Make sure inactive hot physical replication slots don't hold xmin (#3148)

Since `3.2.0` Patroni is able to create physical replication slots on replica nodes just for the case if this node at some moment will become the primary.
There are two potential problems of having such slots:
1. They prevent recycling of WAL files.
2. They may affect vacuum on the primary is hot_standby_feedback is enabled.

The first class of issues is already addressed by periodically calling pg_replication_slot_advance() function.
However the second class of issues doesn't happen instantly, but only when the old primary switched to a replica. In this case physical replication slots that were at some moment activate will hold NOT NULL value of `xmin`, which will be propagated to the primary via hot_standby_feedback mechanism.

To address the second problem we will detect that a physical replication slot is not supposed to be active, but having NOT NULL `xmin` and drop/crecreate it.

Close #3146
Close #3153

Co-authored-by: Polina Bungina <[email protected]>

features/permanent_slots.feature

@@ -81,3 +81,7 @@ Feature: permanent slots
     Then postgres1 has a physical replication slot named test_physical after 10 seconds
     And postgres1 has a physical replication slot named postgres0 after 10 seconds
     And postgres1 has a physical replication slot named postgres3 after 10 seconds
+    When I start postgres0
+    Then postgres0 role is the replica after 20 seconds
+    And physical replication slot named postgres1 on postgres0 has no xmin value after 10 seconds
+    And physical replication slot named postgres2 on postgres0 has no xmin value after 10 seconds

features/steps/slots.py

@@ -94,6 +94,26 @@ def has_physical_replication_slot(context, pg_name, slot_name, time_limit):
     assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds"
 
 
+@step('physical replication slot named {slot_name} on {pg_name:w} has no xmin value after {time_limit:d} seconds')
+def physical_slot_no_xmin(context, pg_name, slot_name, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    query = "SELECT xmin FROM pg_catalog.pg_replication_slots WHERE slot_type = 'physical'"
+    f" AND slot_name = '{slot_name}'"
+    exists = False
+    while time.time() < max_time:
+        try:
+            row = context.pctl.query(pg_name, query).fetchone()
+            exists = bool(row)
+            if exists and row[0] is None:
+                return
+        except Exception:
+            pass
+        time.sleep(1)
+    assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds" if not exists \
+        else f"Physical slot {slot_name} has xmin value after {time_limit} seconds"
+
+
 @step('"{name}" key in DCS has {subkey:w} in {key:w}')
 def dcs_key_contains(context, name, subkey, key):
     response = json.loads(context.dcs_ctl.query(name))