Reword confirmation message

Implement Failover.is_failover/switchover properties
Refactor manual failover checks
2026-08-26 23:50:23 +00:00 · 2023-09-12 15:53:42 +02:00 · 2023-09-12 14:02:17 +02:00 · 2023-09-12 13:13:17 +02:00 · 2023-09-12 08:51:17 +02:00 · 2023-09-11 19:25:01 +02:00
42 changed files with 1002 additions and 490 deletions
@@ -33,7 +33,7 @@ nosetests.xml
 coverage.xml
 htmlcov
 junit.xml
-features/output
+features/output*
 dummy

 # Translations
@@ -48,10 +48,12 @@ pgpass
 scm-source.json

 # Sphinx-generated documentation
+docs/_build/
 docs/build/
 docs/source/_static/
 docs/source/_templates/
 docs/modules/
+docs/pdf/

 # Pycharm IDE
 .idea/
@@ -115,7 +115,7 @@ Kubernetes
 -  **PATRONI\_KUBERNETES\_ROLE\_LABEL**: (optional) name of the label containing role (master or replica or other custom value). Patroni will set this label on the pod it runs in. Default value is ``role``.
 -  **PATRONI\_KUBERNETES\_LEADER\_LABEL\_VALUE**: (optional) value of the pod label when Postgres role is `master`. Default value is `master`.
 -  **PATRONI\_KUBERNETES\_FOLLOWER\_LABEL\_VALUE**: (optional) value of the pod label when Postgres role is `replica`. Default value is `replica`.
-  **PATRONI\_KUBERNETES\_STANDBY\_LEADER\_LABEL\_VALUE**: (optional) value of the pod label when Postgres role is ``standby-leader``. Default value is ``standby-leader``.
+-  **PATRONI\_KUBERNETES\_STANDBY\_LEADER\_LABEL\_VALUE**: (optional) value of the pod label when Postgres role is ``standby_leader``. Default value is ``master``.
 -  **PATRONI\_KUBERNETES\_TMP\_ROLE\_LABEL**: (optional) name of the temporary label containing role (master or replica). Value of this label will always use the default of corresponding role. Set only when necessary.
 -  **PATRONI\_KUBERNETES\_USE\_ENDPOINTS**: (optional) if set to true, Patroni will use Endpoints instead of ConfigMaps to run leader elections and keep cluster state.
 -  **PATRONI\_KUBERNETES\_POD\_IP**: (optional) IP address of the pod Patroni is running in. This value is required when `PATRONI_KUBERNETES_USE_ENDPOINTS` is enabled and is used to populate the leader endpoint subsets when the pod's PostgreSQL is promoted.
@@ -25,82 +25,7 @@ We report new releases information :ref:`here <releases>`.
 Technical Requirements/Installation
 -----------------------------------

-**Pre-requirements for Mac OS**
-
-To install requirements on a Mac, run the following:
-
-::
-
-    brew install postgresql etcd haproxy libyaml python
-
-.. _psycopg2_install_options:
-
-**Psycopg**
-
-Starting from `psycopg2-2.8 <http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/>`__ the binary version of psycopg2 will no longer be installed by default. Installing it from the source code requires C compiler and postgres+python dev packages.
-Since in the python world it is not possible to specify dependency as ``psycopg2 OR psycopg2-binary`` you will have to decide how to install it.
-
-There are a few options available:
-
-1. Use the package manager from your distro
-
-::
-
-    sudo apt-get install python3-psycopg2  # install psycopg2 module on Debian/Ubuntu
-    sudo yum install python3-psycopg2      # install psycopg2 on RedHat/Fedora/CentOS
-
-2. Install psycopg2 from the binary package
-
-::
-
-    pip install psycopg2-binary
-
-3. Install psycopg2 from source
-
-::
-
-    pip install psycopg2>=2.5.4
-
-4. Use psycopg 3.0 instead of psycopg2
-
-::
-
-    pip install psycopg[binary]>=3.0.0
-
-**General installation for pip**
-
-Patroni can be installed with pip:
-
-::
-
-    pip install patroni[dependencies]
-
-where dependencies can be either empty, or consist of one or more of the following:
-
-etcd or etcd3
-    `python-etcd` module in order to use Etcd as Distributed Configuration Store (DCS)
-consul
-    `python-consul` module in order to use Consul as DCS
-zookeeper
-    `kazoo` module in order to use Zookeeper as DCS
-exhibitor
-    `kazoo` module in order to use Exhibitor as DCS (same dependencies as for Zookeeper)
-kubernetes
-    `kubernetes` module in order to use Kubernetes as DCS in Patroni
-raft
-    `pysyncobj` module in order to use python Raft implementation as DCS
-aws
-    `boto3` in order to use AWS callbacks
-
-For example, the command in order to install Patroni together with dependencies for Etcd as a DCS and AWS callbacks is:
-
-::
-
-    pip install patroni[etcd,aws]
-
-Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed
-independently of Patroni.
-
+Go :ref:`here <installation>` for guidance on installing and upgrading Patroni on various platforms.

 .. _running_configuring:

@@ -140,7 +140,7 @@ An example of ``patronictl switchover`` on the worker cluster::
    | work2-1 | 172.27.0.5 | Sync Standby | running |  1 |         0 |
    | work2-2 | 172.27.0.7 | Leader       | running |  1 |           |
    +---------+------------+--------------+---------+----+-----------+
-    Are you sure you want to switchover cluster demo, demoting current primary work2-2? [y/N]: y
+    Are you sure you want to perform a switchover in the cluster demo, demoting current primary work2-2? [y/N]: y
    2022-12-22 07:02:40.33003 Successfully switched over to "work2-1"
    + Citus cluster: demo (group: 2, 7179854924063375386) ------+
    | Member  | Host       | Role    | State   | TL | Lag in MB |
@@ -54,6 +54,13 @@ apidoc_output_dir = 'modules'
 apidoc_excluded_paths = excludes
 apidoc_separate_modules = True

+# Include autodoc for all members, including private ones and the ones that are missing a docstring.
+autodoc_default_options = {
+    "members": True,
+    "undoc-members": True,
+    "private-members": True,
+}
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']

@@ -280,6 +287,14 @@ def doctree_read(app, doctree):
                toc_tree_node['entries'].remove(e)


+def autodoc_skip(app, what, name, obj, would_skip, options):
+    """Include autodoc of ``__init__`` methods, which are skipped by default."""
+    if name == "__init__":
+        return False
+    return would_skip
+
+
+
 # A possibility to have an own stylesheet, to add new rules or override existing ones
 # For the latter case, the CSS specificity of the rules should be higher than the default ones
 def setup(app):
@@ -292,3 +307,4 @@ def setup(app):
    app.connect('builder-inited', builder_inited)
    app.connect('env-get-outdated', env_get_outdated)
    app.connect('doctree-read', doctree_read)
+    app.connect("autodoc-skip-member", autodoc_skip)
@@ -22,6 +22,7 @@ Currently supported PostgreSQL versions: 9.3 to 15.
   :caption: Contents:

   README
+   installation
   patroni_configuration
   rest_api
   replica_bootstrap
@@ -0,0 +1,201 @@
+.. _installation:
+
+Installation
+============
+
+Pre-requirements for Mac OS
+---------------------------
+
+To install requirements on a Mac, run the following:
+
+.. code-block:: shell
+
+    brew install postgresql etcd haproxy libyaml python
+
+.. _psycopg2_install_options:
+
+Psycopg
+-------
+
+Starting from `psycopg2-2.8`_ the binary version of psycopg2 will no longer be installed by default. Installing it from
+the source code requires C compiler and postgres+python dev packages. Since in the python world it is not possible to
+specify dependency as ``psycopg2 OR psycopg2-binary`` you will have to decide how to install it.
+
+There are a few options available:
+
+1. Use the package manager from your distro
+
+.. code-block:: shell
+
+    sudo apt-get install python3-psycopg2  # install psycopg2 module on Debian/Ubuntu
+    sudo yum install python3-psycopg2      # install psycopg2 on RedHat/Fedora/CentOS
+
+2. Install psycopg2 from the binary package
+
+.. code-block:: shell
+
+    pip install psycopg2-binary
+
+3. Install psycopg2 from source
+
+.. code-block:: shell
+
+    pip install psycopg2>=2.5.4
+
+4. Use psycopg 3.0 instead of psycopg2
+
+.. code-block:: shell
+
+    pip install psycopg[binary]>=3.0.0
+
+General installation for pip
+----------------------------
+
+Patroni can be installed with pip:
+
+.. code-block:: shell
+
+    pip install patroni[dependencies]
+
+where ``dependencies`` can be either empty, or consist of one or more of the following:
+
+etcd or etcd3
+    `python-etcd` module in order to use Etcd as Distributed Configuration Store (DCS)
+consul
+    `python-consul` module in order to use Consul as DCS
+zookeeper
+    `kazoo` module in order to use Zookeeper as DCS
+exhibitor
+    `kazoo` module in order to use Exhibitor as DCS (same dependencies as for Zookeeper)
+kubernetes
+    `kubernetes` module in order to use Kubernetes as DCS in Patroni
+raft
+    `pysyncobj` module in order to use python Raft implementation as DCS
+aws
+    `boto3` in order to use AWS callbacks
+
+For example, the command in order to install Patroni together with dependencies for Etcd as a DCS and AWS callbacks is:
+
+.. code-block:: shell
+
+    pip install patroni[etcd,aws]
+
+Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed
+independently of Patroni.
+
+.. _package_installation:
+
+Package installation on Linux
+-----------------------------
+
+Patroni packages may be available for your operating system, produced by the Postgres community for:
+
+* RHEL, RockyLinux, AlmaLinux;
+* Debian and Ubuntu;
+* SUSE Enterprise Linux.
+
+You can also find packages for direct dependencies of Patroni, like python modules that might not be available in
+the official operating system repositories.
+
+For more information see the `PGDG repository`_ documentation.
+
+If you are on a RedHat Enterprise Linux derivative operating system you may also require packages from EPEL, see
+`EPEL repository`_ documentation.
+
+Once you have installed the PGDG repository for your OS you can install patroni.
+
+.. note::
+
+    Patroni packages are not maintained by the Patroni developers, but rather by the Postgres community. If you
+    require support please first try connecting on `Postgres slack`_.
+
+Installing on Debian derivatives
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+With PGDG repo installed, see :ref:`above <package_installation>`, install Patroni via apt run:
+
+.. code-block:: shell
+
+    apt-get install patroni
+
+Installing on RedHat derivatives
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+With PGDG repo installed, see :ref:`above <package_installation>`, install patroni with an etcd DCS via dnf on RHEL 9
+(and derivatives) run:
+
+.. code-block:: shell
+
+    dnf install patroni patroni-etcd
+
+You can install etcd from PGDG if your RedHat derivative distribution does not provide packages. On the nodes that will
+host the DCS run:
+
+.. code-block:: shell
+
+    dnf install 'dnf-command(config-manager)'
+    dnf config-manager --enable pgdg-rhel9-extras
+    dnf install etcd
+
+You can replace the version of RHEL with `8` in the repo to make `pgdg-rhel8-extras` if needed. The repo name is still
+`pgdg-rhelN-extras` on RockyLinux, AlmaLinux, Oracle Linux, etc...
+
+Installing on SUSE Enterprise Linux
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You might need to enable the SUSE PackageHub repositories for some dependencies. see `SUSE PackageHub`_ documentation.
+
+For SLES 15 with PGDG repo installed, see :ref:`above <package_installation>`, you can install patroni using:
+
+.. code-block:: shell
+
+    zypper install patroni patroni-etcd
+
+With the SUSE PackageHub repo enabled you can also install etcd:
+
+.. code-block:: shell
+
+    SUSEConnect -p PackageHub/15.5/x86_64
+    zypper install etcd
+
+Upgrading
+---------
+
+Upgrading patroni is a very simple process, just update the software installation and restart the Patroni daemon on
+each node in the cluster.
+
+However, restarting the Patroni daemon will result in a Postgres database restart. In some situations this may cause
+a failover of the primary node in your cluster, therefore it is recommended to put the cluster into maintenance mode
+until the Patroni daemon restart has been completed.
+
+To put the cluster in maintenance mode, run the following command on one of the patroni nodes:
+
+.. code-block:: shell
+
+    patronictl pause --wait
+
+Then on each node in the cluster, perform the package upgrade required for your OS:
+
+.. code-block:: shell
+
+    apt-get update && apt-get install patroni patroni-etcd
+
+Restart the patroni daemon process on each node:
+
+.. code-block:: shell
+
+    systemctl restart patroni
+
+Then finally resume monitoring of Postgres with patroni to take it out of maintenance mode:
+
+.. code-block:: shell
+
+    patronictl resume --wait
+
+The cluster will now be full operational with the new version of Patroni.
+
+.. _psycopg2-2.8: http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/
+.. _PGDG repository: https://www.postgresql.org/download/linux/
+.. _EPEL repository: https://docs.fedoraproject.org/en-US/epel/
+.. _SUSE PackageHub: https://packagehub.suse.com/how-to-use/
+.. _Postgres slack: http://pgtreats.info/slack-invite
@@ -90,6 +90,42 @@ The parameters would be applied in the following order (run-time are given the h
 This allows configuration for all the nodes (2), configuration for a specific node using ``ALTER SYSTEM`` (3) and ensures that parameters essential to the running of Patroni are enforced (4), as well as leaves room for configuration tools that manage `postgresql.conf` directly without involving Patroni (1).


+PostgreSQL parameters that touch shared memory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+PostgreSQL has some parameters that determine the size of the shared memory used by them:
+
+- **max_connections**
+- **max_prepared_transactions**
+- **max_locks_per_transaction**
+- **max_wal_senders**
+- **max_worker_processes**
+
+Changing these parameters require a PostgreSQL restart to take effect, and their shared memory structures cannot be smaller on the standby nodes than on the primary node.
+
+As explained before, Patroni restrict changing their values through :ref:`dynamic configuration <dynamic_configuration>`, which usually consists of:
+
+1. Applying changes through ``patronictl edit-config`` (or via REST API ``/config`` endpoint)
+2. Restarting nodes through ``patronictl restart`` (or via REST API ``/restart`` endpoint)
+
+**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through ``patronictl restart`` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
+
+However, as those settings manage shared memory, some extra care should be taken when restarting the nodes:
+
+* If you want to **increase** the value of any of those settings:
+
+   1. Restart all standbys first
+   2. Restart the primary after that
+
+* If you want to **decrease** the value of any of those settings:
+
+   1. Restart the primary first
+   2. Restart all standbys after that
+
+**Note:** if you attempt to restart all nodes in one go after **decreasing** the value of any of those settings, Patroni will ignore the change and restart the standby with the original setting value, thus requiring that you restart the standbys again later. Patroni does that to prevent the standby to enter in an infinite crash loop, because PostgreSQL quits with a `FATAL` message if you attempt to set any of those parameters to a value lower than what is visible in ``pg_controldata`` on the Standby node. In other words, we can only decrease the setting on the standby once its ``pg_controldata`` is up-to-date with the primary in regards to these changes on the primary.
+
+More information about that can be found at `PostgreSQL Administrator's Overview <https://www.postgresql.org/docs/current/hot-standby.html#HOT-STANDBY-ADMIN>`__.
+
 Patroni configuration parameters
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@@ -19,7 +19,7 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,

 - For the Postgres primary with the leader lock Patroni updates the lock. If the node with the leader lock stops being the primary (i.e. is demoted manually), Patroni will release the lock instead of promoting the node back.

- Manual unscheduled restart, manual unscheduled failover/switchover and reinitialize are allowed. No scheduled action is allowed. Manual switchover is only allowed if the node to switchover to is specified.
+- Manual unscheduled restart, manual unscheduled failover/switchover and reinitialize are allowed. No scheduled action is allowed. Manual switchover is only allowed if the node to switch over to is specified.

 - If 'parallel' primaries are detected by Patroni, it emits a warning, but does not demote the primary without the leader lock.

@@ -131,7 +131,8 @@ The ``GET /patroni`` is used by Patroni during the leader race. It also could be
      "database_system_identifier": "7268616322854375442",
      "patroni": {
        "version": "3.1.0",
-        "scope": "demo"
+        "scope": "demo",
+        "name": "patroni1"
      }
    }

@@ -178,7 +179,8 @@ The ``GET /patroni`` is used by Patroni during the leader race. It also could be
      "database_system_identifier": "7268616322854375442",
      "patroni": {
        "version": "3.1.0",
-        "scope": "demo"
+        "scope": "demo",
+        "name": "patroni1"
      }
    }

@@ -223,7 +225,8 @@ The ``GET /patroni`` is used by Patroni during the leader race. It also could be
      "database_system_identifier": "7268616322854375442",
      "patroni": {
        "version": "3.1.0",
-        "scope": "demo"
+        "scope": "demo",
+        "name": "patroni1"
      }
    }

@@ -267,7 +270,8 @@ The ``GET /patroni`` is used by Patroni during the leader race. It also could be
      "database_system_identifier": "7268616322854375442",
      "patroni": {
        "version": "3.1.0",
-        "scope": "demo"
+        "scope": "demo",
+        "name": "patroni1"
      }
    }

@@ -279,70 +283,70 @@ Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` e
 	
 	# HELP patroni_version Patroni semver without periods. \
 	# TYPE patroni_version gauge
-	patroni_version{scope="batman"} 020103
+	patroni_version{scope="batman",name="patroni1"} 020103
 	# HELP patroni_postgres_running Value is 1 if Postgres is running, 0 otherwise.
 	# TYPE patroni_postgres_running gauge
-	patroni_postgres_running{scope="batman"} 1
+	patroni_postgres_running{scope="batman",name="patroni1"} 1
 	# HELP patroni_postmaster_start_time Epoch seconds since Postgres started.
 	# TYPE patroni_postmaster_start_time gauge
-	patroni_postmaster_start_time{scope="batman"} 1657656955.179243
+	patroni_postmaster_start_time{scope="batman",name="patroni1"} 1657656955.179243
 	# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.
 	# TYPE patroni_master gauge
-	patroni_master{scope="batman"} 1
+	patroni_master{scope="batman",name="patroni1"} 1
 	# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.
 	# TYPE patroni_primary gauge
-	patroni_primary{scope="batman"} 1
+	patroni_primary{scope="batman",name="patroni1"} 1
 	# HELP patroni_xlog_location Current location of the Postgres transaction log, 0 if this node is not the leader.
 	# TYPE patroni_xlog_location counter
-	patroni_xlog_location{scope="batman"} 22320573386952
+	patroni_xlog_location{scope="batman",name="patroni1"} 22320573386952
 	# HELP patroni_standby_leader Value is 1 if this node is the standby_leader, 0 otherwise.
 	# TYPE patroni_standby_leader gauge
-	patroni_standby_leader{scope="batman"} 0
+	patroni_standby_leader{scope="batman",name="patroni1"} 0
 	# HELP patroni_replica Value is 1 if this node is a replica, 0 otherwise.
 	# TYPE patroni_replica gauge
-	patroni_replica{scope="batman"} 0
+	patroni_replica{scope="batman",name="patroni1"} 0
 	# HELP patroni_sync_standby Value is 1 if this node is a sync standby replica, 0 otherwise.
 	# TYPE patroni_sync_standby gauge
-	patroni_sync_standby{scope="batman"} 0
+	patroni_sync_standby{scope="batman",name="patroni1"} 0
 	# HELP patroni_xlog_received_location Current location of the received Postgres transaction log, 0 if this node is not a replica.
 	# TYPE patroni_xlog_received_location counter
-	patroni_xlog_received_location{scope="batman"} 0
+	patroni_xlog_received_location{scope="batman",name="patroni1"} 0
 	# HELP patroni_xlog_replayed_location Current location of the replayed Postgres transaction log, 0 if this node is not a replica.
 	# TYPE patroni_xlog_replayed_location counter
-	patroni_xlog_replayed_location{scope="batman"} 0
+	patroni_xlog_replayed_location{scope="batman",name="patroni1"} 0
 	# HELP patroni_xlog_replayed_timestamp Current timestamp of the replayed Postgres transaction log, 0 if null.
 	# TYPE patroni_xlog_replayed_timestamp gauge
-	patroni_xlog_replayed_timestamp{scope="batman"} 0
+	patroni_xlog_replayed_timestamp{scope="batman",name="patroni1"} 0
 	# HELP patroni_xlog_paused Value is 1 if the Postgres xlog is paused, 0 otherwise.
 	# TYPE patroni_xlog_paused gauge
-	patroni_xlog_paused{scope="batman"} 0
+	patroni_xlog_paused{scope="batman",name="patroni1"} 0
 	# HELP patroni_postgres_streaming Value is 1 if Postgres is streaming, 0 otherwise.
 	# TYPE patroni_postgres_streaming gauge
-	patroni_postgres_streaming{scope="batman"} 1
+	patroni_postgres_streaming{scope="batman",name="patroni1"} 1
 	# HELP patroni_postgres_in_archive_recovery Value is 1 if Postgres is replicating from archive, 0 otherwise.
 	# TYPE patroni_postgres_in_archive_recovery gauge
-	patroni_postgres_in_archive_recovery{scope="batman"} 0
+	patroni_postgres_in_archive_recovery{scope="batman",name="patroni1"} 0
 	# HELP patroni_postgres_server_version Version of Postgres (if running), 0 otherwise.
 	# TYPE patroni_postgres_server_version gauge
-	patroni_postgres_server_version {scope="batman"} 140004
+	patroni_postgres_server_version{scope="batman",name="patroni1"} 140004
 	# HELP patroni_cluster_unlocked Value is 1 if the cluster is unlocked, 0 if locked.
 	# TYPE patroni_cluster_unlocked gauge
-	patroni_cluster_unlocked{scope="batman"} 0
+	patroni_cluster_unlocked{scope="batman",name="patroni1"} 0
 	# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
 	# TYPE patroni_postgres_timeline counter
-	patroni_failsafe_mode_is_active{scope="batman"} 0
+	patroni_failsafe_mode_is_active{scope="batman",name="patroni1"} 0
 	# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
 	# TYPE patroni_postgres_timeline counter
-	patroni_postgres_timeline{scope="batman"} 24
+	patroni_postgres_timeline{scope="batman",name="patroni1"} 24
 	# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully by Patroni.
 	# TYPE patroni_dcs_last_seen gauge
-	patroni_dcs_last_seen{scope="batman"} 1677658321
+	patroni_dcs_last_seen{scope="batman",name="patroni1"} 1677658321
 	# HELP patroni_pending_restart Value is 1 if the node needs a restart, 0 otherwise.
 	# TYPE patroni_pending_restart gauge
-	patroni_pending_restart{scope="batman"} 1
+	patroni_pending_restart{scope="batman",name="patroni1"} 1
 	# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.
 	# TYPE patroni_is_paused gauge
-	patroni_is_paused{scope="batman"} 1
+	patroni_is_paused{scope="batman",name="patroni1"} 1


 Cluster status endpoints
@@ -381,6 +385,7 @@ Cluster status endpoints
          "lag": 0
        }
      ],
+      "scope": "demo",
      "scheduled_switchover": {
        "at": "2023-09-24T10:36:00+02:00",
        "from": "patroni1",
@@ -489,8 +494,9 @@ Let's check that the node processed this configuration. First of all it should s
 	    "location": 2197818976
 	  },
 	  "patroni": {
+	    "version": "1.0",
 	    "scope": "batman",
-	    "version": "1.0"
+	    "name": "patroni1"
 	  },
 	  "state": "running",
 	  "role": "master",
@@ -559,11 +565,11 @@ Switchover and failover endpoints
 Switchover
 ^^^^^^^^^^

-``/switchover`` endpoint only works when cluster is healthy (there is a leader). It allows to schedule a switchover at a given time.
+``/switchover`` endpoint only works when the cluster is healthy (there is a leader). It also allows to schedule a switchover at a given time.

-When calling ``/switchover`` endpoint candidate can be specified but is not required, in contrast to ``/failover`` endpoint. If candidate is not provided, all the healthy nodes that are allowed to failover participate in the leader race.
+When calling ``/switchover`` endpoint a candidate can be specified but is not required, in contrast to ``/failover`` endpoint. If a candidate is not provided, all the eligible nodes of the cluster will participate in the leader race after the leader stepped down.

-In the JSON body of the ``POST`` request, you must specify at least the ``leader`` field and, optionally, the ``candidate`` and ``scheduled_at`` field if you want to schedule a switchover at a specific time.
+In the JSON body of the ``POST`` request you must specify the ``leader`` field. The ``candidate`` and the ``scheduled_at`` fields are optional and can be used to schedule a switchover at a specific time.

 Depending on the situation, requests might return different HTTP status codes and bodies. Status code **200** is returned when the switchover or failover successfully completed. If the switchover was successfully scheduled, Patroni will return HTTP status code **202**. In case something went wrong, the error status code (one of **400**, **412**, or **503**) will be returned with some details in the response body.

@@ -600,7 +606,7 @@ Failover

 ``/failover`` endpoint can be used to perform a manual failover when there are no healthy nodes (e.g. to an asynchronous standby if all synchronous standbys are not healthy enough to promote). However there is no requirement for a cluster not to have leader - failover can also be run on a healthy cluster.

-In the JSON body of the ``POST`` request you must specify ``candidate`` field. If ``leader`` field is specified, switchover is triggered. 
+In the JSON body of the ``POST`` request you must specify the ``candidate`` field. If the ``leader`` field is specified, a switchover is triggered instead.

 **Example:**

@@ -610,7 +616,7 @@ In the JSON body of the ``POST`` request you must specify ``candidate`` field. I
 	Successfully failed over to "postgresql1"

 .. warning::
-	:ref:`Be very careful <failover_healthcheck>` using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs. 
+	:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs. 


 ``POST /switchover`` and ``POST /failover`` endpoints are used by ``patronictl switchover`` and ``patronictl failover``, respectively.
@@ -644,21 +650,21 @@ Healthy standby

 There are a couple of checks that a member of a cluster should pass to be able to participate in the leader race during a switchover or to become a leader as a failover/switchover candidate:

- be reachable via Patroni API,
- not have ``nofailover`` tag set to ``true``,
- have watchdog fully functional (if required by the configuration),
- in case of a switchover or a failover in a healthy cluster, not exceed maximum replication lag (``maximum_lag_on_failover`` :ref:`configuration parameter <dynamic_configuration>`),
- in case of a switchover or a failover in a healthy cluster, not have a timeline number smaller than the cluster timeline,
+- be reachable via Patroni API;
+- not have ``nofailover`` tag set to ``true``;
+- have watchdog fully functional (if required by the configuration);
+- in case of a switchover in a healthy cluster or an automatic failover, not exceed maximum replication lag (``maximum_lag_on_failover`` :ref:`configuration parameter <dynamic_configuration>`);
+- in case of a switchover in a healthy cluster or an automatic failover, not have a timeline number smaller than the cluster timeline if ``check_timeline`` :ref:`configuration parameter <dynamic_configuration>` is set to ``true``;
 - in :ref:`synchronous mode <synchronous_mode>`:

-  - In case of a switchover (both with and without a candidate): be listed in the ``/sync`` key members.
+  - In case of a switchover (both with and without a candidate): be listed in the ``/sync`` key members;
  - For a failover in both healthy and unhealthy clusters, this check is omitted.

 .. warning::
-    In case of a failover in a cluster without a leader, a candidate will be allowed to promote even if:
-	- it is not in the ``/sync`` key members when synchronous mode is enabled,
-	- its lag exceeds the maximum replication lag allowed,
-	- it has the timeline number smaller than the cluster timeline.
+    In case of a manual failover in a cluster without a leader, a candidate will be allowed to promote even if:
+	- it is not in the ``/sync`` key members when synchronous mode is enabled;
+	- its lag exceeds the maximum replication lag allowed;
+	- it has the timeline number smaller than the last known cluster timeline.


 Restart endpoint
@@ -171,7 +171,7 @@ Kubernetes
 -  **role\_label**: (optional) name of the label containing role (master or replica or other custom value). Patroni will set this label on the pod it runs in. Default value is ``role``.
 -  **leader\_label\_value**: (optional) value of the pod label when Postgres role is ``master``. Default value is ``master``.
 -  **follower\_label\_value**: (optional) value of the pod label when Postgres role is ``replica``. Default value is ``replica``.
-  **standby\_leader\_label\_value**: (optional) value of the pod label when Postgres role is ``standby-leader``. Default value is ``standby-leader``.
+-  **standby\_leader\_label\_value**: (optional) value of the pod label when Postgres role is ``standby_leader``. Default value is ``master``.
 -  **tmp_\role\_label**: (optional) name of the temporary label containing role (master or replica). Value of this label will always use the default of corresponding role. Set only when necessary.
 -  **use\_endpoints**: (optional) if set to true, Patroni will use Endpoints instead of ConfigMaps to run leader elections and keep cluster state.
 -  **pod\_ip**: (optional) IP address of the pod Patroni is running in. This value is required when `use_endpoints` is enabled and is used to populate the leader endpoint subsets when the pod's PostgreSQL is promoted.
@@ -27,8 +27,8 @@ RUN set -ex \
    && apt-get update \
    && apt-get reinstall init-system-helpers \
    && apt-get install -y \
-      python3-pip \
      python3-dev \
+      python3-venv \
      rsync \
      curl \
      gcc \
@@ -40,7 +40,9 @@ RUN set -ex \
      net-tools \
      iputils-ping \
    && rm -rf /var/cache/apt \
-    && python3 -m pip install --no-cache-dir tox \
+    \
+    && python3 -m venv /tox \
+    && /tox/bin/pip install --no-cache-dir tox>=4 \
    \
    && mkdir -p "$PGHOME" \
    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
@@ -50,6 +52,7 @@ RUN set -ex \
    && curl -sL "$ETCDURL/etcd-v$ETCDVERSION-linux-$(dpkg --print-architecture).tar.gz" \
       | tar xz -C /usr/local/bin --strip=1 --wildcards --no-anchored etcd etcdctl

+ENV PATH="/tox/bin:$PATH"

 # This Dockerfile syntax only works with docker buildx and the syntax
 # line at the top of this file.
@@ -13,6 +13,7 @@ from argparse import Namespace
 from typing import Any, Dict, Optional, TYPE_CHECKING

 from patroni.daemon import AbstractPatroniDaemon, abstract_main, get_base_arg_parser
+from patroni.tags import Tags

 if TYPE_CHECKING:  # pragma: no cover
    from .config import Config
@@ -20,7 +21,7 @@ if TYPE_CHECKING:  # pragma: no cover
 logger = logging.getLogger(__name__)


-class Patroni(AbstractPatroniDaemon):
+class Patroni(AbstractPatroniDaemon, Tags):
    """Implement ``patroni`` command daemon.

    :ivar version: Patroni version.
@@ -30,7 +31,6 @@ class Patroni(AbstractPatroniDaemon):
    :ivar api: REST API server instance of this node.
    :ivar request: wrapper for performing HTTP requests.
    :ivar ha: HA handler.
-    :ivar tags: cache of custom tags configured for this node.
    :ivar next_run: time when to run the next HA loop cycle.
    :ivar scheduled_restart: when a restart has been scheduled to occur, if any. In that case, should contain two keys:
        * ``schedule``: timestamp when restart should occur;
@@ -71,7 +71,7 @@ class Patroni(AbstractPatroniDaemon):
        self.api = RestApiServer(self, self.config['restapi'])
        self.ha = Ha(self)

-        self.tags = self.get_tags()
+        self._tags = self._get_tags()
        self.next_run = time.time()
        self.scheduled_restart: Dict[str, Any] = {}

@@ -121,33 +121,12 @@ class Patroni(AbstractPatroniDaemon):
        except Exception:
            return

-    def get_tags(self) -> Dict[str, Any]:
+    def _get_tags(self) -> Dict[str, Any]:
        """Get tags configured for this node, if any.

-        Handle both predefined Patroni tags and custom defined tags.
-
-        .. note::
-            A custom tag is any tag added to the configuration ``tags`` section that is not one of ``clonefrom``,
-            ``nofailover``, ``noloadbalance`` or ``nosync``.
-
-            For the Patroni predefined tags, the returning object will only contain them if they are enabled as they
-            all are boolean values that default to disabled.
-
-        :returns: a dictionary of tags set for this node. The key is the tag name, and the value is the corresponding
-            tag value.
+        :returns: a dictionary of tags set for this node.
        """
-        return {tag: value for tag, value in self.config.get('tags', {}).items()
-                if tag not in ('clonefrom', 'nofailover', 'noloadbalance', 'nosync') or value}
-
-    @property
-    def nofailover(self) -> bool:
-        """``True`` if ``tags.nofailover`` configuration is enabled for this node, else ``False``."""
-        return bool(self.tags.get('nofailover', False))
-
-    @property
-    def nosync(self) -> bool:
-        """``True`` if ``tags.nosync`` configuration is enabled for this node, else ``False``."""
-        return bool(self.tags.get('nosync', False))
+        return self._filter_tags(self.config.get('tags', {}))

    def reload_config(self, sighup: bool = False, local: Optional[bool] = False) -> None:
        """Apply new configuration values for ``patroni`` daemon.
@@ -166,7 +145,7 @@ class Patroni(AbstractPatroniDaemon):
        try:
            super(Patroni, self).reload_config(sighup, local)
            if local:
-                self.tags = self.get_tags()
+                self._tags = self._get_tags()
                self.request.reload_config(self.config)
            if local or sighup and self.api.reload_local_certificate():
                self.api.reload_config(self.config['restapi'])
@@ -177,14 +156,9 @@ class Patroni(AbstractPatroniDaemon):
            logger.exception('Failed to reload config_file=%s', self.config.config_file)

    @property
-    def replicatefrom(self) -> Optional[str]:
-        """Value of ``tags.replicatefrom`` configuration, if any."""
-        return self.tags.get('replicatefrom')
-
-    @property
-    def noloadbalance(self) -> bool:
-        """``True`` if ``tags.noloadbalance`` configuration is enabled for this node, else ``False``."""
-        return bool(self.tags.get('noloadbalance', False))
+    def tags(self) -> Dict[str, Any]:
+        """Tags configured for this node, if any."""
+        return self._tags

    def schedule_next_run(self) -> None:
        """Schedule the next run of the ``patroni`` daemon main loop.
@@ -197,7 +197,11 @@ class RestApiHandler(BaseHTTPRequestHandler):
            response['database_system_identifier'] = patroni.postgresql.sysid
        if patroni.postgresql.pending_restart:
            response['pending_restart'] = True
-        response['patroni'] = {'version': patroni.version, 'scope': patroni.postgresql.scope}
+        response['patroni'] = {
+            'version': patroni.version,
+            'scope': patroni.postgresql.scope,
+            'name': patroni.postgresql.name
+        }
        if patroni.scheduled_restart:
            response['scheduled_restart'] = patroni.scheduled_restart.copy()
            del response['scheduled_restart']['postmaster_start_time']
@@ -448,7 +452,10 @@ class RestApiHandler(BaseHTTPRequestHandler):
        """
        cluster = self.server.patroni.dcs.get_cluster(True)
        global_config = self.server.patroni.config.get_global_config(cluster)
-        self._write_json_response(200, cluster_as_json(cluster, global_config))
+
+        response = cluster_as_json(cluster, global_config)
+        response['scope'] = self.server.patroni.postgresql.scope
+        self._write_json_response(200, response)

    def do_GET_history(self) -> None:
        """Handle a ``GET`` request to ``/history`` path.
@@ -525,113 +532,113 @@ class RestApiHandler(BaseHTTPRequestHandler):

        metrics: List[str] = []

-        scope_label = '{{scope="{0}"}}'.format(patroni.postgresql.scope)
+        labels = f'{{scope="{patroni.postgresql.scope}",name="{patroni.postgresql.name}"}}'
        metrics.append("# HELP patroni_version Patroni semver without periods.")
        metrics.append("# TYPE patroni_version gauge")
        padded_semver = ''.join([x.zfill(2) for x in patroni.version.split('.')])  # 2.0.2 => 020002
-        metrics.append("patroni_version{0} {1}".format(scope_label, padded_semver))
+        metrics.append("patroni_version{0} {1}".format(labels, padded_semver))

        metrics.append("# HELP patroni_postgres_running Value is 1 if Postgres is running, 0 otherwise.")
        metrics.append("# TYPE patroni_postgres_running gauge")
-        metrics.append("patroni_postgres_running{0} {1}".format(scope_label, int(postgres['state'] == 'running')))
+        metrics.append("patroni_postgres_running{0} {1}".format(labels, int(postgres['state'] == 'running')))

        metrics.append("# HELP patroni_postmaster_start_time Epoch seconds since Postgres started.")
        metrics.append("# TYPE patroni_postmaster_start_time gauge")
        postmaster_start_time = postgres.get('postmaster_start_time')
        postmaster_start_time = (postmaster_start_time - epoch).total_seconds() if postmaster_start_time else 0
-        metrics.append("patroni_postmaster_start_time{0} {1}".format(scope_label, postmaster_start_time))
+        metrics.append("patroni_postmaster_start_time{0} {1}".format(labels, postmaster_start_time))

        metrics.append("# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.")
        metrics.append("# TYPE patroni_master gauge")
-        metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
+        metrics.append("patroni_master{0} {1}".format(labels, int(postgres['role'] in ('master', 'primary'))))

        metrics.append("# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.")
        metrics.append("# TYPE patroni_primary gauge")
-        metrics.append("patroni_primary{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
+        metrics.append("patroni_primary{0} {1}".format(labels, int(postgres['role'] in ('master', 'primary'))))

        metrics.append("# HELP patroni_xlog_location Current location of the Postgres"
                       " transaction log, 0 if this node is not the leader.")
        metrics.append("# TYPE patroni_xlog_location counter")
-        metrics.append("patroni_xlog_location{0} {1}".format(scope_label, postgres.get('xlog', {}).get('location', 0)))
+        metrics.append("patroni_xlog_location{0} {1}".format(labels, postgres.get('xlog', {}).get('location', 0)))

        metrics.append("# HELP patroni_standby_leader Value is 1 if this node is the standby_leader, 0 otherwise.")
        metrics.append("# TYPE patroni_standby_leader gauge")
-        metrics.append("patroni_standby_leader{0} {1}".format(scope_label, int(postgres['role'] == 'standby_leader')))
+        metrics.append("patroni_standby_leader{0} {1}".format(labels, int(postgres['role'] == 'standby_leader')))

        metrics.append("# HELP patroni_replica Value is 1 if this node is a replica, 0 otherwise.")
        metrics.append("# TYPE patroni_replica gauge")
-        metrics.append("patroni_replica{0} {1}".format(scope_label, int(postgres['role'] == 'replica')))
+        metrics.append("patroni_replica{0} {1}".format(labels, int(postgres['role'] == 'replica')))

        metrics.append("# HELP patroni_sync_standby Value is 1 if this node is a sync standby replica, 0 otherwise.")
        metrics.append("# TYPE patroni_sync_standby gauge")
-        metrics.append("patroni_sync_standby{0} {1}".format(scope_label, int(postgres.get('sync_standby', False))))
+        metrics.append("patroni_sync_standby{0} {1}".format(labels, int(postgres.get('sync_standby', False))))

        metrics.append("# HELP patroni_xlog_received_location Current location of the received"
                       " Postgres transaction log, 0 if this node is not a replica.")
        metrics.append("# TYPE patroni_xlog_received_location counter")
        metrics.append("patroni_xlog_received_location{0} {1}"
-                       .format(scope_label, postgres.get('xlog', {}).get('received_location', 0)))
+                       .format(labels, postgres.get('xlog', {}).get('received_location', 0)))

        metrics.append("# HELP patroni_xlog_replayed_location Current location of the replayed"
                       " Postgres transaction log, 0 if this node is not a replica.")
        metrics.append("# TYPE patroni_xlog_replayed_location counter")
        metrics.append("patroni_xlog_replayed_location{0} {1}"
-                       .format(scope_label, postgres.get('xlog', {}).get('replayed_location', 0)))
+                       .format(labels, postgres.get('xlog', {}).get('replayed_location', 0)))

        metrics.append("# HELP patroni_xlog_replayed_timestamp Current timestamp of the replayed"
                       " Postgres transaction log, 0 if null.")
        metrics.append("# TYPE patroni_xlog_replayed_timestamp gauge")
        replayed_timestamp = postgres.get('xlog', {}).get('replayed_timestamp')
        replayed_timestamp = (replayed_timestamp - epoch).total_seconds() if replayed_timestamp else 0
-        metrics.append("patroni_xlog_replayed_timestamp{0} {1}".format(scope_label, replayed_timestamp))
+        metrics.append("patroni_xlog_replayed_timestamp{0} {1}".format(labels, replayed_timestamp))

        metrics.append("# HELP patroni_xlog_paused Value is 1 if the Postgres xlog is paused, 0 otherwise.")
        metrics.append("# TYPE patroni_xlog_paused gauge")
        metrics.append("patroni_xlog_paused{0} {1}"
-                       .format(scope_label, int(postgres.get('xlog', {}).get('paused', False) is True)))
+                       .format(labels, int(postgres.get('xlog', {}).get('paused', False) is True)))

        if postgres.get('server_version', 0) >= 90600:
            metrics.append("# HELP patroni_postgres_streaming Value is 1 if Postgres is streaming, 0 otherwise.")
            metrics.append("# TYPE patroni_postgres_streaming gauge")
            metrics.append("patroni_postgres_streaming{0} {1}"
-                           .format(scope_label, int(postgres.get('replication_state') == 'streaming')))
+                           .format(labels, int(postgres.get('replication_state') == 'streaming')))

            metrics.append("# HELP patroni_postgres_in_archive_recovery Value is 1"
                           " if Postgres is replicating from archive, 0 otherwise.")
            metrics.append("# TYPE patroni_postgres_in_archive_recovery gauge")
            metrics.append("patroni_postgres_in_archive_recovery{0} {1}"
-                           .format(scope_label, int(postgres.get('replication_state') == 'in archive recovery')))
+                           .format(labels, int(postgres.get('replication_state') == 'in archive recovery')))

        metrics.append("# HELP patroni_postgres_server_version Version of Postgres (if running), 0 otherwise.")
        metrics.append("# TYPE patroni_postgres_server_version gauge")
-        metrics.append("patroni_postgres_server_version {0} {1}".format(scope_label, postgres.get('server_version', 0)))
+        metrics.append("patroni_postgres_server_version {0} {1}".format(labels, postgres.get('server_version', 0)))

        metrics.append("# HELP patroni_cluster_unlocked Value is 1 if the cluster is unlocked, 0 if locked.")
        metrics.append("# TYPE patroni_cluster_unlocked gauge")
-        metrics.append("patroni_cluster_unlocked{0} {1}".format(scope_label, int(postgres.get('cluster_unlocked', 0))))
+        metrics.append("patroni_cluster_unlocked{0} {1}".format(labels, int(postgres.get('cluster_unlocked', 0))))

        metrics.append("# HELP patroni_failsafe_mode_is_active Value is 1 if failsafe mode is active, 0 if inactive.")
        metrics.append("# TYPE patroni_failsafe_mode_is_active gauge")
        metrics.append("patroni_failsafe_mode_is_active{0} {1}"
-                       .format(scope_label, int(postgres.get('failsafe_mode_is_active', 0))))
+                       .format(labels, int(postgres.get('failsafe_mode_is_active', 0))))

        metrics.append("# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.")
        metrics.append("# TYPE patroni_postgres_timeline counter")
-        metrics.append("patroni_postgres_timeline{0} {1}".format(scope_label, postgres.get('timeline', 0)))
+        metrics.append("patroni_postgres_timeline{0} {1}".format(labels, postgres.get('timeline', 0)))

        metrics.append("# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully"
                       " by Patroni.")
        metrics.append("# TYPE patroni_dcs_last_seen gauge")
-        metrics.append("patroni_dcs_last_seen{0} {1}".format(scope_label, postgres.get('dcs_last_seen', 0)))
+        metrics.append("patroni_dcs_last_seen{0} {1}".format(labels, postgres.get('dcs_last_seen', 0)))

        metrics.append("# HELP patroni_pending_restart Value is 1 if the node needs a restart, 0 otherwise.")
        metrics.append("# TYPE patroni_pending_restart gauge")
        metrics.append("patroni_pending_restart{0} {1}"
-                       .format(scope_label, int(patroni.postgresql.pending_restart)))
+                       .format(labels, int(patroni.postgresql.pending_restart)))

        metrics.append("# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.")
        metrics.append("# TYPE patroni_is_paused gauge")
-        metrics.append("patroni_is_paused{0} {1}".format(scope_label, int(postgres.get('pause', 0))))
+        metrics.append("patroni_is_paused{0} {1}".format(labels, int(postgres.get('pause', 0))))

        self.write_response(200, '\n'.join(metrics) + '\n', content_type='text/plain')

@@ -995,7 +1002,8 @@ class RestApiHandler(BaseHTTPRequestHandler):
            * ``412``: if operation is not possible;
            * ``503``: if unable to register the operation to the DCS;
            * HTTP status returned by :func:`parse_schedule`, if any error was observed while parsing the schedule;
-            * HTTP status returned by :func:`poll_failover_result` if the operation has been processed immediately.
+            * HTTP status returned by :func:`poll_failover_result` if the operation has been processed immediately;
+            * ``400``: if none of the above applies.

        .. note::
            If unable to parse the request body, then the request is silently discarded.
@@ -1282,6 +1290,9 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
    def query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
        """Execute *sql* query with *params* and optionally return results.

+        .. note::
+            Prefer to use own connection to postgres and fallback to ``heartbeat`` when own isn't available.
+
        :param sql: the SQL statement to be run.
        :param params: positional arguments to be used as parameters for *sql*.

@@ -1291,10 +1302,21 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
            :class:`psycopg.Error`: if had issues while executing *sql*.
            :class:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database.
        """
+        # We first try to get a heartbeat connection because it is always required for the main thread.
        try:
-            return self.patroni.postgresql.query(sql, *params, retry=False)
-        except RetryFailedError as e:
-            raise PostgresConnectionException(str(e))
+            heartbeat_connection = self.patroni.postgresql.connection_pool.get('heartbeat')
+            heartbeat_connection.get()  # try to open psycopg connection to postgres
+        except psycopg.Error as exc:
+            raise PostgresConnectionException('connection problems') from exc
+
+        try:
+            connection = self.patroni.postgresql.connection_pool.get('restapi')
+            connection.get()  # try to open psycopg connection to postgres
+        except psycopg.Error:
+            logger.debug('restapi connection to postgres is not available')
+            connection = heartbeat_connection
+
+        return connection.query(sql, *params)

    @staticmethod
    def _set_fd_cloexec(fd: socket.socket) -> None:
@@ -1460,7 +1482,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
        :param listen: IP and port to bind REST API to. It should be a string in the format ``host:port``, where
            ``host`` can be a hostname or IP address. It is the value of ``restapi.listen`` setting.
        :param ssl_options: dictionary that may contain the following keys, depending on what has been configured in
-            ``restapi` section:
+            ``restapi`` section:

            * ``certfile``: path to PEM certificate. If given, will start in HTTPS mode;
            * ``keyfile``: path to key of ``certfile``;
@@ -257,7 +257,7 @@ class Config(object):

              * file or directory path passed as command-line argument (*configfile*), if it exists and the file or
                files found in the directory can be parsed (see :meth:`~Config._load_config_path`), otherwise
-              * YAML file passed via the environment variable (see :cvar:`PATRONI_CONFIG_VARIABLE`), if the referenced
+              * YAML file passed via the environment variable (see :attr:`PATRONI_CONFIG_VARIABLE`), if the referenced
                file exists and can be parsed, otherwise
              * from configuration values defined as environment variables, see
                :meth:`~Config._build_environment_configuration`.
@@ -262,7 +262,9 @@ role_choice = click.Choice(['leader', 'primary', 'standby-leader', 'replica', 's
@click.option('-k', '--insecure', is_flag=True, help='Allow connections to SSL sites without certs')
@click.pass_context
 def ctl(ctx: click.Context, config_file: str, dcs_url: Optional[str], insecure: bool) -> None:
-    """Entry point of ``patronictl`` utility.
+    """Command-line interface for interacting with Patroni.
+    \f
+    Entry point of ``patronictl`` utility.

    Load the configuration file.

@@ -1140,8 +1142,8 @@ def reinit(obj: Dict[str, Any], cluster_name: str, group: Optional[int],


 def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: str,
-                               group: Optional[int], candidate: Optional[str], force: bool,
-                               leader: Optional[str] = None, scheduled: Optional[str] = None) -> None:
+                               group: Optional[int], leader: Optional[str], candidate: Optional[str],
+                               force: bool, scheduled: Optional[str] = None) -> None:
    """Perform a failover or a switchover operation in the cluster.

    Informational messages are printed in the console during the operation, as well as the list of members before and
@@ -1155,9 +1157,9 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
    :param cluster_name: name of the Patroni cluster.
    :param group: filter Citus group within we should perform a failover or switchover. If ``None``, user will be
        prompted for filling it -- unless *force* is ``True``, in which case an exception is raised.
+    :param leader: name of the current leader member.
    :param candidate: name of a standby member to be promoted. Nodes that are tagged with ``nofailover`` cannot be used.
    :param force: perform the failover or switchover without asking for confirmations.
-    :param leader: name of the leader passed to the switchover command if any.
    :param scheduled: timestamp when the switchover should be scheduled to occur. If ``now`` perform immediately.

    :raises:
@@ -1247,11 +1249,12 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
        demote_msg = f', demoting current leader {cluster.leader.name}' if cluster.leader else ''
        if scheduled_at_str:
            # only switchover can be scheduled
-            if not click.confirm(f'Are you sure you want to schedule switchover of cluster'
+            if not click.confirm(f'Are you sure you want to schedule a switchover in the cluster '
                                 f'{cluster_name} at {scheduled_at_str}{demote_msg}?'):
+                # action as a var to catch a regression in the tests
                raise PatroniCtlException('Aborting scheduled ' + action)
        else:
-            if not click.confirm(f'Are you sure you want to {action} cluster {cluster_name}{demote_msg}?'):
+            if not click.confirm(f'Are you sure you want to perform a {action} in the cluster {cluster_name}{demote_msg}?'):
                raise PatroniCtlException('Aborting ' + action)

    # And finally the actual work
@@ -1294,15 +1297,21 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
@ctl.command('failover', help='Failover to a replica')
@arg_cluster_name
@option_citus_group
+@click.option('--leader', '--primary', '--master', 'leader', help='The name of the current leader', default=None)
@click.option('--candidate', help='The name of the candidate', default=None)
@option_force
@click.pass_obj
 def failover(obj: Dict[str, Any], cluster_name: str, group: Optional[int],
-             candidate: Optional[str], force: bool) -> None:
+             leader: Optional[str], candidate: Optional[str], force: bool) -> None:
    """Process ``failover`` command of ``patronictl`` utility.

    Perform a failover operation immediately in the cluster.

+    .. note::
+        If *leader* is given perform a switchover instead of a failover.
+        This behavior is deprecated. ``--leader`` option support will be
+        removed in the next major release.
+
    .. seealso::
        Refer to :func:`_do_failover_or_switchover` for details.

@@ -1311,10 +1320,17 @@ def failover(obj: Dict[str, Any], cluster_name: str, group: Optional[int],
    :param group: filter Citus group within we should perform a failover or switchover. If ``None``, user will be
        prompted for filling it -- unless *force* is ``True``, in which case an exception is raised by
        :func:`_do_failover_or_switchover`.
+    :param leader: name of the current leader member.
    :param candidate: name of a standby member to be promoted. Nodes that are tagged with ``nofailover`` cannot be used.
    :param force: perform the failover or switchover without asking for confirmations.
    """
-    _do_failover_or_switchover(obj, 'failover', cluster_name, group, candidate, force)
+    action = 'failover'
+    if leader:
+        action = 'switchover'
+        click.echo(click.style(
+            'Supplying a leader name using this command is deprecated and will be removed in a future version of'
+            ' Patroni, change your scripts to use `switchover` instead.\nExecuting switchover!', fg='red'))
+    _do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force)


@ctl.command('switchover', help='Switchover to a replica')
@@ -1345,7 +1361,7 @@ def switchover(obj: Dict[str, Any], cluster_name: str, group: Optional[int],
    :param force: perform the switchover without asking for confirmations.
    :param scheduled: timestamp when the switchover should be scheduled to occur. If ``now`` perform immediately.
    """
-    _do_failover_or_switchover(obj, 'switchover', cluster_name, group, candidate, force, leader, scheduled)
+    _do_failover_or_switchover(obj, 'switchover', cluster_name, group, leader, candidate, force, scheduled)


 def generate_topology(level: int, member: Dict[str, Any],
@@ -1531,9 +1547,11 @@ def output_members(obj: Dict[str, Any], cluster: Cluster, name: str,
            rows.append([member.get(n.lower().replace(' ', '_'), '') for n in columns])

    title = 'Citus cluster' if is_citus_cluster else 'Cluster'
-    group_title = '' if group is None else 'group: {0}, '.format(group)
-    title_details = group_title and ' ({0}{1})'.format(group_title, initialize)
-    title = ' {0}: {1}{2} '.format(title, name, title_details)
+    title_details = f' ({initialize})'
+    if is_citus_cluster:
+        title_details = '' if group is None else f' (group: {group}, {initialize})'
+
+    title = f' {title}: {name}{title_details} '
    print_output(columns, rows, {'Group': 'r', 'Lag in MB': 'r', 'TL': 'r'}, fmt, title)

    if fmt not in ('pretty', 'topology'):  # Omit service info when using machine-readable formats
@@ -23,6 +23,7 @@ import dateutil.parser

 from ..exceptions import PatroniFatalException
 from ..utils import deep_compare, uri
+from ..tags import Tags

 if TYPE_CHECKING:  # pragma: no cover
    from ..config import Config
@@ -191,11 +192,11 @@ _Version = Union[int, str]
 _Session = Union[int, float, str, None]


-class Member(NamedTuple('Member',
-                        [('version', _Version),
-                         ('name', str),
-                         ('session', _Session),
-                         ('data', Dict[str, Any])])):
+class Member(Tags, NamedTuple('Member',
+                              [('version', _Version),
+                               ('name', str),
+                               ('session', _Session),
+                               ('data', Dict[str, Any])])):
    """Immutable object (namedtuple) which represents single member of PostgreSQL cluster.

    .. note::
@@ -316,20 +317,10 @@ class Member(NamedTuple('Member',
        """The ``tags`` value from :attr:`~Member.data` if defined, otherwise an empty dictionary."""
        return self.data.get('tags', {})

-    @property
-    def nofailover(self) -> bool:
-        """The value for ``nofailover`` in :attr:`Member`.tags`` if defined, otherwise ``False``."""
-        return self.tags.get('nofailover', False)
-
-    @property
-    def replicatefrom(self) -> Optional[str]:
-        """The value for ``replicatefrom`` in :attr:`Member`.tags`` if defined."""
-        return self.tags.get('replicatefrom')
-
    @property
    def clonefrom(self) -> bool:
        """``True`` if both ``clonefrom`` tag is ``True`` and a connection URL is defined."""
-        return self.tags.get('clonefrom', False) and bool(self.conn_url)
+        return super().clonefrom and bool(self.conn_url)

    @property
    def state(self) -> str:
@@ -460,7 +451,7 @@ class Leader(NamedTuple):


 class Failover(NamedTuple):
-    """Immutable object (namedtuple) representing configuration information required for failover/switchover capability.
+    """Immutable object (namedtuple) which represents failover key.

    :ivar version: version of the object.
    :ivar leader: name of the leader. If value isn't empty we treat it as a switchover from the specified node.
@@ -556,6 +547,13 @@ class Failover(NamedTuple):
        """
        return int(bool(self.leader)) + int(bool(self.candidate))

+    @property
+    def is_switchover(self) -> bool:
+        return bool(self.leader)
+
+    @property
+    def is_failover(self) -> bool:
+        return not self.is_switchover

 class ClusterConfig(NamedTuple):
    """Immutable object (namedtuple) which represents cluster configuration.
@@ -912,8 +910,16 @@ class Cluster(NamedTuple('Cluster',

    @property
    def __permanent_slots(self) -> Dict[str, Union[Dict[str, Any], Any]]:
-        """Dictionary of permanent replication slots."""
-        return self.config and self.config.permanent_slots or {}
+        """Dictionary of permanent replication slots with their known LSN."""
+        ret = deepcopy(self.config.permanent_slots if self.config else {})
+        # If primary reported flush LSN for permanent slots we want to enrich our structure with it
+        for name, lsn in (self.slots or {}).items():
+            if name in ret:
+                if not ret[name]:
+                    ret[name] = {}
+                if isinstance(ret[name], dict):
+                    ret[name]['lsn'] = lsn
+        return ret

    @property
    def __permanent_physical_slots(self) -> Dict[str, Any]:
@@ -938,7 +944,6 @@ class Cluster(NamedTuple('Cluster',

        Will log an error if:

-            * Conflicting slot names between members are found
            * Any logical slots are disabled, due to version compatibility, and *show_error* is ``True``.

        :param my_name: name of this node.
@@ -951,21 +956,9 @@ class Cluster(NamedTuple('Cluster',

        :returns: final dictionary of slot names, after merging with permanent slots and performing sanity checks.
        """
-        slot_members: List[str] = self._get_slot_members(my_name, role)
-
-        slots: Dict[str, Dict[str, str]] = {slot_name_from_member_name(name): {'type': 'physical'}
-                                            for name in slot_members}
-
-        if len(slots) < len(slot_members):
-            # Find which names are conflicting for a nicer error message
-            slot_conflicts: Dict[str, List[str]] = defaultdict(list)
-            for name in slot_members:
-                slot_conflicts[slot_name_from_member_name(name)].append(name)
-            logger.error("Following cluster members share a replication slot name: %s",
-                         "; ".join(f"{', '.join(v)} map to {k}"
-                                   for k, v in slot_conflicts.items() if len(v) > 1))
-
+        slots: Dict[str, Dict[str, str]] = self._get_members_slots(my_name, role)
        permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster, role, nofailover)
+
        disabled_permanent_logical_slots: List[str] = self._merge_permanent_slots(
            slots, permanent_slots, my_name, major_version)

@@ -1025,7 +1018,7 @@ class Cluster(NamedTuple('Cluster',
        return disabled_permanent_logical_slots

    def _get_permanent_slots(self, is_standby_cluster: bool, role: str, nofailover: bool) -> Dict[str, Any]:
-        """Get configured permanent slot names.
+        """Get configured permanent replication slots.

        .. note::
            Permanent replication slots are only considered if ``use_slots`` configuration is enabled.
@@ -1051,35 +1044,48 @@ class Cluster(NamedTuple('Cluster',

        return self.__permanent_slots if role in ('master', 'primary') else self.__permanent_logical_slots

-    def _get_slot_members(self, my_name: str, role: str) -> List[str]:
-        """Get a list of member names that have replication slots sourcing from this node.
+    def _get_members_slots(self, my_name: str, role: str) -> Dict[str, Dict[str, str]]:
+        """Get physical replication slots configuration for members that sourcing from this node.

        If the ``replicatefrom`` tag is set on the member - we should not create the replication slot for it on
        the current primary, because that member would replicate from elsewhere. We still create the slot if
        the ``replicatefrom`` destination member is currently not a member of the cluster (fallback to the
        primary), or if ``replicatefrom`` destination member happens to be the current primary.

+        Will log an error if:
+
+            * Conflicting slot names between members are found
+
        :param my_name: name of this node.
        :param role: role of this node, if this is a ``primary`` or ``standby_leader`` return list of members
                     replicating from this node. If not then return a list of members replicating as cascaded
                     replicas from this node.

-        :returns: list of member names.
+        :returns: dictionary of physical replication slots that should exist on a given node.
        """
        if not self.use_slots:
-            return []
+            return {}
+
+        # we always want to exclude the member with our name from the list
+        members = filter(lambda m: m.name != my_name, self.members)

        if role in ('master', 'primary', 'standby_leader'):
-            slot_members = [m.name for m in self.members
-                            if m.name != my_name
-                            and (m.replicatefrom is None
-                                 or m.replicatefrom == my_name
-                                 or not self.has_member(m.replicatefrom))]
+            members = [m for m in members if m.replicatefrom is None
+                       or m.replicatefrom == my_name or not self.has_member(m.replicatefrom)]
        else:
            # only manage slots for replicas that replicate from this one, except for the leader among them
-            slot_members = [m.name for m in self.members
-                            if m.replicatefrom == my_name and m.name != self.leader_name]
-        return slot_members
+            members = [m for m in members if m.replicatefrom == my_name and m.name != self.leader_name]
+
+        slots = {slot_name_from_member_name(m.name): {'type': 'physical'} for m in members}
+        if len(slots) < len(members):
+            # Find which names are conflicting for a nicer error message
+            slot_conflicts: Dict[str, List[str]] = defaultdict(list)
+            for member in members:
+                slot_conflicts[slot_name_from_member_name(member.name)].append(member.name)
+            logger.error("Following cluster members share a replication slot name: %s",
+                         "; ".join(f"{', '.join(v)} map to {k}"
+                                   for k, v in slot_conflicts.items() if len(v) > 1))
+        return slots

    def has_permanent_logical_slots(self, my_name: str, nofailover: bool, major_version: int = 110000) -> bool:
        """Check if the given member node has permanent ``logical`` replication slots configured.
@@ -141,6 +141,36 @@ class HTTPClient(object):
 class ConsulClient(base.Consul):

    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        """
+        Consul client with Patroni customisations.
+
+        .. note::
+
+            Parameters, *token*, *cert* and *ca_cert* are not passed to the parent class :class:`consul.base.Consul`.
+
+        Original class documentation,
+
+            *token* is an optional ``ACL token``. If supplied it will be used by
+            default for all requests made with this client session. It's still
+            possible to override this token by passing a token explicitly for a
+            request.
+
+            *consistency* sets the consistency mode to use by default for all reads
+            that support the consistency option. It's still possible to override
+            this by passing explicitly for a given request. *consistency* can be
+            either 'default', 'consistent' or 'stale'.
+
+            *dc* is the datacenter that this agent will communicate with.
+            By default, the datacenter of the host is used.
+
+            *verify* is whether to verify the SSL certificate for HTTPS requests
+
+            *cert* client side certificates for HTTPS requests
+
+        :param args: positional arguments to pass to :class:`consul.base.Consul`
+        :param kwargs: keyword arguments, with *cert*, *ca_cert* and *token* removed, passed to
+                       :class:`consul.base.Consul`
+        """
        self._cert = kwargs.pop('cert', None)
        self._ca_cert = kwargs.pop('ca_cert', None)
        self.token = kwargs.get('token')
@@ -756,7 +756,7 @@ class Kubernetes(AbstractDCS):
        self._role_label = config.get('role_label', 'role')
        self._leader_label_value = config.get('leader_label_value', 'master')
        self._follower_label_value = config.get('follower_label_value', 'replica')
-        self._standby_leader_label_value = config.get('standby_leader_label_value', 'standby-leader')
+        self._standby_leader_label_value = config.get('standby_leader_label_value', 'master')
        self._tmp_role_label = config.get('tmp_role_label')
        self._ca_certs = os.environ.get('PATRONI_KUBERNETES_CACERT', config.get('cacert')) or SERVICE_CERT_FILENAME
        super(Kubernetes, self).__init__({**config, 'namespace': ''})
@@ -1140,6 +1140,13 @@ class Kubernetes(AbstractDCS):
        """Unused"""
        raise NotImplementedError  # pragma: no cover

+    def write_leader_optime(self, last_lsn: int) -> None:
+        """Write value for WAL LSN to ``optime`` annotation of the leader object.
+
+        :param last_lsn: absolute WAL LSN in bytes.
+        """
+        self.patch_or_create(self.leader_path, {self._OPTIME: str(last_lsn)}, patch=True, retry=False)
+
    def _update_leader_with_retry(self, annotations: Dict[str, Any],
                                  resource_version: Optional[str], ips: List[str]) -> bool:
        retry = self._retry.copy()
@@ -1269,13 +1276,10 @@ class Kubernetes(AbstractDCS):
    def touch_member(self, data: Dict[str, Any]) -> bool:
        cluster = self.cluster
        if cluster and cluster.leader and cluster.leader.name == self._name:
-            role = self._leader_label_value
+            role = self._standby_leader_label_value if data['role'] == 'standby_leader' else self._leader_label_value
            tmp_role = 'master'
        elif data['state'] == 'running' and data['role'] not in ('master', 'primary'):
-            role = {
-                'replica': self._follower_label_value,
-                'standby-leader': self._standby_leader_label_value,
-            }.get(data['role'], data['role'])
+            role = {'replica': self._follower_label_value}.get(data['role'], data['role'])
            tmp_role = data['role']
        else:
            role = None
@@ -20,31 +20,28 @@ from .postgresql.callback_executor import CallbackAction
 from .postgresql.misc import postgres_version_to_int
 from .postgresql.postmaster import PostmasterProcess
 from .postgresql.rewind import Rewind
+from .tags import Tags
 from .utils import polling_loop, tzutc

 logger = logging.getLogger(__name__)


-class _MemberStatus(NamedTuple):
-    """Node status distilled from API response:
+class _MemberStatus(Tags, NamedTuple('_MemberStatus',
+                                     [('member', Member),
+                                      ('reachable', bool),
+                                      ('in_recovery', Optional[bool]),
+                                      ('wal_position', int),
+                                      ('data', Dict[str, Any])])):
+    """Node status distilled from API response.

-        member - dcs.Member object of the node
-        reachable - `!False` if the node is not reachable or is not responding with correct JSON
-        in_recovery - `!True` if pg_is_in_recovery() == true
-        dcs_last_seen - timestamp from JSON of last succesful communication with DCS
-        timeline - timeline value from JSON
-        wal_position - maximum value of `replayed_location` or `received_location` from JSON
-        tags - dictionary with values of different tags (i.e. nofailover)
-        watchdog_failed - indicates that watchdog is required by configuration but not available or failed
+    Consists of the following fields:
+
+    :ivar member: :class:`~patroni.dcs.Member` object of the node.
+    :ivar reachable: ``False`` if the node is not reachable or is not responding with correct JSON.
+    :ivar in_recovery: ``False`` if the node is running as a primary (`if pg_is_in_recovery() == true`).
+    :ivar wal_position: maximum value of ``replayed_location`` or ``received_location`` from JSON.
+    :ivar data: the whole JSON response for future usage.
    """
-    member: Member
-    reachable: bool
-    in_recovery: Optional[bool]
-    dcs_last_seen: int
-    timeline: int
-    wal_position: int
-    tags: Dict[str, Any]
-    watchdog_failed: bool

    @classmethod
    def from_api_response(cls, member: Member, json: Dict[str, Any]) -> '_MemberStatus':
@@ -57,21 +54,34 @@ class _MemberStatus(NamedTuple):
        wal: Dict[str, Any] = json.get('wal') or json['xlog']
        # abuse difference in primary/replica response format
        in_recovery = not (bool(wal.get('location')) or json.get('role') in ('master', 'primary'))
-        timeline = json.get('timeline', 0)
-        dcs_last_seen = json.get('dcs_last_seen', 0)
        lsn = int(in_recovery and max(wal.get('received_location', 0), wal.get('replayed_location', 0)))
-        return cls(member, True, in_recovery, dcs_last_seen, timeline, lsn,
-                   json.get('tags', {}), json.get('watchdog_failed', False))
+        return cls(member, True, in_recovery, lsn, json)
+
+    @property
+    def tags(self) -> Dict[str, Any]:
+        """Dictionary with values of different tags (i.e. nofailover)."""
+        return self.data.get('tags', {})
+
+    @property
+    def timeline(self) -> int:
+        """Timeline value from JSON."""
+        return self.data.get('timeline', 0)
+
+    @property
+    def watchdog_failed(self) -> bool:
+        """Indicates that watchdog is required by configuration but not available or failed."""
+        return self.data.get('watchdog_failed', False)

    @classmethod
    def unknown(cls, member: Member) -> '_MemberStatus':
-        return cls(member, False, None, 0, 0, 0, {}, False)
+        """Create a new class instance with empty or null values."""
+        return cls(member, False, None, 0, {})

    def failover_limitation(self) -> Optional[str]:
        """Returns reason why this node can't promote or None if everything is ok."""
        if not self.reachable:
            return 'not reachable'
-        if self.tags.get('nofailover', False):
+        if self.nofailover:
            return 'not allowed to promote'
        if self.watchdog_failed:
            return 'not watchdog capable'
@@ -224,7 +234,7 @@ class Ha(object):
        """
        if not self.cluster.failover:
            return 'failover'
-        return 'switchover' if self.cluster.failover.leader else 'manual failover'
+        return 'switchover' if self.cluster.failover.is_switchover else 'manual failover'

    def load_cluster_from_dcs(self) -> None:
        cluster = self.dcs.get_cluster()
@@ -1033,8 +1043,8 @@ class Ha(object):
        elif self.is_paused():
            return False

-        # try to pick some other members to switchover and check that they are healthy
-        if failover.leader:
+        # try to pick some other members for switchover and check that they are healthy
+        if failover.is_switchover:
            if self.state_handler.name == failover.leader:  # I was the leader
                # exclude desired member which is unhealthy if it was specified
                if self.is_failover_possible(exclude_failover_candidate=bool(failover.candidate)):
@@ -1091,7 +1101,7 @@ class Ha(object):

        if self.cluster.failover:
            # When doing a switchover in synchronous mode only synchronous nodes and former leader are allowed to race
-            if self.cluster.failover.leader and self.sync_mode_is_active() \
+            if self.cluster.failover.is_switchover and self.sync_mode_is_active() \
                    and not self.cluster.sync.matches(self.state_handler.name, True):
                return False
            return self.manual_failover_process_no_leader() or False
@@ -1265,7 +1275,7 @@ class Ha(object):
        action = self._get_failover_action_name()
        bare_action = action.replace('manual ', '')

-        # it is not the time for the the scheduled failover yet, do nothing
+        # it is not the time for the scheduled switchover yet, do nothing
        if (failover.scheduled_at and not
            self.should_run_scheduled_action(bare_action, failover.scheduled_at, lambda:
                                             self.dcs.manual_failover('', '', version=failover.version))):
@@ -2012,7 +2022,7 @@ class Ha(object):
        def is_eligible(node: Member) -> bool:
            # in synchronous mode we allow failover (not switchover!) to async node
            if self.sync_mode_is_active() and not self.cluster.sync.matches(node.name)\
-                    and not (failover and not failover.leader):
+                    and not (failover and failover.is_failover):
                return False
            # Don't spend time on "nofailover" nodes checking.
            # We also don't need nodes which we can't query with the api in the list.
@@ -18,7 +18,7 @@ from .bootstrap import Bootstrap
 from .callback_executor import CallbackAction, CallbackExecutor
 from .cancellable import CancellableSubprocess
 from .config import ConfigHandler, mtime
-from .connection import Connection, get_connection_cursor
+from .connection import ConnectionPool, get_connection_cursor
 from .citus import CitusHandler
 from .misc import parse_history, parse_lsn, postgres_major_version_to_int
 from .postmaster import PostmasterProcess
@@ -79,7 +79,8 @@ class Postgresql(object):
        self.set_state('stopped')

        self._pending_restart = False
-        self._connection = Connection()
+        self.connection_pool = ConnectionPool()
+        self._connection = self.connection_pool.get('heartbeat')
        self.citus_handler = CitusHandler(self, config.get('citus'))
        self.config = ConfigHandler(self, config)
        self.config.check_directories()
@@ -277,7 +278,7 @@ class Postgresql(object):

        :returns: 'ok' if PostgreSQL is up, 'reject' if starting up, 'no_resopnse' if not up."""

-        r = self.config.local_connect_kwargs
+        r = self.connection_pool.conn_kwargs
        cmd = [self.pgcommand('pg_isready'), '-p', r['port'], '-d', self._database]

        # Host is not set if we are connecting via default unix socket
@@ -328,10 +329,6 @@ class Postgresql(object):
    def connection(self) -> Union['connection3', 'Connection3[Any]']:
        return self._connection.get()

-    def set_connection_kwargs(self, kwargs: Dict[str, Any]) -> None:
-        self._connection.set_conn_kwargs(kwargs.copy())
-        self.citus_handler.set_conn_kwargs(kwargs.copy())
-
    def _query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
        """Execute *sql* query with *params* and optionally return results.

@@ -694,7 +691,7 @@ class Postgresql(object):
        # the former node, otherwise, we might get a stalled one
        # after kill -9, which would report incorrect data to
        # patroni.
-        self._connection.close()
+        self.connection_pool.close()

        if self.is_running():
            logger.error('Cannot start PostgreSQL because one is already running.')
@@ -765,7 +762,7 @@ class Postgresql(object):
    def checkpoint(self, connect_kwargs: Optional[Dict[str, Any]] = None,
                   timeout: Optional[float] = None) -> Optional[str]:
        check_not_is_in_recovery = connect_kwargs is not None
-        connect_kwargs = connect_kwargs or self.config.local_connect_kwargs
+        connect_kwargs = connect_kwargs or self.connection_pool.conn_kwargs
        for p in ['connect_timeout', 'options']:
            connect_kwargs.pop(p, None)
        if timeout:
@@ -176,7 +176,7 @@ class Bootstrap(object):
        """
        cmd = config.get('post_bootstrap') or config.get('post_init')
        if cmd:
-            r = self._postgresql.config.local_connect_kwargs
+            r = self._postgresql.connection_pool.conn_kwargs
            connstring = self._postgresql.config.format_dsn(r, True)
            if 'host' not in r:
                # https://www.postgresql.org/docs/current/static/libpq-pgpass.html
@@ -6,7 +6,6 @@ from threading import Condition, Event, Thread
 from urllib.parse import urlparse
 from typing import Any, Dict, List, Optional, Union, Tuple, TYPE_CHECKING

-from .connection import Connection
 from ..dcs import CITUS_COORDINATOR_GROUP_ID, Cluster
 from ..psycopg import connect, quote_ident

@@ -71,7 +70,10 @@ class CitusHandler(Thread):
        self.daemon = True
        self._postgresql = postgresql
        self._config = config
-        self._connection = Connection()
+        if config:
+            self._connection = postgresql.connection_pool.get(
+                'citus', {'dbname': config['database'],
+                          'options': '-c statement_timeout=0 -c idle_in_transaction_session_timeout=0'})
        self._pg_dist_node: Dict[int, PgDistNode] = {}  # Cache of pg_dist_node: {groupid: PgDistNode()}
        self._tasks: List[PgDistNode] = []  # Requests to change pg_dist_node, every task is a `PgDistNode`
        self._in_flight: Optional[PgDistNode] = None  # Reference to the `PgDistNode` being changed in a transaction
@@ -91,12 +93,6 @@ class CitusHandler(Thread):
    def is_worker(self) -> bool:
        return self.is_enabled() and not self.is_coordinator()

-    def set_conn_kwargs(self, kwargs: Dict[str, Any]) -> None:
-        if isinstance(self._config, dict):  # self.is_enabled():
-            kwargs.update({'dbname': self._config['database'],
-                           'options': '-c statement_timeout=0 -c idle_in_transaction_session_timeout=0'})
-            self._connection.set_conn_kwargs(kwargs)
-
    def schedule_cache_rebuild(self) -> None:
        with self._condition:
            self._schedule_load_pg_dist_node = True
@@ -359,8 +355,8 @@ class CitusHandler(Thread):
        if not isinstance(self._config, dict):  # self.is_enabled()
            return

-        conn_kwargs = self._postgresql.config.local_connect_kwargs
-        conn_kwargs['options'] = '-c synchronous_commit=local -c statement_timeout=0'
+        conn_kwargs = {**self._postgresql.connection_pool.conn_kwargs,
+                       'options': '-c synchronous_commit=local -c statement_timeout=0'}
        if self._config['database'] != self._postgresql.database:
            conn = connect(**conn_kwargs)
            try:
@@ -14,7 +14,7 @@ from typing import Any, Collection, Dict, Iterator, List, Optional, Union, Tuple
 from .validator import recovery_parameters, transform_postgresql_parameter_value, transform_recovery_parameter_value
 from ..collections import CaseInsensitiveDict, CaseInsensitiveSet
 from ..dcs import Leader, Member, RemoteMember, slot_name_from_member_name
-from ..exceptions import PatroniFatalException
+from ..exceptions import PatroniFatalException, PostgresConnectionException
 from ..file_perm import pg_perm
 from ..utils import compare_values, parse_bool, parse_int, split_host_port, uri, validate_directory, is_subpath
 from ..validator import IntValidator, EnumValidator
@@ -623,7 +623,24 @@ class ConfigHandler(object):
                                          'recovery_target_action', 'standby_mode', self._triggerfile_wrong_name})
        return CaseInsensitiveSet(self._RECOVERY_PARAMETERS - skip_params)

-    def _read_recovery_params(self) -> Tuple[Optional[CaseInsensitiveDict], Optional[bool]]:
+    def _read_recovery_params(self) -> Tuple[Optional[CaseInsensitiveDict], bool]:
+        """Read current recovery parameters values.
+
+        .. note::
+            We query Postgres only if we detected that Postgresql was restarted
+            or when at least one of the following files was updated:
+
+                * ``postgresql.conf``;
+                * ``postgresql.auto.conf``;
+                * ``passfile`` that is used in the ``primary_conninfo``.
+
+        :returns: a tuple with two elements:
+
+            * :class:`CaseInsensitiveDict` object with current values of recovery parameters,
+              or ``None`` if no configuration files were updated;
+
+            * ``True`` if new values of recovery parameters were queried, ``False`` otherwise.
+        """
        if self._postgresql.is_starting():
            return None, False

@@ -644,11 +661,20 @@ class ConfigHandler(object):
            self._postgresql_conf_mtime = pg_conf_mtime
            self._auto_conf_mtime = auto_conf_mtime
            self._postmaster_ctime = postmaster_ctime
-        except Exception:
+        except Exception as exc:
+            if all((isinstance(exc, PostgresConnectionException),
+                    self._postgresql_conf_mtime == pg_conf_mtime,
+                    self._auto_conf_mtime == auto_conf_mtime,
+                    self._passfile_mtime == passfile_mtime,
+                    self._postmaster_ctime != postmaster_ctime)):
+                # We detected that the connection to postgres fails, but the process creation time of the postmaster
+                # doesn't match the old value. It is an indicator that Postgres crashed and either doing crash
+                # recovery or down. In this case we return values like nothing changed in the config.
+                return None, False
            values = None
        return values, True

-    def _read_recovery_params_pre_v12(self) -> Tuple[Optional[CaseInsensitiveDict], Optional[bool]]:
+    def _read_recovery_params_pre_v12(self) -> Tuple[Optional[CaseInsensitiveDict], bool]:
        recovery_conf_mtime = mtime(self._recovery_conf)
        passfile_mtime = mtime(self._passfile) if self._passfile else False
        if recovery_conf_mtime == self._recovery_conf_mtime and passfile_mtime == self._passfile_mtime:
@@ -942,24 +968,32 @@ class ConfigHandler(object):
                return 'localhost'  # connection via localhost is preferred
        return listen_addresses[0].strip()  # can't use localhost, take first address from listen_addresses

-    @property
-    def local_connect_kwargs(self) -> Dict[str, Any]:
-        ret = self._local_address.copy()
-        # add all of the other connection settings that are available
-        ret.update(self._superuser)
-        # if the "username" parameter is present, it actually needs to be "user"
-        # for connecting to PostgreSQL
-        if 'username' in self._superuser:
-            ret['user'] = self._superuser['username']
-            del ret['username']
-        # ensure certain Patroni configurations are available
-        ret.update({'dbname': self._postgresql.database,
-                    'fallback_application_name': 'Patroni',
-                    'connect_timeout': 3,
-                    'options': '-c statement_timeout=2000'})
-        return ret
-
    def resolve_connection_addresses(self) -> None:
+        """Calculates and sets local and remote connection urls and options.
+
+        This method sets:
+            * :attr:`Postgresql.connection_string <patroni.postgresql.Postgresql.connection_string>` attribute, which
+              is later written to the member key in DCS as ``conn_url``.
+            * :attr:`ConfigHandler.local_replication_address` attribute, which is used for replication connections to
+              local postgres.
+            * :attr:`ConnectionPool.conn_kwargs <patroni.postgresql.connection.ConnectionPool.conn_kwargs>` attribute,
+              which is used for superuser connections to local postgres.
+
+        .. note::
+            If there is a valid directory in ``postgresql.parameters.unix_socket_directories`` in the Patroni
+            configuration and ``postgresql.use_unix_socket`` and/or ``postgresql.use_unix_socket_repl``
+            are set to ``True``, we respectively use unix sockets for superuser and replication connections
+            to local postgres.
+
+            If there is a requirement to use unix sockets, but nothing is set in the
+            ``postgresql.parameters.unix_socket_directories``, we omit a ``host`` in connection parameters relying
+            on the ability of ``libpq`` to connect via some default unix socket directory.
+
+            If unix sockets are not requested we "switch" to TCP, prefering to use ``localhost`` if it is possible
+            to deduce that Postgres is listening on a local interface address.
+
+            Otherwise we just used the first address specified in the ``listen_addresses`` GUC.
+        """
        port = self._server_parameters['port']
        tcp_local_address = self._get_tcp_local_address()
        netloc = self._config.get('connect_address') or tcp_local_address + ':' + port
@@ -972,12 +1006,25 @@ class ConfigHandler(object):

        tcp_local_address = {'host': tcp_local_address, 'port': port}

-        self._local_address = unix_local_address if self._config.get('use_unix_socket') else tcp_local_address
        self.local_replication_address = unix_local_address\
            if self._config.get('use_unix_socket_repl') else tcp_local_address

        self._postgresql.connection_string = uri('postgres', netloc, self._postgresql.database)
-        self._postgresql.set_connection_kwargs(self.local_connect_kwargs)
+
+        local_address = unix_local_address if self._config.get('use_unix_socket') else tcp_local_address
+        local_conn_kwargs = {
+            **local_address,
+            **self._superuser,
+            'dbname': self._postgresql.database,
+            'fallback_application_name': 'Patroni',
+            'connect_timeout': 3,
+            'options': '-c statement_timeout=2000'
+        }
+        # if the "username" parameter is present, it actually needs to be "user" for connecting to PostgreSQL
+        if 'username' in local_conn_kwargs:
+            local_conn_kwargs['user'] = local_conn_kwargs.pop('username')
+        # "notify" connection_pool about the "new" local connection address
+        self._postgresql.connection_pool.conn_kwargs = local_conn_kwargs

    def _get_pg_settings(
            self, names: Collection[str]
@@ -2,9 +2,9 @@ import logging

 from contextlib import contextmanager
 from threading import Lock
-from typing import Any, Dict, Iterator, List, Union, Tuple, TYPE_CHECKING
+from typing import Any, Dict, Iterator, List, Optional, Union, Tuple, TYPE_CHECKING
 if TYPE_CHECKING:  # pragma: no cover
-    from psycopg import Connection as Connection3, Cursor
+    from psycopg import Connection, Cursor
    from psycopg2 import connection, cursor

 from .. import psycopg
@@ -13,27 +13,34 @@ from ..exceptions import PostgresConnectionException
 logger = logging.getLogger(__name__)


-class Connection:
-    """Helper class to manage connections from Patroni to PostgreSQL.
+class NamedConnection:
+    """Helper class to manage ``psycopg`` connections from Patroni to PostgreSQL.

    :ivar server_version: PostgreSQL version in integer format where we are connected to.
    """

    server_version: int

-    def __init__(self) -> None:
-        """Create an instance of :class:`Connection` class."""
+    def __init__(self, pool: 'ConnectionPool', name: str, kwargs_override: Optional[Dict[str, Any]]) -> None:
+        """Create an instance of :class:`NamedConnection` class.
+
+        :param pool: reference to a :class:`ConnectionPool` object.
+        :param name: name of the connection.
+        :param kwargs_override: :class:`dict` object with connection parameters that should be
+                                different from default values provided by connection *pool*.
+        """
+        self._pool = pool
+        self._name = name
+        self._kwargs_override = kwargs_override or {}
        self._lock = Lock()  # used to make sure that only one connection to postgres is established
        self._connection = None

-    def set_conn_kwargs(self, conn_kwargs: Dict[str, Any]) -> None:
-        """Set connection parameters, like user, password, host, port and so on.
+    @property
+    def _conn_kwargs(self) -> Dict[str, Any]:
+        """Connection parameters for this :class:`NamedConnection`."""
+        return {**self._pool.conn_kwargs, **self._kwargs_override, 'application_name': f'Patroni {self._name}'}

-        :param conn_kwargs: connection parameters as a dictionary.
-        """
-        self._conn_kwargs = conn_kwargs
-
-    def get(self) -> Union['connection', 'Connection3[Any]']:
+    def get(self) -> Union['connection', 'Connection[Any]']:
        """Get ``psycopg``/``psycopg2`` connection object.

        .. note::
@@ -43,7 +50,7 @@ class Connection:
        """
        with self._lock:
            if not self._connection or self._connection.closed != 0:
-                logger.info("establishing a new patroni connection to postgres")
+                logger.info("establishing a new patroni %s connection to postgres", self._name)
                self._connection = psycopg.connect(**self._conn_kwargs)
                self.server_version = getattr(self._connection, 'server_version', 0)
        return self._connection
@@ -76,12 +83,72 @@ class Connection:
                    raise exc
            raise PostgresConnectionException('connection problems') from exc

-    def close(self) -> None:
-        """Close the psycopg connection to postgres."""
+    def close(self, silent: bool = False) -> bool:
+        """Close the psycopg connection to postgres.
+
+        :param silent: whether the method should not write logs.
+
+        :returns: ``True`` if ``psycopg`` connection was closed, ``False`` otherwise.``
+        """
+        ret = False
        if self._connection and self._connection.closed == 0:
            self._connection.close()
-            logger.info("closed patroni connection to postgres")
+            if not silent:
+                logger.info("closed patroni %s connection to postgres", self._name)
+            ret = True
        self._connection = None
+        return ret
+
+
+class ConnectionPool:
+    """Helper class to manage named connections from Patroni to PostgreSQL.
+
+    The instance keeps named :class:`NamedConnection` objects and parameters that must be used for new connections.
+    """
+
+    def __init__(self) -> None:
+        """Create an instance of :class:`ConnectionPool` class."""
+        self._lock = Lock()
+        self._connections: Dict[str, NamedConnection] = {}
+        self._conn_kwargs: Dict[str, Any] = {}
+
+    @property
+    def conn_kwargs(self) -> Dict[str, Any]:
+        """Connection parameters that must be used for new ``psycopg`` connections."""
+        with self._lock:
+            return self._conn_kwargs.copy()
+
+    @conn_kwargs.setter
+    def conn_kwargs(self, value: Dict[str, Any]) -> None:
+        """Set new connection parameters.
+
+        :param value: :class:`dict` object with connection parameters.
+        """
+        with self._lock:
+            self._conn_kwargs = value
+
+    def get(self, name: str, kwargs_override: Optional[Dict[str, Any]] = None) -> NamedConnection:
+        """Get a new named :class:`NamedConnection` object from the pool.
+
+        .. note::
+            Creates a new :class:`NamedConnection` object if it doesn't yet exist in the pool.
+
+        :param name: name of the connection.
+        :param kwargs_override: :class:`dict` object with connection parameters that should be
+                                different from default values provided by :attr:`conn_kwargs`.
+
+        :returns: :class:`NamedConnection` object.
+        """
+        with self._lock:
+            if name not in self._connections:
+                self._connections[name] = NamedConnection(self, name, kwargs_override)
+        return self._connections[name]
+
+    def close(self) -> None:
+        """Close all named connections from Patroni to PostgreSQL registered in the pool."""
+        with self._lock:
+            if any(conn.close(True) for conn in self._connections.values()):
+                logger.info("closed patroni connections to postgres")


@contextmanager
@@ -384,8 +384,7 @@ class SlotsHandler:

        :yields: connection cursor object, note implementation varies depending on version of :mod:`psycopg`.
        """
-        conn_kwargs = self._postgresql.config.local_connect_kwargs
-        conn_kwargs.update(kwargs)
+        conn_kwargs = {**self._postgresql.connection_pool.conn_kwargs, **kwargs}
        with get_connection_cursor(**conn_kwargs) as cur:
            yield cur

@@ -435,7 +434,7 @@ class SlotsHandler:
            self._advance = SlotsAdvanceThread(self)
        return self._advance.schedule(slots)

-    def _ensure_logical_slots_replica(self, cluster: Cluster, slots: Dict[str, Any]) -> List[str]:
+    def _ensure_logical_slots_replica(self, slots: Dict[str, Any]) -> List[str]:
        """Update logical *slots* on replicas.

        If the logical slot already exists, copy state information into the replication slots structure stored in the
@@ -445,7 +444,6 @@ class SlotsHandler:
        As logical slots can only be created when the primary is available, pass the list of slots that need to be
        copied back to the caller. They will be created on replicas with :meth:`SlotsHandler.copy_logical_slots`.

-        :param cluster: object containing stateful information for the cluster.
        :param slots: A dictionary mapping slot name to slot attributes. This method only considers a slot
                      if the value is a dictionary with the key ``type`` and a value of ``logical``.

@@ -460,15 +458,16 @@ class SlotsHandler:
                continue

            # If the logical already exists, copy some information about it into the original structure
-            if self._replication_slots.get(name, {}).get('datoid'):
+            if name in self._replication_slots and compare_slots(value, self._replication_slots[name]):
                self._copy_items(self._replication_slots[name], value)
-                if cluster.slots and name in cluster.slots:
+                if 'lsn' in value:  # The slot has feedback in DCS
                    try:  # Skip slots that don't need to be advanced
-                        if value['confirmed_flush_lsn'] < int(cluster.slots[name]):
-                            advance_slots[value['database']][name] = int(cluster.slots[name])
+                        if value['confirmed_flush_lsn'] < int(value['lsn']):
+                            advance_slots[value['database']][name] = int(value['lsn'])
                    except Exception as e:
-                        logger.error('Failed to parse "%s": %r', cluster.slots[name], e)
-            elif cluster.slots and name in cluster.slots:  # We want to copy only slots with feedback in a DCS
+                        logger.error('Failed to parse "%s": %r', value['lsn'], e)
+            elif name not in self._replication_slots and 'lsn' in value:
+                # We want to copy only slots with feedback in a DCS
                create_slots.append(name)

        # Slots to be copied from the primary should be removed from the *slots* structure,
@@ -513,10 +512,9 @@ class SlotsHandler:
                if self._postgresql.is_primary():
                    self._logical_slots_processing_queue.clear()
                    self._ensure_logical_slots_primary(slots)
-                elif cluster.slots and slots:
+                else:
                    self.check_logical_slots_readiness(cluster, replicatefrom)
-
-                    ret = self._ensure_logical_slots_replica(cluster, slots)
+                    ret = self._ensure_logical_slots_replica(slots)

                self._replication_slots = slots
            except Exception:
@@ -209,7 +209,7 @@ class _ReplicaList(List[_Replica]):
            # 2. can be mapped to a ``Member`` of the ``Cluster``:
            #   a. ``Member`` doesn't have ``nosync`` tag set;
            #   b. PostgreSQL on the member is known to be running and accepting client connections.
-            if member and row[sort_col] is not None and member.is_running and not member.tags.get('nosync', False):
+            if member and row[sort_col] is not None and member.is_running and not member.nosync:
                self.append(_Replica(row['pid'], row['application_name'],
                                     row['sync_state'], row[sort_col], bool(member.nofailover)))

@@ -290,7 +290,7 @@ def _load_postgres_gucs_validators() -> None:
    Any problem faced while reading or parsing files will be logged as a ``WARNING`` by the child function, and the
    corresponding file or validator will be ignored.

-    By default Patroni only ships the file ``0_postgres.yml``, which contains Community Postgres GUCs validators, but
+    By default, Patroni only ships the file ``0_postgres.yml``, which contains Community Postgres GUCs validators, but
    that behavior can be extended. For example: if a vendor wants to add GUC validators to Patroni for covering a custom
    Postgres build, then they can create their custom YAML files under ``available_parameters`` directory.

@@ -300,8 +300,10 @@ def _load_postgres_gucs_validators() -> None:
            writes them to ``postgresql.conf`` if running PG 12 and above).

    Then, each of these sections, if specified, may contain one or more attributes with the following structure:
+
        * key: the name of a GUC;
        * value: a list of validators. Each item in the list must contain a ``type`` attribute, which must be one among:
+
            * ``Bool``; or
            * ``Integer``; or
            * ``Real``; or
@@ -313,6 +315,7 @@ def _load_postgres_gucs_validators() -> None:
            class in this module.

    .. seealso::
+
        * :class:`Bool`;
        * :class:`Integer`;
        * :class:`Real`;
@@ -325,61 +328,62 @@ def _load_postgres_gucs_validators() -> None:
        This is a sample content for an YAML file based on Postgres GUCs, showing each of the supported types and
        sections:

-        ```yaml
-        parameters:
-          archive_command:
-          - type: String
-            version_from: 90300
-            version_till: null
-          archive_mode:
-          - type: Bool
-            version_from: 90300
-            version_till: 90500
-          - type: EnumBool
-            version_from: 90500
-            version_till: null
-            possible_values:
-            - always
-          archive_timeout:
-          - type: Integer
-            version_from: 90300
-            version_till: null
-            min_val: 0
-            max_val: 1073741823
-            unit: s
-          autovacuum_vacuum_cost_delay:
-          - type: Integer
-            version_from: 90300
-            version_till: 120000
-            min_val: -1
-            max_val: 100
-            unit: ms
-          - type: Real
-            version_from: 120000
-            version_till: null
-            min_val: -1
-            max_val: 100
-            unit: ms
-          client_min_messages:
-          - type: Enum
-            version_from: 90300
-            version_till: null
-            possible_values:
-            - debug5
-            - debug4
-            - debug3
-            - debug2
-            - debug1
-            - log
-            - notice
-            - warning
-            - error
-        recovery_parameters:
-          archive_cleanup_command:
-          - type: String
-            version_from: 90300
-            version_till: null
-        ```
+            .. code-block:: yaml
+
+                parameters:
+                  archive_command:
+                  - type: String
+                    version_from: 90300
+                    version_till: null
+                  archive_mode:
+                  - type: Bool
+                    version_from: 90300
+                    version_till: 90500
+                  - type: EnumBool
+                    version_from: 90500
+                    version_till: null
+                    possible_values:
+                    - always
+                  archive_timeout:
+                  - type: Integer
+                    version_from: 90300
+                    version_till: null
+                    min_val: 0
+                    max_val: 1073741823
+                    unit: s
+                  autovacuum_vacuum_cost_delay:
+                  - type: Integer
+                    version_from: 90300
+                    version_till: 120000
+                    min_val: -1
+                    max_val: 100
+                    unit: ms
+                  - type: Real
+                    version_from: 120000
+                    version_till: null
+                    min_val: -1
+                    max_val: 100
+                    unit: ms
+                  client_min_messages:
+                  - type: Enum
+                    version_from: 90300
+                    version_till: null
+                    possible_values:
+                    - debug5
+                    - debug4
+                    - debug3
+                    - debug2
+                    - debug1
+                    - log
+                    - notice
+                    - warning
+                    - error
+                recovery_parameters:
+                  archive_cleanup_command:
+                  - type: String
+                    version_from: 90300
+                    version_till: null
+
    """
    conf_dir = os.path.join(
        os.path.dirname(os.path.abspath(__file__)),
@@ -434,13 +438,15 @@ def _transform_parameter_value(validators: MutableMapping[str, Tuple[_Transforma
    :param value: value of the Postgres GUC.
    :param available_gucs: a set of all GUCs available in Postgres *version*. Each item is the name of a Postgres
        GUC. Used for a couple purposes:
+
        * Disallow writing GUCs to ``postgresql.conf`` (or ``recovery.conf``) that does not exist in Postgres *version*;
        * Avoid ignoring GUC *name* if it does not have a validator in *validators*, but is a valid GUC in Postgres
-            *version*.
+          *version*.

    :returns: the return value may be one among:
-        * *value* transformed to the expected format for GUC *name* in Postgres *version*, if *name* is present in
-            *available_gucs* and has a validator in *validators* for the corresponding Postgres *version*; or
+
+        * *value* transformed to the expected format for GUC *name* in Postgres *version*, if *name* is present
+          in *available_gucs* and has a validator in *validators* for the corresponding Postgres *version*; or
        * The own *value* if *name* is present in *available_gucs* but not in *validators*; or
        * ``None`` if *name* is not present in *available_gucs*.
    """
@@ -0,0 +1,64 @@
+"""Tags handling."""
+import abc
+
+from typing import Any, Dict, Optional
+
+
+class Tags(abc.ABC):
+    """An abstract class that encapsulates all the ``tags`` logic.
+
+    Child classes that want to use provided facilities must implement ``tags`` abstract property.
+    """
+
+    @staticmethod
+    def _filter_tags(tags: Dict[str, Any]) -> Dict[str, Any]:
+        """Get tags configured for this node, if any.
+
+        Handle both predefined Patroni tags and custom defined tags.
+
+        .. note::
+            A custom tag is any tag added to the configuration ``tags`` section that is not one of ``clonefrom``,
+            ``nofailover``, ``noloadbalance`` or ``nosync``.
+
+            For the Patroni predefined tags, the returning object will only contain them if they are enabled as they
+            all are boolean values that default to disabled.
+
+        :returns: a dictionary of tags set for this node. The key is the tag name, and the value is the corresponding
+            tag value.
+        """
+        return {tag: value for tag, value in tags.items()
+                if tag not in ('clonefrom', 'nofailover', 'noloadbalance', 'nosync') or value}
+
+    @property
+    @abc.abstractmethod
+    def tags(self) -> Dict[str, Any]:
+        """Configured tags.
+
+        Must be implemented in a child class.
+        """
+        raise NotImplementedError  # pragma: no cover
+
+    @property
+    def clonefrom(self) -> bool:
+        """``True`` if ``clonefrom`` tag is ``True``, else ``False``."""
+        return self.tags.get('clonefrom', False)
+
+    @property
+    def nofailover(self) -> bool:
+        """``True`` if ``nofailover`` is ``True``, else ``False``."""
+        return bool(self.tags.get('nofailover', False))
+
+    @property
+    def noloadbalance(self) -> bool:
+        """``True`` if ``noloadbalance`` is ``True``, else ``False``."""
+        return bool(self.tags.get('noloadbalance', False))
+
+    @property
+    def nosync(self) -> bool:
+        """``True`` if ``nosync`` is ``True``, else ``False``."""
+        return bool(self.tags.get('nosync', False))
+
+    @property
+    def replicatefrom(self) -> Optional[str]:
+        """Value of ``replicatefrom`` tag, if any."""
+        return self.tags.get('replicatefrom')
@@ -839,8 +839,9 @@ def cluster_as_json(cluster: 'Cluster', global_config: Optional['GlobalConfig']
        ret['pause'] = True
    if cluster.failover and cluster.failover.scheduled_at:
        ret['scheduled_switchover'] = {'at': cluster.failover.scheduled_at.isoformat()}
-        if cluster.failover.leader:
-            ret['scheduled_switchover']['from'] = cluster.failover.leader
+        if TYPE_CHECKING:  # pragma: no cover
+            assert cluster.failover.leader
+        ret['scheduled_switchover']['from'] = cluster.failover.leader
        if cluster.failover.candidate:
            ret['scheduled_switchover']['to'] = cluster.failover.candidate
    return ret
@@ -333,15 +333,17 @@ class Case(object):
        """Create a :class:`Case` object.

        :param schema: the schema for validating a set of attributes that may be available in the configuration.
-            Each key is the configuration that is available in a given scope and that should be validated, and the
-            related value is the validation function or expected type.
+                       Each key is the configuration that is available in a given scope and that should be validated,
+                       and the related value is the validation function or expected type.

        :Example:

-            Case({
-                "host": validate_host_port,
-                "url": str,
-            })
+            .. code-block:: python
+
+                Case({
+                    "host": validate_host_port,
+                    "url": str,
+                })

        That will check that ``host`` configuration, if given, is valid based on :func:`validate_host_port`, and will
        also check that ``url`` configuration, if given, is a ``str`` instance.
@@ -363,14 +365,16 @@ class Or(object):

        :Example:

-            Or("host", "hosts"): Case({
-                "host": validate_host_port,
-                "hosts": Or(comma_separated_host_port, [validate_host_port]),
-            })
+            .. code-block:: python

-        The outer :class:`Or` is used to define that ``host`` and ``hosts`` are possible options in this scope.
-        The inner :class`Or` in the ``hosts`` key value is used to define that ``hosts`` option is valid if either of
-            :func:`comma_separated_host_port` or :func:`validate_host_port` succeed to validate it.
+                Or("host", "hosts"): Case({
+                    "host": validate_host_port,
+                    "hosts": Or(comma_separated_host_port, [validate_host_port]),
+                })
+
+            The outer :class:`Or` is used to define that ``host`` and ``hosts`` are possible options in this scope.
+            The inner :class`Or` in the ``hosts`` key value is used to define that ``hosts`` option is valid if either
+            of :func:`comma_separated_host_port` or :func:`validate_host_port` succeed to validate it.
        """
        self.args = args

@@ -535,32 +539,34 @@ class Schema(object):

        :Example:

-            Schema({
-                "application_name": str,
-                "bind": {
-                    "host": validate_host,
-                    "port": int,
-                },
-                "aliases": [str],
-                Optional("data_directory"): "/var/lib/myapp",
-                Or("log_to_file", "log_to_db"): Case({
-                    "log_to_file": bool,
-                    "log_to_db": bool,
-                }),
-                "version": Or(int, float),
-            })
+            .. code-block:: python

-        This sample schema defines that your YAML configuration follows these rules:
+                Schema({
+                    "application_name": str,
+                    "bind": {
+                        "host": validate_host,
+                        "port": int,
+                    },
+                    "aliases": [str],
+                    Optional("data_directory"): "/var/lib/myapp",
+                    Or("log_to_file", "log_to_db"): Case({
+                        "log_to_file": bool,
+                        "log_to_db": bool,
+                    }),
+                    "version": Or(int, float),
+                })

-            * It must contain an ``application_name`` entry which value should be a :class:`str` instance;
-            * It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``;
-            * It must contain a ``bind.port`` entry which value should be an :class:`int` instance;
-            * It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances;
-            * It may optionally contain a ``data_directory`` entry, with a value which should be a string;
-            * It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a
-              :class:`bool` instance;
-            * It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float`
-              instance.
+            This sample schema defines that your YAML configuration follows these rules:
+
+                * It must contain an ``application_name`` entry which value should be a :class:`str` instance;
+                * It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``;
+                * It must contain a ``bind.port`` entry which value should be an :class:`int` instance;
+                * It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances;
+                * It may optionally contain a ``data_directory`` entry, with a value which should be a string;
+                * It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a
+                  :class:`bool` instance;
+                * It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float`
+                  instance.
        """
        self.validator = validator

@@ -1,5 +1,4 @@
 sphinx>=4
 sphinx_rtd_theme>1
 sphinxcontrib-apidoc
-sphinx-github-style
-pyyaml
+sphinx-github-style<1.0.3
@@ -104,7 +104,7 @@ class MockCursor(object):
        elif sql.startswith('SELECT slot_name, slot_type, datname, plugin, catalog_xmin'):
            self.results = [('ls', 'logical', 'a', 'b', 100, 500, b'123456')]
        elif sql.startswith('SELECT slot_name'):
-            self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'a', 'b', 5, 100, 500)]
+            self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'b', 'a', 5, 100, 500)]
        elif sql.startswith('WITH slots AS (SELECT slot_name, active'):
            self.results = [(False, True)] if self.rowcount == 1 else []
        elif sql.startswith('SELECT CASE WHEN pg_catalog.pg_is_in_recovery()'):
@@ -11,10 +11,13 @@ from socketserver import ThreadingMixIn
 from patroni.api import RestApiHandler, RestApiServer
 from patroni.config import GlobalConfig
 from patroni.dcs import ClusterConfig, Member
+from patroni.exceptions import PostgresConnectionException
 from patroni.ha import _MemberStatus
 from patroni.manual_failover import ManualFailoverPrecheckStatus
+from patroni.psycopg import OperationalError
 from patroni.utils import ParseScheduleErrors, RetryFailedError, tzutc

+from . import MockConnect, psycopg_connect
 from .test_ha import get_cluster_initialized_without_leader, get_cluster_initialized_with_leader


@@ -22,8 +25,29 @@ future_restart_time = datetime.datetime.now(tzutc) + datetime.timedelta(days=5)
 postmaster_start_time = datetime.datetime.now(tzutc)


-class MockPostgresql(object):
+class MockConnection:

+    @staticmethod
+    def get(*args):
+        return psycopg_connect()
+
+    @staticmethod
+    def query(sql, *params):
+        return [(postmaster_start_time, 0, '', 0, '', False, postmaster_start_time, 'streaming', None,
+                 '[{"application_name":"walreceiver","client_addr":"1.2.3.4",'
+                 + '"state":"streaming","sync_state":"async","sync_priority":0}]')]
+
+
+class MockConnectionPool:
+
+    @staticmethod
+    def get(*args):
+        return MockConnection()
+
+
+class MockPostgresql:
+
+    connection_pool = MockConnectionPool()
    name = 'test'
    state = 'running'
    role = 'primary'
@@ -55,12 +79,6 @@ class MockPostgresql(object):
    def replication_state_from_parameters(*args):
        return 'streaming'

-    @staticmethod
-    def query(sql, *params, retry=False):
-        return [(postmaster_start_time, 0, '', 0, '', False, postmaster_start_time, 'streaming', None,
-                 '[{"application_name":"walreceiver","client_addr":"1.2.3.4",'
-                 + '"state":"streaming","sync_state":"async","sync_priority":0}]')]
-

 class MockWatchdog(object):
    is_healthy = False
@@ -101,7 +119,7 @@ class MockHa(object):

    @staticmethod
    def fetch_nodes_statuses(members):
-        return [_MemberStatus(None, True, None, 0, 0, None, {}, False)]
+        return [_MemberStatus(None, True, None, 0, {})]

    @staticmethod
    def schedule_future_restart(data):
@@ -491,7 +509,7 @@ class TestRestApiHandler(unittest.TestCase):

    @patch('time.sleep', Mock())
    def test_RestApiServer_query(self):
-        with patch.object(MockPostgresql, 'query', Mock(side_effect=RetryFailedError('bla'))):
+        with patch.object(MockConnection, 'query', Mock(side_effect=RetryFailedError('bla'))):
            self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))

    @patch('time.sleep', Mock())
@@ -762,3 +780,11 @@ class TestRestApiServer(unittest.TestCase):

    def test_get_certificate_serial_number(self):
        self.assertIsNone(self.srv.get_certificate_serial_number())
+
+    def test_query(self):
+        with patch.object(MockConnection, 'get', Mock(side_effect=OperationalError)):
+            self.assertRaises(PostgresConnectionException, self.srv.query, 'SELECT 1')
+        with patch.object(MockConnection, 'get', Mock(side_effect=[MockConnect(), OperationalError])), \
+                patch.object(MockConnection, 'query') as mock_query:
+            self.srv.query('SELECT 1')
+            mock_query.assert_called_once_with('SELECT 1')
@@ -250,7 +250,7 @@ class TestBootstrap(BaseTestPostgresql):
        self.assertFalse(self.b.call_post_bootstrap({'post_init': '/bin/false'}))

        mock_cancellable_subprocess_call.return_value = 0
-        self.p.config.superuser.pop('username')
+        self.p.connection_pool._conn_kwargs.pop('user')
        self.assertTrue(self.b.call_post_bootstrap({'post_init': '/bin/false'}))
        mock_cancellable_subprocess_call.assert_called()
        args, kwargs = mock_cancellable_subprocess_call.call_args
@@ -258,7 +258,7 @@ class TestBootstrap(BaseTestPostgresql):
        self.assertEqual(args[0], ['/bin/false', 'dbname=postgres host=127.0.0.2 port=5432'])

        mock_cancellable_subprocess_call.reset_mock()
-        self.p.config._local_address.pop('host')
+        self.p.connection_pool._conn_kwargs.pop('host')
        self.assertTrue(self.b.call_post_bootstrap({'post_init': '/bin/false'}))
        mock_cancellable_subprocess_call.assert_called()
        self.assertEqual(mock_cancellable_subprocess_call.call_args[0][0], ['/bin/false', 'dbname=postgres port=5432'])
@@ -13,7 +13,7 @@ class TestCitus(BaseTestPostgresql):
    def setUp(self):
        super(TestCitus, self).setUp()
        self.c = self.p.citus_handler
-        self.c.set_conn_kwargs({'host': 'localhost', 'dbname': 'postgres'})
+        self.p.connection_pool.conn_kwargs = {'host': 'localhost', 'dbname': 'postgres'}
        self.cluster = get_cluster_initialized_with_leader()
        self.cluster.workers[1] = self.cluster

@@ -23,10 +23,17 @@ from .test_ha import get_cluster_initialized_without_leader, get_cluster_initial
    get_cluster_initialized_with_only_leader, get_cluster_not_initialized_without_leader, get_cluster, Member


-@patch('patroni.ctl.load_config', Mock(return_value={
-    'scope': 'alpha', 'restapi': {'listen': '::', 'certfile': 'a'}, 'ctl': {'certfile': 'a'},
-    'etcd': {'host': 'localhost:2379'}, 'citus': {'database': 'citus', 'group': 0},
-    'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}}))
+DEFAULT_CONFIG = {
+    'scope': 'alpha',
+    'restapi': {'listen': '::', 'certfile': 'a'},
+    'ctl': {'certfile': 'a'},
+    'etcd': {'host': 'localhost:2379'},
+    'citus': {'database': 'citus', 'group': 0},
+    'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}
+}
+
+
+@patch('patroni.ctl.load_config', Mock(return_value=DEFAULT_CONFIG))
 class TestCtl(unittest.TestCase):
    TEST_ROLES = ('master', 'primary', 'leader')

@@ -140,7 +147,7 @@ class TestCtl(unittest.TestCase):
        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'],
                                    input=f'leader\nother\n{self.SCHEDULED_TS}\ny')
        self.assertEqual(result.exit_code, 0)
-        self.assertIn(f'Are you sure you want to schedule switchover of cluster dummy '
+        self.assertIn(f'Are you sure you want to schedule a switchover in the cluster dummy '
                      f'at {self.SCHEDULED_TS}, demoting current leader', result.output)

        # Scheduled (abort)
@@ -271,6 +278,15 @@ class TestCtl(unittest.TestCase):

        # Failover to an async member in sync mode (confirm)
        cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+
+        # Temp test to check a fallback to switchover if leader is specified
+        with patch('patroni.ctl._do_failover_or_switchover') as failover_func_mock:
+            result = self.runner.invoke(ctl, ['failover', '--leader', 'leader', 'dummy'], input='0\n')
+            self.assertIn('Supplying a leader name using this command is deprecated', result.output)
+            failover_func_mock.assert_called_once_with(
+                DEFAULT_CONFIG, 'switchover', 'dummy', None, 'leader', None, False)
+
+        # Failover to an async member in sync mode (confirm)
        cluster.members.append(Member(0, 'async', 28, {'api_url': 'http://127.0.0.1:8012/patroni'}))
        cluster.config.data['synchronous_mode'] = True
        mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
@@ -520,9 +536,19 @@ class TestCtl(unittest.TestCase):
    @patch('patroni.ctl.get_dcs')
    def test_members(self, mock_get_dcs):
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
+
        result = self.runner.invoke(ctl, ['list'])
        assert '127.0.0.1' in result.output
        assert result.exit_code == 0
+        assert 'Citus cluster: alpha -' in result.output
+
+        result = self.runner.invoke(ctl, ['list', '--group', '0'])
+        assert 'Citus cluster: alpha (group: 0, 12345678901) -' in result.output
+
+        with patch('patroni.ctl.load_config', Mock(return_value={'scope': 'alpha'})):
+            result = self.runner.invoke(ctl, ['list'])
+            assert 'Cluster: alpha (12345678901) -' in result.output
+
        with patch('patroni.ctl.load_config', Mock(return_value={})):
            self.runner.invoke(ctl, ['list'])

@@ -99,7 +99,9 @@ def get_node_status(reachable=True, in_recovery=True, dcs_last_seen=0,
        tags = {}
        if nofailover:
            tags['nofailover'] = True
-        return _MemberStatus(e, reachable, in_recovery, dcs_last_seen, timeline, wal_position, tags, watchdog_failed)
+        return _MemberStatus(e, reachable, in_recovery, wal_position,
+                             {'tags': tags, 'watchdog_failed': watchdog_failed,
+                              'dcs_last_seen': dcs_last_seen, 'timeline': timeline})
    return fetch_node_status


@@ -1461,14 +1463,16 @@ class TestHa(PostgresInit):
        mock_restart.assert_called_once()
        self.ha.dcs.get_cluster.assert_not_called()

+    @patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
    def test_enable_synchronous_mode(self):
        self.ha.is_synchronous_mode = true
        self.ha.has_lock = true
        self.p.name = 'leader'
+        self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(), CaseInsensitiveSet()))
        self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
        with patch('patroni.ha.logger.info') as mock_logger:
            self.ha.run_cycle()
-            self.assertEqual(mock_logger.call_args[0][0], 'Enabled synchronous replication')
+            self.assertEqual(mock_logger.call_args_list[0][0][0], 'Enabled synchronous replication')
        self.ha.dcs.write_sync_state = Mock(return_value=None)
        with patch('patroni.ha.logger.warning') as mock_logger:
            self.ha.run_cycle()
@@ -308,13 +308,15 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
        mock_patch_namespaced_pod.assert_called()
        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['isMaster'], 'false')
        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['tmp_role'], 'replica')
-
-        self.k.touch_member({'state': 'running', 'role': 'standby-leader'})
-        mock_patch_namespaced_pod.assert_called()
-        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['isMaster'], 'false')
-        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['tmp_role'], 'standby-leader')
+        mock_patch_namespaced_pod.rest_mock()

        self.k._name = 'p-0'
+        self.k.touch_member({'role': 'standby_leader'})
+        mock_patch_namespaced_pod.assert_called()
+        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['isMaster'], 'false')
+        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['tmp_role'], 'master')
+        mock_patch_namespaced_pod.rest_mock()
+
        self.k.touch_member({'role': 'primary'})
        mock_patch_namespaced_pod.assert_called()
        self.assertEqual(mock_patch_namespaced_pod.call_args[0][2].metadata.labels['isMaster'], 'true')
@@ -434,6 +436,10 @@ class TestKubernetesEndpoints(BaseTestKubernetes):
        mock_logger_exception.assert_called_once()
        self.assertEqual(('create_config_service failed',), mock_logger_exception.call_args[0])

+    @patch.object(k8s_client.CoreV1Api, 'patch_namespaced_endpoints', mock_namespaced_kind, create=True)
+    def test_write_leader_optime(self):
+        self.k.write_leader_optime(12345)
+

 def mock_watch(*args):
    return urllib3.HTTPResponse()
@@ -185,7 +185,7 @@ class TestPatroni(unittest.TestCase):

    def test_reload_config(self):
        self.p.reload_config()
-        self.p.get_tags = Mock(side_effect=Exception)
+        self.p._get_tags = Mock(side_effect=Exception)
        self.p.reload_config(local=True)

    def test_nosync(self):
@@ -310,6 +310,17 @@ class TestPostgresql(BaseTestPostgresql):
        self.p.config.write_postgresql_conf()
        self.assertEqual(self.p.config.check_recovery_conf(None), (False, False))
        self.assertEqual(self.p.config.check_recovery_conf(None), (False, False))
+
+        # Config files changed, but can't connect to postgres
+        mock_get_pg_settings.side_effect = PostgresConnectionException('')
+        with patch('patroni.postgresql.config.mtime', mock_mtime):
+            self.assertEqual(self.p.config.check_recovery_conf(None), (True, True))
+
+        # Config files didn't change, but postgres crashed or in crash recovery
+        with patch.object(MockPostmaster, 'create_time', Mock(return_value=1234568), create=True):
+            self.assertEqual(self.p.config.check_recovery_conf(None), (False, False))
+
+        # Any other exception raised when executing the query
        mock_get_pg_settings.side_effect = Exception
        with patch('patroni.postgresql.config.mtime', mock_mtime):
            self.assertEqual(self.p.config.check_recovery_conf(None), (True, True))
@@ -577,7 +588,10 @@ class TestPostgresql(BaseTestPostgresql):
        self.assertEqual(self.p.config.local_replication_address, {'host': '/tmp', 'port': '5432'})
        self.p.config._server_parameters.pop('unix_socket_directories')
        self.p.config.resolve_connection_addresses()
-        self.assertEqual(self.p.config._local_address, {'port': '5432'})
+        self.assertEqual(self.p.connection_pool.conn_kwargs, {'connect_timeout': 3, 'dbname': 'postgres',
+                                                              'fallback_application_name': 'Patroni',
+                                                              'options': '-c statement_timeout=2000',
+                                                              'password': 'test', 'port': '5432', 'user': 'foo'})

    @patch.object(Postgresql, '_version_file_exists', Mock(return_value=True))
    def test_get_major_version(self):
@@ -32,9 +32,9 @@ class TestSlotsHandler(BaseTestPostgresql):
        self.p._global_config = GlobalConfig({})
        self.s = self.p.slots_handler
        self.p.start()
-        config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}}}, 1)
+        config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}, 'ls2': None}}, 1)
        self.cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem],
-                               None, SyncState.empty(), None, {'ls': 12345}, None)
+                               None, SyncState.empty(), None, {'ls': 12345, 'ls2': 12345}, None)

    def test_sync_replication_slots(self):
        config = ClusterConfig(1, {'slots': {'test_3': {'database': 'a', 'plugin': 'b'},
@@ -123,6 +123,7 @@ class TestSlotsHandler(BaseTestPostgresql):
            self.assertEqual(self.s.sync_replication_slots(self.cluster, False), ['ls'])
        self.cluster.slots['ls'] = 'a'
        self.assertEqual(self.s.sync_replication_slots(self.cluster, False), [])
+        self.cluster.config.data['slots']['ls']['database'] = 'b'
        with patch.object(MockCursor, 'rowcount', PropertyMock(return_value=1), create=True):
            self.assertEqual(self.s.sync_replication_slots(self.cluster, False), ['ls'])
Author	SHA1	Message	Date
Polina Bungina	e013c1a6ee	Reword confirmation message	2023-09-12 15:53:42 +02:00
Polina Bungina	03d9226633	Implement Failover.is_failover/switchover properties	2023-09-12 14:02:17 +02:00
Polina Bungina	5b4291bfab	Refactor manual failover checks - Implement the dedicated class that represents a manual failover request - Move manual failover/switchover prechecks to the class method and use for both ctl and api - Use a single parse_schedule function in both ctl and api - Implement has_members_eligible_to_promote Ha method - Fix get_members + role='any' exception msg	2023-09-12 13:13:17 +02:00
Polina BunginaandGitHub	b31a4d55c9	Ensure strict failover/switchover definition difference (#2784 ) - Don't set leader in failover key from patronictl failover - Show warning and execute switchover if leader option is provided for patronictl failover command - Be more precise in the log messages - Allow to failover to an async candidate in sync mode - Check if candidate is the same as the leader specified in api - Fix and extend some tests - Add documentation	2023-09-12 08:51:17 +02:00
IsraelandGitHub	3c24c33e59	Document how to change Postgres settings that touch shared memory (#2843 ) Some special handling is required when changing either of these settings in a Postgres cluster that has standby nodes: * `max_connections` * `max_prepared_transactions` * `max_locks_per_transaction` * `max_wal_senders` * `max_worker_processes` If one attempts to decrease `max_connections` dynamic setting and restart all nodes at the same time (primary and standbys), Patroni will refuse to apply the new value on the standbys and require the user to restart it again later, once replication catches up. That behavior is correct, but it is not documented. This commit adds information to documentation about that behavior and why it's required. References: PAT-166.	2023-09-11 19:25:01 +02:00
Matt BakerandGitHub	83a060fc15	Extend documentation with package installation and upgrade process (#2854 )	2023-09-11 19:02:13 +02:00
Israel GitHub Matt Baker	a2ceff1517	Generate documentation of private members through sphinx docs (#2831 ) * Generate documentation of private members through sphinx docs With this commit we make sphinx build API docs for the following things, which were missing up to this point: * `__init__` method of classes; * "private" members (properties, functions, methods, attributes, etc., which name starts with an underscore); * members that are missing a docstring, so we can still reference them with links in the documentation. The third point can be removed later, if we wish, when we reach a point where everything has proper docstrings in the Patroni code base. * Fix documentation problems found after enabling private methods in sphinx * `:cvar:` is not a valid domain role. Replaced with `:attr:`. * documentation for `consul.base.Consul.__init__` has a single backtick quoted string which is interpreted as a reference which cannot be found. Therefore, the docstring has been copied as a block quote. * various list spacing problems and indentation problems. * code blocks added where indentation is interpreted incorrectly * literal string quoting issues. --------- Signed-off-by: Israel Barth Rubio <[email protected]> Co-authored-by: Matt Baker <[email protected]>	2023-09-11 15:41:34 +02:00
Alexander KukushkinandGitHub	19f20ec2eb	Refactor replication slots handling (#2851 ) 1. make _get_members_slots() method return data in the same format as _get_permanent_slots() method 2. move conflicting name handling from get_replication_slots() to _get_members_slots() method 3. enrich structure returned by get_replication_slots() with the LSN of permanent logical slots reported by primary 4. use the added information in the SlotsHandler instead of fetching it from the Cluster.slots 5. bugfix: don't try to advance logical slot that doesn't match required configuration	2023-09-07 12:56:07 +02:00
Alexander KukushkinandGitHub	30f0f132e8	Don't start stopped postgres in pause (#2848 ) Due to a race condition Patroni was falsely assuming that the standby should be restarted because some recovery parameters (primary_conninfo or similar) were changed. Close https://github.com/zalando/patroni/issues/2834	2023-09-06 08:57:56 +02:00
Alexander KukushkinandGitHub	941e883dde	Override write_leader_optime method in K8s implementation (#2850 ) It is being called when postgres is already shut down cleanly but there are no healthy replicas to take it over. Close https://github.com/zalando/patroni/issues/2837 Close https://github.com/zalando/patroni/pull/2838	2023-09-05 07:41:45 +02:00
Polina BunginaandGitHub	89a162e000	Return system id to the ctl list title (#2840 )	2023-09-05 07:27:34 +02:00
Alexander KukushkinandGitHub	0ab5b49757	Introduce a dedicated postgres connection for REST API (#2833 ) Sharing a single connection between REST API and the main thread (doing heartbeats) was working mostly fine, except when Postgres becomes so slow that REST API queries start blocking the main loop. If the dedicated REST API connection isn't available we use the heartbeat connection as a fallback.	2023-09-05 07:26:44 +02:00
SKandGitHub	80a03a4892	Enreach some endpoints with the scope and name (#2846 ) - monitoring endpoints - added `name` to the `patroni`, next to the `scope` and `version` - metrics endpoint - added name to labels	2023-09-05 07:24:17 +02:00
Matt BakerandGitHub	d2603402ea	Debian docker image pip error (#2849 ) * Use virtualenv to install tox in behave Dockerfile Upstream change in postgres docker image uses debian restriction on installing system-wide non-debian python packages. Debian doesn't provide a tox>=4, so we need to install with pip. * Exclude all output directories generated using `tox-wrapper.sh` The `tox-wrapper.sh` script created by `features/Dockerfile` creates directories like features/output-tox-pg14-docker-behave-etcd-lin-973719674/ * Reduce footprint of tox behave docker image	2023-09-04 21:24:26 +02:00
Alexander KukushkinandGitHub	6b7f914da7	Fix bug with kubernetes.standby_leader_label_value (#2832 ) When running with the leader lock Patroni was just setting the `role` label to `master` and effectively `kubernetes.standby_leader_label_value` feature never worked. Now it is fixed, but in order to not introduce breaking changes we just update default value of the `standby_leader_label_value` to the `master`.	2023-09-04 10:03:37 +02:00
IsraelandGitHub	03107e6d8b	`patronictl --help` was showing `ctl` function's docstring (#2845 ) `patronictl` is implemented using `click` module, and that module uses the functions' docstrings for creating a helper text. As a consequence the docstring for `ctl` function was being shown to the user, which doesn't make sense. This PR fixes that issue by adding a user-friendly description to be shown on `patronictl --help`. We use a `\f` to tell `click` when to stop capturing text to show in the helper. Note that `patronictl` commands are implemented using `@ctl.command` decorator, and we always provide them with `help` argument. That said, none of the subcommands are affected by the aforementioned issue, only the entry point of the CLI. References: PAT-201.	2023-09-04 09:27:46 +02:00
Alexander KukushkinandGitHub	77dba39585	Pin version of sphinx-github-style (#2847 ) 1.0.3 removed support of `top_level` configuration parameter and builds now are failing. Besides that remove redundant pyyaml from requirements.docs.txt	2023-09-04 09:00:24 +02:00
Alexander KukushkinandGitHub	89d794facc	Introduce connection pool (#2829 ) Make it hold connection kwargs for local connections and all `NamedConnection` objects use them automatically. Also get rid of redundant `ConfigHandler.local_connect_kwargs`. On top of that we will introduce a dedicated connection for the REST API thread.	2023-08-24 16:13:22 +02:00
Alexander KukushkinandGitHub	3333e78500	Factor out tags handling into a dedicated class (#2823 ) The same (almost) logic was used in three different places: 1. `Patroni` class 2. `Member` class 3. `_MemberStatus` class Now they all inherit newly intoduced `Tags` class.	2023-08-21 17:03:14 +02:00
Matt BakerandGitHub	0ab4bc9d27	Exclude sphinx build files from git (#2828 )	2023-08-21 16:23:04 +02:00