Release 1.5.4 (#931 )

* Bump version * Update release notes * Make it possible to configure registration of Service in Consul via env variables
Optionally consider node not healthy if it is not on the latest timeline (#892 )
2026-08-27 08:00:28 +00:00 · 2019-01-15 12:14:19 +01:00 · 2019-01-15 11:16:30 +01:00 · 2019-01-15 11:15:35 +01:00 · 2019-01-15 08:42:13 +01:00 · 2019-01-15 08:37:06 +01:00
49 changed files with 1716 additions and 494 deletions
@@ -51,3 +51,6 @@ scm-source.json
 docs/build/
 docs/source/_static/
 docs/source/_templates/
+
+# Pycharm IDE
+.idea/
@@ -11,8 +11,13 @@ Global/Universal
 -  **PATRONI\_NAME**: name of the node where the current instance of Patroni is running. Must be unique for the cluster.
 -  **PATRONI\_NAMESPACE**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **PATRONI\_SCOPE**: cluster name
-  **PATRONI\_LOGLEVEL**: sets the general logging level (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
-  **PATRONI\_REQUESTS_LOGLEVEL**: sets the logging level for all HTTP requests e.g. Kubernetes API calls (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **PATRONI\_LOG\_LEVEL**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **PATRONI\_LOG\_FORMAT**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
+-  **PATRONI\_LOG\_DATEFORMAT**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
+-  **PATRONI\_LOG\_DIR**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this env variable, the application will retain 4 25MB logs by default. You can tune those retention values with `PATRONI_LOG_FILE_NUM` and `PATRONI_LOG_FILE_SIZE` (see below).
+-  **PATRONI\_LOG\_FILE\_NUM**: The number of application logs to retain.
+-  **PATRONI\_LOG\_FILE\_SIZE**: Size of patroni.log file (in bytes) that triggers a log rolling.
+-  **PATRONI\_LOG\_LOGGERS**: Redefine logging level per python module. Example ``PATRONI_LOG_LOGGERS="{patroni.postmaster: WARNING, urllib3: DEBUG}"``

 Bootstrap configuration
 -----------------------
@@ -36,11 +41,13 @@ Consul
 -  **PATRONI\_CONSUL\_KEY**: (optional) File with the client key. Can be empty if the key is part of certificate.
 -  **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable.
+-  **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
+-  **PATRONI\_CONSUL\_SERVICE\_CHECK\_INTERVAL**: (optional) how often to perform health check against registered url

 Etcd
 ----
 -  **PATRONI\_ETCD\_HOST**: the host:port for the etcd endpoint.
-  **PATRONI\_ETCD\_HOSTS**: list of etcd endpoints in format host1:port1,host2:port2,etc...
+-  **PATRONI\_ETCD\_HOSTS**: list of etcd endpoints in format 'host1:port1','host2:port2',etc...
 -  **PATRONI\_ETCD\_URL**: url for the etcd, in format: http(s)://(username:password@)host:port
 -  **PATRONI\_ETCD\_PROXY**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **PATRONI\_ETCD\_URL**
 -  **PATRONI\_ETCD\_SRV**: Domain to search the SRV record(s) for cluster autodiscovery.
@@ -79,7 +86,7 @@ PostgreSQL
 -  **PATRONI\_SUPERUSER\_PASSWORD**: password for the superuser, set during initialization (initdb).

 REST API
-------- 
+--------
 -  **PATRONI\_RESTAPI\_CONNECT\_ADDRESS**: IP address and port to access the REST API.
 -  **PATRONI\_RESTAPI\_LISTEN**: IP address and port that Patroni will listen to, to provide health-check information for HAProxy.
 -  **PATRONI\_RESTAPI\_USERNAME**: Basic-auth username to protect unsafe REST API endpoints.
@@ -10,6 +10,18 @@ Global/Universal
 -  **namespace**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **scope**: cluster name

+Log
+---
+-  **level**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **format**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
+-  **dateformat**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
+-  **dir**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this value, the application will retain 4 25MB logs by default. You can tune those retention values with `file_num` and `file_size` (see below).
+-  **file\_num**: The number of application logs to retain.
+-  **file\_size**: Size of patroni.log file (in bytes) that triggers a log rolling.
+-  **loggers**: This section allows redefining logging level per python module
+    -  **patroni.postmaster: WARNING**
+    -  **urllib3: DEBUG**
+
 Bootstrap configuration
 -----------------------
 -  **dcs**: This section will be written into `/<namespace>/<scope>/config` of a given configuration store after initializing of new cluster. This is the global configuration for the cluster. If you want to change some parameters for all cluster nodes - just do it in DCS (or via Patroni API) and all nodes will apply this configuration.
@@ -33,6 +45,11 @@ Bootstrap configuration
        -  **restore\_command**: command to restore WAL records from the remote master to standby leader, can be different from the list defined in :ref:`postgresql_settings`
        -  **archive\_cleanup\_command**: cleanup command for standby leader
        -  **recovery\_min\_apply\_delay**: how long to wait before actually apply WAL records on a standby leader
+    -  **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Patroni will try to create slots before opening connections to the cluster.
+        -  **my_slot_name**: the name of replication slot. It is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots.
+            -  **type**: slot type. Could be ``physical`` or ``logical``. If the slot is logical, you have to additionally define ``database`` and ``plugin``.
+               **database**: the database name where logical slots should be created.
+               **plugin**: the plugin name for the logical slot.
 -  **method**: custom script to use for bootstrapping this cluster.
   See :ref:`custom bootstrap methods documentation <custom_bootstrap>` for details.
   When ``initdb`` is specified revert to the default ``initdb`` command. ``initdb`` is also triggered when no ``method``
@@ -69,6 +86,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **key**: (optional) file with the client key. Can be empty if the key is part of **cert**.
 -  **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **checks**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable
+-  **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
+-  **service\_check\_interval**: (optional) how often to perform health check against registered url

 Etcd
 ----
@@ -144,8 +163,10 @@ PostgreSQL

 REST API
 -------- 
-  **connect\_address**: IP address (or hostname) and port, to access the Patroni's REST API. It can serve as a endpoint for HTTP health checks (read below about the "listen" REST API parameter), and also for user queries (either directly or via the REST API), as well as for the health checks done by the cluster members during leader elections (for example, to determine whether the master is still running, or if there is a node which has a WAL position that is ahead of the one doing the query; etc.) The connect_address is put in the member key in DCS, making it possible to translate the member name into the address to connect to its REST API.
-  **listen**: IP address (or hostname) and port that Patroni will listen to for the REST API - to provide also the same health checks and cluster messaging between the participating nodes, as described above. to provide health-check information for HAProxy (or any other load balancer capable of doing a HTTP "OPTION" or "GET" checks)  
+-  **connect\_address**: IP address (or hostname) and port, to access the Patroni's REST API. All the members of the cluster must be able to connect to this address, so unless the Patroni setup is intended for a demo inside the localhost, this address must be a non "localhost" or loopback addres (ie: "localhost" or "127.0.0.1"). It can serve as a endpoint for HTTP health checks (read below about the "listen" REST API parameter), and also for user queries (either directly or via the REST API), as well as for the health checks done by the cluster members during leader elections (for example, to determine whether the master is still running, or if there is a node which has a WAL position that is ahead of the one doing the query; etc.) The connect_address is put in the member key in DCS, making it possible to translate the member name into the address to connect to its REST API.
+
+-  **listen**: IP address (or hostname) and port that Patroni will listen to for the REST API - to provide also the same health checks and cluster messaging between the participating nodes, as described above. to provide health-check information for HAProxy (or any other load balancer capable of doing a HTTP "OPTION" or "GET" checks).
+
 -  **Optional**:
        -  **authentication**:
            -  **username**: Basic-auth username to protect unsafe REST API endpoints.
@@ -76,6 +76,7 @@ Also, the following Patroni configuration options can be changed only dynamicall
 - loop_wait: 10
 - retry_timeouts: 10
 - maximum_lag_on_failover: 1048576
+- check_timeline: false
 - postgresql.use_slots: true

 Upon changing these options, Patroni will read the relevant section of the configuration stored in DCS and change its
@@ -3,19 +3,133 @@
 Release notes
 =============

-Version 1.5.0                                                                                                                           
-------------                                                                                                                           
-                                                                                                                                        
+Version 1.5.4
+-------------
+
+This version implements flexible logging and fixes a number of bugs.
+
+**New features**
+
+- Improvements in logging infrastructure (Alexander Kukushkin, Lucas Capistrant, Alexander Anikin)
+
+  Logging configuration could be configured not only from environment variables but also from Patroni config file. It makes it possible to change logging configuration in runtime by updating config and doing reload or sending SIGHUP to the Patroni process. By default Patroni writes logs to stderr, but now it becomes possible to write logs directly into the file and rotate when it reaches a certain size. In addition to that added support of custom dateformat and the possibility to fine-tune log level for each python module.
+
+- Make it possible to take into account the current timeline during leader elections (Alexander Kukushkin)
+
+  It could happen that the node is considering itself as a healthiest one although it is currently not on the latest known timeline. In some cases we want to avoid promoting of such node, which could be achieved by setting `check_timeline` parameter to `true` (default behavior remains unchanged).
+
+- Relaxed requirements on superuser credentials
+
+  Libpq allows opening connections without explicitly specifying neither username nor password. Depending on situation it relies either on pgpass file or trust authentication method in pg_hba.conf. Since pg_rewind is also using libpq, it will work the same way.
+
+- Implemented possibility to configure Consul Service registration and check interval via environment variables (Alexander Kukushkin)
+
+  Registration of service in Consul was added in the 1.5.0, but so far it was only possible to turn it on via patroni.yaml.
+
+**Stability Improvements**
+
+- Set archive_mode to off during the custom bootstrap (Alexander Kukushkin)
+
+  We want to avoid archiving wals and history files until the cluster is fully functional.  It really helps if the custom bootstrap involves pg_upgrade.
+
+- Apply five seconds backoff when loading global config on start (Alexander Kukushkin)
+
+  It helps to avoid hammering DCS when Patroni just starting up.
+
+- Reduce amount of error messages generated on shutdown (Alexander Kukushkin)
+
+  They were harmless but rather annoying and sometimes scary.
+
+- Explicitly secure rw perms for recovery.conf at creation time (Lucas)
+
+  We don't want anybody except patroni/postgres user reading this file, because it contains replication user and password.
+
+- Redirect HTTPServer exceptions to logger (Julien Riou)
+
+  By default, such exceptions were logged on standard output messing with regular logs.
+
+**Bug fixes**
+
+- Removed stderr pipe to stdout on pg_ctl process (Cody Coons)
+
+  Inheriting stderr from the main Patroni process allows all Postgres logs to be seen along with all patroni logs. This is very useful in a container environment as Patroni and Postgres logs may be consumed using standard tools (docker logs, kubectl, etc). In addition to that, this change fixes a bug with Patroni not being able to catch postmaster pid when postgres writing some warnings into stderr.
+
+- Set Consul service check deregister timeout in Go time format (Pavel Kirillov)
+
+  Without explicitly mentioned time unit registration was failing.
+
+- Relax checks of standby_cluster cluster configuration (Dmitry Dolgov, Alexander Kukushkin)
+
+  It was accepting only strings as valid values and therefore it was not possible to specify the port as integer and create_replica_methods as a list.
+
+Version 1.5.3
+-------------
+
+Compatibility and bugfix release.
+
+- Improve stability when running with python3 against zookeeper (Alexander Kukushkin)
+
+  Change of `loop_wait` was causing Patroni to disconnect from zookeeper and never reconnect back.
+
+- Fix broken compatibility with postgres 9.3 (Alexander)
+
+  When opening a replication connection we should specify replication=1, beacuse 9.3 does not understand replication='database'
+
+- Make sure we refresh Consul session at least once per HA loop and improve handling of consul sessions exceptions (Alexander)
+
+  Restart of local consul agent invalidates all sessions related to the node. Not calling session refresh on time and not doing proper handling of session errors was causing demote of the primary.
+
+Version 1.5.2
+-------------
+
+Compatibility and bugfix release.
+
+- Compatibility with kazoo-2.6.0 (Alexander Kukushkin)
+
+  In order to make sure that requests are performed with an appropriate timeout, Patroni redefines create_connection method from python-kazoo module. The last release of kazoo slightly changed the way how create_connection method is called.
+
+- Fix Patroni crash when Consul cluster loses the leader (Alexander)
+
+  The crash was happening due to incorrect implementation of touch_member method, it should return boolean and not raise any exceptions.
+
+Version 1.5.1
+-------------
+
+This version implements support of permanent replication slots, adds support of pgBackRest and fixes number of bugs.
+
+**New features**
+
+- Permanent replication slots (Alexander Kukushkin)
+
+  Permanent replication slots are preserved on failover/switchover, that is, Patroni on the new primary will create configured replication slots right after doing promote. Slots could be configured with the help of `patronictl edit-config`. The initial configuration could be also done in the :ref:`bootstrap.dcs <settings>`.
+
+- Add pgbackrest support (Yogesh Sharma)
+
+  pgBackrest can restore in existing $PGDATA folder, this allows speedy restore as files which have not changed since last backup are skipped, to support this feature new parameter `keep_data` has been introduced. See :ref:`replica creation method <custom_replica_creation>` section for additional examples.
+
+**Bug fixes**
+
+- A few bugfixes in the "standby cluster" workflow (Alexander)
+
+  Please see https://github.com/zalando/patroni/pull/823 for more details.
+
+- Fix REST API health check when cluster management is paused and DCS is not accessible (Alexander)
+
+  Regression was introduced in https://github.com/zalando/patroni/commit/90cf930036a9d5249265af15d2b787ec7517cf57
+
+Version 1.5.0
+-------------
+
 This version enables Patroni HA cluster to operate in a standby mode, introduces experimental support for running on Windows, and provides a new configuration parameter to register PostgreSQL service in Consul.

-**New features**                                                                                                                        
+**New features**

- Standby cluster (Dmitry Dolgov)                                                                                                       
+- Standby cluster (Dmitry Dolgov)

  One or more Patroni nodes can form a standby cluster that runs alongside the primary one (i.e. in another datacenter) and consists of standby nodes that replicate from the master in the primary cluster. All PostgreSQL nodes in the standby cluster are replicas; one of those replicas elects itself to replicate directly from the remote master, while the others replicate from it in a cascading manner. More detailed description of this feature and some configuration examples can be found at :ref:`here <standby_cluster>`.

 - Register Services in Consul (Pavel Kirillov, Alexander Kukushkin)
-                                                                                                                                        
+
  If `register_service` parameter in the consul :ref:`configuration <consul_settings>` is enabled, the node will register a service with the name `scope` and the tag `master`, `replica` or `standby-leader`.

 - Experimental Windows support (Pavel Golub)
@@ -299,8 +413,8 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i

 - Minimize the amount of SELECT's issued by Patroni on every loop of HA cylce (Alexander Kukushkin)

-  On every iteration of HA loop Patroni needs to know recovery status and absolute wal position. From now on Patroni will run only single SELECT to get this information instead of two on the replica and three on the master. 
-  
+  On every iteration of HA loop Patroni needs to know recovery status and absolute wal position. From now on Patroni will run only single SELECT to get this information instead of two on the replica and three on the master.
+
 - Remove leader key on shutdown only when we have the lock (Ants)

  Unconditional removal was generating unnecessary and missleading exceptions.
@@ -310,7 +424,7 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i
 - Add version command to patronictl (Ants)

  It will show the version of installed Patroni and versions of running Patroni instances (if the cluster name is specified).
-  
+
 - Make optional specifying cluster_name argument for some of patronictl commands (Alexander, Ants)

  It will work if patronictl is using usual Patroni configuration file with the ``scope`` defined.
@@ -318,11 +432,11 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i
 - Show information about scheduled switchover and maintenance mode (Alexander)

  Before that it was possible to get this information only from Patroni logs or directly from DCS.
-  
+
 - Improve ``patronictl reinit`` (Alexander)

  Sometimes ``patronictl reinit`` refused to proceed when Patroni was busy with other actions, namely trying to start postgres. `patronictl` didn't provide any commands to cancel such long running actions and the only (dangerous) workarond was removing a data directory manually. The new implementation of `reinit` forcefully cancells other long-running actions before proceeding with reinit.
-  
+
 - Implement ``--wait`` flag in ``patronictl pause`` and ``patronictl resume`` (Alexander)

  It will make ``patronictl`` wait until the requested action is acknowledged by all nodes in the cluster.
@@ -369,7 +483,7 @@ Version 1.3.6
  After a crash that doesn't clean up postmaster.pid there could be a new process with the same pid, resulting in a false positive for is_running(), which will lead to all kinds of bad behavior.

 - Shutdown postgresql before bootstrap when we lost data directory (ainlolcat)
-  
+
  When data directory on the master is forcefully removed, postgres process can still stay alive for some time and prevent the replica created in place of that former master from starting or replicating.
  The fix makes Patroni cache the postmaster pid and its start time and let it terminate the old postmaster in case it is still running after the corresponding data directory has been removed.

@@ -439,7 +553,7 @@ Version 1.3.4
 - Pass the consul token as a header (Andrew Colin Kissa)

  Headers are now the prefered way to pass the token to the consul `API <https://www.consul.io/api/index.html#authentication>`__.
-  
+

 - Advanced configuration for Consul (Alexander Kukushkin)

@@ -65,6 +65,19 @@ for outdated backup files. Some people prefer other backup solutions, such as ``
 others, or simply roll their own scripts. In order to accommodate all those use-cases Patroni supports running custom
 scripts to clone a new replica. Those are configured in the ``postgresql`` configuration block:

+.. code:: YAML
+
+    postgresql:
+        create_replica_methods:
+            - <method name>
+        <method name>:
+            command: <command name>
+            keep_data: True
+            no_params: True
+            no_master: 1
+
+example: wal_e
+
 .. code:: YAML

    postgresql:
@@ -79,6 +92,21 @@ scripts to clone a new replica. Those are configured in the ``postgresql`` confi
        basebackup:
            max-rate: '100M'

+example: pgbackrest
+
+.. code:: YAML
+
+    postgresql:
+        create_replica_methods:
+            - pgbackrest
+            - basebackup
+        pgbackrest:
+            command: /usr/bin/pgbackrest --stanza=<scope> --delta restore
+            keep_data: True
+            no_params: True
+        basebackup:
+            max-rate: '100M'
+

 The ``create_replica_methods`` defines available replica creation methods and the order of executing them. Patroni will
 stop on the first one that returns 0. Each method should define a separate section in the configuration file, listing the command
@@ -99,6 +127,10 @@ A special ``no_master`` parameter, if defined, allows Patroni to call the replic
 running master or replicas. In that case, an empty string will be passed in a connection string. This is useful for
 restoring the formerly running cluster from the binary backup.

+A special ``keep_data`` parameter, if defined, will instuct Patroni to  not clean PGDATA folder before calling restore.
+
+A special ``no_params`` parameter, if defined, restricts passing parameters to custom command.
+
 A ``basebackup`` method is a special case: it will be used if
 ``create_replica_methods`` is empty, although it is possible
 to list it explicitly among the ``create_replica_methods`` methods. This method initializes a new replica with the
@@ -163,3 +195,5 @@ in a patroni configuration:

 Note, that these options will be applied only once during cluster bootstrap,
 and the only way to change them afterwards is through DCS.
+
+If you use replication slots on the standby cluster, you must also create the corresponding replication slot on the primary cluster.  It will not be done automatically by the standby cluster implementation.  You can use Patroni's permenant replication slots feature on the primary cluster to maintain a replication slot with the same name as ``primary_slot_name``, or its default value if ``primary_slot_name`` is not provided.
@@ -13,6 +13,8 @@ In asynchronous mode the cluster is allowed to lose some committed transactions

 The amount of transactions that can be lost is controlled via ``maximum_lag_on_failover`` parameter. Because the primary transaction log position is not sampled in real time, in reality the amount of lost data on failover is worst case bounded by  ``maximum_lag_on_failover`` bytes of transaction log plus the amount that is written in the last ``ttl`` seconds (``loop_wait``/2 seconds in the average case). However typical steady state replication delay is well under a second.

+By default, when running leader elections, Patroni does not take into account the current timeline of replicas, what in some cases could be undesirable behavior. You can prevent the node not having the same timeline as a former master become the new leader by changing the value of ``check_timeline`` parameter to ``true``.
+
 PostgreSQL synchronous replication
 ----------------------------------

@@ -1,11 +1,11 @@
 ### confd

-`confd` directory contains haproxy template files for the [confd](https://github.com/kelseyhightower/confd) -- lightweight configuration management tool
+`confd` directory contains haproxy and pgbouncer template files for the [confd](https://github.com/kelseyhightower/confd) -- lightweight configuration management tool
 You need to copy content of `confd` directory into /etcd/confd and run confd service:
 ```bash
 $ confd -prefix=/service/$PATRONI_SCOPE -backend etcd -node $PATRONI_ETCD_URL -interval=10
 ```
-It will periodically update haproxy.cfg with the actual list of Patroni nodes from `etcd` and "reload" haproxy when it is necessary.
+It will periodically update haproxy.cfg and pgbouncer.ini with the actual list of Patroni nodes from `etcd` and "reload" haproxy and pgbouncer.ini when it is necessary.


 ### startup-scripts
@@ -0,0 +1,12 @@
+[template]
+prefix = "/service/batman"
+owner = "postgres"
+mode = "0644"
+src = "pgbouncer.tmpl"
+dest = "/etc/pgbouncer/pgbouncer.ini"
+
+reload_cmd = "systemctl reload pgbouncer"
+
+keys = [
+    "/members/","/leader"
+]
@@ -0,0 +1,17 @@
+[databases]
+{{with get "/leader"}}{{$leader := .Value}}{{$leadkey := printf "/members/%s" $leader}}{{with get $leadkey}}{{$data := json .Value}}{{$hostport := base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}}{{ $host := base (index (split $hostport ":") 0)}}{{ $port := base (index (split $hostport ":") 1)}}* = host={{ $host }} port={{ $port }} pool_size=10{{end}}{{end}}
+
+[pgbouncer]
+logfile = /var/log/postgresql/pgbouncer.log
+pidfile = /var/run/postgresql/pgbouncer.pid
+listen_addr = *
+listen_port = 6432
+unix_socket_dir = /var/run/postgresql
+auth_type = trust
+auth_file = /etc/pgbouncer/userlist.txt
+auth_hba_file = /etc/pgbouncer/pg_hba.txt
+admin_users = pgbouncer
+stats_users = pgbouncer
+pool_mode = session
+max_client_conn = 100
+default_pool_size = 20
@@ -18,8 +18,14 @@ WorkingDirectory=~

 # Where to send early-startup messages from the server
 # This is normally controlled by the global default set by systemd
-# StandardOutput=syslog
+#StandardOutput=syslog

+# Pre-commands to start watchdog device
+# Uncomment if watchdog is part of your patroni setup
+#ExecStartPre=-/usr/bin/sudo /sbin/modprobe softdog
+#ExecStartPre=-/usr/bin/sudo /bin/chown postgres /dev/watchdog
+
+# Start the patroni process
 ExecStart=/bin/patroni /etc/patroni.yml

 # Send HUP to reload from patroni.yml
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+[[ "$3" == "master" ]] || exit
+
+PGPASSWORD=zalando psql -h localhost -U postgres -p $1 -w -tAc "SELECT slot_name FROM pg_replication_slots WHERE slot_type = 'logical'" >> data/postgres0/label
@@ -117,12 +117,24 @@ class PatroniController(AbstractController):
        except IOError:
            return None

-    def add_tag_to_config(self, tag, value):
+    @staticmethod
+    def recursive_update(dst, src):
+        for k, v in src.items():
+            if k in dst and isinstance(dst[k], dict):
+                PatroniController.recursive_update(dst[k], v)
+            else:
+                dst[k] = v
+
+    def update_config(self, custom_config):
        with open(self._config) as r:
            config = yaml.safe_load(r)
-            config['tags']['tag'] = value
+            self.recursive_update(config, custom_config)
            with open(self._config, 'w') as w:
                yaml.safe_dump(config, w, default_flow_style=False)
+        self._scope = config.get('scope', 'batman')
+
+    def add_tag_to_config(self, tag, value):
+        self.update_config({'tags': {tag: value}})

    def _start(self):
        if self.watchdog:
@@ -174,13 +186,10 @@ class PatroniController(AbstractController):
                config['bootstrap']['initdb'].extend([{'auth': 'md5'}, {'auth-host': 'md5'}])

        if custom_config is not None:
-            def recursive_update(dst, src):
-                for k, v in src.items():
-                    if k in dst and isinstance(dst[k], dict):
-                        recursive_update(dst[k], v)
-                    else:
-                        dst[k] = v
-            recursive_update(config, custom_config)
+            self.recursive_update(config, custom_config)
+
+        if config['postgresql'].get('callbacks', {}).get('on_role_change'):
+            config['postgresql']['callbacks']['on_role_change'] += ' ' + str(self.__PORT)

        with open(patroni_config_path, 'w') as f:
            yaml.safe_dump(config, f, default_flow_style=False)
@@ -363,6 +372,7 @@ class ConsulController(AbstractDcsController):
    def __init__(self, context):
        super(ConsulController, self).__init__(context)
        os.environ['PATRONI_CONSUL_HOST'] = 'localhost:8500'
+        os.environ['PATRONI_CONSUL_REGISTER_SERVICE'] = 'on'
        self._client = consul.Consul()
        self._config_file = None

@@ -43,7 +43,7 @@ Scenario: check dynamic configuration change via DCS
 	And I receive a response loop_wait 2
 	When I issue a GET request to http://127.0.0.1:8008/patroni
 	Then I receive a response code 200
-	And I receive a response tags {'tag': 'new_value'}
+	And I receive a response tags {'new_tag': 'new_value'}

 Scenario: check API requests for the primary-replica pair in the pause mode
 	Given I run patronictl.py pause batman
@@ -1,9 +1,26 @@
 Feature: standby cluster
+  Scenario: check permanent logical slots are preserved on failover/switchover
+    Given I start postgres1
+    Then postgres1 is a leader after 10 seconds
+    And I sleep for 2 seconds
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"loop_wait": 2, "slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
+    Then I receive a response code 200
+    And Response on GET http://127.0.0.1:8009/config contains slots after 10 seconds
+    And I sleep for 2 seconds
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"test_logical": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
+    Then I receive a response code 200
+    When I start postgres0 with callback configured
+    Then "members/postgres0" key in DCS has state=running after 10 seconds
+    And replication works from postgres1 to postgres0 after 15 seconds
+    When I shut down postgres1
+    Then postgres0 is a leader after 10 seconds
+    And I sleep for 2 seconds
+    When I issue a GET request to http://127.0.0.1:8008/
+    Then I receive a response code 200
+    And there is a label with "test_logical" in postgres0 data directory

  Scenario: check replication of a single table in a standby cluster
-    Given I start postgres0 without slots sync
-    And I create a replication slot postgres1 on postgres0
-    And I start postgres1 in a standby cluster batman1 as a clone of postgres0
+    Given I start postgres1 in a standby cluster batman1 as a clone of postgres0
    Then postgres1 is a leader of batman1 after 10 seconds
    When I add the table foo to postgres0
    Then table foo is present on postgres1 after 20 seconds
@@ -13,4 +30,5 @@ Feature: standby cluster

  Scenario: check failover
    When I kill postgres1
-	Then postgres2 is replicating from postgres0 after 20 seconds
+    And I kill postmaster on postgres1
+    Then postgres2 is replicating from postgres0 after 20 seconds
@@ -1,3 +1,4 @@
+import os
 import time

 from behave import step
@@ -8,19 +9,13 @@ SELECT * FROM pg_catalog.pg_stat_replication
 WHERE application_name = '{0}'
 """

-create_replication_slot_query = """
-SELECT pg_create_physical_replication_slot('{0}')
-"""

-
-@step('I start {name:w} without slots sync')
-def start_patroni_without_slots_sync(context, name):
+@step('I start {name:w} with callback configured')
+def start_patroni_with_callbacks(context, name):
    return context.pctl.start(name, custom_config={
-        "bootstrap": {
-            "dcs": {
-                "postgresql": {
-                    "use_slots": False
-                }
+        "postgresql": {
+            "callbacks": {
+                "on_role_change": "features/callback.sh"
            }
        }
    })
@@ -35,19 +30,22 @@ def start_patroni(context, name, cluster_name):

@step('I start {name:w} in a standby cluster {cluster_name:w} as a clone of {name2:w}')
 def start_patroni_stanby_cluster(context, name, cluster_name, name2):
+    # we need to remove patroni.dynamic.json in order to "bootstrap" standby cluster with existing PGDATA
+    os.unlink(os.path.join(context.pctl._processes[name]._data_dir, 'patroni.dynamic.json'))
    port = context.pctl._processes[name2]._connkwargs.get('port')
-    return context.pctl.start(name, custom_config={
+    context.pctl._processes[name].update_config({
        "scope": cluster_name,
        "bootstrap": {
            "dcs": {
                "standby_cluster": {
                    "host": "localhost",
                    "port": port,
-                    "primary_slot_name": "postgres1",
+                    "primary_slot_name": "pm_1",
                }
            }
        }
    })
+    return context.pctl.start(name)


@step('{pg_name1:w} is replicating from {pg_name2:w} after {timeout:d} seconds')
@@ -67,12 +65,3 @@ def check_replication_status(context, pg_name1, pg_name2, timeout):
        time.sleep(1)

    return False
-
-
-@step('I create a replication slot {slot_name:w} on {pg_name:w}')
-def create_replication_slot(context, slot_name, pg_name):
-    return context.pctl.query(
-        pg_name,
-        create_replication_slot_query.format(slot_name),
-        fail_ok=True
-    )
@@ -1,32 +1,34 @@
-FROM postgres:9.6
+FROM postgres:11
 MAINTAINER Alexander Kukushkin <[email protected]>

 RUN export DEBIAN_FRONTEND=noninteractive \
    && echo 'APT::Install-Recommends "0";\nAPT::Install-Suggests "0";' > /etc/apt/apt.conf.d/01norecommend \
    && apt-get update -y \
    && apt-get upgrade -y \
-    && apt-get install -y git curl jq python-psycopg2 python-yaml python-requests python-six python-pysocks \
-        python-dateutil python-pip python-prettytable python-wheel python-psutil python locales \
-
+    && apt-cache depends patroni | sed -n -e 's/.* Depends: \(python3-.\+\)$/\1/p' \
+            | grep -Ev '^python3-(sphinx|etcd|consul|kazoo|kubernetes)' \
+            | xargs apt-get install -y vim-tiny curl jq locales git python3-pip python3-wheel \
    ## Make sure we have a en_US.UTF-8 locale available
    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
-
-    && pip install setuptools pip --upgrade \
-    && pip install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
-
-    && mkdir -p /home/postgres \
-    && chown postgres:postgres /home/postgres \
-
+    && pip3 install setuptools \
+    && pip3 install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
+    && PGHOME=/home/postgres \
+    && mkdir -p $PGHOME \
+    && chown postgres $PGHOME \
+    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
+    # Set permissions for OpenShift
+    && chmod 775 $PGHOME \
+    && chmod 664 /etc/passwd \
    # Clean up
-    && apt-get remove -y git python-pip python-setuptools \
+    && apt-get remove -y git python3-pip python3-wheel \
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* /root/.cache

-ADD entrypoint.sh callback.py /
+ADD entrypoint.sh /

 EXPOSE 5432 8008
-ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
+ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 EDITOR=/usr/bin/editor
 USER postgres
 WORKDIR /home/postgres
 CMD ["/bin/bash", "/entrypoint.sh"]
@@ -1,62 +0,0 @@
-#!/usr/bin/env python
-
-import logging
-import os
-import socket
-import sys
-import time
-
-from kubernetes import client as k8s_client, config as k8s_config
-from urllib3.exceptions import HTTPError
-from six.moves.http_client import HTTPException
-
-logger = logging.getLogger(__name__)
-
-
-class CoreV1Api(k8s_client.CoreV1Api):
-
-    def retry(func):
-        def wrapped(*args, **kwargs):
-            count = 0
-            while True:
-                try:
-                    return func(*args, **kwargs)
-                except (HTTPException, HTTPError, socket.error, socket.timeout):
-                    if count >= 10:
-                        raise
-                    logger.info('Throttling API requests...')
-                    time.sleep(2 ** count * 0.5)
-                    count += 1
-        return wrapped
-
-    @retry
-    def patch_namespaced_endpoints(self, *args, **kwargs):
-        return super(CoreV1Api, self).patch_namespaced_endpoints(*args, **kwargs)
-
-
-def patch_master_endpoint(api, namespace, cluster):
-    addresses = [k8s_client.V1EndpointAddress(ip=os.environ['POD_IP'])]
-    ports = [k8s_client.V1EndpointPort(port=5432)]
-    subsets = [k8s_client.V1EndpointSubset(addresses=addresses, ports=ports)]
-    body = k8s_client.V1Endpoints(subsets=subsets)
-    return api.patch_namespaced_endpoints(cluster, namespace, body)
-
-
-def main():
-    logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.INFO)
-    if len(sys.argv) != 4 or sys.argv[1] not in ('on_start', 'on_stop', 'on_role_change'):
-        sys.exit('Usage: %s <action> <role> <cluster_name>', sys.argv[0])
-
-    action, role, cluster = sys.argv[1:4]
-
-    k8s_config.load_incluster_config()
-    k8s_api = CoreV1Api()
-
-    namespace = os.environ['KUBERNETES_NAMESPACE']
-
-    if role == 'master' and action in ('on_start', 'on_role_change'):
-        patch_master_endpoint(k8s_api, namespace, cluster)
-
-
-if __name__ == '__main__':
-    main()
@@ -1,5 +1,12 @@
 #!/bin/bash

+if [[ $UID -ge 10000 ]]; then
+    GID=$(id -g)
+    sed -e "s/^postgres:x:[^:]*:[^:]*:/postgres:x:$UID:$GID:/" /etc/passwd > /tmp/passwd
+    cat /tmp/passwd > /etc/passwd
+    rm /tmp/passwd
+fi
+
 cat > /home/postgres/patroni.yml <<__EOF__
 bootstrap:
  dcs:
@@ -13,24 +20,20 @@ bootstrap:
  - data-checksums
  pg_hba:
  - host all all 0.0.0.0/0 md5
-  - host replication ${PATRONI_REPLICATION_USERNAME} ${POD_IP}/16    md5
+  - host replication ${PATRONI_REPLICATION_USERNAME} ${PATRONI_KUBERNETES_POD_IP}/16 md5
 restapi:
-  connect_address: '${POD_IP}:8008'
+  connect_address: '${PATRONI_KUBERNETES_POD_IP}:8008'
 postgresql:
-  connect_address: '${POD_IP}:5432'
+  connect_address: '${PATRONI_KUBERNETES_POD_IP}:5432'
  authentication:
    superuser:
      password: '${PATRONI_SUPERUSER_PASSWORD}'
    replication:
      password: '${PATRONI_REPLICATION_PASSWORD}'
-  callbacks:
-    on_start: /callback.py
-    on_stop: /callback.py
-    on_role_change: /callback.py
 __EOF__

 unset PATRONI_SUPERUSER_PASSWORD PATRONI_REPLICATION_PASSWORD
 export KUBERNETES_NAMESPACE=$PATRONI_KUBERNETES_NAMESPACE
 export POD_NAME=$PATRONI_NAME

-exec /usr/bin/python /usr/local/bin/patroni /home/postgres/patroni.yml
+exec /usr/bin/python3 /usr/local/bin/patroni /home/postgres/patroni.yml
@@ -0,0 +1,49 @@
+# Patroni OpenShift Configuration
+Patroni can be run in OpenShift. Based on the kubernetes configuration, the Dockerfile and Entrypoint has been modified to support the dynamic UID/GID configuration that is applied in OpenShift. This can be run under the standard `restricted` SCC. 
+
+# Examples
+
+## Create test project
+
+```
+oc new-project patroni-test
+```
+
+## Build the image
+
+Note: Update the references when merged upstream. 
+Note: If deploying as a template for multiple users, the following commands should be performed in a shared namespace like `openshift`. 
+
+```
+oc import-image postgres:10 --confirm -n openshift
+oc new-build https://github.com/zalando/patroni --context-dir=kubernetes -n openshift
+```
+
+## Deploy the Image 
+Two configuration templates exist in [templates](templates) directory: 
+- Patroni Ephemeral
+- Patroni Persistent
+
+The only difference is whether or not the statefulset requests persistent storage. 
+
+## Create the Template
+Install the template into the `openshift` namespace if this should be shared across projects: 
+
+```
+oc create -f templates/template_patroni_ephemeral.yml -n openshift
+```
+
+Then, from your own project: 
+
+```
+oc new-app patroni-pgsql-ephemeral
+```
+
+Once the pods are running, two configmaps should be available: 
+
+```
+$ oc get configmap
+NAME                DATA      AGE
+patroniocp-config   0         1m
+patroniocp-leader   0         1m
+```
@@ -0,0 +1,287 @@
+apiVersion: v1
+kind: Template
+metadata:
+  name: patroni-pgsql-ephemeral
+  annotations:
+    description: |-
+      Patroni Postgresql database cluster, without persistent storage.
+
+      WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing.
+    iconClass: icon-postgresql
+    openshift.io/display-name: Patroni Postgresql (Ephemeral)
+    openshift.io/long-description: This template deploys a a patroni postgresql HA cluster without persistent storage.
+    tags: postgresql
+objects:
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_CLUSTER_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_MASTER_SERVICE_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    selector:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+      role: master
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: v1
+  kind: Secret
+  metadata:
+    name: ${PATRONI_CLUSTER_NAME}
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+  stringData:
+    superuser-password: ${PATRONI_SUPERUSER_PASSWORD}
+    replication-password: ${PATRONI_REPLICATION_PASSWORD}
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_REPLICA_SERVICE_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    selector:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+      role: replica
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: apps/v1
+  kind: StatefulSet
+  metadata:
+    creationTimestamp: null
+    generation: 3
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${APPLICATION_NAME}
+  spec:
+    podManagementPolicy: OrderedReady
+    replicas: 3
+    revisionHistoryLimit: 10
+    selector:
+      matchLabels:
+        application: ${APPLICATION_NAME}
+        cluster-name: ${PATRONI_CLUSTER_NAME}
+    serviceName: ${APPLICATION_NAME}
+    template:
+      metadata:
+        creationTimestamp: null
+        labels:
+          application: ${APPLICATION_NAME}
+          cluster-name: ${PATRONI_CLUSTER_NAME}
+      spec:
+        containers:
+        - env:
+          - name: PATRONI_KUBERNETES_POD_IP
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: status.podIP
+          - name: PATRONI_KUBERNETES_NAMESPACE
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: metadata.namespace
+          - name: PATRONI_KUBERNETES_LABELS
+            value: '{application: ${APPLICATION_NAME}, cluster-name: ${PATRONI_CLUSTER_NAME}}'
+          - name: PATRONI_SUPERUSER_USERNAME
+            value: ${PATRONI_SUPERUSER_USERNAME}
+          - name: PATRONI_SUPERUSER_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                key: superuser-password
+                name: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_REPLICATION_USERNAME
+            value: ${PATRONI_REPLICATION_USERNAME}
+          - name: PATRONI_REPLICATION_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                key: replication-password
+                name: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_SCOPE
+            value: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_NAME
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: metadata.name
+          - name: PATRONI_POSTGRESQL_DATA_DIR
+            value: /home/postgres/pgdata/pgroot/data
+          - name: PATRONI_POSTGRESQL_PGPASS
+            value: /tmp/pgpass
+          - name: PATRONI_POSTGRESQL_LISTEN
+            value: 0.0.0.0:5432
+          - name: PATRONI_RESTAPI_LISTEN
+            value: 0.0.0.0:8008
+          image: docker-registry.default.svc:5000/${NAMESPACE}/patroni:latest
+          imagePullPolicy: IfNotPresent
+          name: ${APPLICATION_NAME}
+          ports:
+          - containerPort: 8008
+            protocol: TCP
+          - containerPort: 5432
+            protocol: TCP
+          resources: {}
+          terminationMessagePath: /dev/termination-log
+          terminationMessagePolicy: File
+          volumeMounts:
+          - mountPath: /home/postgres/pgdata
+            name: pgdata
+        dnsPolicy: ClusterFirst
+        restartPolicy: Always
+        schedulerName: default-scheduler
+        securityContext: {}
+        serviceAccount: ${SERVICE_ACCOUNT}
+        serviceAccountName: ${SERVICE_ACCOUNT}
+        terminationGracePeriodSeconds: 0
+        volumes:
+        - name: pgdata
+          emptyDir: {}
+    updateStrategy:
+      type: OnDelete
+- apiVersion: v1
+  kind: Endpoints
+  metadata:
+    name: ${APPLICATION_NAME}
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+  subsets: []
+- apiVersion: v1
+  kind: ServiceAccount
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: Role
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+  rules:
+  - apiGroups:
+    - ""
+    resources:
+    - configmaps
+    verbs:
+    - create
+    - get
+    - list
+    - patch
+    - update
+    - watch
+    # delete is required only for 'patronictl remove'
+    - delete
+  - apiGroups:
+    - ""
+    resources:
+    - endpoints
+    verbs:
+    - get
+    - patch
+    - update
+    # the following three privileges are necessary only when using endpoints
+    - create
+    - list
+    - watch
+    # delete is required only for for 'patronictl remove'
+    - delete
+  - apiGroups:
+    - ""
+    resources:
+    - pods
+    verbs:
+    - get
+    - list
+    - patch
+    - update
+    - watch
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: RoleBinding
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+  roleRef:
+    apiGroup: rbac.authorization.k8s.io
+    kind: Role
+    name: ${SERVICE_ACCOUNT}
+  subjects:
+  - kind: ServiceAccount
+    name: ${SERVICE_ACCOUNT}
+parameters:
+- description: The name of the application for labelling all artifacts.
+  displayName: Application Name
+  name: APPLICATION_NAME
+  value: patroni-ephemeral
+- description: The name of the patroni-pgsql cluster.
+  displayName: Cluster Name
+  name: PATRONI_CLUSTER_NAME
+  value: patroni-ephemeral
+- description: The name of the OpenShift Service exposed for the patroni-ephemeral-master container.
+  displayName: Master service name.  
+  name: PATRONI_MASTER_SERVICE_NAME
+  value: patroni-ephemeral-master
+- description: The name of the OpenShift Service exposed for the patroni-ephemeral-replica containers.
+  displayName: Replica service name. 
+  name: PATRONI_REPLICA_SERVICE_NAME
+  value: patroni-ephemeral-replica
+- description: Maximum amount of memory the container can use.
+  displayName: Memory Limit
+  name: MEMORY_LIMIT
+  value: 512Mi
+- description: The OpenShift Namespace where the patroni and postgresql ImageStream resides.
+  displayName: ImageStream Namespace
+  name: NAMESPACE
+  value: openshift
+- description: Username of the superuser account for initialization.
+  displayName: Superuser Username
+  name: PATRONI_SUPERUSER_USERNAME
+  value: postgres
+- description: Password of the superuser account for initialization.
+  displayName: Superuser Passsword
+  name: PATRONI_SUPERUSER_PASSWORD
+  value: postgres
+- description: Username of the replication account for initialization.
+  displayName: Replication Username
+  name: PATRONI_REPLICATION_USERNAME
+  value: postgres
+- description: Password of the replication account for initialization.
+  displayName: Repication Passsword
+  name: PATRONI_REPLICATION_PASSWORD
+  value: postgres
+- description: Service account name used for pods and rolebindings to form a cluster in the project. 
+  displayName: Service Account
+  name: SERVICE_ACCOUNT
+  value: patroniocp
@@ -0,0 +1,303 @@
+apiVersion: v1
+kind: Template
+metadata:
+  name: patroni-pgsql-persistent
+  annotations:
+    description: |-
+      Patroni Postgresql database cluster, with persistent storage.
+
+      WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing.
+    iconClass: icon-postgresql
+    openshift.io/display-name: Patroni Postgresql (Persistent)
+    openshift.io/long-description: This template deploys a a patroni postgresql HA cluster without persistent storage.
+    tags: postgresql
+objects:
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_CLUSTER_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_MASTER_SERVICE_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    selector:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+      role: master
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: v1
+  kind: Secret
+  metadata:
+    name: ${PATRONI_CLUSTER_NAME}
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+  stringData:
+    superuser-password: ${PATRONI_SUPERUSER_PASSWORD}
+    replication-password: ${PATRONI_REPLICATION_PASSWORD}
+- apiVersion: v1
+  kind: Service
+  metadata:
+    creationTimestamp: null
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${PATRONI_REPLICA_SERVICE_NAME}
+  spec:
+    ports:
+    - port: 5432
+      protocol: TCP
+      targetPort: 5432
+    selector:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+      role: replica
+    sessionAffinity: None
+    type: ClusterIP
+  status:
+    loadBalancer: {}
+- apiVersion: apps/v1
+  kind: StatefulSet
+  metadata:
+    creationTimestamp: null
+    generation: 3
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+    name: ${APPLICATION_NAME}
+  spec:
+    podManagementPolicy: OrderedReady
+    replicas: 3
+    revisionHistoryLimit: 10
+    selector:
+      matchLabels:
+        application: ${APPLICATION_NAME}
+        cluster-name: ${PATRONI_CLUSTER_NAME}
+    serviceName: ${APPLICATION_NAME}
+    template:
+      metadata:
+        creationTimestamp: null
+        labels:
+          application: ${APPLICATION_NAME}
+          cluster-name: ${PATRONI_CLUSTER_NAME}
+      spec:
+        containers:
+        - env:
+          - name: PATRONI_KUBERNETES_POD_IP
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: status.podIP
+          - name: PATRONI_KUBERNETES_NAMESPACE
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: metadata.namespace
+          - name: PATRONI_KUBERNETES_LABELS
+            value: '{application: ${APPLICATION_NAME}, cluster-name: ${PATRONI_CLUSTER_NAME}}'
+          - name: PATRONI_SUPERUSER_USERNAME
+            value: ${PATRONI_SUPERUSER_USERNAME}
+          - name: PATRONI_SUPERUSER_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                key: superuser-password
+                name: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_REPLICATION_USERNAME
+            value: ${PATRONI_REPLICATION_USERNAME}
+          - name: PATRONI_REPLICATION_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                key: replication-password
+                name: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_SCOPE
+            value: ${PATRONI_CLUSTER_NAME}
+          - name: PATRONI_NAME
+            valueFrom:
+              fieldRef:
+                apiVersion: v1
+                fieldPath: metadata.name
+          - name: PATRONI_POSTGRESQL_DATA_DIR
+            value: /home/postgres/pgdata/pgroot/data
+          - name: PATRONI_POSTGRESQL_PGPASS
+            value: /tmp/pgpass
+          - name: PATRONI_POSTGRESQL_LISTEN
+            value: 0.0.0.0:5432
+          - name: PATRONI_RESTAPI_LISTEN
+            value: 0.0.0.0:8008
+          image: docker-registry.default.svc:5000/${NAMESPACE}/patroni:latest
+          imagePullPolicy: IfNotPresent
+          name: ${APPLICATION_NAME}
+          ports:
+          - containerPort: 8008
+            protocol: TCP
+          - containerPort: 5432
+            protocol: TCP
+          resources: {}
+          terminationMessagePath: /dev/termination-log
+          terminationMessagePolicy: File
+          volumeMounts:
+          - mountPath: /home/postgres/pgdata
+            name: ${APPLICATION_NAME}
+        dnsPolicy: ClusterFirst
+        restartPolicy: Always
+        schedulerName: default-scheduler
+        securityContext: {}
+        serviceAccount: ${SERVICE_ACCOUNT}
+        serviceAccountName: ${SERVICE_ACCOUNT}
+        terminationGracePeriodSeconds: 0
+        volumes:
+        - name: ${APPLICATION_NAME}
+          persistentVolumeClaim: 
+            claimName: ${APPLICATION_NAME}
+    volumeClaimTemplates:
+    - metadata:
+        labels:
+          application: ${APPLICATION_NAME}
+        name: ${APPLICATION_NAME}
+      spec:
+        accessModes:
+        - ReadWriteOnce
+        resources:
+          requests:
+            storage: ${PVC_SIZE}
+    updateStrategy:
+      type: OnDelete
+- apiVersion: v1
+  kind: Endpoints
+  metadata:
+    name: ${APPLICATION_NAME}
+    labels:
+      application: ${APPLICATION_NAME}
+      cluster-name: ${PATRONI_CLUSTER_NAME}
+  subsets: []
+- apiVersion: v1
+  kind: ServiceAccount
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: Role
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+  rules:
+  - apiGroups:
+    - ""
+    resources:
+    - configmaps
+    verbs:
+    - create
+    - get
+    - list
+    - patch
+    - update
+    - watch
+    # delete is required only for 'patronictl remove'
+    - delete
+  - apiGroups:
+    - ""
+    resources:
+    - endpoints
+    verbs:
+    - get
+    - patch
+    - update
+    # the following three privileges are necessary only when using endpoints
+    - create
+    - list
+    - watch
+    # delete is required only for for 'patronictl remove'
+    - delete
+  - apiGroups:
+    - ""
+    resources:
+    - pods
+    verbs:
+    - get
+    - list
+    - patch
+    - update
+    - watch
+- apiVersion: rbac.authorization.k8s.io/v1
+  kind: RoleBinding
+  metadata:
+    name: ${SERVICE_ACCOUNT}
+  roleRef:
+    apiGroup: rbac.authorization.k8s.io
+    kind: Role
+    name: ${SERVICE_ACCOUNT}
+  subjects:
+  - kind: ServiceAccount
+    name: ${SERVICE_ACCOUNT}
+parameters:
+- description: The name of the application for labelling all artifacts.
+  displayName: Application Name
+  name: APPLICATION_NAME
+  value: patroni-persistent
+- description: The name of the patroni-pgsql cluster.
+  displayName: Cluster Name
+  name: PATRONI_CLUSTER_NAME
+  value: patroni-persistent
+- description: The name of the OpenShift Service exposed for the patroni-persistent-master container.
+  displayName: Master service name.  
+  name: PATRONI_MASTER_SERVICE_NAME
+  value: patroni-persistent-master
+- description: The name of the OpenShift Service exposed for the patroni-persistent-replica containers.
+  displayName: Replica service name. 
+  name: PATRONI_REPLICA_SERVICE_NAME
+  value: patroni-persistent-replica
+- description: Maximum amount of memory the container can use.
+  displayName: Memory Limit
+  name: MEMORY_LIMIT
+  value: 512Mi
+- description: The OpenShift Namespace where the patroni and postgresql ImageStream resides.
+  displayName: ImageStream Namespace
+  name: NAMESPACE
+  value: openshift
+- description: Username of the superuser account for initialization.
+  displayName: Superuser Username
+  name: PATRONI_SUPERUSER_USERNAME
+  value: postgres
+- description: Password of the superuser account for initialization.
+  displayName: Superuser Passsword
+  name: PATRONI_SUPERUSER_PASSWORD
+  value: postgres
+- description: Username of the replication account for initialization.
+  displayName: Replication Username
+  name: PATRONI_REPLICATION_USERNAME
+  value: postgres
+- description: Password of the replication account for initialization.
+  displayName: Repication Passsword
+  name: PATRONI_REPLICATION_PASSWORD
+  value: postgres
+- description: Service account name used for pods and rolebindings to form a cluster in the project. 
+  displayName: Service Account
+  name: SERVICE_ACCOUNT
+  value: patroni-persistent
+- description: The size of the persistent volume to create. 
+  displayName: Persistent Volume Size
+  name: PVC_SIZE
+  value: 5Gi
@@ -0,0 +1,43 @@
+pipeline {
+  agent any
+  stages {
+    stage ('Deploy test pod'){
+      when {
+          expression {
+            openshift.withCluster() {
+              openshift.withProject() {
+                return !openshift.selector( "dc", "pgbench" ).exists()
+              }
+            }
+          }
+        }
+      steps {
+        script {
+            openshift.withCluster() {
+                openshift.withProject() {
+                  def pgbench = openshift.newApp( "https://github.com/stewartshea/docker-pgbench/", "--name=pgbench", "-e PGPASSWORD=postgres", "-e PGUSER=postgres", "-e PGHOST=patroni-persistent-master", "-e PGDATABASE=postgres", "-e TEST_CLIENT_COUNT=20", "-e TEST_DURATION=120" )
+                  def pgbenchdc = openshift.selector( "dc", "pgbench" )
+                  timeout(5) { 
+                    pgbenchdc.rollout().status()
+                    }
+                  }
+                }
+            }
+        }
+    }
+    stage ('Run benchmark Test'){
+      steps {
+          sh '''
+          oc exec $(oc get pods -l app=pgbench | grep Running | awk '{print $1}')  ./test.sh
+          '''
+      }
+    }
+    stage ('Clean up pgtest pod'){
+      steps {
+          sh '''
+          oc delete all -l app=pgbench
+          '''
+      }
+    }
+  }
+}
@@ -0,0 +1,2 @@
+# Jenkins Test
+This pipeline test will create a separate deployment config for a pgbench pod and execute a test against the patroni cluster. This is a sample and should be customized. 
@@ -28,7 +28,7 @@ spec:
        - mountPath: /home/postgres/pgdata
          name: pgdata
        env:
-        - name: POD_IP
+        - name: PATRONI_KUBERNETES_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
@@ -36,6 +36,8 @@ spec:
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
+        - name: PATRONI_KUBERNETES_USE_ENDPOINTS
+          value: 'true'
        - name: PATRONI_KUBERNETES_LABELS
          value: '{application: patroni, cluster-name: patronidemo}'
        - name: PATRONI_SUPERUSER_USERNAME
@@ -14,6 +14,7 @@ class Patroni(object):
        from patroni.config import Config
        from patroni.dcs import get_dcs
        from patroni.ha import Ha
+        from patroni.log import PatroniLogger
        from patroni.postgresql import Postgresql
        from patroni.version import __version__
        from patroni.watchdog import Watchdog
@@ -21,7 +22,9 @@ class Patroni(object):
        self.setup_signal_handlers()

        self.version = __version__
+        self.logger = PatroniLogger()
        self.config = Config()
+        self.logger.reload_config(self.config.get('log', {}))
        self.dcs = get_dcs(self.config)
        self.watchdog = Watchdog(self.config)
        self.load_dynamic_configuration()
@@ -49,6 +52,7 @@ class Patroni(object):
                break
            except DCSError:
                logger.warning('Can not get cluster from dcs')
+                time.sleep(5)

    def get_tags(self):
        return {tag: value for tag, value in self.config.get('tags', {}).items()
@@ -65,6 +69,7 @@ class Patroni(object):
    def reload_config(self):
        try:
            self.tags = self.get_tags()
+            self.logger.reload_config(self.config.get('log', {}))
            self.dcs.reload_config(self.config)
            self.watchdog.reload_config(self.config)
            self.api.reload_config(self.config['restapi'])
@@ -138,12 +143,6 @@ class Patroni(object):


 def patroni_main():
-    logformat = os.environ.get('PATRONI_LOGFORMAT', '%(asctime)s %(levelname)s: %(message)s')
-    loglevel = os.environ.get('PATRONI_LOGLEVEL', 'INFO')
-    requests_loglevel = os.environ.get('PATRONI_REQUESTS_LOGLEVEL', 'WARNING')
-    logging.basicConfig(format=logformat, level=loglevel)
-    logging.getLogger('requests').setLevel(requests_loglevel)
-
    patroni = Patroni()
    try:
        patroni.run()
@@ -151,6 +150,7 @@ def patroni_main():
        pass
    finally:
        patroni.shutdown()
+        logging.shutdown()


 def pg_ctl_start(args):
@@ -3,6 +3,7 @@ import json
 import logging
 import psycopg2
 import time
+import traceback
 import dateutil.parser
 import datetime
 import os
@@ -87,10 +88,14 @@ class RestApiHandler(BaseHTTPRequestHandler):
        replica_status_code = 200 if not patroni.noloadbalance and response.get('role') == 'replica' else 503
        status_code = 503

-        if patroni.config.is_standby_cluster and ('standby_leader' in path or 'standby-leader' in path):
+        if patroni.ha.is_standby_cluster() and ('standby_leader' in path or 'standby-leader' in path):
            status_code = 200 if patroni.ha.is_leader() else 503
        elif 'master' in path or 'leader' in path or 'primary' in path:
-            status_code = 200 if patroni.ha.is_leader() else 503
+            # Round-robing across all masters in pause mode if DCS is not accessible
+            if not cluster and patroni.ha.is_paused():
+                status_code = 200 if response['role'] == 'master' else 503
+            else:
+                status_code = 200 if patroni.ha.is_leader() else 503
        elif 'replica' in path:
            status_code = replica_status_code
        elif cluster:  # dcs is available
@@ -408,17 +413,20 @@ class RestApiHandler(BaseHTTPRequestHandler):
                raise RetryFailedError('')
            stmt = ("WITH replication_info AS ("
                    "SELECT usename, application_name, client_addr, state, sync_state, sync_priority"
-                    " FROM pg_stat_replication) SELECT"
-                    " to_char(pg_postmaster_start_time(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),"
-                    " CASE WHEN pg_is_in_recovery() THEN 0"
-                    " ELSE ('x' || SUBSTR(pg_{0}file_name(pg_current_{0}_{1}()), 1, 8))::bit(32)::int END,"
-                    " CASE WHEN pg_is_in_recovery() THEN 0"
-                    " ELSE pg_{0}_{1}_diff(pg_current_{0}_{1}(), '0/0')::bigint END,"
-                    " pg_{0}_{1}_diff(COALESCE(pg_last_{0}_receive_{1}(), pg_last_{0}_replay_{1}()), '0/0')::bigint,"
-                    " pg_{0}_{1}_diff(pg_last_{0}_replay_{1}(), '0/0')::bigint,"
-                    " to_char(pg_last_xact_replay_timestamp(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),"
-                    " pg_is_in_recovery() AND pg_is_{0}_replay_paused(),"
-                    " (SELECT array_to_json(array_agg(row_to_json(ri))) FROM replication_info ri)")
+                    " FROM pg_catalog.pg_stat_replication) SELECT"
+                    " pg_catalog.to_char(pg_catalog.pg_postmaster_start_time(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),"
+                    " CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0"
+                    " ELSE ('x' || pg_catalog.substr(pg_catalog.pg_{0}file_name("
+                    "pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END,"
+                    " CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0"
+                    " ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), '0/0')::bigint END,"
+                    " pg_catalog.pg_{0}_{1}_diff(COALESCE(pg_catalog.pg_last_{0}_receive_{1}(),"
+                    " pg_catalog.pg_last_{0}_replay_{1}()), '0/0')::bigint,"
+                    " pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), '0/0')::bigint,"
+                    " pg_catalog.to_char(pg_catalog.pg_last_xact_replay_timestamp(), 'YYYY-MM-DD HH24:MI:SS.MS TZ'),"
+                    " pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_{0}_replay_paused(), "
+                    "(SELECT pg_catalog.array_to_json(pg_catalog.array_agg("
+                    "pg_catalog.row_to_json(ri))) FROM replication_info ri)")

            row = self.query(stmt.format(self.server.patroni.postgresql.wal_name,
                                         self.server.patroni.postgresql.lsn_name), retry=retry)[0]
@@ -540,3 +548,9 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
                and self.__initialize(config):
            self.start()
        self.__set_config_parameters(config)
+
+    @staticmethod
+    def handle_error(request, client_address):
+        address, port = client_address
+        logger.warning('Exception happened during processing of request from {}:{}'.format(address, port))
+        logger.warning(traceback.format_exc())
@@ -2,14 +2,13 @@ import json
 import logging
 import os
 import shutil
-import six
 import sys
 import tempfile
 import yaml

 from collections import defaultdict
 from copy import deepcopy
-from patroni.dcs import ClusterConfig, is_standby_cluster
+from patroni.dcs import ClusterConfig
 from patroni.postgresql import Postgresql
 from patroni.utils import deep_compare, parse_bool, parse_int, patch_config
 from requests.structures import CaseInsensitiveDict
@@ -44,6 +43,7 @@ class Config(object):
    __DEFAULT_CONFIG = {
        'ttl': 30, 'loop_wait': 10, 'retry_timeout': 10,
        'maximum_lag_on_failover': 1048576,
+        'check_timeline': False,
        'master_start_timeout': 300,
        'synchronous_mode': False,
        'synchronous_mode_strict': False,
@@ -99,10 +99,6 @@ class Config(object):
    def dynamic_configuration(self):
        return deepcopy(self._dynamic_configuration)

-    @property
-    def is_standby_cluster(self):
-        return is_standby_cluster(self._dynamic_configuration.get('standby_cluster'))
-
    def check_mode(self, mode):
        return bool(parse_bool(self._dynamic_configuration.get(mode)))

@@ -199,12 +195,9 @@ class Config(object):
                    elif name not in ('connect_address', 'listen', 'data_dir', 'pgpass', 'authentication'):
                        config['postgresql'][name] = deepcopy(value)
            elif name == 'standby_cluster':
-                allowed_keys = self.__DEFAULT_CONFIG['standby_cluster'].keys()
-                expected = {
-                    k: v for k, v in (value or {}).items()
-                    if (k in allowed_keys and isinstance(v, six.string_types))
-                }
-                config['standby_cluster'].update(expected)
+                for name, value in (value or {}).items():
+                    if name in self.__DEFAULT_CONFIG['standby_cluster']:
+                        config['standby_cluster'][name] = deepcopy(value)
            elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overriden from DCS
                if name in ('synchronous_mode', 'synchronous_mode_strict'):
                    config[name] = value
@@ -224,6 +217,15 @@ class Config(object):
            if value:
                ret[param] = value

+        def _fix_log_env(name, oldname):
+            value = _popenv(oldname)
+            name = Config.PATRONI_ENV_PREFIX + 'LOG_' + name.upper()
+            if value and name not in os.environ:
+                os.environ[name] = value
+
+        for name, oldname in (('level', 'loglevel'), ('format', 'logformat'), ('dateformat', 'log_datefmt')):
+            _fix_log_env(name, oldname)
+
        def _set_section_values(section, params):
            for param in params:
                value = _popenv(section + '_' + param)
@@ -232,6 +234,22 @@ class Config(object):

        _set_section_values('restapi', ['listen', 'connect_address', 'certfile', 'keyfile'])
        _set_section_values('postgresql', ['listen', 'connect_address', 'data_dir', 'pgpass', 'bin_dir'])
+        _set_section_values('log', ['level', 'format', 'dateformat', 'dir', 'file_size', 'file_num', 'loggers'])
+
+        def _parse_dict(value):
+            if not value.strip().startswith('{'):
+                value = '{{{0}}}'.format(value)
+            try:
+                return yaml.safe_load(value)
+            except Exception:
+                logger.exception('Exception when parsing dict %s', value)
+                return None
+
+        value = ret.get('log', {}).pop('loggers', None)
+        if value:
+            value = _parse_dict(value)
+            if value:
+                ret['log']['loggers'] = value

        def _get_auth(name):
            ret = {}
@@ -270,22 +288,18 @@ class Config(object):
                name, suffix = (param[8:].split('_', 1) + [''])[:2]
                if name and suffix:
                    # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
-                    if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT',
-                                  'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'NAMESPACE', 'CONTEXT',
-                                  'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS'):
+                    if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY', 'VERIFY',
+                                  'TOKEN', 'CHECKS', 'DC', 'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'NAMESPACE',
+                                  'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS'):
                        value = os.environ.pop(param)
                        if suffix == 'PORT':
                            value = value and parse_int(value)
                        elif suffix in ('HOSTS', 'PORTS', 'CHECKS'):
                            value = value and _parse_list(value)
                        elif suffix == 'LABELS':
-                            if not value.strip().startswith('{'):
-                                value = '{{{0}}}'.format(value)
-                            try:
-                                value = yaml.safe_load(value)
-                            except Exception:
-                                logger.exception('Exception when parsing dict %s', value)
-                                value = None
+                            value = _parse_dict(value)
+                        elif suffix == 'REGISTER_SERVICE':
+                            value = parse_bool(value)
                        if value:
                            ret[name.lower()][suffix.lower()] = value
                    # PATRONI_<username>_PASSWORD=<password>, PATRONI_<username>_OPTIONS=<option1,option2,...>
@@ -344,7 +358,7 @@ class Config(object):
            'scope',
            'retry_timeout',
            'synchronous_mode',
-            'maximum_lag_on_failover'
+            'synchronous_mode_strict',
        )

        pg_config.update({p: config[p] for p in updated_fields if p in config})
@@ -252,7 +252,7 @@ def get_cursor(cluster, connect_parameters, role='master', member=None):
    if role == 'any':
        return cursor

-    cursor.execute('SELECT pg_is_in_recovery()')
+    cursor.execute('SELECT pg_catalog.pg_is_in_recovery()')
    in_recovery = cursor.fetchone()[0]

    if in_recovery and role == 'replica' or not in_recovery and role == 'master':
@@ -392,7 +392,7 @@ def query_member(cluster, cursor, member, role, command, connect_parameters):
            logging.debug(message)
            return [[timestamp(0), message]], None

-        cursor.execute('SELECT pg_is_in_recovery()')
+        cursor.execute('SELECT pg_catalog.pg_is_in_recovery()')
        in_recovery = cursor.fetchone()[0]

        if in_recovery and role == 'master' or not in_recovery and role == 'replica':
@@ -10,16 +10,34 @@ import re
 import six
 import sys

-from collections import namedtuple
+from collections import defaultdict, namedtuple
+from copy import deepcopy
 from patroni.exceptions import PatroniException
 from patroni.utils import parse_bool
 from random import randint
 from six.moves.urllib_parse import urlparse, urlunparse, parse_qsl
 from threading import Event, Lock

+slot_name_re = re.compile('^[a-z0-9_]{1,63}$')
 logger = logging.getLogger(__name__)


+def slot_name_from_member_name(member_name):
+    """Translate member name to valid PostgreSQL slot name.
+
+    PostgreSQL replication slot names must be valid PostgreSQL names. This function maps the wider space of
+    member names to valid PostgreSQL names. Names are lowercased, dashes and periods common in hostnames
+    are replaced with underscores, other characters are encoded as their unicode codepoint. Name is truncated
+    to 64 characters. Multiple different member names may map to a single slot name."""
+
+    def replace_char(match):
+        c = match.group(0)
+        return '_' if c in '-.' else "u{:04d}".format(ord(c))
+
+    slot_name = re.sub('[^a-z0-9_]', replace_char, member_name.lower())
+    return slot_name[0:63]
+
+
 def parse_connection_string(value):
    """Original Governor stores connection strings for each cluster members if a following format:
        postgres://{username}:{password}@{connect_address}/postgres
@@ -189,13 +207,12 @@ class RemoteMember(Member):
                'create_replica_methods',
                'restore_command',
                'archive_cleanup_command',
-                'recovery_min_apply_delay')
+                'recovery_min_apply_delay',
+                'no_replication_slot')

    def __getattr__(self, name):
-        if name not in RemoteMember.allowed_keys():
-            return
-
-        return self.data.get(name)
+        if name in RemoteMember.allowed_keys():
+            return self.data.get(name)


 class Leader(namedtuple('Leader', 'index,session,member')):
@@ -275,14 +292,24 @@ class ClusterConfig(namedtuple('ClusterConfig', 'index,data,modify_index')):
    def from_node(index, data, modify_index=None):
        """
        >>> ClusterConfig.from_node(1, '{') is None
-        True
+        False
        """

        try:
            data = json.loads(data)
        except (TypeError, ValueError):
-            return None
-        return ClusterConfig(index, data, modify_index or index)
+            data = None
+            modify_index = 0
+        if not isinstance(data, dict):
+            data = {}
+        return ClusterConfig(index, data, index if modify_index is None else modify_index)
+
+    @property
+    def permanent_slots(self):
+        return isinstance(self.data, dict) and (
+                self.data.get('permanent_replication_slots') or
+                self.data.get('permanent_slots') or self.data.get('slots')
+        ) or {}


 class SyncState(namedtuple('SyncState', 'index,leader,sync_standby')):
@@ -341,7 +368,7 @@ class SyncState(namedtuple('SyncState', 'index,leader,sync_standby')):
        return name is not None and name in (self.leader, self.sync_standby)


-class TimelineHistory(namedtuple('TimelineHistory', 'index,lines')):
+class TimelineHistory(namedtuple('TimelineHistory', 'index,value,lines')):
    """Object representing timeline history file"""

    @staticmethod
@@ -357,7 +384,7 @@ class TimelineHistory(namedtuple('TimelineHistory', 'index,lines')):
            lines = None
        if not isinstance(lines, list):
            lines = []
-        return TimelineHistory(index, lines)
+        return TimelineHistory(index, value, lines)


 class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operation,members,failover,sync,history')):
@@ -398,8 +425,84 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
    def is_synchronous_mode(self):
        return self.check_mode('synchronous_mode')

-    def is_standby_cluster(self):
-        return is_standby_cluster(self.config and self.config.data.get('standby_cluster'))
+    def get_replication_slots(self, name, role):
+        # if the replicatefrom tag is set on the member - we should not create the replication slot for it on
+        # the current master, because that member would replicate from elsewhere. We still create the slot if
+        # the replicatefrom destination member is currently not a member of the cluster (fallback to the
+        # master), or if replicatefrom destination member happens to be the current master
+        if role in ('master', 'standby_leader'):
+            slot_members = [m.name for m in self.members if m.name != name and
+                            (m.replicatefrom is None or m.replicatefrom == name or
+                             not self.has_member(m.replicatefrom))]
+            permanent_slots = (self.config and self.config.permanent_slots or {}).copy()
+        else:
+            # only manage slots for replicas that replicate from this one, except for the leader among them
+            slot_members = [m.name for m in self.members if m.replicatefrom == name and m.name != self.leader.name]
+            permanent_slots = {}
+
+        slots = {slot_name_from_member_name(name): {'type': 'physical'} for name in slot_members}
+
+        if len(slots) < len(slot_members):
+            # Find which names are conflicting for a nicer error message
+            slot_conflicts = defaultdict(list)
+            for name in slot_members:
+                slot_conflicts[slot_name_from_member_name(name)].append(name)
+            logger.error("Following cluster members share a replication slot name: %s",
+                         "; ".join("{} map to {}".format(", ".join(v), k)
+                                   for k, v in slot_conflicts.items() if len(v) > 1))
+
+        # "merge" replication slots for members with permanent_replication_slots
+        for name, value in permanent_slots.items():
+            if not slot_name_re.match(name):
+                logger.error("Invalid permanent replication slot name '%s'", name)
+                logger.error("Slot name may only contain lower case letters, numbers, and the underscore chars")
+                continue
+
+            if name in slots:
+                logger.error("Permanent replication slot {'%s': %s} is conflicting with" +
+                             " physical replication slot for cluster member", name, value)
+                continue
+
+            value = deepcopy(value)
+            if not value:
+                value = {'type': 'physical'}
+
+            if isinstance(value, dict):
+                if 'type' not in value:
+                    value['type'] = 'logical' if value.get('database') and value.get('plugin') else 'physical'
+
+                if value['type'] == 'physical' or value['type'] == 'logical' \
+                        and value.get('database') and value.get('plugin'):
+                    slots[name] = value
+                    continue
+
+            logger.error("Bad value for slot '%s' in permanent_slots: %s", name, permanent_slots[name])
+
+        return slots
+
+    def has_permanent_logical_slots(self, name):
+        slots = self.get_replication_slots(name, 'master').values()
+        return any(v for v in slots if v.get("type") == "logical")
+
+    @property
+    def timeline(self):
+        """
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, 0).timeline
+        0
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[]')).timeline
+        1
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[["a"]]')).timeline
+        0
+        """
+        if self.history:
+            if self.history.lines:
+                try:
+                    return int(self.history.lines[-1][0]) + 1
+                except Exception:
+                    logger.error('Failed to parse cluster history from DCS: %s', self.history.lines)
+            elif self.history.value == '[]':
+                return 1
+        return 0


@six.add_metaclass(abc.ABCMeta)
@@ -537,7 +640,7 @@ class AbstractDCS(object):
        You have to use CAS (Compare And Swap) operation in order to update leader key,
        for example for etcd `prevValue` parameter must be used."""

-    def update_leader(self, last_operation):
+    def update_leader(self, last_operation, access_is_restricted=False):
        """Update leader key (or session) ttl and optime/leader

        :param last_operation: absolute xlog location in bytes
@@ -654,14 +757,3 @@ class AbstractDCS(object):

        self.event.wait(timeout)
        return self.event.isSet()
-
-
-def is_standby_cluster(config):
-    """ Check whether or not provided configuration describes a standby cluster.
-        Config can be both patroni config or cluster.config.data
-    """
-    return isinstance(config, dict) and (
-        config.get('host') or
-        config.get('port') or
-        config.get('restore_command')
-    )
@@ -27,10 +27,14 @@ class ConsulInternalError(ConsulException):
    """An internal Consul server error occurred"""


-class InvalidSessionTTL(ConsulInternalError):
+class InvalidSessionTTL(ConsulException):
    """Session TTL is too small or too big"""


+class InvalidSession(ConsulException):
+    """invalid session"""
+
+
 class HTTPClient(object):

    def __init__(self, host='127.0.0.1', port=8500, token=None, scheme='http', verify=True, cert=None, ca_cert=None):
@@ -72,6 +76,8 @@ class HTTPClient(object):
            msg = '{0} {1}'.format(response.status, data)
            if data.startswith('Invalid Session TTL'):
                raise InvalidSessionTTL(msg)
+            elif data.startswith('invalid session'):
+                raise InvalidSession(msg)
            else:
                raise ConsulInternalError(msg)
        return base.Response(response.status, response.headers, data)
@@ -338,17 +344,15 @@ class Consul(AbstractDCS):
            logger.exception('get_cluster')
            raise ConsulError('Consul is not responding properly')

+    @catch_consul_errors
    def touch_member(self, data, ttl=None, permanent=False):
        cluster = self.cluster
        member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
        create_member = not permanent and self.refresh_session()

        if member and (create_member or member.session != self._session):
-            try:
-                self._client.kv.delete(self.member_path)
-                create_member = True
-            except Exception:
-                return False
+            self._client.kv.delete(self.member_path)
+            create_member = True

        if not create_member and member and deep_compare(data, member.data):
            return True
@@ -359,6 +363,9 @@ class Consul(AbstractDCS):
            if self._register_service:
                self.update_service(not create_member and member and member.data or {}, data)
            return True
+        except InvalidSession:
+            self._session = None
+            logger.error('Our session disappeared from Consul, can not "touch_member"')
        except Exception:
            logger.exception('touch_member')
        return False
@@ -382,7 +389,8 @@ class Consul(AbstractDCS):
        api_parts = urlparse(data['api_url'])
        api_parts = api_parts._replace(path='/{0}'.format(role))
        conn_parts = urlparse(data['conn_url'])
-        check = base.Check.http(api_parts.geturl(), self._service_check_interval, deregister=self._client.http.ttl * 10)
+        check = base.Check.http(api_parts.geturl(), self._service_check_interval,
+                                deregister='{0}s'.format(self._client.http.ttl * 10))
        params = {
            'service_id': '{0}/{1}'.format(self._scope, self._name),
            'address': conn_parts.hostname,
@@ -416,14 +424,21 @@ class Consul(AbstractDCS):
            return self._update_service(new_data)

    @catch_consul_errors
-    def _do_attempt_to_acquire_leader(self, kwargs):
-        return self.retry(self._client.kv.put, self.leader_path, self._name, **kwargs)
+    def _do_attempt_to_acquire_leader(self, permanent):
+        try:
+            kwargs = {} if permanent else {'acquire': self._session}
+            return self.retry(self._client.kv.put, self.leader_path, self._name, **kwargs)
+        except InvalidSession:
+            self._session = None
+            logger.error('Our session disappeared from Consul. Will try to get a new one and retry attempt')
+            self.refresh_session()
+            return self.retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)

    def attempt_to_acquire_leader(self, permanent=False):
        if not self._session and not permanent:
            self.refresh_session()

-        ret = self._do_attempt_to_acquire_leader({} if permanent else {'acquire': self._session})
+        ret = self._do_attempt_to_acquire_leader(permanent)
        if not ret:
            logger.info('Could not take out TTL lock')

@@ -501,4 +516,5 @@ class Consul(AbstractDCS):
        try:
            return super(Consul, self).watch(None, timeout)
        finally:
+            self._last_session_refresh = 0
            self.event.clear()
@@ -152,7 +152,8 @@ class Kubernetes(AbstractDCS):

            # get global dynamic configuration
            config = ClusterConfig.from_node(metadata and metadata.resource_version,
-                                             annotations.get(self._CONFIG) or '{}')
+                                             annotations.get(self._CONFIG) or '{}',
+                                             metadata.resource_version if self._CONFIG in annotations else 0)

            # get timeline history
            history = TimelineHistory.from_node(metadata and metadata.resource_version,
@@ -279,7 +280,7 @@ class Kubernetes(AbstractDCS):
    def _update_leader(self):
        """Unused"""

-    def update_leader(self, last_operation):
+    def update_leader(self, last_operation, access_is_restricted=False):
        now = datetime.datetime.now(tzutc).isoformat()
        annotations = {self._LEADER: self._name, 'ttl': str(self._ttl), 'renewTime': now,
                       'acquireTime': self._leader_observed_record.get('acquireTime') or now,
@@ -287,7 +288,11 @@ class Kubernetes(AbstractDCS):
        if last_operation:
            annotations[self._OPTIME] = last_operation

-        ret = self.patch_or_create(self.leader_path, annotations, self._leader_resource_version, subsets=self.__subsets)
+        subsets = self.__subsets
+        if subsets is not None and access_is_restricted:
+            subsets = []
+
+        ret = self.patch_or_create(self.leader_path, annotations, self._leader_resource_version, subsets=subsets)
        if ret:
            self._leader_resource_version = ret.metadata.resource_version
        return ret
@@ -307,7 +312,8 @@ class Kubernetes(AbstractDCS):
            else:
                annotations['acquireTime'] = self._leader_observed_record.get('acquireTime') or now
            annotations['transitions'] = str(transitions)
-        ret = self.patch_or_create(self.leader_path, annotations, self._leader_resource_version, subsets=self.__subsets)
+        subsets = [] if self.__subsets else None
+        ret = self.patch_or_create(self.leader_path, annotations, self._leader_resource_version, subsets=subsets)
        if ret:
            self._leader_resource_version = ret.metadata.resource_version
        else:
@@ -1,5 +1,6 @@
 import json
 import logging
+import select
 import time

 from kazoo.client import KazooClient, KazooState, KazooRetry
@@ -37,12 +38,21 @@ class PatroniSequentialThreadingHandler(SequentialThreadingHandler):
                     `connect_timeout` (negotiated session timeout) as the second element."""

        args = list(args)
-        if len(args) == 1:
+        if len(args) == 0:  # kazoo 2.6.0 slightly changed the way how it calls create_connection method
+            kwargs['timeout'] = max(self._connect_timeout, kwargs.get('timeout', self._connect_timeout*10)/10.0)
+        elif len(args) == 1:
            args.append(self._connect_timeout)
        else:
            args[1] = max(self._connect_timeout, args[1]/10.0)
        return super(PatroniSequentialThreadingHandler, self).create_connection(*args, **kwargs)

+    def select(self, *args, **kwargs):
+        """Python3 raises `ValueError` if socket is closed, because fd == -1"""
+        try:
+            return super(PatroniSequentialThreadingHandler, self).select(*args, **kwargs)
+        except ValueError as e:
+            raise select.error(9, str(e))
+

 class ZooKeeper(AbstractDCS):

@@ -12,7 +12,7 @@ from collections import namedtuple
 from multiprocessing.pool import ThreadPool
 from patroni.async_executor import AsyncExecutor, CriticalTask
 from patroni.exceptions import DCSError, PostgresConnectionException, PatroniException
-from patroni.postgresql import ACTION_ON_START
+from patroni.postgresql import ACTION_ON_START, ACTION_ON_ROLE_CHANGE
 from patroni.utils import polling_loop, tzutc
 from patroni.dcs import RemoteMember
 from threading import RLock
@@ -20,25 +20,28 @@ from threading import RLock
 logger = logging.getLogger(__name__)


-class _MemberStatus(namedtuple('_MemberStatus', 'member,reachable,in_recovery,wal_position,tags,watchdog_failed')):
+class _MemberStatus(namedtuple('_MemberStatus', ['member', 'reachable', 'in_recovery', 'timeline',
+                                                 'wal_position', 'tags', 'watchdog_failed'])):
    """Node status distilled from API response:

        member - dcs.Member object of the node
        reachable - `!False` if the node is not reachable or is not responding with correct JSON
        in_recovery - `!True` if pg_is_in_recovery() == true
-        wal_position - value of `replayed_location` or `location` from JSON, dependin on its role.
+        timeline - timeline value from JSON
+        wal_position - maximum value of `replayed_location` or `received_location` from JSON
        tags - dictionary with values of different tags (i.e. nofailover)
        watchdog_failed - indicates that watchdog is required by configuration but not available or failed
    """
    @classmethod
    def from_api_response(cls, member, json):
        is_master = json['role'] == 'master'
+        timeline = json.get('timeline', 0)
        wal = not is_master and max(json['xlog'].get('received_location', 0), json['xlog'].get('replayed_location', 0))
-        return cls(member, True, not is_master, wal, json.get('tags', {}), json.get('watchdog_failed', False))
+        return cls(member, True, not is_master, timeline, wal, json.get('tags', {}), json.get('watchdog_failed', False))

    @classmethod
    def unknown(cls, member):
-        return cls(member, False, None, 0, {}, False)
+        return cls(member, False, None, 0, 0, {}, False)

    def failover_limitation(self):
        """Returns reason why this node can't promote or None if everything is ok."""
@@ -61,6 +64,7 @@ class Ha(object):
        self.old_cluster = None
        self._is_leader = False
        self._is_leader_lock = RLock()
+        self._leader_access_is_restricted = False
        self._was_paused = False
        self._leader_timeline = None
        self.recovering = False
@@ -91,14 +95,33 @@ class Ha(object):
    def is_paused(self):
        return self.check_mode('pause')

+    def check_timeline(self):
+        return self.check_mode('check_timeline')
+
+    def get_standby_cluster_config(self):
+        if self.cluster and self.cluster.config and self.cluster.config.modify_index:
+            config = self.cluster.config.data
+        else:
+            config = self.patroni.config.dynamic_configuration
+        return config.get('standby_cluster')
+
+    def is_standby_cluster(self):
+        config = self.get_standby_cluster_config()
+        # Check whether or not provided configuration describes a standby cluster
+        return isinstance(config, dict) and (config.get('host') or config.get('port') or config.get('restore_command'))
+
    def is_leader(self):
        with self._is_leader_lock:
-            return self._is_leader
+            return self._is_leader and not self._leader_access_is_restricted

    def set_is_leader(self, value):
        with self._is_leader_lock:
            self._is_leader = value

+    def set_leader_access_is_restricted(self, value):
+        with self._is_leader_lock:
+            self._leader_access_is_restricted = value
+
    def load_cluster_from_dcs(self):
        cluster = self.dcs.get_cluster()

@@ -113,6 +136,7 @@ class Ha(object):
        self._leader_timeline = None if cluster.is_unlocked() else cluster.leader.timeline

    def acquire_lock(self):
+        self.set_leader_access_is_restricted(self.cluster.has_permanent_logical_slots(self.state_handler.name))
        ret = self.dcs.attempt_to_acquire_leader()
        self.set_is_leader(ret)
        return ret
@@ -124,7 +148,7 @@ class Ha(object):
                last_operation = self.state_handler.last_operation()
            except Exception:
                logger.exception('Exception when called state_handler.last_operation()')
-        ret = self.dcs.update_leader(last_operation)
+        ret = self.dcs.update_leader(last_operation, self._leader_access_is_restricted)
        self.set_is_leader(ret)
        if ret:
            self.watchdog.keepalive()
@@ -151,6 +175,10 @@ class Ha(object):
                'state': self.state_handler.state,
                'role': self.state_handler.role
            }
+
+            # following two lines are mainly necessary for consul, to avoid creation of master service
+            if data['role'] == 'master' and not self.is_leader():
+                data['role'] = 'promoted'
            tags = self.get_effective_tags()
            if tags:
                data['tags'] = tags
@@ -202,7 +230,7 @@ class Ha(object):
                self.state_handler.bootstrapping = True
                self._post_bootstrap_task = CriticalTask()

-                if self.patroni.config.is_standby_cluster:
+                if self.is_standby_cluster():
                    self._async_executor.schedule('bootstrap_standby_leader')
                    self._async_executor.run_async(self.bootstrap_standby_leader)
                    return 'trying to bootstrap a new standby leader'
@@ -228,8 +256,7 @@ class Ha(object):
            not a real master, but a 'standby leader', that will take base backup
            from a remote master and start follow it.
        """
-        patroni_config = self.patroni.config.dynamic_configuration
-        clone_source = self.get_remote_master(patroni_config)
+        clone_source = self.get_remote_master()
        msg = 'clone from remote master {0}'.format(clone_source.conn_url)
        result = self.clone(clone_source, msg)
        self._post_bootstrap_task.complete(result)
@@ -239,9 +266,10 @@ class Ha(object):
        return result

    def _handle_rewind(self):
-        if self.state_handler.rewind_needed_and_possible(self.cluster.leader):
-            self._async_executor.schedule('running pg_rewind from ' + self.cluster.leader.name)
-            self._async_executor.run_async(self.state_handler.rewind, (self.cluster.leader,))
+        leader = self.get_remote_master() if self.is_standby_cluster() else self.cluster.leader
+        if self.state_handler.rewind_needed_and_possible(leader):
+            self._async_executor.schedule('running pg_rewind from ' + leader.name)
+            self._async_executor.run_async(self.state_handler.rewind, (leader,))
            return True

    def recover(self):
@@ -273,16 +301,24 @@ class Ha(object):

        self.load_cluster_from_dcs()

-        if self.has_lock():
-            msg = "starting as readonly because i had the session lock"
-            node_to_follow = None
-        else:
+        if self.is_standby_cluster() or not self.has_lock():
            if not self.state_handler.rewind_executed:
                self.state_handler.trigger_check_diverged_lsn()
            if self._handle_rewind():
                return self._async_executor.scheduled_action
-            msg = "starting as a secondary"
-            node_to_follow = self._get_node_to_follow(self.cluster)
+
+            if self.has_lock():  # in standby cluster
+                msg = "starting as a standby leader because i had the session lock"
+                node_to_follow = self._get_node_to_follow(self.cluster)
+            elif self.is_standby_cluster() and self.cluster.is_unlocked():
+                msg = "trying to follow a remote master because standby cluster is unhealthy"
+                node_to_follow = self.get_remote_master()
+            else:
+                msg = "starting as a secondary"
+                node_to_follow = self._get_node_to_follow(self.cluster)
+        elif self.has_lock():
+            msg = "starting as readonly because i had the session lock"
+            node_to_follow = None

        self.recovering = True

@@ -295,8 +331,8 @@ class Ha(object):
        # try to follow the node mentioned there, otherwise, follow the leader.
        is_leader = self.cluster.leader and self.state_handler.name == self.cluster.leader.name

-        if self.cluster.is_standby_cluster() and is_leader:
-            node_to_follow = self.get_remote_master(cluster.config.data)
+        if self.is_standby_cluster() and (is_leader or self.cluster.is_unlocked()):
+            node_to_follow = self.get_remote_master()
        elif self.patroni.replicatefrom and self.patroni.replicatefrom != self.state_handler.name:
            node_to_follow = cluster.get_member(self.patroni.replicatefrom)
        else:
@@ -479,8 +515,10 @@ class Ha(object):
                    return 'Postponing promotion because synchronous replication state was updated by somebody else'
                self.state_handler.set_synchronous_standby('*' if self.is_synchronous_mode_strict() else None)
            if self.state_handler.role != 'master':
+                self.set_leader_access_is_restricted(self.cluster.has_permanent_logical_slots(self.state_handler.name))
                self._async_executor.schedule('promote')
-                self._async_executor.run_async(self.state_handler.promote, args=(self.dcs.loop_wait,))
+                self._async_executor.run_async(self.state_handler.promote,
+                                               args=(self.dcs.loop_wait, self._leader_access_is_restricted))
            return promote_message

    @staticmethod
@@ -511,15 +549,23 @@ class Ha(object):
        :returns True when node is lagging
        """
        lag = (self.cluster.last_leader_operation or 0) - wal_position
-        return lag > self.state_handler.config.get('maximum_lag_on_failover', 0)
+        return lag > self.patroni.config.get('maximum_lag_on_failover', 0)

    def _is_healthiest_node(self, members, check_replication_lag=True):
        """This method tries to determine whether I am healthy enough to became a new leader candidate or not."""

        _, my_wal_position = self.state_handler.timeline_wal_position()
        if check_replication_lag and self.is_lagging(my_wal_position):
+            logger.info('My wal position exceeds maximum replication lag')
            return False  # Too far behind last reported wal position on master

+        if not self.is_standby_cluster() and self.check_timeline():
+            cluster_timeline = self.cluster.timeline
+            my_timeline = self.state_handler.replica_cached_timeline(cluster_timeline)
+            if my_timeline < cluster_timeline:
+                logger.info('My timeline %s is behind last known cluster timeline %s', my_timeline, cluster_timeline)
+                return False
+
        # Prepare list of nodes to run check against
        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]

@@ -530,11 +576,13 @@ class Ha(object):
                        logger.warning('Master (%s) is still alive', st.member.name)
                        return False
                    if my_wal_position < st.wal_position:
+                        logger.info('Wal position of %s is ahead of my wal position', st.member.name)
                        return False
        return True

    def is_failover_possible(self, members):
        ret = False
+        cluster_timeline = self.cluster.timeline
        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
        if members:
            for st in self.fetch_nodes_statuses(members):
@@ -543,6 +591,9 @@ class Ha(object):
                    logger.info('Member %s is %s', st.member.name, not_allowed_reason)
                elif self.is_lagging(st.wal_position):
                    logger.info('Member %s exceeds maximum replication lag', st.member.name)
+                elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline):
+                    logger.info('Timeline %s of member %s is behind the cluster timeline %s',
+                                st.timeline, st.member.name, cluster_timeline)
                else:
                    ret = True
        else:
@@ -780,7 +831,7 @@ class Ha(object):
                        self.dcs.manual_failover('', '')
                self.load_cluster_from_dcs()

-                if self.cluster.is_standby_cluster():
+                if self.is_standby_cluster():
                    # standby leader disappeared, and this is a healthiest
                    # replica, so it should become a new standby leader.
                    # This imply that we need to start following a remote master
@@ -817,12 +868,17 @@ class Ha(object):
                self.dcs.reset_cluster()
                return 'removed leader lock because postgres is not running as master'

+            if self.state_handler.is_leader() and self._leader_access_is_restricted:
+                self.state_handler.sync_replication_slots(self.cluster)
+                self.state_handler.call_nowait(ACTION_ON_ROLE_CHANGE)
+                self.set_leader_access_is_restricted(False)
+
            if self.update_lock(True):
                msg = self.process_manual_failover_from_leader()
                if msg is not None:
                    return msg

-                if self.cluster.is_standby_cluster():
+                if self.is_standby_cluster():
                    # in case of standby cluster we don't really need to
                    # enforce anything, since the leader is not a master.
                    # So just remind the role.
@@ -962,7 +1018,8 @@ class Ha(object):

    def _do_reinitialize(self, cluster):
        self.state_handler.stop('immediate')
-        self.state_handler.remove_data_directory()
+        # Commented redundant data directory cleanup here
+        # self.state_handler.remove_data_directory()

        clone_member = self.cluster.get_clone_member(self.state_handler.name)
        member_role = 'leader' if clone_member == self.cluster.leader else 'replica'
@@ -1055,6 +1112,7 @@ class Ha(object):
        if not self.watchdog.activate():
            logger.error('Cancelling bootstrap because watchdog activation failed')
            self.cancel_initialization()
+        self.state_handler.sync_replication_slots(self.cluster)
        self.dcs.take_leader()
        self.set_is_leader(True)
        self.state_handler.call_nowait(ACTION_ON_START)
@@ -1213,11 +1271,11 @@ class Ha(object):
                # stops PostgreSQL, therefore, we only reload replication slots if no
                # asynchronous processes are running (should be always the case for the master)
                if not self._async_executor.busy and not self.state_handler.is_starting():
+                    self.state_handler.sync_replication_slots(self.cluster)
                    if not self.state_handler.cb_called:
                        if not self.state_handler.is_leader():
                            self.state_handler.trigger_check_diverged_lsn()
                        self.state_handler.call_nowait(ACTION_ON_START)
-                    self.state_handler.sync_replication_slots(self.cluster)
        except DCSError:
            dcs_failed = True
            logger.error('Error communicating with DCS')
@@ -1275,15 +1333,14 @@ class Ha(object):
        This usually happens on the master or if the node is running async action"""
        self.dcs.event.set()

-    def get_remote_master(self, config):
+    def get_remote_master(self):
        """ In case of standby cluster this will tel us from which remote
            master to stream. Config can be both patroni config or
            cluster.config.data
        """
-        config = config or (self.config is not None and self.config.data)
+        cluster_params = self.get_standby_cluster_config()

-        if config and config.get('standby_cluster'):
-            cluster_params = config.get('standby_cluster')
+        if cluster_params:
            unique_name = 'remote_master:{}'.format(uuid.uuid1())
            data = {
                'conn_kwargs': {
@@ -0,0 +1,66 @@
+import logging
+import os
+
+from copy import deepcopy
+from logging.handlers import RotatingFileHandler
+from patroni.utils import deep_compare
+
+
+class PatroniLogger(object):
+
+    DEFAULT_LEVEL = 'INFO'
+    DEFAULT_FORMAT = '%(asctime)s %(levelname)s: %(message)s'
+
+    def __init__(self):
+        self.root_logger = logging.getLogger()
+        self.config = None
+        self.handler = None
+        self.reload_config({'level': 'DEBUG'})
+
+    def update_loggers(self):
+        loggers = deepcopy(self.config.get('loggers') or {})
+        for name, logger in self.root_logger.manager.loggerDict.items():
+            if not isinstance(logger, logging.PlaceHolder):
+                level = loggers.pop(name, logging.NOTSET)
+                logger.setLevel(level)
+
+        for name, level in loggers.items():
+            logger = self.root_logger.manager.getLogger(name)
+            logger.setLevel(level)
+
+    def reload_config(self, config):
+        if self.config is None or not deep_compare(self.config, config):
+            self.root_logger.setLevel(config.get('level', PatroniLogger.DEFAULT_LEVEL))
+
+            add_handler = None
+            if 'dir' in config:
+                if not isinstance(self.handler, RotatingFileHandler):
+                    add_handler = RotatingFileHandler(os.path.join(config['dir'], __name__))
+                handler = add_handler or self.handler
+                handler.maxBytes = int(config.get('file_size', 25000000))
+                handler.backupCount = int(config.get('file_num', 4))
+            else:
+                if self.handler is None or isinstance(self.handler, RotatingFileHandler):
+                    add_handler = logging.StreamHandler()
+                handler = add_handler or self.handler
+
+            oldlogformat = (self.config or {}).get('format', PatroniLogger.DEFAULT_FORMAT)
+            logformat = config.get('format', PatroniLogger.DEFAULT_FORMAT)
+
+            olddateformat = (self.config or {}).get('dateformat') or None
+            dateformat = config.get('dateformat') or None  # Convert empty string to `None`
+
+            if oldlogformat != logformat or olddateformat != dateformat or add_handler:
+                handler.setFormatter(logging.Formatter(logformat, dateformat))
+
+            if add_handler:
+                self.root_logger.addHandler(add_handler)
+
+                if self.handler is not None:
+                    self.root_logger.removeHandler(self.handler)
+                    self.handler.close()
+
+                self.handler = add_handler
+
+            self.config = config.copy()
+            self.update_loggers()
@@ -5,6 +5,7 @@ import re
 import shlex
 import shutil
 import socket
+import stat
 import subprocess
 import tempfile
 import time
@@ -15,7 +16,7 @@ from patroni.callback_executor import CallbackExecutor
 from patroni.exceptions import PostgresConnectionException, PostgresException
 from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop, split_host_port
 from patroni.postmaster import PostmasterProcess
-from patroni.dcs import RemoteMember
+from patroni.dcs import slot_name_from_member_name, RemoteMember
 from requests.structures import CaseInsensitiveDict
 from six import string_types
 from six.moves.urllib.parse import quote_plus
@@ -37,14 +38,16 @@ STATE_UNKNOWN = 'unknown'

 STOP_POLLING_INTERVAL = 1
 REWIND_STATUS = type('Enum', (), {'INITIAL': 0, 'CHECK': 1, 'NEED': 2, 'NOT_NEED': 3, 'SUCCESS': 4, 'FAILED': 5})
-sync_standby_name_re = re.compile('^[A-Za-z_][A-Za-z_0-9\$]*$')
+sync_standby_name_re = re.compile(r'^[A-Za-z_][A-Za-z_0-9\$]*$')

-cluster_info_query = ("SELECT CASE WHEN pg_is_in_recovery() THEN 0 "
-                      "ELSE ('x' || SUBSTR(pg_{0}file_name(pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, "
-                      "CASE WHEN pg_is_in_recovery() THEN GREATEST("
-                      " pg_{0}_{1}_diff(COALESCE(pg_last_{0}_receive_{1}(), '0/0'), '0/0')::bigint,"
-                      " pg_{0}_{1}_diff(pg_last_{0}_replay_{1}(), '0/0')::bigint)"
-                      "ELSE pg_{0}_{1}_diff(pg_current_{0}_{1}(), '0/0')::bigint END")
+cluster_info_query = ("SELECT CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
+                      "ELSE ('x' || pg_catalog.substr(pg_catalog.pg_{0}file_name("
+                      "pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, "
+                      "CASE WHEN pg_catalog.pg_is_in_recovery() THEN GREATEST("
+                      " pg_catalog.pg_{0}_{1}_diff(COALESCE("
+                      "pg_catalog.pg_last_{0}_receive_{1}(), '0/0'), '0/0')::bigint,"
+                      " pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), '0/0')::bigint)"
+                      "ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), '0/0')::bigint END")


 def quote_ident(value):
@@ -52,22 +55,6 @@ def quote_ident(value):
    return value if sync_standby_name_re.match(value) else '"' + value + '"'


-def slot_name_from_member_name(member_name):
-    """Translate member name to valid PostgreSQL slot name.
-
-    PostgreSQL replication slot names must be valid PostgreSQL names. This function maps the wider space of
-    member names to valid PostgreSQL names. Names are lowercased, dashes and periods common in hostnames
-    are replaced with underscores, other characters are encoded as their unicode codepoint. Name is truncated
-    to 64 characters. Multiple different member names may map to a single slot name."""
-
-    def replace_char(match):
-        c = match.group(0)
-        return '_' if c in '-.' else "u{:04d}".format(ord(c))
-
-    slot_name = re.sub('[^a-z0-9_]', replace_char, member_name.lower())
-    return slot_name[0:63]
-
-
@contextmanager
 def null_context():
    yield
@@ -154,7 +141,7 @@ class Postgresql(object):
        self._connection = None
        self._cursor_holder = None
        self._sysid = None
-        self._replication_slots = []  # list of already existing replication slots
+        self._replication_slots = {}  # already existing replication slots
        self.retry = Retry(max_tries=-1, deadline=config['retry_timeout']/2.0, max_delay=1,
                           retry_exceptions=PostgresConnectionException)

@@ -324,8 +311,8 @@ class Postgresql(object):
                    changes['wal_segment_size'] = '16384kB'
                # XXX: query can raise an exception
                for r in self.query("""SELECT name, setting, unit, vartype, context
-                                         FROM pg_settings
-                                        WHERE LOWER(name) IN (""" + ', '.join(['%s'] * len(changes)) + """)
+                                         FROM pg_catalog.pg_settings
+                                        WHERE pg_catalog.lower(name) IN (""" + ', '.join(['%s'] * len(changes)) + """)
                                        ORDER BY 1 DESC""", *(k.lower() for k in changes.keys())):
                    if r[4] == 'internal':
                        if r[0] == 'wal_segment_size':
@@ -405,7 +392,7 @@ class Postgresql(object):
            we have either wal_log_hints or checksums turned on
        """
        # low-hanging fruit: check if pg_rewind configuration is there
-        if not (self.config.get('use_pg_rewind') and all(self._superuser.get(n) for n in ('username', 'password'))):
+        if not self.config.get('use_pg_rewind'):
            return False

        cmd = [self._pgcommand('pg_rewind'), '--help']
@@ -706,8 +693,10 @@ class Postgresql(object):
                    # if basebackup succeeds, exit with success
                    break
            else:
-                if not self.data_directory_empty():
+                if not self.data_directory_empty() and not self.config.get(replica_method, {}).get('keep_data', False):
                    self.remove_data_directory()
+                else:
+                    logger.info('Leaving data directory uncleaned')

                cmd = replica_method
                method_config = {}
@@ -720,10 +709,14 @@ class Postgresql(object):
                    cmd = method_config.pop('command', cmd)

                # add the default parameters
-                method_config.update({"scope": self.scope,
-                                      "role": "replica",
-                                      "datadir": self._data_dir,
-                                      "connstring": connstring})
+                if not method_config.get('no_params', False):
+                    method_config.update({"scope": self.scope,
+                                          "role": "replica",
+                                          "datadir": self._data_dir,
+                                          "connstring": connstring})
+                else:
+                    for param in ('no_params', 'no_master', 'keep_data'):
+                        method_config.pop(param, None)
                params = ["--{0}={1}".format(arg, val) for arg, val in method_config.items()]
                try:
                    # call script with the full set of parameters
@@ -951,7 +944,7 @@ class Postgresql(object):
            with self._get_connection_cursor(**connect_kwargs) as cur:
                cur.execute("SET statement_timeout = 0")
                if check_not_is_in_recovery:
-                    cur.execute('SELECT pg_is_in_recovery()')
+                    cur.execute('SELECT pg_catalog.pg_is_in_recovery()')
                    if cur.fetchone()[0]:
                        return 'is_in_recovery=true'
                return cur.execute('CHECKPOINT')
@@ -1118,8 +1111,12 @@ class Postgresql(object):
            f.write(self._CONFIG_WARNING_HEADER)
            f.write("include '{0}'\n\n".format(self.config.get('custom_conf') or self._postgresql_base_conf_name))
            for name, value in sorted((configuration or self._server_parameters).items()):
-                if not self._running_custom_bootstrap or name != 'hba_file':
+                if not self._running_custom_bootstrap or name not in ('hba_file', 'archive_mode'):
                    f.write("{0} = '{1}'\n".format(name, value))
+            # we want to set archive_mode to 'off' during the custom bootstrap
+            # in order to avoid premature archiving of wals and history files
+            if self._running_custom_bootstrap:
+                f.write("archive_mode = 'off'\n")
            # when we are doing custom bootstrap we assume that we don't know superuser password
            # and in order to be able to change it, we are opening trust access from a certain address
            # therefore we need to make sure that hba_file is not overriden
@@ -1162,8 +1159,10 @@ class Postgresql(object):
            with open(self._pg_hba_conf, 'w') as f:
                f.write(self._CONFIG_WARNING_HEADER)
                for address, t in addresses.items():
-                    f.write('{0}\t{1}\t{2}\t{3}\ttrust\n'.format(t, 'all',
-                                                                 self._superuser.get('username') or 'all', address))
+                    f.write((
+                        '{0}\treplication\t{1}\t{3}\ttrust\n'
+                        '{0}\tall\t{2}\t{3}\ttrust\n'
+                    ).format(t, self._replication['username'], self._superuser.get('username') or 'all', address))
        elif not self._server_parameters.get('hba_file') and self.config.get('pg_hba'):
            with open(self._pg_hba_conf, 'w') as f:
                f.write(self._CONFIG_WARNING_HEADER)
@@ -1194,6 +1193,7 @@ class Postgresql(object):

    def write_recovery_conf(self, recovery_params):
        with open(self._recovery_conf, 'w') as f:
+            os.chmod(self._recovery_conf, stat.S_IWRITE | stat.S_IREAD)
            for name, value in recovery_params.items():
                f.write("{0} = '{1}'\n".format(name, value))

@@ -1204,7 +1204,7 @@ class Postgresql(object):
            ('user', r.get('user')),
            ('host', r.get('host')),
            ('port', r.get('port')),
-            ('dbname', r.get('database')),
+            ('dbname', r.get('database') or self._database),
            ('sslmode', 'prefer'),
            ('sslcompression', '1'),
        ]
@@ -1249,8 +1249,10 @@ class Postgresql(object):
                yield cur

    @contextmanager
-    def _get_replication_connection_cursor(self, host='localhost', port=5432, **kwargs):
-        with self._get_connection_cursor(host=host, port=int(port), database=self._database, replication=1,
+    def _get_replication_connection_cursor(self, host='localhost', port=5432, database=None, **kwargs):
+        database = database or self._database
+        replication = 'database' if self._major_version >= 90400 else 1
+        with self._get_connection_cursor(host=host, port=int(port), database=database, replication=replication,
                                         user=self._replication['username'], password=self._replication['password'],
                                         connect_timeout=3, options='-c statement_timeout=2000') as cur:
            yield cur
@@ -1258,7 +1260,7 @@ class Postgresql(object):
    def check_leader_is_not_in_recovery(self, **kwargs):
        try:
            with self._get_connection_cursor(connect_timeout=3, options='-c statement_timeout=2000', **kwargs) as cur:
-                cur.execute('SELECT pg_is_in_recovery()')
+                cur.execute('SELECT pg_catalog.pg_is_in_recovery()')
                if not cur.fetchone()[0]:
                    return True
                logger.info('Leader is still in_recovery and therefore can\'t be used for rewind')
@@ -1369,10 +1371,10 @@ class Postgresql(object):
        history_path = 'pg_{0}/{1:08X}.history'.format(self.wal_name, timeline)
        try:
            cursor = self._cursor()
-            cursor.execute('SELECT isdir, modification FROM pg_stat_file(%s)', (history_path,))
+            cursor.execute('SELECT isdir, modification FROM pg_catalog.pg_stat_file(%s)', (history_path,))
            isdir, modification = cursor.fetchone()
            if not isdir:
-                cursor.execute('SELECT pg_read_file(%s)', (history_path,))
+                cursor.execute('SELECT pg_catalog.pg_read_file(%s)', (history_path,))
                history = list(self.parse_history(cursor.fetchone()[0]))
                if history[-1][0] == timeline - 1:
                    history[-1].append(modification.isoformat())
@@ -1501,7 +1503,7 @@ class Postgresql(object):
            if data.get('Database cluster state') == 'in production':
                return True

-    def promote(self, wait_seconds):
+    def promote(self, wait_seconds, access_is_restricted=False):
        if self.role == 'master':
            return True
        ret = self.pg_ctl('promote', '-W')
@@ -1509,7 +1511,8 @@ class Postgresql(object):
            self.set_role('master')
            logger.info("cleared rewind state after becoming the leader")
            self._rewind_state = REWIND_STATUS.INITIAL
-            self.call_nowait(ACTION_ON_ROLE_CHANGE)
+            if not access_is_restricted:
+                self.call_nowait(ACTION_ON_ROLE_CHANGE)
            ret = self._wait_promote(wait_seconds)
        return ret

@@ -1542,61 +1545,88 @@ $$""".format(name, ' '.join(options)), name, password, password)

    def load_replication_slots(self):
        if self.use_slots and self._schedule_load_slots:
-            cursor = self._query("SELECT slot_name FROM pg_replication_slots WHERE slot_type='physical'")
-            self._replication_slots = [r[0] for r in cursor]
+            replication_slots = {}
+            cursor = self._query('SELECT slot_name, slot_type, plugin, database FROM pg_catalog.pg_replication_slots')
+            for r in cursor:
+                value = {'type': r[1]}
+                if r[1] == 'logical':
+                    value.update({'plugin': r[2], 'database': r[3]})
+                replication_slots[r[0]] = value
+            self._replication_slots = replication_slots
            self._schedule_load_slots = False

    def postmaster_start_time(self):
        try:
-            cursor = self.query("""SELECT to_char(pg_postmaster_start_time(), 'YYYY-MM-DD HH24:MI:SS.MS TZ')""")
+            cursor = self.query("SELECT pg_catalog.to_char(pg_catalog.pg_postmaster_start_time(),"
+                                " 'YYYY-MM-DD HH24:MI:SS.MS TZ')")
            return cursor.fetchone()[0]
        except psycopg2.Error:
            return None

+    def drop_replication_slot(self, name):
+        cursor = self._query(('SELECT pg_catalog.pg_drop_replication_slot(%s) WHERE EXISTS (SELECT 1 ' +
+                              'FROM pg_catalog.pg_replication_slots WHERE slot_name = %s AND NOT active)'), name, name)
+        # In normal situation rowcount should be 1, otherwise either slot doesn't exists or it is still active
+        return cursor.rowcount == 1
+
+    @staticmethod
+    def compare_slots(s1, s2):
+        return s1['type'] == s2['type'] and\
+                (s1['type'] == 'physical' or s1['database'] == s2['database'] and s1['plugin'] == s2['plugin'])
+
    def sync_replication_slots(self, cluster):
        if self.use_slots:
            try:
                self.load_replication_slots()
-                # if the replicatefrom tag is set on the member - we should not create the replication slot for it on
-                # the current master, because that member would replicate from elsewhere. We still create the slot if
-                # the replicatefrom destination member is currently not a member of the cluster (fallback to the
-                # master), or if replicatefrom destination member happens to be the current master
-                if self.role in ('master', 'standby_leader'):
-                    slot_members = [m.name for m in cluster.members if m.name != self.name and
-                                    (m.replicatefrom is None or m.replicatefrom == self.name or
-                                     not cluster.has_member(m.replicatefrom))]
-                else:
-                    # only manage slots for replicas that replicate from this one, except for the leader among them
-                    slot_members = [m.name for m in cluster.members if m.replicatefrom == self.name and
-                                    m.name != cluster.leader.name]
-                slots = set(slot_name_from_member_name(name) for name in slot_members)

-                if len(slots) < len(slot_members):
-                    # Find which names are conflicting for a nicer error message
-                    slot_conflicts = defaultdict(list)
-                    for name in slot_members:
-                        slot_conflicts[slot_name_from_member_name(name)].append(name)
-                    logger.error("Following cluster members share a replication slot name: %s",
-                                 "; ".join("{} map to {}".format(", ".join(v), k)
-                                           for k, v in slot_conflicts.items() if len(v) > 1))
+                slots = cluster.get_replication_slots(self.name, self.role)

-                # drop unused slots
-                for slot in set(self._replication_slots) - slots:
-                    cursor = self._query("""SELECT pg_drop_replication_slot(%s)
-                                             WHERE EXISTS(SELECT 1 FROM pg_replication_slots
-                                             WHERE slot_name = %s AND NOT active)""", slot, slot)
-
-                    if cursor.rowcount != 1:  # Either slot doesn't exists or it is still active
-                        self._schedule_load_slots = True  # schedule load_replication_slots on the next iteration
+                # drop old replication slots which are not presented in desired slots
+                for name in set(self._replication_slots) - set(slots):
+                    if not self.drop_replication_slot(name):
+                        logger.error("Failed to drop replication slot '%s'", name)
+                        self._schedule_load_slots = True

                immediately_reserve = ', true' if self._major_version >= 90600 else ''

-                # create new slots
-                for slot in slots - set(self._replication_slots):
-                    self._query("""SELECT pg_create_physical_replication_slot(%s{0})
-                                    WHERE NOT EXISTS (SELECT 1 FROM pg_replication_slots
-                                    WHERE slot_name = %s)""".format(immediately_reserve), slot, slot)
+                logical_slots = defaultdict(dict)
+                for name, value in slots.items():
+                    if name in self._replication_slots and not self.compare_slots(value, self._replication_slots[name]):
+                        logger.info("Trying to drop replication slot '%s' because value is changing from %s to %s",
+                                    name, self._replication_slots[name], value)
+                        if not self.drop_replication_slot(name):
+                            logger.error("Failed to drop replication slot '%s'", name)
+                            self._schedule_load_slots = True
+                            continue
+                        self._replication_slots.pop(name)
+                    if name not in self._replication_slots:
+                        if value['type'] == 'physical':
+                            try:
+                                self._query(("SELECT pg_catalog.pg_create_physical_replication_slot(%s{0})" +
+                                             " WHERE NOT EXISTS (SELECT 1 FROM pg_catalog.pg_replication_slots" +
+                                             " WHERE slot_type = 'physical' AND slot_name = %s)").format(
+                                                 immediately_reserve), name, name)
+                            except Exception:
+                                logger.exception("Failed to create physical replication slot '%s'", name)
+                                self._schedule_load_slots = True
+                        elif value['type'] == 'logical' and name not in self._replication_slots:
+                            logical_slots[value['database']][name] = value

+                # create new logical slots
+                for database, values in logical_slots.items():
+                    conn_kwargs = self._local_connect_kwargs
+                    conn_kwargs['database'] = database
+                    with self._get_connection_cursor(**conn_kwargs) as cur:
+                        for name, value in values.items():
+                            try:
+                                cur.execute("SELECT pg_catalog.pg_create_logical_replication_slot(%s, %s)" +
+                                            " WHERE NOT EXISTS (SELECT 1 FROM pg_catalog.pg_replication_slots" +
+                                            " WHERE slot_type = 'logical' AND slot_name = %s)",
+                                            (name, value['plugin'], name))
+                            except Exception:
+                                logger.exception("Failed to create logical replication slot '%s' plugin='%s'",
+                                                 name, value['plugin'])
+                                self._schedule_load_slots = True
                self._replication_slots = slots
            except Exception:
                logger.exception('Exception when changing replication slots')
@@ -1643,7 +1673,8 @@ $$""".format(name, ' '.join(options)), name, password, password)

    def post_bootstrap(self, config, task):
        try:
-            self.create_or_update_role(self._superuser['username'], self._superuser['password'], ['SUPERUSER'])
+            if 'username' in self._superuser and 'password' in self._superuser:
+                self.create_or_update_role(self._superuser['username'], self._superuser['password'], ['SUPERUSER'])

            task.complete(self.run_bootstrap_post_init(config))
            if task.result:
@@ -1662,17 +1693,11 @@ $$""".format(name, ' '.join(options)), name, password, password)
                        os.unlink(self._pg_hba_conf)
                        self.restore_configuration_files()
                    self._write_postgresql_conf()
-                    if self._server_parameters.get('hba_file') and \
-                            self._server_parameters['hba_file'] != self._pg_hba_conf:
-                        self.restart()
-                    else:
-                        self._replace_pg_hba()
-                        if self.pending_restart:
-                            self.restart()
-                        else:
-                            self.reload()
-                            time.sleep(1)  # give a time to postgres to "reload" configuration files
-                            self.close_connection()  # close connection to reconnect with a new password
+                    self._replace_pg_hba()
+                    # at this point there should be no recovery.conf
+                    if os.path.isfile(self._recovery_conf) or os.path.islink(self._recovery_conf):
+                        os.unlink(self._recovery_conf)
+                    self.restart()
        except Exception:
            logger.exception('post_bootstrap')
            task.complete(False)
@@ -1753,9 +1778,9 @@ $$""".format(name, ' '.join(options)), name, password, password)
        # Pick candidates based on who has flushed WAL farthest.
        # TODO: for synchronous_commit = remote_write we actually want to order on write_location
        for app_name, state, sync_state in self.query(
-                """SELECT LOWER(application_name), state, sync_state
-                     FROM pg_stat_replication
-                    ORDER BY flush_{0} DESC""".format(self.lsn_name)):
+                "SELECT pg_catalog.lower(application_name), state, sync_state"
+                " FROM pg_catalog.pg_stat_replication"
+                " ORDER BY flush_{0} DESC".format(self.lsn_name)):
            member = members.get(app_name)
            if state != 'streaming' or not member or member.tags.get('nosync', False):
                continue
@@ -102,28 +102,31 @@ class PostmasterProcess(psutil.Process):
        return None

    def wait_for_user_backends_to_close(self):
-        # These regexps are cross checked against versions PostgreSQL 9.1 .. 9.6
-        aux_proc_re = re.compile("(?:postgres:)( .*:)? (?:""(?:startup|logger|checkpointer|writer|wal writer|"
-                                 "autovacuum launcher|autovacuum worker|stats collector|wal receiver|archiver|"
-                                 "wal sender) process|bgworker: )")
+        # These regexps are cross checked against versions PostgreSQL 9.1 .. 11
+        aux_proc_re = re.compile("(?:postgres:)( .*:)? (?:(?:archiver|startup|autovacuum launcher|autovacuum worker|"
+                                 "checkpointer|logger|stats collector|wal receiver|wal writer|writer)(?: process  )?|"
+                                 "walreceiver|wal sender process|walsender|walwriter|background writer|"
+                                 "logical replication launcher|logical replication worker for|bgworker:) ")

        try:
-            user_backends = []
-            user_backends_cmdlines = []
-            for child in self.children():
-                try:
-                    cmdline = child.cmdline()[0]
-                    if not aux_proc_re.match(cmdline):
-                        user_backends.append(child)
-                        user_backends_cmdlines.append(cmdline)
-                except psutil.NoSuchProcess:
-                    pass
-            if user_backends:
-                logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines))
-                psutil.wait_procs(user_backends)
-            logger.debug("Backends closed")
+            children = self.children()
        except psutil.Error:
-            logger.exception('wait_for_user_backends_to_close')
+            return logger.debug('Failed to get list of postmaster children')
+
+        user_backends = []
+        user_backends_cmdlines = []
+        for child in children:
+            try:
+                cmdline = child.cmdline()[0]
+                if not aux_proc_re.match(cmdline):
+                    user_backends.append(child)
+                    user_backends_cmdlines.append(cmdline)
+            except psutil.NoSuchProcess:
+                pass
+        if user_backends:
+            logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines))
+            psutil.wait_procs(user_backends)
+        logger.debug("Backends closed")

    @staticmethod
    def start(pgcommand, data_dir, conf, options):
@@ -157,7 +160,7 @@ class PostmasterProcess(psutil.Process):
        cmdline = [pgcommand, '-D', data_dir, '--config-file={}'.format(conf)] + options
        logger.debug("Starting postgres: %s", " ".join(cmdline))
        proc = call_self(['pg_ctl_start'] + cmdline, close_fds=(os.name != 'nt'),
-                         stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env)
+                         stdout=subprocess.PIPE, env=env)
        pid = int(proc.stdout.readline().strip())
        proc.wait()
        logger.info('postmaster pid=%s', pid)
@@ -224,13 +224,12 @@ class WALERestore(object):
                            lsn_name = 'location'
                        con.autocommit = True
                        with con.cursor() as cur:
-                            cur.execute("""SELECT CASE WHEN pg_is_in_recovery()
-                                                       THEN GREATEST(
-                                                                pg_{0}_{1}_diff(COALESCE(
-                                                                  pg_last_{0}_receive_{1}(), '0/0'), %s)::bigint,
-                                                                pg_{0}_{1}_diff(pg_last_{0}_replay_{1}(), %s)::bigint)
-                                                       ELSE pg_{0}_{1}_diff(pg_current_{0}_{1}(), %s)::bigint
-                                                   END""".format(wal_name, lsn_name),
+                            cur.execute(("SELECT CASE WHEN pg_catalog.pg_is_in_recovery()"
+                                         " THEN GREATEST(pg_catalog.pg_{0}_{1}_diff(COALESCE("
+                                         "pg_last_{0}_receive_{1}(), '0/0'), %s)::bigint, "
+                                         "pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), %s)::bigint)"
+                                         " ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), %s)::bigint"
+                                         " END").format(wal_name, lsn_name),
                                        (backup_start_lsn, backup_start_lsn, backup_start_lsn))

                            diff_in_bytes = int(cur.fetchone()[0])
@@ -1 +1 @@
-__version__ = '1.5.0'
+__version__ = '1.5.4'
@@ -73,7 +73,7 @@ class MockHa(object):

    @staticmethod
    def fetch_nodes_statuses(members):
-        return [_MemberStatus(None, True, None, None, {}, False)]
+        return [_MemberStatus(None, True, None, 0, None, {}, False)]

    @staticmethod
    def schedule_future_restart(data):
@@ -95,6 +95,10 @@ class MockHa(object):
    def is_paused():
        return True

+    @staticmethod
+    def is_standby_cluster():
+        return False
+

 class MockPatroni(object):

@@ -167,8 +171,8 @@ class TestRestApiHandler(unittest.TestCase):
        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /master'))
        with patch.object(RestApiServer, 'query', Mock(return_value=[('', 1, '', '', '', '', False, '')])):
            self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))
-        MockPatroni.config.is_standby_cluster = PropertyMock(return_value=True)
-        MockRestApiServer(RestApiHandler, 'GET /standby_leader')
+        with patch.object(MockHa, 'is_standby_cluster', Mock(return_value=True)):
+            MockRestApiServer(RestApiHandler, 'GET /standby_leader')

    def test_do_OPTIONS(self):
        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'OPTIONS / HTTP/1.0'))
@@ -397,3 +401,6 @@ class TestRestApiServer(unittest.TestCase):
        self.assertRaises(ValueError, srv.reload_config, bad_config)
        self.assertRaises(ValueError, srv.reload_config, {})
        srv.reload_config({'listen': '127.0.0.2:8008'})
+
+    def test_handle_error(self):
+        self.assertIsNone(MockRestApiServer.handle_error(None, ('127.0.0.1', 55555)))
@@ -1,6 +1,6 @@
 import os
-import unittest
 import sys
+import unittest

 from mock import MagicMock, Mock, patch
 from patroni.config import Config
@@ -30,6 +30,8 @@ class TestConfig(unittest.TestCase):
            'PATRONI_NAME': 'postgres0',
            'PATRONI_NAMESPACE': '/patroni/',
            'PATRONI_SCOPE': 'batman2',
+            'PATRONI_LOGLEVEL': 'ERROR',
+            'PATRONI_LOG_LOGGERS': 'patroni.postmaster: WARNING, urllib3: DEBUG',
            'PATRONI_RESTAPI_USERNAME': 'username',
            'PATRONI_RESTAPI_PASSWORD': 'password',
            'PATRONI_RESTAPI_LISTEN': '0.0.0.0:8008',
@@ -49,6 +51,7 @@ class TestConfig(unittest.TestCase):
            'PATRONI_ETCD_CERT': '/cert',
            'PATRONI_ETCD_KEY': '/key',
            'PATRONI_CONSUL_HOST': '127.0.0.1:8500',
+            'PATRONI_CONSUL_REGISTER_SERVICE': 'on',
            'PATRONI_KUBERNETES_LABELS': 'a:b:c',
            'PATRONI_KUBERNETES_SCOPE_LABEL': 'a',
            'PATRONI_KUBERNETES_PORTS': '[{"name": "postgresql"}]',
@@ -84,3 +87,15 @@ class TestConfig(unittest.TestCase):
            self.config.save_cache()
        with patch('os.fdopen', MagicMock()):
            self.config.save_cache()
+
+    def test_standby_cluster_parameters(self):
+        dynamic_configuration = {
+            'standby_cluster': {
+                'create_replica_methods': ['wal_e', 'basebackup'],
+                'host': 'localhost',
+                'port': 5432
+            }
+        }
+        self.config.set_dynamic_configuration(dynamic_configuration)
+        for name, value in dynamic_configuration['standby_cluster'].items():
+            self.assertEqual(self.config['standby_cluster'][name], value)
@@ -4,7 +4,7 @@ import unittest
 from consul import ConsulException, NotFound
 from mock import Mock, patch
 from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, \
-                                ConsulError, HTTPClient, InvalidSessionTTL
+                                ConsulError, HTTPClient, InvalidSessionTTL, InvalidSession
 from test_etcd import SleepException


@@ -52,6 +52,8 @@ class TestHTTPClient(unittest.TestCase):
        self.assertRaises(ConsulInternalError, self.client.get, Mock(), '')
        self.client.http.request.return_value.data = b"Invalid Session TTL '3000000000', must be between [10s=24h0m0s]"
        self.assertRaises(InvalidSessionTTL, self.client.get, Mock(), '')
+        self.client.http.request.return_value.data = b"invalid session '16492f43-c2d6-5307-432f-e32d6f7bcbd0'"
+        self.assertRaises(InvalidSession, self.client.get, Mock(), '')

    def test_unknown_method(self):
        try:
@@ -110,19 +112,18 @@ class TestConsul(unittest.TestCase):
        self.c._session = 'fd4f44fe-2cac-bba5-a60b-304b51ff39b8'
        self.assertIsInstance(self.c.get_cluster(), Cluster)

-    @patch.object(consul.Consul.KV, 'delete', Mock(side_effect=[ConsulException, True, True]))
-    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=[True, ConsulException]))
+    @patch.object(consul.Consul.KV, 'delete', Mock(side_effect=[ConsulException, True, True, True]))
+    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=[True, ConsulException, InvalidSession]))
    def test_touch_member(self):
-        self.c._register_service = True
-        self.c.refresh_session = Mock(return_value=True)
-        self.c.touch_member({'balbla': 'blabla'})
-        self.c.touch_member({'balbla': 'blabla'})
-        self.c.touch_member({'balbla': 'blabla'})
        self.c.refresh_session = Mock(return_value=False)
        self.c.touch_member({'conn_url': 'postgres://replicator:[email protected]:5433/postgres',
                             'api_url': 'http://127.0.0.1:8009/patroni'})
+        self.c._register_service = True
+        self.c.refresh_session = Mock(return_value=True)
+        for _ in range(0, 4):
+            self.c.touch_member({'balbla': 'blabla'})

-    @patch.object(consul.Consul.KV, 'put', Mock(return_value=False))
+    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=InvalidSession))
    def test_take_leader(self):
        self.c.set_ttl(20)
        self.c.refresh_session = Mock()
@@ -192,24 +192,24 @@ class TestCtl(unittest.TestCase):

    def test_query_member(self):
        with patch('patroni.ctl.get_cursor', Mock(return_value=MockConnect().cursor())):
-            rows = query_member(None, None, None, 'master', 'SELECT pg_is_in_recovery()', {})
+            rows = query_member(None, None, None, 'master', 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertTrue('False' in str(rows))

-            rows = query_member(None, None, None, 'replica', 'SELECT pg_is_in_recovery()', {})
+            rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertEqual(rows, (None, None))

            with patch('test_postgresql.MockCursor.execute', Mock(side_effect=OperationalError('bla'))):
-                rows = query_member(None, None, None, 'replica', 'SELECT pg_is_in_recovery()', {})
+                rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})

        with patch('patroni.ctl.get_cursor', Mock(return_value=None)):
-            rows = query_member(None, None, None, None, 'SELECT pg_is_in_recovery()', {})
+            rows = query_member(None, None, None, None, 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertTrue('No connection to' in str(rows))

-            rows = query_member(None, None, None, 'replica', 'SELECT pg_is_in_recovery()', {})
+            rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertTrue('No connection to' in str(rows))

        with patch('patroni.ctl.get_cursor', Mock(side_effect=OperationalError('bla'))):
-            rows = query_member(None, None, None, 'replica', 'SELECT pg_is_in_recovery()', {})
+            rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})

    @patch('patroni.ctl.get_dcs')
    def test_dsn(self, mock_get_dcs):
@@ -5,7 +5,6 @@ import unittest
 import sys

 from mock import Mock, MagicMock, PropertyMock, patch
-from patroni.async_executor import CriticalTask
 from patroni.config import Config
 from patroni.dcs import Cluster, ClusterConfig, Failover, Leader, Member, get_dcs, SyncState, TimelineHistory
 from patroni.dcs.etcd import Client
@@ -29,8 +28,10 @@ def false(*args, **kwargs):


 def get_cluster(initialize, leader, members, failover, sync, cluster_config=None):
-    history = TimelineHistory(1, [(1, 67197376, 'no recovery target specified', datetime.datetime.now().isoformat())])
-    cluster_config = cluster_config or ClusterConfig(1, {1: 2}, 1)
+    t = datetime.datetime.now().isoformat()
+    history = TimelineHistory(1, '[[1,67197376,"no recovery target specified","' + t + '"]]',
+                              [(1, 67197376, 'no recovery target specified', t)])
+    cluster_config = cluster_config or ClusterConfig(1, {'check_timeline': True}, 1)
    return Cluster(initialize, cluster_config, leader, 10, members, failover, sync, history)


@@ -62,17 +63,6 @@ def get_cluster_initialized_with_only_leader(failover=None, cluster_config=None)
    return get_cluster(True, leader, [leader], failover, None, cluster_config)


-def get_cluster_not_initialized_standby(failover=None, sync=None):
-    return get_cluster_not_initialized_without_leader(
-        cluster_config=ClusterConfig(1, {
-            "standby_cluster": {
-                "host": "localhost",
-                "port": 5432,
-                "primary_slot_name": "",
-            }}, 1)
-    )
-
-
 def get_standby_cluster_initialized_with_only_leader(failover=None, sync=None):
    return get_cluster_initialized_with_only_leader(
        cluster_config=ClusterConfig(1, {
@@ -84,12 +74,13 @@ def get_standby_cluster_initialized_with_only_leader(failover=None, sync=None):
    )


-def get_node_status(reachable=True, in_recovery=True, wal_position=10, nofailover=False, watchdog_failed=False):
+def get_node_status(reachable=True, in_recovery=True, timeline=2,
+                    wal_position=10, nofailover=False, watchdog_failed=False):
    def fetch_node_status(e):
        tags = {}
        if nofailover:
            tags['nofailover'] = True
-        return _MemberStatus(e, reachable, in_recovery, wal_position, tags, watchdog_failed)
+        return _MemberStatus(e, reachable, in_recovery, timeline, wal_position, tags, watchdog_failed)
    return fetch_node_status


@@ -127,6 +118,7 @@ zookeeper:
        sys.argv = sys.argv[:1]

        self.config = Config()
+        self.config.set_dynamic_configuration({'maximum_lag_on_failover': 5})
        self.postgresql = p
        self.dcs = d
        self.api = Mock()
@@ -158,8 +150,8 @@ def run_async(self, func, args=()):
@patch.object(Postgresql, 'write_recovery_conf', Mock())
@patch.object(Postgresql, 'query', Mock())
@patch.object(Postgresql, 'checkpoint', Mock())
-@patch.object(Postgresql, 'call_nowait', Mock())
@patch.object(Postgresql, 'cancellable_subprocess_call', Mock(return_value=0))
+@patch.object(Postgresql, '_get_local_timeline_lsn_from_replication_connection', Mock(return_value=[2, 10]))
@patch.object(etcd.Client, 'write', etcd_write)
@patch.object(etcd.Client, 'read', etcd_read)
@patch.object(etcd.Client, 'delete', Mock(side_effect=etcd.EtcdException))
@@ -179,7 +171,6 @@ class TestHa(unittest.TestCase):
            mock_machines.__get__ = Mock(return_value=['http://remotehost:2379'])
            self.p = Postgresql({'name': 'postgresql0', 'scope': 'dummy', 'listen': '127.0.0.1:5432',
                                 'data_dir': 'data/postgresql0', 'retry_timeout': 10,
-                                 'maximum_lag_on_failover': 5,
                                 'authentication': {'superuser': {'username': 'foo', 'password': 'bar'},
                                                    'replication': {'username': '', 'password': ''}},
                                 'parameters': {'wal_level': 'hot_standby', 'max_replication_slots': 5, 'foo': 'bar',
@@ -214,17 +205,10 @@ class TestHa(unittest.TestCase):
    @patch('patroni.dcs.etcd.Etcd.initialize', return_value=True)
    def test_start_as_standby_leader(self, initialize):
        self.p.data_directory_empty = true
-        self.ha.cluster = get_cluster_not_initialized_standby()
+        self.ha.cluster = get_cluster_not_initialized_without_leader(cluster_config=ClusterConfig(0, {}, 0))
        self.ha.cluster.is_unlocked = true
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
-        self.assertEqual(
-            self.ha.run_cycle(),
-            'trying to bootstrap a new standby leader'
-        )
+        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {"port": 5432}}
+        self.assertEqual(self.ha.run_cycle(), 'trying to bootstrap a new standby leader')

    @patch.object(Cluster, 'get_clone_member',
                  Mock(return_value=Member(0, 'test', 1, {'api_url': 'http://127.0.0.1:8011/patroni'})))
@@ -233,27 +217,7 @@ class TestHa(unittest.TestCase):
        self.p.data_directory_empty = true
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        self.ha.cluster.is_unlocked = false
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
-        self.assertEqual(
-            self.ha.run_cycle(),
-            "trying to bootstrap from replica 'test'"
-        )
-
-    @patch.object(Postgresql, 'create_replica', Mock(return_value=0))
-    def test_bootstrap_standby_leader(self):
-        self.ha.cluster = get_cluster_not_initialized_standby()
-        self.ha.cluster.is_unlocked = true
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
-        self.ha._post_bootstrap_task = CriticalTask()
-        self.assertEqual(self.ha.bootstrap_standby_leader(), True)
+        self.assertEqual(self.ha.run_cycle(), "trying to bootstrap from replica 'test'")

    def test_recover_replica_failed(self):
        self.p.controldata = lambda: {'Database cluster state': 'in recovery', 'Database system identifier': SYSID}
@@ -501,6 +465,8 @@ class TestHa(unittest.TestCase):
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        self.ha.fetch_node_status = get_node_status(watchdog_failed=True)
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
+        self.ha.fetch_node_status = get_node_status(timeline=1)
+        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        self.ha.fetch_node_status = get_node_status(wal_position=1)
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        # manual failover from the previous leader to us won't happen if we hold the nofailover flag
@@ -612,6 +578,8 @@ class TestHa(unittest.TestCase):
        self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        with patch('patroni.postgresql.Postgresql.timeline_wal_position', return_value=(1, 1)):
            self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
+        with patch('patroni.postgresql.Postgresql.replica_cached_timeline', return_value=1):
+            self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        self.ha.patroni.nofailover = True
        self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        self.ha.patroni.nofailover = False
@@ -692,11 +660,6 @@ class TestHa(unittest.TestCase):
    def test_process_healthy_standby_cluster_as_standby_leader(self):
        self.p.is_leader = false
        self.p.name = 'leader'
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        msg = 'no action.  i am the standby leader with the lock'
        self.assertEqual(self.ha.run_cycle(), msg)
@@ -704,24 +667,13 @@ class TestHa(unittest.TestCase):
    def test_process_healthy_standby_cluster_as_cascade_replica(self):
        self.p.is_leader = false
        self.p.name = 'replica'
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        msg = 'no action.  i am a secondary and i am following a leader'
        self.assertEqual(self.ha.run_cycle(), msg)

-    @patch('patroni.dcs.etcd.Etcd.initialize', return_value=True)
-    def test_process_unhealthy_standby_cluster_as_standby_leader(self, initialize):
+    def test_process_unhealthy_standby_cluster_as_standby_leader(self):
        self.p.is_leader = false
        self.p.name = 'leader'
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        self.ha.cluster.is_unlocked = true
        self.ha.sysid_valid = true
@@ -730,19 +682,30 @@ class TestHa(unittest.TestCase):
        self.assertEqual(self.ha.run_cycle(), msg)

    @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
-    @patch('patroni.dcs.etcd.Etcd.initialize', return_value=True)
-    def test_process_unhealthy_standby_cluster_as_cascade_replica(self, initialize):
+    def test_process_unhealthy_standby_cluster_as_cascade_replica(self):
        self.p.is_leader = false
        self.p.name = 'replica'
-        self.ha.patroni.config._dynamic_configuration = {"standby_cluster": {
-            "host": "localhost",
-            "port": 5432,
-            "primary_slot_name": "",
-        }}
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        self.ha.is_unlocked = true
-        msg = 'running pg_rewind from leader'
-        self.assertEqual(self.ha.run_cycle(), msg)
+        self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_master:'))
+
+    def test_recover_unhealthy_leader_in_standby_cluster(self):
+        self.p.is_leader = false
+        self.p.name = 'leader'
+        self.p.is_running = false
+        self.p.follow = false
+        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
+        self.assertEqual(self.ha.run_cycle(), 'starting as a standby leader because i had the session lock')
+
+    def test_recover_unhealthy_unlocked_standby_cluster(self):
+        self.p.is_leader = false
+        self.p.name = 'leader'
+        self.p.is_running = false
+        self.p.follow = false
+        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
+        self.ha.cluster.is_unlocked = true
+        self.ha.has_lock = false
+        self.assertEqual(self.ha.run_cycle(), 'trying to follow a remote master because standby cluster is unhealthy')

    def test_failed_to_update_lock_in_pause(self):
        self.ha.update_lock = false
@@ -1045,3 +1008,12 @@ class TestHa(unittest.TestCase):
        self.assertEqual(self.ha.run_cycle(), 'PAUSE: no action.  i am the leader with the lock')
        self.ha.is_paused = false
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
+
+    @patch('psycopg2.connect', psycopg2_connect)
+    def test_permanent_logical_slots_after_promote(self):
+        config = ClusterConfig(1, {'slots': {'l': {'database': 'postgres', 'plugin': 'test_decoding'}}}, 1)
+        self.ha.cluster = get_cluster_initialized_without_leader(cluster_config=config)
+        self.assertEqual(self.ha.run_cycle(), 'acquired session lock as a leader')
+        self.ha.cluster = get_cluster_initialized_without_leader(leader=True, cluster_config=config)
+        self.ha.has_lock = true
+        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
@@ -50,6 +50,13 @@ class TestKubernetes(unittest.TestCase):
                        'labels': {'f': 'b'}, 'use_endpoints': True, 'pod_ip': '10.0.0.0'})
        self.assertIsNotNone(k.update_leader('123'))

+    @patch('kubernetes.config.load_kube_config', Mock())
+    @patch.object(k8s_client.CoreV1Api, 'create_namespaced_endpoints', Mock())
+    def test_update_leader_with_restricted_access(self):
+        k = Kubernetes({'ttl': 30, 'scope': 'test', 'name': 'p-0', 'retry_timeout': 10,
+                        'labels': {'f': 'b'}, 'use_endpoints': True, 'pod_ip': '10.0.0.0'})
+        self.assertIsNotNone(k.update_leader('123', True))
+
    def test_take_leader(self):
        self.k.take_leader()
        self.k._leader_observed_record['leader'] = 'test'
@@ -0,0 +1,38 @@
+import os
+import sys
+import unittest
+import yaml
+
+from mock import Mock, patch
+from patroni.config import Config
+from patroni.log import PatroniLogger
+
+
+class TestPatroniLogger(unittest.TestCase):
+
+    @patch('logging.FileHandler._open', Mock())
+    def setUp(self):
+        self.config = {
+            'log': {
+                'dir': 'foo',
+                'file_size': 4096,
+                'file_num': 5,
+                'loggers': {
+                    'foo.bar': 'INFO'
+                }
+            },
+            'restapi': {}, 'postgresql': {'data_dir': 'foo'}
+        }
+        sys.argv = ['patroni.py']
+        os.environ[Config.PATRONI_CONFIG_VARIABLE] = yaml.dump(self.config, default_flow_style=False)
+        self.logger = PatroniLogger()
+        config = Config()
+        self.logger.reload_config(config['log'])
+
+    def test_rotating_handler(self):
+        self.assertEqual(self.logger.handler.maxBytes, self.config['log']['file_size'])
+        self.assertEqual(self.logger.handler.backupCount, self.config['log']['file_num'])
+
+    def test_reload_config(self):
+        self.config['log'].pop('dir')
+        self.logger.reload_config(self.config['log'])
@@ -8,7 +8,7 @@ import unittest

 from mock import Mock, MagicMock, PropertyMock, patch, mock_open
 from patroni.async_executor import CriticalTask
-from patroni.dcs import Cluster, Leader, Member, RemoteMember, SyncState
+from patroni.dcs import Cluster, ClusterConfig, Leader, Member, RemoteMember, SyncState
 from patroni.exceptions import PostgresConnectionException, PostgresException
 from patroni.postgresql import Postgresql, STATE_REJECT, STATE_NO_RESPONSE
 from patroni.postmaster import PostmasterProcess
@@ -28,15 +28,15 @@ class MockCursor(object):
    def execute(self, sql, *params):
        if sql.startswith('blabla'):
            raise psycopg2.ProgrammingError()
-        elif sql == 'CHECKPOINT':
+        elif sql == 'CHECKPOINT' or sql.startswith('SELECT pg_catalog.pg_create_'):
            raise psycopg2.OperationalError()
        elif sql.startswith('RetryFailedError'):
            raise RetryFailedError('retry')
        elif sql.startswith('SELECT slot_name'):
-            self.results = [('blabla',), ('foobar',)]
-        elif sql.startswith('SELECT CASE WHEN pg_is_in_recovery()'):
+            self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'a', 'b')]
+        elif sql.startswith('SELECT CASE WHEN pg_catalog.pg_is_in_recovery()'):
            self.results = [(1, 2)]
-        elif sql.startswith('SELECT pg_is_in_recovery()'):
+        elif sql.startswith('SELECT pg_catalog.pg_is_in_recovery()'):
            self.results = [(False, 2)]
        elif sql.startswith('WITH replication_info AS ('):
            replication_info = '[{"application_name":"walreceiver","client_addr":"1.2.3.4",' +\
@@ -53,7 +53,7 @@ class MockCursor(object):
            self.results = [('1', 2, '0/402EEC0', '')]
        elif sql.startswith('SELECT isdir, modification'):
            self.results = [(False, datetime.datetime.now())]
-        elif sql.startswith('SELECT pg_read_file'):
+        elif sql.startswith('SELECT pg_catalog.pg_read_file'):
            self.results = [('1\t0/40159C0\tno recovery target specified\n\n' +
                             '2\t1/40159C0\tno recovery target specified\n',)]
        elif sql.startswith('TIMELINE_HISTORY '):
@@ -421,9 +421,13 @@ class TestPostgresql(unittest.TestCase):
    def test_create_replica(self, mock_cancellable_subprocess_call):
        self.p.delete_trigger_file = Mock(side_effect=OSError)

+        self.p.config['create_replica_methods'] = ['pgBackRest']
+        self.p.config['pgBackRest'] = {'command': 'pgBackRest', 'keep_data': True, 'no_params': True}
+        mock_cancellable_subprocess_call.return_value = 0
+        self.assertEqual(self.p.create_replica(self.leader), 0)
+
        self.p.config['create_replica_methods'] = ['wale', 'basebackup']
        self.p.config['wale'] = {'command': 'foo'}
-        mock_cancellable_subprocess_call.return_value = 0
        self.assertEqual(self.p.create_replica(self.leader), 0)
        del self.p.config['wale']
        self.assertEqual(self.p.create_replica(self.leader), 0)
@@ -498,23 +502,25 @@ class TestPostgresql(unittest.TestCase):
    @patch.object(Postgresql, 'is_running', Mock(return_value=True))
    def test_sync_replication_slots(self):
        self.p.start()
-        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None, None, None)
+        config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'},
+                                             'A': 0, 'test_3': 0, 'b': {'type': 'logical', 'plugin': '1'}}}, 1)
+        cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem], None, None, None)
        with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg2.OperationalError)):
            self.p.sync_replication_slots(cluster)
        self.p.sync_replication_slots(cluster)
        with mock.patch('patroni.postgresql.Postgresql.role', new_callable=PropertyMock(return_value='replica')):
            self.p.sync_replication_slots(cluster)
-        with mock.patch('patroni.postgresql.logger.error', new_callable=Mock()) as errorlog_mock:
+        with patch.object(Postgresql, 'drop_replication_slot', Mock(return_value=True)),\
+                patch('patroni.dcs.logger.error', new_callable=Mock()) as errorlog_mock:
            self.p.query = Mock()
            alias1 = Member(0, 'test-3', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres'})
            alias2 = Member(0, 'test.3', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres'})
            cluster.members.extend([alias1, alias2])
            self.p.sync_replication_slots(cluster)
-            errorlog_mock.assert_called_once()
-            self.assertTrue("test-3" in errorlog_mock.call_args[0][1],
-                            "non matching {0}".format(errorlog_mock.call_args[0][1]))
-            self.assertTrue("test.3" in errorlog_mock.call_args[0][1],
-                            "non matching {0}".format(errorlog_mock.call_args[0][1]))
+            self.assertEqual(errorlog_mock.call_count, 5)
+            ca = errorlog_mock.call_args_list[0][0][1]
+            self.assertTrue("test-3" in ca, "non matching {0}".format(ca))
+            self.assertTrue("test.3" in ca, "non matching {0}".format(ca))

    @patch.object(MockCursor, 'execute', Mock(side_effect=psycopg2.OperationalError))
    def test__query(self):
@@ -645,6 +651,7 @@ class TestPostgresql(unittest.TestCase):

    @patch('time.sleep', Mock())
    @patch('os.unlink', Mock())
+    @patch('os.path.isfile', Mock(return_value=True))
    @patch.object(Postgresql, 'run_bootstrap_post_init', Mock(return_value=True))
    @patch.object(Postgresql, '_custom_bootstrap', Mock(return_value=True))
    @patch.object(Postgresql, 'start', Mock(return_value=True))
@@ -67,7 +67,7 @@ class TestPostmasterProcess(unittest.TestCase):
    @patch('psutil.wait_procs')
    def test_wait_for_user_backends_to_close(self, mock_wait):
        c1 = Mock()
-        c1.cmdline = Mock(return_value=["postgres: startup process"])
+        c1.cmdline = Mock(return_value=["postgres: startup process   "])
        c2 = Mock()
        c2.cmdline = Mock(return_value=["postgres: postgres postgres [local] idle"])
        c3 = Mock()
@@ -77,8 +77,7 @@ class TestPostmasterProcess(unittest.TestCase):
            self.assertIsNone(proc.wait_for_user_backends_to_close())
            mock_wait.assert_called_with([c2])

-        c3.cmdline = Mock(side_effect=psutil.AccessDenied(123))
-        with patch('psutil.Process.children', Mock(return_value=[c3])):
+        with patch('psutil.Process.children', Mock(side_effect=psutil.NoSuchProcess(123))):
            proc = PostmasterProcess(123)
            self.assertIsNone(proc.wait_for_user_backends_to_close())

@@ -1,3 +1,4 @@
+import select
 import six
 import unittest

@@ -113,6 +114,11 @@ class TestPatroniSequentialThreadingHandler(unittest.TestCase):
    def test_create_connection(self):
        self.assertIsNotNone(self.handler.create_connection(()))
        self.assertIsNotNone(self.handler.create_connection((), 40))
+        self.assertIsNotNone(self.handler.create_connection(timeout=40))
+
+    @patch.object(SequentialThreadingHandler, 'select', Mock(side_effect=ValueError))
+    def test_select(self):
+        self.assertRaises(select.error, self.handler.select)


 class TestZooKeeper(unittest.TestCase):
Author	SHA1	Message	Date
Alexander KukushkinandGitHub	381a5b80d2	Release 1.5.4 (#931 ) * Bump version * Update release notes * Make it possible to configure registration of Service in Consul via env variables	2019-01-15 12:14:19 +01:00
Alexander KukushkinandGitHub	71dae6a905	Optionally consider node not healthy if it is not on the latest timeline (#892 ) The latest timeline is calculated from the `/history` key in DCS. In case there is no such key or it contains some garbage we consider the node healthy. Closes https://github.com/zalando/patroni/issues/890	2019-01-15 11:16:30 +01:00
Alexander KukushkinandGitHub	cf34fb3934	Relax requirements on superuser credentials (#930 ) libpq allows opening connection without explicitly specifying neither username nor password. Depending on situation it would rely either on `pgpass` file or trust authentication method in pg_hba.conf. Since pg_rewind is also using libpq, it could work the same way. Fixes https://github.com/zalando/patroni/issues/928	2019-01-15 11:15:35 +01:00
Alexander KukushkinandGitHub	e080ded44b	Make logging configurable via YAML file (#927 ) It allows changing logging settings in runtime by updating config and doing reload or sending `SIGHUP` to the Patroni process. Important! Environment configuration names related to logging were renamed and documentation accordingly updated. For compatibility reasons Patroni still accepts `PATRONI_LOGLEVEL` and `PATRONI_FORMAT`, but some other variables related to logging, which were introduced only recently (between releases), will stop working. I think it is ok, since we didn't release the new version yet and therefore it is very unlikely that somebody is using them except authors of corresponding PRs. Example of log section in the config file: ```yaml log: dir: /where/to/write/patroni/logs # if not specified, write logs to stderr file_size: 50000000 # 50MB file_num: 10 # keep history of 10 files dateformat: '%Y-%m-%d %H:%M:%S' loggers: # increase log verbosity for etcd.client and urllib3 etcd.client: DEBUG urllib3: DEBUG ```	2019-01-15 08:42:13 +01:00
jouirandAlexander Kukushkin	dec3656f6e	Redirect HTTPServer exceptions to logger (#900 ) (#925 ) By default they were written to stdout	2019-01-15 08:37:06 +01:00
Alexander KukushkinandGitHub	1e2d89fa58	Apply 5 second backoff when loading global config up on start (#922 ) It doesn't make much sense to hammer DCS when we just starting up. Fixes https://github.com/zalando/patroni/issues/919	2019-01-14 14:55:56 +01:00
Alexander KukushkinandGitHub	994863c18d	Refactor wait_for_user_backends_to_close method (#917 ) 1. Log only debug level messages on any kind of error 2. Update regexp for matching postgres aux processes to make it compatible with postgres 11 Fixes https://github.com/zalando/patroni/issues/914	2019-01-14 14:55:45 +01:00
Alexander KukushkinandGitHub	3fce982909	Set archive_mode to off during the custom bootstrap (#911 ) We want to avoid archiving WALs and history files until the cluster is fully functional. It should really help if the custom bootstrap involves pg_upgrade.	2019-01-14 14:55:23 +01:00
Lucas CapistrantandAlexander Kukushkin	d306092cbc	Explicitly secure rw perms for recovery.conf at creation time (#910 ) We don't want anybody except patroni/postgres user reading this file, it contains replication user and password.	2019-01-14 14:22:04 +01:00
Dmitry DolgovandAlexander Kukushkin	11f7ceb521	Do not check types of standby_cluster configuration (#924 ) Simply allow valid keys	2019-01-14 14:16:15 +01:00
bradnicholsonandAlexander Kukushkin	05a13839aa	Update replica_bootstrap.rst (#915 ) Add some docs about replication slots for standby clusters	2019-01-14 12:57:21 +01:00
Étienne MandAlexander Kukushkin	04ac199fc8	Single quotes are mandatory around each host in PATRONI_ETCD_HOSTS (#926 ) Otherwise YAML parser fails	2019-01-14 11:56:15 +01:00
anikin-aaandAlexander Kukushkin	0be8a9527b	possibility to set `logdatefmt` via env. (#904 ) The value could be configured via `PATRONI_LOGDATEFMT` environment variable. The default value is `%Y-%m-%d %H:%M:%S`	2019-01-14 08:46:38 +01:00
Pavel KirillovandAlexander Kukushkin	929ff08bfd	Service deregister timeout must be in Go time format (#893 )	2018-12-21 15:42:15 +01:00
Cody CoonsandAlexander Kukushkin	7bc8d0aac9	Removed stderr pipe to stdout on pg_ctl process (#896 ) Inheriting stderr from the main Patroni process allows all Postgres logs to be seen along with all patroni logs. This is very useful in a container environment as Patroni and Postgres logs may be consumed using standard tools (docker logs, kubectl, etc). In addition to that, this change fixes a bug with Patroni not being able to catch postmaster pid when postgres writing some warnings into stderr	2018-12-21 15:41:34 +01:00
Lucas CapistrantandAlexander Kukushkin	f3da6de129	Add ability to configure app logs to be written to a file (#903 ) It gives users the option to send Patroni application logs to a File instead of Standard Out. There are three environment variables that can be set to enable and configure file logging. 1. `PATRONI_FILE_LOG_DIR`: Path to a directory that is writeable by the executing user. Having this variable set is what activates file logging. 2. `PATRONI_FILE_LOG_NUM`: This is a rolling file logger. This variable dictates how many log files are retained. 3. `PATRONI_FILE_LOG_SIZE`: This variable dictates the size at which the logs will roll. If `PATRONI_FILE_LOG_DIR` is not set than Patroni will log to stderr (default behavior does not change) Closes https://github.com/zalando/patroni/issues/902	2018-12-21 15:38:29 +01:00
Alexander KukushkinandGitHub	491f230711	Release 1.5.3 (#889 ) * Bump version * Update release notes	2018-12-03 17:12:53 +01:00
Alexander KukushkinandGitHub	f1d7ccf36e	Make sure we refresh session at least once per HA loop (#880 ) Fixes https://github.com/zalando/patroni/issues/879	2018-12-03 16:35:14 +01:00
Kostiantyn NemchenkoandAlexander Kukushkin	96ea01bee4	Fix kubernetes demo files (#885 ) - Update postgres docker image to the latest 11 version. - Remove empty lines inside the `RUN` command to make the Dockerfile compatible with future docker versions. - Set the `PATRONI_KUBERNETES_POD_IP` environment variable, which is required when _use_endpoints_ is enabled. Otherwise, the `KeyError` is raised [here](https://github.com/zalando/patroni/blob/master/patroni/dcs/kubernetes.py#L95). - Set `EDITOR` environment variable to make configuration changes via `patronictl edit-config`.	2018-12-03 15:46:25 +01:00
Alexander KukushkinandGitHub	e684ca66e5	Compatibility with postgres 9.3 (#882 ) use replication=1 instead of replication='database' when opening replication connection, 9.3 doesn't understand it	2018-11-30 15:46:05 +01:00
Alexander KukushkinandGitHub	1a0876e5ca	Refactor acceptance tests to improve stability (#884 ) Hope it will crash less often when executed on travis against k8s	2018-11-30 12:40:56 +01:00
Alexander KukushkinandGitHub	9bf074acfb	Compatibility with python3 (#883 ) Change of `loop_wait` was causing Patroni to disconnect from zookeeper and never reconnect back. The error was happening only with python3 due to a difference in implementation of `select.select` function.	2018-11-30 11:40:34 +01:00
Alexander KukushkinandGitHub	f8f928420d	Release 1.5.2 (#875 ) * Update release notes * Bump version	2018-11-26 10:31:14 +01:00
Shea StewartandAlexander Kukushkin	6519a192b1	add openshift customizations, templates, and test (#871 ) - It modifies the Dockerfile and entrypoint slightly to allow for OpenShift SCCs to operate correctly - It adds 2 template examples that can be easily modified by changing parameters Fixes #572	2018-11-21 18:01:39 +01:00
Alexander KukushkinandGitHub	8c7e1892ee	touch_member method should not raise exceptions (#859 ) Fixes https://github.com/zalando/patroni/issues/853	2018-11-21 12:51:52 +01:00
Lardière SébastienandAlexander Kukushkin	a1ba2cdca7	Add confd template for pgbouncer (#844 )	2018-11-21 12:03:22 +01:00
Kostiantyn NemchenkoandAlexander Kukushkin	ce9c7bdadc	Describe Consul registration parameters (#870 ) Mention Consul related parameters introduced by #802	2018-11-21 12:00:40 +01:00
Alexander KukushkinandGitHub	fb01aaebc5	Compatibility with kazoo-2.6.0 (#872 ) Recently 2.6.0 was release which changes the way how create_connection method is called. Before it was passing two arguments, and in the new version all argument names are specified explicitly.	2018-11-19 14:26:20 +01:00
alago197andAlexander Kukushkin	a13cc8b847	Update SETTINGS.rst with connect_address clarifications (#858 )	2018-11-12 16:56:51 +01:00
lwsboxandAlexander Kukushkin	bd9f30372e	fix bug: change pip to pip3 (#851 ) should use python3 to install in postgres10 image.	2018-11-05 14:17:22 +01:00
Josh BerkusandAlexander Kukushkin	d247b5ae17	Fix several build problems with the Docker image for testing Kubernet… (#735 ) 1. Update to Postgres 10 2. Install most of the python modules from deb packages 3. Do not upgrade pip	2018-11-04 08:31:11 +01:00
Alexander KukushkinandGitHub	b4f35ecca0	Release 1.5.1 (#846 ) * Bump version * Update release notes	2018-11-01 16:18:41 +01:00
Alexander KukushkinandGitHub	0f666e69f3	Prefix system tables, views and functions with pg_catalog (#845 ) and implement missing unit tests	2018-11-01 16:17:40 +01:00
Alexander KukushkinandGitHub	2efd97baab	Permanent replication slots (#819 ) Permanent replication slots are preserved on failover/switchover, that is Patroni on the new primary will create configured replication slots right after doing promote. Slots could be configured with the help of `patronictl edit-config`. The initial configuration could be also done in the `bootstrap.dcs` ```yaml slots: permanent_physical_1: type: physical permanent_logical_1: type: logical database: foo plugin: pgoutput ``` It is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots. Closes https://github.com/zalando/patroni/issues/656	2018-10-31 11:37:42 +01:00
Alexander KukushkinandGitHub	c65c4f1ffe	Round-robing across all masters in pause mode if DCS is not accessible (#842 ) Regression was introduced in https://github.com/zalando/patroni/commit/90cf930036a9d5249265af15d2b787ec7517cf57 Fixes https://github.com/zalando/patroni/issues/841	2018-10-30 16:48:53 +01:00
SrikanthandAlexander Kukushkin	2264190e79	Add pre-command to start watchdog (#835 ) Add `ExecStartPre` commands to systemd unit to help start watchdog devices.	2018-10-26 07:55:01 +01:00
Alexander KukushkinandGitHub	f70edefc65	A few bugfixes in the "standby cluster" workflow (#823 ) * Always run `pg_rewind` against the remote master * Always use the remote master as the source when "recovering" stopped standby leader * Use remote master as the source when "recovering" the node in the unhealthy cluster * Use the local dbname as the fallback when doing `pg_rewind` from the remote master * `no_replication_slot` is the allowed key in the `RemoteMember` object * Make it possible to "bootstrap" the new `standby_cluster` with existing (and valid) data directory. There is one prerequisite though, there should be no `patroni.dynamic.json` file in it!	2018-10-09 13:30:48 +02:00
Yogesh SharmaandAlexander Kukushkin	6567f509b1	Add pgbackrest support (#1 ) (#822 ) * pgbackrest support pgBackrest can restore in existing $PGDATA folder, this allows speedy restore as files which have not changed since last backup are skipped, to support this feature new param keep_data has been introduced. When keep_data=True, cleanup of $PGDATA will be skipped. Patroni passes some extra parameters to custom_replica_methods when calling, this causes an error due to pgbackrest strict parameter checking. New param no_params=True can be set to skip parameters passing. Fixes https://github.com/zalando/patroni/issues/625	2018-10-08 19:00:30 +02:00