Release 1.5.5 (#979 )

* Bump version * Update release notes
Don't write primary_conninfo into recovery.conf for wal only standby cluster (#971 )
2026-08-26 07:30:14 +00:00 · 2019-02-15 16:14:39 +01:00 · 2019-02-15 13:35:34 +01:00 · 2019-02-15 13:35:22 +01:00 · 2019-02-15 13:35:04 +01:00 · 2019-02-13 16:02:12 +01:00
43 changed files with 758 additions and 229 deletions
@@ -51,3 +51,6 @@ scm-source.json
 docs/build/
 docs/source/_static/
 docs/source/_templates/
+
+# Pycharm IDE
+.idea/
@@ -137,7 +137,7 @@ script:
            echo Running acceptance tests using python${pv}
            if ! PATH=.:/usr/lib/postgresql/9.6/bin:$PATH $TEST_SUITE; then
                # output all log files when tests are failing
-                grep . features/output/*/*postgres?.*
+                grep . features/output/*_failed/*postgres?.*
                exit 1
            fi
        fi
@@ -11,8 +11,13 @@ Global/Universal
 -  **PATRONI\_NAME**: name of the node where the current instance of Patroni is running. Must be unique for the cluster.
 -  **PATRONI\_NAMESPACE**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **PATRONI\_SCOPE**: cluster name
-  **PATRONI\_LOGLEVEL**: sets the general logging level (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
-  **PATRONI\_REQUESTS_LOGLEVEL**: sets the logging level for all HTTP requests e.g. Kubernetes API calls (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **PATRONI\_LOG\_LEVEL**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **PATRONI\_LOG\_FORMAT**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
+-  **PATRONI\_LOG\_DATEFORMAT**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
+-  **PATRONI\_LOG\_DIR**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this env variable, the application will retain 4 25MB logs by default. You can tune those retention values with `PATRONI_LOG_FILE_NUM` and `PATRONI_LOG_FILE_SIZE` (see below).
+-  **PATRONI\_LOG\_FILE\_NUM**: The number of application logs to retain.
+-  **PATRONI\_LOG\_FILE\_SIZE**: Size of patroni.log file (in bytes) that triggers a log rolling.
+-  **PATRONI\_LOG\_LOGGERS**: Redefine logging level per python module. Example ``PATRONI_LOG_LOGGERS="{patroni.postmaster: WARNING, urllib3: DEBUG}"``

 Bootstrap configuration
 -----------------------
@@ -36,16 +41,22 @@ Consul
 -  **PATRONI\_CONSUL\_KEY**: (optional) File with the client key. Can be empty if the key is part of certificate.
 -  **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable.
+-  **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
+-  **PATRONI\_CONSUL\_SERVICE\_CHECK\_INTERVAL**: (optional) how often to perform health check against registered url

 Etcd
 ----
-  **PATRONI\_ETCD\_HOST**: the host:port for the etcd endpoint.
-  **PATRONI\_ETCD\_HOSTS**: list of etcd endpoints in format host1:port1,host2:port2,etc...
-  **PATRONI\_ETCD\_URL**: url for the etcd, in format: http(s)://(username:password@)host:port
+
 -  **PATRONI\_ETCD\_PROXY**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **PATRONI\_ETCD\_URL**
+-  **PATRONI\_ETCD\_URL**: url for the etcd, in format: http(s)://(username:password@)host:port
+-  **PATRONI\_ETCD\_HOSTS**: list of etcd endpoints in format 'host1:port1','host2:port2',etc...
+-  **PATRONI\_ETCD\_PROTOCOL**: http or https, if not specified http is used. If the **url** or **proxy** is specified - will take protocol from them.
+-  **PATRONI\_ETCD\_HOST**: the host:port for the etcd endpoint.
 -  **PATRONI\_ETCD\_SRV**: Domain to search the SRV record(s) for cluster autodiscovery.
+-  **PATRONI\_ETCD\_USERNAME**: username for etcd authentication.
+-  **PATRONI\_ETCD\_PASSWORD**: password for etcd authentication.
 -  **PATRONI\_ETCD\_CACERT**: The ca certificate. If present it will enable validation.
-  **PATRONI\_ETCD\_CERT**: File with the client certificate
+-  **PATRONI\_ETCD\_CERT**: File with the client certificate.
 -  **PATRONI\_ETCD\_KEY**: File with the client key. Can be empty if the key is part of certificate.

 Exhibitor
@@ -79,7 +90,7 @@ PostgreSQL
 -  **PATRONI\_SUPERUSER\_PASSWORD**: password for the superuser, set during initialization (initdb).

 REST API
-------- 
+--------
 -  **PATRONI\_RESTAPI\_CONNECT\_ADDRESS**: IP address and port to access the REST API.
 -  **PATRONI\_RESTAPI\_LISTEN**: IP address and port that Patroni will listen to, to provide health-check information for HAProxy.
 -  **PATRONI\_RESTAPI\_USERNAME**: Basic-auth username to protect unsafe REST API endpoints.
@@ -66,6 +66,8 @@ Note that external tools to call in the replica creation or custom bootstrap scr
 independently of Patroni.


+.. _running_configuring:
+
 Running and Configuring
 -----------------------

@@ -10,6 +10,20 @@ Global/Universal
 -  **namespace**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
 -  **scope**: cluster name

+Log
+---
+-  **level**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
+-  **format**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
+-  **dateformat**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
+-  **dir**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this value, the application will retain 4 25MB logs by default. You can tune those retention values with `file_num` and `file_size` (see below).
+-  **file\_num**: The number of application logs to retain.
+-  **file\_size**: Size of patroni.log file (in bytes) that triggers a log rolling.
+-  **loggers**: This section allows redefining logging level per python module
+    -  **patroni.postmaster: WARNING**
+    -  **urllib3: DEBUG**
+
+.. _bootstrap_settings:
+
 Bootstrap configuration
 -----------------------
 -  **dcs**: This section will be written into `/<namespace>/<scope>/config` of a given configuration store after initializing of new cluster. This is the global configuration for the cluster. If you want to change some parameters for all cluster nodes - just do it in DCS (or via Patroni API) and all nodes will apply this configuration.
@@ -23,7 +37,7 @@ Bootstrap configuration
    -  **postgresql**:
        -  **use\_pg\_rewind**: whether or not to use pg_rewind
        -  **use\_slots**: whether or not to use replication_slots. Must be False for PostgreSQL 9.3. You should comment out max_replication_slots before it becomes ineligible for leader status.
-        -  **recovery\_conf**: additional configuration settings written to recovery.conf when configuring follower. 
+        -  **recovery\_conf**: additional configuration settings written to recovery.conf when configuring follower.
        -  **parameters**: list of configuration settings for Postgres. Many of these are required for replication to work.
    -  **standby\_cluster**: if this section is defined, we want to bootstrap a standby cluster.
        -  **host**: an address of remote master
@@ -87,10 +101,10 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **proxy**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **url**
 -  **srv**: Domain to search the SRV record(s) for cluster autodiscovery.
 -  **protocol**: (optional) http or https, if not specified http is used. If the **url** or **proxy** is specified - will take protocol from them.
-  **username**: (optional) username for etcd authentication
+-  **username**: (optional) username for etcd authentication.
 -  **password**: (optional) password for etcd authentication.
 -  **cacert**: (optional) The ca certificate. If present it will enable validation.
-  **cert**: (optional) file with the client certificate
+-  **cert**: (optional) file with the client certificate.
 -  **key**: (optional) file with the client key. Can be empty if the key is part of **cert**.

 Exhibitor
@@ -132,7 +146,7 @@ PostgreSQL
 -  **create\_replica\_methods**: an ordered list of the create methods for turning a Patroni node into a new replica.
   "basebackup" is the default method; other methods are assumed to refer to scripts, each of which is configured as its
   own config item. See :ref:`custom replica creation methods documentation <custom_replica_creation>` for further explanation.
-  **data\_dir**: The location of the Postgres data directory, either existing or to be initialized by Patroni.
+-  **data\_dir**: The location of the Postgres data directory, either :ref:`existing <existing_data>` or to be initialized by Patroni.
 -  **config\_dir**: The location of the Postgres configuration directory, defaults to the data directory. Must be writable by Patroni.
 -  **bin\_dir**: Path to PostgreSQL binaries (pg_ctl, pg_rewind, pg_basebackup, postgres). The default value is an empty string meaning that PATH environment variable will be used to find the executables.
 -  **listen**: IP address + port that Postgres listens to; must be accessible from other nodes in the cluster, if you're using streaming replication. Multiple comma-separated addresses are permitted, as long as the port component is appended after to the last one with a colon, i.e. ``listen: 127.0.0.1,127.0.0.2:5432``. Patroni will use the first address from this list to establish local connections to the PostgreSQL node.
@@ -147,10 +161,11 @@ PostgreSQL
 -  **pg\_ctl\_timeout**: How long should pg_ctl wait when doing ``start``, ``stop`` or ``restart``. Default value is 60 seconds.
 -  **use\_pg\_rewind**: try to use pg\_rewind on the former leader when it joins cluster as a replica.
 -  **remove\_data\_directory\_on\_rewind\_failure**: If this option is enabled, Patroni will remove postgres data directory and recreate replica. Otherwise it will try to follow the new leader. Default value is **false**.
+-  **remove\_data\_directory\_on\_diverged\_timelines**: Patroni will remove postgres data directory and recreate replica if it notices that timelines are diverging and the former master can not start streaming from the new master. This option is useful when ``pg_rewind`` can not be used. Default value is **false**.
 -  **replica\_method**: for each create_replica_methods other than basebackup, you would add a configuration section of the same name. At a minimum, this should include "command" with a full path to the actual script to be executed. Other configuration parameters will be passed along to the script in the form "parameter=value".

 REST API
-------- 
+--------
 -  **connect\_address**: IP address (or hostname) and port, to access the Patroni's REST API. All the members of the cluster must be able to connect to this address, so unless the Patroni setup is intended for a demo inside the localhost, this address must be a non "localhost" or loopback addres (ie: "localhost" or "127.0.0.1"). It can serve as a endpoint for HTTP health checks (read below about the "listen" REST API parameter), and also for user queries (either directly or via the REST API), as well as for the health checks done by the cluster members during leader elections (for example, to determine whether the master is still running, or if there is a node which has a WAL position that is ahead of the one doing the query; etc.) The connect_address is put in the member key in DCS, making it possible to translate the member name into the address to connect to its REST API.

 -  **listen**: IP address (or hostname) and port that Patroni will listen to for the REST API - to provide also the same health checks and cluster messaging between the participating nodes, as described above. to provide health-check information for HAProxy (or any other load balancer capable of doing a HTTP "OPTION" or "GET" checks).
@@ -168,7 +183,7 @@ REST API
 CTL
 ---
 -  **Optional**:
-    -  **insecure**: Allow connections to REST API without verifying SSL certs. 
+    -  **insecure**: Allow connections to REST API without verifying SSL certs.
    -  **cacert**: Specifices the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs.
    -  **certfile**: Specifies the file with the certificate in the PEM format to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "certfile" parameter.

@@ -76,6 +76,7 @@ Also, the following Patroni configuration options can be changed only dynamicall
 - loop_wait: 10
 - retry_timeouts: 10
 - maximum_lag_on_failover: 1048576
+- check_timeline: false
 - postgresql.use_slots: true

 Upon changing these options, Patroni will read the relevant section of the configuration stored in DCS and change its
@@ -0,0 +1,35 @@
+.. _existing_data:
+
+Convert a Standalone to a Patroni Cluster
+=========================================
+
+This section describes the process for converting a standalone PostgreSQL instance into a Patroni cluster.
+
+To deploy a Patroni cluster without using a pre-existing PostgreSQL instance, see :ref:`Running and Configuring <running_configuring>` instead.
+
+Procedure
+---------
+
+A Patroni cluster can be started with a data directory from a single-node PostgreSQL database. This is achieved by following closely these steps:
+
+#. Manually start PostgreSQL daemon
+#. Create Patroni superuser and replication users as defined in the :ref:`authentication <postgresql_settings>` section of the Patroni configuration. If this user is created in SQL, the following queries achieve this:
+
+.. code-block:: sql
+  CREATE USER $PATRONI_SUPERUSER_USERNAME WITH SUPERUSER ENCRYPTED PASSWORD '$PATRONI_SUPERUSER_PASSWORD';
+  CREATE USER $PATRONI_REPLICATION_USERNAME WITH REPLICATION ENCRYPTED PASSWORD '$PATRONI_REPLICATION_PASSWORD';
+
+#. Start Patroni (e.g. ``patroni /etc/patroni/patroni.yml``). It automatically detects that PostgreSQL daemon is already running but its configuration might be out-of-date.
+#. Ask Patroni to restart the node with ``patronictl restart cluster-name node-name``.
+
+
+FAQ
+---
+
+#. During Patroni startup, Patroni complains that it cannot bind to the PostgreSQL port.
+
+You need to verify ``listen_addresses`` and ``port`` in ``postgresql.conf`` and ``postgresql.listen`` in ``patroni.yml``. Don't forget that ``pg_hba.conf`` should allow such access.
+
+#. After asking Patroni to restart the node, PostgreSQL displays the error message ``could not open configuration file "/etc/postgresql/10/main/pg_hba.conf": No such file or directory``
+
+It can mean various things depending on how you manage PostgreSQL configuration. If you specified `postgresql.config_dir`, Patroni generates the ``pg_hba.conf`` based on the settings in the :ref:`bootstrap <bootstrap_settings>` section only when it bootstraps a new cluster. In this scenario the ``PGDATA`` was not empty, therefore no bootstrap happened. This file must exist beforehand.
@@ -3,6 +3,120 @@
 Release notes
 =============

+Version 1.5.5
+-------------
+
+This version introduces the possibility of automatic reinit of the former master, improves patronictl list output and fixes a number of bugs.
+
+**New features**
+
+- Add support of `PATRONI_ETCD_PROTOCOL`, `PATRONI_ETCD_USERNAME` and `PATRONI_ETCD_PASSWORD` environment variables (Étienne M)
+
+  Before it was possible to configure them only in the config file or as a part of `PATRONI_ETCD_URL`, which is not always convenient.
+
+- Make it possible to automatically reinit the former master (Alexander Kukushkin)
+
+  If the pg_rewind is disabled or can't be used, the former master could fail to start as a new replica due to diverged timelines. In this case, the only way to fix it is wiping the data directory and reinitializing. This behavior could be changed by setting `postgresql.remove_data_directory_on_diverged_timelines`. When it is set, Patroni will wipe the data directory and reinitialize the former master automatically.
+
+- Show information about timelines in patronictl list (Alexander)
+
+  It helps to detect stale replicas. In addition to that, `Host` will include ':{port}' if the port value isn't default or there is more than one member running on the same host.
+
+- Create a headless service associated with the $SCOPE-config endpoint (Alexander)
+
+  The "config" endpoint keeps information about the cluster-wide Patroni and Postgres configuration, history file, and last but the most important, it holds the `initialize` key. When the Kubernetes master node is restarted or upgraded, it removes endpoints without services. The headless service will prevent it from being removed.
+
+**Bug fixes**
+
+- Adjust the read timeout for the leader watch blocking query (Alexander)
+
+  According to the Consul documentation, the actual response timeout is increased by a small random amount of additional wait time added to the supplied maximum wait time to spread out the wake up time of any concurrent requests. It adds up to `wait / 16` additional time to the maximum duration. In our case we are adding `wait / 15` or 1 second depending on what is bigger.
+
+- Always use replication=1 when connecting via replication protocol to the postgres (Alexander)
+
+  Starting from Postgres 10 the line in the pg_hba.conf with database=replication doesn't accept connections with the parameter replication=database.
+
+- Don't write primary_conninfo into recovery.conf for wal-only standby cluster (Alexander)
+
+  Despite not having neither `host` nor `port` defined in the `standby_cluster` config, Patroni was putting the `primary_conninfo` into the `recovery.conf`, which is useless and generating a lot of errors.
+
+
+Version 1.5.4
+-------------
+
+This version implements flexible logging and fixes a number of bugs.
+
+**New features**
+
+- Improvements in logging infrastructure (Alexander Kukushkin, Lucas Capistrant, Alexander Anikin)
+
+  Logging configuration could be configured not only from environment variables but also from Patroni config file. It makes it possible to change logging configuration in runtime by updating config and doing reload or sending SIGHUP to the Patroni process. By default Patroni writes logs to stderr, but now it becomes possible to write logs directly into the file and rotate when it reaches a certain size. In addition to that added support of custom dateformat and the possibility to fine-tune log level for each python module.
+
+- Make it possible to take into account the current timeline during leader elections (Alexander Kukushkin)
+
+  It could happen that the node is considering itself as a healthiest one although it is currently not on the latest known timeline. In some cases we want to avoid promoting of such node, which could be achieved by setting `check_timeline` parameter to `true` (default behavior remains unchanged).
+
+- Relaxed requirements on superuser credentials
+
+  Libpq allows opening connections without explicitly specifying neither username nor password. Depending on situation it relies either on pgpass file or trust authentication method in pg_hba.conf. Since pg_rewind is also using libpq, it will work the same way.
+
+- Implemented possibility to configure Consul Service registration and check interval via environment variables (Alexander Kukushkin)
+
+  Registration of service in Consul was added in the 1.5.0, but so far it was only possible to turn it on via patroni.yaml.
+
+**Stability Improvements**
+
+- Set archive_mode to off during the custom bootstrap (Alexander Kukushkin)
+
+  We want to avoid archiving wals and history files until the cluster is fully functional.  It really helps if the custom bootstrap involves pg_upgrade.
+
+- Apply five seconds backoff when loading global config on start (Alexander Kukushkin)
+
+  It helps to avoid hammering DCS when Patroni just starting up.
+
+- Reduce amount of error messages generated on shutdown (Alexander Kukushkin)
+
+  They were harmless but rather annoying and sometimes scary.
+
+- Explicitly secure rw perms for recovery.conf at creation time (Lucas)
+
+  We don't want anybody except patroni/postgres user reading this file, because it contains replication user and password.
+
+- Redirect HTTPServer exceptions to logger (Julien Riou)
+
+  By default, such exceptions were logged on standard output messing with regular logs.
+
+**Bug fixes**
+
+- Removed stderr pipe to stdout on pg_ctl process (Cody Coons)
+
+  Inheriting stderr from the main Patroni process allows all Postgres logs to be seen along with all patroni logs. This is very useful in a container environment as Patroni and Postgres logs may be consumed using standard tools (docker logs, kubectl, etc). In addition to that, this change fixes a bug with Patroni not being able to catch postmaster pid when postgres writing some warnings into stderr.
+
+- Set Consul service check deregister timeout in Go time format (Pavel Kirillov)
+
+  Without explicitly mentioned time unit registration was failing.
+
+- Relax checks of standby_cluster cluster configuration (Dmitry Dolgov, Alexander Kukushkin)
+
+  It was accepting only strings as valid values and therefore it was not possible to specify the port as integer and create_replica_methods as a list.
+
+Version 1.5.3
+-------------
+
+Compatibility and bugfix release.
+
+- Improve stability when running with python3 against zookeeper (Alexander Kukushkin)
+
+  Change of `loop_wait` was causing Patroni to disconnect from zookeeper and never reconnect back.
+
+- Fix broken compatibility with postgres 9.3 (Alexander)
+
+  When opening a replication connection we should specify replication=1, beacuse 9.3 does not understand replication='database'
+
+- Make sure we refresh Consul session at least once per HA loop and improve handling of consul sessions exceptions (Alexander)
+
+  Restart of local consul agent invalidates all sessions related to the node. Not calling session refresh on time and not doing proper handling of session errors was causing demote of the primary.
+
 Version 1.5.2
 -------------

@@ -177,9 +177,15 @@ standby nodes replicating from some remote master. This type of clusters has:

 Standby leader holds and updates a leader lock in DCS. If the leader lock
 expires, cascade replicas will perform an election to choose another leader
-from the standbys. For the sake of flexibility, you can specify different
-methods of creating a replica and recovery WAL records when a cluster is in the
-"standby mode", and after it was detached to function as a normal cluster.
+from the standbys.
+
+For the sake of flexibility, you can specify methods of creating a replica and
+recovery WAL records when a cluster is in the "standby mode" by providing
+`create_replica_methods` key in `standby_cluster` section. It is distinct from
+creating replicas, when cluster is detached and functions as a normal cluster,
+which is controlled by `create_replica_methods` in `postgresql` section. Both
+"standby" and "normal" `create_replica_methods` reference  keys in `postgresql`
+section.

 To configure such cluster you need to specify the section ``standby_cluster``
 in a patroni configuration:
@@ -192,6 +198,10 @@ in a patroni configuration:
                host: 1.2.3.4
                port: 5432
                primary_slot_name: patroni
+                create_replica_methods:
+                - basebackup

 Note, that these options will be applied only once during cluster bootstrap,
 and the only way to change them afterwards is through DCS.
+
+If you use replication slots on the standby cluster, you must also create the corresponding replication slot on the primary cluster.  It will not be done automatically by the standby cluster implementation.  You can use Patroni's permenant replication slots feature on the primary cluster to maintain a replication slot with the same name as ``primary_slot_name``, or its default value if ``primary_slot_name`` is not provided.
@@ -13,6 +13,8 @@ In asynchronous mode the cluster is allowed to lose some committed transactions

 The amount of transactions that can be lost is controlled via ``maximum_lag_on_failover`` parameter. Because the primary transaction log position is not sampled in real time, in reality the amount of lost data on failover is worst case bounded by  ``maximum_lag_on_failover`` bytes of transaction log plus the amount that is written in the last ``ttl`` seconds (``loop_wait``/2 seconds in the average case). However typical steady state replication delay is well under a second.

+By default, when running leader elections, Patroni does not take into account the current timeline of replicas, what in some cases could be undesirable behavior. You can prevent the node not having the same timeline as a former master become the new leader by changing the value of ``check_timeline`` parameter to ``true``.
+
 PostgreSQL synchronous replication
 ----------------------------------

@@ -12,6 +12,7 @@ import shutil
 import signal
 import six
 import subprocess
+import sys
 import tempfile
 import threading
 import time
@@ -84,7 +85,7 @@ class AbstractController(object):


 class PatroniController(AbstractController):
-    __PORT = 5440
+    __PORT = 5360
    PATRONI_CONFIG = '{}.yml'
    """ starts and stops individual patronis"""

@@ -117,12 +118,24 @@ class PatroniController(AbstractController):
        except IOError:
            return None

-    def add_tag_to_config(self, tag, value):
+    @staticmethod
+    def recursive_update(dst, src):
+        for k, v in src.items():
+            if k in dst and isinstance(dst[k], dict):
+                PatroniController.recursive_update(dst[k], v)
+            else:
+                dst[k] = v
+
+    def update_config(self, custom_config):
        with open(self._config) as r:
            config = yaml.safe_load(r)
-            config['tags']['tag'] = value
+            self.recursive_update(config, custom_config)
            with open(self._config, 'w') as w:
                yaml.safe_dump(config, w, default_flow_style=False)
+        self._scope = config.get('scope', 'batman')
+
+    def add_tag_to_config(self, tag, value):
+        self.update_config({'tags': {tag: value}})

    def _start(self):
        if self.watchdog:
@@ -130,7 +143,8 @@ class PatroniController(AbstractController):
        if isinstance(self._context.dcs_ctl, KubernetesController):
            self._context.dcs_ctl.create_pod(self._name[8:], self._scope)
            os.environ['PATRONI_KUBERNETES_POD_IP'] = '10.0.0.' + self._name[-1]
-        return subprocess.Popen(['coverage', 'run', '--source=patroni', '-p', 'patroni.py', self._config],
+        return subprocess.Popen([sys.executable, '-m', 'coverage', 'run',
+                                '--source=patroni', '-p', 'patroni.py', self._config],
                                stdout=self._log, stderr=subprocess.STDOUT, cwd=self._work_directory)

    def stop(self, kill=False, timeout=15, postgres=False):
@@ -174,13 +188,7 @@ class PatroniController(AbstractController):
                config['bootstrap']['initdb'].extend([{'auth': 'md5'}, {'auth-host': 'md5'}])

        if custom_config is not None:
-            def recursive_update(dst, src):
-                for k, v in src.items():
-                    if k in dst and isinstance(dst[k], dict):
-                        recursive_update(dst[k], v)
-                    else:
-                        dst[k] = v
-            recursive_update(config, custom_config)
+            self.recursive_update(config, custom_config)

        if config['postgresql'].get('callbacks', {}).get('on_role_change'):
            config['postgresql']['callbacks']['on_role_change'] += ' ' + str(self.__PORT)
@@ -366,6 +374,7 @@ class ConsulController(AbstractDcsController):
    def __init__(self, context):
        super(ConsulController, self).__init__(context)
        os.environ['PATRONI_CONSUL_HOST'] = 'localhost:8500'
+        os.environ['PATRONI_CONSUL_REGISTER_SERVICE'] = 'on'
        self._client = consul.Consul()
        self._config_file = None

@@ -802,8 +811,8 @@ def before_all(context):

 def after_all(context):
    context.dcs_ctl.stop()
-    subprocess.call(['coverage', 'combine'])
-    subprocess.call(['coverage', 'report'])
+    subprocess.call([sys.executable, '-m', 'coverage', 'combine'])
+    subprocess.call([sys.executable, '-m', 'coverage', 'report'])


 def before_feature(context, feature):
@@ -816,3 +825,5 @@ def after_feature(context, feature):
    context.pctl.stop_all()
    shutil.rmtree(os.path.join(context.pctl.patroni_path, 'data'))
    context.dcs_ctl.cleanup_service_tree()
+    if feature.status == 'failed':
+        shutil.copytree(context.pctl.output_dir, context.pctl.output_dir + '_failed')
@@ -43,7 +43,7 @@ Scenario: check dynamic configuration change via DCS
 	And I receive a response loop_wait 2
 	When I issue a GET request to http://127.0.0.1:8008/patroni
 	Then I receive a response code 200
-	And I receive a response tags {'tag': 'new_value'}
+	And I receive a response tags {'new_tag': 'new_value'}

 Scenario: check API requests for the primary-replica pair in the pause mode
 	Given I run patronictl.py pause batman
@@ -3,13 +3,15 @@ Feature: standby cluster
    Given I start postgres1
    Then postgres1 is a leader after 10 seconds
    And I sleep for 2 seconds
-    When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"loop_wait": 2, "slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
    Then I receive a response code 200
    And Response on GET http://127.0.0.1:8009/config contains slots after 10 seconds
+    And I sleep for 2 seconds
    When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"test_logical": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
    Then I receive a response code 200
    When I start postgres0 with callback configured
    Then "members/postgres0" key in DCS has state=running after 10 seconds
+    And replication works from postgres1 to postgres0 after 15 seconds
    When I shut down postgres1
    Then postgres0 is a leader after 10 seconds
    And I sleep for 2 seconds
@@ -20,8 +22,7 @@ Feature: standby cluster
  Scenario: check replication of a single table in a standby cluster
    Given I start postgres1 in a standby cluster batman1 as a clone of postgres0
    Then postgres1 is a leader of batman1 after 10 seconds
-    When I issue a PATCH request to http://127.0.0.1:8009/config with {"ttl": 20, "loop_wait": 2}
-    And I add the table foo to postgres0
+    When I add the table foo to postgres0
    Then table foo is present on postgres1 after 20 seconds
    When I start postgres2 in a cluster batman1
    Then postgres2 role is the replica after 24 seconds
@@ -5,6 +5,7 @@ import parse
 import requests
 import shlex
 import subprocess
+import sys
 import time
 import yaml

@@ -95,7 +96,7 @@ def do_request(context, request_method, url, data):

@step('I run {cmd}')
 def do_run(context, cmd):
-    cmd = ['coverage', 'run', '--source=patroni', '-p'] + shlex.split(cmd)
+    cmd = [sys.executable, '-m', 'coverage', 'run', '--source=patroni', '-p'] + shlex.split(cmd)
    try:
        # XXX: Dirty hack! We need to take name/passwd from the config!
        env = os.environ.copy()
@@ -30,12 +30,10 @@ def start_patroni(context, name, cluster_name):

@step('I start {name:w} in a standby cluster {cluster_name:w} as a clone of {name2:w}')
 def start_patroni_stanby_cluster(context, name, cluster_name, name2):
-    ctl = context.pctl._processes.pop(name, None)
    # we need to remove patroni.dynamic.json in order to "bootstrap" standby cluster with existing PGDATA
-    if ctl:
-        os.unlink(os.path.join(ctl._data_dir, 'patroni.dynamic.json'))
+    os.unlink(os.path.join(context.pctl._processes[name]._data_dir, 'patroni.dynamic.json'))
    port = context.pctl._processes[name2]._connkwargs.get('port')
-    return context.pctl.start(name, custom_config={
+    context.pctl._processes[name].update_config({
        "scope": cluster_name,
        "bootstrap": {
            "dcs": {
@@ -47,6 +45,7 @@ def start_patroni_stanby_cluster(context, name, cluster_name, name2):
            }
        }
    })
+    return context.pctl.start(name)


@step('{pg_name1:w} is replicating from {pg_name2:w} after {timeout:d} seconds')
@@ -1,4 +1,4 @@
-FROM postgres:10
+FROM postgres:11
 MAINTAINER Alexander Kukushkin <[email protected]>

 RUN export DEBIAN_FRONTEND=noninteractive \
@@ -7,23 +7,18 @@ RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get upgrade -y \
    && apt-cache depends patroni | sed -n -e 's/.* Depends: \(python3-.\+\)$/\1/p' \
            | grep -Ev '^python3-(sphinx|etcd|consul|kazoo|kubernetes)' \
-            | xargs apt-get install -y curl jq locales git python3-pip python3-wheel \
-
+            | xargs apt-get install -y vim-tiny curl jq locales git python3-pip python3-wheel \
    ## Make sure we have a en_US.UTF-8 locale available
    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
-
    && pip3 install setuptools \
    && pip3 install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
-
    && PGHOME=/home/postgres \
    && mkdir -p $PGHOME \
    && chown postgres $PGHOME \
    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
-
    # Set permissions for OpenShift
    && chmod 775 $PGHOME \
    && chmod 664 /etc/passwd \
-
    # Clean up
    && apt-get remove -y git python3-pip python3-wheel \
    && apt-get autoremove -y \
@@ -33,7 +28,7 @@ RUN export DEBIAN_FRONTEND=noninteractive \
 ADD entrypoint.sh /

 EXPOSE 5432 8008
-ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
+ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 EDITOR=/usr/bin/editor
 USER postgres
 WORKDIR /home/postgres
-CMD ["/bin/bash", "/entrypoint.sh"]
+CMD ["/bin/bash", "/entrypoint.sh"]
@@ -20,11 +20,11 @@ bootstrap:
  - data-checksums
  pg_hba:
  - host all all 0.0.0.0/0 md5
-  - host replication ${PATRONI_REPLICATION_USERNAME} ${POD_IP}/16    md5
+  - host replication ${PATRONI_REPLICATION_USERNAME} ${PATRONI_KUBERNETES_POD_IP}/16 md5
 restapi:
-  connect_address: '${POD_IP}:8008'
+  connect_address: '${PATRONI_KUBERNETES_POD_IP}:8008'
 postgresql:
-  connect_address: '${POD_IP}:5432'
+  connect_address: '${PATRONI_KUBERNETES_POD_IP}:5432'
  authentication:
    superuser:
      password: '${PATRONI_SUPERUSER_PASSWORD}'
@@ -108,7 +108,7 @@ objects:
      spec:
        containers:
        - env:
-          - name: POD_IP
+          - name: PATRONI_KUBERNETES_POD_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
@@ -108,7 +108,7 @@ objects:
      spec:
        containers:
        - env:
-          - name: POD_IP
+          - name: PATRONI_KUBERNETES_POD_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
@@ -1,3 +1,15 @@
+# headless service to avoid deletion of patronidemo-config endpoint
+apiVersion: v1
+kind: Service
+metadata:
+  name: patronidemo-config
+  labels:
+    application: patroni
+    cluster-name: patronidemo
+spec:
+  clusterIP: None
+
+---
 apiVersion: apps/v1beta1
 kind: StatefulSet
 metadata:
@@ -28,7 +40,7 @@ spec:
        - mountPath: /home/postgres/pgdata
          name: pgdata
        env:
-        - name: POD_IP
+        - name: PATRONI_KUBERNETES_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
@@ -173,6 +185,16 @@ rules:
  - patch
  - update
  - watch
+# The following privilege is only necessary for creation of headless service
+# for patronidemo-config endpoint, in order to prevent cleaning it up by the
+# k8s master. You can avoid giving this privilege by explicitly creating the
+# service like it is done in this manifest (lines 2..10)
+- apiGroups:
+  - ""
+  resources:
+  - services
+  verbs:
+  - create

 ---
 apiVersion: rbac.authorization.k8s.io/v1
@@ -14,6 +14,7 @@ class Patroni(object):
        from patroni.config import Config
        from patroni.dcs import get_dcs
        from patroni.ha import Ha
+        from patroni.log import PatroniLogger
        from patroni.postgresql import Postgresql
        from patroni.version import __version__
        from patroni.watchdog import Watchdog
@@ -21,7 +22,9 @@ class Patroni(object):
        self.setup_signal_handlers()

        self.version = __version__
+        self.logger = PatroniLogger()
        self.config = Config()
+        self.logger.reload_config(self.config.get('log', {}))
        self.dcs = get_dcs(self.config)
        self.watchdog = Watchdog(self.config)
        self.load_dynamic_configuration()
@@ -49,6 +52,7 @@ class Patroni(object):
                break
            except DCSError:
                logger.warning('Can not get cluster from dcs')
+                time.sleep(5)

    def get_tags(self):
        return {tag: value for tag, value in self.config.get('tags', {}).items()
@@ -65,6 +69,7 @@ class Patroni(object):
    def reload_config(self):
        try:
            self.tags = self.get_tags()
+            self.logger.reload_config(self.config.get('log', {}))
            self.dcs.reload_config(self.config)
            self.watchdog.reload_config(self.config)
            self.api.reload_config(self.config['restapi'])
@@ -138,12 +143,6 @@ class Patroni(object):


 def patroni_main():
-    logformat = os.environ.get('PATRONI_LOGFORMAT', '%(asctime)s %(levelname)s: %(message)s')
-    loglevel = os.environ.get('PATRONI_LOGLEVEL', 'INFO')
-    requests_loglevel = os.environ.get('PATRONI_REQUESTS_LOGLEVEL', 'WARNING')
-    logging.basicConfig(format=logformat, level=loglevel)
-    logging.getLogger('requests').setLevel(requests_loglevel)
-
    patroni = Patroni()
    try:
        patroni.run()
@@ -151,6 +150,7 @@ def patroni_main():
        pass
    finally:
        patroni.shutdown()
+        logging.shutdown()


 def pg_ctl_start(args):
@@ -3,6 +3,7 @@ import json
 import logging
 import psycopg2
 import time
+import traceback
 import dateutil.parser
 import datetime
 import os
@@ -547,3 +548,9 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
                and self.__initialize(config):
            self.start()
        self.__set_config_parameters(config)
+
+    @staticmethod
+    def handle_error(request, client_address):
+        address, port = client_address
+        logger.warning('Exception happened during processing of request from {}:{}'.format(address, port))
+        logger.warning(traceback.format_exc())
@@ -2,7 +2,6 @@ import json
 import logging
 import os
 import shutil
-import six
 import sys
 import tempfile
 import yaml
@@ -44,6 +43,7 @@ class Config(object):
    __DEFAULT_CONFIG = {
        'ttl': 30, 'loop_wait': 10, 'retry_timeout': 10,
        'maximum_lag_on_failover': 1048576,
+        'check_timeline': False,
        'master_start_timeout': 300,
        'synchronous_mode': False,
        'synchronous_mode_strict': False,
@@ -195,12 +195,9 @@ class Config(object):
                    elif name not in ('connect_address', 'listen', 'data_dir', 'pgpass', 'authentication'):
                        config['postgresql'][name] = deepcopy(value)
            elif name == 'standby_cluster':
-                allowed_keys = self.__DEFAULT_CONFIG['standby_cluster'].keys()
-                expected = {
-                    k: v for k, v in (value or {}).items()
-                    if (k in allowed_keys and isinstance(v, six.string_types))
-                }
-                config['standby_cluster'].update(expected)
+                for name, value in (value or {}).items():
+                    if name in self.__DEFAULT_CONFIG['standby_cluster']:
+                        config['standby_cluster'][name] = deepcopy(value)
            elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overriden from DCS
                if name in ('synchronous_mode', 'synchronous_mode_strict'):
                    config[name] = value
@@ -220,6 +217,15 @@ class Config(object):
            if value:
                ret[param] = value

+        def _fix_log_env(name, oldname):
+            value = _popenv(oldname)
+            name = Config.PATRONI_ENV_PREFIX + 'LOG_' + name.upper()
+            if value and name not in os.environ:
+                os.environ[name] = value
+
+        for name, oldname in (('level', 'loglevel'), ('format', 'logformat'), ('dateformat', 'log_datefmt')):
+            _fix_log_env(name, oldname)
+
        def _set_section_values(section, params):
            for param in params:
                value = _popenv(section + '_' + param)
@@ -228,6 +234,22 @@ class Config(object):

        _set_section_values('restapi', ['listen', 'connect_address', 'certfile', 'keyfile'])
        _set_section_values('postgresql', ['listen', 'connect_address', 'data_dir', 'pgpass', 'bin_dir'])
+        _set_section_values('log', ['level', 'format', 'dateformat', 'dir', 'file_size', 'file_num', 'loggers'])
+
+        def _parse_dict(value):
+            if not value.strip().startswith('{'):
+                value = '{{{0}}}'.format(value)
+            try:
+                return yaml.safe_load(value)
+            except Exception:
+                logger.exception('Exception when parsing dict %s', value)
+                return None
+
+        value = ret.get('log', {}).pop('loggers', None)
+        if value:
+            value = _parse_dict(value)
+            if value:
+                ret['log']['loggers'] = value

        def _get_auth(name):
            ret = {}
@@ -250,8 +272,6 @@ class Config(object):
        if authentication:
            ret['postgresql']['authentication'] = authentication

-        users = {}
-
        def _parse_list(value):
            if not (value.strip().startswith('-') or '[' in value):
                value = '[{0}]'.format(value)
@@ -263,37 +283,40 @@ class Config(object):

        for param in list(os.environ.keys()):
            if param.startswith(Config.PATRONI_ENV_PREFIX):
+                # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
                name, suffix = (param[8:].split('_', 1) + [''])[:2]
-                if name and suffix:
-                    # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
-                    if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT',
-                                  'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'NAMESPACE', 'CONTEXT',
-                                  'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS'):
-                        value = os.environ.pop(param)
-                        if suffix == 'PORT':
-                            value = value and parse_int(value)
-                        elif suffix in ('HOSTS', 'PORTS', 'CHECKS'):
-                            value = value and _parse_list(value)
-                        elif suffix == 'LABELS':
-                            if not value.strip().startswith('{'):
-                                value = '{{{0}}}'.format(value)
-                            try:
-                                value = yaml.safe_load(value)
-                            except Exception:
-                                logger.exception('Exception when parsing dict %s', value)
-                                value = None
-                        if value:
-                            ret[name.lower()][suffix.lower()] = value
-                    # PATRONI_<username>_PASSWORD=<password>, PATRONI_<username>_OPTIONS=<option1,option2,...>
-                    # CREATE USER "<username>" WITH <OPTIONS> PASSWORD '<password>'
-                    elif suffix == 'PASSWORD':
-                        password = os.environ.pop(param)
-                        if password:
-                            users[name] = {'password': password}
-                            options = os.environ.pop(param[:-9] + '_OPTIONS', None)
-                            options = options and _parse_list(options)
-                            if options:
-                                users[name]['options'] = options
+                if suffix in ('HOST', 'HOSTS', 'PORT', 'PROTOCOL', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY',
+                              'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL',
+                              'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP',
+                              'PORTS', 'LABELS') and name:
+                    value = os.environ.pop(param)
+                    if suffix == 'PORT':
+                        value = value and parse_int(value)
+                    elif suffix in ('HOSTS', 'PORTS', 'CHECKS'):
+                        value = value and _parse_list(value)
+                    elif suffix == 'LABELS':
+                        value = _parse_dict(value)
+                    elif suffix == 'REGISTER_SERVICE':
+                        value = parse_bool(value)
+                    if value:
+                        ret[name.lower()][suffix.lower()] = value
+        if 'etcd' in ret:
+            ret['etcd'].update(_get_auth('etcd'))
+
+        users = {}
+        for param in list(os.environ.keys()):
+            if param.startswith(Config.PATRONI_ENV_PREFIX):
+                name, suffix = (param[8:].rsplit('_', 1) + [''])[:2]
+                # PATRONI_<username>_PASSWORD=<password>, PATRONI_<username>_OPTIONS=<option1,option2,...>
+                # CREATE USER "<username>" WITH <OPTIONS> PASSWORD '<password>'
+                if name and suffix == 'PASSWORD':
+                    password = os.environ.pop(param)
+                    if password:
+                        users[name] = {'password': password}
+                        options = os.environ.pop(param[:-9] + '_OPTIONS', None)
+                        options = options and _parse_list(options)
+                        if options:
+                            users[name]['options'] = options
        if users:
            ret['bootstrap']['users'] = users

@@ -340,7 +363,7 @@ class Config(object):
            'scope',
            'retry_timeout',
            'synchronous_mode',
-            'maximum_lag_on_failover'
+            'synchronous_mode_strict',
        )

        pg_config.update({p: config[p] for p in updated_fields if p in config})
@@ -180,7 +180,7 @@ def print_output(columns, rows=None, alignment=None, fmt='pretty', header=True,

    if fmt == 'tsv':
        if columns is not None and header:
-            click.echo(delimiter.join(columns) + '\n')
+            click.echo(delimiter.join(columns))

        for r in rows:
            c = [str(c) for c in r]
@@ -716,6 +716,10 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
    has_scheduled_restarts = any(m.data.get('scheduled_restart') for m in cluster.members)
    has_pending_restarts = any(m.data.get('pending_restart') for m in cluster.members)

+    # Show Host as 'host:port' if somebody is running on non-standard port or two nodes are running on the same host
+    append_port = any(str(m.conn_kwargs()['port']) != '5432' for m in cluster.members) or\
+        len(set(m.conn_kwargs()['host'] for m in cluster.members)) < len(cluster.members)
+
    for m in cluster.members:
        logging.debug(m)

@@ -732,7 +736,11 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
        elif xlog_location_cluster >= xlog_location:
            lag = round((xlog_location_cluster - xlog_location)/1024/1024)

-        row = [name, m.name, m.conn_kwargs()['host'], role, m.data.get('state', ''), lag]
+        host = m.conn_kwargs()['host']
+        if append_port:
+            host += ':{0}'.format(m.conn_kwargs()['port'])
+
+        row = [name, m.name, host, role, m.data.get('state', ''), m.data.get('timeline', ''), lag]

        if extended or has_pending_restarts:
            row.append('*' if m.data.get('pending_restart') else '')
@@ -749,8 +757,8 @@ def output_members(cluster, name, extended=False, fmt='pretty'):

        rows.append(row)

-    columns = ['Cluster', 'Member', 'Host', 'Role', 'State', 'Lag in MB']
-    alignment = {'Lag in MB': 'r'}
+    columns = ['Cluster', 'Member', 'Host', 'Role', 'State', 'TL', 'Lag in MB']
+    alignment = {'Lag in MB': 'r', 'TL': 'r'}

    if extended or has_pending_restarts:
        columns.append('Pending restart')
@@ -368,7 +368,7 @@ class SyncState(namedtuple('SyncState', 'index,leader,sync_standby')):
        return name is not None and name in (self.leader, self.sync_standby)


-class TimelineHistory(namedtuple('TimelineHistory', 'index,lines')):
+class TimelineHistory(namedtuple('TimelineHistory', 'index,value,lines')):
    """Object representing timeline history file"""

    @staticmethod
@@ -384,7 +384,7 @@ class TimelineHistory(namedtuple('TimelineHistory', 'index,lines')):
            lines = None
        if not isinstance(lines, list):
            lines = []
-        return TimelineHistory(index, lines)
+        return TimelineHistory(index, value, lines)


 class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operation,members,failover,sync,history')):
@@ -484,6 +484,26 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
        slots = self.get_replication_slots(name, 'master').values()
        return any(v for v in slots if v.get("type") == "logical")

+    @property
+    def timeline(self):
+        """
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, 0).timeline
+        0
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[]')).timeline
+        1
+        >>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[["a"]]')).timeline
+        0
+        """
+        if self.history:
+            if self.history.lines:
+                try:
+                    return int(self.history.lines[-1][0]) + 1
+                except Exception:
+                    logger.error('Failed to parse cluster history from DCS: %s', self.history.lines)
+            elif self.history.value == '[]':
+                return 1
+        return 0
+

@six.add_metaclass(abc.ABCMeta)
 class AbstractDCS(object):
@@ -27,10 +27,14 @@ class ConsulInternalError(ConsulException):
    """An internal Consul server error occurred"""


-class InvalidSessionTTL(ConsulInternalError):
+class InvalidSessionTTL(ConsulException):
    """Session TTL is too small or too big"""


+class InvalidSession(ConsulException):
+    """invalid session"""
+
+
 class HTTPClient(object):

    def __init__(self, host='127.0.0.1', port=8500, token=None, scheme='http', verify=True, cert=None, ca_cert=None):
@@ -72,6 +76,8 @@ class HTTPClient(object):
            msg = '{0} {1}'.format(response.status, data)
            if data.startswith('Invalid Session TTL'):
                raise InvalidSessionTTL(msg)
+            elif data.startswith('invalid session'):
+                raise InvalidSession(msg)
            else:
                raise ConsulInternalError(msg)
        return base.Response(response.status, response.headers, data)
@@ -96,7 +102,12 @@ class HTTPClient(object):
                params = {k: v for k, v in params}
            kwargs = {'retries': 0, 'preload_content': False, 'body': data}
            if method == 'get' and isinstance(params, dict) and 'index' in params:
-                kwargs['timeout'] = (float(params['wait'][:-1]) if 'wait' in params else 300) + 1
+                timeout = float(params['wait'][:-1]) if 'wait' in params else 300
+                # According to the documentation a small random amount of additional wait time is added to the
+                # supplied maximum wait time to spread out the wake up time of any concurrent requests. This adds
+                # up to wait / 16 additional time to the maximum duration. Since our goal is actually getting a
+                # response rather read timeout we will add to the timeout a sligtly bigger value.
+                kwargs['timeout'] = timeout + max(timeout/15.0, 1)
            else:
                kwargs['timeout'] = self._read_timeout
            token = params.pop('token', self.token) if isinstance(params, dict) else self.token
@@ -357,6 +368,9 @@ class Consul(AbstractDCS):
            if self._register_service:
                self.update_service(not create_member and member and member.data or {}, data)
            return True
+        except InvalidSession:
+            self._session = None
+            logger.error('Our session disappeared from Consul, can not "touch_member"')
        except Exception:
            logger.exception('touch_member')
        return False
@@ -380,7 +394,8 @@ class Consul(AbstractDCS):
        api_parts = urlparse(data['api_url'])
        api_parts = api_parts._replace(path='/{0}'.format(role))
        conn_parts = urlparse(data['conn_url'])
-        check = base.Check.http(api_parts.geturl(), self._service_check_interval, deregister=self._client.http.ttl * 10)
+        check = base.Check.http(api_parts.geturl(), self._service_check_interval,
+                                deregister='{0}s'.format(self._client.http.ttl * 10))
        params = {
            'service_id': '{0}/{1}'.format(self._scope, self._name),
            'address': conn_parts.hostname,
@@ -414,14 +429,21 @@ class Consul(AbstractDCS):
            return self._update_service(new_data)

    @catch_consul_errors
-    def _do_attempt_to_acquire_leader(self, kwargs):
-        return self.retry(self._client.kv.put, self.leader_path, self._name, **kwargs)
+    def _do_attempt_to_acquire_leader(self, permanent):
+        try:
+            kwargs = {} if permanent else {'acquire': self._session}
+            return self.retry(self._client.kv.put, self.leader_path, self._name, **kwargs)
+        except InvalidSession:
+            self._session = None
+            logger.error('Our session disappeared from Consul. Will try to get a new one and retry attempt')
+            self.refresh_session()
+            return self.retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)

    def attempt_to_acquire_leader(self, permanent=False):
        if not self._session and not permanent:
            self.refresh_session()

-        ret = self._do_attempt_to_acquire_leader({} if permanent else {'acquire': self._session})
+        ret = self._do_attempt_to_acquire_leader(permanent)
        if not ret:
            logger.info('Could not take out TTL lock')

@@ -499,4 +521,5 @@ class Consul(AbstractDCS):
        try:
            return super(Consul, self).watch(None, timeout)
        finally:
+            self._last_session_refresh = 0
            self.event.clear()
@@ -92,13 +92,14 @@ class Kubernetes(AbstractDCS):
        self.__subsets = None
        use_endpoints = config.get('use_endpoints') and (config.get('patronictl') or 'pod_ip' in config)
        if use_endpoints:
-            addresses = [k8s_client.V1EndpointAddress(ip=config['pod_ip'])]
+            addresses = [k8s_client.V1EndpointAddress(ip='127.0.0.1' if config.get('patronictl') else config['pod_ip'])]
            ports = []
            for p in config.get('ports', [{}]):
                port = {'port': int(p.get('port', '5432'))}
                port.update({n: p[n] for n in ('name', 'protocol') if p.get(n)})
                ports.append(k8s_client.V1EndpointPort(**port))
            self.__subsets = [k8s_client.V1EndpointSubset(addresses=addresses, ports=ports)]
+            self._should_create_config_service = True
        self._api = CoreV1ApiProxy(use_endpoints)
        self.set_retry_timeout(config['retry_timeout'])
        self.set_ttl(config.get('ttl') or 30)
@@ -274,6 +275,24 @@ class Kubernetes(AbstractDCS):
            body = k8s_client.V1ConfigMap(metadata=metadata)
        return self.retry(func, self._namespace, body) if retry else func(self._namespace, body)

+    def patch_or_create_config(self, annotations, resource_version=None, patch=False, retry=True):
+        # SCOPE-config endpoint requires corresponding service otherwise it might be "cleaned" by k8s master
+        if self.__subsets and not patch and not resource_version:
+            self._should_create_config_service = True
+            self._create_config_service()
+        return self.patch_or_create(self.config_path, annotations, resource_version, patch, retry)
+
+    def _create_config_service(self):
+        metadata = k8s_client.V1ObjectMeta(namespace=self._namespace, name=self.config_path, labels=self._labels)
+        body = k8s_client.V1Service(metadata=metadata, spec=k8s_client.V1ServiceSpec(cluster_ip='None'))
+        try:
+            if not self._api.create_namespaced_service(self._namespace, body):
+                return
+        except Exception as e:
+            if not isinstance(e, k8s_client.rest.ApiException) or e.status != 409:  # Service already exists
+                return logger.exception('create_config_service failed')
+        self._should_create_config_service = False
+
    def _write_leader_optime(self, last_operation):
        """Unused"""

@@ -288,9 +307,7 @@ class Kubernetes(AbstractDCS):
        if last_operation:
            annotations[self._OPTIME] = last_operation

-        subsets = self.__subsets
-        if subsets is not None and access_is_restricted:
-            subsets = []
+        subsets = [] if access_is_restricted else self.__subsets

        ret = self.patch_or_create(self.leader_path, annotations, self._leader_resource_version, subsets=subsets)
        if ret:
@@ -333,13 +350,13 @@ class Kubernetes(AbstractDCS):

    def set_config_value(self, value, index=None):
        patch = bool(index or self.cluster and self.cluster.config and self.cluster.config.index)
-        return self.patch_or_create(self.config_path, {self._CONFIG: value}, index, patch, False)
+        return self.patch_or_create_config({self._CONFIG: value}, index, patch, False)

    @catch_kubernetes_errors
    def touch_member(self, data, ttl=None, permanent=False):
        cluster = self.cluster
        if cluster and cluster.leader and cluster.leader.name == self._name:
-            role = 'master'
+            role = 'promoted' if data['role'] in ('replica', 'promoted') else 'master'
        elif data['state'] == 'running' and data['role'] != 'master':
            role = data['role']
        else:
@@ -354,12 +371,14 @@ class Kubernetes(AbstractDCS):
                        'annotations': {'status': json.dumps(data, separators=(',', ':'))}}
            body = k8s_client.V1Pod(metadata=k8s_client.V1ObjectMeta(**metadata))
            ret = self._api.patch_namespaced_pod(self._name, self._namespace, body)
+        if self.__subsets and self._should_create_config_service:
+            self._create_config_service()
        return ret

    def initialize(self, create_new=True, sysid=""):
        cluster = self.cluster
        resource_version = cluster.config.index if cluster and cluster.config and cluster.config.index else None
-        return self.patch_or_create(self.config_path, {self._INITIALIZE: sysid}, resource_version)
+        return self.patch_or_create_config({self._INITIALIZE: sysid}, resource_version)

    def delete_leader(self):
        if self.cluster and isinstance(self.cluster.leader, Leader) and self.cluster.leader.name == self._name:
@@ -367,7 +386,7 @@ class Kubernetes(AbstractDCS):
            self.reset_cluster()

    def cancel_initialization(self):
-        self.patch_or_create(self.config_path, {self._INITIALIZE: None}, self.cluster.config.index, True)
+        self.patch_or_create_config({self._INITIALIZE: None}, self.cluster.config.index, True)

    @catch_kubernetes_errors
    def delete_cluster(self):
@@ -375,7 +394,7 @@ class Kubernetes(AbstractDCS):

    def set_history_value(self, value):
        patch = bool(self.cluster and self.cluster.config and self.cluster.config.index)
-        return self.patch_or_create(self.config_path, {self._HISTORY: value}, None, patch, False)
+        return self.patch_or_create_config({self._HISTORY: value}, None, patch, False)

    def set_sync_state_value(self, value, index=None):
        """Unused"""
@@ -1,5 +1,6 @@
 import json
 import logging
+import select
 import time

 from kazoo.client import KazooClient, KazooState, KazooRetry
@@ -45,6 +46,13 @@ class PatroniSequentialThreadingHandler(SequentialThreadingHandler):
            args[1] = max(self._connect_timeout, args[1]/10.0)
        return super(PatroniSequentialThreadingHandler, self).create_connection(*args, **kwargs)

+    def select(self, *args, **kwargs):
+        """Python3 raises `ValueError` if socket is closed, because fd == -1"""
+        try:
+            return super(PatroniSequentialThreadingHandler, self).select(*args, **kwargs)
+        except ValueError as e:
+            raise select.error(9, str(e))
+

 class ZooKeeper(AbstractDCS):

@@ -20,25 +20,28 @@ from threading import RLock
 logger = logging.getLogger(__name__)


-class _MemberStatus(namedtuple('_MemberStatus', 'member,reachable,in_recovery,wal_position,tags,watchdog_failed')):
+class _MemberStatus(namedtuple('_MemberStatus', ['member', 'reachable', 'in_recovery', 'timeline',
+                                                 'wal_position', 'tags', 'watchdog_failed'])):
    """Node status distilled from API response:

        member - dcs.Member object of the node
        reachable - `!False` if the node is not reachable or is not responding with correct JSON
        in_recovery - `!True` if pg_is_in_recovery() == true
-        wal_position - value of `replayed_location` or `location` from JSON, dependin on its role.
+        timeline - timeline value from JSON
+        wal_position - maximum value of `replayed_location` or `received_location` from JSON
        tags - dictionary with values of different tags (i.e. nofailover)
        watchdog_failed - indicates that watchdog is required by configuration but not available or failed
    """
    @classmethod
    def from_api_response(cls, member, json):
        is_master = json['role'] == 'master'
+        timeline = json.get('timeline', 0)
        wal = not is_master and max(json['xlog'].get('received_location', 0), json['xlog'].get('replayed_location', 0))
-        return cls(member, True, not is_master, wal, json.get('tags', {}), json.get('watchdog_failed', False))
+        return cls(member, True, not is_master, timeline, wal, json.get('tags', {}), json.get('watchdog_failed', False))

    @classmethod
    def unknown(cls, member):
-        return cls(member, False, None, 0, {}, False)
+        return cls(member, False, None, 0, 0, {}, False)

    def failover_limitation(self):
        """Returns reason why this node can't promote or None if everything is ok."""
@@ -92,6 +95,9 @@ class Ha(object):
    def is_paused(self):
        return self.check_mode('pause')

+    def check_timeline(self):
+        return self.check_mode('check_timeline')
+
    def get_standby_cluster_config(self):
        if self.cluster and self.cluster.config and self.cluster.config.modify_index:
            config = self.cluster.config.data
@@ -259,13 +265,22 @@ class Ha(object):

        return result

-    def _handle_rewind(self):
+    def _handle_rewind_or_reinitialize(self):
        leader = self.get_remote_master() if self.is_standby_cluster() else self.cluster.leader
-        if self.state_handler.rewind_needed_and_possible(leader):
+        if not self.state_handler.rewind_or_reinitialize_needed_and_possible(leader):
+            return None
+
+        if self.state_handler.can_rewind:
            self._async_executor.schedule('running pg_rewind from ' + leader.name)
            self._async_executor.run_async(self.state_handler.rewind, (leader,))
            return True

+        # remove_data_directory_on_diverged_timelines is set
+        if not self.is_standby_cluster():
+            self._async_executor.schedule('reinitializing due to diverged timelines')
+            self._async_executor.run_async(self._do_reinitialize, args=(self.cluster, ))
+            return True
+
    def recover(self):
        # Postgres is not running and we will restart in standby mode. Watchdog is not needed until we promote.
        self.watchdog.disable()
@@ -298,7 +313,7 @@ class Ha(object):
        if self.is_standby_cluster() or not self.has_lock():
            if not self.state_handler.rewind_executed:
                self.state_handler.trigger_check_diverged_lsn()
-            if self._handle_rewind():
+            if self._handle_rewind_or_reinitialize():
                return self._async_executor.scheduled_action

            if self.has_lock():  # in standby cluster
@@ -332,9 +347,7 @@ class Ha(object):
        else:
            node_to_follow = cluster.leader

-        return (node_to_follow if
-                node_to_follow and
-                node_to_follow.name != self.state_handler.name else None)
+        return node_to_follow if node_to_follow and node_to_follow.name != self.state_handler.name else None

    def follow(self, demote_reason, follow_reason, refresh=True):
        if refresh:
@@ -345,7 +358,8 @@ class Ha(object):
        node_to_follow = self._get_node_to_follow(self.cluster)

        if self.is_paused():
-            if not (self.state_handler.need_rewind and self.state_handler.can_rewind) or self.cluster.is_unlocked():
+            if not (self.state_handler.need_rewind and self.state_handler.can_rewind_or_reinitialize_allowed)\
+                    or self.cluster.is_unlocked():
                self.state_handler.set_role('master' if is_leader else 'replica')
                if is_leader:
                    return 'continue to run as master without lock'
@@ -355,7 +369,7 @@ class Ha(object):
            self.demote('immediate-nolock')
            return demote_reason

-        if self._handle_rewind():
+        if self._handle_rewind_or_reinitialize():
            return self._async_executor.scheduled_action

        if not self.state_handler.check_recovery_conf(node_to_follow):
@@ -543,15 +557,23 @@ class Ha(object):
        :returns True when node is lagging
        """
        lag = (self.cluster.last_leader_operation or 0) - wal_position
-        return lag > self.state_handler.config.get('maximum_lag_on_failover', 0)
+        return lag > self.patroni.config.get('maximum_lag_on_failover', 0)

    def _is_healthiest_node(self, members, check_replication_lag=True):
        """This method tries to determine whether I am healthy enough to became a new leader candidate or not."""

        _, my_wal_position = self.state_handler.timeline_wal_position()
        if check_replication_lag and self.is_lagging(my_wal_position):
+            logger.info('My wal position exceeds maximum replication lag')
            return False  # Too far behind last reported wal position on master

+        if not self.is_standby_cluster() and self.check_timeline():
+            cluster_timeline = self.cluster.timeline
+            my_timeline = self.state_handler.replica_cached_timeline(cluster_timeline)
+            if my_timeline < cluster_timeline:
+                logger.info('My timeline %s is behind last known cluster timeline %s', my_timeline, cluster_timeline)
+                return False
+
        # Prepare list of nodes to run check against
        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]

@@ -562,11 +584,13 @@ class Ha(object):
                        logger.warning('Master (%s) is still alive', st.member.name)
                        return False
                    if my_wal_position < st.wal_position:
+                        logger.info('Wal position of %s is ahead of my wal position', st.member.name)
                        return False
        return True

    def is_failover_possible(self, members):
        ret = False
+        cluster_timeline = self.cluster.timeline
        members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
        if members:
            for st in self.fetch_nodes_statuses(members):
@@ -575,6 +599,9 @@ class Ha(object):
                    logger.info('Member %s is %s', st.member.name, not_allowed_reason)
                elif self.is_lagging(st.wal_position):
                    logger.info('Member %s exceeds maximum replication lag', st.member.name)
+                elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline):
+                    logger.info('Timeline %s of member %s is behind the cluster timeline %s',
+                                st.timeline, st.member.name, cluster_timeline)
                else:
                    ret = True
        else:
@@ -714,7 +741,7 @@ class Ha(object):
        else:
            if self.is_synchronous_mode():
                self.state_handler.set_synchronous_standby(None)
-            if self.state_handler.rewind_needed_and_possible(leader):
+            if self.state_handler.rewind_or_reinitialize_needed_and_possible(leader):
                return False  # do not start postgres, but run pg_rewind on the next iteration
            self.state_handler.follow(node_to_follow)

@@ -1234,8 +1261,8 @@ class Ha(object):
                    # the demote code follows through to starting Postgres right away, however, in the rewind case
                    # it returns from demote and reaches this point to start PostgreSQL again after rewind. In that
                    # case it makes no sense to continue to recover() unless rewind has finished successfully.
-                    elif (self.state_handler.rewind_failed or
-                          not (self.state_handler.need_rewind and self.state_handler.can_rewind)):
+                    elif self.state_handler.rewind_failed or not self.state_handler.need_rewind \
+                            or not self.state_handler.can_rewind_or_reinitialize_allowed:
                        return 'postgres is not running'

                # try to start dead postgres
@@ -1323,16 +1350,11 @@ class Ha(object):

        if cluster_params:
            unique_name = 'remote_master:{}'.format(uuid.uuid1())
-            data = {
-                'conn_kwargs': {
-                    "host": cluster_params.get('host'),
-                    "port": cluster_params.get('port'),
-                },
-                'no_replication_slot': 'primary_slot_name' not in cluster_params,
-            }
-            data.update({
-                k: v for k, v in cluster_params.items()
-                if k in RemoteMember.allowed_keys()
-            })
+
+            data = {k: v for k, v in cluster_params.items() if k in RemoteMember.allowed_keys()}
+            data['no_replication_slot'] = 'primary_slot_name' not in cluster_params
+            conn_kwargs = {k: cluster_params[k] for k in ('host', 'port') if k in cluster_params}
+            if conn_kwargs:
+                data['conn_kwargs'] = conn_kwargs

            return RemoteMember(unique_name, data)
@@ -0,0 +1,66 @@
+import logging
+import os
+
+from copy import deepcopy
+from logging.handlers import RotatingFileHandler
+from patroni.utils import deep_compare
+
+
+class PatroniLogger(object):
+
+    DEFAULT_LEVEL = 'INFO'
+    DEFAULT_FORMAT = '%(asctime)s %(levelname)s: %(message)s'
+
+    def __init__(self):
+        self.root_logger = logging.getLogger()
+        self.config = None
+        self.handler = None
+        self.reload_config({'level': 'DEBUG'})
+
+    def update_loggers(self):
+        loggers = deepcopy(self.config.get('loggers') or {})
+        for name, logger in self.root_logger.manager.loggerDict.items():
+            if not isinstance(logger, logging.PlaceHolder):
+                level = loggers.pop(name, logging.NOTSET)
+                logger.setLevel(level)
+
+        for name, level in loggers.items():
+            logger = self.root_logger.manager.getLogger(name)
+            logger.setLevel(level)
+
+    def reload_config(self, config):
+        if self.config is None or not deep_compare(self.config, config):
+            self.root_logger.setLevel(config.get('level', PatroniLogger.DEFAULT_LEVEL))
+
+            add_handler = None
+            if 'dir' in config:
+                if not isinstance(self.handler, RotatingFileHandler):
+                    add_handler = RotatingFileHandler(os.path.join(config['dir'], __name__))
+                handler = add_handler or self.handler
+                handler.maxBytes = int(config.get('file_size', 25000000))
+                handler.backupCount = int(config.get('file_num', 4))
+            else:
+                if self.handler is None or isinstance(self.handler, RotatingFileHandler):
+                    add_handler = logging.StreamHandler()
+                handler = add_handler or self.handler
+
+            oldlogformat = (self.config or {}).get('format', PatroniLogger.DEFAULT_FORMAT)
+            logformat = config.get('format', PatroniLogger.DEFAULT_FORMAT)
+
+            olddateformat = (self.config or {}).get('dateformat') or None
+            dateformat = config.get('dateformat') or None  # Convert empty string to `None`
+
+            if oldlogformat != logformat or olddateformat != dateformat or add_handler:
+                handler.setFormatter(logging.Formatter(logformat, dateformat))
+
+            if add_handler:
+                self.root_logger.addHandler(add_handler)
+
+                if self.handler is not None:
+                    self.root_logger.removeHandler(self.handler)
+                    self.handler.close()
+
+                self.handler = add_handler
+
+            self.config = config.copy()
+            self.update_loggers()
@@ -5,6 +5,7 @@ import re
 import shlex
 import shutil
 import socket
+import stat
 import subprocess
 import tempfile
 import time
@@ -15,7 +16,7 @@ from patroni.callback_executor import CallbackExecutor
 from patroni.exceptions import PostgresConnectionException, PostgresException
 from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop, split_host_port
 from patroni.postmaster import PostmasterProcess
-from patroni.dcs import slot_name_from_member_name, RemoteMember
+from patroni.dcs import slot_name_from_member_name, RemoteMember, Leader
 from requests.structures import CaseInsensitiveDict
 from six import string_types
 from six.moves.urllib.parse import quote_plus
@@ -391,7 +392,7 @@ class Postgresql(object):
            we have either wal_log_hints or checksums turned on
        """
        # low-hanging fruit: check if pg_rewind configuration is there
-        if not (self.config.get('use_pg_rewind') and all(self._superuser.get(n) for n in ('username', 'password'))):
+        if not self.config.get('use_pg_rewind'):
            return False

        cmd = [self._pgcommand('pg_rewind'), '--help']
@@ -403,6 +404,10 @@ class Postgresql(object):
            return False
        return self.configuration_allows_rewind(self.controldata())

+    @property
+    def can_rewind_or_reinitialize_allowed(self):
+        return self.config.get('remove_data_directory_on_diverged_timelines') or self.can_rewind
+
    @property
    def sysid(self):
        if not self._sysid and not self.bootstrapping:
@@ -1110,8 +1115,12 @@ class Postgresql(object):
            f.write(self._CONFIG_WARNING_HEADER)
            f.write("include '{0}'\n\n".format(self.config.get('custom_conf') or self._postgresql_base_conf_name))
            for name, value in sorted((configuration or self._server_parameters).items()):
-                if not self._running_custom_bootstrap or name != 'hba_file':
+                if not self._running_custom_bootstrap or name not in ('hba_file', 'archive_mode'):
                    f.write("{0} = '{1}'\n".format(name, value))
+            # we want to set archive_mode to 'off' during the custom bootstrap
+            # in order to avoid premature archiving of wals and history files
+            if self._running_custom_bootstrap:
+                f.write("archive_mode = 'off'\n")
            # when we are doing custom bootstrap we assume that we don't know superuser password
            # and in order to be able to change it, we are opening trust access from a certain address
            # therefore we need to make sure that hba_file is not overriden
@@ -1154,8 +1163,10 @@ class Postgresql(object):
            with open(self._pg_hba_conf, 'w') as f:
                f.write(self._CONFIG_WARNING_HEADER)
                for address, t in addresses.items():
-                    f.write('{0}\t{1}\t{2}\t{3}\ttrust\n'.format(t, 'all',
-                                                                 self._superuser.get('username') or 'all', address))
+                    f.write((
+                        '{0}\treplication\t{1}\t{3}\ttrust\n'
+                        '{0}\tall\t{2}\t{3}\ttrust\n'
+                    ).format(t, self._replication['username'], self._superuser.get('username') or 'all', address))
        elif not self._server_parameters.get('hba_file') and self.config.get('pg_hba'):
            with open(self._pg_hba_conf, 'w') as f:
                f.write(self._CONFIG_WARNING_HEADER)
@@ -1186,6 +1197,7 @@ class Postgresql(object):

    def write_recovery_conf(self, recovery_params):
        with open(self._recovery_conf, 'w') as f:
+            os.chmod(self._recovery_conf, stat.S_IWRITE | stat.S_IREAD)
            for name, value in recovery_params.items():
                f.write("{0} = '{1}'\n".format(name, value))

@@ -1242,8 +1254,7 @@ class Postgresql(object):

    @contextmanager
    def _get_replication_connection_cursor(self, host='localhost', port=5432, database=None, **kwargs):
-        database = database or self._database
-        with self._get_connection_cursor(host=host, port=int(port), database=database, replication='database',
+        with self._get_connection_cursor(host=host, port=int(port), database=database or self._database, replication=1,
                                         user=self._replication['username'], password=self._replication['password'],
                                         connect_timeout=3, options='-c statement_timeout=2000') as cur:
            yield cur
@@ -1314,7 +1325,11 @@ class Postgresql(object):
        if local_timeline is None or local_lsn is None:
            return

-        if not self.check_leader_is_not_in_recovery(**leader.conn_kwargs(self._superuser)):
+        if isinstance(leader, Leader):
+            if leader.member.data.get('role') != 'master':
+                return
+        # standby cluster
+        elif not self.check_leader_is_not_in_recovery(**leader.conn_kwargs(self._superuser)):
            return

        history = need_rewind = None
@@ -1394,19 +1409,21 @@ class Postgresql(object):
        else:
            logger.error('Failed to rewind from healty master: %s', leader.name)

-            if self.config.get('remove_data_directory_on_rewind_failure', False):
-                logger.warning('remove_data_directory_on_rewind_failure is set. removing...')
-                self.remove_data_directory()
-                self._rewind_state = REWIND_STATUS.INITIAL
+            for name in ('remove_data_directory_on_rewind_failure', 'remove_data_directory_on_diverged_timelines'):
+                if self.config.get(name):
+                    logger.warning('%s is set. removing...', name)
+                    self.remove_data_directory()
+                    self._rewind_state = REWIND_STATUS.INITIAL
+                    break
            else:
                self._rewind_state = REWIND_STATUS.FAILED
        return False

    def trigger_check_diverged_lsn(self):
-        if self.can_rewind and self._rewind_state != REWIND_STATUS.NEED:
+        if self.can_rewind_or_reinitialize_allowed and self._rewind_state != REWIND_STATUS.NEED:
            self._rewind_state = REWIND_STATUS.CHECK

-    def rewind_needed_and_possible(self, leader):
+    def rewind_or_reinitialize_needed_and_possible(self, leader):
        if leader and leader.name != self.name and leader.conn_url and self._rewind_state == REWIND_STATUS.CHECK:
            self._check_timeline_and_lsn(leader)
        return leader and leader.conn_url and self._rewind_state == REWIND_STATUS.NEED
@@ -1643,6 +1660,7 @@ $$""".format(name, ' '.join(options)), name, password, password)
               base backup)
        """

+        self._rewind_state = REWIND_STATUS.INITIAL
        ret = self.create_replica(clone_member) == 0
        if ret:
            self._post_restore()
@@ -1664,7 +1682,8 @@ $$""".format(name, ' '.join(options)), name, password, password)

    def post_bootstrap(self, config, task):
        try:
-            self.create_or_update_role(self._superuser['username'], self._superuser['password'], ['SUPERUSER'])
+            if 'username' in self._superuser and 'password' in self._superuser:
+                self.create_or_update_role(self._superuser['username'], self._superuser['password'], ['SUPERUSER'])

            task.complete(self.run_bootstrap_post_init(config))
            if task.result:
@@ -1683,17 +1702,11 @@ $$""".format(name, ' '.join(options)), name, password, password)
                        os.unlink(self._pg_hba_conf)
                        self.restore_configuration_files()
                    self._write_postgresql_conf()
-                    if self._server_parameters.get('hba_file') and \
-                            self._server_parameters['hba_file'] != self._pg_hba_conf:
-                        self.restart()
-                    else:
-                        self._replace_pg_hba()
-                        if self.pending_restart:
-                            self.restart()
-                        else:
-                            self.reload()
-                            time.sleep(1)  # give a time to postgres to "reload" configuration files
-                            self.close_connection()  # close connection to reconnect with a new password
+                    self._replace_pg_hba()
+                    # at this point there should be no recovery.conf
+                    if os.path.isfile(self._recovery_conf) or os.path.islink(self._recovery_conf):
+                        os.unlink(self._recovery_conf)
+                    self.restart()
        except Exception:
            logger.exception('post_bootstrap')
            task.complete(False)
@@ -102,28 +102,31 @@ class PostmasterProcess(psutil.Process):
        return None

    def wait_for_user_backends_to_close(self):
-        # These regexps are cross checked against versions PostgreSQL 9.1 .. 9.6
-        aux_proc_re = re.compile("(?:postgres:)( .*:)? (?:""(?:startup|logger|checkpointer|writer|wal writer|"
-                                 "autovacuum launcher|autovacuum worker|stats collector|wal receiver|archiver|"
-                                 "wal sender) process|bgworker: )")
+        # These regexps are cross checked against versions PostgreSQL 9.1 .. 11
+        aux_proc_re = re.compile("(?:postgres:)( .*:)? (?:(?:archiver|startup|autovacuum launcher|autovacuum worker|"
+                                 "checkpointer|logger|stats collector|wal receiver|wal writer|writer)(?: process  )?|"
+                                 "walreceiver|wal sender process|walsender|walwriter|background writer|"
+                                 "logical replication launcher|logical replication worker for|bgworker:) ")

        try:
-            user_backends = []
-            user_backends_cmdlines = []
-            for child in self.children():
-                try:
-                    cmdline = child.cmdline()[0]
-                    if not aux_proc_re.match(cmdline):
-                        user_backends.append(child)
-                        user_backends_cmdlines.append(cmdline)
-                except psutil.NoSuchProcess:
-                    pass
-            if user_backends:
-                logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines))
-                psutil.wait_procs(user_backends)
-            logger.debug("Backends closed")
+            children = self.children()
        except psutil.Error:
-            logger.exception('wait_for_user_backends_to_close')
+            return logger.debug('Failed to get list of postmaster children')
+
+        user_backends = []
+        user_backends_cmdlines = []
+        for child in children:
+            try:
+                cmdline = child.cmdline()[0]
+                if not aux_proc_re.match(cmdline):
+                    user_backends.append(child)
+                    user_backends_cmdlines.append(cmdline)
+            except psutil.NoSuchProcess:
+                pass
+        if user_backends:
+            logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines))
+            psutil.wait_procs(user_backends)
+        logger.debug("Backends closed")

    @staticmethod
    def start(pgcommand, data_dir, conf, options):
@@ -157,7 +160,7 @@ class PostmasterProcess(psutil.Process):
        cmdline = [pgcommand, '-D', data_dir, '--config-file={}'.format(conf)] + options
        logger.debug("Starting postgres: %s", " ".join(cmdline))
        proc = call_self(['pg_ctl_start'] + cmdline, close_fds=(os.name != 'nt'),
-                         stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env)
+                         stdout=subprocess.PIPE, env=env)
        pid = int(proc.stdout.readline().strip())
        proc.wait()
        logger.info('postmaster pid=%s', pid)
@@ -1 +1 @@
-__version__ = '1.5.2'
+__version__ = '1.5.5'
@@ -11,6 +11,6 @@ click>=4.1
 prettytable>=0.7
 tzlocal
 python-dateutil
-psutil
+psutil>=2.0.0
 cdiff
 kubernetes>=2.0.0,<=7.0.0,!=4.0.*,!=5.0.*
@@ -73,7 +73,7 @@ class MockHa(object):

    @staticmethod
    def fetch_nodes_statuses(members):
-        return [_MemberStatus(None, True, None, None, {}, False)]
+        return [_MemberStatus(None, True, None, 0, None, {}, False)]

    @staticmethod
    def schedule_future_restart(data):
@@ -401,3 +401,9 @@ class TestRestApiServer(unittest.TestCase):
        self.assertRaises(ValueError, srv.reload_config, bad_config)
        self.assertRaises(ValueError, srv.reload_config, {})
        srv.reload_config({'listen': '127.0.0.2:8008'})
+
+    def test_handle_error(self):
+        try:
+            raise Exception()
+        except Exception:
+            self.assertIsNone(MockRestApiServer.handle_error(None, ('127.0.0.1', 55555)))
@@ -1,6 +1,6 @@
 import os
-import unittest
 import sys
+import unittest

 from mock import MagicMock, Mock, patch
 from patroni.config import Config
@@ -30,6 +30,8 @@ class TestConfig(unittest.TestCase):
            'PATRONI_NAME': 'postgres0',
            'PATRONI_NAMESPACE': '/patroni/',
            'PATRONI_SCOPE': 'batman2',
+            'PATRONI_LOGLEVEL': 'ERROR',
+            'PATRONI_LOG_LOGGERS': 'patroni.postmaster: WARNING, urllib3: DEBUG',
            'PATRONI_RESTAPI_USERNAME': 'username',
            'PATRONI_RESTAPI_PASSWORD': 'password',
            'PATRONI_RESTAPI_LISTEN': '0.0.0.0:8008',
@@ -49,6 +51,7 @@ class TestConfig(unittest.TestCase):
            'PATRONI_ETCD_CERT': '/cert',
            'PATRONI_ETCD_KEY': '/key',
            'PATRONI_CONSUL_HOST': '127.0.0.1:8500',
+            'PATRONI_CONSUL_REGISTER_SERVICE': 'on',
            'PATRONI_KUBERNETES_LABELS': 'a:b:c',
            'PATRONI_KUBERNETES_SCOPE_LABEL': 'a',
            'PATRONI_KUBERNETES_PORTS': '[{"name": "postgresql"}]',
@@ -84,3 +87,15 @@ class TestConfig(unittest.TestCase):
            self.config.save_cache()
        with patch('os.fdopen', MagicMock()):
            self.config.save_cache()
+
+    def test_standby_cluster_parameters(self):
+        dynamic_configuration = {
+            'standby_cluster': {
+                'create_replica_methods': ['wal_e', 'basebackup'],
+                'host': 'localhost',
+                'port': 5432
+            }
+        }
+        self.config.set_dynamic_configuration(dynamic_configuration)
+        for name, value in dynamic_configuration['standby_cluster'].items():
+            self.assertEqual(self.config['standby_cluster'][name], value)
@@ -4,7 +4,7 @@ import unittest
 from consul import ConsulException, NotFound
 from mock import Mock, patch
 from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, \
-                                ConsulError, HTTPClient, InvalidSessionTTL
+                                ConsulError, HTTPClient, InvalidSessionTTL, InvalidSession
 from test_etcd import SleepException


@@ -52,6 +52,8 @@ class TestHTTPClient(unittest.TestCase):
        self.assertRaises(ConsulInternalError, self.client.get, Mock(), '')
        self.client.http.request.return_value.data = b"Invalid Session TTL '3000000000', must be between [10s=24h0m0s]"
        self.assertRaises(InvalidSessionTTL, self.client.get, Mock(), '')
+        self.client.http.request.return_value.data = b"invalid session '16492f43-c2d6-5307-432f-e32d6f7bcbd0'"
+        self.assertRaises(InvalidSession, self.client.get, Mock(), '')

    def test_unknown_method(self):
        try:
@@ -110,19 +112,18 @@ class TestConsul(unittest.TestCase):
        self.c._session = 'fd4f44fe-2cac-bba5-a60b-304b51ff39b8'
        self.assertIsInstance(self.c.get_cluster(), Cluster)

-    @patch.object(consul.Consul.KV, 'delete', Mock(side_effect=[ConsulException, True, True]))
-    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=[True, ConsulException]))
+    @patch.object(consul.Consul.KV, 'delete', Mock(side_effect=[ConsulException, True, True, True]))
+    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=[True, ConsulException, InvalidSession]))
    def test_touch_member(self):
-        self.c._register_service = True
-        self.c.refresh_session = Mock(return_value=True)
-        self.c.touch_member({'balbla': 'blabla'})
-        self.c.touch_member({'balbla': 'blabla'})
-        self.c.touch_member({'balbla': 'blabla'})
        self.c.refresh_session = Mock(return_value=False)
        self.c.touch_member({'conn_url': 'postgres://replicator:[email protected]:5433/postgres',
                             'api_url': 'http://127.0.0.1:8009/patroni'})
+        self.c._register_service = True
+        self.c.refresh_session = Mock(return_value=True)
+        for _ in range(0, 4):
+            self.c.touch_member({'balbla': 'blabla'})

-    @patch.object(consul.Consul.KV, 'put', Mock(return_value=False))
+    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=InvalidSession))
    def test_take_leader(self):
        self.c.set_ttl(20)
        self.c.refresh_session = Mock()
@@ -28,8 +28,10 @@ def false(*args, **kwargs):


 def get_cluster(initialize, leader, members, failover, sync, cluster_config=None):
-    history = TimelineHistory(1, [(1, 67197376, 'no recovery target specified', datetime.datetime.now().isoformat())])
-    cluster_config = cluster_config or ClusterConfig(1, {1: 2}, 1)
+    t = datetime.datetime.now().isoformat()
+    history = TimelineHistory(1, '[[1,67197376,"no recovery target specified","' + t + '"]]',
+                              [(1, 67197376, 'no recovery target specified', t)])
+    cluster_config = cluster_config or ClusterConfig(1, {'check_timeline': True}, 1)
    return Cluster(initialize, cluster_config, leader, 10, members, failover, sync, history)


@@ -72,12 +74,13 @@ def get_standby_cluster_initialized_with_only_leader(failover=None, sync=None):
    )


-def get_node_status(reachable=True, in_recovery=True, wal_position=10, nofailover=False, watchdog_failed=False):
+def get_node_status(reachable=True, in_recovery=True, timeline=2,
+                    wal_position=10, nofailover=False, watchdog_failed=False):
    def fetch_node_status(e):
        tags = {}
        if nofailover:
            tags['nofailover'] = True
-        return _MemberStatus(e, reachable, in_recovery, wal_position, tags, watchdog_failed)
+        return _MemberStatus(e, reachable, in_recovery, timeline, wal_position, tags, watchdog_failed)
    return fetch_node_status


@@ -115,6 +118,7 @@ zookeeper:
        sys.argv = sys.argv[:1]

        self.config = Config()
+        self.config.set_dynamic_configuration({'maximum_lag_on_failover': 5})
        self.postgresql = p
        self.dcs = d
        self.api = Mock()
@@ -147,6 +151,7 @@ def run_async(self, func, args=()):
@patch.object(Postgresql, 'query', Mock())
@patch.object(Postgresql, 'checkpoint', Mock())
@patch.object(Postgresql, 'cancellable_subprocess_call', Mock(return_value=0))
+@patch.object(Postgresql, '_get_local_timeline_lsn_from_replication_connection', Mock(return_value=[2, 10]))
@patch.object(etcd.Client, 'write', etcd_write)
@patch.object(etcd.Client, 'read', etcd_read)
@patch.object(etcd.Client, 'delete', Mock(side_effect=etcd.EtcdException))
@@ -166,7 +171,6 @@ class TestHa(unittest.TestCase):
            mock_machines.__get__ = Mock(return_value=['http://remotehost:2379'])
            self.p = Postgresql({'name': 'postgresql0', 'scope': 'dummy', 'listen': '127.0.0.1:5432',
                                 'data_dir': 'data/postgresql0', 'retry_timeout': 10,
-                                 'maximum_lag_on_failover': 5,
                                 'authentication': {'superuser': {'username': 'foo', 'password': 'bar'},
                                                    'replication': {'username': '', 'password': ''}},
                                 'parameters': {'wal_level': 'hot_standby', 'max_replication_slots': 5, 'foo': 'bar',
@@ -237,12 +241,20 @@ class TestHa(unittest.TestCase):
        self.p.controldata = lambda: {'Database cluster state': 'in production', 'Database system identifier': SYSID}
        self.assertEqual(self.ha.run_cycle(), 'doing crash recovery in a single user mode')

-    @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
    def test_recover_with_rewind(self):
        self.p.is_running = false
        self.ha.cluster = get_cluster_initialized_with_leader()
        self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')

+    @patch.object(Postgresql, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'create_replica', Mock(return_value=1))
+    def test_recover_with_reinitialize(self):
+        self.p.is_running = false
+        self.ha.cluster = get_cluster_initialized_with_leader()
+        self.assertEqual(self.ha.run_cycle(), 'reinitializing due to diverged timelines')
+
    @patch('sys.exit', return_value=1)
    @patch('patroni.ha.Ha.sysid_valid', MagicMock(return_value=True))
    def test_sysid_no_match(self, exit_mock):
@@ -341,7 +353,8 @@ class TestHa(unittest.TestCase):
        self.p.is_leader = false
        self.assertEqual(self.ha.run_cycle(), 'PAUSE: no action')

-    @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
    def test_follow_triggers_rewind(self):
        self.p.is_leader = false
        self.p.trigger_check_diverged_lsn()
@@ -455,12 +468,14 @@ class TestHa(unittest.TestCase):
        f = Failover(0, self.p.name, '', None)
        self.ha.cluster = get_cluster_initialized_with_leader(f)
        self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
-        self.p.rewind_needed_and_possible = true
+        self.p.rewind_or_reinitialize_needed_and_possible = true
        self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
        self.ha.fetch_node_status = get_node_status(nofailover=True)
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        self.ha.fetch_node_status = get_node_status(watchdog_failed=True)
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
+        self.ha.fetch_node_status = get_node_status(timeline=1)
+        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        self.ha.fetch_node_status = get_node_status(wal_position=1)
        self.assertEqual(self.ha.run_cycle(), 'no action.  i am the leader with the lock')
        # manual failover from the previous leader to us won't happen if we hold the nofailover flag
@@ -572,6 +587,8 @@ class TestHa(unittest.TestCase):
        self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        with patch('patroni.postgresql.Postgresql.timeline_wal_position', return_value=(1, 1)):
            self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
+        with patch('patroni.postgresql.Postgresql.replica_cached_timeline', return_value=1):
+            self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        self.ha.patroni.nofailover = True
        self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
        self.ha.patroni.nofailover = False
@@ -673,7 +690,8 @@ class TestHa(unittest.TestCase):
        msg = 'promoted self to a standby leader because i had the session lock'
        self.assertEqual(self.ha.run_cycle(), msg)

-    @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
+    @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
    def test_process_unhealthy_standby_cluster_as_cascade_replica(self):
        self.p.is_leader = false
        self.p.name = 'replica'
@@ -72,7 +72,7 @@ class TestKubernetes(unittest.TestCase):

    @patch.object(k8s_client.CoreV1Api, 'patch_namespaced_pod', Mock(return_value=True))
    def test_touch_member(self):
-        self.k.touch_member({})
+        self.k.touch_member({'role': 'replica'})
        self.k._name = 'p-1'
        self.k.touch_member({'state': 'running', 'role': 'replica'})
        self.k.touch_member({'state': 'stopped', 'role': 'master'})
@@ -112,3 +112,15 @@ class TestKubernetes(unittest.TestCase):

    def test_set_history_value(self):
        self.k.set_history_value('{}')
+
+    @patch('kubernetes.config.load_kube_config', Mock())
+    @patch.object(k8s_client.CoreV1Api, 'patch_namespaced_pod', Mock(return_value=True))
+    @patch.object(k8s_client.CoreV1Api, 'create_namespaced_endpoints', Mock())
+    @patch.object(k8s_client.CoreV1Api, 'create_namespaced_service',
+                  Mock(side_effect=[True, False, k8s_client.rest.ApiException(500, '')]))
+    def test__create_config_service(self):
+        k = Kubernetes({'ttl': 30, 'scope': 'test', 'name': 'p-0', 'retry_timeout': 10,
+                        'labels': {'f': 'b'}, 'use_endpoints': True, 'pod_ip': '10.0.0.0'})
+        self.assertIsNotNone(k.patch_or_create_config({'foo': 'bar'}))
+        self.assertIsNotNone(k.patch_or_create_config({'foo': 'bar'}))
+        k.touch_member({'state': 'running', 'role': 'replica'})
@@ -0,0 +1,36 @@
+import os
+import sys
+import unittest
+import yaml
+
+from mock import Mock, patch
+from patroni.config import Config
+from patroni.log import PatroniLogger
+
+
+class TestPatroniLogger(unittest.TestCase):
+
+    @patch('logging.FileHandler._open', Mock())
+    def test_patroni_logger(self):
+        config = {
+            'log': {
+                'dir': 'foo',
+                'file_size': 4096,
+                'file_num': 5,
+                'loggers': {
+                    'foo.bar': 'INFO'
+                }
+            },
+            'restapi': {}, 'postgresql': {'data_dir': 'foo'}
+        }
+        sys.argv = ['patroni.py']
+        os.environ[Config.PATRONI_CONFIG_VARIABLE] = yaml.dump(config, default_flow_style=False)
+        logger = PatroniLogger()
+        patroni_config = Config()
+        logger.reload_config(patroni_config['log'])
+
+        self.assertEqual(logger.handler.maxBytes, config['log']['file_size'])
+        self.assertEqual(logger.handler.backupCount, config['log']['file_num'])
+
+        config['log'].pop('dir')
+        logger.reload_config(config['log'])
@@ -345,10 +345,10 @@ class TestPostgresql(unittest.TestCase):
                          Mock(return_value={'Database cluster state': 'shut down in recovery',
                                             'Minimum recovery ending location': '0/0',
                                             "Min recovery ending loc's timeline": '0'})):
-            self.p.rewind_needed_and_possible(self.leader)
+            self.p.rewind_or_reinitialize_needed_and_possible(self.leader)
        with patch.object(Postgresql, 'is_running', Mock(return_value=True)):
            with patch.object(MockCursor, 'fetchone', Mock(side_effect=[(False, ), Exception])):
-                self.p.rewind_needed_and_possible(self.leader)
+                self.p.rewind_or_reinitialize_needed_and_possible(self.leader)

    @patch.object(Postgresql, 'start', Mock())
    @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
@@ -357,21 +357,23 @@ class TestPostgresql(unittest.TestCase):
    def test__check_timeline_and_lsn(self, mock_check_leader_is_not_in_recovery):
        mock_check_leader_is_not_in_recovery.return_value = False
        self.p.trigger_check_diverged_lsn()
-        self.assertFalse(self.p.rewind_needed_and_possible(self.leader))
+        self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
+        self.leader = self.leader.member
+        self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
        mock_check_leader_is_not_in_recovery.return_value = True
-        self.assertFalse(self.p.rewind_needed_and_possible(self.leader))
+        self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
        self.p.trigger_check_diverged_lsn()
        with patch('psycopg2.connect', Mock(side_effect=Exception)):
-            self.assertFalse(self.p.rewind_needed_and_possible(self.leader))
+            self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
        self.p.trigger_check_diverged_lsn()
        with patch.object(MockCursor, 'fetchone', Mock(side_effect=[('', 2, '0/0'), ('', b'3\t0/40159C0\tn\n')])):
-            self.assertFalse(self.p.rewind_needed_and_possible(self.leader))
+            self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
        self.p.trigger_check_diverged_lsn()
        with patch.object(MockCursor, 'fetchone', Mock(return_value=('', 1, '0/0'))):
            with patch.object(Postgresql, '_get_local_timeline_lsn', Mock(return_value=(1, '0/0'))):
-                self.assertFalse(self.p.rewind_needed_and_possible(self.leader))
+                self.assertFalse(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))
            self.p.trigger_check_diverged_lsn()
-            self.assertTrue(self.p.rewind_needed_and_possible(self.leader))
+            self.assertTrue(self.p.rewind_or_reinitialize_needed_and_possible(self.leader))

    @patch.object(MockCursor, 'fetchone', Mock(side_effect=[(True,), Exception]))
    def test_check_leader_is_not_in_recovery(self):
@@ -651,6 +653,7 @@ class TestPostgresql(unittest.TestCase):

    @patch('time.sleep', Mock())
    @patch('os.unlink', Mock())
+    @patch('os.path.isfile', Mock(return_value=True))
    @patch.object(Postgresql, 'run_bootstrap_post_init', Mock(return_value=True))
    @patch.object(Postgresql, '_custom_bootstrap', Mock(return_value=True))
    @patch.object(Postgresql, 'start', Mock(return_value=True))
@@ -48,7 +48,7 @@ class TestPostmasterProcess(unittest.TestCase):
        mock_init.side_effect = psutil.NoSuchProcess(123)
        self.assertEqual(PostmasterProcess.from_pid(123), None)
        mock_init.side_effect = None
-        self.assertNotEquals(PostmasterProcess.from_pid(123), None)
+        self.assertNotEqual(PostmasterProcess.from_pid(123), None)

    @patch('psutil.Process.__init__', Mock())
    @patch('psutil.Process.send_signal')
@@ -67,7 +67,7 @@ class TestPostmasterProcess(unittest.TestCase):
    @patch('psutil.wait_procs')
    def test_wait_for_user_backends_to_close(self, mock_wait):
        c1 = Mock()
-        c1.cmdline = Mock(return_value=["postgres: startup process"])
+        c1.cmdline = Mock(return_value=["postgres: startup process   "])
        c2 = Mock()
        c2.cmdline = Mock(return_value=["postgres: postgres postgres [local] idle"])
        c3 = Mock()
@@ -77,8 +77,7 @@ class TestPostmasterProcess(unittest.TestCase):
            self.assertIsNone(proc.wait_for_user_backends_to_close())
            mock_wait.assert_called_with([c2])

-        c3.cmdline = Mock(side_effect=psutil.AccessDenied(123))
-        with patch('psutil.Process.children', Mock(return_value=[c3])):
+        with patch('psutil.Process.children', Mock(side_effect=psutil.NoSuchProcess(123))):
            proc = PostmasterProcess(123)
            self.assertIsNone(proc.wait_for_user_backends_to_close())

@@ -1,3 +1,4 @@
+import select
 import six
 import unittest

@@ -115,6 +116,10 @@ class TestPatroniSequentialThreadingHandler(unittest.TestCase):
        self.assertIsNotNone(self.handler.create_connection((), 40))
        self.assertIsNotNone(self.handler.create_connection(timeout=40))

+    @patch.object(SequentialThreadingHandler, 'select', Mock(side_effect=ValueError))
+    def test_select(self):
+        self.assertRaises(select.error, self.handler.select)
+

 class TestZooKeeper(unittest.TestCase):
Author	SHA1	Message	Date
Alexander KukushkinandGitHub	c6e70a9910	Release 1.5.5 (#979 ) * Bump version * Update release notes	2019-02-15 16:14:39 +01:00
Alexander KukushkinandGitHub	0ec1760397	Don't write primary_conninfo into recovery.conf for wal only standby cluster (#971 ) It is useless and makes postgres to generate a lot of errors	2019-02-15 13:35:34 +01:00
Alexander KukushkinandGitHub	317721991b	Fix handling of PATRONI_*_PASSWORD environment variables (#970 ) Bug was introduced in https://github.com/zalando/patroni/pull/947 and https://github.com/zalando/patroni/pull/500	2019-02-15 13:35:22 +01:00
Alexander KukushkinandGitHub	0c516de147	Create headless service associated with $SCOPE-config endpoint (#958 ) if there is no service defined k8s assumes that endpoint is orphaned and removes it. Patroni tries to create the service only in case if use_endpoints is enabled if the following cases: 1. Upon start 2. When it tries to (re-)create the config endpoint If for some reason creation of the service has failed, Patroni will retry it on every cycle of HA loop. Usually it fails due to lack of permissions and if you don't want to give such permissions to the service account used by Patroni, you can create the service explicitly in the deployment manifest.	2019-02-15 13:35:04 +01:00
Michael BanckandAlexander Kukushkin	073074f83e	Run coverage as python -m coverage (#968 ) Depending on the platform the coverage binary might not always be available under the standard name.	2019-02-13 16:02:12 +01:00
Michael BanckandAlexander Kukushkin	345e6d3131	Copy away output directories of failed acceptance tests. (#967 ) And dump logs on travis from only failed features	2019-02-13 16:00:15 +01:00
Michael BanckandAlexander Kukushkin	d01a9bdcd5	Change base port for acceptance tests from 5440 to 5360 (#966 )	2019-02-13 15:59:13 +01:00
Alexander KukushkinandGitHub	10bbf0c3c5	Always use replication=1, otherwise it is considered a logical walsender (#952 ) Fixes: https://github.com/zalando/patroni/issues/894 Fixes: https://github.com/zalando/patroni/issues/951	2019-01-30 12:38:51 +01:00
Alexander KukushkinandGitHub	4304560ce2	Adjust read timeout for leader watch blocking query (#950 ) According to the Consul documentation the actual response timeout is increased by a small random amount of additional wait time added to the supplied maximum wait time to spread out the wake up time of any concurrent requests. It adds up to wait / 16 additional time to the maximum duration. In our case we will add wait/15 or 1 second depending on what is bigger. Fixes: https://github.com/zalando/patroni/issues/945	2019-01-30 12:38:24 +01:00
Alexander KukushkinandGitHub	254ee2acfc	Show information about timelines in patronictl list (#949 ) This information will help to detect stale replicas. In addition to that Host will include ':{port}' if the port value isn't default or more than one member running on the same host. Fixes: https://github.com/zalando/patroni/issues/942	2019-01-30 12:37:51 +01:00
Alexander KukushkinandGitHub	739329b590	Make it possible to automatically reinit the former master (#948 ) If the pg_rewind is disabled or can't be used, the former master could fail to start as a new replica due to diverged timelines. In this case, the only way to fix it is wiping the data directory and reinitializing. So far Patroni was able to remove the data directory only after failed attempt to run pg_rewind. This commit fixes it. If the `postgresql.remove_data_directory_on_diverged_timelines` is set, Patroni will wipe the data directory and reinitialize the former master automatically. Fixes: https://github.com/zalando/patroni/issues/941	2019-01-30 12:37:21 +01:00
Étienne MandAlexander Kukushkin	bd2c54581a	Add ETCD_(PROTOCOL\|USERNAME\|PASSWORD) env variables (#947 ) Fix #944	2019-01-30 12:36:50 +01:00
Maxim IvanovandAlexander Kukushkin	f0b12b7e2e	Document create_replicas_methods in standby_cluster section (#939 ) Fixes https://github.com/zalando/patroni/issues/935	2019-01-30 12:36:24 +01:00
Étienne MandAlexander Kukushkin	93d157dea3	Document how to start Patroni with an existing data directory (#918 )	2019-01-30 12:35:57 +01:00
Alexander KukushkinandGitHub	2c128520cf	Python34 compatibility (#933 ) and some other minor fixes. Closes https://github.com/zalando/patroni/issues/932	2019-01-16 14:40:05 +01:00
Alexander KukushkinandGitHub	381a5b80d2	Release 1.5.4 (#931 ) * Bump version * Update release notes * Make it possible to configure registration of Service in Consul via env variables	2019-01-15 12:14:19 +01:00
Alexander KukushkinandGitHub	71dae6a905	Optionally consider node not healthy if it is not on the latest timeline (#892 ) The latest timeline is calculated from the `/history` key in DCS. In case there is no such key or it contains some garbage we consider the node healthy. Closes https://github.com/zalando/patroni/issues/890	2019-01-15 11:16:30 +01:00
Alexander KukushkinandGitHub	cf34fb3934	Relax requirements on superuser credentials (#930 ) libpq allows opening connection without explicitly specifying neither username nor password. Depending on situation it would rely either on `pgpass` file or trust authentication method in pg_hba.conf. Since pg_rewind is also using libpq, it could work the same way. Fixes https://github.com/zalando/patroni/issues/928	2019-01-15 11:15:35 +01:00
Alexander KukushkinandGitHub	e080ded44b	Make logging configurable via YAML file (#927 ) It allows changing logging settings in runtime by updating config and doing reload or sending `SIGHUP` to the Patroni process. Important! Environment configuration names related to logging were renamed and documentation accordingly updated. For compatibility reasons Patroni still accepts `PATRONI_LOGLEVEL` and `PATRONI_FORMAT`, but some other variables related to logging, which were introduced only recently (between releases), will stop working. I think it is ok, since we didn't release the new version yet and therefore it is very unlikely that somebody is using them except authors of corresponding PRs. Example of log section in the config file: ```yaml log: dir: /where/to/write/patroni/logs # if not specified, write logs to stderr file_size: 50000000 # 50MB file_num: 10 # keep history of 10 files dateformat: '%Y-%m-%d %H:%M:%S' loggers: # increase log verbosity for etcd.client and urllib3 etcd.client: DEBUG urllib3: DEBUG ```	2019-01-15 08:42:13 +01:00
jouirandAlexander Kukushkin	dec3656f6e	Redirect HTTPServer exceptions to logger (#900 ) (#925 ) By default they were written to stdout	2019-01-15 08:37:06 +01:00
Alexander KukushkinandGitHub	1e2d89fa58	Apply 5 second backoff when loading global config up on start (#922 ) It doesn't make much sense to hammer DCS when we just starting up. Fixes https://github.com/zalando/patroni/issues/919	2019-01-14 14:55:56 +01:00
Alexander KukushkinandGitHub	994863c18d	Refactor wait_for_user_backends_to_close method (#917 ) 1. Log only debug level messages on any kind of error 2. Update regexp for matching postgres aux processes to make it compatible with postgres 11 Fixes https://github.com/zalando/patroni/issues/914	2019-01-14 14:55:45 +01:00
Alexander KukushkinandGitHub	3fce982909	Set archive_mode to off during the custom bootstrap (#911 ) We want to avoid archiving WALs and history files until the cluster is fully functional. It should really help if the custom bootstrap involves pg_upgrade.	2019-01-14 14:55:23 +01:00
Lucas CapistrantandAlexander Kukushkin	d306092cbc	Explicitly secure rw perms for recovery.conf at creation time (#910 ) We don't want anybody except patroni/postgres user reading this file, it contains replication user and password.	2019-01-14 14:22:04 +01:00
Dmitry DolgovandAlexander Kukushkin	11f7ceb521	Do not check types of standby_cluster configuration (#924 ) Simply allow valid keys	2019-01-14 14:16:15 +01:00
bradnicholsonandAlexander Kukushkin	05a13839aa	Update replica_bootstrap.rst (#915 ) Add some docs about replication slots for standby clusters	2019-01-14 12:57:21 +01:00
Étienne MandAlexander Kukushkin	04ac199fc8	Single quotes are mandatory around each host in PATRONI_ETCD_HOSTS (#926 ) Otherwise YAML parser fails	2019-01-14 11:56:15 +01:00
anikin-aaandAlexander Kukushkin	0be8a9527b	possibility to set `logdatefmt` via env. (#904 ) The value could be configured via `PATRONI_LOGDATEFMT` environment variable. The default value is `%Y-%m-%d %H:%M:%S`	2019-01-14 08:46:38 +01:00
Pavel KirillovandAlexander Kukushkin	929ff08bfd	Service deregister timeout must be in Go time format (#893 )	2018-12-21 15:42:15 +01:00
Cody CoonsandAlexander Kukushkin	7bc8d0aac9	Removed stderr pipe to stdout on pg_ctl process (#896 ) Inheriting stderr from the main Patroni process allows all Postgres logs to be seen along with all patroni logs. This is very useful in a container environment as Patroni and Postgres logs may be consumed using standard tools (docker logs, kubectl, etc). In addition to that, this change fixes a bug with Patroni not being able to catch postmaster pid when postgres writing some warnings into stderr	2018-12-21 15:41:34 +01:00
Lucas CapistrantandAlexander Kukushkin	f3da6de129	Add ability to configure app logs to be written to a file (#903 ) It gives users the option to send Patroni application logs to a File instead of Standard Out. There are three environment variables that can be set to enable and configure file logging. 1. `PATRONI_FILE_LOG_DIR`: Path to a directory that is writeable by the executing user. Having this variable set is what activates file logging. 2. `PATRONI_FILE_LOG_NUM`: This is a rolling file logger. This variable dictates how many log files are retained. 3. `PATRONI_FILE_LOG_SIZE`: This variable dictates the size at which the logs will roll. If `PATRONI_FILE_LOG_DIR` is not set than Patroni will log to stderr (default behavior does not change) Closes https://github.com/zalando/patroni/issues/902	2018-12-21 15:38:29 +01:00
Alexander KukushkinandGitHub	491f230711	Release 1.5.3 (#889 ) * Bump version * Update release notes	2018-12-03 17:12:53 +01:00
Alexander KukushkinandGitHub	f1d7ccf36e	Make sure we refresh session at least once per HA loop (#880 ) Fixes https://github.com/zalando/patroni/issues/879	2018-12-03 16:35:14 +01:00
Kostiantyn NemchenkoandAlexander Kukushkin	96ea01bee4	Fix kubernetes demo files (#885 ) - Update postgres docker image to the latest 11 version. - Remove empty lines inside the `RUN` command to make the Dockerfile compatible with future docker versions. - Set the `PATRONI_KUBERNETES_POD_IP` environment variable, which is required when _use_endpoints_ is enabled. Otherwise, the `KeyError` is raised [here](https://github.com/zalando/patroni/blob/master/patroni/dcs/kubernetes.py#L95). - Set `EDITOR` environment variable to make configuration changes via `patronictl edit-config`.	2018-12-03 15:46:25 +01:00
Alexander KukushkinandGitHub	e684ca66e5	Compatibility with postgres 9.3 (#882 ) use replication=1 instead of replication='database' when opening replication connection, 9.3 doesn't understand it	2018-11-30 15:46:05 +01:00
Alexander KukushkinandGitHub	1a0876e5ca	Refactor acceptance tests to improve stability (#884 ) Hope it will crash less often when executed on travis against k8s	2018-11-30 12:40:56 +01:00
Alexander KukushkinandGitHub	9bf074acfb	Compatibility with python3 (#883 ) Change of `loop_wait` was causing Patroni to disconnect from zookeeper and never reconnect back. The error was happening only with python3 due to a difference in implementation of `select.select` function.	2018-11-30 11:40:34 +01:00