Release 2.0.1 (#1722)

* Bump version
* Update release notes

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,48 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Environment**
+- Patroni version:
+- PostgreSQL version:
+- DCS (and its version):
+
+**Patroni configuration file**
+```
+Please copy&paste your Patroni configuration file here
+```
+
+**patronictl show-config**
+```
+Please copy&paste the output of "patronictl show-config" command here
+```
+
+**Have you checked Patroni logs?**
+Please provide a snippet of Patroni log files here
+
+**Have you checked PostgreSQL logs?**
+Please provide a snippet here
+
+**Have you tried to use GitHub issue search?**
+Maybe there is already a similar issue solved.
+
+
+**Additional context**
+Add any other context about the problem here.

docs/ENVIRONMENT.rst

@@ -109,9 +109,11 @@ Kubernetes
 Raft
 ----
 
--  **PATRONI\_RAFT\_SELF\_ADDR**: ``ip:port`` to listen on for Raft connections. If not set, the node will not participate in consensus.
+-  **PATRONI\_RAFT\_SELF\_ADDR**: ``ip:port`` to listen on for Raft connections. The ``self_addr`` must be accessible from other nodes of the cluster. If not set, the node will not participate in consensus.
+-  **PATRONI\_RAFT\_BIND\_ADDR**: (optional) ``ip:port`` to listen on for Raft connections. If not specified the ``self_addr`` will be used.
 -  **PATRONI\_RAFT\_PARTNER\_ADDRS**: list of other Patroni nodes in the cluster in format ``"'ip1:port1','ip2:port2'"``. It is important to quote every single entity!
 -  **PATRONI\_RAFT\_DATA\_DIR**: directory where to store Raft log and snapshot. If not specified the current working directory is used.
+-  **PATRONI\_RAFT\_PASSWORD**: (optional) Encrypt Raft traffic with a specified password, requires ``cryptography`` python module.
 
 PostgreSQL
 ----------
@@ -125,6 +127,7 @@ PostgreSQL
 -  **PATRONI\_REPLICATION\_PASSWORD**: replication password; the user will be created during initialization.
 -  **PATRONI\_REPLICATION\_SSLMODE**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
 -  **PATRONI\_REPLICATION\_SSLKEY**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+-  **PATRONI\_REPLICATION\_SSLPASSWORD**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``PATRONI_REPLICATION_SSLKEY``.
 -  **PATRONI\_REPLICATION\_SSLCERT**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
 -  **PATRONI\_REPLICATION\_SSLROOTCERT**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
 -  **PATRONI\_REPLICATION\_SSLCRL**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.
@@ -134,6 +137,7 @@ PostgreSQL
 -  **PATRONI\_SUPERUSER\_PASSWORD**: password for the superuser, set during initialization (initdb).
 -  **PATRONI\_SUPERUSER\_SSLMODE**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
 -  **PATRONI\_SUPERUSER\_SSLKEY**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+-  **PATRONI\_SUPERUSER\_SSLPASSWORD**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``PATRONI_SUPERUSER_SSLKEY``.
 -  **PATRONI\_SUPERUSER\_SSLCERT**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
 -  **PATRONI\_SUPERUSER\_SSLROOTCERT**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
 -  **PATRONI\_SUPERUSER\_SSLCRL**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.
@@ -143,6 +147,7 @@ PostgreSQL
 -  **PATRONI\_REWIND\_PASSWORD**: password for the user for ``pg_rewind``; the user will be created during initialization.
 -  **PATRONI\_REWIND\_SSLMODE**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
 -  **PATRONI\_REWIND\_SSLKEY**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+-  **PATRONI\_REWIND\_SSLPASSWORD**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``PATRONI_REWIND_SSLKEY``.
 -  **PATRONI\_REWIND\_SSLCERT**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
 -  **PATRONI\_REWIND\_SSLROOTCERT**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
 -  **PATRONI\_REWIND\_SSLCRL**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.

docs/SETTINGS.rst

@@ -183,9 +183,11 @@ Kubernetes
 
 Raft
 ----
--  **self\_addr**: ``ip:port`` to listen on for Raft connections. If not set, the node will not participate in consensus.
+-  **self\_addr**: ``ip:port`` to listen on for Raft connections. The ``self_addr`` must be accessible from other nodes of the cluster. If not set, the node will not participate in consensus.
+-  **bind\_addr**: (optional) ``ip:port`` to listen on for Raft connections. If not specified the ``self_addr`` will be used.
 -  **partner\_addrs**: list of other Patroni nodes in the cluster in format: ['ip1:port', 'ip2:port', 'etc...']
 -  **data\_dir**: directory where to store Raft log and snapshot. If not specified the current working directory is used.
+-  **password**: (optional) Encrypt Raft traffic with a specified password, requires ``cryptography`` python module.
 
   Short FAQ about Raft implementation
 
@@ -221,6 +223,7 @@ PostgreSQL
             -  **password**: password for the superuser, set during initialization (initdb).
             -  **sslmode**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
             -  **sslkey**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+            -  **sslpassword**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``sslkey``.
             -  **sslcert**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
             -  **sslrootcert**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
             -  **sslcrl**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.
@@ -231,6 +234,7 @@ PostgreSQL
             -  **password**: replication password; the user will be created during initialization.
             -  **sslmode**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
             -  **sslkey**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+            -  **sslpassword**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``sslkey``.
             -  **sslcert**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
             -  **sslrootcert**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
             -  **sslcrl**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.
@@ -241,6 +245,7 @@ PostgreSQL
             -  **password**: password for the user for ``pg_rewind``; the user will be created during initialization.
             -  **sslmode**: (optional) maps to the `sslmode <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLMODE>`__ connection parameter, which allows a client to specify the type of TLS negotiation mode with the server. For more information on how each mode works, please visit the `PostgreSQL documentation <https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-SSLMODE-STATEMENTS>`__. The default mode is ``prefer``.
             -  **sslkey**: (optional) maps to the `sslkey <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLKEY>`__ connection parameter, which specifies the location of the secret key used with the client's certificate.
+            -  **sslpassword**: (optional) maps to the `sslpassword <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLPASSWORD>`__ connection parameter, which specifies the password for the secret key specified in ``sslkey``.
             -  **sslcert**: (optional) maps to the `sslcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCERT>`__ connection parameter, which specifies the location of the client certificate.
             -  **sslrootcert**: (optional) maps to the `sslrootcert <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLROOTCERT>`__ connection parameter, which specifies the location of a file containing one ore more certificate authorities (CA) certificates that the client will use to verify a server's certificate.
             -  **sslcrl**: (optional) maps to the `sslcrl <https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-SSLCRL>`__ connection parameter, which specifies the location of a file containing a certificate revocation list. A client will reject connecting to any server that has a certificate present in this list.

docs/releases.rst

@@ -3,6 +3,54 @@
 Release notes
 =============
 
+Version 2.0.1
+-------------
+
+**New features**
+
+- Use ``more`` as pager in ``patronictl edit-config`` if ``less`` is not available (Pavel Golub)
+
+  On Windows it would be the ``more.com``. In addition to that, ``cdiff`` was changed to ``ydiff`` in ``requirements.txt``, but ``patronictl`` still supports both for compatibility.
+
+- Added support of ``raft`` ``bind_addr`` and ``password`` (Alexander Kukushkin)
+
+  ``raft.bind_addr`` might be useful when running behind NAT. ``raft.password`` enables traffic encryption (requires the ``cryptography`` module).
+
+- Added ``sslpassword`` connection parameter support (Kostiantyn Nemchenko)
+
+  The connection parameter was introduced in PostgreSQL 13.
+
+**Stability improvements**
+
+- Changed the behavior in pause (Alexander)
+
+  1. Patroni will not call the ``bootstrap`` method if the ``PGDATA`` directory is missing/empty.
+  2. Patroni will not exit on sysid mismatch in pause, only log a warning.
+  3. The node will not try to grab the leader key in pause mode if Postgres is running not in recovery (accepting writes) but the sysid doesn't match with the initialize key.
+
+- Apply ``master_start_timeout`` when executing crash recovery (Alexander)
+
+  If Postgres crashed on the leader node, Patroni does a crash-recovery by starting Postgres in single-user mode. During the crash-recovery the leader lock is being updated. If the crash-recovery didn't finish in ``master_start_timeout`` seconds, Patroni will stop it forcefully and release the leader lock.
+
+- Removed the ``secure`` extra from the ``urllib3`` requirements (Alexander)
+
+  The only reason for adding it there was the ``ipaddress`` dependency for python 2.7.
+
+**Bugfixes**
+
+- Fixed a bug in the ``Kubernetes.update_leader()`` (Alexander)
+
+  An unhandled exception was preventing demoting the primary when the update of the leader object failed.
+
+- Fixed hanging ``patronictl`` when RAFT is being used (Alexander)
+
+  When using ``patronictl`` with Patroni config, ``self_addr`` should be added to the ``partner_addrs``.
+
+- Fixed bug in ``get_guc_value()`` (Alexander)
+
+  Patroni was failing to get the value of ``restore_command`` on PostgreSQL 12, therefore fetching missing WALs for ``pg_rewind`` didn't work.
+
+
 Version 2.0.0
 -------------
 

extras/startup-scripts/patroni.service

@@ -14,7 +14,8 @@ Group=postgres
 # Read in configuration file if it exists, otherwise proceed
 EnvironmentFile=-/etc/patroni_env.conf
 
-WorkingDirectory=~
+# the default is the user's home directory, and if you want to change it, you must provide an absolute path.
+# WorkingDirectory=/home/sameuser
 
 # Where to send early-startup messages from the server
 # This is normally controlled by the global default set by systemd

features/patroni_api.feature

@@ -61,9 +61,15 @@ Scenario: check the scheduled restart
 		And postgres0 role is the primary after 10 seconds
 
 Scenario: check API requests for the primary-replica pair in the pause mode
-	Given I run patronictl.py pause batman
+	Given I start postgres1
+	Then replication works from postgres0 to postgres1 after 20 seconds
+	When I run patronictl.py pause batman
+	Then I receive a response returncode 0
+	When I kill postmaster on postgres1
+	And I issue a GET request to http://127.0.0.1:8009/replica
+	Then I receive a response code 503
+	When I run patronictl.py restart batman postgres1 --force
 	Then I receive a response returncode 0
-	When I start postgres1
 	Then replication works from postgres0 to postgres1 after 20 seconds
 	When I issue a GET request to http://127.0.0.1:8009/replica
 	Then I receive a response code 200

patroni/config.py

@@ -21,6 +21,7 @@ _AUTH_ALLOWED_PARAMETERS = (
     'sslmode',
     'sslcert',
     'sslkey',
+    'sslpassword',
     'sslrootcert',
     'sslcrl',
     'gssencmode',
@@ -316,7 +317,7 @@ class Config(object):
                 logger.exception('Exception when parsing list %s', value)
                 return None
 
-        _set_section_values('raft', ['data_dir', 'self_addr', 'partner_addrs'])
+        _set_section_values('raft', ['data_dir', 'self_addr', 'partner_addrs', 'password', 'bind_addr'])
         if 'raft' in ret and 'partner_addrs' in ret['raft']:
             ret['raft']['partner_addrs'] = _parse_list(ret['raft']['partner_addrs'])
 

patroni/ctl.py

@@ -7,7 +7,6 @@ import codecs
 import datetime
 import dateutil.parser
 import dateutil.tz
-import cdiff
 import copy
 import difflib
 import io
@@ -34,6 +33,10 @@ from patroni.request import PatroniRequest
 from patroni.version import __version__
 from prettytable import ALL, FRAME, PrettyTable
 from six.moves.urllib_parse import urlparse
+try:
+    from ydiff import markup_to_pager, PatchStream
+except ImportError:  # pragma: no cover
+    from cdiff import markup_to_pager, PatchStream
 
 CONFIG_DIR_PATH = click.get_app_dir('patroni')
 CONFIG_FILE_PATH = os.path.join(CONFIG_DIR_PATH, 'patronictl.yaml')
@@ -1086,7 +1089,14 @@ def show_diff(before_editing, after_editing):
             side_by_side = False
             width = 80
             tab_width = 8
-        cdiff.markup_to_pager(cdiff.PatchStream(buf), opts)
+            wrap = True
+            if find_executable('less'):
+                pager = None
+            else:
+                pager = 'more.com' if sys.platform == 'win32' else 'more'
+            pager_options = None
+
+        markup_to_pager(PatchStream(buf), opts)
     else:
         for line in unified_diff:
             click.echo(line.rstrip('\n'))

patroni/dcs/kubernetes.py

@@ -901,7 +901,7 @@ class Kubernetes(AbstractDCS):
             else:
                 logger.exception('Permission denied' if e.status == 403 else 'Unexpected error from Kubernetes API')
                 return False
-        except RetryFailedError:
+        except (RetryFailedError, K8sException):
             return False
 
         deadline = retry.stoptime - time.time()

patroni/dcs/raft.py

@@ -87,15 +87,14 @@ class DynMemberSyncObj(SyncObj):
 
     def __init__(self, selfAddress, partnerAddrs, conf):
         add_self = False
-        if selfAddress:
-            utility = SyncObjUtility(partnerAddrs, conf)
-            for node in utility._SyncObj__otherNodes:
-                utility.setPartnerNode(node)
-                response = utility.sendMessage(['members'])
-                if response:
-                    partnerAddrs = [member['addr'] for member in response if member['addr'] != selfAddress]
-                    add_self = len(partnerAddrs) == len(response)
-                    break
+        utility = SyncObjUtility(partnerAddrs, conf)
+        for node in utility._SyncObj__otherNodes:
+            utility.setPartnerNode(node)
+            response = utility.sendMessage(['members'])
+            if response:
+                partnerAddrs = [member['addr'] for member in response if member['addr'] != selfAddress]
+                add_self = selfAddress and len(partnerAddrs) == len(response)
+                break
 
         super(DynMemberSyncObj, self).__init__(selfAddress, partnerAddrs, conf, transportClass=MyTCPTransport)
         if add_self:
@@ -269,14 +268,22 @@ class Raft(AbstractDCS):
         super(Raft, self).__init__(config)
         self._ttl = int(config.get('ttl') or 30)
 
-        self_addr = None if self._ctl else config.get('self_addr')
-        template = os.path.join(config.get('data_dir', ''), self_addr or '')
-        files = {'journalFile': template + '.journal', 'fullDumpFile': template + '.dump'} if self_addr else {}
+        self_addr = config.get('self_addr')
+        partner_addrs = config.get('partner_addrs', [])
+        if self._ctl:
+            if self_addr:
+                partner_addrs.append(self_addr)
+            self_addr = None
 
         ready_event = threading.Event()
-        conf = SyncObjConf(commandsWaitLeader=False, appendEntriesUseBatch=False, onReady=ready_event.set,
-                           dynamicMembershipChange=True, **files)
-        self._sync_obj = KVStoreTTL(self_addr, config.get('partner_addrs', []), conf, self._on_set, self._on_delete)
+        file_template = os.path.join(config.get('data_dir', ''), (self_addr or ''))
+        conf = SyncObjConf(password=config.get('password'), appendEntriesUseBatch=False,
+                           bindAddress=config.get('bind_addr'), commandsWaitLeader=False,
+                           fullDumpFile=(file_template + '.dump' if self_addr else None),
+                           journalFile=(file_template + '.journal' if self_addr else None),
+                           onReady=ready_event.set, dynamicMembershipChange=True)
+
+        self._sync_obj = KVStoreTTL(self_addr, partner_addrs, conf, self._on_set, self._on_delete)
         while True:
             ready_event.wait(5)
             if ready_event.isSet() or self._sync_obj.applied_local_log:

patroni/ha.py

@@ -72,6 +72,7 @@ class Ha(object):
         self.recovering = False
         self._async_response = CriticalTask()
         self._crash_recovery_executed = False
+        self._crash_recovery_started = None
         self._start_timeout = None
         self._async_executor = AsyncExecutor(self.state_handler.cancellable, self.wakeup)
         self.watchdog = patroni.watchdog
@@ -156,7 +157,11 @@ class Ha(object):
                 last_operation = self.state_handler.last_operation()
             except Exception:
                 logger.exception('Exception when called state_handler.last_operation()')
-        ret = self.dcs.update_leader(last_operation, self._leader_access_is_restricted)
+        try:
+            ret = self.dcs.update_leader(last_operation, self._leader_access_is_restricted)
+        except Exception:
+            logger.exception('Unexpected exception raised from update_leader, please report it as a BUG')
+            ret = False
         self.set_is_leader(ret)
         if ret:
             self.watchdog.keepalive()
@@ -329,6 +334,7 @@ class Ha(object):
                 and not self._crash_recovery_executed and \
                 (self.cluster.is_unlocked() or self._rewind.can_rewind):
             self._crash_recovery_executed = True
+            self._crash_recovery_started = time.time()
             msg = 'doing crash recovery in a single user mode'
             return self._async_executor.try_run_async(msg, self._rewind.ensure_clean_shutdown) or msg
 
@@ -761,8 +767,10 @@ class Ha(object):
         if self.state_handler.is_starting():  # postgresql still starting up is unhealthy
             return False
 
-        if self.state_handler.is_leader():  # leader is always the healthiest
-            return True
+        if self.state_handler.is_leader():
+            # in pause leader is the healthiest only when no initialize or sysid matches with initialize!
+            return not self.is_paused() or not self.cluster.initialize\
+                    or self.state_handler.sysid == self.cluster.initialize
 
         if self.is_paused():
             return False
@@ -1165,6 +1173,14 @@ class Ha(object):
         Figure out what to do with the task AsyncExecutor is performing.
         """
         if self.has_lock() and self.update_lock():
+            if self._async_executor.scheduled_action == 'doing crash recovery in a single user mode':
+                time_left = self.patroni.config['master_start_timeout'] - (time.time() - self._crash_recovery_started)
+                if time_left <= 0 and self.is_failover_possible(self.cluster.members):
+                    logger.info("Demoting self because crash recovery is taking too long")
+                    self.state_handler.cancellable.cancel(True)
+                    self.demote('immediate')
+                    return 'terminated crash recovery because of startup timeout'
+
             return 'updated leader lock during ' + self._async_executor.scheduled_action
         elif not self.state_handler.bootstrapping:
             # Don't have lock, make sure we are not promoting or starting up a master in the background
@@ -1348,6 +1364,8 @@ class Ha(object):
                     self.release_leader_key_voluntarily()
                     return 'released leader key voluntarily as data dir empty and currently leader'
 
+                if self.is_paused():
+                    return 'running with empty data directory'
                 return self.bootstrap()  # new node
             else:
                 # check if we are allowed to join
@@ -1359,9 +1377,16 @@ class Ha(object):
 
                 if self.sysid_valid(self.cluster.initialize):
                     if self.cluster.initialize != data_sysid:
-                        logger.fatal("system ID mismatch, node %s belongs to a different cluster: %s != %s",
-                                     self.state_handler.name, self.cluster.initialize, data_sysid)
-                        sys.exit(1)
+                        if self.is_paused():
+                            logger.warning('system ID has changed while in paused mode. Patroni will exit when resuming'
+                                           ' unless system ID is reset: %s != %s', self.cluster.initialize, data_sysid)
+                            if self.has_lock():
+                                self.release_leader_key_voluntarily()
+                                return 'released leader key voluntarily due to the system ID mismatch'
+                        else:
+                            logger.fatal('system ID mismatch, node %s belongs to a different cluster: %s != %s',
+                                         self.state_handler.name, self.cluster.initialize, data_sysid)
+                            sys.exit(1)
                 elif self.cluster.is_unlocked() and not self.is_paused():
                     # "bootstrap", but data directory is not empty
                     if not self.state_handler.cb_called and self.state_handler.is_running() \

patroni/postgresql/__init__.py

@@ -459,6 +459,8 @@ class Postgresql(object):
         self._pending_restart = False
 
         try:
+            if not self._major_version:
+                self.configure_server_parameters()
             configuration = self.config.effective_configuration
         except Exception:
             return None
@@ -698,7 +700,7 @@ class Postgresql(object):
         return True
 
     def get_guc_value(self, name):
-        cmd = [self.pgcommand('postgres'), self._data_dir, '-C', name]
+        cmd = [self.pgcommand('postgres'), '-D', self._data_dir, '-C', name]
         try:
             data = subprocess.check_output(cmd)
             if data:
@@ -1001,8 +1003,11 @@ class Postgresql(object):
     def schedule_sanity_checks_after_pause(self):
         """
             After coming out of pause we have to:
-            1. sync replication slots, because it might happen that slots were removed
-            2. get new 'Database system identifier' to make sure that it wasn't changed
+            1. configure server parameters if necessary
+            2. sync replication slots, because it might happen that slots were removed
+            3. get new 'Database system identifier' to make sure that it wasn't changed
         """
+        if not self._major_version:
+            self.configure_server_parameters()
         self.slots_handler.schedule()
         self._sysid = None

patroni/postgresql/cancellable.py

@@ -114,7 +114,7 @@ class CancellableSubprocess(CancellableExecutor):
         with self._lock:
             return self._is_cancelled
 
-    def cancel(self):
+    def cancel(self, kill=False):
         with self._lock:
             self._is_cancelled = True
             if self._process is None or not self._process.is_running():
@@ -127,5 +127,7 @@ class CancellableSubprocess(CancellableExecutor):
             with self._lock:
                 if self._process is None or not self._process.is_running():
                     return
+            if kill:
+                break
 
         self._kill_process()

patroni/postgresql/config.py

@@ -478,7 +478,7 @@ class ConfigHandler(object):
     def format_dsn(self, params, include_dbname=False):
         # A list of keywords that can be found in a conninfo string. Follows what is acceptable by libpq
         keywords = ('dbname', 'user', 'passfile' if params.get('passfile') else 'password', 'host', 'port',
-                    'sslmode', 'sslcompression', 'sslcert', 'sslkey', 'sslrootcert', 'sslcrl',
+                    'sslmode', 'sslcompression', 'sslcert', 'sslkey', 'sslpassword', 'sslrootcert', 'sslcrl',
                     'application_name', 'krbsrvname', 'gssencmode', 'channel_binding')
         if include_dbname:
             params = params.copy()

patroni/postgresql/rewind.py

@@ -1,5 +1,6 @@
 import logging
 import os
+import shlex
 import six
 import subprocess
 
@@ -273,7 +274,7 @@ class Rewind(object):
             i += 1
 
         logger.info('Trying to fetch the missing wal: %s', cmd)
-        return self._postgresql.cancellable.call(cmd, shell=True) == 0
+        return self._postgresql.cancellable.call(shlex.split(cmd)) == 0
 
     def _find_missing_wal(self, data):
         # could not open file "$PGDATA/pg_wal/0000000A00006AA100000068": No such file or directory
@@ -427,4 +428,4 @@ class Rewind(object):
         opts = self.read_postmaster_opts()
         opts.update({'archive_mode': 'on', 'archive_command': 'false'})
         self._postgresql.config.remove_recovery_conf()
-        return self.single_user_mode(options=opts) == 0 or None
+        return self.single_user_mode(communicate={}, options=opts) == 0 or None

patroni/postgresql/validator.py

@@ -484,7 +484,7 @@ recovery_parameters = CaseInsensitiveDict({
     'promote_trigger_file': String(120000, None),
     'recovery_end_command': String(90300, None),
     'recovery_min_apply_delay': Integer(90400, None, 0, 2147483647, 'ms'),
-    'recovery_target': Enum(90400, None, ('immediate',)),
+    'recovery_target': Enum(90400, None, ('immediate', '')),
     'recovery_target_action': Enum(90500, None, ('pause', 'promote', 'shutdown')),
     'recovery_target_inclusive': Bool(90300, None),
     'recovery_target_lsn': String(100000, None),

patroni/version.py

@@ -1 +1 @@
-__version__ = '2.0.0'
+__version__ = '2.0.1'

requirements.txt

@@ -1,4 +1,5 @@
-urllib3[secure]>=1.19.1,!=1.21
+urllib3>=1.19.1,!=1.21
+ipaddress; python_version=="2.7"
 boto
 PyYAML
 six >= 1.7
@@ -8,6 +9,6 @@ python-consul>=0.7.1
 click>=4.1
 prettytable>=0.7
 python-dateutil
-pysyncobj>=0.3.5
+pysyncobj>=0.3.7
 psutil>=2.0.0
-cdiff
+ydiff>=1.2.0

setup.py

@@ -23,7 +23,7 @@ KEYWORDS = 'etcd governor patroni postgresql postgres ha haproxy confd' +\
     ' zookeeper exhibitor consul streaming replication kubernetes k8s'
 
 EXTRAS_REQUIRE = {'aws': ['boto'], 'etcd': ['python-etcd'], 'etcd3': ['python-etcd'], 'consul': ['python-consul'],
-                  'exhibitor': ['kazoo'], 'zookeeper': ['kazoo'], 'kubernetes': [], 'raft': ['pysyncobj']}
+                  'exhibitor': ['kazoo'], 'zookeeper': ['kazoo'], 'kubernetes': ['ipaddress'], 'raft': ['pysyncobj']}
 COVERAGE_XML = True
 COVERAGE_HTML = False
 
@@ -143,9 +143,13 @@ class PyTest(Command):
 
     def run(self):
         from pkg_resources import evaluate_marker
-        requirements = self.distribution.install_requires + ['mock>=2.0.0', 'pytest-cov', 'pytest'] +\
-            [v for k, v in self.distribution.extras_require.items() if not k.startswith(':') or evaluate_marker(k[1:])]
-        self.distribution.fetch_build_eggs(requirements)
+
+        requirements = set(self.distribution.install_requires + ['mock>=2.0.0', 'pytest-cov', 'pytest'])
+        for k, v in self.distribution.extras_require.items():
+            if not k.startswith(':') or evaluate_marker(k[1:]):
+                requirements.update(v)
+
+        self.distribution.fetch_build_eggs(list(requirements))
         self.run_tests()
 
 
@@ -167,7 +171,7 @@ def setup_package(version):
         extra = False
         for e, v in EXTRAS_REQUIRE.items():
             if v and r.startswith(v[0]):
-                EXTRAS_REQUIRE[e] = [r]
+                EXTRAS_REQUIRE[e] = [r] if e != 'kubernetes' or sys.version_info < (3, 0, 0) else []
                 extra = True
         if not extra:
             install_requires.append(r)

tests/test_ctl.py

@@ -561,7 +561,7 @@ class TestCtl(unittest.TestCase):
         self.assertRaises(PatroniCtlException, apply_config_changes, before_editing, config, ['a'])
 
     @patch('sys.stdout.isatty', return_value=False)
-    @patch('cdiff.markup_to_pager')
+    @patch('patroni.ctl.markup_to_pager')
     def test_show_diff(self, mock_markup_to_pager, mock_isatty):
         show_diff("foo:\n  bar: 1\n", "foo:\n  bar: 2\n")
         mock_markup_to_pager.assert_not_called()
@@ -570,6 +570,9 @@ class TestCtl(unittest.TestCase):
         show_diff("foo:\n  bar: 1\n", "foo:\n  bar: 2\n")
         mock_markup_to_pager.assert_called_once()
 
+        with patch('patroni.ctl.find_executable', Mock(return_value=None)):
+            show_diff("foo:\n  bar: 1\n", "foo:\n  bar: 2\n")
+
         # Test that unicode handling doesn't fail with an exception
         show_diff(b"foo:\n  bar: \xc3\xb6\xc3\xb6\n".decode('utf-8'),
                   b"foo:\n  bar: \xc3\xbc\xc3\xbc\n".decode('utf-8'))

tests/test_ha.py

@@ -3,7 +3,7 @@ import etcd
 import os
 import sys
 
-from mock import call, Mock, MagicMock, PropertyMock, patch, mock_open
+from mock import Mock, MagicMock, PropertyMock, patch, mock_open
 from patroni.config import Config
 from patroni.dcs import Cluster, ClusterConfig, Failover, Leader, Member, get_dcs, SyncState, TimelineHistory
 from patroni.dcs.etcd import AbstractEtcdClientWithFailover
@@ -200,7 +200,8 @@ class TestHa(PostgresInit):
 
     def test_update_lock(self):
         self.p.last_operation = Mock(side_effect=PostgresConnectionException(''))
-        self.assertTrue(self.ha.update_lock(True))
+        self.ha.dcs.update_leader = Mock(side_effect=Exception)
+        self.assertFalse(self.ha.update_lock(True))
 
     @patch.object(Postgresql, 'received_timeline', Mock(return_value=None))
     def test_touch_member(self):
@@ -260,9 +261,17 @@ class TestHa(PostgresInit):
 
     @patch.object(Rewind, 'ensure_clean_shutdown', Mock())
     def test_crash_recovery(self):
+        self.ha.has_lock = true
         self.p.is_running = false
         self.p.controldata = lambda: {'Database cluster state': 'in production', 'Database system identifier': SYSID}
         self.assertEqual(self.ha.run_cycle(), 'doing crash recovery in a single user mode')
+        with patch('patroni.async_executor.AsyncExecutor.busy', PropertyMock(return_value=True)),\
+                patch.object(Ha, 'check_timeline', Mock(return_value=False)):
+            self.ha._async_executor.schedule('doing crash recovery in a single user mode')
+            self.ha.state_handler.cancellable._process = Mock()
+            self.ha._crash_recovery_started -= 600
+            self.ha.patroni.config.set_dynamic_configuration({'maximum_lag_on_failover': 10})
+            self.assertEqual(self.ha.run_cycle(), 'terminated crash recovery because of startup timeout')
 
     @patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
     @patch.object(Rewind, 'can_rewind', PropertyMock(return_value=True))
@@ -901,9 +910,8 @@ class TestHa(PostgresInit):
         self.p.pick_synchronous_standby = Mock(return_value=(['other2', 'other3'], ['other2']))
         self.ha.dcs.write_sync_state = Mock(return_value=True)
         self.ha.run_cycle()
-        # mock_set_sync.assert_called_once_with(['other2'])
-        calls = [call(['other2']), call(['other2', 'other3'])]
-        mock_set_sync.assert_has_calls(calls)
+        self.assertEqual(mock_set_sync.call_args_list[0][0], (['other2'],))
+        self.assertEqual(mock_set_sync.call_args_list[1][0], (['other2', 'other3'],))
 
         mock_set_sync.reset_mock()
         # Test sync standby is not disabled when updating dcs fails
@@ -1111,3 +1119,18 @@ class TestHa(PostgresInit):
         self.assertEqual(self.ha.run_cycle(), 'Unexpected exception raised, please report it as a BUG')
         self.ha.dcs.touch_member = Mock(side_effect=PatroniFatalException('foo'))
         self.assertRaises(PatroniFatalException, self.ha.run_cycle)
+
+    def test_empty_directory_in_pause(self):
+        self.ha.is_paused = true
+        self.p.data_directory_empty = true
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: running with empty data directory')
+        self.assertEqual(self.p.role, 'uninitialized')
+
+    @patch('patroni.ha.Ha.sysid_valid', MagicMock(return_value=True))
+    def test_sysid_no_match_in_pause(self):
+        self.ha.is_paused = true
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery', 'Database system identifier': '123'}
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
+
+        self.ha.has_lock = true
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: released leader key voluntarily due to the system ID mismatch')

tests/test_raft.py

@@ -152,4 +152,4 @@ class TestRaft(unittest.TestCase):
     def test_init(self, mock_event, mock_kvstore):
         mock_kvstore.return_value.applied_local_log = False
         mock_event.return_value.isSet.side_effect = [False, True]
-        self.assertIsNotNone(Raft({'ttl': 30, 'scope': 'test', 'name': 'pg', 'patronictl': True}))
+        self.assertIsNotNone(Raft({'ttl': 30, 'scope': 'test', 'name': 'pg', 'patronictl': True, 'self_addr': '1'}))