Bump version and write release notes (#560)

and implement missing unit-tests

docs/ENVIRONMENT.rst

@@ -33,6 +33,8 @@ Consul
 -  **PATRONI\_CONSUL\_CACERT**: (optional) The ca certificate. If pressent it will enable validation.
 -  **PATRONI\_CONSUL\_CERT**: (optional) File with the client certificate
 -  **PATRONI\_CONSUL\_KEY**: (optional) File with the client key. Can be empty if the key is part of certificate.
+-  **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
+-  **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable.
 
 Etcd
 ----

docs/SETTINGS.rst

@@ -58,6 +58,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **cacert**: (optional) The ca certificate. If pressent it will enable validation.
 -  **cert**: (optional) file with the client certificate
 -  **key**: (optional) file with the client key. Can be empty if the key is part of **cert**.
+-  **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
+-  **checks**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable
 
 Etcd
 ----

docs/releases.rst

@@ -3,6 +3,78 @@
 Release notes
 =============
 
+Version 1.3.6
+-------------
+
+**Stability improvements**
+
+- Verify process start time when checking if postgres is running. (Ants Aasma)
+
+  After a crash that doesn't clean up postmaster.pid there could be a new process with the same pid, resulting in a false positive for is_running(), which will lead to all kinds of bad behavior.
+
+- Shutdown postgresql before bootstrap when we lost data directory (ainlolcat)
+  
+  When data directory on the master is forcefully removed, postgres process can still stay alive for some time and prevent the replica created in place of that former master from starting or replicating.
+  The fix makes Patroni cache the postmaster pid and its start time and let it terminate the old postmaster in case it is still running after the corresponding data directory has been removed.
+
+- Perform crash recovery in a single user mode if postgres master dies (Alexander Kukushkin)
+
+  It is unsafe to start immediately as a standby and not possible to run ``pg_rewind`` if postgres hasn't been shut down cleanly.
+  The single user crash recovery only kicks in if ``pg_rewind`` is enabled or there is no master at the moment.
+
+**Consul improvements**
+
+- Make it possible to provide datacenter configuration for Consul (DeathBorn, Alexander)
+
+  Before that Patroni was always communicating with datacenter of the host it runs on.
+
+- Always send a token in X-Consul-Token http header (Alexander)
+
+  If ``consul.token`` is defined in Patroni configuration, we will always send it in the 'X-Consul-Token' http header.
+  python-consul module tries to be "consistent" with Consul REST API, which doesn't accept token as a query parameter for `session API <https://www.consul.io/api/session.html>`__, but it still works with 'X-Consul-Token' header.
+
+- Adjust session TTL if supplied value is smaller than the minimum possible (Stas Fomin, Alexander)
+
+  It could happen that the TTL provided in the Patroni configuration is smaller than the minimum one supported by Consul. In that case, Consul agent fails to create a new session.
+  Without a session Patroni cannot create member and leader keys in the Consul KV store, resulting in an unhealthy cluster.
+
+**Other improvements**
+
+- Define custom log format via environment variable ``PATRONI_LOGFORMAT`` (Stas)
+
+  Allow disabling timestamps and other similar fields in Patroni logs if they are already added by the system logger (usually when Patroni runs as a service).
+
+Version 1.3.5
+-------------
+
+**Bugfix**
+
+- Set role to 'uninitialized' if data directory was removed (Alexander Kukushkin)
+
+  If the node was running as a master it was preventing from failover.
+
+**Stability improvement**
+
+- Try to run postmaster in a single-user mode if we tried and failed to start postgres (Alexander)
+
+  Usually such problem happens when node running as a master was terminated and timelines were diverged.
+  If ``recovery.conf`` has ``restore_command`` defined, there are really high chances that postgres will abort startup and leave controldata unchanged.
+  It makes impossible to use ``pg_rewind``, which requires a clean shutdown.
+
+**Consul improvements**
+
+- Make it possible to specify health checks when creating session (Alexander)
+
+  If not specified, Consul will use "serfHealth". From one side it allows fast detection of isolated master, but from another side it makes it impossible for Patroni to tolerate short network lags.
+
+**Bugfix**
+
+- Fix watchdog on Python 3 (Ants Aasma)
+
+  A misunderstanding of the ioctl() call interface. If mutable=False then fcntl.ioctl() actually returns the arg buffer back.
+  This accidentally worked on Python2 because int and str comparison did not return an error.
+  Error reporting is actually done by raising IOError on Python2 and OSError on Python3.
+
 Version 1.3.4
 -------------
 

extras/startup-scripts/patroni

@@ -1,144 +1,144 @@
-#!/bin/sh
-#
-### BEGIN INIT INFO
-# Provides:          patroni
-# Required-Start:    $remote_fs $syslog
-# Required-Stop:     $remote_fs $syslog
-# Default-Start:     2 3 4 5
-# Default-Stop:      0 1 6
-# Short-Description: Patroni init script
-# Description:       Runners to orchestrate a high-availability PostgreSQL
-### END INIT INFO
-
-### BEGIN USER CONFIGURATION
-
-CONF="/etc/patroni/postgres.yml"
-LOGFILE="/var/log/patroni.log"
-USER="postgres"
-GROUP="postgres"
-
-NAME=patroni
-PATRONI="/opt/patroni/$NAME.py"
-PIDFILE="/var/run/$NAME.pid"
-
-# Set this parameter, if you have several Postgres versions installed
-# POSTGRES_VERSION="9.4"
-POSTGRES_VERSION=""
-
-### END USER CONFIGURATION
-
-. /lib/lsb/init-functions
-
-# Loading this library for get_versions() function
-if test ! -e /usr/share/postgresql-common/init.d-functions; then
-  log_failure_msg "Probably postgresql-common does not installed."
-  exit 1
-else
-  . /usr/share/postgresql-common/init.d-functions
-fi
-
-# Is there Patroni executable?
-if test ! -e $PATRONI; then
-  log_failure_msg "Patroni executable $PATRONI does not exist."
-  exit 1
-fi
-
-# Is there Patroni configuration file?
-if test ! -e $CONF; then
-  log_failure_msg "Patroni configuration file $CONF does not exist."
-  exit 1
-fi
-
-# Create logfile if doesn't exist
-if test ! -e $LOGFILE; then
-  log_action_msg "Creating logfile for Patroni..."
-  touch $LOGFILE
-  chown $USER:$GROUP $LOGFILE
-fi
-
-prepare_pgpath() {
-  if [ "$POSTGRES_VERSION" != "" ]; then
-    if [ -x /usr/lib/postgresql/$POSTGRES_VERSION/bin/pg_ctl ]; then
-      PGPATH="/usr/lib/postgresql/$POSTGRES_VERSION/bin"
-    else
-      log_failure_msg "Postgres version incorrect, check POSTGRES_VERSION variable."
-      exit 0
-    fi
-  else
-    get_versions
-    if echo $versions | grep -q -e "\s"; then
-      log_warning_msg "You have several Postgres versions installed. Please, use POSTGRES_VERSION to define correct environment."
-    else
-      versions=`echo $versions | sed -e 's/^[ \t]*//'`
-      PGPATH="/usr/lib/postgresql/$versions/bin"
-    fi
-  fi
-}
-
-get_pid() {
-  if test -e $PIDFILE; then
-    PID=`cat $PIDFILE`
-    CHILDPID=`ps --ppid $PID -o %p --no-headers`
-  else
-    log_failure_msg "Could not find PID file. Patroni probably down."
-    exit 1
-  fi
-}
-
-
-case "$1" in
-  start)
-    prepare_pgpath
-    PGPATH=$PATH:$PGPATH
-    log_success_msg "Starting Patroni\n"
-    exec start-stop-daemon --start --quiet \
-        --background \
-        --pidfile $PIDFILE --make-pidfile \
-        --chuid $USER:$GROUP \
-        --chdir `eval echo ~$USER` \
-        --exec $PATRONI \
-        --startas /bin/sh -- \
-        -c "/usr/bin/env PATH=$PGPATH /usr/bin/python $PATRONI $CONF >> $LOGFILE 2>&1"
-    ;;
-
-  stop)
-    log_success_msg "Stopping Patroni"
-    get_pid
-    start-stop-daemon --stop --pid $CHILDPID
-    start-stop-daemon --stop --pidfile $PIDFILE --remove-pidfile --quiet
-    ;;
-
-  reload)
-    log_success_msg "Reloading Patroni configuration"
-    get_pid
-    kill -HUP $CHILDPID
-    ;;
-
-  status)
-    get_pid
-    if start-stop-daemon -T --pid $CHILDPID; then
-      log_success_msg "Patroni is running\n"
-      exit 0
-    else
-      log_warning_msg "Patroni in not running\n"
-    fi
-    ;;
-
-  restart)
-    $0 stop
-    $0 start
-    ;;
-
-  *)
-    echo "Usage: /etc/init.d/$NAME {start|stop|restart|reload|status}"
-    exit 1
-    ;;
-esac
-
-if [ $? -eq 0 ]; then
-        echo .
-        exit 0
-else
-        echo " failed"
-        exit 1
-fi
+#!/bin/sh
+#
+### BEGIN INIT INFO
+# Provides:          patroni
+# Required-Start:    $remote_fs $syslog
+# Required-Stop:     $remote_fs $syslog
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: Patroni init script
+# Description:       Runners to orchestrate a high-availability PostgreSQL
+### END INIT INFO
+
+### BEGIN USER CONFIGURATION
+
+CONF="/etc/patroni/postgres.yml"
+LOGFILE="/var/log/patroni.log"
+USER="postgres"
+GROUP="postgres"
+
+NAME=patroni
+PATRONI="/opt/patroni/$NAME.py"
+PIDFILE="/var/run/$NAME.pid"
+
+# Set this parameter, if you have several Postgres versions installed
+# POSTGRES_VERSION="9.4"
+POSTGRES_VERSION=""
+
+### END USER CONFIGURATION
+
+. /lib/lsb/init-functions
+
+# Loading this library for get_versions() function
+if test ! -e /usr/share/postgresql-common/init.d-functions; then
+  log_failure_msg "Probably postgresql-common does not installed."
+  exit 1
+else
+  . /usr/share/postgresql-common/init.d-functions
+fi
+
+# Is there Patroni executable?
+if test ! -e $PATRONI; then
+  log_failure_msg "Patroni executable $PATRONI does not exist."
+  exit 1
+fi
+
+# Is there Patroni configuration file?
+if test ! -e $CONF; then
+  log_failure_msg "Patroni configuration file $CONF does not exist."
+  exit 1
+fi
+
+# Create logfile if doesn't exist
+if test ! -e $LOGFILE; then
+  log_action_msg "Creating logfile for Patroni..."
+  touch $LOGFILE
+  chown $USER:$GROUP $LOGFILE
+fi
+
+prepare_pgpath() {
+  if [ "$POSTGRES_VERSION" != "" ]; then
+    if [ -x /usr/lib/postgresql/$POSTGRES_VERSION/bin/pg_ctl ]; then
+      PGPATH="/usr/lib/postgresql/$POSTGRES_VERSION/bin"
+    else
+      log_failure_msg "Postgres version incorrect, check POSTGRES_VERSION variable."
+      exit 0
+    fi
+  else
+    get_versions
+    if echo $versions | grep -q -e "\s"; then
+      log_warning_msg "You have several Postgres versions installed. Please, use POSTGRES_VERSION to define correct environment."
+    else
+      versions=`echo $versions | sed -e 's/^[ \t]*//'`
+      PGPATH="/usr/lib/postgresql/$versions/bin"
+    fi
+  fi
+}
+
+get_pid() {
+  if test -e $PIDFILE; then
+    PID=`cat $PIDFILE`
+    CHILDPID=`ps --ppid $PID -o %p --no-headers`
+  else
+    log_failure_msg "Could not find PID file. Patroni probably down."
+    exit 1
+  fi
+}
+
+
+case "$1" in
+  start)
+    prepare_pgpath
+    PGPATH=$PATH:$PGPATH
+    log_success_msg "Starting Patroni\n"
+    exec start-stop-daemon --start --quiet \
+        --background \
+        --pidfile $PIDFILE --make-pidfile \
+        --chuid $USER:$GROUP \
+        --chdir `eval echo ~$USER` \
+        --exec $PATRONI \
+        --startas /bin/sh -- \
+        -c "/usr/bin/env PATH=$PGPATH /usr/bin/python $PATRONI $CONF >> $LOGFILE 2>&1"
+    ;;
+
+  stop)
+    log_success_msg "Stopping Patroni"
+    get_pid
+    start-stop-daemon --stop --pid $CHILDPID
+    start-stop-daemon --stop --pidfile $PIDFILE --remove-pidfile --quiet
+    ;;
+
+  reload)
+    log_success_msg "Reloading Patroni configuration"
+    get_pid
+    kill -HUP $CHILDPID
+    ;;
+
+  status)
+    get_pid
+    if start-stop-daemon -T --pid $CHILDPID; then
+      log_success_msg "Patroni is running\n"
+      exit 0
+    else
+      log_warning_msg "Patroni in not running\n"
+    fi
+    ;;
+
+  restart)
+    $0 stop
+    $0 start
+    ;;
+
+  *)
+    echo "Usage: /etc/init.d/$NAME {start|stop|restart|reload|status}"
+    exit 1
+    ;;
+esac
+
+if [ $? -eq 0 ]; then
+        echo .
+        exit 0
+else
+        echo " failed"
+        exit 1
+fi

features/environment.py

@@ -228,7 +228,7 @@ class PatroniController(AbstractController):
             if not os.path.exists(pidfile):
                 return None
             return int(open(pidfile).readline().strip())
-        except:
+        except Exception:
             return None
 
     def database_is_running(self):

features/steps/watchdog.py

@@ -31,7 +31,7 @@ def watchdog_was_closed(context, name):
     assert context.pctl.get_watchdog(name).was_closed
 
 
-@step('I wait for next {name:w} watchdog ping')
+@step('I reset {name:w} watchdog state')
 def watchdog_reset_pinged(context, name):
     context.pctl.get_watchdog(name).reset()
 

features/watchdog.feature

@@ -14,7 +14,8 @@ Feature: watchdog
     Then postgres0 watchdog has been closed
 
   Scenario: watchdog is opened and pinged after resume
-    Given I run patronictl.py resume batman
+    Given I reset postgres0 watchdog state
+    And I run patronictl.py resume batman
     Then I receive a response returncode 0
     And postgres0 watchdog has been pinged after 10 seconds
 
@@ -23,7 +24,8 @@ Feature: watchdog
     Then postgres0 watchdog has been closed
 
   Scenario: watchdog is triggered if patroni stops responding
-    Given I start postgres0 with watchdog
+    Given I reset postgres0 watchdog state
+    And I start postgres0 with watchdog
     Then postgres0 role is the primary after 10 seconds
     When postgres0 hangs for 30 seconds
     Then postgres0 watchdog is triggered after 30 seconds

patroni/__init__.py

@@ -134,7 +134,8 @@ class Patroni(object):
 
 
 def patroni_main():
-    logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.INFO)
+    logformat = os.environ.get('PATRONI_LOGFORMAT', '%(asctime)s %(levelname)s: %(message)s')
+    logging.basicConfig(format=logformat, level=logging.INFO)
     logging.getLogger('requests').setLevel(logging.WARNING)
 
     patroni = Patroni()

patroni/async_executor.py

@@ -85,7 +85,7 @@ class AsyncExecutor(object):
             # if the func returned something (not None) - wake up main HA loop
             wakeup = func(*args) if args else func()
             return wakeup
-        except:
+        except Exception:
             logger.exception('Exception during execution of long running task %s', self.scheduled_action)
         finally:
             with self:

patroni/config.py

@@ -243,11 +243,11 @@ class Config(object):
                 if name and suffix:
                     # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
                     if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY',
-                                  'VERIFY', 'TOKEN') and '_' not in name:
+                                  'VERIFY', 'TOKEN', 'CHECKS', 'DC') and '_' not in name:
                         value = os.environ.pop(param)
                         if suffix == 'PORT':
                             value = value and parse_int(value)
-                        elif suffix == 'HOSTS':
+                        elif suffix in ('HOSTS', 'CHECKS'):
                             value = value and _parse_list(value)
                         if value:
                             ret[name.lower()][suffix.lower()] = value

patroni/dcs/__init__.py

@@ -424,7 +424,7 @@ class AbstractDCS(object):
         with self._cluster_thread_lock:
             try:
                 self._load_cluster()
-            except:
+            except Exception:
                 self._cluster = None
                 raise
             return self._cluster

patroni/dcs/consul.py

@@ -25,9 +25,14 @@ class ConsulInternalError(ConsulException):
     """An internal Consul server error occurred"""
 
 
+class InvalidSessionTTL(ConsulInternalError):
+    """Session TTL is too small or too big"""
+
+
 class HTTPClient(object):
 
-    def __init__(self, host='127.0.0.1', port=8500, scheme='http', verify=True, cert=None, ca_cert=None):
+    def __init__(self, host='127.0.0.1', port=8500, token=None, scheme='http', verify=True, cert=None, ca_cert=None):
+        self.token = token
         self._read_timeout = 10
         self.base_uri = '{0}://{1}:{2}'.format(scheme, host, port)
         kwargs = {}
@@ -49,6 +54,10 @@ class HTTPClient(object):
     def set_read_timeout(self, timeout):
         self._read_timeout = timeout/3.0
 
+    @property
+    def ttl(self):
+        return self._ttl
+
     def set_ttl(self, ttl):
         ret = self._ttl != ttl
         self._ttl = ttl
@@ -58,7 +67,11 @@ class HTTPClient(object):
     def response(response):
         data = response.data.decode('utf-8')
         if response.status == 500:
-            raise ConsulInternalError('{0} {1}'.format(response.status, data))
+            msg = '{0} {1}'.format(response.status, data)
+            if data.startswith('Invalid Session TTL'):
+                raise InvalidSessionTTL(msg)
+            else:
+                raise ConsulInternalError(msg)
         return base.Response(response.status, response.headers, data)
 
     def uri(self, path, params=None):
@@ -82,8 +95,9 @@ class HTTPClient(object):
                 kwargs['timeout'] = (float(params['wait'][:-1]) if 'wait' in params else 300) + 1
             else:
                 kwargs['timeout'] = self._read_timeout
-            if isinstance(params, dict) and 'token' in params and params['token']:
-                kwargs['headers'] = {'X-Consul-Token': params.pop('token')}
+            token = params.pop('token', self.token) if isinstance(params, dict) else self.token
+            if token:
+                kwargs['headers'] = {'X-Consul-Token': token}
             return callback(self.response(self.http.request(method.upper(), self.uri(path, params), **kwargs)))
         return wrapper
 
@@ -93,6 +107,7 @@ class ConsulClient(base.Consul):
     def __init__(self, *args, **kwargs):
         self._cert = kwargs.pop('cert', None)
         self._ca_cert = kwargs.pop('ca_cert', None)
+        self._token = kwargs.get('token')
         super(ConsulClient, self).__init__(*args, **kwargs)
 
     def connect(self, *args, **kwargs):
@@ -101,6 +116,8 @@ class ConsulClient(base.Consul):
             kwargs['cert'] = self._cert
         if self._ca_cert:
             kwargs['ca_cert'] = self._ca_cert
+        if self._token:
+            kwargs['token'] = self._token
         return HTTPClient(**kwargs)
 
 
@@ -141,7 +158,8 @@ class Consul(AbstractDCS):
         if config.get('key') and config.get('cert'):
             config['cert'] = (config['cert'], config['key'])
 
-        kwargs = {p: config.get(p) for p in ('host', 'port', 'token', 'scheme', 'cert', 'ca_cert') if config.get(p)}
+        config_keys = ('host', 'port', 'token', 'scheme', 'cert', 'ca_cert', 'dc')
+        kwargs = {p: config.get(p) for p in config_keys if config.get(p)}
 
         verify = config.get('verify')
         if not isinstance(verify, bool):
@@ -153,6 +171,7 @@ class Consul(AbstractDCS):
         self.set_retry_timeout(config['retry_timeout'])
         self.set_ttl(config.get('ttl') or 30)
         self._last_session_refresh = 0
+        self.__session_checks = config.get('checks')
         if not self._ctl:
             self.create_session()
 
@@ -176,6 +195,15 @@ class Consul(AbstractDCS):
         self._retry.deadline = retry_timeout
         self._client.http.set_read_timeout(retry_timeout)
 
+    def adjust_ttl(self):
+        try:
+            settings = self._client.agent.self()
+            min_ttl = (settings['Config']['SessionTTLMin'] or 10000000000)/1000000000.0
+            logger.warning('Changing Session TTL from %s to %s', self._client.http.ttl, min_ttl)
+            self._client.http.set_ttl(min_ttl)
+        except Exception:
+            logger.exception('adjust_ttl')
+
     def _do_refresh_session(self):
         """:returns: `!True` if it had to create new session"""
         if self._session and self._last_session_refresh + self._loop_wait > time.time():
@@ -188,8 +216,15 @@ class Consul(AbstractDCS):
                 self._session = None
         ret = not self._session
         if ret:
-            self._session = self._client.session.create(name=self._scope + '-' + self._name,
-                                                        lock_delay=0.001, behavior='delete')
+            try:
+                self._session = self._client.session.create(name=self._scope + '-' + self._name,
+                                                            checks=self.__session_checks,
+                                                            lock_delay=0.001, behavior='delete')
+            except InvalidSessionTTL:
+                logger.exception('session.create')
+                self.adjust_ttl()
+                raise
+
         self._last_session_refresh = time.time()
         return ret
 
@@ -260,14 +295,14 @@ class Consul(AbstractDCS):
             self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover, sync)
         except NotFound:
             self._cluster = Cluster(None, None, None, None, [], None, None)
-        except:
+        except Exception:
             logger.exception('get_cluster')
             raise ConsulError('Consul is not responding properly')
 
-    def touch_member(self, data, **kwargs):
+    def touch_member(self, data, ttl=None, permanent=False):
         cluster = self.cluster
         member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
-        create_member = self.refresh_session()
+        create_member = not permanent and self.refresh_session()
 
         if member and (create_member or member.session != self._session):
             try:
@@ -280,7 +315,7 @@ class Consul(AbstractDCS):
             return True
 
         try:
-            args = {} if kwargs.get('permanent', False) else {'acquire': self._session}
+            args = {} if permanent else {'acquire': self._session}
             self._client.kv.put(self.member_path, data, **args)
             self._my_member_data = data
             return True

patroni/dcs/zookeeper.py

@@ -206,7 +206,7 @@ class ZooKeeper(AbstractDCS):
         try:
             self._client.retry(self._client.create, path, value.encode('utf-8'), **kwargs)
             return True
-        except:
+        except Exception:
             return False
 
     def attempt_to_acquire_leader(self, permanent=False):
@@ -221,7 +221,7 @@ class ZooKeeper(AbstractDCS):
             return True
         except NoNodeError:
             return value == '' or (index is None and self._create(self.failover_path, value))
-        except:
+        except Exception:
             logging.exception('set_failover_value')
             return False
 
@@ -248,7 +248,7 @@ class ZooKeeper(AbstractDCS):
                 self._client.delete_async(self.member_path).get(timeout=1)
             except NoNodeError:
                 pass
-            except:
+            except Exception:
                 return False
             member = None
 
@@ -268,7 +268,7 @@ class ZooKeeper(AbstractDCS):
             self._client.set_async(self.member_path, data).get(timeout=1)
             self._my_member_data = data
             return True
-        except:
+        except Exception:
             logger.exception('touch_member')
 
         return False
@@ -285,9 +285,9 @@ class ZooKeeper(AbstractDCS):
             try:
                 self._client.create_async(self.leader_optime_path, last_operation, makepath=True).get(timeout=1)
                 return True
-            except:
+            except Exception:
                 logger.exception('Failed to create %s', self.leader_optime_path)
-        except:
+        except Exception:
             logger.exception('Failed to update %s', self.leader_optime_path)
         return False
 
@@ -307,7 +307,7 @@ class ZooKeeper(AbstractDCS):
     def cancel_initialization(self):
         try:
             self._client.retry(self._cancel_initialization)
-        except:
+        except Exception:
             logger.exception("Unable to delete initialize key")
 
     def delete_cluster(self):
@@ -322,7 +322,7 @@ class ZooKeeper(AbstractDCS):
             return True
         except NoNodeError:
             return value == '' or (index is None and self._create(self.sync_path, value))
-        except:
+        except Exception:
             logging.exception('set_sync_state_value')
             return False
 

patroni/ha.py

@@ -59,6 +59,7 @@ class Ha(object):
         self.old_cluster = None
         self.recovering = False
         self._post_bootstrap_task = None
+        self._crash_recovery_executed = False
         self._start_timeout = None
         self._async_executor = AsyncExecutor(self.wakeup)
         self.watchdog = patroni.watchdog
@@ -91,7 +92,7 @@ class Ha(object):
             if write_leader_optime:
                 try:
                     self.dcs.write_leader_optime(self.state_handler.last_operation())
-                except:
+                except Exception:
                     pass
         return ret
 
@@ -124,7 +125,7 @@ class Ha(object):
             if not self._async_executor.busy and data['state'] in ['running', 'restarting', 'starting']:
                 try:
                     data['xlog_location'] = self.state_handler.wal_position(retry=False)
-                except:
+                except Exception:
                     pass
             if self.patroni.scheduled_restart:
                 scheduled_restart_data = self.patroni.scheduled_restart.copy()
@@ -175,6 +176,11 @@ class Ha(object):
             self._async_executor.run_async(self.state_handler.rewind, (self.cluster.leader,))
             return True
 
+    def _start_crash_recovery(self, msg):
+        self._async_executor.schedule(msg)
+        self._async_executor.run_async(self.state_handler.fix_cluster_state)
+        return msg
+
     def recover(self):
         # Postgres is not running and we will restart in standby mode. Watchdog is not needed until we promote.
         self.watchdog.disable()
@@ -194,6 +200,12 @@ class Ha(object):
         else:
             timeout = None
 
+        data = self.state_handler.controldata()
+        if data.get('Database cluster state') == 'in production' and not self._crash_recovery_executed and \
+                (self.cluster.is_unlocked() or self.state_handler.can_rewind):
+            self._crash_recovery_executed = True
+            return self._start_crash_recovery('doing crash recovery in a single user mode')
+
         self.load_cluster_from_dcs()
 
         if self.has_lock():
@@ -207,6 +219,13 @@ class Ha(object):
             msg = "starting as a secondary"
             node_to_follow = self._get_node_to_follow(self.cluster)
 
+        # once we already tried to start postgres but failed, single user mode is a rescue in this case
+        if self.recovering and not self.state_handler.rewind_executed \
+                and not self._crash_recovery_executed and self.state_handler.can_rewind \
+                and data.get('Database cluster state') not in ('shut down', 'shut down in recovery'):
+            self.recovering = False
+            return self._start_crash_recovery('fixing cluster state in a single user mode')
+
         self.recovering = True
 
         self._async_executor.schedule('restarting after failure')
@@ -886,6 +905,7 @@ class Ha(object):
                 self.dcs.reset_cluster()
                 return 'removed leader key after trying and failing to start postgres'
             return 'failed to start postgres'
+        self._crash_recovery_executed = False
         return None
 
     def cancel_initialization(self):
@@ -1004,6 +1024,8 @@ class Ha(object):
 
             # is data directory empty?
             if self.state_handler.data_directory_empty():
+                self.state_handler.set_role('uninitialized')
+                self.state_handler.stop()
                 # In case datadir went away while we were master. TODO: check for this and try to stop postgresql.
                 self.watchdog.disable()
 

patroni/postgresql.py

@@ -17,7 +17,7 @@ from contextlib import contextmanager
 from patroni import call_self
 from patroni.callback_executor import CallbackExecutor
 from patroni.exceptions import PostgresConnectionException
-from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop
+from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop, int_or_none
 from six import string_types
 from six.moves.urllib.parse import quote_plus
 from threading import current_thread, Lock
@@ -71,6 +71,25 @@ def null_context():
     yield
 
 
+def _update_postmaster_cached_info(func):
+    def wrapper(self):
+        ret = func(self)
+        if ret and 'pid' in ret and 'start_time' in ret:
+            old_pid = self._postmaster_cached_info.get('pid', 0)
+            old_start_time = self._postmaster_cached_info.get('start_time', 0)
+            try:
+                pmpid = int(ret['pid'])
+                pmstart = int(ret['start_time'])
+                if pmpid != old_pid or pmstart != old_start_time:  # this check removes repeating messages from logs
+                    self._postmaster_cached_info = {'pid': pmpid, 'start_time': pmstart}
+                    logger.info("Updated postmaster info: %s .", self._postmaster_cached_info)
+            except ValueError:
+                logger.warning('Cannot update postmaster info with data due garbage in pid file: %s', ret)
+        return ret
+
+    return wrapper
+
+
 class Postgresql(object):
 
     # List of parameters which must be always passed to postmaster as command line options
@@ -142,6 +161,7 @@ class Postgresql(object):
         self._pg_hba_conf = os.path.join(self._config_dir, 'pg_hba.conf')
         self._recovery_conf = os.path.join(self._data_dir, 'recovery.conf')
         self._postmaster_pid = os.path.join(self._data_dir, 'postmaster.pid')
+        self._postmaster_cached_info = {'pid': 0, 'start_time': 0}
         self._trigger_file = config.get('recovery_conf', {}).get('trigger_file') or 'promote'
         self._trigger_file = os.path.abspath(os.path.join(self._data_dir, self._trigger_file))
 
@@ -707,8 +727,12 @@ class Postgresql(object):
         if not (self._version_file_exists() and os.path.isfile(self._postmaster_pid)):
             # XXX: This is dangerous in case somebody deletes the data directory while PostgreSQL is still running.
             return False
-        return self.is_pid_running(self.get_pid())
 
+        pidfile = self.read_pid_file()
+        return self._is_postmaster_pid_running(int_or_none(pidfile.get('pid')),
+                                               start_time=int_or_none(pidfile.get('start_time')))
+
+    @_update_postmaster_cached_info
     def read_pid_file(self):
         """Reads and parses postmaster.pid from the data directory
 
@@ -733,14 +757,48 @@ class Postgresql(object):
             logger.warning("Garbage pid in postmaster.pid: {0!r}".format(pid))
             return 0
 
-    @staticmethod
-    def is_pid_running(pid):
+    def get_pid_with_lost_data_dir(self):
+        logger.info("Trying to check if process running without directory "
+                    "with cached postmaster info: %s .", self._postmaster_cached_info)
         try:
-            if pid < 0:
-                pid = -pid
-            return pid > 0 and pid != os.getpid() and pid != os.getppid() and (os.kill(pid, 0) or True)
-        except Exception:
+            process = psutil.Process(self._postmaster_cached_info['pid'])
+            # check difference instead of values because of rounding issues
+            if abs(self._postmaster_cached_info["start_time"] - process.create_time()) < 2:
+                return process.pid
+            else:
+                logger.info("Process with pid %s was started at different time %s .",
+                            process.pid, process.create_time())
+        except psutil.NoSuchProcess:
+            logger.info("Cannot find process %s .", self._postmaster_cached_info['pid'])
+        return 0
+
+    def clean_postmaster_cached_info(self):
+        self._postmaster_cached_info = {'pid': 0, 'start_time': 0}
+        logger.info("postmaster info was cleaned.")
+
+    @staticmethod
+    def _is_postmaster_pid_running(pid, start_time=None):
+        # Normalize pid handling missing values and negative pids from postmaster.pid
+        if not pid:
             return False
+        if pid < 0:
+            pid = -pid
+
+        try:
+            proc = psutil.Process(pid)
+        except psutil.NoSuchProcess:
+            return False
+
+        # If the process is Patroni or Patronis host process or Patronis child process then it's a false positive
+        my_pid = os.getpid()
+        if pid == my_pid or pid == os.getppid() or proc.parent() == my_pid:
+            return False
+
+        # If process start time differs by more than 3 seconds it's a false positive
+        if start_time is not None and abs(proc.create_time() - start_time) > 3:
+            return False
+
+        return True
 
     @property
     def cb_called(self):
@@ -805,7 +863,7 @@ class Postgresql(object):
                     # Garbage in the pid file
                     pass
 
-            if not self.is_pid_running(pid):
+            if not self._is_postmaster_pid_running(pid, start_time=initiated):
                 logger.error('postmaster is not running')
                 self.set_state('start failed')
                 return False
@@ -933,6 +991,12 @@ class Postgresql(object):
 
     def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint):
         if not self.is_running():
+            if self.data_directory_empty() and self._postmaster_cached_info['pid']:
+                pid = self.get_pid_with_lost_data_dir()
+                if pid > 0:
+                    self.terminate_starting_postmaster(pid)
+                    self.clean_postmaster_cached_info()
+                    return True, True
             if on_safepoint:
                 on_safepoint()
             return True, False
@@ -958,13 +1022,14 @@ class Postgresql(object):
             on_safepoint()
 
         self._wait_for_postmaster_stop(pid)
+        self.clean_postmaster_cached_info()
 
         return True, True
 
     def _wait_for_postmaster_stop(self, pid):
         # This wait loop differs subtly from pg_ctl as we check for both the pid file going
         # away and if the pid is running. This seems safer.
-        while pid == self.get_pid() and self.is_pid_running(pid):
+        while pid == self.get_pid() and self._is_postmaster_pid_running(pid):
             time.sleep(STOP_POLLING_INTERVAL)
 
     def _signal_postmaster_stop(self, mode):
@@ -994,13 +1059,13 @@ class Postgresql(object):
                 return
             logger.warning("Could not send stop signal to PostgreSQL (error: {0})".format(e.errno))
 
-        while self.is_pid_running(pid):
+        while self._is_postmaster_pid_running(pid):
             time.sleep(STOP_POLLING_INTERVAL)
 
     def _wait_for_connection_close(self, pid):
         try:
             with self.connection().cursor() as cur:
-                while pid == self.get_pid() and self.is_pid_running(pid):  # Need a timeout here?
+                while pid == self.get_pid() and self._is_postmaster_pid_running(pid):  # Need a timeout here?
                     cur.execute("SELECT 1")
                     time.sleep(STOP_POLLING_INTERVAL)
         except psycopg2.Error:
@@ -1261,12 +1326,14 @@ class Postgresql(object):
         else:  # otherwise analyze pg_controldata output
             data = self.controldata()
             try:
+                if data.get('Database cluster state') == 'shut down in recovery':
+                    lsn = data.get('Minimum recovery ending location')
+                    timeline = int(data.get("Min recovery ending loc's timeline"))
+                    if lsn == '0/0' or timeline == 0:  # it was a master when it crashed
+                        data['Database cluster state'] = 'shut down'
                 if data.get('Database cluster state') == 'shut down':
                     lsn = data.get('Latest checkpoint location')
                     timeline = int(data.get("Latest checkpoint's TimeLineID"))
-                elif data.get('Database cluster state') == 'shut down in recovery':
-                    lsn = data.get('Minimum recovery ending location')
-                    timeline = int(data.get("Min recovery ending loc's timeline"))
             except (TypeError, ValueError):
                 logger.exception('Failed to get local timeline and lsn from pg_controldata output')
         logger.info('Local timeline=%s lsn=%s', timeline, lsn)
@@ -1744,3 +1811,57 @@ $$""".format(name, ' '.join(options)), name, password, password)
         90600
         """
         return Postgresql.postgres_version_to_int(pg_version + '.0')
+
+    def read_postmaster_opts(self):
+        """returns the list of option names/values from postgres.opts, Empty dict if read failed or no file"""
+        result = {}
+        try:
+            with open(os.path.join(self._data_dir, 'postmaster.opts')) as f:
+                data = f.read()
+                for opt in data.split('" "'):
+                    if '=' in opt and opt.startswith('--'):
+                        name, val = opt.split('=', 1)
+                        result[name.strip('-')] = val.rstrip('"\n')
+        except IOError:
+            logger.exception('Error when reading postmaster.opts')
+        return result
+
+    def single_user_mode(self, command=None, options=None):
+        """run a given command in a single-user mode. If the command is empty - then just start and stop"""
+        cmd = [self._pgcommand('postgres'), '--single', '-D', self._data_dir]
+        for opt, val in sorted((options or {}).items()):
+            cmd.extend(['-c', '{0}={1}'.format(opt, val)])
+        # need a database name to connect
+        cmd.append(self._database)
+        p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT)
+        if p:
+            if command:
+                p.communicate('{0}\n'.format(command))
+            p.stdin.close()
+            return p.wait()
+        return 1
+
+    def cleanup_archive_status(self):
+        status_dir = os.path.join(self._data_dir, 'pg_' + self.wal_name, 'archive_status')
+        try:
+            for f in os.listdir(status_dir):
+                path = os.path.join(status_dir, f)
+                try:
+                    if os.path.islink(path):
+                        os.unlink(path)
+                    elif os.path.isfile(path):
+                        os.remove(path)
+                except OSError:
+                    logger.exception('Unable to remove %s', path)
+        except OSError:
+            logger.exception('Unable to list %s', status_dir)
+
+    def fix_cluster_state(self):
+        self.cleanup_archive_status()
+
+        # Start in a single user mode and stop to produce a clean shutdown
+        opts = self.read_postmaster_opts()
+        opts.update({'archive_mode': 'on', 'archive_command': 'false'})
+        if os.path.isfile(self._recovery_conf) or os.path.islink(self._recovery_conf):
+            os.unlink(self._recovery_conf)
+        return self.single_user_mode(options=opts) == 0 or None

patroni/utils.py

@@ -95,17 +95,17 @@ def strtol(value, strict=True):
     True
     """
     value = str(value).strip()
-    l = len(value)
+    ln = len(value)
     i = 0
     # skip sign:
-    if i < l and value[i] in ('-', '+'):
+    if i < ln and value[i] in ('-', '+'):
         i += 1
 
     # we always expect to get digit in the beginning
-    if i < l and value[i].isdigit():
+    if i < ln and value[i].isdigit():
         if value[i] == '0':
             i += 1
-            if i < l and value[i] in ('x', 'X'):  # '0' followed by 'x': HEX
+            if i < ln and value[i] in ('x', 'X'):  # '0' followed by 'x': HEX
                 base = 16
                 i += 1
             else:  # just starts with '0': OCT
@@ -114,7 +114,7 @@ def strtol(value, strict=True):
             base = 10
 
         ret = None
-        while i <= l:
+        while i <= ln:
             try:  # try to find maximally long number
                 i += 1  # by giving to `int` longer and longer strings
                 ret = int(value[:i], base)
@@ -280,3 +280,11 @@ def polling_loop(timeout, interval=1):
         yield iteration
         iteration += 1
         time.sleep(interval)
+
+
+def int_or_none(val):
+    """Returns integer value of the parameter if convertible to int, None otherwise."""
+    try:
+        return int(val)
+    except (ValueError, TypeError):
+        return None

patroni/version.py

@@ -1 +1 @@
-__version__ = '1.3.4'
+__version__ = '1.3.6'

patroni/watchdog/base.py

@@ -133,6 +133,7 @@ class Watchdog(object):
 
         try:
             self.impl.open()
+            actual_timeout = self._set_timeout()
         except WatchdogError as e:
             logger.warning("Could not activate %s: %s", self.impl.describe(), e)
             self.impl = NullWatchdog()
@@ -141,8 +142,6 @@ class Watchdog(object):
             logger.warning("Watchdog implementation can't be disabled."
                            " Watchdog will trigger after Patroni loses leader key.")
 
-        actual_timeout = self._set_timeout()
-
         if not self.impl.is_running or actual_timeout > self.config.timeout:
             if self.config.mode == MODE_REQUIRED:
                 if self.impl.is_null:

patroni/watchdog/linux.py

@@ -155,21 +155,25 @@ class LinuxWatchdogDevice(WatchdogBase):
     def can_be_disabled(self):
         return self.get_support().has_MAGICCLOSE
 
-    def _ioctl(self, func, arg, mutate_arg=False):
+    def _ioctl(self, func, arg):
+        """Runs the specified ioctl on the underlying fd.
+
+        Raises WatchdogError if the device is closed.
+        Raises OSError or IOError (Python 2) when the ioctl fails."""
         if self._fd is None:
             raise WatchdogError("Watchdog device is closed")
-
-        result = fcntl.ioctl(self._fd, func, arg, mutate_arg)
-        if result < 0:
-            raise IOError(result)
+        fcntl.ioctl(self._fd, func, arg, True)
 
     def get_support(self):
         if self._support_cache is None:
             info = watchdog_info()
-            self._ioctl(WDIOC_GETSUPPORT, info, True)
+            try:
+                self._ioctl(WDIOC_GETSUPPORT, info)
+            except (WatchdogError, OSError, IOError) as e:
+                raise WatchdogError("Could not get information about watchdog device: {}".format(e))
             self._support_cache = WatchdogInfo(info.options,
                                                info.firmware_version,
-                                               str(bytearray(info.identity)).rstrip('\x00'))
+                                               bytearray(info.identity).decode(errors='ignore').rstrip('\x00'))
         return self._support_cache
 
     def describe(self):
@@ -180,7 +184,7 @@ class LinuxWatchdogDevice(WatchdogBase):
             try:
                 _, version, identity = self.get_support()
                 ver_str = " (firmware {0})".format(version) if version else ""
-            except WatchdogError:  # XXX: Can it really be raise when self._fd is not None?
+            except WatchdogError:
                 pass
 
         return identity + ver_str + dev_str
@@ -199,11 +203,17 @@ class LinuxWatchdogDevice(WatchdogBase):
         timeout = int(timeout)
         if not 0 < timeout < 0xFFFF:
             raise WatchdogError("Invalid timeout {0}. Supported values are between 1 and 65535".format(timeout))
-        self._ioctl(WDIOC_SETTIMEOUT, ctypes.c_int(timeout))
+        try:
+            self._ioctl(WDIOC_SETTIMEOUT, ctypes.c_int(timeout))
+        except (WatchdogError, OSError, IOError) as e:
+            raise WatchdogError("Could not set timeout on watchdog device: {}".format(e))
 
     def get_timeout(self):
         timeout = ctypes.c_int()
-        self._ioctl(WDIOC_GETTIMEOUT, timeout, True)
+        try:
+            self._ioctl(WDIOC_GETTIMEOUT, timeout)
+        except (WatchdogError, OSError, IOError) as e:
+            raise WatchdogError("Could not get timeout on watchdog device: {}".format(e))
         return timeout.value
 
 

setup.py

@@ -87,7 +87,7 @@ class PyTest(TestCommand):
     def run_tests(self):
         try:
             import pytest
-        except:
+        except Exception:
             raise RuntimeError('py.test is not installed, run: pip install pytest')
         params = {'args': self.test_args}
         if self.cov:

tests/test_consul.py

@@ -3,7 +3,8 @@ import unittest
 
 from consul import ConsulException, NotFound
 from mock import Mock, patch
-from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, ConsulError, HTTPClient
+from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, \
+                                ConsulError, HTTPClient, InvalidSessionTTL
 from test_etcd import SleepException
 
 
@@ -47,7 +48,10 @@ class TestHTTPClient(unittest.TestCase):
         self.client.get(Mock(), '')
         self.client.get(Mock(), '', {'wait': '1s', 'index': 1, 'token': 'foo'})
         self.client.http.request.return_value.status = 500
+        self.client.http.request.return_value.data = b'Foo'
         self.assertRaises(ConsulInternalError, self.client.get, Mock(), '')
+        self.client.http.request.return_value.data = b"Invalid Session TTL '3000000000', must be between [10s=24h0m0s]"
+        self.assertRaises(InvalidSessionTTL, self.client.get, Mock(), '')
 
     def test_unknown_method(self):
         try:
@@ -70,7 +74,7 @@ class TestConsul(unittest.TestCase):
     @patch.object(consul.Consul.KV, 'delete', Mock())
     def setUp(self):
         Consul({'ttl': 30, 'scope': 't', 'name': 'p', 'url': 'https://l:1', 'retry_timeout': 10,
-                'verify': 'on', 'key': 'foo', 'cert': 'bar', 'cacert': 'buz'})
+                'verify': 'on', 'key': 'foo', 'cert': 'bar', 'cacert': 'buz', 'token': 'asd', 'dc': 'dc1'})
         Consul({'ttl': 30, 'scope': 't', 'name': 'p', 'url': 'https://l:1', 'retry_timeout': 10,
                 'verify': 'on', 'cert': 'bar', 'cacert': 'buz'})
         self.c = Consul({'ttl': 30, 'scope': 'test', 'name': 'postgresql1', 'host': 'localhost:1', 'retry_timeout': 10})
@@ -84,7 +88,9 @@ class TestConsul(unittest.TestCase):
         self.assertRaises(SleepException, self.c.create_session)
 
     @patch.object(consul.Consul.Session, 'renew', Mock(side_effect=NotFound))
-    @patch.object(consul.Consul.Session, 'create', Mock(side_effect=ConsulException))
+    @patch.object(consul.Consul.Session, 'create', Mock(side_effect=[InvalidSessionTTL, ConsulException]))
+    @patch.object(consul.Consul.Agent, 'self', Mock(return_value={'Config': {'SessionTTLMin': 0}}))
+    @patch.object(HTTPClient, 'set_ttl', Mock(side_effect=ValueError))
     def test_referesh_session(self):
         self.c._session = '1'
         self.assertFalse(self.c.refresh_session())

tests/test_ha.py

@@ -35,7 +35,7 @@ def get_cluster_not_initialized_without_leader():
 def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None):
     m1 = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres',
                                   'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4})
-    l = Leader(0, 0, m1) if leader else None
+    leader = Leader(0, 0, m1) if leader else None
     m2 = Member(0, 'other', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres',
                                  'api_url': 'http://127.0.0.1:8011/patroni',
                                  'state': 'running',
@@ -43,7 +43,7 @@ def get_cluster_initialized_without_leader(leader=False, failover=None, sync=Non
                                  'scheduled_restart': {'schedule': "2100-01-01 10:53:07.560445+00:00",
                                                        'postgres_version': '99.0.0'}})
     syncstate = SyncState(0 if sync else None, sync and sync[0], sync and sync[1])
-    return get_cluster(True, l, [m1, m2], failover, syncstate)
+    return get_cluster(True, leader, [m1, m2], failover, syncstate)
 
 
 def get_cluster_initialized_with_leader(failover=None, sync=None):
@@ -51,8 +51,8 @@ def get_cluster_initialized_with_leader(failover=None, sync=None):
 
 
 def get_cluster_initialized_with_only_leader(failover=None):
-    l = get_cluster_initialized_without_leader(leader=True, failover=failover).leader
-    return get_cluster(True, l, [l], failover, None)
+    leader = get_cluster_initialized_without_leader(leader=True, failover=failover).leader
+    return get_cluster(True, leader, [leader], failover, None)
 
 
 def get_node_status(reachable=True, in_recovery=True, wal_position=10, nofailover=False, watchdog_failed=False):
@@ -136,7 +136,7 @@ class TestHa(unittest.TestCase):
 
     @patch('socket.getaddrinfo', socket_getaddrinfo)
     @patch('psycopg2.connect', psycopg2_connect)
-    @patch('patroni.dcs.dcs_modules', Mock(return_value=['foo', 'patroni.dcs.etcd']))
+    @patch('patroni.dcs.dcs_modules', Mock(return_value=['patroni.dcs.foo', 'patroni.dcs.etcd']))
     @patch.object(etcd.Client, 'read', etcd_read)
     def setUp(self):
         with patch.object(Client, 'machines') as mock_machines:
@@ -172,27 +172,42 @@ class TestHa(unittest.TestCase):
         self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')
 
     def test_recover_replica_failed(self):
-        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery'}
         self.p.is_running = false
         self.p.follow = false
         self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')
         self.assertEquals(self.ha.run_cycle(), 'failed to start postgres')
 
-    def test_recover_master_failed(self):
+    def test_recover_former_master(self):
         self.p.follow = false
         self.p.is_running = false
         self.p.name = 'leader'
         self.p.set_role('master')
-        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.p.controldata = lambda: {'Database cluster state': 'shut down'}
         self.ha.cluster = get_cluster_initialized_with_leader()
         self.assertEquals(self.ha.run_cycle(), 'starting as readonly because i had the session lock')
 
+    @patch.object(Postgresql, 'fix_cluster_state', Mock())
+    def test_crash_recovery(self):
+        self.p.is_running = false
+        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.assertEquals(self.ha.run_cycle(), 'doing crash recovery in a single user mode')
+
     @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
     def test_recover_with_rewind(self):
         self.p.is_running = false
         self.ha.cluster = get_cluster_initialized_with_leader()
         self.assertEquals(self.ha.run_cycle(), 'running pg_rewind from leader')
 
+    @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
+    @patch.object(Postgresql, 'fix_cluster_state', Mock())
+    def test_single_user_after_recover_failed(self):
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery'}
+        self.p.is_running = false
+        self.p.follow = false
+        self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')
+        self.assertEquals(self.ha.run_cycle(), 'fixing cluster state in a single user mode')
+
     @patch('sys.exit', return_value=1)
     @patch('patroni.ha.Ha.sysid_valid', MagicMock(return_value=True))
     def test_sysid_no_match(self, exit_mock):
@@ -853,6 +868,7 @@ class TestHa(unittest.TestCase):
         self.ha.has_lock = true
         self.p.data_directory_empty = true
         self.assertEquals(self.ha.run_cycle(), 'released leader key voluntarily as data dir empty and currently leader')
+        self.assertEquals(self.p.role, 'uninitialized')
 
         # as has_lock is mocked out, we need to fake the leader key release
         self.ha.has_lock = false

tests/test_patroni.py

@@ -104,6 +104,7 @@ class TestPatroni(unittest.TestCase):
     @patch('patroni.config.Config.save_cache', Mock())
     @patch('patroni.config.Config.reload_local_configuration', Mock(return_value=True))
     @patch.object(Postgresql, 'state', PropertyMock(return_value='running'))
+    @patch.object(Postgresql, 'data_directory_empty', Mock(return_value=False))
     def test_run(self):
         self.p.postgresql.set_role('replica')
         self.p.sighup_handler()

tests/test_postgresql.py

@@ -2,6 +2,7 @@ import errno
 import mock  # for the mock.call method, importing it without a namespace breaks python3
 import os
 import psycopg2
+import psutil
 import shutil
 import subprocess
 import unittest
@@ -174,7 +175,7 @@ class TestPostgresql(unittest.TestCase):
         if not os.path.exists(self.data_dir):
             os.makedirs(self.data_dir)
         self.p = Postgresql({'name': 'test0', 'scope': 'batman', 'data_dir': self.data_dir,
-                             'config_dir': self.config_dir, 'retry_timeout': 10,
+                             'config_dir': self.config_dir, 'retry_timeout': 10, 'pgpass': '/tmp/pgpass0',
                              'listen': '127.0.0.2, 127.0.0.3:5432', 'connect_address': '127.0.0.2:5432',
                              'authentication': {'superuser': {'username': 'test', 'password': 'test'},
                                                 'replication': {'username': 'replicator', 'password': 'rep-pass'}},
@@ -238,17 +239,17 @@ class TestPostgresql(unittest.TestCase):
 
     @patch.object(Postgresql, 'pg_isready')
     @patch.object(Postgresql, 'read_pid_file')
-    @patch.object(Postgresql, 'is_pid_running')
+    @patch.object(Postgresql, '_is_postmaster_pid_running')
     @patch('patroni.postgresql.polling_loop', Mock(return_value=range(1)))
-    def test_wait_for_port_open(self, mock_is_pid_running, mock_read_pid_file, mock_pg_isready):
-        mock_is_pid_running.return_value = False
+    def test_wait_for_port_open(self, mock_is_postmaster_pid_running, mock_read_pid_file, mock_pg_isready):
+        mock_is_postmaster_pid_running.return_value = False
         mock_pg_isready.return_value = STATE_NO_RESPONSE
 
         # No pid file and postmaster death
         mock_read_pid_file.return_value = {}
         self.assertFalse(self.p.wait_for_port_open(42, 100., 1))
 
-        mock_is_pid_running.return_value = True
+        mock_is_postmaster_pid_running.return_value = True
 
         # timeout
         mock_read_pid_file.return_value = {'pid', 1}
@@ -276,6 +277,21 @@ class TestPostgresql(unittest.TestCase):
         mock_is_running.return_value = False
         self.assertTrue(self.p.stop(on_safepoint=mock_callback))
         mock_callback.assert_called()
+
+        with patch.object(Postgresql, '_is_postmaster_pid_running', Mock(return_value=False)), \
+                patch.object(Postgresql, 'data_directory_empty', Mock(return_value=True)):
+            with patch('psutil.Process') as mock_psutil:
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.return_value.pid = 1
+                mock_psutil.return_value.create_time.return_value = 1
+                self.assertTrue(self.p.stop())
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.return_value.create_time.return_value = 100
+                self.assertTrue(self.p.stop())
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.side_effect = psutil.NoSuchProcess('')
+                self.assertTrue(self.p.stop())
+
         mock_is_running.return_value = True
         mock_get_pid.return_value = 0
         mock_callback.reset_mock()
@@ -289,7 +305,7 @@ class TestPostgresql(unittest.TestCase):
             self.assertFalse(self.p.stop())
             self.assertTrue(self.p.stop())
         with patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None))):
-            with patch.object(Postgresql, 'is_pid_running', Mock(side_effect=[True, False, False])):
+            with patch.object(Postgresql, '_is_postmaster_pid_running', Mock(side_effect=[True, False, False])):
                 self.assertTrue(self.p.stop())
 
     def test_restart(self):
@@ -327,10 +343,10 @@ class TestPostgresql(unittest.TestCase):
     @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
     def test__get_local_timeline_lsn(self):
         self.p.trigger_check_diverged_lsn()
-        with patch.object(Postgresql, 'controldata', Mock(return_value={'Database cluster state': 'shut down'})):
-            self.p.rewind_needed_and_possible(self.leader)
         with patch.object(Postgresql, 'controldata',
-                          Mock(return_value={'Database cluster state': 'shut down in recovery'})):
+                          Mock(return_value={'Database cluster state': 'shut down in recovery',
+                                             'Minimum recovery ending location': '0/0',
+                                             "Min recovery ending loc's timeline": '0'})):
             self.p.rewind_needed_and_possible(self.leader)
         with patch.object(Postgresql, 'is_running', Mock(return_value=True)):
             with patch.object(MockCursor, 'fetchone', Mock(side_effect=[(False, ), Exception])):
@@ -772,12 +788,19 @@ class TestPostgresql(unittest.TestCase):
             os.remove(pidfile)
         self.assertEquals(self.p.read_pid_file(), {})
 
-    @patch('os.kill')
-    def test_is_pid_running(self, mock_kill):
-        mock_kill.return_value = True
-        self.assertTrue(self.p.is_pid_running(-100))
-        self.assertFalse(self.p.is_pid_running(0))
-        self.assertFalse(self.p.is_pid_running(None))
+    @patch.object(Postgresql, '_version_file_exists', Mock(return_value=True))
+    @patch('os.path.isfile', Mock(return_value=True))
+    @patch.object(Postgresql, 'read_pid_file')
+    @patch('psutil.Process')
+    def test_is_postmaster_pid_running(self, mock_psutil, mock_read_pid_file):
+        mock_psutil.return_value.create_time.return_value = 1
+        mock_read_pid_file.return_value = {'pid': -100, 'start_time': 1}
+        self.assertTrue(self.p.is_running())
+        with patch('os.getpid', Mock(return_value=100)):
+            mock_read_pid_file.return_value = {'pid': 100, 'start_time': 1}
+            self.assertFalse(self.p.is_running())
+        mock_read_pid_file.return_value = {'pid': 100, 'start_time': 100}
+        self.assertFalse(self.p.is_running())
 
     def test_pick_sync_standby(self):
         cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None,
@@ -855,20 +878,20 @@ class TestPostgresql(unittest.TestCase):
     @patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None)))
     @patch.object(Postgresql, 'get_pid', Mock(return_value=123))
     @patch('time.sleep', Mock())
-    @patch.object(Postgresql, 'is_pid_running')
-    def test__wait_for_connection_close(self, mock_is_pid_running):
-        mock_is_pid_running.side_effect = [True, False, False]
+    @patch.object(Postgresql, '_is_postmaster_pid_running')
+    def test__wait_for_connection_close(self, mock_is_postmaster_pid_running):
+        mock_is_postmaster_pid_running.side_effect = [True, False, False]
         mock_callback = Mock()
         self.p.stop(on_safepoint=mock_callback)
 
-        mock_is_pid_running.side_effect = [True, False, False]
+        mock_is_postmaster_pid_running.side_effect = [True, False, False]
         with patch.object(MockCursor, "execute", Mock(side_effect=psycopg2.Error)):
             self.p.stop(on_safepoint=mock_callback)
 
     @patch.object(Postgresql, 'is_running', Mock(return_value=True))
     @patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None)))
     @patch.object(Postgresql, 'get_pid', Mock(return_value=123))
-    @patch.object(Postgresql, 'is_pid_running', Mock(return_value=False))
+    @patch.object(Postgresql, '_is_postmaster_pid_running', Mock(return_value=False))
     @patch('psutil.Process')
     def test__wait_for_user_backends_to_close(self, mock_psutil):
         child = Mock()
@@ -878,8 +901,55 @@ class TestPostgresql(unittest.TestCase):
         self.p.stop(on_safepoint=mock_callback)
 
     @patch('os.kill', Mock(side_effect=[OSError(errno.ESRCH, ''), OSError]))
+    @patch('psutil.Process', Mock(side_effect=[psutil.NoSuchProcess]))
     @patch('time.sleep', Mock())
-    @patch.object(Postgresql, 'is_pid_running', Mock(side_effect=[True, False]))
+    @patch.object(Postgresql, '_is_postmaster_pid_running', Mock(side_effect=[True, False]))
     def test_terminate_starting_postmaster(self):
         self.p.terminate_starting_postmaster(123)
         self.p.terminate_starting_postmaster(123)
+
+    def test_read_postmaster_opts(self):
+        m = mock_open(read_data='/usr/lib/postgres/9.6/bin/postgres "-D" "data/postgresql0" \
+"--listen_addresses=127.0.0.1" "--port=5432" "--hot_standby=on" "--wal_level=hot_standby" \
+"--wal_log_hints=on" "--max_wal_senders=5" "--max_replication_slots=5"\n')
+        with patch.object(builtins, 'open', m):
+            data = self.p.read_postmaster_opts()
+            self.assertEquals(data['wal_level'], 'hot_standby')
+            self.assertEquals(int(data['max_replication_slots']), 5)
+            self.assertEqual(data.get('D'), None)
+
+            m.side_effect = IOError
+            data = self.p.read_postmaster_opts()
+            self.assertEqual(data, dict())
+
+    @patch('subprocess.Popen')
+    @patch.object(builtins, 'open', Mock(return_value=42))
+    def test_single_user_mode(self, subprocess_popen_mock):
+        subprocess_popen_mock.return_value.wait.return_value = 0
+        self.assertEquals(self.p.single_user_mode(command="CHECKPOINT"), 0)
+        subprocess_popen_mock.return_value = None
+        self.assertEquals(self.p.single_user_mode(), 1)
+        self.assertEquals(self.p.single_user_mode(options={'archive_mode': 'on'}), 1)
+
+    @patch('os.listdir', Mock(side_effect=[OSError, ['a', 'b']]))
+    @patch('os.unlink', Mock(side_effect=OSError))
+    @patch('os.remove', Mock())
+    @patch('os.path.islink', Mock(side_effect=[True, False]))
+    @patch('os.path.isfile', Mock(return_value=True))
+    def test_cleanup_archive_status(self):
+        self.p.cleanup_archive_status()
+        self.p.cleanup_archive_status()
+
+    @patch('os.unlink', Mock())
+    @patch('os.path.isfile', Mock(return_value=True))
+    @patch.object(Postgresql, 'single_user_mode', Mock(return_value=0))
+    def test_fix_cluster_state(self):
+        self.assertTrue(self.p.fix_cluster_state())
+
+    def test__update_postmaster_cached_info(self):
+        with open(os.path.join(self.data_dir, 'postmaster.pid'), 'w') as f:
+            f.write('1\n\n1\n')
+        self.p.read_pid_file()
+        with open(os.path.join(self.data_dir, 'postmaster.pid'), 'w') as f:
+            f.write('a\n\n1\n')
+        self.p.read_pid_file()

tests/test_watchdog.py

@@ -194,15 +194,24 @@ class TestLinuxWatchdogDevice(unittest.TestCase):
         self.assertRaises(WatchdogError, self.impl.set_timeout, -1)
 
     @patch('os.open', Mock(return_value=3))
-    @patch('fcntl.ioctl', Mock(return_value=-1))
+    @patch('fcntl.ioctl', Mock(side_effect=OSError))
     def test__ioctl(self):
         self.assertRaises(WatchdogError, self.impl.get_support)
         self.impl.open()
-        self.assertRaises(IOError, self.impl.get_support)
+        self.assertRaises(WatchdogError, self.impl.get_support)
 
     def test_is_healthy(self):
         self.assertFalse(self.impl.is_healthy)
 
+    @patch('os.open', Mock(return_value=3))
+    @patch('fcntl.ioctl', Mock(side_effect=OSError))
+    def test_error_handling(self):
+        self.impl.open()
+        self.assertRaises(WatchdogError, self.impl.get_timeout)
+        self.assertRaises(WatchdogError, self.impl.set_timeout, 10)
+        # We still try to output a reasonable string even if getting info errors
+        self.assertEquals(self.impl.describe(), "Linux watchdog device")
+
     @patch('os.open', Mock(side_effect=OSError))
     def test_open(self):
         self.assertRaises(WatchdogError, self.impl.open)