Bump version and write release notes (#560)

and implement missing unit-tests

docs/ENVIRONMENT.rst

@@ -33,6 +33,7 @@ Consul
 -  **PATRONI\_CONSUL\_CACERT**: (optional) The ca certificate. If pressent it will enable validation.
 -  **PATRONI\_CONSUL\_CERT**: (optional) File with the client certificate
 -  **PATRONI\_CONSUL\_KEY**: (optional) File with the client key. Can be empty if the key is part of certificate.
+-  **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable.
 
 Etcd

docs/SETTINGS.rst

@@ -58,6 +58,7 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **cacert**: (optional) The ca certificate. If pressent it will enable validation.
 -  **cert**: (optional) file with the client certificate
 -  **key**: (optional) file with the client key. Can be empty if the key is part of **cert**.
+-  **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **checks**: (optional) list of Consul health checks used for the session. If not specified Consul will use "serfHealth" in additional to the TTL based check created by Patroni. Additional checks, in particular the "serfHealth", may cause the leader lock to expire faster than in `ttl` seconds when the leader instance becomes unavailable
 
 Etcd

docs/releases.rst

@@ -3,6 +3,47 @@
 Release notes
 =============
 
+Version 1.3.6
+-------------
+
+**Stability improvements**
+
+- Verify process start time when checking if postgres is running. (Ants Aasma)
+
+  After a crash that doesn't clean up postmaster.pid there could be a new process with the same pid, resulting in a false positive for is_running(), which will lead to all kinds of bad behavior.
+
+- Shutdown postgresql before bootstrap when we lost data directory (ainlolcat)
+  
+  When data directory on the master is forcefully removed, postgres process can still stay alive for some time and prevent the replica created in place of that former master from starting or replicating.
+  The fix makes Patroni cache the postmaster pid and its start time and let it terminate the old postmaster in case it is still running after the corresponding data directory has been removed.
+
+- Perform crash recovery in a single user mode if postgres master dies (Alexander Kukushkin)
+
+  It is unsafe to start immediately as a standby and not possible to run ``pg_rewind`` if postgres hasn't been shut down cleanly.
+  The single user crash recovery only kicks in if ``pg_rewind`` is enabled or there is no master at the moment.
+
+**Consul improvements**
+
+- Make it possible to provide datacenter configuration for Consul (DeathBorn, Alexander)
+
+  Before that Patroni was always communicating with datacenter of the host it runs on.
+
+- Always send a token in X-Consul-Token http header (Alexander)
+
+  If ``consul.token`` is defined in Patroni configuration, we will always send it in the 'X-Consul-Token' http header.
+  python-consul module tries to be "consistent" with Consul REST API, which doesn't accept token as a query parameter for `session API <https://www.consul.io/api/session.html>`__, but it still works with 'X-Consul-Token' header.
+
+- Adjust session TTL if supplied value is smaller than the minimum possible (Stas Fomin, Alexander)
+
+  It could happen that the TTL provided in the Patroni configuration is smaller than the minimum one supported by Consul. In that case, Consul agent fails to create a new session.
+  Without a session Patroni cannot create member and leader keys in the Consul KV store, resulting in an unhealthy cluster.
+
+**Other improvements**
+
+- Define custom log format via environment variable ``PATRONI_LOGFORMAT`` (Stas)
+
+  Allow disabling timestamps and other similar fields in Patroni logs if they are already added by the system logger (usually when Patroni runs as a service).
+
 Version 1.3.5
 -------------
 

extras/startup-scripts/patroni

@@ -1,144 +1,144 @@
-#!/bin/sh
-#
-### BEGIN INIT INFO
-# Provides:          patroni
-# Required-Start:    $remote_fs $syslog
-# Required-Stop:     $remote_fs $syslog
-# Default-Start:     2 3 4 5
-# Default-Stop:      0 1 6
-# Short-Description: Patroni init script
-# Description:       Runners to orchestrate a high-availability PostgreSQL
-### END INIT INFO
-
-### BEGIN USER CONFIGURATION
-
-CONF="/etc/patroni/postgres.yml"
-LOGFILE="/var/log/patroni.log"
-USER="postgres"
-GROUP="postgres"
-
-NAME=patroni
-PATRONI="/opt/patroni/$NAME.py"
-PIDFILE="/var/run/$NAME.pid"
-
-# Set this parameter, if you have several Postgres versions installed
-# POSTGRES_VERSION="9.4"
-POSTGRES_VERSION=""
-
-### END USER CONFIGURATION
-
-. /lib/lsb/init-functions
-
-# Loading this library for get_versions() function
-if test ! -e /usr/share/postgresql-common/init.d-functions; then
-  log_failure_msg "Probably postgresql-common does not installed."
-  exit 1
-else
-  . /usr/share/postgresql-common/init.d-functions
-fi
-
-# Is there Patroni executable?
-if test ! -e $PATRONI; then
-  log_failure_msg "Patroni executable $PATRONI does not exist."
-  exit 1
-fi
-
-# Is there Patroni configuration file?
-if test ! -e $CONF; then
-  log_failure_msg "Patroni configuration file $CONF does not exist."
-  exit 1
-fi
-
-# Create logfile if doesn't exist
-if test ! -e $LOGFILE; then
-  log_action_msg "Creating logfile for Patroni..."
-  touch $LOGFILE
-  chown $USER:$GROUP $LOGFILE
-fi
-
-prepare_pgpath() {
-  if [ "$POSTGRES_VERSION" != "" ]; then
-    if [ -x /usr/lib/postgresql/$POSTGRES_VERSION/bin/pg_ctl ]; then
-      PGPATH="/usr/lib/postgresql/$POSTGRES_VERSION/bin"
-    else
-      log_failure_msg "Postgres version incorrect, check POSTGRES_VERSION variable."
-      exit 0
-    fi
-  else
-    get_versions
-    if echo $versions | grep -q -e "\s"; then
-      log_warning_msg "You have several Postgres versions installed. Please, use POSTGRES_VERSION to define correct environment."
-    else
-      versions=`echo $versions | sed -e 's/^[ \t]*//'`
-      PGPATH="/usr/lib/postgresql/$versions/bin"
-    fi
-  fi
-}
-
-get_pid() {
-  if test -e $PIDFILE; then
-    PID=`cat $PIDFILE`
-    CHILDPID=`ps --ppid $PID -o %p --no-headers`
-  else
-    log_failure_msg "Could not find PID file. Patroni probably down."
-    exit 1
-  fi
-}
-
-
-case "$1" in
-  start)
-    prepare_pgpath
-    PGPATH=$PATH:$PGPATH
-    log_success_msg "Starting Patroni\n"
-    exec start-stop-daemon --start --quiet \
-        --background \
-        --pidfile $PIDFILE --make-pidfile \
-        --chuid $USER:$GROUP \
-        --chdir `eval echo ~$USER` \
-        --exec $PATRONI \
-        --startas /bin/sh -- \
-        -c "/usr/bin/env PATH=$PGPATH /usr/bin/python $PATRONI $CONF >> $LOGFILE 2>&1"
-    ;;
-
-  stop)
-    log_success_msg "Stopping Patroni"
-    get_pid
-    start-stop-daemon --stop --pid $CHILDPID
-    start-stop-daemon --stop --pidfile $PIDFILE --remove-pidfile --quiet
-    ;;
-
-  reload)
-    log_success_msg "Reloading Patroni configuration"
-    get_pid
-    kill -HUP $CHILDPID
-    ;;
-
-  status)
-    get_pid
-    if start-stop-daemon -T --pid $CHILDPID; then
-      log_success_msg "Patroni is running\n"
-      exit 0
-    else
-      log_warning_msg "Patroni in not running\n"
-    fi
-    ;;
-
-  restart)
-    $0 stop
-    $0 start
-    ;;
-
-  *)
-    echo "Usage: /etc/init.d/$NAME {start|stop|restart|reload|status}"
-    exit 1
-    ;;
-esac
-
-if [ $? -eq 0 ]; then
-        echo .
-        exit 0
-else
-        echo " failed"
-        exit 1
-fi
+#!/bin/sh
+#
+### BEGIN INIT INFO
+# Provides:          patroni
+# Required-Start:    $remote_fs $syslog
+# Required-Stop:     $remote_fs $syslog
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: Patroni init script
+# Description:       Runners to orchestrate a high-availability PostgreSQL
+### END INIT INFO
+
+### BEGIN USER CONFIGURATION
+
+CONF="/etc/patroni/postgres.yml"
+LOGFILE="/var/log/patroni.log"
+USER="postgres"
+GROUP="postgres"
+
+NAME=patroni
+PATRONI="/opt/patroni/$NAME.py"
+PIDFILE="/var/run/$NAME.pid"
+
+# Set this parameter, if you have several Postgres versions installed
+# POSTGRES_VERSION="9.4"
+POSTGRES_VERSION=""
+
+### END USER CONFIGURATION
+
+. /lib/lsb/init-functions
+
+# Loading this library for get_versions() function
+if test ! -e /usr/share/postgresql-common/init.d-functions; then
+  log_failure_msg "Probably postgresql-common does not installed."
+  exit 1
+else
+  . /usr/share/postgresql-common/init.d-functions
+fi
+
+# Is there Patroni executable?
+if test ! -e $PATRONI; then
+  log_failure_msg "Patroni executable $PATRONI does not exist."
+  exit 1
+fi
+
+# Is there Patroni configuration file?
+if test ! -e $CONF; then
+  log_failure_msg "Patroni configuration file $CONF does not exist."
+  exit 1
+fi
+
+# Create logfile if doesn't exist
+if test ! -e $LOGFILE; then
+  log_action_msg "Creating logfile for Patroni..."
+  touch $LOGFILE
+  chown $USER:$GROUP $LOGFILE
+fi
+
+prepare_pgpath() {
+  if [ "$POSTGRES_VERSION" != "" ]; then
+    if [ -x /usr/lib/postgresql/$POSTGRES_VERSION/bin/pg_ctl ]; then
+      PGPATH="/usr/lib/postgresql/$POSTGRES_VERSION/bin"
+    else
+      log_failure_msg "Postgres version incorrect, check POSTGRES_VERSION variable."
+      exit 0
+    fi
+  else
+    get_versions
+    if echo $versions | grep -q -e "\s"; then
+      log_warning_msg "You have several Postgres versions installed. Please, use POSTGRES_VERSION to define correct environment."
+    else
+      versions=`echo $versions | sed -e 's/^[ \t]*//'`
+      PGPATH="/usr/lib/postgresql/$versions/bin"
+    fi
+  fi
+}
+
+get_pid() {
+  if test -e $PIDFILE; then
+    PID=`cat $PIDFILE`
+    CHILDPID=`ps --ppid $PID -o %p --no-headers`
+  else
+    log_failure_msg "Could not find PID file. Patroni probably down."
+    exit 1
+  fi
+}
+
+
+case "$1" in
+  start)
+    prepare_pgpath
+    PGPATH=$PATH:$PGPATH
+    log_success_msg "Starting Patroni\n"
+    exec start-stop-daemon --start --quiet \
+        --background \
+        --pidfile $PIDFILE --make-pidfile \
+        --chuid $USER:$GROUP \
+        --chdir `eval echo ~$USER` \
+        --exec $PATRONI \
+        --startas /bin/sh -- \
+        -c "/usr/bin/env PATH=$PGPATH /usr/bin/python $PATRONI $CONF >> $LOGFILE 2>&1"
+    ;;
+
+  stop)
+    log_success_msg "Stopping Patroni"
+    get_pid
+    start-stop-daemon --stop --pid $CHILDPID
+    start-stop-daemon --stop --pidfile $PIDFILE --remove-pidfile --quiet
+    ;;
+
+  reload)
+    log_success_msg "Reloading Patroni configuration"
+    get_pid
+    kill -HUP $CHILDPID
+    ;;
+
+  status)
+    get_pid
+    if start-stop-daemon -T --pid $CHILDPID; then
+      log_success_msg "Patroni is running\n"
+      exit 0
+    else
+      log_warning_msg "Patroni in not running\n"
+    fi
+    ;;
+
+  restart)
+    $0 stop
+    $0 start
+    ;;
+
+  *)
+    echo "Usage: /etc/init.d/$NAME {start|stop|restart|reload|status}"
+    exit 1
+    ;;
+esac
+
+if [ $? -eq 0 ]; then
+        echo .
+        exit 0
+else
+        echo " failed"
+        exit 1
+fi

features/environment.py

@@ -228,7 +228,7 @@ class PatroniController(AbstractController):
             if not os.path.exists(pidfile):
                 return None
             return int(open(pidfile).readline().strip())
-        except:
+        except Exception:
             return None
 
     def database_is_running(self):

patroni/__init__.py

@@ -134,7 +134,8 @@ class Patroni(object):
 
 
 def patroni_main():
-    logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.INFO)
+    logformat = os.environ.get('PATRONI_LOGFORMAT', '%(asctime)s %(levelname)s: %(message)s')
+    logging.basicConfig(format=logformat, level=logging.INFO)
     logging.getLogger('requests').setLevel(logging.WARNING)
 
     patroni = Patroni()

patroni/async_executor.py

@@ -85,7 +85,7 @@ class AsyncExecutor(object):
             # if the func returned something (not None) - wake up main HA loop
             wakeup = func(*args) if args else func()
             return wakeup
-        except:
+        except Exception:
             logger.exception('Exception during execution of long running task %s', self.scheduled_action)
         finally:
             with self:

patroni/config.py

@@ -243,7 +243,7 @@ class Config(object):
                 if name and suffix:
                     # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
                     if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY',
-                                  'VERIFY', 'TOKEN', 'CHECKS') and '_' not in name:
+                                  'VERIFY', 'TOKEN', 'CHECKS', 'DC') and '_' not in name:
                         value = os.environ.pop(param)
                         if suffix == 'PORT':
                             value = value and parse_int(value)

patroni/dcs/__init__.py

@@ -424,7 +424,7 @@ class AbstractDCS(object):
         with self._cluster_thread_lock:
             try:
                 self._load_cluster()
-            except:
+            except Exception:
                 self._cluster = None
                 raise
             return self._cluster

patroni/dcs/consul.py

@@ -25,9 +25,14 @@ class ConsulInternalError(ConsulException):
     """An internal Consul server error occurred"""
 
 
+class InvalidSessionTTL(ConsulInternalError):
+    """Session TTL is too small or too big"""
+
+
 class HTTPClient(object):
 
-    def __init__(self, host='127.0.0.1', port=8500, scheme='http', verify=True, cert=None, ca_cert=None):
+    def __init__(self, host='127.0.0.1', port=8500, token=None, scheme='http', verify=True, cert=None, ca_cert=None):
+        self.token = token
         self._read_timeout = 10
         self.base_uri = '{0}://{1}:{2}'.format(scheme, host, port)
         kwargs = {}
@@ -49,6 +54,10 @@ class HTTPClient(object):
     def set_read_timeout(self, timeout):
         self._read_timeout = timeout/3.0
 
+    @property
+    def ttl(self):
+        return self._ttl
+
     def set_ttl(self, ttl):
         ret = self._ttl != ttl
         self._ttl = ttl
@@ -58,7 +67,11 @@ class HTTPClient(object):
     def response(response):
         data = response.data.decode('utf-8')
         if response.status == 500:
-            raise ConsulInternalError('{0} {1}'.format(response.status, data))
+            msg = '{0} {1}'.format(response.status, data)
+            if data.startswith('Invalid Session TTL'):
+                raise InvalidSessionTTL(msg)
+            else:
+                raise ConsulInternalError(msg)
         return base.Response(response.status, response.headers, data)
 
     def uri(self, path, params=None):
@@ -82,8 +95,9 @@ class HTTPClient(object):
                 kwargs['timeout'] = (float(params['wait'][:-1]) if 'wait' in params else 300) + 1
             else:
                 kwargs['timeout'] = self._read_timeout
-            if isinstance(params, dict) and 'token' in params and params['token']:
-                kwargs['headers'] = {'X-Consul-Token': params.pop('token')}
+            token = params.pop('token', self.token) if isinstance(params, dict) else self.token
+            if token:
+                kwargs['headers'] = {'X-Consul-Token': token}
             return callback(self.response(self.http.request(method.upper(), self.uri(path, params), **kwargs)))
         return wrapper
 
@@ -93,6 +107,7 @@ class ConsulClient(base.Consul):
     def __init__(self, *args, **kwargs):
         self._cert = kwargs.pop('cert', None)
         self._ca_cert = kwargs.pop('ca_cert', None)
+        self._token = kwargs.get('token')
         super(ConsulClient, self).__init__(*args, **kwargs)
 
     def connect(self, *args, **kwargs):
@@ -101,6 +116,8 @@ class ConsulClient(base.Consul):
             kwargs['cert'] = self._cert
         if self._ca_cert:
             kwargs['ca_cert'] = self._ca_cert
+        if self._token:
+            kwargs['token'] = self._token
         return HTTPClient(**kwargs)
 
 
@@ -141,7 +158,8 @@ class Consul(AbstractDCS):
         if config.get('key') and config.get('cert'):
             config['cert'] = (config['cert'], config['key'])
 
-        kwargs = {p: config.get(p) for p in ('host', 'port', 'token', 'scheme', 'cert', 'ca_cert') if config.get(p)}
+        config_keys = ('host', 'port', 'token', 'scheme', 'cert', 'ca_cert', 'dc')
+        kwargs = {p: config.get(p) for p in config_keys if config.get(p)}
 
         verify = config.get('verify')
         if not isinstance(verify, bool):
@@ -177,6 +195,15 @@ class Consul(AbstractDCS):
         self._retry.deadline = retry_timeout
         self._client.http.set_read_timeout(retry_timeout)
 
+    def adjust_ttl(self):
+        try:
+            settings = self._client.agent.self()
+            min_ttl = (settings['Config']['SessionTTLMin'] or 10000000000)/1000000000.0
+            logger.warning('Changing Session TTL from %s to %s', self._client.http.ttl, min_ttl)
+            self._client.http.set_ttl(min_ttl)
+        except Exception:
+            logger.exception('adjust_ttl')
+
     def _do_refresh_session(self):
         """:returns: `!True` if it had to create new session"""
         if self._session and self._last_session_refresh + self._loop_wait > time.time():
@@ -189,9 +216,15 @@ class Consul(AbstractDCS):
                 self._session = None
         ret = not self._session
         if ret:
-            self._session = self._client.session.create(name=self._scope + '-' + self._name,
-                                                        checks=self.__session_checks,
-                                                        lock_delay=0.001, behavior='delete')
+            try:
+                self._session = self._client.session.create(name=self._scope + '-' + self._name,
+                                                            checks=self.__session_checks,
+                                                            lock_delay=0.001, behavior='delete')
+            except InvalidSessionTTL:
+                logger.exception('session.create')
+                self.adjust_ttl()
+                raise
+
         self._last_session_refresh = time.time()
         return ret
 
@@ -262,14 +295,14 @@ class Consul(AbstractDCS):
             self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover, sync)
         except NotFound:
             self._cluster = Cluster(None, None, None, None, [], None, None)
-        except:
+        except Exception:
             logger.exception('get_cluster')
             raise ConsulError('Consul is not responding properly')
 
-    def touch_member(self, data, **kwargs):
+    def touch_member(self, data, ttl=None, permanent=False):
         cluster = self.cluster
         member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
-        create_member = self.refresh_session()
+        create_member = not permanent and self.refresh_session()
 
         if member and (create_member or member.session != self._session):
             try:
@@ -282,7 +315,7 @@ class Consul(AbstractDCS):
             return True
 
         try:
-            args = {} if kwargs.get('permanent', False) else {'acquire': self._session}
+            args = {} if permanent else {'acquire': self._session}
             self._client.kv.put(self.member_path, data, **args)
             self._my_member_data = data
             return True

patroni/dcs/zookeeper.py

@@ -206,7 +206,7 @@ class ZooKeeper(AbstractDCS):
         try:
             self._client.retry(self._client.create, path, value.encode('utf-8'), **kwargs)
             return True
-        except:
+        except Exception:
             return False
 
     def attempt_to_acquire_leader(self, permanent=False):
@@ -221,7 +221,7 @@ class ZooKeeper(AbstractDCS):
             return True
         except NoNodeError:
             return value == '' or (index is None and self._create(self.failover_path, value))
-        except:
+        except Exception:
             logging.exception('set_failover_value')
             return False
 
@@ -248,7 +248,7 @@ class ZooKeeper(AbstractDCS):
                 self._client.delete_async(self.member_path).get(timeout=1)
             except NoNodeError:
                 pass
-            except:
+            except Exception:
                 return False
             member = None
 
@@ -268,7 +268,7 @@ class ZooKeeper(AbstractDCS):
             self._client.set_async(self.member_path, data).get(timeout=1)
             self._my_member_data = data
             return True
-        except:
+        except Exception:
             logger.exception('touch_member')
 
         return False
@@ -285,9 +285,9 @@ class ZooKeeper(AbstractDCS):
             try:
                 self._client.create_async(self.leader_optime_path, last_operation, makepath=True).get(timeout=1)
                 return True
-            except:
+            except Exception:
                 logger.exception('Failed to create %s', self.leader_optime_path)
-        except:
+        except Exception:
             logger.exception('Failed to update %s', self.leader_optime_path)
         return False
 
@@ -307,7 +307,7 @@ class ZooKeeper(AbstractDCS):
     def cancel_initialization(self):
         try:
             self._client.retry(self._cancel_initialization)
-        except:
+        except Exception:
             logger.exception("Unable to delete initialize key")
 
     def delete_cluster(self):
@@ -322,7 +322,7 @@ class ZooKeeper(AbstractDCS):
             return True
         except NoNodeError:
             return value == '' or (index is None and self._create(self.sync_path, value))
-        except:
+        except Exception:
             logging.exception('set_sync_state_value')
             return False
 

patroni/ha.py

@@ -59,6 +59,7 @@ class Ha(object):
         self.old_cluster = None
         self.recovering = False
         self._post_bootstrap_task = None
+        self._crash_recovery_executed = False
         self._start_timeout = None
         self._async_executor = AsyncExecutor(self.wakeup)
         self.watchdog = patroni.watchdog
@@ -91,7 +92,7 @@ class Ha(object):
             if write_leader_optime:
                 try:
                     self.dcs.write_leader_optime(self.state_handler.last_operation())
-                except:
+                except Exception:
                     pass
         return ret
 
@@ -124,7 +125,7 @@ class Ha(object):
             if not self._async_executor.busy and data['state'] in ['running', 'restarting', 'starting']:
                 try:
                     data['xlog_location'] = self.state_handler.wal_position(retry=False)
-                except:
+                except Exception:
                     pass
             if self.patroni.scheduled_restart:
                 scheduled_restart_data = self.patroni.scheduled_restart.copy()
@@ -175,6 +176,11 @@ class Ha(object):
             self._async_executor.run_async(self.state_handler.rewind, (self.cluster.leader,))
             return True
 
+    def _start_crash_recovery(self, msg):
+        self._async_executor.schedule(msg)
+        self._async_executor.run_async(self.state_handler.fix_cluster_state)
+        return msg
+
     def recover(self):
         # Postgres is not running and we will restart in standby mode. Watchdog is not needed until we promote.
         self.watchdog.disable()
@@ -194,6 +200,12 @@ class Ha(object):
         else:
             timeout = None
 
+        data = self.state_handler.controldata()
+        if data.get('Database cluster state') == 'in production' and not self._crash_recovery_executed and \
+                (self.cluster.is_unlocked() or self.state_handler.can_rewind):
+            self._crash_recovery_executed = True
+            return self._start_crash_recovery('doing crash recovery in a single user mode')
+
         self.load_cluster_from_dcs()
 
         if self.has_lock():
@@ -208,14 +220,11 @@ class Ha(object):
             node_to_follow = self._get_node_to_follow(self.cluster)
 
         # once we already tried to start postgres but failed, single user mode is a rescue in this case
-        if self.recovering and not self.state_handler.rewind_executed and self.state_handler.can_rewind:
-            data = self.state_handler.controldata()
-            if data.get('Database cluster state') not in ('shut down', 'shut down in recovery'):
-                self.recovering = False
-                msg = 'fixing cluster state in a single user mode'
-                self._async_executor.schedule(msg)
-                self._async_executor.run_async(self.state_handler.fix_cluster_state)
-                return msg
+        if self.recovering and not self.state_handler.rewind_executed \
+                and not self._crash_recovery_executed and self.state_handler.can_rewind \
+                and data.get('Database cluster state') not in ('shut down', 'shut down in recovery'):
+            self.recovering = False
+            return self._start_crash_recovery('fixing cluster state in a single user mode')
 
         self.recovering = True
 
@@ -896,6 +905,7 @@ class Ha(object):
                 self.dcs.reset_cluster()
                 return 'removed leader key after trying and failing to start postgres'
             return 'failed to start postgres'
+        self._crash_recovery_executed = False
         return None
 
     def cancel_initialization(self):
@@ -1015,6 +1025,7 @@ class Ha(object):
             # is data directory empty?
             if self.state_handler.data_directory_empty():
                 self.state_handler.set_role('uninitialized')
+                self.state_handler.stop()
                 # In case datadir went away while we were master. TODO: check for this and try to stop postgresql.
                 self.watchdog.disable()
 

patroni/postgresql.py

@@ -17,7 +17,7 @@ from contextlib import contextmanager
 from patroni import call_self
 from patroni.callback_executor import CallbackExecutor
 from patroni.exceptions import PostgresConnectionException
-from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop
+from patroni.utils import compare_values, parse_bool, parse_int, Retry, RetryFailedError, polling_loop, int_or_none
 from six import string_types
 from six.moves.urllib.parse import quote_plus
 from threading import current_thread, Lock
@@ -71,6 +71,25 @@ def null_context():
     yield
 
 
+def _update_postmaster_cached_info(func):
+    def wrapper(self):
+        ret = func(self)
+        if ret and 'pid' in ret and 'start_time' in ret:
+            old_pid = self._postmaster_cached_info.get('pid', 0)
+            old_start_time = self._postmaster_cached_info.get('start_time', 0)
+            try:
+                pmpid = int(ret['pid'])
+                pmstart = int(ret['start_time'])
+                if pmpid != old_pid or pmstart != old_start_time:  # this check removes repeating messages from logs
+                    self._postmaster_cached_info = {'pid': pmpid, 'start_time': pmstart}
+                    logger.info("Updated postmaster info: %s .", self._postmaster_cached_info)
+            except ValueError:
+                logger.warning('Cannot update postmaster info with data due garbage in pid file: %s', ret)
+        return ret
+
+    return wrapper
+
+
 class Postgresql(object):
 
     # List of parameters which must be always passed to postmaster as command line options
@@ -142,6 +161,7 @@ class Postgresql(object):
         self._pg_hba_conf = os.path.join(self._config_dir, 'pg_hba.conf')
         self._recovery_conf = os.path.join(self._data_dir, 'recovery.conf')
         self._postmaster_pid = os.path.join(self._data_dir, 'postmaster.pid')
+        self._postmaster_cached_info = {'pid': 0, 'start_time': 0}
         self._trigger_file = config.get('recovery_conf', {}).get('trigger_file') or 'promote'
         self._trigger_file = os.path.abspath(os.path.join(self._data_dir, self._trigger_file))
 
@@ -707,8 +727,12 @@ class Postgresql(object):
         if not (self._version_file_exists() and os.path.isfile(self._postmaster_pid)):
             # XXX: This is dangerous in case somebody deletes the data directory while PostgreSQL is still running.
             return False
-        return self.is_pid_running(self.get_pid())
 
+        pidfile = self.read_pid_file()
+        return self._is_postmaster_pid_running(int_or_none(pidfile.get('pid')),
+                                               start_time=int_or_none(pidfile.get('start_time')))
+
+    @_update_postmaster_cached_info
     def read_pid_file(self):
         """Reads and parses postmaster.pid from the data directory
 
@@ -733,14 +757,48 @@ class Postgresql(object):
             logger.warning("Garbage pid in postmaster.pid: {0!r}".format(pid))
             return 0
 
-    @staticmethod
-    def is_pid_running(pid):
+    def get_pid_with_lost_data_dir(self):
+        logger.info("Trying to check if process running without directory "
+                    "with cached postmaster info: %s .", self._postmaster_cached_info)
         try:
-            if pid < 0:
-                pid = -pid
-            return pid > 0 and pid != os.getpid() and pid != os.getppid() and (os.kill(pid, 0) or True)
-        except Exception:
+            process = psutil.Process(self._postmaster_cached_info['pid'])
+            # check difference instead of values because of rounding issues
+            if abs(self._postmaster_cached_info["start_time"] - process.create_time()) < 2:
+                return process.pid
+            else:
+                logger.info("Process with pid %s was started at different time %s .",
+                            process.pid, process.create_time())
+        except psutil.NoSuchProcess:
+            logger.info("Cannot find process %s .", self._postmaster_cached_info['pid'])
+        return 0
+
+    def clean_postmaster_cached_info(self):
+        self._postmaster_cached_info = {'pid': 0, 'start_time': 0}
+        logger.info("postmaster info was cleaned.")
+
+    @staticmethod
+    def _is_postmaster_pid_running(pid, start_time=None):
+        # Normalize pid handling missing values and negative pids from postmaster.pid
+        if not pid:
             return False
+        if pid < 0:
+            pid = -pid
+
+        try:
+            proc = psutil.Process(pid)
+        except psutil.NoSuchProcess:
+            return False
+
+        # If the process is Patroni or Patronis host process or Patronis child process then it's a false positive
+        my_pid = os.getpid()
+        if pid == my_pid or pid == os.getppid() or proc.parent() == my_pid:
+            return False
+
+        # If process start time differs by more than 3 seconds it's a false positive
+        if start_time is not None and abs(proc.create_time() - start_time) > 3:
+            return False
+
+        return True
 
     @property
     def cb_called(self):
@@ -805,7 +863,7 @@ class Postgresql(object):
                     # Garbage in the pid file
                     pass
 
-            if not self.is_pid_running(pid):
+            if not self._is_postmaster_pid_running(pid, start_time=initiated):
                 logger.error('postmaster is not running')
                 self.set_state('start failed')
                 return False
@@ -933,6 +991,12 @@ class Postgresql(object):
 
     def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint):
         if not self.is_running():
+            if self.data_directory_empty() and self._postmaster_cached_info['pid']:
+                pid = self.get_pid_with_lost_data_dir()
+                if pid > 0:
+                    self.terminate_starting_postmaster(pid)
+                    self.clean_postmaster_cached_info()
+                    return True, True
             if on_safepoint:
                 on_safepoint()
             return True, False
@@ -958,13 +1022,14 @@ class Postgresql(object):
             on_safepoint()
 
         self._wait_for_postmaster_stop(pid)
+        self.clean_postmaster_cached_info()
 
         return True, True
 
     def _wait_for_postmaster_stop(self, pid):
         # This wait loop differs subtly from pg_ctl as we check for both the pid file going
         # away and if the pid is running. This seems safer.
-        while pid == self.get_pid() and self.is_pid_running(pid):
+        while pid == self.get_pid() and self._is_postmaster_pid_running(pid):
             time.sleep(STOP_POLLING_INTERVAL)
 
     def _signal_postmaster_stop(self, mode):
@@ -994,13 +1059,13 @@ class Postgresql(object):
                 return
             logger.warning("Could not send stop signal to PostgreSQL (error: {0})".format(e.errno))
 
-        while self.is_pid_running(pid):
+        while self._is_postmaster_pid_running(pid):
             time.sleep(STOP_POLLING_INTERVAL)
 
     def _wait_for_connection_close(self, pid):
         try:
             with self.connection().cursor() as cur:
-                while pid == self.get_pid() and self.is_pid_running(pid):  # Need a timeout here?
+                while pid == self.get_pid() and self._is_postmaster_pid_running(pid):  # Need a timeout here?
                     cur.execute("SELECT 1")
                     time.sleep(STOP_POLLING_INTERVAL)
         except psycopg2.Error:

patroni/utils.py

@@ -95,17 +95,17 @@ def strtol(value, strict=True):
     True
     """
     value = str(value).strip()
-    l = len(value)
+    ln = len(value)
     i = 0
     # skip sign:
-    if i < l and value[i] in ('-', '+'):
+    if i < ln and value[i] in ('-', '+'):
         i += 1
 
     # we always expect to get digit in the beginning
-    if i < l and value[i].isdigit():
+    if i < ln and value[i].isdigit():
         if value[i] == '0':
             i += 1
-            if i < l and value[i] in ('x', 'X'):  # '0' followed by 'x': HEX
+            if i < ln and value[i] in ('x', 'X'):  # '0' followed by 'x': HEX
                 base = 16
                 i += 1
             else:  # just starts with '0': OCT
@@ -114,7 +114,7 @@ def strtol(value, strict=True):
             base = 10
 
         ret = None
-        while i <= l:
+        while i <= ln:
             try:  # try to find maximally long number
                 i += 1  # by giving to `int` longer and longer strings
                 ret = int(value[:i], base)
@@ -280,3 +280,11 @@ def polling_loop(timeout, interval=1):
         yield iteration
         iteration += 1
         time.sleep(interval)
+
+
+def int_or_none(val):
+    """Returns integer value of the parameter if convertible to int, None otherwise."""
+    try:
+        return int(val)
+    except (ValueError, TypeError):
+        return None

patroni/version.py

@@ -1 +1 @@
-__version__ = '1.3.5'
+__version__ = '1.3.6'

setup.py

@@ -87,7 +87,7 @@ class PyTest(TestCommand):
     def run_tests(self):
         try:
             import pytest
-        except:
+        except Exception:
             raise RuntimeError('py.test is not installed, run: pip install pytest')
         params = {'args': self.test_args}
         if self.cov:

tests/test_consul.py

@@ -3,7 +3,8 @@ import unittest
 
 from consul import ConsulException, NotFound
 from mock import Mock, patch
-from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, ConsulError, HTTPClient
+from patroni.dcs.consul import AbstractDCS, Cluster, Consul, ConsulInternalError, \
+                                ConsulError, HTTPClient, InvalidSessionTTL
 from test_etcd import SleepException
 
 
@@ -47,7 +48,10 @@ class TestHTTPClient(unittest.TestCase):
         self.client.get(Mock(), '')
         self.client.get(Mock(), '', {'wait': '1s', 'index': 1, 'token': 'foo'})
         self.client.http.request.return_value.status = 500
+        self.client.http.request.return_value.data = b'Foo'
         self.assertRaises(ConsulInternalError, self.client.get, Mock(), '')
+        self.client.http.request.return_value.data = b"Invalid Session TTL '3000000000', must be between [10s=24h0m0s]"
+        self.assertRaises(InvalidSessionTTL, self.client.get, Mock(), '')
 
     def test_unknown_method(self):
         try:
@@ -70,7 +74,7 @@ class TestConsul(unittest.TestCase):
     @patch.object(consul.Consul.KV, 'delete', Mock())
     def setUp(self):
         Consul({'ttl': 30, 'scope': 't', 'name': 'p', 'url': 'https://l:1', 'retry_timeout': 10,
-                'verify': 'on', 'key': 'foo', 'cert': 'bar', 'cacert': 'buz'})
+                'verify': 'on', 'key': 'foo', 'cert': 'bar', 'cacert': 'buz', 'token': 'asd', 'dc': 'dc1'})
         Consul({'ttl': 30, 'scope': 't', 'name': 'p', 'url': 'https://l:1', 'retry_timeout': 10,
                 'verify': 'on', 'cert': 'bar', 'cacert': 'buz'})
         self.c = Consul({'ttl': 30, 'scope': 'test', 'name': 'postgresql1', 'host': 'localhost:1', 'retry_timeout': 10})
@@ -84,7 +88,9 @@ class TestConsul(unittest.TestCase):
         self.assertRaises(SleepException, self.c.create_session)
 
     @patch.object(consul.Consul.Session, 'renew', Mock(side_effect=NotFound))
-    @patch.object(consul.Consul.Session, 'create', Mock(side_effect=ConsulException))
+    @patch.object(consul.Consul.Session, 'create', Mock(side_effect=[InvalidSessionTTL, ConsulException]))
+    @patch.object(consul.Consul.Agent, 'self', Mock(return_value={'Config': {'SessionTTLMin': 0}}))
+    @patch.object(HTTPClient, 'set_ttl', Mock(side_effect=ValueError))
     def test_referesh_session(self):
         self.c._session = '1'
         self.assertFalse(self.c.refresh_session())

tests/test_ha.py

@@ -35,7 +35,7 @@ def get_cluster_not_initialized_without_leader():
 def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None):
     m1 = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres',
                                   'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4})
-    l = Leader(0, 0, m1) if leader else None
+    leader = Leader(0, 0, m1) if leader else None
     m2 = Member(0, 'other', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres',
                                  'api_url': 'http://127.0.0.1:8011/patroni',
                                  'state': 'running',
@@ -43,7 +43,7 @@ def get_cluster_initialized_without_leader(leader=False, failover=None, sync=Non
                                  'scheduled_restart': {'schedule': "2100-01-01 10:53:07.560445+00:00",
                                                        'postgres_version': '99.0.0'}})
     syncstate = SyncState(0 if sync else None, sync and sync[0], sync and sync[1])
-    return get_cluster(True, l, [m1, m2], failover, syncstate)
+    return get_cluster(True, leader, [m1, m2], failover, syncstate)
 
 
 def get_cluster_initialized_with_leader(failover=None, sync=None):
@@ -51,8 +51,8 @@ def get_cluster_initialized_with_leader(failover=None, sync=None):
 
 
 def get_cluster_initialized_with_only_leader(failover=None):
-    l = get_cluster_initialized_without_leader(leader=True, failover=failover).leader
-    return get_cluster(True, l, [l], failover, None)
+    leader = get_cluster_initialized_without_leader(leader=True, failover=failover).leader
+    return get_cluster(True, leader, [leader], failover, None)
 
 
 def get_node_status(reachable=True, in_recovery=True, wal_position=10, nofailover=False, watchdog_failed=False):
@@ -136,7 +136,7 @@ class TestHa(unittest.TestCase):
 
     @patch('socket.getaddrinfo', socket_getaddrinfo)
     @patch('psycopg2.connect', psycopg2_connect)
-    @patch('patroni.dcs.dcs_modules', Mock(return_value=['foo', 'patroni.dcs.etcd']))
+    @patch('patroni.dcs.dcs_modules', Mock(return_value=['patroni.dcs.foo', 'patroni.dcs.etcd']))
     @patch.object(etcd.Client, 'read', etcd_read)
     def setUp(self):
         with patch.object(Client, 'machines') as mock_machines:
@@ -172,21 +172,27 @@ class TestHa(unittest.TestCase):
         self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')
 
     def test_recover_replica_failed(self):
-        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery'}
         self.p.is_running = false
         self.p.follow = false
         self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')
         self.assertEquals(self.ha.run_cycle(), 'failed to start postgres')
 
-    def test_recover_master_failed(self):
+    def test_recover_former_master(self):
         self.p.follow = false
         self.p.is_running = false
         self.p.name = 'leader'
         self.p.set_role('master')
-        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.p.controldata = lambda: {'Database cluster state': 'shut down'}
         self.ha.cluster = get_cluster_initialized_with_leader()
         self.assertEquals(self.ha.run_cycle(), 'starting as readonly because i had the session lock')
 
+    @patch.object(Postgresql, 'fix_cluster_state', Mock())
+    def test_crash_recovery(self):
+        self.p.is_running = false
+        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.assertEquals(self.ha.run_cycle(), 'doing crash recovery in a single user mode')
+
     @patch.object(Postgresql, 'rewind_needed_and_possible', Mock(return_value=True))
     def test_recover_with_rewind(self):
         self.p.is_running = false
@@ -196,7 +202,7 @@ class TestHa(unittest.TestCase):
     @patch.object(Postgresql, 'can_rewind', PropertyMock(return_value=True))
     @patch.object(Postgresql, 'fix_cluster_state', Mock())
     def test_single_user_after_recover_failed(self):
-        self.p.controldata = lambda: {'Database cluster state': 'in production'}
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery'}
         self.p.is_running = false
         self.p.follow = false
         self.assertEquals(self.ha.run_cycle(), 'starting as a secondary')

tests/test_patroni.py

@@ -104,6 +104,7 @@ class TestPatroni(unittest.TestCase):
     @patch('patroni.config.Config.save_cache', Mock())
     @patch('patroni.config.Config.reload_local_configuration', Mock(return_value=True))
     @patch.object(Postgresql, 'state', PropertyMock(return_value='running'))
+    @patch.object(Postgresql, 'data_directory_empty', Mock(return_value=False))
     def test_run(self):
         self.p.postgresql.set_role('replica')
         self.p.sighup_handler()

tests/test_postgresql.py

@@ -2,6 +2,7 @@ import errno
 import mock  # for the mock.call method, importing it without a namespace breaks python3
 import os
 import psycopg2
+import psutil
 import shutil
 import subprocess
 import unittest
@@ -238,17 +239,17 @@ class TestPostgresql(unittest.TestCase):
 
     @patch.object(Postgresql, 'pg_isready')
     @patch.object(Postgresql, 'read_pid_file')
-    @patch.object(Postgresql, 'is_pid_running')
+    @patch.object(Postgresql, '_is_postmaster_pid_running')
     @patch('patroni.postgresql.polling_loop', Mock(return_value=range(1)))
-    def test_wait_for_port_open(self, mock_is_pid_running, mock_read_pid_file, mock_pg_isready):
-        mock_is_pid_running.return_value = False
+    def test_wait_for_port_open(self, mock_is_postmaster_pid_running, mock_read_pid_file, mock_pg_isready):
+        mock_is_postmaster_pid_running.return_value = False
         mock_pg_isready.return_value = STATE_NO_RESPONSE
 
         # No pid file and postmaster death
         mock_read_pid_file.return_value = {}
         self.assertFalse(self.p.wait_for_port_open(42, 100., 1))
 
-        mock_is_pid_running.return_value = True
+        mock_is_postmaster_pid_running.return_value = True
 
         # timeout
         mock_read_pid_file.return_value = {'pid', 1}
@@ -276,6 +277,21 @@ class TestPostgresql(unittest.TestCase):
         mock_is_running.return_value = False
         self.assertTrue(self.p.stop(on_safepoint=mock_callback))
         mock_callback.assert_called()
+
+        with patch.object(Postgresql, '_is_postmaster_pid_running', Mock(return_value=False)), \
+                patch.object(Postgresql, 'data_directory_empty', Mock(return_value=True)):
+            with patch('psutil.Process') as mock_psutil:
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.return_value.pid = 1
+                mock_psutil.return_value.create_time.return_value = 1
+                self.assertTrue(self.p.stop())
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.return_value.create_time.return_value = 100
+                self.assertTrue(self.p.stop())
+                self.p._postmaster_cached_info = {'pid': 1, 'start_time': 1}
+                mock_psutil.side_effect = psutil.NoSuchProcess('')
+                self.assertTrue(self.p.stop())
+
         mock_is_running.return_value = True
         mock_get_pid.return_value = 0
         mock_callback.reset_mock()
@@ -289,7 +305,7 @@ class TestPostgresql(unittest.TestCase):
             self.assertFalse(self.p.stop())
             self.assertTrue(self.p.stop())
         with patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None))):
-            with patch.object(Postgresql, 'is_pid_running', Mock(side_effect=[True, False, False])):
+            with patch.object(Postgresql, '_is_postmaster_pid_running', Mock(side_effect=[True, False, False])):
                 self.assertTrue(self.p.stop())
 
     def test_restart(self):
@@ -772,12 +788,19 @@ class TestPostgresql(unittest.TestCase):
             os.remove(pidfile)
         self.assertEquals(self.p.read_pid_file(), {})
 
-    @patch('os.kill')
-    def test_is_pid_running(self, mock_kill):
-        mock_kill.return_value = True
-        self.assertTrue(self.p.is_pid_running(-100))
-        self.assertFalse(self.p.is_pid_running(0))
-        self.assertFalse(self.p.is_pid_running(None))
+    @patch.object(Postgresql, '_version_file_exists', Mock(return_value=True))
+    @patch('os.path.isfile', Mock(return_value=True))
+    @patch.object(Postgresql, 'read_pid_file')
+    @patch('psutil.Process')
+    def test_is_postmaster_pid_running(self, mock_psutil, mock_read_pid_file):
+        mock_psutil.return_value.create_time.return_value = 1
+        mock_read_pid_file.return_value = {'pid': -100, 'start_time': 1}
+        self.assertTrue(self.p.is_running())
+        with patch('os.getpid', Mock(return_value=100)):
+            mock_read_pid_file.return_value = {'pid': 100, 'start_time': 1}
+            self.assertFalse(self.p.is_running())
+        mock_read_pid_file.return_value = {'pid': 100, 'start_time': 100}
+        self.assertFalse(self.p.is_running())
 
     def test_pick_sync_standby(self):
         cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None,
@@ -855,20 +878,20 @@ class TestPostgresql(unittest.TestCase):
     @patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None)))
     @patch.object(Postgresql, 'get_pid', Mock(return_value=123))
     @patch('time.sleep', Mock())
-    @patch.object(Postgresql, 'is_pid_running')
-    def test__wait_for_connection_close(self, mock_is_pid_running):
-        mock_is_pid_running.side_effect = [True, False, False]
+    @patch.object(Postgresql, '_is_postmaster_pid_running')
+    def test__wait_for_connection_close(self, mock_is_postmaster_pid_running):
+        mock_is_postmaster_pid_running.side_effect = [True, False, False]
         mock_callback = Mock()
         self.p.stop(on_safepoint=mock_callback)
 
-        mock_is_pid_running.side_effect = [True, False, False]
+        mock_is_postmaster_pid_running.side_effect = [True, False, False]
         with patch.object(MockCursor, "execute", Mock(side_effect=psycopg2.Error)):
             self.p.stop(on_safepoint=mock_callback)
 
     @patch.object(Postgresql, 'is_running', Mock(return_value=True))
     @patch.object(Postgresql, '_signal_postmaster_stop', Mock(return_value=(123, None)))
     @patch.object(Postgresql, 'get_pid', Mock(return_value=123))
-    @patch.object(Postgresql, 'is_pid_running', Mock(return_value=False))
+    @patch.object(Postgresql, '_is_postmaster_pid_running', Mock(return_value=False))
     @patch('psutil.Process')
     def test__wait_for_user_backends_to_close(self, mock_psutil):
         child = Mock()
@@ -878,8 +901,9 @@ class TestPostgresql(unittest.TestCase):
         self.p.stop(on_safepoint=mock_callback)
 
     @patch('os.kill', Mock(side_effect=[OSError(errno.ESRCH, ''), OSError]))
+    @patch('psutil.Process', Mock(side_effect=[psutil.NoSuchProcess]))
     @patch('time.sleep', Mock())
-    @patch.object(Postgresql, 'is_pid_running', Mock(side_effect=[True, False]))
+    @patch.object(Postgresql, '_is_postmaster_pid_running', Mock(side_effect=[True, False]))
     def test_terminate_starting_postmaster(self):
         self.p.terminate_starting_postmaster(123)
         self.p.terminate_starting_postmaster(123)
@@ -921,3 +945,11 @@ class TestPostgresql(unittest.TestCase):
     @patch.object(Postgresql, 'single_user_mode', Mock(return_value=0))
     def test_fix_cluster_state(self):
         self.assertTrue(self.p.fix_cluster_state())
+
+    def test__update_postmaster_cached_info(self):
+        with open(os.path.join(self.data_dir, 'postmaster.pid'), 'w') as f:
+            f.write('1\n\n1\n')
+        self.p.read_pid_file()
+        with open(os.path.join(self.data_dir, 'postmaster.pid'), 'w') as f:
+            f.write('a\n\n1\n')
+        self.p.read_pid_file()