Add synchronous replication support. (#314)

Adds a new configuration variable synchronous_mode. When enabled Patroni will manage synchronous_standby_names to enable synchronous replication whenever there are healthy standbys available. With synchronous mode enabled Patroni will automatically fail over only to a standby that was synchronously replicating at the time of master failure. This effectively means zero lost user visible transactions.

To enforce the synchronous failover guarantee Patroni stores current synchronous replication state in the DCS, using strict ordering, first enable synchronous replication, then publish the information. Standby can use this to verify that it was indeed a synchronous standby before master failed and is allowed to fail over.

We can't enable multiple standbys as synchronous, allowing PostreSQL to pick one because we can't know which one was actually set to be synchronous on the master when it failed. This means that on standby failure commits will be blocked on the master until next run_cycle iteration. TODO: figure out a way to poke Patroni to run sooner or allow for PostgreSQL to pick one without the possibility of lost transactions.

On graceful shutdown standbys will disable themselves by setting a nosync tag for themselves and waiting for the master to notice and pick another standby. This adds a new mechanism for Ha to publish dynamic tags to the DCS.

When the synchronous standby goes away or disconnects a new one is picked and Patroni switches master over to the new one. If no synchronous standby exists Patroni disables synchronous replication (synchronous_standby_names=''), but not synchronous_mode. In this case, only the node that was previously master is allowed to acquire the leader lock.

Added acceptance tests and documentation.

Implementation by @ants with extensive review by @CyberDem0n.

README.rst

@@ -89,24 +89,7 @@ Go `here <https://github.com/zalando/patroni/blob/master/docs/ENVIRONMENT.rst>`_
 Replication Choices
 ===============
 
-Patroni uses Postgres' streaming replication, which is asynchronous by default. For more information, see the `Postgres documentation on streaming replication <http://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION>`__.
-
-Patroni's asynchronous replication configuration allows for ``maximum_lag_on_failover`` settings. This setting ensures failover will not occur if a follower is more than a certain number of bytes behind the follower. This setting should be increased or decreased based on business requirements.
-
-When asynchronous replication is not optimal for your use case, investigate Postgres's `synchronous replication <http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION>`__. Synchronous replication ensures consistency across a cluster by confirming that writes are written to a secondary before returning to the connecting client with a success. The cost of synchronous replication: reduced throughput on writes. This throughput will be entirely based on network performance.
-
-In hosted datacenter environments (like AWS, Rackspace, or any network you do not control), synchronous replication significantly increases the variability of write performance. If followers become inaccessible from the leader, the leader effectively becomes read-only.
-
-To enable a simple synchronous replication test, add the follow lines to the ``parameters`` section of your YAML configuration files:
-
-.. code:: YAML
-
-        synchronous_commit: "on"
-        synchronous_standby_names: "*"
-
-When using synchronous replication, use at least three Postgres data nodes to ensure write availability if one host fails.
-
-Choosing your replication schema is dependent on your business considerations. Investigate both async and sync replication, as well as other HA solutions, to determine which solution is best for you.
+Patroni uses Postgres' streaming replication, which is asynchronous by default. Patroni's asynchronous replication configuration allows for ``maximum_lag_on_failover`` settings. This setting ensures failover will not occur if a follower is more than a certain number of bytes behind the leader. This setting should be increased or decreased based on business requirements. It's also possible to use synchronous replication for better durability guarantees. See `replication modes documentation <https://github.com/zalando/patroni/blob/master/docs/replication_modes.rst>` for details.
 
 ===============================
 Applications Should Not Use Superusers

docs/SETTINGS.rst

@@ -14,6 +14,7 @@ Bootstrap configuration
     -  **loop\_wait**: the number of seconds the loop will sleep. Default value: 10
     -  **ttl**: the TTL to acquire the leader lock. Think of it as the length of time before initiation of the automatic failover process. Default value: 30
     -  **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
+    -  **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that succesfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See `replication modes documentation <https://github.com/zalando/patroni/blob/master/docs/replication_modes.rst>`__ for details.
     -  **postgresql**:
         -  **use\_pg\_rewind**:whether or not to use pg_rewind
         -  **use\_slots**: whether or not to use replication_slots. Must be False for PostgreSQL 9.3. You should comment out max_replication_slots before it becomes ineligible for leader status.

docs/replication_modes.rst

@@ -0,0 +1,64 @@
+=================
+Replication modes
+=================
+
+Patroni uses PostgreSQL streaming replication. For more information about streaming replication, see the `Postgres documentation <http://www.postgresql.org/docs/current/static/warm-standby.html#STREAMING-REPLICATION>`__. By default Patroni configures PostgreSQL for asynchronous replication. Choosing your replication schema is dependent on your business considerations. Investigate both async and sync replication, as well as other HA solutions, to determine which solution is best for you.
+
+Asynchronous mode durability
+----------------------------
+
+In asynchronous mode the cluster is allowed to lose some committed transactions to ensure availability. When master server fails or becomes unavailable for any other reason Patroni will automatically promote a sufficiently healthy standby to master. Any transactions that have not been replicated to that standby remain in a "forked timeline" on the master, and are effectively unrecoverable [1]_.
+
+The amount of transactions that can be lost is controlled via ``maximum_lag_on_failover`` parameter. Because master transaction log position is not sampled in real time, in reality the amount of lost data on failover is worst case bounded by  ``maximum_lag_on_failover`` bytes of transaction log plus the amount that is written in the last ``ttl`` seconds (``loop_wait``/2 seconds in the average case). However typical steady state replication delay is well under a second.
+
+PostgreSQL synchronous replication
+----------------------------------
+
+You can use Postgres's `synchronous replication <http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION>`__ with Patroni. Synchronous replication ensures consistency across a cluster by confirming that writes are written to a secondary before returning to the connecting client with a success. The cost of synchronous replication: reduced throughput on writes. This throughput will be entirely based on network performance.
+
+In hosted datacenter environments (like AWS, Rackspace, or any network you do not control), synchronous replication significantly increases the variability of write performance. If followers become inaccessible from the leader, the leader effectively becomes read-only.
+
+To enable a simple synchronous replication test, add the follow lines to the ``parameters`` section of your YAML configuration files:
+
+.. code:: YAML
+
+        synchronous_commit: "on"
+        synchronous_standby_names: "*"
+
+When using PostgreSQL synchronous replication, use at least three Postgres data nodes to ensure write availability if one host fails.
+
+Using PostgreSQL synchronous replication does not guarantee zero lost transactions under all circumstances. When master and standby that is currently acting as synchronous fail simultaneously a third node that might not contain all transactions will be promoted.
+
+Synchronous mode
+----------------
+
+For use cases where losing committed transactions is not permissible you can turn on Patronis ``synchronous_mode``. When ``synchronous_mode`` is turned on Patroni will not promote a standby unless it is certain that the standby contains all transactions that may have returned a successful commit status to client [2]_. This means that the system may be unavailable for writes even though some servers are available. System administrators can still use manual failover commmands to promote a standby even if it results in transaction loss.
+
+Turning on ``synchronous_mode`` does not guarantee multi node durability of commits under all circumstances. When no suitable standby is available, master server will still accept writes, but does not guarantee their replication. When the master fails in this mode no standby will be promote.  When the host that used to be master comes back it will get promoted automatically, unless system administrator performed a manual failover. This behavior makes synchronous mode usable with 2 node clusters.
+
+When ``synchronous_mode`` is on and a standby crashes, commits will block until next iteration of Patroni runs and switches master to standalone mode (worst case delay for writes ``ttl`` seconds, average case ``loop_wait``/2 seconds). Manually shutting down or restarting a standby will not cause a commit service interruption. Standby will signal the master to release itself from synchronous standby duties before PostgreSQL shutdown is initiated.
+
+You can ensure that a standby never becomes the synchronous standby by setting ``nosync`` tag to true. This is recommended to set for standbys that are behind slow network connections and would cause performance degradation when becoming a synchronous standby.
+
+Synchronous mode can be switched on and off via Patroni REST interface. See `dynamic configuration <https://github.com/zalando/patroni/blob/master/docs/dynamic_configuration.rst>`__ for instructions.
+
+
+Synchronous mode implementation
+-------------------------------
+
+When in synchronous mode Patroni maintains synchronization state in the DCS, containing the latest master and current synchronous standby. This state is updated with strict ordering constraints to ensure the following invariants:
+
+- A node must be marked as the latest leader whenever it can accept write transactions. Patroni crashing or PostgreSQL not shutting down can cause violations of this invariant.
+
+- A node must be set as the synchronous standby in PostgreSQL as long as it is published as the synchronous standby.
+
+- A node that is not the leader or current synchronous standby is not allowed to promote itself automatically.
+
+Patroni will only ever assign one standby to ``synchronous_standby_names`` because with multiple candidates it is not possible to know which node was acting as synchronous during the failure.
+
+On each HA loop iteration Patroni re-evaluates synchronous standby choice. If the current synchronous standby is connected and has not requested its synchronous status to be removed it remains picked. Otherwise the cluster member avaiable for sync that is furthest ahead in replication is picked.
+
+
+.. [1] The data is still there, but recovering it requires a manual recovery effort by data recovery specialists. When Patroni is allowed to rewind with ``use_pg_rewind`` the forked timeline will be automatically erased to rejoin the failed master with the cluster.
+
+.. [2] Clients can change the behavior per transaction using PostgreSQL's ``synchronous_commit`` setting. Transactions with ``synchronous_commit`` values of ``off`` and ``local`` may be lost on fail over, but will not be blocked by replication delays.

features/basic_replication.feature

@@ -3,15 +3,38 @@ Feature: basic replication
 
   Scenario: check replication of a single table
     Given I start postgres0
-    And postgres0 is a leader after 10 seconds
-    And I start postgres1
-    When I add the table foo to postgres0
+    Then postgres0 is a leader after 10 seconds
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "loop_wait": 2, "synchronous_mode": true}
+    Then I receive a response code 200
+    When I start postgres1
+    And I configure and start postgres2 with a tag replicatefrom postgres0
+    And "sync" key in DCS has leader=postgres0 after 20 seconds
+    And I add the table foo to postgres0
     Then table foo is present on postgres1 after 20 seconds
+    Then table foo is present on postgres2 after 20 seconds
+
+  Scenario: check restart of sync replica
+    Given I run patronictl.py restart batman postgres2 --force
+    And "sync" key in DCS has sync_standby=postgres1 after 2 seconds
+    And I run patronictl.py restart batman postgres1 --force
+    Then I receive a response returncode 0
+    And "sync" key in DCS has sync_standby=postgres2 after 10 seconds
+
+  Scenario: check the basic failover in synchronous mode
+    When I kill postgres0
+    Then postgres2 role is the primary after 22 seconds
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_mode": null}
+    Then I receive a response code 200
+    When I add the table bar to postgres2
+    Then table bar is present on postgres1 after 20 seconds
 
   Scenario: check the basic failover
-    When I kill postgres0
-    Then postgres1 role is the primary after 32 seconds
-    When I start postgres0
+    Given I shut down postgres2
+    Then postgres1 is a leader after 10 seconds
+    And postgres1 role is the primary after 10 seconds
+
+  Scenario: check rejoin of the former master with pg_rewind
+    Given I start postgres0
     Then postgres0 role is the secondary after 20 seconds
-    When I add the table bar to postgres1
-    Then table bar is present on postgres0 after 20 seconds
+    When I add the table buz to postgres1
+    Then table buz is present on postgres0 after 20 seconds

features/cascading_replication.feature

@@ -8,7 +8,7 @@ Scenario: check a base backup and streaming replication from a replica
 	And replication works from postgres0 to postgres1 after 20 seconds
 	And I create label with "postgres0" in postgres0 data directory
 	And I create label with "postgres1" in postgres1 data directory
-	And postgres1 has state=running in dcs after 12 seconds
+	And "members/postgres1" key in DCS has state=running after 12 seconds
 	And I configure and start postgres2 with a tag replicatefrom postgres1
 	Then replication works from postgres0 to postgres2 after 30 seconds
 	And there is a label with "postgres1" in postgres2 data directory

features/steps/cascading_replication.py

@@ -20,13 +20,13 @@ def write_label(context, content, name):
     context.pctl.write_label(name, content)
 
 
-@step('{name:w} has {key:w}={value:w} in dcs after {time_limit:d} seconds')
+@step('"{name}" key in DCS has {key:w}={value:w} after {time_limit:d} seconds')
 def check_member(context, name, key, value, time_limit):
     time_limit *= context.timeout_multiplier
     max_time = time.time() + int(time_limit)
     while time.time() < max_time:
         try:
-            response = json.loads(context.dcs_ctl.query('members/' + name))
+            response = json.loads(context.dcs_ctl.query(name))
             if response.get(key) == value:
                 return
         except Exception:

features/steps/patroni_api.py

@@ -1,4 +1,5 @@
 import json
+import os
 import parse
 import pytz
 import requests
@@ -91,7 +92,10 @@ def do_request(context, request_method, url, data):
 def do_run(context, cmd):
     cmd = ['coverage', 'run', '--source=patroni', '-p'] + shlex.split(cmd)
     try:
-        response = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        # XXX: Dirty hack! We need to take name/passwd from the config!
+        env = os.environ.copy()
+        env.update({'PATRONI_RESTAPI_USERNAME': 'username', 'PATRONI_RESTAPI_PASSWORD': 'password'})
+        response = subprocess.check_output(cmd, stderr=subprocess.STDOUT, env=env)
         context.status_code = 0
     except subprocess.CalledProcessError as e:
         response = e.output

patroni/__init__.py

@@ -49,12 +49,16 @@ class Patroni(object):
 
     def get_tags(self):
         return {tag: value for tag, value in self.config.get('tags', {}).items()
-                if tag not in ('clonefrom', 'nofailover', 'noloadbalance') or value}
+                if tag not in ('clonefrom', 'nofailover', 'noloadbalance', 'nosync') or value}
 
     @property
     def nofailover(self):
         return bool(self.tags.get('nofailover', False))
 
+    @property
+    def nosync(self):
+        return bool(self.tags.get('nosync', False))
+
     def reload_config(self):
         try:
             self.tags = self.get_tags()
@@ -139,5 +143,5 @@ def main():
         if patroni.ha.is_paused():
             logger.info('Leader key is not deleted and Postgresql is not stopped due paused state')
         else:
-            patroni.postgresql.stop(checkpoint=False)
+            patroni.ha.while_not_sync_standby(lambda: patroni.postgresql.stop(checkpoint=False))
             patroni.dcs.delete_leader()

patroni/api.py

@@ -57,7 +57,9 @@ class RestApiHandler(BaseHTTPRequestHandler):
 
     def _write_status_response(self, status_code, response):
         patroni = self.server.patroni
-        response.update({'tags': patroni.tags} if patroni.tags else {})
+        tags = patroni.ha.get_effective_tags()
+        if tags:
+            response['tags'] = tags
         if patroni.postgresql.sysid:
             response['database_system_identifier'] = patroni.postgresql.sysid
         if patroni.postgresql.pending_restart:

patroni/config.py

@@ -41,6 +41,7 @@ class Config(object):
     __DEFAULT_CONFIG = {
         'ttl': 30, 'loop_wait': 10, 'retry_timeout': 10,
         'maximum_lag_on_failover': 1048576,
+        'synchronous_mode': False,
         'postgresql': {
             'bin_dir': '',
             'use_slots': True,
@@ -172,7 +173,10 @@ class Config(object):
                     elif name not in ('connect_address', 'listen', 'data_dir', 'pgpass', 'authentication'):
                         config['postgresql'][name] = deepcopy(value)
             elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overriden from DCS
-                config[name] = int(value)
+                if name == 'synchronous_mode':
+                    config[name] = value
+                else:
+                    config[name] = int(value)
         return config
 
     @staticmethod
@@ -293,7 +297,7 @@ class Config(object):
             config['name'] = pg_config['name']
 
         pg_config.update({p: config[p] for p in ('name', 'scope', 'retry_timeout',
-                          'maximum_lag_on_failover') if p in config})
+                          'synchronous_mode', 'maximum_lag_on_failover') if p in config})
 
         return config
 

patroni/ctl.py

@@ -631,9 +631,11 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
     for m in cluster.members:
         logging.debug(m)
 
-        leader = ''
+        role = ''
         if m.name == leader_name:
-            leader = '*'
+            role = 'Leader'
+        elif m.name == cluster.sync.sync_standby:
+            role = 'Sync standby'
 
         host = m.conn_kwargs()['host']
 
@@ -646,7 +648,7 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
             name,
             m.name,
             host,
-            leader,
+            role,
             m.data.get('state', ''),
             lag,
         ]
@@ -666,7 +668,7 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
         'Cluster',
         'Member',
         'Host',
-        'Leader',
+        'Role',
         'State',
         'Lag in MB',
     ]

patroni/dcs/__init__.py

@@ -230,7 +230,59 @@ class ClusterConfig(namedtuple('ClusterConfig', 'index,data,modify_index')):
         return ClusterConfig(index, data, modify_index or index)
 
 
-class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operation,members,failover')):
+class SyncState(namedtuple('SyncState', 'index,leader,sync_standby')):
+    """Immutable object (namedtuple) which represents last observed synhcronous replication state
+
+    :param index: modification index of a synchronization key in a Configuration Store
+    :param leader: reference to member that was leader
+    :param sync_standby: standby that was last synchronized to leader
+    """
+
+    @staticmethod
+    def from_node(index, value):
+        """
+        >>> SyncState.from_node(1, None).leader is None
+        True
+        >>> SyncState.from_node(1, '{}').leader is None
+        True
+        >>> SyncState.from_node(1, '{').leader is None
+        True
+        >>> SyncState.from_node(1, '[]').leader is None
+        True
+        >>> SyncState.from_node(1, '{"leader": "leader"}').leader == "leader"
+        True
+        """
+        if value:
+            try:
+                data = json.loads(value)
+                if not isinstance(data, dict):
+                    data = {}
+            except (TypeError, ValueError):
+                data = {}
+        else:
+            data = {}
+        return SyncState(index, data.get('leader'), data.get('sync_standby'))
+
+    def matches(self, name):
+        """
+        Returns if a node name matches one of the nodes in the sync state
+
+        >>> s = SyncState(1, 'foo', 'bar')
+        >>> s.matches('foo')
+        True
+        >>> s.matches('bar')
+        True
+        >>> s.matches('baz')
+        False
+        >>> s.matches(None)
+        False
+        >>> SyncState(1, None, None).matches('foo')
+        False
+        """
+        return name is not None and name in (self.leader, self.sync_standby)
+
+
+class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operation,members,failover,sync')):
 
     """Immutable object (namedtuple) which represents PostgreSQL cluster.
     Consists of the following fields:
@@ -240,7 +292,9 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
     :param last_leader_operation: int or long object containing position of last known leader operation.
         This value is stored in `/optime/leader` key
     :param members: list of Member object, all PostgreSQL cluster members including leader
-    :param failover: reference to `Failover` object"""
+    :param failover: reference to `Failover` object
+    :param sync: reference to `SyncState` object, last observed synchronous replication state.
+    """
 
     def is_unlocked(self):
         return not (self.leader and self.leader.name)
@@ -270,6 +324,7 @@ class AbstractDCS(object):
     _MEMBERS = 'members/'
     _OPTIME = 'optime'
     _LEADER_OPTIME = _OPTIME + '/' + _LEADER
+    _SYNC = 'sync'
 
     def __init__(self, config):
         """
@@ -318,6 +373,10 @@ class AbstractDCS(object):
     def leader_optime_path(self):
         return self.client_path(self._LEADER_OPTIME)
 
+    @property
+    def sync_path(self):
+        return self.client_path(self._SYNC)
+
     @abc.abstractmethod
     def set_ttl(self, ttl):
         """Set the new ttl value for leader key"""
@@ -461,6 +520,17 @@ class AbstractDCS(object):
     def delete_cluster(self):
         """Delete cluster from DCS"""
 
+    def write_sync_state(self, leader, sync_standby, index=None):
+        return self.set_sync_state_value(json.dumps({'leader': leader, 'sync_standby': sync_standby}), index=index)
+
+    @abc.abstractmethod
+    def set_sync_state_value(self, value, index=None):
+        """"""
+
+    @abc.abstractmethod
+    def delete_sync_state(self, index=None):
+        """"""
+
     def watch(self, timeout):
         """If the current node is a master it should just sleep.
         Any other node should watch for changes of leader key with a given timeout

patroni/dcs/consul.py

@@ -6,7 +6,7 @@ import time
 import urllib3
 
 from consul import ConsulException, NotFound, base
-from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member
+from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState
 from patroni.exceptions import DCSError
 from patroni.utils import Retry, RetryFailedError, sleep
 from urllib3.exceptions import HTTPError
@@ -209,9 +209,13 @@ class Consul(AbstractDCS):
             if failover:
                 failover = Failover.from_node(failover['ModifyIndex'], failover['Value'])
 
-            self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover)
+            # get synchronization state
+            sync = nodes.get(self._SYNC)
+            sync = SyncState.from_node(sync and sync['ModifyIndex'], sync and sync['Value'])
+
+            self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover, sync)
         except NotFound:
-            self._cluster = Cluster(None, None, None, None, [], None)
+            self._cluster = Cluster(None, None, None, None, [], None, None)
         except:
             logger.exception('get_cluster')
             raise ConsulError('Consul is not responding properly')
@@ -292,6 +296,14 @@ class Consul(AbstractDCS):
         if cluster and isinstance(cluster.leader, Leader) and cluster.leader.name == self._name:
             return self._client.kv.delete(self.leader_path, cas=cluster.leader.index)
 
+    @catch_consul_errors
+    def set_sync_state_value(self, value, index=None):
+        return self._client.kv.put(self.sync_path, value, cas=index)
+
+    @catch_consul_errors
+    def delete_sync_state(self, index=None):
+        return self._client.kv.delete(self.sync_path, cas=index)
+
     def watch(self, timeout):
         if self.__do_not_watch:
             self.__do_not_watch = False

patroni/dcs/etcd.py

@@ -9,7 +9,7 @@ import time
 
 from dns.exception import DNSException
 from dns import resolver
-from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member
+from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState
 from patroni.exceptions import DCSError
 from patroni.utils import Retry, RetryFailedError, sleep
 from urllib3.exceptions import HTTPError, ReadTimeoutError
@@ -300,9 +300,13 @@ class Etcd(AbstractDCS):
             if failover:
                 failover = Failover.from_node(failover.modifiedIndex, failover.value)
 
-            self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover)
+            # get synchronization state
+            sync = nodes.get(self._SYNC)
+            sync = SyncState.from_node(sync and sync.modifiedIndex, sync and sync.value)
+
+            self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover, sync)
         except etcd.EtcdKeyNotFound:
-            self._cluster = Cluster(None, None, None, None, [], None)
+            self._cluster = Cluster(None, None, None, None, [], None, None)
         except:
             logger.exception('get_cluster')
             raise EtcdError('Etcd is not responding properly')
@@ -360,6 +364,14 @@ class Etcd(AbstractDCS):
     def delete_cluster(self):
         return self.retry(self._client.delete, self.client_path(''), recursive=True)
 
+    @catch_etcd_errors
+    def set_sync_state_value(self, value, index=None):
+        return self._client.write(self.sync_path, value, prevIndex=index or 0)
+
+    @catch_etcd_errors
+    def delete_sync_state(self, index=None):
+        return self.retry(self._client.delete, self.sync_path, prevIndex=index or 0)
+
     def watch(self, timeout):
         if self.__do_not_watch:
             self.__do_not_watch = False

patroni/dcs/zookeeper.py

@@ -3,7 +3,7 @@ import logging
 from kazoo.client import KazooClient, KazooState
 from kazoo.exceptions import NoNodeError, NodeExistsError
 from kazoo.handlers.threading import SequentialThreadingHandler
-from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member
+from patroni.dcs import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState
 from patroni.exceptions import DCSError
 
 logger = logging.getLogger(__name__)
@@ -133,10 +133,11 @@ class ZooKeeper(AbstractDCS):
         except NoNodeError:
             return []
 
-    def load_members(self):
+    def load_members(self, sync_standby):
         members = []
         for member in self.get_children(self.members_path, self.cluster_watcher):
-            data = self.get_node(self.members_path + member)
+            watch = member == sync_standby and self.cluster_watcher or None
+            data = self.get_node(self.members_path + member, watch)
             if data is not None:
                 members.append(self.member(member, *data))
         return members
@@ -159,8 +160,13 @@ class ZooKeeper(AbstractDCS):
         last_leader_operation = self._OPTIME in nodes and self._fetch_cluster and self.get_node(self.leader_optime_path)
         last_leader_operation = last_leader_operation and int(last_leader_operation[0]) or 0
 
+        # get synchronization state
+        sync = self.get_node(self.sync_path, watch=self.cluster_watcher) if self._SYNC in nodes else None
+        sync = SyncState.from_node(sync and sync[1].version, sync and sync[0])
+
         # get list of members
-        members = self.load_members() if self._MEMBERS[:-1] in nodes else []
+        sync_standby = sync.leader == self._name and sync.sync_standby or None
+        members = self.load_members(sync_standby) if self._MEMBERS[:-1] in nodes else []
 
         # get leader
         leader = self.get_node(self.leader_path) if self._LEADER in nodes else None
@@ -182,7 +188,7 @@ class ZooKeeper(AbstractDCS):
         failover = self.get_node(self.failover_path, watch=self.cluster_watcher) if self._FAILOVER in nodes else None
         failover = failover and Failover.from_node(failover[1].version, failover[0])
 
-        self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover)
+        self._cluster = Cluster(initialize, config, leader, last_leader_operation, members, failover, sync)
 
     def _load_cluster(self):
         if self._fetch_cluster or self._cluster is None:
@@ -228,7 +234,7 @@ class ZooKeeper(AbstractDCS):
 
     def initialize(self, create_new=True, sysid=""):
         return self._create(self.initialize_path, sysid, makepath=True) if create_new \
-            else self._client.retry(self._client.set, self.initialize_path,  sysid.encode("utf-8"))
+            else self._client.retry(self._client.set, self.initialize_path, sysid.encode("utf-8"))
 
     def touch_member(self, data, ttl=None, permanent=False):
         cluster = self.cluster
@@ -307,6 +313,19 @@ class ZooKeeper(AbstractDCS):
         except NoNodeError:
             return True
 
+    def set_sync_state_value(self, value, index=None):
+        try:
+            self._client.retry(self._client.set, self.sync_path, value.encode('utf-8'), version=index or -1)
+            return True
+        except NoNodeError:
+            return value == '' or (index is None and self._create(self.sync_path, value))
+        except:
+            logging.exception('set_sync_state_value')
+            return False
+
+    def delete_sync_state(self, index=None):
+        return self.set_sync_state_value("{}", index)
+
     def watch(self, timeout):
         if super(ZooKeeper, self).watch(timeout):
             self._fetch_cluster = True

patroni/ha.py

@@ -1,3 +1,4 @@
+import functools
 import json
 import logging
 import psycopg2
@@ -5,12 +6,13 @@ import requests
 import sys
 import datetime
 import pytz
+from threading import RLock
 
 from multiprocessing.pool import ThreadPool
 from patroni.async_executor import AsyncExecutor
 from patroni.exceptions import DCSError, PostgresConnectionException
 from patroni.postgresql import ACTION_ON_START
-from patroni.utils import sleep
+from patroni.utils import polling_loop, sleep
 
 logger = logging.getLogger(__name__)
 
@@ -26,6 +28,13 @@ class Ha(object):
         self.recovering = False
         self._async_executor = AsyncExecutor()
 
+        # Each member publishes various pieces of information to the DCS using touch_member. This lock protects
+        # the state and publishing procedure to have consistent ordering and avoid publishing stale values.
+        self._member_state_lock = RLock()
+        # Count of concurrent sync disabling requests. Value above zero means that we don't want to be synchronous
+        # standby. Changes protected by _member_state_lock.
+        self._disable_sync = 0
+
     def is_paused(self):
         return self.cluster and self.cluster.is_paused()
 
@@ -54,28 +63,38 @@ class Ha(object):
         logger.info('Lock owner: %s; I am %s', lock_owner, self.state_handler.name)
         return lock_owner == self.state_handler.name
 
-    def touch_member(self):
-        data = {
-            'conn_url': self.state_handler.connection_string,
-            'api_url': self.patroni.api.connection_string,
-            'state': self.state_handler.state,
-            'role': self.state_handler.role
-        }
-        if self.patroni.tags:
-            data['tags'] = self.patroni.tags
-        if self.state_handler.pending_restart:
-            data['pending_restart'] = True
-        if not self._async_executor.busy and data['state'] in ['running', 'restarting', 'starting']:
-            try:
-                data['xlog_location'] = self.state_handler.xlog_position(retry=False)
-            except:
-                pass
-        if self.patroni.scheduled_restart:
-            scheduled_restart_data = self.patroni.scheduled_restart.copy()
-            scheduled_restart_data['schedule'] = scheduled_restart_data['schedule'].isoformat()
-            data['scheduled_restart'] = scheduled_restart_data
+    def get_effective_tags(self):
+        """Return configuration tags merged with dynamically applied tags."""
+        tags = self.patroni.tags.copy()
+        # _disable_sync could be modified concurrently, but we don't care as attribute get and set are atomic.
+        if self._disable_sync > 0:
+            tags['nosync'] = True
+        return tags
 
-        self.dcs.touch_member(json.dumps(data, separators=(',', ':')))
+    def touch_member(self):
+        with self._member_state_lock:
+            data = {
+                'conn_url': self.state_handler.connection_string,
+                'api_url': self.patroni.api.connection_string,
+                'state': self.state_handler.state,
+                'role': self.state_handler.role
+            }
+            tags = self.get_effective_tags()
+            if tags:
+                data['tags'] = tags
+            if self.state_handler.pending_restart:
+                data['pending_restart'] = True
+            if not self._async_executor.busy and data['state'] in ['running', 'restarting', 'starting']:
+                try:
+                    data['xlog_location'] = self.state_handler.xlog_position(retry=False)
+                except:
+                    pass
+            if self.patroni.scheduled_restart:
+                scheduled_restart_data = self.patroni.scheduled_restart.copy()
+                scheduled_restart_data['schedule'] = scheduled_restart_data['schedule'].isoformat()
+                data['scheduled_restart'] = scheduled_restart_data
+
+            return self.dcs.touch_member(json.dumps(data, separators=(',', ':')))
 
     def clone(self, clone_member=None, msg='(without leader)'):
         if self.state_handler.clone(clone_member):
@@ -157,13 +176,106 @@ class Ha(object):
 
         return ret
 
+    def is_synchronous_mode(self):
+        return bool(self.cluster and self.cluster.config and self.cluster.config.data.get('synchronous_mode'))
+
+    def process_sync_replication(self):
+        """Process synchronous standby beahvior.
+
+        Synchronous standbys are registered in two places postgresql.conf and DCS. The order of updating them must
+        be right. The invariant that should be kept is that if a node is master and sync_standby is set in DCS,
+        then that node must have synchronous_standby set to that value. Or more simple, first set in postgresql.conf
+        and then in DCS. When removing, first remove in DCS, then in postgresql.conf. This is so we only consider
+        promoting standbys that were guaranteed to be replicating synchronously.
+        """
+        if self.is_synchronous_mode():
+            current = self.cluster.sync.leader and self.cluster.sync.sync_standby
+            picked, allow_promote = self.state_handler.pick_synchronous_standby(self.cluster)
+            if picked != current:
+                # We need to revoke privilege from current before replacing it in the config
+                if current:
+                    logger.info("Removing synchronous privilege from %s", current)
+                    if not self.dcs.write_sync_state(self.state_handler.name, None, index=self.cluster.sync.index):
+                        logger.info('Synchronous replication key updated by someone else.')
+                        return
+                logger.info("Assigning synchronous standby status to %s", picked)
+                self.state_handler.set_synchronous_standby(picked)
+
+                if picked and not allow_promote:
+                    # Wait for PostgreSQL to enable synchronous mode and see if we can immediately set sync_standby
+                    sleep(2)
+                    picked, allow_promote = self.state_handler.pick_synchronous_standby(self.cluster)
+                if allow_promote:
+                    cluster = self.dcs.get_cluster()
+                    if cluster.sync.leader and cluster.sync.leader != self.state_handler.name:
+                        logger.info("Synchronous replication key updated by someone else")
+                        return
+                    if not self.dcs.write_sync_state(self.state_handler.name, picked, index=cluster.sync.index):
+                        logger.info("Synchronous replication key updated by someone else")
+                        return
+                    logger.info("Synchronous standby status assigned to %s", picked)
+        else:
+            if self.cluster.sync.leader and self.dcs.delete_sync_state(index=self.cluster.sync.index):
+                logger.info("Disabled synchronous replication")
+            self.state_handler.set_synchronous_standby(None)
+
+    def is_sync_standby(self, cluster):
+        return cluster.leader and cluster.sync.leader == cluster.leader.name \
+            and cluster.sync.sync_standby == self.state_handler.name
+
+    def while_not_sync_standby(self, func):
+        """Runs specified action while trying to make sure that the node is not assigned synchronous standby status.
+
+        Tags us as not allowed to be a sync standby as we are going to go away, if we currently are wait for
+        leader to notice and pick an alternative one or if the leader changes or goes away we are also free.
+
+        If the connection to DCS fails we run the action anyway, as this is only a hint.
+
+        There is a small race window where this function runs between a master picking us the sync standby and
+        publishing it to the DCS. As the window is rather tiny consequences are holding up commits for one cycle
+        period we don't worry about it here."""
+
+        if not self.is_synchronous_mode() or self.patroni.nosync:
+            return func()
+
+        with self._member_state_lock:
+            self._disable_sync += 1
+        try:
+            if self.touch_member():
+                # Master should notice the updated value during the next cycle. We will wait double that, if master
+                # hasn't noticed the value by then not disabling sync replication is not likely to matter.
+                for _ in polling_loop(timeout=self.dcs.loop_wait*2, interval=2):
+                    try:
+                        if not self.is_sync_standby(self.dcs.get_cluster()):
+                            break
+                    except DCSError:
+                        logger.warning("Could not get cluster state, skipping synchronous standby disable")
+                        break
+                    logger.info("Waiting for master to release us from synchronous standby")
+            else:
+                logger.warning("Updating member state failed, skipping synchronous standby disable")
+
+            return func()
+        finally:
+            with self._member_state_lock:
+                self._disable_sync -= 1
+
     def enforce_master_role(self, message, promote_message):
         if self.state_handler.is_leader() or self.state_handler.role == 'master':
             # Inform the state handler about its master role.
             # It may be unaware of it if postgres is promoted manually.
             self.state_handler.set_role('master')
+            self.process_sync_replication()
             return message
         else:
+            if self.is_synchronous_mode():
+                # Just set ourselves as the authoritative source of truth for now. We don't want to wait for standbys
+                # to connect. We will try finding a synchronous standby in the next cycle.
+                if not self.dcs.write_sync_state(self.state_handler.name, None, index=self.cluster.sync.index):
+                    # Somebody else updated sync state, it may be due to us losing the lock. To be safe, postpone
+                    # promotion until next cycle. TODO: trigger immediate retry of run_cycle
+                    return 'Postponing promotion because synchronous replication state was updated by somebody else'
+                self.state_handler.set_synchronous_standby(None)
             self.state_handler.promote()
             return promote_message
 
@@ -299,8 +411,17 @@ class Ha(object):
         if self.cluster.failover:
             return self.manual_failover_process_no_leader()
 
-        # run usual health check
-        members = {m.name: m for m in self.cluster.members + self.old_cluster.members}
+        # When in sync mode, only last known master and sync standby are allowed to promote automatically.
+        all_known_members = self.cluster.members + self.old_cluster.members
+        if self.is_synchronous_mode() and self.cluster.sync.leader:
+            if not self.cluster.sync.matches(self.state_handler.name):
+                return False
+            # pick between synchronous candidates so we minimize unnecessary failovers/demotions
+            members = {m.name: m for m in all_known_members if self.cluster.sync.matches(m.name)}
+        else:
+            # run usual health check
+            members = {m.name: m for m in all_known_members}
+
         return self._is_healthiest_node(members.values())
 
     def demote(self, delete_leader=True):
@@ -519,10 +640,14 @@ class Ha(object):
             if prev is not None:
                 return (False, prev + ' already in progress')
 
+        do_restart = self.state_handler.restart
+        if self.is_synchronous_mode() and not self.has_lock():
+            do_restart = functools.partial(self.while_not_sync_standby, do_restart)
+
         if run_async:
-            self._async_executor.run_async(self.state_handler.restart)
+            self._async_executor.run_async(do_restart)
             return (True, 'restart initiated')
-        elif self._async_executor.run(self.state_handler.restart):
+        elif self._async_executor.run(do_restart):
             return (True, 'restarted successfully')
         else:
             return (False, 'restart failed')
@@ -592,6 +717,7 @@ class Ha(object):
 
             if not (self.cluster.is_unlocked() or self.cluster.config and self.cluster.config.data) and self.has_lock():
                 self.dcs.set_config_value(json.dumps(self.patroni.config.dynamic_configuration, separators=(',', ':')))
+                self.cluster = self.dcs.get_cluster()
 
             if self._async_executor.busy:
                 return self.handle_long_action_in_progress()

patroni/postgresql.py

@@ -83,6 +83,7 @@ class Postgresql(object):
 
         self._version_file = os.path.join(self._data_dir, 'PG_VERSION')
         self._major_version = self.get_major_version()
+        self._synchronous_standby_names = None
         self._server_parameters = self.get_server_parameters(config)
 
         self._connect_address = config.get('connect_address')
@@ -155,6 +156,11 @@ class Postgresql(object):
         parameters = config['parameters'].copy()
         listen_addresses, port = (config['listen'] + ':5432').split(':')[:2]
         parameters.update({'cluster_name': self.scope, 'listen_addresses': listen_addresses, 'port': port})
+        if config.get('synchronous_mode', False):
+            if self._synchronous_standby_names is None:
+                parameters.pop('synchronous_standby_names', None)
+            else:
+                parameters['synchronous_standby_names'] = self._synchronous_standby_names
         return {k: v for k, v in parameters.items() if not self._major_version or
                 self._major_version >= self.CMDLINE_OPTIONS.get(k, (0, 1, 9.1))[2]}
 
@@ -1066,6 +1072,49 @@ $$""".format(name, ' '.join(options)), name, password, password)
 
         return ret
 
+    def pick_synchronous_standby(self, cluster):
+        """Finds the best candidate to be the synchronous standby.
+
+        Current synchronous standby is always preferred, unless it has disconnected or does not want to be a
+        synchronous standby any longer.
+
+        :returns tuple of candidate name or None, and bool showing if the member is the active synchronous standby.
+        """
+        current = cluster.sync.sync_standby
+        members = {m.name: m for m in cluster.members}
+        candidates = []
+        # Pick candidates based on who has flushed WAL farthest.
+        # TODO: for synchronous_commit = remote_write we actually want to order on write_location
+        for app_name, state, sync_state in self.query(
+                """SELECT application_name, state, sync_state
+                    FROM pg_stat_replication
+                    ORDER BY flush_location DESC"""):
+            member = members.get(app_name)
+            if state != 'streaming' or not member or member.tags.get('nosync', False):
+                continue
+            if sync_state == 'sync':
+                return app_name, True
+            if sync_state == 'potential' and app_name == current:
+                # Prefer current even if not the best one any more to avoid indecisivness and spurious swaps.
+                return current, False
+            if sync_state == 'async':
+                candidates.append(app_name)
+
+        if candidates:
+            return candidates[0], False
+        return None, False
+
+    def set_synchronous_standby(self, name):
+        """Sets a node to be synchronous standby and if changed does a reload for PostgreSQL."""
+        if name != self._synchronous_standby_names:
+            if name is None:
+                self._server_parameters.pop('synchronous_standby_names', None)
+            else:
+                self._server_parameters['synchronous_standby_names'] = name
+            self._synchronous_standby_names = name
+            self._write_postgresql_conf()
+            self.reload()
+
     @staticmethod
     def postgres_version_to_int(pg_version):
         """ Convert the server_version to integer

patroni/utils.py

@@ -302,3 +302,14 @@ class Retry(object):
                 else:
                     self.sleep_func(sleeptime)
                 self._cur_delay = min(self._cur_delay * self.backoff, self.max_delay)
+
+
+def polling_loop(timeout, interval=1):
+    """Returns an iterator that returns values until timeout has passed. Timeout is measured from start of iteration."""
+    start_time = time.time()
+    iteration = 0
+    end_time = start_time + timeout
+    while time.time() < end_time:
+        yield iteration
+        iteration += 1
+        sleep(interval)

postgres0.yml

@@ -22,6 +22,7 @@ bootstrap:
     loop_wait: 10
     retry_timeout: 10
     maximum_lag_on_failover: 1048576
+    synchronous_mode: false
     postgresql:
       use_pg_rewind: true
 #      use_slots: true
@@ -78,3 +79,4 @@ tags:
     nofailover: false
     noloadbalance: false
     clonefrom: false
+    nosync: false

tests/test_api.py

@@ -63,6 +63,10 @@ class MockHa(object):
     def schedule_future_restart(data):
         return True
 
+    @staticmethod
+    def get_effective_tags():
+        return {'nosync': True}
+
 
 class MockPatroni(object):
 

tests/test_consul.py

@@ -31,7 +31,9 @@ def kv_get(self, key, **kwargs):
                   'Value': ('postgres://replicator:[email protected]:5433/postgres' +
                             '?application_name=http://127.0.0.1:8009/patroni').encode('utf-8')},
                  {'CreateIndex': 1085, 'Flags': 0, 'Key': key + 'optime/leader', 'LockIndex': 0,
-                  'ModifyIndex': 6429, 'Value': b'4496294792'}])
+                  'ModifyIndex': 6429, 'Value': b'4496294792'},
+                 {'CreateIndex': 1085, 'Flags': 0, 'Key': key + 'sync', 'LockIndex': 0,
+                  'ModifyIndex': 6429, 'Value': b'{"leader": "leader", "sync_standby": null}'}])
     raise ConsulException
 
 
@@ -154,3 +156,7 @@ class TestConsul(unittest.TestCase):
 
     def test_set_retry_timeout(self):
         self.c.set_retry_timeout(10)
+
+    def test_sync_state(self):
+        self.assertFalse(self.c.set_sync_state_value('{}'))
+        self.assertFalse(self.c.delete_sync_state())

tests/test_ctl.py

@@ -379,7 +379,8 @@ class TestCtl(unittest.TestCase):
     @patch('patroni.ctl.get_dcs')
     def test_list_extended(self, mock_get_dcs):
         mock_get_dcs.return_value = self.e
-        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
+        cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+        mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
 
         result = self.runner.invoke(ctl, ['list', 'dummy', '--extended'])
         assert '2100' in result.output

tests/test_etcd.py

@@ -91,6 +91,8 @@ def etcd_read(self, key, **kwargs):
                     {"key": "/service/batman5/optime/leader", "value": "2164261704",
                      "modifiedIndex": 20729, "createdIndex": 20729}],
                  "modifiedIndex": 20437, "createdIndex": 20437},
+                {"key": "/service/batman5/sync", "value": '{"leader": "leader"}',
+                 "modifiedIndex": 1582, "createdIndex": 1582},
                 {"key": "/service/batman5/members", "dir": True, "nodes": [
                     {"key": "/service/batman5/members/postgresql1",
                      "value": "postgres://replicator:[email protected]:5434/postgres" +
@@ -283,3 +285,7 @@ class TestEtcd(unittest.TestCase):
     def test_set_ttl(self):
         self.etcd.set_ttl(20)
         self.assertTrue(self.etcd.watch(1))
+
+    def test_sync_state(self):
+        self.assertFalse(self.etcd.write_sync_state('leader', None))
+        self.assertFalse(self.etcd.delete_sync_state())

tests/test_ha.py

@@ -6,7 +6,7 @@ import unittest
 
 from mock import Mock, MagicMock, PropertyMock, patch
 from patroni.config import Config
-from patroni.dcs import Cluster, Failover, Leader, Member, get_dcs
+from patroni.dcs import Cluster, ClusterConfig, Failover, Leader, Member, get_dcs, SyncState
 from patroni.dcs.etcd import Client
 from patroni.exceptions import DCSError, PostgresException
 from patroni.ha import Ha
@@ -23,15 +23,15 @@ def false(*args, **kwargs):
     return False
 
 
-def get_cluster(initialize, leader, members, failover):
-    return Cluster(initialize, None, leader, 10, members, failover)
+def get_cluster(initialize, leader, members, failover, sync):
+    return Cluster(initialize, ClusterConfig(1, {1: 2}, 1), leader, 10, members, failover, sync)
 
 
 def get_cluster_not_initialized_without_leader():
-    return get_cluster(None, None, [], None)
+    return get_cluster(None, None, [], None, SyncState(None, None, None))
 
 
-def get_cluster_initialized_without_leader(leader=False, failover=None):
+def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None):
     m1 = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres',
                                   'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4})
     l = Leader(0, 0, m1) if leader else None
@@ -41,16 +41,17 @@ def get_cluster_initialized_without_leader(leader=False, failover=None):
                                  'tags': {'clonefrom': True},
                                  'scheduled_restart': {'schedule': "2100-01-01 10:53:07.560445+00:00",
                                                        'postgres_version': '99.0.0'}})
-    return get_cluster(True, l, [m1, m2], failover)
+    syncstate = SyncState(0 if sync else None, sync and sync[0], sync and sync[1])
+    return get_cluster(True, l, [m1, m2], failover, syncstate)
 
 
-def get_cluster_initialized_with_leader(failover=None):
-    return get_cluster_initialized_without_leader(leader=True, failover=failover)
+def get_cluster_initialized_with_leader(failover=None, sync=None):
+    return get_cluster_initialized_without_leader(leader=True, failover=failover, sync=sync)
 
 
 def get_cluster_initialized_with_only_leader(failover=None):
     l = get_cluster_initialized_without_leader(leader=True, failover=failover).leader
-    return get_cluster(True, l, [l], failover)
+    return get_cluster(True, l, [l], failover, None)
 
 future_restart_time = datetime.datetime.now(pytz.utc) + datetime.timedelta(days=5)
 postmaster_start_time = datetime.datetime.now(pytz.utc)
@@ -88,6 +89,7 @@ zookeeper:
         self.replicatefrom = None
         self.api.connection_string = 'http://127.0.0.1:8008'
         self.clonefrom = None
+        self.nosync = False
         self.scheduled_restart = {'schedule': future_restart_time,
                                   'postmaster_start_time': str(postmaster_start_time)}
 
@@ -139,6 +141,7 @@ class TestHa(unittest.TestCase):
             self.ha.old_cluster = self.e.get_cluster()
             self.ha.cluster = get_cluster_not_initialized_without_leader()
             self.ha.load_cluster_from_dcs = Mock()
+            self.ha.is_synchronous_mode = false
 
     def test_update_lock(self):
         self.p.last_operation = Mock(side_effect=PostgresException(''))
@@ -510,3 +513,171 @@ class TestHa(unittest.TestCase):
         self.ha.is_paused = true
         self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
         self.assertEquals(self.ha.run_cycle(), 'PAUSE: DCS is not accessible')
+
+    @patch('patroni.ha.sleep', Mock())
+    def test_process_sync_replication(self):
+        self.ha.has_lock = true
+        mock_set_sync = self.p.set_synchronous_standby = Mock()
+        self.p.name = 'leader'
+
+        # Test sync key removed when sync mode disabled
+        self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+        with patch.object(self.ha.dcs, 'delete_sync_state') as mock_delete_sync:
+            self.ha.run_cycle()
+            mock_delete_sync.assert_called_once()
+            mock_set_sync.assert_called_once_with(None)
+
+        mock_set_sync.reset_mock()
+        # Test sync key not touched when not there
+        self.ha.cluster = get_cluster_initialized_with_leader()
+        with patch.object(self.ha.dcs, 'delete_sync_state') as mock_delete_sync:
+            self.ha.run_cycle()
+            mock_delete_sync.assert_not_called()
+            mock_set_sync.assert_called_once_with(None)
+
+        mock_set_sync.reset_mock()
+
+        self.ha.is_synchronous_mode = true
+
+        # Test sync standby not touched when picking the same node
+        self.p.pick_synchronous_standby = Mock(return_value=('other', True))
+        self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+        self.ha.run_cycle()
+        mock_set_sync.assert_not_called()
+
+        mock_set_sync.reset_mock()
+
+        # Test sync standby is replaced when switching standbys
+        self.p.pick_synchronous_standby = Mock(return_value=('other2', False))
+        self.ha.dcs.write_sync_state = Mock(return_value=True)
+        self.ha.run_cycle()
+        mock_set_sync.assert_called_once_with('other2')
+
+        mock_set_sync.reset_mock()
+        # Test sync standby is not disabled when updating dcs fails
+        self.ha.dcs.write_sync_state = Mock(return_value=False)
+        self.ha.run_cycle()
+        mock_set_sync.assert_not_called()
+
+        mock_set_sync.reset_mock()
+        # Test changing sync standby
+        self.ha.dcs.write_sync_state = Mock(return_value=True)
+        self.ha.dcs.get_cluster = Mock(return_value=get_cluster_initialized_with_leader(sync=('leader', 'other')))
+        # self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+        self.p.pick_synchronous_standby = Mock(return_value=('other2', True))
+        self.ha.run_cycle()
+        self.ha.dcs.get_cluster.assert_called_once()
+        self.assertEquals(self.ha.dcs.write_sync_state.call_count, 2)
+
+        # Test updating sync standby key failed due to race
+        self.ha.dcs.write_sync_state = Mock(side_effect=[True, False])
+        self.ha.run_cycle()
+        self.assertEquals(self.ha.dcs.write_sync_state.call_count, 2)
+
+        # Test changing sync standby failed due to race
+        self.ha.dcs.write_sync_state = Mock(return_value=True)
+        self.ha.dcs.get_cluster = Mock(return_value=get_cluster_initialized_with_leader(sync=('somebodyelse', None)))
+        self.ha.run_cycle()
+        self.assertEquals(self.ha.dcs.write_sync_state.call_count, 1)
+
+    def test_sync_replication_become_master(self):
+        self.ha.is_synchronous_mode = true
+
+        mock_set_sync = self.p.set_synchronous_standby = Mock()
+        self.p.is_leader = false
+        self.p.set_role('replica')
+        self.ha.has_lock = true
+        mock_write_sync = self.ha.dcs.write_sync_state = Mock(return_value=True)
+        self.p.name = 'leader'
+        self.ha.cluster = get_cluster_initialized_with_leader(sync=('other', None))
+
+        # When we just became master nobody is sync
+        self.assertEquals(self.ha.enforce_master_role('msg', 'promote msg'), 'promote msg')
+        mock_set_sync.assert_called_once_with(None)
+        mock_write_sync.assert_called_once_with('leader', None, index=0)
+
+        mock_set_sync.reset_mock()
+
+        # When we just became master nobody is sync
+        self.p.set_role('replica')
+        mock_write_sync.return_value = False
+        self.assertTrue(self.ha.enforce_master_role('msg', 'promote msg') != 'promote msg')
+        mock_set_sync.assert_not_called()
+
+    def test_unhealthy_sync_mode(self):
+        self.ha.is_synchronous_mode = true
+
+        self.p.is_leader = false
+        self.p.set_role('replica')
+        self.p.name = 'other'
+        self.ha.cluster = get_cluster_initialized_without_leader(sync=('leader', 'other2'))
+        mock_write_sync = self.ha.dcs.write_sync_state = Mock(return_value=True)
+        mock_acquire = self.ha.acquire_lock = Mock(return_value=True)
+        mock_follow = self.p.follow = Mock()
+        mock_promote = self.p.promote = Mock()
+
+        # If we don't match the sync replica we are not allowed to acquire lock
+        self.ha.run_cycle()
+        mock_acquire.assert_not_called()
+        mock_follow.assert_called_once()
+        self.assertEquals(mock_follow.call_args[0][0], None)
+        mock_write_sync.assert_not_called()
+
+        mock_follow.reset_mock()
+        # If we do match we will try to promote
+        self.ha._is_healthiest_node = true
+
+        self.ha.cluster = get_cluster_initialized_without_leader(sync=('leader', 'other'))
+        self.ha.run_cycle()
+        mock_acquire.assert_called_once()
+        mock_follow.assert_not_called()
+        mock_promote.assert_called_once()
+        mock_write_sync.assert_called_once_with('other', None, index=0)
+
+    @patch('patroni.utils.sleep')
+    def test_disable_sync_when_restarting(self, mock_sleep):
+        self.ha.is_synchronous_mode = true
+
+        self.p.name = 'other'
+        self.p.is_leader = false
+        self.p.set_role('replica')
+        mock_restart = self.p.restart = Mock(return_value=True)
+        self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
+        self.ha.touch_member = Mock(return_value=True)
+        self.ha.dcs.get_cluster = Mock(side_effect=[
+            get_cluster_initialized_with_leader(sync=('leader', syncstandby))
+            for syncstandby in ['other', None]])
+
+        self.ha.restart()
+
+        mock_restart.assert_called_once()
+        mock_sleep.assert_called()
+
+        # Restart is still called when DCS connection fails
+        mock_restart.reset_mock()
+        self.ha.dcs.get_cluster = Mock(side_effect=DCSError("foo"))
+        self.ha.restart()
+
+        mock_restart.assert_called_once()
+
+        # We don't try to fetch the cluster state when touch_member fails
+        mock_restart.reset_mock()
+        self.ha.dcs.get_cluster.reset_mock()
+        self.ha.touch_member = Mock(return_value=False)
+
+        self.ha.restart()
+
+        mock_restart.assert_called_once()
+        self.ha.dcs.get_cluster.assert_not_called()
+
+    def test_effective_tags(self):
+        self.ha._disable_sync = True
+        self.assertEquals(self.ha.get_effective_tags(), {'foo': 'bar', 'nosync': True})
+        self.ha._disable_sync = False
+        self.assertEquals(self.ha.get_effective_tags(), {'foo': 'bar'})
+
+    def test_restore_cluster_config(self):
+        self.ha.cluster.config.data.clear()
+        self.ha.has_lock = true
+        self.ha.cluster.is_unlocked = false
+        self.assertEquals(self.ha.run_cycle(), 'no action.  i am the leader with the lock')

tests/test_patroni.py

@@ -105,3 +105,9 @@ class TestPatroni(unittest.TestCase):
         self.p.reload_config()
         self.p.get_tags = Mock(side_effect=Exception)
         self.p.reload_config()
+
+    def test_nosync(self):
+        self.p.tags['nosync'] = True
+        self.assertTrue(self.p.nosync)
+        self.p.tags['nosync'] = None
+        self.assertFalse(self.p.nosync)

tests/test_postgresql.py

@@ -6,7 +6,7 @@ import subprocess
 import unittest
 
 from mock import Mock, MagicMock, PropertyMock, patch, mock_open
-from patroni.dcs import Cluster, Leader, Member
+from patroni.dcs import Cluster, Leader, Member, SyncState
 from patroni.exceptions import PostgresException, PostgresConnectionException
 from patroni.postgresql import Postgresql
 from patroni.utils import RetryFailedError
@@ -313,7 +313,7 @@ class TestPostgresql(unittest.TestCase):
     @patch.object(Postgresql, 'is_running', Mock(return_value=True))
     def test_sync_replication_slots(self):
         self.p.start()
-        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None)
+        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None, None)
         with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg2.OperationalError)):
             self.p.sync_replication_slots(cluster)
         self.p.sync_replication_slots(cluster)
@@ -570,3 +570,69 @@ class TestPostgresql(unittest.TestCase):
             self.assertEqual(self.p.postmaster_start_time(), 'foo')
         with patch.object(MockCursor, "execute", side_effect=psycopg2.Error):
             self.assertIsNone(self.p.postmaster_start_time())
+
+    def test_pick_sync_standby(self):
+        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None,
+                          SyncState(0, self.me.name, self.leadermem.name))
+
+        with patch.object(Postgresql, "query", return_value=[
+                    (self.leadermem.name, 'streaming', 'sync'),
+                    (self.me.name, 'streaming', 'async'),
+                    (self.other.name, 'streaming', 'async'),
+                ]):
+            self.assertEquals(self.p.pick_synchronous_standby(cluster), (self.leadermem.name, True))
+
+        with patch.object(Postgresql, "query", return_value=[
+                    (self.me.name, 'streaming', 'async'),
+                    (self.leadermem.name, 'streaming', 'potential'),
+                    (self.other.name, 'streaming', 'async'),
+                ]):
+            self.assertEquals(self.p.pick_synchronous_standby(cluster), (self.leadermem.name, False))
+
+        with patch.object(Postgresql, "query", return_value=[
+                    (self.me.name, 'streaming', 'async'),
+                    (self.other.name, 'streaming', 'async'),
+                ]):
+            self.assertEquals(self.p.pick_synchronous_standby(cluster), (self.me.name, False))
+
+        with patch.object(Postgresql, "query", return_value=[
+                    ('missing', 'streaming', 'sync'),
+                    (self.me.name, 'streaming', 'async'),
+                    (self.other.name, 'streaming', 'async'),
+                ]):
+            self.assertEquals(self.p.pick_synchronous_standby(cluster), (self.me.name, False))
+
+        with patch.object(Postgresql, "query", return_value=[]):
+            self.assertEquals(self.p.pick_synchronous_standby(cluster), (None, False))
+
+    def test_set_sync_standby(self):
+        def value_in_conf():
+            with open(os.path.join(self.data_dir, 'postgresql.conf')) as f:
+                for line in f:
+                    if line.startswith('synchronous_standby_names'):
+                        return line.strip()
+
+        mock_reload = self.p.reload = Mock()
+        self.p.set_synchronous_standby('n1')
+        self.assertEquals(value_in_conf(), "synchronous_standby_names = 'n1'")
+        mock_reload.assert_called()
+
+        mock_reload.reset_mock()
+        self.p.set_synchronous_standby('n1')
+        mock_reload.assert_not_called()
+        self.assertEquals(value_in_conf(), "synchronous_standby_names = 'n1'")
+
+        self.p.set_synchronous_standby('n2')
+        mock_reload.assert_called()
+        self.assertEquals(value_in_conf(), "synchronous_standby_names = 'n2'")
+
+        mock_reload.reset_mock()
+        self.p.set_synchronous_standby(None)
+        mock_reload.assert_called()
+        self.assertEquals(value_in_conf(), None)
+
+    def test_get_server_parameters(self):
+        config = {'synchronous_mode': True, 'parameters': {}, 'listen': '0'}
+        self.p.get_server_parameters(config)
+        self.p.set_synchronous_standby('foo')
+        self.p.get_server_parameters(config)

tests/test_zookeeper.py

@@ -50,7 +50,7 @@ class MockKazooClient(Mock):
         if path.startswith('/no_node'):
             raise NoNodeError
         elif path in ['/service/bla/', '/service/test/']:
-            return ['initialize', 'leader', 'members', 'optime', 'failover']
+            return ['initialize', 'leader', 'members', 'optime', 'failover', 'sync']
         return ['foo', 'bar', 'buzz']
 
     def create(self, path, value=b"", acl=None, ephemeral=False, sequence=False, makepath=False):
@@ -78,7 +78,7 @@ class MockKazooClient(Mock):
             raise Exception
         if path == '/service/test/members/bar' and value == b'retry':
             return
-        if path in ('/service/test/failover', '/service/test/config'):
+        if path in ('/service/test/failover', '/service/test/config', '/service/test/sync'):
             if value == b'Exception':
                 raise Exception
             elif value == b'ok':
@@ -211,3 +211,9 @@ class TestZooKeeper(unittest.TestCase):
         self.zk._client._retry.deadline = 1
         self.zk._orig_kazoo_connect = Mock(return_value=(0, 0))
         self.zk._kazoo_connect(None, None)
+
+    def test_sync_state(self):
+        self.zk.set_sync_state_value('')
+        self.zk.set_sync_state_value('ok')
+        self.zk.set_sync_state_value('Exception')
+        self.zk.delete_sync_state()