Add pgbackrest support (#1) (#822)

* pgbackrest support

pgBackrest can restore in existing $PGDATA folder, this allows speedy restore as files which have not changed since last backup are skipped, to support this feature new param keep_data has been introduced. When keep_data=True, cleanup of $PGDATA will be skipped.

Patroni passes some extra parameters to custom_replica_methods when calling, this causes an error due to pgbackrest strict parameter checking. New param no_params=True can be set to skip parameters passing.

Fixes https://github.com/zalando/patroni/issues/625

docs/replica_bootstrap.rst

@@ -65,6 +65,19 @@ for outdated backup files. Some people prefer other backup solutions, such as ``
 others, or simply roll their own scripts. In order to accommodate all those use-cases Patroni supports running custom
 scripts to clone a new replica. Those are configured in the ``postgresql`` configuration block:
 
+.. code:: YAML
+
+    postgresql:
+        create_replica_methods:
+            - <method name>
+        <method name>:
+            command: <command name>
+            keep_data: True
+            no_params: True
+            no_master: 1
+
+example: wal_e
+
 .. code:: YAML
 
     postgresql:
@@ -79,6 +92,21 @@ scripts to clone a new replica. Those are configured in the ``postgresql`` confi
         basebackup:
             max-rate: '100M'
 
+example: pgbackrest
+
+.. code:: YAML
+
+    postgresql:
+        create_replica_methods:
+            - pgbackrest
+            - basebackup
+        pgbackrest:
+            command: /usr/bin/pgbackrest --stanza=mydb --deltarestore
+            keep_data: True
+            no_params: True
+        basebackup:
+            max-rate: '100M'
+
 
 The ``create_replica_methods`` defines available replica creation methods and the order of executing them. Patroni will
 stop on the first one that returns 0. Each method should define a separate section in the configuration file, listing the command
@@ -99,6 +127,10 @@ A special ``no_master`` parameter, if defined, allows Patroni to call the replic
 running master or replicas. In that case, an empty string will be passed in a connection string. This is useful for
 restoring the formerly running cluster from the binary backup.
 
+A special ``keep_data`` parameter, if defined, will instuct Patroni to  not clean PGDATA folder before calling restore.
+
+A special ``no_params`` parameter, if defined, restricts passing parameters to custom command.
+
 A ``basebackup`` method is a special case: it will be used if
 ``create_replica_methods`` is empty, although it is possible
 to list it explicitly among the ``create_replica_methods`` methods. This method initializes a new replica with the

patroni/ha.py

@@ -962,7 +962,8 @@ class Ha(object):
 
     def _do_reinitialize(self, cluster):
         self.state_handler.stop('immediate')
-        self.state_handler.remove_data_directory()
+        # Commented redundant data directory cleanup here
+        # self.state_handler.remove_data_directory()
 
         clone_member = self.cluster.get_clone_member(self.state_handler.name)
         member_role = 'leader' if clone_member == self.cluster.leader else 'replica'

patroni/postgresql.py

@@ -706,8 +706,10 @@ class Postgresql(object):
                     # if basebackup succeeds, exit with success
                     break
             else:
-                if not self.data_directory_empty():
+                if not self.data_directory_empty() and not self.config.get(replica_method, {}).get('keep_data', False):
                     self.remove_data_directory()
+                else:
+                    logger.info('Leaving data directory uncleaned')
 
                 cmd = replica_method
                 method_config = {}
@@ -720,10 +722,18 @@ class Postgresql(object):
                     cmd = method_config.pop('command', cmd)
 
                 # add the default parameters
-                method_config.update({"scope": self.scope,
-                                      "role": "replica",
-                                      "datadir": self._data_dir,
-                                      "connstring": connstring})
+                if not method_config.get('no_params', False):
+                    method_config.update({"scope": self.scope,
+                                          "role": "replica",
+                                          "datadir": self._data_dir,
+                                          "connstring": connstring})
+                else:
+                    if 'no_params' in method_config:
+                        del method_config['no_params']
+                    if 'no_master' in method_config:
+                        del method_config['no_master']
+                    if 'keep_data' in method_config:
+                        del method_config['keep_data']
                 params = ["--{0}={1}".format(arg, val) for arg, val in method_config.items()]
                 try:
                     # call script with the full set of parameters