Merge pull request #281 from CartoDB/feature/add_custom_conf_location

Add configuration parameter to specify a path to a custom postgresql.base.conf and disable its backup

docs/SETTINGS.rst

@@ -69,6 +69,7 @@ PostgreSQL
 -  **listen**: IP address + port that Postgres listens to; must be accessible from other nodes in the cluster, if you're using streaming replication. Multiple comma-separated addresses are permitted, as long as the port component is appended after to the last one with a colon, i.e. ``listen: 127.0.0.1,127.0.0.2:5432``. Patroni will use the first address from this list to establish local connections to the PostgreSQL node.
 -  **pgpass**: path to the `.pgpass <https://www.postgresql.org/docs/current/static/libpq-pgpass.html>`__ password file. Patroni creates this file before executing pg\_basebackup and under some other circumstances. The location must be writable by Patroni.
 -  **recovery\_conf**: additional configuration settings written to recovery.conf when configuring follower.
+-  **custom_conf** : path to an optional custom ``postgresql.conf`` file, that will be used in place of ``postgresql.base.conf``. The file must exist on all cluster nodes, be readable by PostgreSQL and will be included from its location on the real ``postgresql.conf``. Note that Patroni will not monitor this file for changes, nor backup it. However, its settings can still be overriden by Patroni's own configuration facilities - see `dynamic configuration <https://github.com/zalando/patroni/blob/master/docs/dynamic_configuration.rst>`__ for details.
 -  **parameters**: list of configuration settings for Postgres. Many of these are required for replication to work.
 -  **pg\_ctl\_timeout**: How long should pg_ctl wait when doing ``start``, ``stop`` or ``restart``. Default value is 60 seconds.
 -  **use\_pg\_rewind**: try to use pg\_rewind on the former leader when it joins cluster as a replica.

docs/dynamic_configuration.rst

@@ -48,23 +48,24 @@ To be on the safe side parameters from the above lists are not written into ``po
 
 When applying the local or dynamic configuration options, the following actions are taken:
 
-- The node first checks if there is a postgresql.base.conf.
-- If it exists, it contains the renamed "original" configuration.
-- If it doesn't, the original postgresql.conf is taken and renamed to postgresql.base.conf.
+- The node first checks if there is a postgresql.base.conf or if the ``custom_conf`` parameter is set.
+- If the `custom_conf` parameter is set, it will take the file specified on it as a base configuration, ignoring `postgresql.base.conf` and `postgresql.conf`.
+- If the `custom_conf` parameter is not set and `postgresql.base.conf` exists, it contains the renamed "original" configuration and it will be used as a base configuration.
+- If there is no `custom_conf` nor `postgresql.base.conf`, the original postgresql.conf is taken and renamed to postgresql.base.conf.
 - The dynamic options (with the exceptions above) are dumped into the postgresql.conf and an include is set in
-  postgresql.conf to postgresql.base.conf. Therefore, we would be able to apply new options without re-reading the configuration file to check if the include is present not.
+  postgresql.conf to the used base configuration (either postgresql.base.conf or what is on ``custom_conf``). Therefore, we would be able to apply new options without re-reading the configuration file to check if the include is present not.
 - Some parameters that are essential for Patroni to manage the cluster are overridden using the command line.
 - If some of the options that require restart are changed (we should look at the context in pg_settings and at the actual
   values of those options), a pending_restart flag of a given node is set. This flag is reset on any restart.
 
 The parameters would be applied in the following order (run-time are given the highest priority):
 
-1. load parameters from file `postgresql.base.conf`
+1. load parameters from file `postgresql.base.conf` (or from a `custom_conf` file, if set)
 2. load parameters from file `postgresql.conf`
 3. load parameters from file `postgresql.auto.conf`
 4. run-time parameter using `-o --name=value`
 
-This allows configuration for all the nodes (2), configuration for a specific node using `ALTER SYSTEM` (3) and ensures that parameters essential to the running of Patroni are enforced. (4)
+This allows configuration for all the nodes (2), configuration for a specific node using `ALTER SYSTEM` (3) and ensures that parameters essential to the running of Patroni are enforced (4), as well as leaves room for configuration tools that manage `postgresql.conf` directly without involving Patroni (1).
 
 
 Also, the following Patroni configuration options can be changed only dynamically:

patroni/postgresql.py

@@ -100,8 +100,6 @@ class Postgresql(object):
         self._postgresql_base_conf_name = config_base_name + '.base.conf'
         self._postgresql_base_conf = os.path.join(self._data_dir, self._postgresql_base_conf_name)
         self._recovery_conf = os.path.join(self._data_dir, 'recovery.conf')
-        self._configuration_to_save = (self._postgresql_conf, self._postgresql_base_conf,
-                                       os.path.join(self._data_dir, 'pg_hba.conf'))
         self._postmaster_pid = os.path.join(self._data_dir, 'postmaster.pid')
         self._trigger_file = config.get('recovery_conf', {}).get('trigger_file') or 'promote'
         self._trigger_file = os.path.abspath(os.path.join(self._data_dir, self._trigger_file))
@@ -123,6 +121,15 @@ class Postgresql(object):
             self.set_role('master' if self.is_leader() else 'replica')
             self._write_postgresql_conf()  # we are "joining" already running postgres
 
+    @property
+    def _configuration_to_save(self):
+        configuration = [self._postgresql_conf]
+        if 'custom_conf' not in self.config:
+            configuration.append(self._postgresql_base_conf)
+        if not self.config['parameters'].get('hba_file'):
+            configuration.append(os.path.join(self._data_dir, 'pg_hba.conf'))
+        return configuration
+
     @property
     def use_slots(self):
         return self._use_slots and self._major_version >= 9.4
@@ -625,12 +632,12 @@ class Postgresql(object):
 
     def _write_postgresql_conf(self):
         # rename the original configuration if it is necessary
-        if not os.path.exists(self._postgresql_base_conf):
+        if 'custom_conf' not in self.config and not os.path.exists(self._postgresql_base_conf):
             os.rename(self._postgresql_conf, self._postgresql_base_conf)
 
         with open(self._postgresql_conf, 'w') as f:
             f.write('# Do not edit this file manually!\n# It will be overwritten by Patroni!\n')
-            f.write("include '{0}'\n\n".format(self._postgresql_base_conf_name))
+            f.write("include '{0}'\n\n".format(self.config.get('custom_conf') or self._postgresql_base_conf_name))
             for name, value in sorted(self._server_parameters.items()):
                 if name not in self.CMDLINE_OPTIONS:
                     f.write("{0} = '{1}'\n".format(name, value))