diff --git a/docs/dynamic_configuration.rst b/docs/dynamic_configuration.rst index ffcf299f..af871eec 100644 --- a/docs/dynamic_configuration.rst +++ b/docs/dynamic_configuration.rst @@ -1,46 +1,59 @@ -Patroni configuration reload -============================ +Patroni configuration +===================== -Patroni configuration should be stored in the DCS. There will be 3 types of configuration: +Patroni configuration is stored in the DCS (Distributed Configuration Store). There are 3 types of configuration: -- bootstrap configuration set in Patroni - That should be applied during the initialization time and written to etcd. - -- startup configuration (also in patroni.yml). - They should be applied during the initialization time. Unlike other options, they are not written in etcd and - any attempts to change them dynamically are blocked. - -- dynamic configuration. - Those options can be set in etcd at any time. If the options changed are not part of the startup configuration, +- Dynamic configuration. + Those options can be set in DCS at any time. If the options changed are not part of the startup configuration, they are applied asynchronously (upon the next wake up cycle) to every node, which gets subsequently reloaded. - If the node requires a restart to apply the configuration (for options with context postmaster or internal, if - their values have changed), a special flag indicating this should be set in the members.data JSON. A new API - endpoint should return whether the given node requires a restart. Additionally, the node status should also - indicate this. + If the node requires a restart to apply the configuration (for options with context postmaster, if their values + have changed), a special flag, ``pending_restart`` indicating this, is set in the members.data JSON. + Additionally, the node status also indicates this, by showing ``"restart_pending": true``. -Some options may not be increased on the master independently of the replicas (master-dependent options): +- Local `configuration `__ (patroni.yml). + Those options are defined in the configuration file and take precedence over dynamic configuration. + patroni.yml could be changed and reload in runtime (without restart of Patroni) by sending SIGHUP to the Patroni process or by performing ``POST /reload`` REST-API request. -- max_connections -- max_locks_per_transactions -- max_worker_processes -- max_prepared_transactions +- Environment `configuration ` + It is possible to set/override some of the "Local" configuration parameters with environment variables. + Environment configuration is very useful when you are running in dynamic environment and you don't know some of the parameters in advance (for example it's not possible to know you external IP address when you are running inside ``docker``). -Regarding the options that can be set, the following restrictions apply: +Some of PostgreSQL parameters must be set to the same value on master and replicas and therefore controlled only via "Dynamic configuration". Any attemt to change them via "Local configuration" are blocked: -- dynamic configuration options that are also listed in the startup configuration will not be changed, - except for the case of master-dependent options. +- max_connections: 100 +- max_locks_per_transaction: 64 +- max_worker_processes: 8 +- max_prepared_transactions: 0 +- wal_level: hot_standby +- wal_log_hints: on +- track_commit_timestamp: off -When applying the startup or dynamic configuration options, the following actions should be taken: +Although following parameters are not necessarily must be the same on master and replicas, but it is better to have them eqals, therefore they are also controlled by Patroni and can be set only via Dynamic configuration. +- max_wal_senders: 5 +- max_replication_slots: 5 +- wal_keep_segments: 8 -- The node should first check if there is a postgresql.base.conf. +Important! Patroni does some simple checks of new values obtained from DCS before applying them. Above lists containing some sane default values and it is not allowed to set new value smaller. + +There are some other PostgreSQL parameters controlled by Patroni: +- listen_addresses - is set either from ``postgresql.listen`` or from ``PATRONI_POSTGRESQL_LISTEN`` environment variable +- port - is set either from ``postgresql.listen`` or from ``PATRONI_POSTGRESQL_LISTEN`` environment variable +- cluster_name - is set either from ``scope`` or from ``PATRRONI_SCOPE`` environment variable +- hot_standby: on + +To be on the safe side parameters from the above lists are not written into ``postgresql.conf``, but passed as a list of arguments to the ``pg_ctl start`` what makes it not possible to change them even with 'ALTER SYSTEM' + + +When applying the local or dynamic configuration options, the following actions are taken: + +- The node first checks if there is a postgresql.base.conf. - If it exists, it contains the renamed "original" configuration. - If it doesn't, the original postgresql.conf is taken and renamed to postgresql.base.conf. - The dynamic options (with the exceptions above) are dumped into the postgresql.conf and an include is set in postgresql.conf to postgresql.base.conf. Therefore, we would be able to apply new options without re-reading the configuration file to check if the include is present not. -- Some parameters that are essential for Patroni to manage the cluster are overridden using the command line, these - include at least `port`, `listen_addresses`, `wal_level` +- Some parameters that are essential for Patroni to manage the cluster are overridden using the command line. - If some of the options that require restart are changed (we should look at the context in pg_settings and at the actual - values of those options), a restart_pending flag of a given node should be set. This flag is reset on any restart. + values of those options), a pending_restart flag of a given node is set. This flag is reset on any restart. Parameters would be applied in the following order (run-time are given the highest priority): @@ -49,19 +62,162 @@ Parameters would be applied in the following order (run-time are given the highe 3. load parameters from file `postgresql.auto.conf` 4. run-time parameter using `-o --name=value` -This allows configuration for all the nodes (2), configuration for a specific node using `ALTER SYSTEM` (3) and ensures that -parameters essential to the running of Patroni are enforced. (4) +This allows configuration for all the nodes (2), configuration for a specific node using `ALTER SYSTEM` (3) and ensures that parameters essential to the running of Patroni are enforced. (4) -Also, the following patroni configuration options can be changed dynamically: +Also, the following patroni configuration options can be changed only dynamically: -- ttl -- loop_wait -- retry_timeouts (to be defined first in patroni.yaml) +- ttl: 30 +- loop_wait: 10 +- retry_timeouts: 10 +- maximum_lag_on_failover: 1048576 +- postgresql.use_slots: true Upon changing those options, Patroni should read the relevant section of the configuration stored in DCS and change their run-time values. -Patroni nodes should dump the state of the DCS options to disk on startup and upon every change of the configuration. -Only master is allowed to restore those options from the on-disk dump if those are completely absent from the DCS or invalid. +Patroni nodes are dumping the state of the DCS options to disk upon every change of the configuration into file ``patroni.dynamic.json`` located in the postgres data directory. Only master is allowed to restore those options from the on-disk dump if those are completely absent from the DCS or invalid. +REST API +======== + +We are providing REST API endpoint for working with dynamic configuration. + +GET /config +----------- +Get current version of dynamic configuration. + +.. code-block:: bash + + $ curl -s localhost:8008/config | jq . + { + "ttl": 30, + "loop_wait": 10, + "retry_timeout": 10, + "maximum_lag_on_failover": 1048576, + "postgresql": { + "use_slots": true, + "use_pg_rewind": true, + "parameters": { + "hot_standby": "on", + "wal_log_hints": "on", + "wal_keep_segments": 8, + "wal_level": "hot_standby", + "max_wal_senders": 5, + "max_replication_slots": 5, + "max_connections": "100" + } + } + } + +PATCH /config +------------- +Patch existing configuration. + +.. code-block:: bash + + $ curl -s -XPATCH -d \ + '{"loop_wait":5,"ttl":20,"postgresql":{"parameters":{"max_connections":"101"}}}' \ + http://localhost:8008/config | jq . + { + "ttl": 20, + "loop_wait": 5, + "maximum_lag_on_failover": 1048576, + "retry_timeout": 10, + "postgresql": { + "use_slots": true, + "use_pg_rewind": true, + "parameters": { + "hot_standby": "on", + "wal_log_hints": "on", + "wal_keep_segments": 8, + "wal_level": "hot_standby", + "max_wal_senders": 5, + "max_replication_slots": 5, + "max_connections": "101" + } + } + } + +Above REST API call patches existing configuration and returns the new configuration. + +Let's check that node processed this configuration. First of all it should start printing logs lines every 5 seconds (loop_wait=5). Change of "max_connections" requires restart, so "restart_pending" flag should be exposed: + +.. code-block:: bash + + $ curl -s http://localhost:8008/patroni | jq . + { + "pending_restart": true, + "database_system_identifier": "6287881213849985952", + "postmaster_start_time": "2016-06-13 13:13:05.211 CEST", + "xlog": { + "location": 2197818976 + }, + "patroni": { + "scope": "batman", + "version": "1.0" + }, + "state": "running", + "role": "master", + "server_version": 90503 + } + +Removing parameters: + +If you want to remove (reset) some setting just patch it with ``null``: + +.. code-block:: bash + + $ curl -s -XPATCH -d \ + '{"postgresql":{"parameters":{"max_connections":null}}}' \ + http://localhost:8008/config | jq . + { + "ttl": 20, + "loop_wait": 5, + "retry_timeout": 10, + "maximum_lag_on_failover": 1048576, + "postgresql": { + "use_slots": true, + "use_pg_rewind": true, + "parameters": { + "hot_standby": "on", + "unix_socket_directories": ".", + "wal_keep_segments": 8, + "wal_level": "hot_standby", + "wal_log_hints": "on", + "max_wal_senders": 5, + "max_replication_slots": 5 + } + } + } + +Above call removes ``postgresql.parameters.max_connections`` from dynaminc configuration. + +PUT /config +----------- + +It's also possible to perform the full rewrite of existing dynamic configuration unconditionally: + +.. code-block:: bash + + $ curl -s -XPUT -d \ + '{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_log_hints":"on","wal_keep_segments":8,"wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \ + http://localhost:8008/config | jq . + { + "ttl": 20, + "maximum_lag_on_failover": 1048576, + "retry_timeout": 10, + "postgresql": { + "use_slots": true, + "parameters": { + "hot_standby": "on", + "unix_socket_directories": ".", + "wal_keep_segments": 8, + "wal_level": "hot_standby", + "wal_log_hints": "on", + "max_wal_senders": 5 + }, + "use_pg_rewind": true + }, + "loop_wait": 3 + }