Merge branch 'master' of github.com:zalando/patroni into feature/quorum-commit

This commit is contained in:
Alexander Kukushkin
2024-04-02 12:10:17 +02:00
75 changed files with 3302 additions and 1272 deletions
+9 -1
View File
@@ -14,10 +14,18 @@ Global/Universal
Log
---
- **PATRONI\_LOG\_TYPE**: sets the format of logs. Can be either **plain** or **json**. To use **json** format, you must have the :ref:`jsonlogger <extras>` installed. The default value is **plain**.
- **PATRONI\_LOG\_LEVEL**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
- **PATRONI\_LOG\_TRACEBACK\_LEVEL**: sets the level where tracebacks will be visible. Default value is **ERROR**. Set it to **DEBUG** if you want to see tracebacks only if you enable **PATRONI\_LOG\_LEVEL=DEBUG**.
- **PATRONI\_LOG\_FORMAT**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
- **PATRONI\_LOG\_FORMAT**: sets the log formatting string. If the log type is **plain**, the log format should be a string.
Refer to `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_ for
available attributes. If the log type is **json**, the log format can be a list in addition to a string. Each list
item should correspond to LogRecord attributes. Be cautious that only the field name is required, and the **%(**
and **)** should be omitted. If you wish to print a log field with a different key name, use a dictionary where
the dictionary key is the log field, and the value is the name of the field you want to be printed in the log.
Default value is **%(asctime)s %(levelname)s: %(message)s**
- **PATRONI\_LOG\_DATEFORMAT**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
- **PATRONI\_LOG\_STATIC\_FIELDS**: add additional fields to the log. This option is only available when the log type is set to **json**. Example ``PATRONI_LOG_STATIC_FIELDS="{app: patroni}"``
- **PATRONI\_LOG\_MAX\_QUEUE\_SIZE**: Patroni is using two-step logging. Log records are written into the in-memory queue and there is a separate thread which pulls them from the queue and writes to stderr or file. The maximum size of the internal queue is limited by default by **1000** records, which is enough to keep logs for the past 1h20m.
- **PATRONI\_LOG\_DIR**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this env variable, the application will retain 4 25MB logs by default. You can tune those retention values with `PATRONI_LOG_FILE_NUM` and `PATRONI_LOG_FILE_SIZE` (see below).
- **PATRONI\_LOG\_FILE\_NUM**: The number of application logs to retain.
+4
View File
@@ -108,3 +108,7 @@ Note: if cluster topology is static (fixed number of nodes that never change the
.. warning::
Permanent replication slots are synchronized only from the ``primary``/``standby_leader`` to replica nodes. That means, applications are supposed to be using them only from the leader node. Using them on replica nodes will cause indefinite growth of ``pg_wal`` on all other nodes in the cluster.
An exception to that rule are permanent physical slots that match the Patroni member names, if you happen to configure any. Those will be synchronized among all nodes as they are used for replication among them.
.. warning::
Setting ``nostream`` tag on standby disables copying and synchronization of permanent logical replication slots on the node itself and all its cascading replicas if any.
+2
View File
@@ -28,12 +28,14 @@ Currently supported PostgreSQL versions: 9.3 to 16.
patronictl
replica_bootstrap
replication_modes
standby_cluster
watchdog
pause
dcs_failsafe_mode
kubernetes
citus
existing_data
tools_integration
security
ha_multi_dc
faq
+2
View File
@@ -60,6 +60,8 @@ raft
`pysyncobj` module in order to use python Raft implementation as DCS
aws
`boto3` in order to use AWS callbacks
jsonlogger
`python-json-logger` module in order to enable :ref:`logging <log_settings>` in json format
all
all of the above (except psycopg family)
psycopg
+50
View File
@@ -3,6 +3,56 @@
Release notes
=============
Version 3.2.2
-------------
**Bugfixes**
- Don't let replica restore initialize key when DCS was wiped (Alexander Kukushkin)
It was happening in the method where Patroni was supposed to take over a standalone PG cluster.
- Use consistent read when fetching just updated sync key from Consul (Alexander Kukushkin)
Consul doesn't provide any interface to immediately get ``ModifyIndex`` for the key that we just updated, therefore we have to perform an explicit read operation. Since stale reads are allowed by default, we sometimes used to get an outdated version of the key.
- Reload Postgres config if a parameter that requires restart was reset to the original value (Polina Bungina)
Previously Patroni wasn't updating the config, but only resetting the ``pending_restart``.
- Fix erroneous inverted logic of the confirmation prompt message when doing a failover to an async candidate in synchronous mode (Polina Bungina)
The problem existed only in ``patronictl``.
- Exclude leader from failover candidates in ``patronictl`` (Polina Bungina)
If the cluster is healthy, failing over to an existing leader is no-op.
- Create Citus database and extension idempotently (Alexander Kukushkin, Zhao Junwang)
It will allow to create them in the ``post_bootstrap`` script in case if there is a need to add some more dependencies to the Citus database.
- Don't filter our contradictory ``nofailover`` tag (Polina Bungina)
The configuration ``{nofailover: false, failover_priority: 0}`` set on a node didn't allow it to participate in the race, while it should, because ``nofailover`` tag should take precedence.
- Fixed PyInstaller frozen issue (Sophia Ruan)
The ``freeze_support()`` was called after ``argparse`` and as a result, Patroni wasn't able to start Postgres.
- Fixed bug in the config generator for ``patronictl`` and ``Citus`` configuration (Israel Barth Rubio)
It prevented ``patronictl`` and ``Citus`` configuration parameters set via environment variables from being written into the generated config.
- Restore recovery GUCs and some Patroni-managed parameters when joining a running standby (Alexander Kukushkin)
Patroni was failing to restart Postgres v12 onwards with an error about missing ``port`` in one of the internal structures.
- Fixes around ``pending_restart`` flag (Polina Bungina)
Don't expose ``pending_restart`` when in custom bootstrap with ``recovery_target_action = promote`` or when someone changed ``hot_standby`` or ``wal_log_hints`` using for example ``ALTER SYSTEM``.
Version 3.2.1
-------------
+8 -64
View File
@@ -1,3 +1,5 @@
.. _replica_imaging_and_bootstrap:
Replica imaging and bootstrap
=============================
@@ -79,14 +81,13 @@ As an example, you are able to bootstrap a fresh Patroni cluster from a Barman b
method: barman
barman:
keep_existing_recovery_conf: true
command: patroni_barman_recover
api-url: https://barman-host:7480
command: patroni_barman --api-url https://barman-host:7480 recover
barman-server: my_server
ssh-command: ssh postgres@patroni-host
.. note::
``patroni_barman_recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
The above example uses a subset of the available parameters. You can get more information running ``patroni_barman_recover --help``.
``patroni_barman recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
The above example uses a subset of the available parameters. You can get more information running ``patroni_barman recover --help``.
.. _custom_replica_creation:
@@ -150,16 +151,15 @@ example: Barman
- barman
- basebackup
barman:
command: patroni_barman_recover
api-url: https://barman-host:7480
command: patroni_barman --api-url https://barman-host:7480 recover
barman-server: my_server
ssh-command: ssh postgres@patroni-host
basebackup:
max-rate: '100M'
.. note::
``patroni_barman_recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
The above example uses a subset of the available parameters. You can get more information running ``patroni_barman_recover --help``.
``patroni_barman recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
The above example uses a subset of the available parameters. You can get more information running ``patroni_barman recover --help``.
The ``create_replica_methods`` defines available replica creation methods and the order of executing them. Patroni will
stop on the first one that returns 0. Each method should define a separate section in the configuration file, listing the command
@@ -219,59 +219,3 @@ and
- waldir: /pg-wal-mount/external-waldir
If all replica creation methods fail, Patroni will try again all methods in order during the next event loop cycle.
.. _standby_cluster:
Standby cluster
---------------
Another available option is to run a "standby cluster", that contains only of
standby nodes replicating from some remote node. This type of clusters has:
* "standby leader", that behaves pretty much like a regular cluster leader,
except it replicates from a remote node.
* cascade replicas, that are replicating from standby leader.
Standby leader holds and updates a leader lock in DCS. If the leader lock
expires, cascade replicas will perform an election to choose another leader
from the standbys.
There is no further relationship between the standby cluster and the primary
cluster it replicates from, in particular, they must not share the same DCS
scope if they use the same DCS. They do not know anything else from each other
apart from replication information. Also, the standby cluster is not being
displayed in :ref:`patronictl_list` or :ref:`patronictl_topology` output on the
primary cluster.
For the sake of flexibility, you can specify methods of creating a replica and
recovery WAL records when a cluster is in the "standby mode" by providing
`create_replica_methods` key in `standby_cluster` section. It is distinct from
creating replicas, when cluster is detached and functions as a normal cluster,
which is controlled by `create_replica_methods` in `postgresql` section. Both
"standby" and "normal" `create_replica_methods` reference keys in `postgresql`
section.
To configure such cluster you need to specify the section ``standby_cluster``
in a patroni configuration:
.. code:: YAML
bootstrap:
dcs:
standby_cluster:
host: 1.2.3.4
port: 5432
primary_slot_name: patroni
create_replica_methods:
- basebackup
Note, that these options will be applied only once during cluster bootstrap,
and the only way to change them afterwards is through DCS.
Patroni expects to find `postgresql.conf` or `postgresql.conf.backup` in PGDATA
of the remote primary and will not start if it does not find it after a
basebackup. If the remote primary keeps its `postgresql.conf` elsewhere, it is
your responsibility to copy it to PGDATA.
If you use replication slots on the standby cluster, you must also create the corresponding replication slot on the primary cluster. It will not be done automatically by the standby cluster implementation. You can use Patroni's permanent replication slots feature on the primary cluster to maintain a replication slot with the same name as ``primary_slot_name``, or its default value if ``primary_slot_name`` is not provided.
+1 -1
View File
@@ -56,7 +56,7 @@ are available. As a downside, the primary is not be available for writes
blocking all client write requests until at least one synchronous replica comes
up.
You can ensure that a standby never becomes the synchronous standby by setting ``nosync`` tag to true. This is recommended to set for standbys that are behind slow network connections and would cause performance degradation when becoming a synchronous standby.
You can ensure that a standby never becomes the synchronous standby by setting ``nosync`` tag to true. This is recommended to set for standbys that are behind slow network connections and would cause performance degradation when becoming a synchronous standby. Setting tag ``nostream`` to true will also have the same effect.
Synchronous mode can be switched on and off using ``patronictl edit-config`` command or via Patroni REST interface. See :ref:`dynamic configuration <dynamic_configuration>` for instructions.
+78
View File
@@ -0,0 +1,78 @@
.. _standby_cluster:
Standby cluster
---------------
Patroni also support running cascading replication to a remote datacenter
(region) using a feature that is called "standby cluster". This type of
clusters has:
* "standby leader", that behaves pretty much like a regular cluster leader,
except it replicates from a remote node.
* cascade replicas, that are replicating from standby leader.
Standby leader holds and updates a leader lock in DCS. If the leader lock
expires, cascade replicas will perform an election to choose another leader
from the standbys.
There is no further relationship between the standby cluster and the primary
cluster it replicates from, in particular, they must not share the same DCS
scope if they use the same DCS. They do not know anything else from each other
apart from replication information. Also, the standby cluster is not being
displayed in :ref:`patronictl_list` or :ref:`patronictl_topology` output on the
primary cluster.
For the sake of flexibility, you can specify methods of creating a replica and
recovery WAL records when a cluster is in the "standby mode" by providing
:ref:`create_replica_methods <custom_replica_creation>` key in
`standby_cluster` section. It is distinct from creating replicas, when cluster
is detached and functions as a normal cluster, which is controlled by
`create_replica_methods` in `postgresql` section. Both "standby" and "normal"
`create_replica_methods` reference keys in `postgresql` section.
To configure such cluster you need to specify the section ``standby_cluster``
in a patroni configuration:
.. code:: YAML
bootstrap:
dcs:
standby_cluster:
host: 1.2.3.4
port: 5432
primary_slot_name: patroni
create_replica_methods:
- basebackup
Note, that these options will be applied only once during cluster bootstrap,
and the only way to change them afterwards is through DCS.
Patroni expects to find `postgresql.conf` or `postgresql.conf.backup` in PGDATA
of the remote primary and will not start if it does not find it after a
basebackup. If the remote primary keeps its `postgresql.conf` elsewhere, it is
your responsibility to copy it to PGDATA.
If you use replication slots on the standby cluster, you must also create the
corresponding replication slot on the primary cluster. It will not be done
automatically by the standby cluster implementation. You can use Patroni's
permanent replication slots feature on the primary cluster to maintain a
replication slot with the same name as ``primary_slot_name``, or its default
value if ``primary_slot_name`` is not provided.
In case the remote site doesn't provide a single endpoint that connects to a
primary, one could list all hosts of the source cluster in the
``standby_cluster.host`` section. When ``standby_cluster.host`` contains
multiple hosts separated by commas, Patroni will:
* add ``target_session_attrs=read-write`` to the ``primary_conninfo`` on the
standby leader node.
* use ``target_session_attrs=read-write`` when trying to determine whether we
need to run ``pg_rewind`` or when executing ``pg_rewind`` on all nodes of the
standby cluster.
There is also a possibility to replicate the standby cluster from another
standby cluster or from a standby member of the primary cluster: for that, you
need to define a single host in the ``standby_cluster.host`` section. However,
you need to beware that in this case ``pg_rewind`` will fail to execute on the
standby cluster.
+62
View File
@@ -0,0 +1,62 @@
Integration with other tools
============================
Patroni is able to integrate with other tools in your stack. In this section you
will find a list of examples, which although not an exhaustive list, might
provide you with ideas on how Patroni can integrate with other tools.
Barman
------
Patroni delivers an application named ``patroni_barman`` which has logic to
communicate with ``pg-backup-api``, so you are able to perform Barman operations
remotely.
This application currently has a couple of sub-commands: ``recover`` and
``config-switch``.
patroni_barman recover
^^^^^^^^^^^^^^^^^^^^^^
The ``recover`` sub-command can be used as a custom bootstrap or custom replica
creation method. You can find more information about that in
:ref:`replica_imaging_and_bootstrap`.
patroni_barman config-switch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``config-switch`` sub-command is designed to be used as an ``on_role_change``
callback in Patroni. As an example, assume you are streaming WALs from your
current primary to your Barman host. In the event of a failover in the cluster
you might want to start streaming WALs from the new primary. You can accomplish
this by using ``patroni_barman config-switch`` as the ``on_role_change`` callback.
.. note::
That sub-command relies on the ``barman config-switch`` command, which is in
charge of overriding the configuration of a Barman server by applying a
pre-defined model on top of it. This command is available since Barman 3.10.
Please consult the Barman documentation for more details.
This is an example of how you can configure Patroni to apply a configuration
model in case this Patroni node is promoted to primary:
.. code:: YAML
postgresql:
callbacks:
on_role_change: >
patroni_barman
--api-url YOUR_API_URL
config-switch
--barman-server YOUR_BARMAN_SERVER_NAME
--barman-model YOUR_BARMAN_MODEL_NAME
--switch-when promoted
.. note::
``patroni_barman config-switch`` requires that you have both Barman and
``pg-backup-api`` configured in the Barman host, so it can execute a remote
``barman config-switch`` through the backup API. Also, it requires that you
have pre-configured Barman models to be applied. The above example uses a
subset of the available parameters. You can get more information running
``patroni_barman config-switch --help``, and by consulting the Barman
documentation.
+26 -1
View File
@@ -11,12 +11,22 @@ Global/Universal
- **namespace**: path within the configuration store where Patroni will keep information about the cluster. Default value: "/service"
- **scope**: cluster name
.. _log_settings:
Log
---
- **type**: sets the format of logs. Can be either **plain** or **json**. To use **json** format, you must have the :ref:`jsonlogger <extras>` installed. The default value is **plain**.
- **level**: sets the general logging level. Default value is **INFO** (see `the docs for Python logging <https://docs.python.org/3.6/library/logging.html#levels>`_)
- **traceback\_level**: sets the level where tracebacks will be visible. Default value is **ERROR**. Set it to **DEBUG** if you want to see tracebacks only if you enable **log.level=DEBUG**.
- **format**: sets the log formatting string. Default value is **%(asctime)s %(levelname)s: %(message)s** (see `the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_)
- **format**: sets the log formatting string. If the log type is **plain**, the log format should be a string. Refer to
`the LogRecord attributes <https://docs.python.org/3.6/library/logging.html#logrecord-attributes>`_ for
available attributes. If the log type is **json**, the log format can be a list in addition to a string. Each list
item should correspond to LogRecord attributes. Be cautious that only the field name is required, and the **%(**
and **)** should be omitted. If you wish to print a log field with a different key name, use a dictionary where
the dictionary key is the log field, and the value is the name of the field you want to be printed in the log.
Default value is **%(asctime)s %(levelname)s: %(message)s**
- **dateformat**: sets the datetime formatting string. (see the `formatTime() documentation <https://docs.python.org/3.6/library/logging.html#logging.Formatter.formatTime>`_)
- **static_fields**: add additional fields to the log. This option is only available when the log type is set to **json**.
- **max\_queue\_size**: Patroni is using two-step logging. Log records are written into the in-memory queue and there is a separate thread which pulls them from the queue and writes to stderr or file. The maximum size of the internal queue is limited by default by **1000** records, which is enough to keep logs for the past 1h20m.
- **dir**: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this value, the application will retain 4 25MB logs by default. You can tune those retention values with `file_num` and `file_size` (see below).
- **file\_num**: The number of application logs to retain.
@@ -26,6 +36,20 @@ Log
- **patroni.postmaster: WARNING**
- **urllib3: DEBUG**
Here is an example of how to config patroni to log in json format.
.. code:: YAML
log:
type: json
format:
- message
- module
- asctime: '@timestamp'
- levelname: level
static_fields:
app: patroni
.. _bootstrap_settings:
Bootstrap configuration
@@ -375,6 +399,7 @@ Tags
- **nosync**: ``true`` or ``false``. If set to ``true`` the node will never be selected as a synchronous replica.
- **nofailover**: ``true`` or ``false``, controls whether this node is allowed to participate in the leader race and become a leader. Defaults to ``false``, meaning this node _can_ participate in leader races.
- **failover_priority**: integer, controls the priority that this node should have during failover. Nodes with higher priority will be preferred over lower priority nodes if they received/replayed the same amount of WAL. However, nodes with higher values of receive/replay LSN are preferred regardless of their priority. If the ``failover_priority`` is 0 or negative - such node is not allowed to participate in the leader race and to become a leader (similar to ``nofailover: true``).
- **nostream**: ``true`` or ``false``. If set to ``true`` the node will not use replication protocol to stream WAL. It will rely instead on archive recovery (if ``restore_command`` is configured) and ``pg_wal``/``pg_xlog`` polling. It also disables copying and synchronization of permanent logical replication slots on the node itself and all its cascading replicas. Setting this tag on primary node has no effect.
.. warning::
Provide only one of ``nofailover`` or ``failover_priority``. Providing ``nofailover: true`` is the same as ``failover_priority: 0``, and providing ``nofailover: false`` will give the node priority 1.