Compare commits

..
28 Commits
Author SHA1 Message Date
Alexander Kukushkin 710afd5952 Release v3.1.2 (#2885)
- bump version
- update release notes
2023-09-26 12:31:18 +02:00
Alexander Kukushkin 096ee8f36f Read GUC's values when joining running Postgres (#2876)
If restarted in pause Patroni was discarding `synchronous_standby_names` from `postgresql.conf` because in the internal cache this values was set to `None`. As a result synchronous replication transitioned to a broken state, with no synchronous replicas according to the `synchronous_standby_names` and Patroni not selecting/setting the new synchronous replicas (another bug).

To solve the problem of broken initial state and to avoid similar issues with other GUC's we will read GUC's value if Patroni is joining running Postgres.
2023-09-26 11:34:19 +02:00
Alexander Kukushkin 91e2be092c Detect and solve inconsistency between /sync and actual sync nodes (#2877)
Patroni is changing `synchronous_standby_names` and the `/sync` key in a very specific order, first we add nodes to `synchronous_standby_names` and only after, when they are recognized as synchronous they are added to the `/sync` key. When removing nodes the order is different: they are first removed from the `/sync` key and only after that from the `synchronous_standby_names`.

As a result Patroni expects that either actual synchronous nodes will match with the nodes listed in the `/sync` key or that new candidates to synchronous nodes will not match with nodes listed in the `/sync` key. In case if `synchronous_standby_names` was removed from the `postgresql.conf`, manually, or due the the bug (#2876), the state becomes inconsistent because of the wrong order of updates.

To solve inconsistent state we introduce additional checks and will update the `/sync` key with actual names of synchronous nodes (usually empty set).
2023-09-26 11:17:12 +02:00
Alexander Kukushkin 4148e0b5b2 Take into account current role when deciding on removal of member ZNode (#2884)
Patroni doesn't watch on all changes of member keys in order to not create too much load on ZooKeeper, but only subscribes to changes (ZNodes added or deleted) in the `/member` directory. Therefore when some important fields in the value are updated we remove and recreate ZNode in order to notify the leader or other members.

The leader should remove the member key only when the `checkpoint_after_promote` value is changed and replicas when the `state` is changed to/from `running`.

We don't care about the `version` field, because Patroni version can't be changed without restart, what will case ZooKeeper `session_id` to change it anyway.

This fix hopefully will reduce failures of behave tests on GH Actions.
2023-09-26 11:16:52 +02:00
Alexander Kukushkin 5ceba81269 Bugfix for GUC's values with units (#2883)
Despite being validated by `IntValidator` some GUC's couldn't be casted directly to `int` because they include suffix. Example: `128MB`.

Close https://github.com/zalando/patroni/issues/2879
2023-09-26 11:16:47 +02:00
Alexander Kukushkin dbbe065a27 Silence annoying warnings when checking for node uniqueness (#2878)
WARNING messages are produced by `urllib3` if Patroni is quickly restarted.
Instead we will check that the node is listen on a given port. This fact is actually enough to detect names clashes, while HTTP request could raise an exception is a few other cases, what might case false negatives.

Close https://github.com/zalando/patroni/issues/2881
2023-09-26 11:16:38 +02:00
Alexander Kukushkin 4a4a7dab45 Update supported Postgres versions (#2857) 2023-09-20 15:05:53 +02:00
Alexander Kukushkin 2f8d0f9662 Stick with sphinx_rtd_theme (#2873)
by default they are using something else
2023-09-20 14:59:35 +02:00
Polina BunginaandAlexander Kukushkin 40f9c02606 Pin sphinx_rtd_theme to >1 (#2825)
Earlier versions are incompatible with sphinx>7
2023-09-20 14:58:35 +02:00
Alexander Kukushkin ce51eb02d0 Mock request() method when running tests (#2802)
1. Unit tests should not really try accessing any resources.
2. Not doing so results in significant execution time of unit tests on Windows

In addition to that perform a request with timeout 3s. Usually this is more than enough to figure out whether resource is accessible.

Followup on #2724
2023-09-20 14:26:12 +02:00
Matt BakerandAlexander Kukushkin e796198045 Generate API docs from code with sphinx autodoc (#2699)
Expanding on the addition of docstrings in code, this adds python module API docs to sphinx documentation.

A developer can preview what this might look like by running this locally:

```
tox -m docs
```

The option `-W` is added to the tox env so that warning messages are considered errors.

Adds doc generation using the above method to the test GitHub workflow to catch documentation problems on PRs.

Some docstrings have been reformatted and fixed to satisfy errors generated with the above setup.
2023-09-20 12:35:14 +02:00
Matt BakerandAlexander Kukushkin dde2331160 Add docs to patroni.dcs.__init__.py (#2777)
Also, made some small code changes to satisfy formatting and pylint.
2023-09-20 12:31:55 +02:00
Matt BakerandAlexander Kukushkin bcafe91a55 Add docstrings to patroni.postgresql.slots.py (#2778)
Also, made some small code changes to satisfy formatting and pylint.
2023-09-20 12:30:57 +02:00
Alexander Kukushkin f7e99749ef Release v3.1.1 (#2872)
* Bump version
* Update release notes
* Update tox.ini (include v16)
* Enable tests for `REL*` branches
2023-09-20 12:13:52 +02:00
IsraelandAlexander Kukushkin b31f590700 patronictl --help was showing ctl function's docstring (#2845)
`patronictl` is implemented using `click` module, and that module uses the functions' docstrings for creating a helper text.

As a consequence the docstring for `ctl` function was being shown to the user, which doesn't make sense.

This PR fixes that issue by adding a user-friendly description to be shown on `patronictl --help`. We use a `\f` to tell `click` when to stop capturing text to show in the helper.

Note that `patronictl` commands are implemented using `@ctl.command` decorator, and we always provide them with `help` argument. That said, none of the subcommands are affected by the aforementioned issue, only the entry point of the CLI.

References: PAT-201.
2023-09-20 12:13:42 +02:00
Alexander Kukushkin 9d7e7174fc Bump pyright version (#2871)
and fix all reported issues.

We aren't sticking to the latest version this time because it has [a bug](https://github.com/microsoft/pyright/issues/5968).
2023-09-20 12:13:23 +02:00
Alexander Kukushkin 6e82de8751 Don't rely on pg_stat_wal_receiver when deciding on pg_rewind (#2863)
As was reported by @ants on Slack it could happen that `received_tli` is ahead of replayed timeline, therefore we should stop using it when deciding on pg_rewind if postgres is running and use only `IDENTIFY_SYSTEM` via replication connection.
2023-09-18 13:01:08 +02:00
Polina BunginaandAlexander Kukushkin 85db209c19 Always store CMDLINE_OPTIONS config values as int (#2861) 2023-09-18 12:59:55 +02:00
IsraelandAlexander Kukushkin 564dd7e7af Fix bug in patronictl query command (#2859)
Previous to this commit `patronictl query` was working only if `-r` argument was provided to the command. Otherwise it would face issues:

* If neither `-r` nor `-m` were provided:

```
 PGPASSWORD=zalando patronictl -c postgres0.yml query -U postgres -c "SHOW PORT"
2023-09-12 17:45:38	No connection to role=None is available
```

* If only `-m` was provided:

```
$ PGPASSWORD=zalando patronictl -c postgres0.yml query -U postgres -c "SHOW PORT" -m postgresql0
2023-09-12 17:46:15	No connection to member postgresql0 is available
```

This issue was a regression introduced by `4c3e0b9382820524239d2aa4d6b95379ef1291db` through PR #2687.

Through that PR we decided to move the common logic used to check mutually exclusiveness of `--role` and `--member` arguments to `get_any_member` function.

However, previous to that change `role` variable would assume the default value of `any` in `query` method, before `get_any_member` was called, which was not the case after the change.

This commit fixes that issue by adding a handler in `get_cursor` function to `role=None`. As `role` defaulting to `any` is handled in a sub-call to `get_any_member`, we are apparently safe in `get_cursor` to return the cursor if `role=None`.

Unit tests were updated accordingly.

References: PAT-204.
2023-09-18 12:59:55 +02:00
Alexander Kukushkin 95ed90183b Don't start stopped postgres in pause (#2848)
Due to a race condition Patroni was falsely assuming that the standby should be restarted because some recovery parameters (primary_conninfo or similar) were changed.

Close https://github.com/zalando/patroni/issues/2834
2023-09-18 12:59:55 +02:00
Alexander Kukushkin 33e02e14ca Override write_leader_optime method in K8s implementation (#2850)
It is being called when postgres is already shut down cleanly but there are no healthy replicas to take it over.

Close https://github.com/zalando/patroni/issues/2837
Close https://github.com/zalando/patroni/pull/2838
2023-09-18 12:59:55 +02:00
Polina BunginaandAlexander Kukushkin e38d7d4e9f Return system id to the ctl list title (#2840) 2023-09-18 12:59:55 +02:00
Alexander Kukushkin 24bf2f3fa0 Fix bug with kubernetes.standby_leader_label_value (#2832)
When running with the leader lock Patroni was just setting the `role` label to `master` and effectively `kubernetes.standby_leader_label_value` feature never worked.

Now it is fixed, but in order to not introduce breaking changes we just update default value of the `standby_leader_label_value` to the `master`.
2023-09-18 12:59:55 +02:00
Alexander Kukushkin 1849bd1a56 Don't return logical slots for standby cluster (#2816)
Cluster.get_replication_slots() didn't take into account that there can not be logical replication slots in a standby cluster replicas. It was only skipping logical slots for the standby_leader, but replicas were expecting that they will have to copy them over.

Also on replicas in a standby cluster these logical slots were falsely added to the `_replication_slots` dict.
2023-09-18 12:59:55 +02:00
IsraelandAlexander Kukushkin 37643b5a8b Fix IntValidator regarding validation of value 0 (#2818)
Previous to this commit `IntValidator` would always consider the value `0` invalid, even if in the allowed range.

The problem was that `parse_int` was returning `0` in the following line:

```python
value = parse_int(value, self.base_unit) or ""
```

However the `or ""` was evaluating to an empty string.

As `parse_int` returns either an `int` if able to parse, or `None` otherwise, the `isinstance(value, int)` is enough to error out when not a valid `int`.

Closes #2817
2023-09-18 12:59:54 +02:00
Alexander KukushkinandPolina Bungina f659edd60f Explicitly enable synchronous mode (#2820)
Close https://github.com/zalando/patroni/issues/2819

Co-authored-by: Polina Bungina <[email protected]>
2023-09-18 12:59:36 +02:00
Alexander Kukushkin 6d548aefbe Silence useless warnings in patronictl (#2808)
Close https://github.com/zalando/patroni/issues/2805
2023-09-18 12:59:18 +02:00
ChenChangAoandAlexander Kukushkin 783112385f reset failsafe state when promote (#2803)
consider the scenario(enable failsafe_mode):

0. node1(primary) - node2(replica)
1. stop all etcd nodes; wait ttl seconds; start all etcd nodes; (node2's failsafe will contain the info about node1)
2. switchover to node2; (node2's failsafe still contain the info about node1)
3. stop all etcd nodes; wait ttl seconds; start all etcd nodes;
4. node2 will demote because it consider node1 as primary

Resetting failsafe state when running as a primary fixes the issue.
2023-09-18 12:59:17 +02:00
98 changed files with 1875 additions and 7160 deletions
+2 -2
View File
@@ -45,8 +45,8 @@ def install_packages(what):
packages['exhibitor'] = packages['zookeeper']
packages = packages.get(what, [])
ver = versions.get(what)
if float(ver) >= 15:
packages += ['postgresql-{0}-citus-12.1'.format(ver)]
if float(ver) == 15:
packages += ['postgresql-{0}-citus-12.0'.format(ver)]
subprocess.call(['sudo', 'apt-get', 'update', '-y'])
return subprocess.call(['sudo', 'apt-get', 'install', '-y', 'postgresql-' + ver, 'expect-dev'] + packages)
+1 -4
View File
@@ -24,11 +24,8 @@ jobs:
- name: Run tests and flake8
run: python .github/workflows/run_tests.py
- name: Install Python packaging build frontend
run: python -m pip install build
- name: Build a binary wheel and a source tarball
run: python -m build
run: python setup.py sdist bdist_wheel
- name: Publish distribution to Test PyPI
if: github.event_name == 'push'
+1 -1
View File
@@ -174,7 +174,7 @@ jobs:
- uses: jakebailey/pyright-action@v1
with:
version: 1.1.333
version: 1.1.326
docs:
runs-on: ubuntu-latest
+1 -6
View File
@@ -33,7 +33,7 @@ nosetests.xml
coverage.xml
htmlcov
junit.xml
features/output*
features/output
dummy
# Translations
@@ -48,12 +48,10 @@ pgpass
scm-source.json
# Sphinx-generated documentation
docs/_build/
docs/build/
docs/source/_static/
docs/source/_templates/
docs/modules/
docs/pdf/
# Pycharm IDE
.idea/
@@ -66,6 +64,3 @@ venv*/
# Default test data directory
data/
# macOS
**/.DS_Store
+21 -13
View File
@@ -74,11 +74,27 @@ There are a few options available:
::
sudo apt-get install python3-psycopg2 # install psycopg2 module on Debian/Ubuntu
sudo yum install python3-psycopg2 # install psycopg2 on RedHat/Fedora/CentOS
sudo apt-get install python-psycopg2 # install python2 psycopg2 module on Debian/Ubuntu
sudo apt-get install python3-psycopg2 # install python3 psycopg2 module on Debian/Ubuntu
sudo yum install python-psycopg2 # install python2 psycopg2 on RedHat/Fedora/CentOS
2. Specify one of `psycopg`, `psycopg2`, or `psycopg2-binary` in the list of dependencies when installing Patroni with pip (see below).
2. Install psycopg2 from the binary package
::
pip install psycopg2-binary
3. Install psycopg2 from source
::
pip install psycopg2>=2.5.4
4. Use psycopg 3.0 instead of psycopg2
::
pip install psycopg[binary]
**General installation for pip**
@@ -104,20 +120,12 @@ raft
`pysyncobj` module in order to use python Raft implementation as DCS
aws
`boto3` in order to use AWS callbacks
all
all of the above (except psycopg family)
psycopg3
`psycopg[binary]>=3.0.0` module
psycopg2
`psycopg2>=2.5.4` module
psycopg2-binary
`psycopg2-binary` module
For example, the command in order to install Patroni together with psycopg3, dependencies for Etcd as a DCS, and AWS callbacks is:
For example, the command in order to install Patroni together with dependencies for Etcd as a DCS and AWS callbacks is:
::
pip install patroni[psycopg3,etcd3,aws]
pip install patroni[etcd,aws]
Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed independently of Patroni.
+1 -1
View File
@@ -78,7 +78,7 @@ Example session:
| demo | patroni3 | 172.22.0.4 | | running | 1 | 0 |
+---------+----------+------------+--------+---------+----+-----------+
postgres@patroni1:~$ etcdctl get --keys-only --prefix /service/demo
postgres@patroni1:~$ etcdctl ls --recursive --sort -p /service/demo
/service/demo/config
/service/demo/initialize
/service/demo/leader
BIN
View File
Binary file not shown.
+12 -3
View File
@@ -24,6 +24,15 @@ Log
- **PATRONI\_LOG\_FILE\_SIZE**: Size of patroni.log file (in bytes) that triggers a log rolling.
- **PATRONI\_LOG\_LOGGERS**: Redefine logging level per python module. Example ``PATRONI_LOG_LOGGERS="{patroni.postmaster: WARNING, urllib3: DEBUG}"``
Bootstrap configuration
-----------------------
It is possible to create new database users right after the successful initialization of a new cluster. This process is defined by the following variables:
- **PATRONI\_<username>\_PASSWORD='<password>'**
- **PATRONI\_<username>\_OPTIONS='list,of,options'**
Example: defining ``PATRONI_admin_PASSWORD=strongpasswd`` and ``PATRONI_admin_OPTIONS='createrole,createdb'`` will cause creation of the user **admin** with the password **strongpasswd** that is allowed to create other users and databases.
Citus
-----
Enables integration Patroni with `Citus <https://docs.citusdata.com>`__. If configured, Patroni will take care of registering Citus worker nodes on the coordinator. You can find more information about Citus support :ref:`here <citus>`.
@@ -200,10 +209,10 @@ REST API
CTL
---
- **PATRONICTL\_CONFIG\_FILE**: (optional) location of the configuration file.
- **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
- **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
- **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
- **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
- **PATRONI\_CTL\_INSECURE**: (optional) Allow connections to REST API without verifying SSL certs.
- **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
- **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
- **PATRONI\_CTL\_CERTFILE**: (optional) Specifies the file with the client certificate in the PEM format.
- **PATRONI\_CTL\_KEYFILE**: (optional) Specifies the file with the client secret key in the PEM format.
- **PATRONI\_CTL\_KEYFILE\_PASSWORD**: (optional) Specifies a password for decrypting the client keyfile.
+81 -1
View File
@@ -25,7 +25,83 @@ We report new releases information :ref:`here <releases>`.
Technical Requirements/Installation
-----------------------------------
Go :ref:`here <installation>` for guidance on installing and upgrading Patroni on various platforms.
**Pre-requirements for Mac OS**
To install requirements on a Mac, run the following:
::
brew install postgresql etcd haproxy libyaml python
.. _psycopg2_install_options:
**Psycopg**
Starting from `psycopg2-2.8 <http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/>`__ the binary version of psycopg2 will no longer be installed by default. Installing it from the source code requires C compiler and postgres+python dev packages.
Since in the python world it is not possible to specify dependency as ``psycopg2 OR psycopg2-binary`` you will have to decide how to install it.
There are a few options available:
1. Use the package manager from your distro
::
sudo apt-get install python-psycopg2 # install python2 psycopg2 module on Debian/Ubuntu
sudo apt-get install python3-psycopg2 # install python3 psycopg2 module on Debian/Ubuntu
sudo yum install python-psycopg2 # install python2 psycopg2 on RedHat/Fedora/CentOS
2. Install psycopg2 from the binary package
::
pip install psycopg2-binary
3. Install psycopg2 from source
::
pip install psycopg2>=2.5.4
4. Use psycopg 3.0 instead of psycopg2
::
pip install psycopg[binary]>=3.0.0
**General installation for pip**
Patroni can be installed with pip:
::
pip install patroni[dependencies]
where dependencies can be either empty, or consist of one or more of the following:
etcd or etcd3
`python-etcd` module in order to use Etcd as Distributed Configuration Store (DCS)
consul
`python-consul` module in order to use Consul as DCS
zookeeper
`kazoo` module in order to use Zookeeper as DCS
exhibitor
`kazoo` module in order to use Exhibitor as DCS (same dependencies as for Zookeeper)
kubernetes
`kubernetes` module in order to use Kubernetes as DCS in Patroni
raft
`pysyncobj` module in order to use python Raft implementation as DCS
aws
`boto3` in order to use AWS callbacks
For example, the command in order to install Patroni together with dependencies for Etcd as a DCS and AWS callbacks is:
::
pip install patroni[etcd,aws]
Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed
independently of Patroni.
.. _running_configuring:
@@ -89,6 +165,10 @@ Applications Should Not Use Superusers
When connecting from an application, always use a non-superuser. Patroni requires access to the database to function properly. By using a superuser from an application, you can potentially use the entire connection pool, including the connections reserved for superusers, with the ``superuser_reserved_connections`` setting. If Patroni cannot access the Primary because the connection pool is full, behavior will be undesirable.
.. |Build Status| image:: https://travis-ci.org/zalando/patroni.svg?branch=master
:target: https://travis-ci.org/zalando/patroni
.. |Coverage Status| image:: https://coveralls.io/repos/zalando/patroni/badge.svg?branch=master
:target: https://coveralls.io/r/zalando/patroni?branch=master
Testing Your HA Solution
--------------------------------------
+11 -15
View File
@@ -38,18 +38,14 @@ After that you just need to start Patroni and it will handle the rest:
2. If ``max_prepared_transactions`` isn't explicitly set in the global
:ref:`dynamic configuration <dynamic_configuration>` Patroni will
automatically set it to ``2*max_connections``.
3. The ``citus.local_hostname`` GUC value will be adjusted from ``localhost`` to the
value that Patroni is using in order to connect to the local PostgreSQL
instance. The value sometimes should be different from the ``localhost``
because PostgreSQL might be not listening on it.
4. The ``citus.database`` will be automatically created followed by ``CREATE EXTENSION citus``.
5. Current superuser :ref:`credentials <postgresql_settings>` will be added to the ``pg_dist_authinfo``
3. The ``citus.database`` will be automatically created followed by ``CREATE EXTENSION citus``.
4. Current superuser :ref:`credentials <postgresql_settings>` will be added to the ``pg_dist_authinfo``
table to allow cross-node communication. Don't forget to update them if
later you decide to change superuser username/password/sslcert/sslkey!
6. The coordinator primary node will automatically discover worker primary
5. The coordinator primary node will automatically discover worker primary
nodes and add them to the ``pg_dist_node`` table using the
``citus_add_node()`` function.
7. Patroni will also maintain ``pg_dist_node`` in case failover/switchover
6. Patroni will also maintain ``pg_dist_node`` in case failover/switchover
on the coordinator or worker clusters occurs.
patronictl
@@ -61,7 +57,7 @@ clusters that are just logically groupped together using the
PostgreSQL. Therefore in most cases it is not possible to manage them as a
single entity.
It results in two major differences in :ref:`patronictl` behaviour when
It results in two major differences in ``patronictl`` behaviour when
``patroni.yaml`` has the ``citus`` section comparing with the usual:
1. The ``list`` and the ``topology`` by default output all members of the Citus
@@ -69,12 +65,12 @@ It results in two major differences in :ref:`patronictl` behaviour when
which Citus group they belong to.
2. For all ``patronictl`` commands the new option is introduced, named
``--group``. For some commands the default value for the group might be
taken from the ``patroni.yaml``. For example, :ref:`patronictl_pause` will
taken from the ``patroni.yaml``. For example, ``patronictl pause`` will
enable the maintenance mode by default for the ``group`` that is set in the
``citus`` section, but for example for :ref:`patronictl_switchover` or
:ref:`patronictl_remove` the group must be explicitly specified.
``citus`` section, but for example for ``patronictl switchover`` or
``patronictl remove`` the group must be explicitly specified.
An example of :ref:`patronictl_list` output for the Citus cluster::
An example of ``patronictl list`` output for the Citus cluster::
postgres@coord1:~$ patronictl list demo
+ Citus cluster: demo ----------+--------------+---------+----+-----------+
@@ -119,7 +115,7 @@ the coordinator for the shards hosted on a worker node. The switchover then
happens while the traffic is kept on the coordinator, and resumes as soon as a
new primary worker node is ready to accept read-write queries.
An example of :ref:`patronictl_switchover` on the worker cluster::
An example of ``patronictl switchover`` on the worker cluster::
postgres@coord1:~$ patronictl switchover demo
+ Citus cluster: demo ----------+--------------+---------+----+-----------+
@@ -347,7 +343,7 @@ Citus upgrades and PostgreSQL major upgrades
First, please read about upgrading Citus version in the `documentation`__.
There is one minor change in the process. When executing upgrade, you have to
use :ref:`patronictl_restart` instead of ``systemctl restart`` to restart
use ``patronictl restart`` instead of ``systemctl restart`` to restart
PostgreSQL.
__ https://docs.citusdata.com/en/latest/admin_guide/upgrading_citus.html
-16
View File
@@ -54,13 +54,6 @@ apidoc_output_dir = 'modules'
apidoc_excluded_paths = excludes
apidoc_separate_modules = True
# Include autodoc for all members, including private ones and the ones that are missing a docstring.
autodoc_default_options = {
"members": True,
"undoc-members": True,
"private-members": True,
}
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
@@ -287,14 +280,6 @@ def doctree_read(app, doctree):
toc_tree_node['entries'].remove(e)
def autodoc_skip(app, what, name, obj, would_skip, options):
"""Include autodoc of ``__init__`` methods, which are skipped by default."""
if name == "__init__":
return False
return would_skip
# A possibility to have an own stylesheet, to add new rules or override existing ones
# For the latter case, the CSS specificity of the rules should be higher than the default ones
def setup(app):
@@ -307,4 +292,3 @@ def setup(app):
app.connect('builder-inited', builder_inited)
app.connect('env-get-outdated', env_get_outdated)
app.connect('doctree-read', doctree_read)
app.connect("autodoc-skip-member", autodoc_skip)
+18 -22
View File
@@ -3,25 +3,21 @@
Contributing guidelines
=======================
Wanna contribute to Patroni? Yay - here is how!
Chatting
--------
If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
Reporting bugs
--------------
Before reporting a bug please make sure to **reproduce it with the latest Patroni version**!
Also please double check if the issue already exists in our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
Running tests
-------------
Requirements for running behave tests:
#. PostgreSQL packages including `contrib <https://www.postgresql.org/docs/current/contrib.html>`__ modules need to be installed.
#. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
#. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
1. PostgreSQL packages need to be installed.
2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
Install dependencies:
@@ -43,13 +39,6 @@ After you have all dependencies installed, you can run the various test suites:
# Run the pytest suite in tests/:
python setup.py test
# Moreover, you may want to run tests in different scopes for debugging purposes,
# the -s option include print output during test execution.
# Tests in pytest typically follow the pattern: FILEPATH::CLASSNAME::TESTNAME.
pytest -s tests/test_api.py
pytest -s tests/test_api.py::TestRestApiHandler
pytest -s tests/test_api.py::TestRestApiHandler::test_do_GET
# Run the behave (https://behave.readthedocs.io/en/latest/) test suite in features/;
# modify DCS as desired (raft has no dependencies so is the easiest to start with):
DCS=raft python -m behave
@@ -154,12 +143,12 @@ If you want to disable this facility set the env var `OPEN_CMD` to the `:` no-op
Behave tests
^^^^^^^^^^^^
Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 16 and then run all
Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 15 and then run all
behave tests. This can take quite a long time to run so you might want to limit the scope to a select version of
Postgres or to a specific feature set or steps.
To specify the version of postgres include the full name of the dependent image build env that you want and then the
behave env name. For instance if you want Postgres 14 use:
behave env name. For instance if you want Postgres 15 use:
.. code-block:: bash
@@ -174,12 +163,19 @@ the watchdog behave feature test scenario with all versions of Postgres.
Of course you can combine the two.
Reporting issues
----------------
If you have a question about patroni or have a problem using it, please read the :ref:`README <readme>` before filing an issue.
Also double check with the current issues on our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Contributing a pull request
---------------------------
#. Fork the repository, develop and test your code changes.
#. Reflect changes in the user documentation.
#. Submit a pull request with a clear description of the changes objective. Link an existing issue if necessary.
1) Submit a comment to the relevant issue or create a new issue describing your proposed change.
2) Do a fork, develop and test your code changes.
3) Include documentation
4) Submit a pull request.
You'll get feedback about your pull request as soon as possible.
+1 -1
View File
@@ -60,4 +60,4 @@ F.A.Q.
- How to enable the Failsafe Mode?
Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or :ref:`patronictl edit-config -s failsafe_mode=true <patronictl_edit_config_parameters>`
Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or ``patronictl edit-config -s failsafe_mode=true``
+7 -34
View File
@@ -6,20 +6,11 @@ Dynamic Configuration Settings
Dynamic configuration is stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
In order to change the dynamic configuration you can use either :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
- **loop\_wait**: the number of seconds the loop will sleep. Default value: 10, minimum possible value: 1
- **ttl**: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30, minimum possible value: 20
- **retry\_timeout**: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10, minimum possible value: 3
.. warning::
when changing values of **loop_wait**, **retry_timeout**, or **ttl** you have to follow the rule:
.. code-block:: python
loop_wait + 2 * retry_timeout <= ttl
In order to change the dynamic configuration you can use either ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
- **loop\_wait**: the number of seconds the loop will sleep. Default value: 10
- **ttl**: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30
- **retry\_timeout**: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10
- **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
- **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
- **max\_timelines\_history**: maximum number of timeline history items kept in DCS. Default value: 0. When set to 0, it keeps the full history in DCS.
@@ -55,9 +46,9 @@ In order to change the dynamic configuration you can use either :ref:`patronictl
- **archive\_cleanup\_command**: cleanup command for standby leader
- **recovery\_min\_apply\_delay**: how long to wait before actually apply WAL records on a standby leader
- **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Permanent slots that don't exist will be created by Patroni. With PostgreSQL 11 onwards permanent physical slots are created on all nodes and their position is advanced every **loop_wait** seconds. For PostgreSQL versions older than 11 permanent physical replication slots are maintained only on the current primary. The logical slots are copied from the primary to a standby with restart, and after that their position advanced every **loop_wait** seconds (if necessary). Copying logical slot files performed via ``libpq`` connection and using either rewind or superuser credentials (see **postgresql.authentication** section). There is always a chance that the logical slot position on the replica is a bit behind the former primary, therefore application should be prepared that some messages could be received the second time after the failover. The easiest way of doing so - tracking ``confirmed_flush_lsn``. Enabling permanent replication slots requires **postgresql.use_slots** to be set to ``true``. If there are permanent logical replication slots defined Patroni will automatically enable the ``hot_standby_feedback``. Since the failover of logical replication slots is unsafe on PostgreSQL 9.6 and older and PostgreSQL version 10 is missing some important functions, the feature only works with PostgreSQL 11+.
- **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Permanent slots that don't exist will be created by Patroni. The physical slots are maintained only in the current primary. The logical slots are copied from the primary to a standby with restart, and after that their position advanced every **loop_wait** seconds (if necessary). Copying logical slot files performed via ``libpq`` connection and using either rewind or superuser credentials (see **postgresql.authentication** section). There is always a chance that the logical slot position on the replica is a bit behind the former primary, therefore application should be prepared that some messages could be received the second time after the failover. The easiest way of doing so - tracking ``confirmed_flush_lsn``. Enabling permanent logical replication slots requires **postgresql.use_slots** to be set and will also automatically enable the ``hot_standby_feedback``. Since the failover of logical replication slots is unsafe on PostgreSQL 9.6 and older and PostgreSQL version 10 is missing some important functions, the feature only works with PostgreSQL 11+.
- **my\_slot\_name**: the name of the permanent replication slot. If the permanent slot name matches with the name of the current node it will not be created on this node. If you add a permanent physical replication slot which name matches the name of a Patroni member, Patroni will ensure that the slot that was created is not removed even if the corresponding member becomes unresponsive, situation which would normally result in the slot's removal by Patroni. Although this can be useful in some situations, such as when you want replication slots used by members to persist during temporary failures or when importing existing members to a new Patroni cluster (see :ref:`Convert a Standalone to a Patroni Cluster <existing_data>` for details), caution should be exercised by the operator that these clashes in names are not persisted in the DCS, when the slot is no longer required, due to its effect on normal functioning of Patroni.
- **my\_slot\_name**: the name of the permanent replication slot. If the permanent slot name matches with the name of the current leader it will not be created. Please note that Patroni does not make checks for permanent slot names added to this configuration matching those that Patroni creates automatically for members. If those names are added, Patroni will ensure that any slots that were created are not removed even if the member becomes unresponsive, situation which would normally result in the slot's removal by Patroni. Although this can be useful in some situations, such as when importing existing members to a new Patroni cluster (see :ref:`Convert a Standalone to a Patroni Cluster <existing_data>` for details), caution should be exercised by the operator that these clashes in names are not persisted in the DCS due to its effect on normal functioning of Patroni.
- **type**: slot type. Could be ``physical`` or ``logical``. If the slot is logical, you have to additionally define ``database`` and ``plugin``.
- **database**: the database name where logical slots should be created.
@@ -89,22 +80,4 @@ Note: **slots** is a hashmap while **ignore_slots** is an array. For example:
plugin: test_decoding
- name: ignored_physical_slot_name
type: physical
...
Note: if cluster topology is static (fixed number of nodes that never change their names) you can configure permanent physical replication slots with names corresponding to names of nodes to avoid recycling of WAL files while replica is temporary down:
.. code:: YAML
slots:
node_name1:
type: physical
node_name2:
type: physical
node_name3:
type: physical
...
.. warning::
Permanent replication slots are synchronized only from the ``primary``/``standby_leader`` to replica nodes. That means, applications are supposed to be using them only from the leader node. Using them on replica nodes will cause indefinite growth of ``pg_wal`` on all other nodes in the cluster.
An exception to that rule are permanent physical slots that match the Patroni member names, if you happen to configure any. Those will be synchronized among all nodes as they are used for replication among them.
...
+4 -4
View File
@@ -36,18 +36,18 @@ You can find below an overview of steps for converting an existing Postgres clus
#. If you are running Postgres through systemd, then disable the Postgres systemd unit. This is performed as Patroni manages starting and stopping the Postgres daemon.
#. Create a YAML configuration file for Patroni. You can use :ref:`Patroni configuration generation and validation tooling <validate_generate_config>` for that.
#. Create a YAML configuration file for Patroni.
* **Note (specific for the primary node):** If you have replication slots being used for replication between cluster members, then it is recommended that you enable ``use_slots`` and configure the existing replication slots as permanent via the ``slots`` configuration item. Be aware that Patroni automatically creates replication slots for replication between members, and drops replication slots that it does not recognize, when ``use_slots`` is enabled. The idea of using permanent slots here is to allow your existing slots to persist while the migration to Patroni is in progress. See :ref:`YAML Configuration Settings <yaml_configuration>` for details.
#. Start Patroni using the ``patroni`` systemd service unit. It automatically detects that Postgres is already running and starts monitoring the instance.
#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through :ref:`patronictl restart cluster-name member-name <patronictl_restart_parameters>` command. For minimal downtime you might want to split this step into:
#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through ``patronictl restart cluster-name member-name`` command. For minimal downtime you might want to split this step into:
#. Immediate restart of the standby nodes.
#. Scheduled restart of the primary node within a maintenance window.
#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through :ref:`patronictl edit-config cluster-name member-name <patronictl_edit_config_parameters>` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through ``patronictl edit-config cluster-name member-name`` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
.. code-block:: sql
@@ -73,7 +73,7 @@ The only possible way to do a major upgrade currently is:
#. Stop Patroni
#. Upgrade PostgreSQL binaries and perform `pg_upgrade <https://www.postgresql.org/docs/current/pgupgrade.html>`_ on the primary node
#. Update patroni.yml
#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running :ref:`patronictl remove cluster-name <patronictl_remove_parameters>` . It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running ``patronictl remove <cluster-name>``. It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
#. If you wiped the cluster state in the previous step, you may wish to copy patroni.dynamic.json from old data dir to the new one. It will help you to retain some PostgreSQL parameters you had set before.
#. Start Patroni on the primary node.
#. Upgrade PostgreSQL binaries, update patroni.yml and wipe the data_dir on standby nodes.
-2
View File
@@ -22,10 +22,8 @@ Currently supported PostgreSQL versions: 9.3 to 16.
:caption: Contents:
README
installation
patroni_configuration
rest_api
patronictl
replica_bootstrap
replication_modes
watchdog
-196
View File
@@ -1,196 +0,0 @@
.. _installation:
Installation
============
Pre-requirements for Mac OS
---------------------------
To install requirements on a Mac, run the following:
.. code-block:: shell
brew install postgresql etcd haproxy libyaml python
.. _psycopg2_install_options:
Psycopg
-------
Starting from `psycopg2-2.8`_ the binary version of psycopg2 will no longer be installed by default. Installing it from
the source code requires C compiler and postgres+python dev packages. Since in the python world it is not possible to
specify dependency as ``psycopg2 OR psycopg2-binary`` you will have to decide how to install it.
There are a few options available:
1. Use the package manager from your distro
.. code-block:: shell
sudo apt-get install python3-psycopg2 # install psycopg2 module on Debian/Ubuntu
sudo yum install python3-psycopg2 # install psycopg2 on RedHat/Fedora/CentOS
2. Specify one of `psycopg`, `psycopg2`, or `psycopg2-binary` in the :ref:`list of dependencies <extras>` when installing Patroni with pip.
.. _extras:
General installation for pip
----------------------------
Patroni can be installed with pip:
.. code-block:: shell
pip install patroni[dependencies]
where ``dependencies`` can be either empty, or consist of one or more of the following:
etcd or etcd3
`python-etcd` module in order to use Etcd as Distributed Configuration Store (DCS)
consul
`python-consul` module in order to use Consul as DCS
zookeeper
`kazoo` module in order to use Zookeeper as DCS
exhibitor
`kazoo` module in order to use Exhibitor as DCS (same dependencies as for Zookeeper)
kubernetes
`kubernetes` module in order to use Kubernetes as DCS in Patroni
raft
`pysyncobj` module in order to use python Raft implementation as DCS
aws
`boto3` in order to use AWS callbacks
all
all of the above (except psycopg family)
psycopg
`psycopg[binary]>=3.0.0` module
psycopg2
`psycopg2>=2.5.4` module
psycopg2-binary
`psycopg2-binary` module
For example, the command in order to install Patroni together with psycopg3, dependencies for Etcd as a DCS, and AWS callbacks is:
.. code-block:: shell
pip install patroni[psycopg3,etcd3,aws]
Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed
independently of Patroni.
.. _package_installation:
Package installation on Linux
-----------------------------
Patroni packages may be available for your operating system, produced by the Postgres community for:
* RHEL, RockyLinux, AlmaLinux;
* Debian and Ubuntu;
* SUSE Enterprise Linux.
You can also find packages for direct dependencies of Patroni, like python modules that might not be available in
the official operating system repositories.
For more information see the `PGDG repository`_ documentation.
If you are on a RedHat Enterprise Linux derivative operating system you may also require packages from EPEL, see
`EPEL repository`_ documentation.
Once you have installed the PGDG repository for your OS you can install patroni.
.. note::
Patroni packages are not maintained by the Patroni developers, but rather by the Postgres community. If you
require support please first try connecting on `Postgres slack`_.
Installing on Debian derivatives
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
With PGDG repo installed, see :ref:`above <package_installation>`, install Patroni via apt run:
.. code-block:: shell
apt-get install patroni
Installing on RedHat derivatives
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
With PGDG repo installed, see :ref:`above <package_installation>`, install patroni with an etcd DCS via dnf on RHEL 9
(and derivatives) run:
.. code-block:: shell
dnf install patroni patroni-etcd
You can install etcd from PGDG if your RedHat derivative distribution does not provide packages. On the nodes that will
host the DCS run:
.. code-block:: shell
dnf install 'dnf-command(config-manager)'
dnf config-manager --enable pgdg-rhel9-extras
dnf install etcd
You can replace the version of RHEL with `8` in the repo to make `pgdg-rhel8-extras` if needed. The repo name is still
`pgdg-rhelN-extras` on RockyLinux, AlmaLinux, Oracle Linux, etc...
Installing on SUSE Enterprise Linux
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You might need to enable the SUSE PackageHub repositories for some dependencies. see `SUSE PackageHub`_ documentation.
For SLES 15 with PGDG repo installed, see :ref:`above <package_installation>`, you can install patroni using:
.. code-block:: shell
zypper install patroni patroni-etcd
With the SUSE PackageHub repo enabled you can also install etcd:
.. code-block:: shell
SUSEConnect -p PackageHub/15.5/x86_64
zypper install etcd
Upgrading
---------
Upgrading patroni is a very simple process, just update the software installation and restart the Patroni daemon on
each node in the cluster.
However, restarting the Patroni daemon will result in a Postgres database restart. In some situations this may cause
a failover of the primary node in your cluster, therefore it is recommended to put the cluster into maintenance mode
until the Patroni daemon restart has been completed.
To put the cluster in maintenance mode, run the following command on one of the patroni nodes:
.. code-block:: shell
patronictl pause --wait
Then on each node in the cluster, perform the package upgrade required for your OS:
.. code-block:: shell
apt-get update && apt-get install patroni patroni-etcd
Restart the patroni daemon process on each node:
.. code-block:: shell
systemctl restart patroni
Then finally resume monitoring of Postgres with patroni to take it out of maintenance mode:
.. code-block:: shell
patronictl resume --wait
The cluster will now be full operational with the new version of Patroni.
.. _psycopg2-2.8: http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/
.. _PGDG repository: https://www.postgresql.org/download/linux/
.. _EPEL repository: https://docs.fedoraproject.org/en-US/epel/
.. _SUSE PackageHub: https://packagehub.suse.com/how-to-use/
.. _Postgres slack: http://pgtreats.info/slack-invite
+11 -152
View File
@@ -15,7 +15,7 @@ There are 3 types of Patroni configuration:
- Global :ref:`dynamic configuration <dynamic_configuration>`.
These options are stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
Dynamic configuration can be set at any time using :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
Dynamic configuration can be set at any time using ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
If the options changed are not part of the startup configuration, they are applied asynchronously (upon the next wake up cycle)
to every node, which gets subsequently reloaded.
If the node requires a restart to apply the configuration (for `PostgreSQL parameters <https://www.postgresql.org/docs/current/view-pg-settings.html>`__ with context postmaster, if their values
@@ -24,7 +24,7 @@ There are 3 types of Patroni configuration:
- Local :ref:`configuration file <yaml_configuration>` (patroni.yml).
These options are defined in the configuration file and take precedence over dynamic configuration.
``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing :ref:`patronictl_reload`. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing ``patronictl reload``. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
- :ref:`Environment configuration <environment>`.
It is possible to set/override some of the "Local" configuration parameters with environment variables.
@@ -44,6 +44,7 @@ Some of the PostgreSQL parameters **must hold the same values on the primary and
- **max_worker_processes**: 8
- **max_prepared_transactions**: 0
- **wal_level**: hot_standby
- **wal_log_hints**: on
- **track_commit_timestamp**: off
For the parameters below, PostgreSQL does not require equal values among the primary and all the replicas. However, considering the possibility of a replica to become the primary at any time, it doesn't really make sense to set them differently; therefore, **Patroni restricts setting their values to the** :ref:`dynamic configuration <dynamic_configuration>`.
@@ -61,7 +62,6 @@ There are some other Postgres parameters controlled by Patroni:
- **port** - is set either from ``postgresql.listen`` or from ``PATRONI_POSTGRESQL_LISTEN`` environment variable
- **cluster_name** - is set either from ``scope`` or from ``PATRONI_SCOPE`` environment variable
- **hot_standby: on**
- **wal_log_hints: on** - for Postgres 9.4 and newer.
To be on the safe side parameters from the above lists are not written into ``postgresql.conf``, but passed as a list of arguments to the ``pg_ctl start`` which gives them the highest precedence, even above `ALTER SYSTEM <https://www.postgresql.org/docs/current/static/sql-altersystem.html>`__
@@ -70,16 +70,15 @@ There also are some parameters like **postgresql.listen**, **postgresql.data_dir
When applying the local or dynamic configuration options, the following actions are taken:
- The node first checks if there is a `postgresql.base.conf` file or if the ``custom_conf`` parameter is set.
- If the ``custom_conf`` parameter is set, the file it specifies is used as the base configuration, ignoring `postgresql.base.conf` and `postgresql.conf`.
- If the ``custom_conf`` parameter is not set and `postgresql.base.conf` exists, it contains the renamed "original" configuration and is used as the base configuration.
- If there is no ``custom_conf`` nor `postgresql.base.conf`, the original `postgresql.conf` is renamed to `postgresql.base.conf` and used as the base configuration.
- The dynamic options (with the exceptions above) are dumped into the `postgresql.conf` and an include is set in
`postgresql.conf` to the base configuration (either `postgresql.base.conf` or the file at ``custom_conf``).
Therefore, we would be able to apply new options without re-reading the configuration file to check if the include is present or not.
- The node first checks if there is a `postgresql.base.conf` or if the ``custom_conf`` parameter is set.
- If the ``custom_conf`` parameter is set, it will take the file specified on it as a base configuration, ignoring `postgresql.base.conf` and `postgresql.conf`.
- If the ``custom_conf`` parameter is not set and `postgresql.base.conf` exists, it contains the renamed "original" configuration and it will be used as a base configuration.
- If there is no ``custom_conf``` nor `postgresql.base.conf`, the original `postgresql.conf`` is taken and renamed to postgresql.base.conf.
- The dynamic options (with the exceptions above) are dumped into the `postgresql.conf`` and an include is set in
postgresql.conf to the used base configuration (either `postgresql.base.conf` or what is on ``custom_conf``). Therefore, we would be able to apply new options without re-reading the configuration file to check if the include is present not.
- Some parameters that are essential for Patroni to manage the cluster are overridden using the command line.
- If an option that requires restart is changed (we should look at the context in pg_settings and at the actual
values of those options), a pending_restart flag is set on that node. This flag is reset on any restart.
- If some of the options that require restart are changed (we should look at the context in pg_settings and at the actual
values of those options), a pending_restart flag of a given node is set. This flag is reset on any restart.
The parameters would be applied in the following order (run-time are given the highest priority):
@@ -91,42 +90,6 @@ The parameters would be applied in the following order (run-time are given the h
This allows configuration for all the nodes (2), configuration for a specific node using ``ALTER SYSTEM`` (3) and ensures that parameters essential to the running of Patroni are enforced (4), as well as leaves room for configuration tools that manage `postgresql.conf` directly without involving Patroni (1).
PostgreSQL parameters that touch shared memory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PostgreSQL has some parameters that determine the size of the shared memory used by them:
- **max_connections**
- **max_prepared_transactions**
- **max_locks_per_transaction**
- **max_wal_senders**
- **max_worker_processes**
Changing these parameters require a PostgreSQL restart to take effect, and their shared memory structures cannot be smaller on the standby nodes than on the primary node.
As explained before, Patroni restrict changing their values through :ref:`dynamic configuration <dynamic_configuration>`, which usually consists of:
1. Applying changes through :ref:`patronictl_edit_config` (or via REST API ``/config`` endpoint)
2. Restarting nodes through :ref:`patronictl_restart` (or via REST API ``/restart`` endpoint)
**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through :ref:`patronictl_restart` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
However, as those settings manage shared memory, some extra care should be taken when restarting the nodes:
* If you want to **increase** the value of any of those settings:
1. Restart all standbys first
2. Restart the primary after that
* If you want to **decrease** the value of any of those settings:
1. Restart the primary first
2. Restart all standbys after that
**Note:** if you attempt to restart all nodes in one go after **decreasing** the value of any of those settings, Patroni will ignore the change and restart the standby with the original setting value, thus requiring that you restart the standbys again later. Patroni does that to prevent the standby to enter in an infinite crash loop, because PostgreSQL quits with a `FATAL` message if you attempt to set any of those parameters to a value lower than what is visible in ``pg_controldata`` on the Standby node. In other words, we can only decrease the setting on the standby once its ``pg_controldata`` is up-to-date with the primary in regards to these changes on the primary.
More information about that can be found at `PostgreSQL Administrator's Overview <https://www.postgresql.org/docs/current/hot-standby.html#HOT-STANDBY-ADMIN>`__.
Patroni configuration parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -143,107 +106,3 @@ Also the following Patroni configuration options **can be changed only dynamical
Upon changing these options, Patroni will read the relevant section of the configuration stored in DCS and change its run-time values.
Patroni nodes are dumping the state of the DCS options to disk upon for every change of the configuration into the file ``patroni.dynamic.json`` located in the Postgres data directory. Only the leader is allowed to restore these options from the on-disk dump if these are completely absent from the DCS or if they are invalid.
.. _validate_generate_config:
Configuration generation and validation
---------------------------------------
Patroni provides command-line interfaces for a Patroni :ref:`local configuration <yaml_configuration>` generation and validation. Using the ``patroni`` executable you can:
- Create a sample local Patroni configuration;
- Create a Patroni configuration file for the locally running PostgreSQL instance (e.g. as a preparation step for the :ref:`Patroni integration <existing_data>`);
- Validate a given Patroni configuration file.
Sample Patroni configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: text
patroni --generate-sample-config [configfile]
Description
"""""""""""
Generate a sample Patroni configuration file in ``yaml`` format.
Parameter values are defined using the :ref:`Environment configuration <environment>`, otherwise, if not set, the defaults used in Patroni or the ``#FIXME`` string for the values that should be later defined by the user.
Some default values are defined based on the local setup:
- **postgresql.listen**: the IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``5432`` port.
- **postgresql.connect_address**: the IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``5432`` port.
- **postgresql.authentication.rewind**: is only defined if the PostgreSQL version can be defined from the binary and the version is 11 or later.
- **restapi.listen**: IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``8008`` port.
- **restapi.connect_address**: IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``8008`` port.
Parameters
""""""""""
``configfile`` - full path to the configuration file used to store the result. If not provided, the result is sent to ``stdout``.
Patroni configuration for a running instance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: text
patroni --generate-config [--dsn DSN] [configfile]
Description
"""""""""""
Generate a Patroni configuration in ``yaml`` format for the locally running PostgreSQL instance.
Either the provided DSN (takes precedence) or PostgreSQL `environment variables <https://www.postgresql.org/docs/current/libpq-envars.html>`__ will be used for the PostgreSQL connection. If the password is not provided, it should be entered via prompt.
All the non-internal GUCs defined in the source Postgres instance, independently if they were set through a configuration file, through the postmaster command-line, or through environment variables, will be used as the source for the following Patroni configuration parameters:
- **scope**: ``cluster_name`` GUC value;
- **postgresql.listen**: ``listen_addresses`` and ``port`` GUC values;
- **postgresql.datadir**: ``data_directory`` GUC value;
- **postgresql.parameters**: ``archive_command``, ``restore_command``, ``archive_cleanup_command``, ``recovery_end_command``, ``ssl_passphrase_command``, ``hba_file``, ``ident_file``, ``config_file`` GUC values;
- **bootstrap.dcs**: all other gathered PostgreSQL GUCs.
If ``scope``, ``postgresql.listen`` or ``postgresql.datadir`` is not set from the Postgres GUCs, the respective :ref:`Environment configuration <environment>` value is used.
Other rules applied for the values definition:
- **name**: ``PATRONI_NAME`` environment variable value if set, otherwise the current machine's hostname.
- **postgresql.bin_dir**: path to the Postgres binaries gathered from the running instance.
- **postgresql.connect_address**: the IP address returned by ``gethostname`` call for the current machine's hostname and the port used for the instance connection or the ``port`` GUC value.
- **postgresql.authentication.superuser**: the configuration used for the instance connection;
- **postgresql.pg_hba**: the lines gathered from the source instance's ``hba_file``.
- **postgresql.pg_ident**: the lines gathered from the source instance's ``ident_file``.
- **restapi.listen**: IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``8008`` port.
- **restapi.connect_address**: IP address returned by ``gethostname`` call for the current machine's hostname and the standard ``8008`` port.
Other parameters defined using :ref:`Environment configuration <environment>` are also included into the configuration.
Parameters
""""""""""
``configfile``
Full path to the configuration file used to store the result. If not provided, result is sent to ``stdout``.
``dsn``
Optional DSN string for the local PostgreSQL instance to get GUC values from.
Validate Patroni configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: text
patroni --validate-config [configfile]
Description
"""""""""""
Validate the given Patroni configuration and print the information about the failed checks.
Parameters
""""""""""
``configfile``
Full path to the configuration file to check. If not given or file does not exist, will try to read from the ``PATRONI_CONFIG_VARIABLE`` environment variable or, if not set, from the :ref:`Patroni environment variables <environment>`.
-1975
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -19,7 +19,7 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,
- For the Postgres primary with the leader lock Patroni updates the lock. If the node with the leader lock stops being the primary (i.e. is demoted manually), Patroni will release the lock instead of promoting the node back.
- Manual unscheduled restart, manual unscheduled failover/switchover and reinitialize are allowed. No scheduled action is allowed. Manual switchover is only allowed if the node to switch over to is specified.
- Manual unscheduled restart, reinitialize and manual failover are allowed. Manual failover is only allowed if the node to failover to is specified. In the paused mode, manual failover does not require a running primary node.
- If 'parallel' primaries are detected by Patroni, it emits a warning, but does not demote the primary without the leader lock.
@@ -32,6 +32,6 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,
User guide
----------
``patronictl`` supports :ref:`pause <patronictl_pause>` and :ref:`resume <patronictl_resume>` commands.
``patronictl`` supports ``pause`` and ``resume`` commands.
One can also issue a ``PATCH`` request to the ``{namespace}/{cluster}/config`` key with ``{"pause": true/false/null}``
-72
View File
@@ -3,78 +3,6 @@
Release notes
=============
Version 3.2.0
-------------
**Deprecation notice**
- The ``bootstrap.users`` support will be removed in version 4.0.0. If you need to create users after deploying a new cluster please use the ``bootstrap.post_bootstrap`` hook for that.
**Breaking changes**
- Enforce ``loop_wait + 2*retry_timeout <= ttl`` rule and hard-code minimal possible values (Alexander Kukushkin)
Minimal values: ``loop_wait=2``, ``retry_timeout=3``, ``ttl=20``. In case values are smaller or violate the rule they are adjusted and a warning is written to Patroni logs.
**New features**
- Failover priority (Mark Pekala)
With the help of ``tags.failover_priority`` it's now possible to make a node more preferred during the leader race. More details in the documentation (ref tags).
- Implemented ``patroni --generate-config [--dsn DSN]`` and ``patroni --generate-sample-config`` (Polina Bungina)
It allows to generate a config file for the running PostgreSQL cluster or a sample config file for the new Patroni cluster.
- Use a dedicated connection to Postgres for Patroni REST API (Alexander Kukushkin)
It helps to avoid blocking the main heartbeat loop if the system is under stress.
- Enrich some endpoints with the ``name`` of the node (sskserk)
For the monitoring endpoint ``name`` is added next to the ``scope`` and for metrics endpoint the ``name`` is added to tags.
- Ensure strict failover/switchover difference (Polina Bungina)
Be more precise in log messages and allow failing over to an asynchronous node in a healthy synchronous cluster.
- Make permanent physical replication slots behave similarly to permanent logical slots (Alexander Kukushkin)
Create permanent physical replication slots on all nodes that are allowed to become the leader and use ``pg_replication_slot_advance()`` function to advance ``restart_lsn`` for slots on standby nodes.
- Add capability of specifying namespace through ``--dcs`` argument in ``patronictl`` (Israel Barth Rubio)
It could be handy if ``patronictl`` is used without a configuration file.
- Add support for additional parameters in custom bootstrap configuration (Israel Barth Rubio)
Previously it was only possible to add custom arguments to the ``command`` and now one could list them as a mapping.
**Improvements**
- Set ``citus.local_hostname`` GUC to the same value which is used by Patroni to connect to the Postgres (Alexander Kukushkin)
There are cases when Citus wants to have a connection to the local Postgres. By default it uses ``localhost``, which is not always available.
**Bugfixes**
- Ignore ``synchronous_mode`` setting in a standby cluster (Polina Bungina)
Postgres doesn't support cascading synchronous replication and not ignoring ``synchronous_mode`` was breaking a switchover in a standby cluster.
- Handle SIGCHLD for ``on_reload`` callback (Alexander Kukushkin)
Not doing so results in a zombie process, which is reaped only when the next ``on_reload`` is executed.
- Handle ``AuthOldRevision`` error when working with Etcd v3 (Alexander Kukushkin, Kenny Do)
The error is raised if Etcd is configured to use JWT and when the user database in Etcd is updated.
Version 3.1.2
-------------
+6 -21
View File
@@ -43,31 +43,16 @@ in the configuration files, Patroni supplies two cluster-specific ones:
Passing these two additional flags can be disabled by setting a special ``no_params`` parameter to ``True``.
If the bootstrap script returns ``0``, Patroni tries to configure and start the PostgreSQL instance produced by it. If any
If the bootstrap script returns 0, Patroni tries to configure and start the PostgreSQL instance produced by it. If any
of the intermediate steps fail, or the script returns a non-zero value, Patroni assumes that the bootstrap has failed,
cleans up after itself and releases the initialize lock to give another node the opportunity to bootstrap.
If a ``recovery_conf`` block is defined in the same section as the custom bootstrap method, Patroni will generate a
``recovery.conf`` before starting the newly bootstrapped instance (or set the recovery settings on Postgres configuration if
running PostgreSQL >= 12).
Typically, such recovery configuration should contain at least one of the ``recovery_target_*`` parameters, together with the ``recovery_target_timeline`` set to ``promote``.
``recovery.conf`` before starting the newly bootstrapped instance. Typically, such recovery.conf should contain at least
one of the ``recovery_target_*`` parameters, together with the ``recovery_target_timeline`` set to ``promote``.
If ``keep_existing_recovery_conf`` is defined and set to ``True``, Patroni will not remove the existing ``recovery.conf`` file if it exists (PostgreSQL <= 11).
Similarly, in that case Patroni will not remove the existing ``recovery.signal`` or ``standby.signal`` if either exists, nor will it override the configured recovery settings (PostgreSQL >= 12).
This is useful when bootstrapping from a backup with tools like pgBackRest that generate the appropriate recovery configuration for you.
Besides that, any additional key/value pairs informed in the custom bootstrap method configuration will be passed as arguments to ``command`` in the format ``--name=value``. For example:
.. code:: YAML
bootstrap:
method: <custom_bootstrap_method_name>
<custom_bootstrap_method_name>:
command: <path_to_custom_bootstrap_script>
arg1: value1
arg2: value2
Makes the configured ``command`` to be called additionally with ``--arg1=value1 --arg2=value2`` command-line arguments.
If ``keep_existing_recovery_conf`` is defined and set to ``True``, Patroni will not remove the existing ``recovery.conf`` file if it exists.
This is useful when bootstrapping from a backup with tools like pgBackRest that generate the appropriate ``recovery.conf`` for you.
.. note:: Bootstrap methods are neither chained, nor fallen-back to the default one in case the primary one fails
@@ -206,7 +191,7 @@ There is no further relationship between the standby cluster and the primary
cluster it replicates from, in particular, they must not share the same DCS
scope if they use the same DCS. They do not know anything else from each other
apart from replication information. Also, the standby cluster is not being
displayed in :ref:`patronictl_list` or :ref:`patronictl_topology` output on the
displayed in ``patronictl list`` or ``patronictl topology`` output on the
primary cluster.
For the sake of flexibility, you can specify methods of creating a replica and
+66 -305
View File
@@ -3,7 +3,7 @@
Patroni REST API
================
Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
Health check endpoints
----------------------
@@ -92,188 +92,26 @@ Monitoring endpoint
The ``GET /patroni`` is used by Patroni during the leader race. It also could be used by your monitoring system. The JSON document produced by this endpoint has the same structure as the JSON produced by the health check endpoints.
**Example:** A healthy cluster
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2023-08-18 11:03:37.966359+00:00",
"postmaster_start_time": "2019-09-24 09:22:32.555 CEST",
"role": "master",
"server_version": 150004,
"server_version": 110005,
"cluster_unlocked": false,
"xlog": {
"location": 67395656
"location": 25624640
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"dcs_last_seen": 1692356718,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"timeline": 3,
"database_system_identifier": "6739877027151648096",
"patroni": {
"version": "3.1.0",
"scope": "demo",
"name": "patroni1"
"version": "1.6.0",
"scope": "batman"
}
}
**Example:** An unlocked cluster
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2023-08-18 11:09:08.615242+00:00",
"role": "replica",
"server_version": 150004,
"xlog": {
"received_location": 67419744,
"replayed_location": 67419744,
"replayed_timestamp": null,
"paused": false
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"cluster_unlocked": true,
"dcs_last_seen": 1692356928,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "3.1.0",
"scope": "demo",
"name": "patroni1"
}
}
**Example:** An unlocked cluster with :ref:`DCS failsafe mode <dcs_failsafe_mode>` enabled
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2023-08-18 11:09:08.615242+00:00",
"role": "replica",
"server_version": 150004,
"xlog": {
"location": 67420024
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"cluster_unlocked": true,
"failsafe_mode_is_active": true,
"dcs_last_seen": 1692356928,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "3.1.0",
"scope": "demo",
"name": "patroni1"
}
}
**Example:** A cluster with the :ref:`pause mode <pause>` enabled
.. code-block:: bash
$ curl -s http://localhost:8008/patroni | jq .
{
"state": "running",
"postmaster_start_time": "2023-08-18 11:09:08.615242+00:00",
"role": "replica",
"server_version": 150004,
"xlog": {
"location": 67420024
},
"timeline": 1,
"replication": [
{
"usename": "replicator",
"application_name": "patroni2",
"client_addr": "10.89.0.6",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
},
{
"usename": "replicator",
"application_name": "patroni3",
"client_addr": "10.89.0.2",
"state": "streaming",
"sync_state": "async",
"sync_priority": 0
}
],
"pause": true,
"dcs_last_seen": 1692356928,
"tags": {
"clonefrom": true
},
"database_system_identifier": "7268616322854375442",
"patroni": {
"version": "3.1.0",
"scope": "demo",
"name": "patroni1"
}
}
Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` endpoint.
@@ -283,70 +121,64 @@ Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` e
# HELP patroni_version Patroni semver without periods. \
# TYPE patroni_version gauge
patroni_version{scope="batman",name="patroni1"} 020103
patroni_version{scope="batman"} 020103
# HELP patroni_postgres_running Value is 1 if Postgres is running, 0 otherwise.
# TYPE patroni_postgres_running gauge
patroni_postgres_running{scope="batman",name="patroni1"} 1
patroni_postgres_running{scope="batman"} 1
# HELP patroni_postmaster_start_time Epoch seconds since Postgres started.
# TYPE patroni_postmaster_start_time gauge
patroni_postmaster_start_time{scope="batman",name="patroni1"} 1657656955.179243
patroni_postmaster_start_time{scope="batman"} 1657656955.179243
# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.
# TYPE patroni_master gauge
patroni_master{scope="batman",name="patroni1"} 1
# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.
# TYPE patroni_primary gauge
patroni_primary{scope="batman",name="patroni1"} 1
patroni_master{scope="batman"} 1
# HELP patroni_xlog_location Current location of the Postgres transaction log, 0 if this node is not the leader.
# TYPE patroni_xlog_location counter
patroni_xlog_location{scope="batman",name="patroni1"} 22320573386952
patroni_xlog_location{scope="batman"} 22320573386952
# HELP patroni_standby_leader Value is 1 if this node is the standby_leader, 0 otherwise.
# TYPE patroni_standby_leader gauge
patroni_standby_leader{scope="batman",name="patroni1"} 0
patroni_standby_leader{scope="batman"} 0
# HELP patroni_replica Value is 1 if this node is a replica, 0 otherwise.
# TYPE patroni_replica gauge
patroni_replica{scope="batman",name="patroni1"} 0
patroni_replica{scope="batman"} 0
# HELP patroni_sync_standby Value is 1 if this node is a sync standby replica, 0 otherwise.
# TYPE patroni_sync_standby gauge
patroni_sync_standby{scope="batman",name="patroni1"} 0
patroni_sync_standby{scope="batman"} 0
# HELP patroni_xlog_received_location Current location of the received Postgres transaction log, 0 if this node is not a replica.
# TYPE patroni_xlog_received_location counter
patroni_xlog_received_location{scope="batman",name="patroni1"} 0
patroni_xlog_received_location{scope="batman"} 0
# HELP patroni_xlog_replayed_location Current location of the replayed Postgres transaction log, 0 if this node is not a replica.
# TYPE patroni_xlog_replayed_location counter
patroni_xlog_replayed_location{scope="batman",name="patroni1"} 0
patroni_xlog_replayed_location{scope="batman"} 0
# HELP patroni_xlog_replayed_timestamp Current timestamp of the replayed Postgres transaction log, 0 if null.
# TYPE patroni_xlog_replayed_timestamp gauge
patroni_xlog_replayed_timestamp{scope="batman",name="patroni1"} 0
patroni_xlog_replayed_timestamp{scope="batman"} 0
# HELP patroni_xlog_paused Value is 1 if the Postgres xlog is paused, 0 otherwise.
# TYPE patroni_xlog_paused gauge
patroni_xlog_paused{scope="batman",name="patroni1"} 0
patroni_xlog_paused{scope="batman"} 0
# HELP patroni_postgres_streaming Value is 1 if Postgres is streaming, 0 otherwise.
# TYPE patroni_postgres_streaming gauge
patroni_postgres_streaming{scope="batman",name="patroni1"} 1
patroni_postgres_streaming{scope="batman"} 1
# HELP patroni_postgres_in_archive_recovery Value is 1 if Postgres is replicating from archive, 0 otherwise.
# TYPE patroni_postgres_in_archive_recovery gauge
patroni_postgres_in_archive_recovery{scope="batman",name="patroni1"} 0
patroni_postgres_in_archive_recovery{scope="batman"} 0
# HELP patroni_postgres_server_version Version of Postgres (if running), 0 otherwise.
# TYPE patroni_postgres_server_version gauge
patroni_postgres_server_version{scope="batman",name="patroni1"} 140004
patroni_postgres_server_version {scope="batman"} 140004
# HELP patroni_cluster_unlocked Value is 1 if the cluster is unlocked, 0 if locked.
# TYPE patroni_cluster_unlocked gauge
patroni_cluster_unlocked{scope="batman",name="patroni1"} 0
patroni_cluster_unlocked{scope="batman"} 0
# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
# TYPE patroni_postgres_timeline counter
patroni_failsafe_mode_is_active{scope="batman",name="patroni1"} 0
# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
# TYPE patroni_postgres_timeline counter
patroni_postgres_timeline{scope="batman",name="patroni1"} 24
patroni_postgres_timeline{scope="batman"} 24
# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully by Patroni.
# TYPE patroni_dcs_last_seen gauge
patroni_dcs_last_seen{scope="batman",name="patroni1"} 1677658321
patroni_dcs_last_seen{scope="batman"} 1677658321
# HELP patroni_pending_restart Value is 1 if the node needs a restart, 0 otherwise.
# TYPE patroni_pending_restart gauge
patroni_pending_restart{scope="batman",name="patroni1"} 1
patroni_pending_restart{scope="batman"} 1
# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.
# TYPE patroni_is_paused gauge
patroni_is_paused{scope="batman",name="patroni1"} 1
patroni_is_paused{scope="batman"} 1
Cluster status endpoints
@@ -360,24 +192,24 @@ Cluster status endpoints
{
"members": [
{
"name": "patroni1",
"name": "postgresql0",
"host": "127.0.0.1",
"port": 5432,
"role": "leader",
"state": "running",
"api_url": "http://10.89.0.4:8008/patroni",
"host": "10.89.0.4",
"port": 5432,
"api_url": "http://127.0.0.1:8008/patroni",
"timeline": 5,
"tags": {
"clonefrom": true
}
},
{
"name": "patroni2",
"role": "replica",
"state": "streaming",
"api_url": "http://10.89.0.6:8008/patroni",
"host": "10.89.0.6",
"name": "postgresql1",
"host": "127.0.0.1",
"port": 5433,
"role": "replica",
"state": "running",
"api_url": "http://127.0.0.1:8009/patroni",
"timeline": 5,
"tags": {
"clonefrom": true
@@ -385,11 +217,9 @@ Cluster status endpoints
"lag": 0
}
],
"scope": "demo",
"scheduled_switchover": {
"at": "2023-09-24T10:36:00+02:00",
"from": "patroni1",
"to": "patroni3"
"at": "2019-09-24T10:36:00+02:00",
"from": "postgresql0"
}
}
@@ -434,7 +264,7 @@ Config endpoint
.. code-block:: bash
$ curl -s http://localhost:8008/config | jq .
$ curl -s localhost:8008/config | jq .
{
"ttl": 30,
"loop_wait": 10,
@@ -445,6 +275,7 @@ Config endpoint
"use_pg_rewind": true,
"parameters": {
"hot_standby": "on",
"wal_log_hints": "on",
"wal_level": "hot_standby",
"max_wal_senders": 5,
"max_replication_slots": 5,
@@ -471,6 +302,7 @@ Config endpoint
"use_pg_rewind": true,
"parameters": {
"hot_standby": "on",
"wal_log_hints": "on",
"wal_level": "hot_standby",
"max_wal_senders": 5,
"max_replication_slots": 5,
@@ -494,9 +326,8 @@ Let's check that the node processed this configuration. First of all it should s
"location": 2197818976
},
"patroni": {
"version": "1.0",
"scope": "batman",
"name": "patroni1"
"version": "1.0"
},
"state": "running",
"role": "master",
@@ -524,6 +355,7 @@ If you want to remove (reset) some setting just patch it with ``null``:
"hot_standby": "on",
"unix_socket_directories": ".",
"wal_level": "hot_standby",
"wal_log_hints": "on",
"max_wal_senders": 5,
"max_replication_slots": 5
}
@@ -537,7 +369,7 @@ The above call removes ``postgresql.parameters.max_connections`` from the dynami
.. code-block:: bash
$ curl -s -XPUT -d \
'{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \
'{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_log_hints":"on","wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \
http://localhost:8008/config | jq .
{
"ttl": 20,
@@ -549,6 +381,7 @@ The above call removes ``postgresql.parameters.max_connections`` from the dynami
"hot_standby": "on",
"unix_socket_directories": ".",
"wal_level": "hot_standby",
"wal_log_hints": "on",
"max_wal_senders": 5
},
"use_pg_rewind": true
@@ -560,111 +393,39 @@ The above call removes ``postgresql.parameters.max_connections`` from the dynami
Switchover and failover endpoints
---------------------------------
.. _switchover_api:
``POST /switchover`` or ``POST /failover``. These endpoints are very similar to each other. There are a couple of minor differences though:
Switchover
^^^^^^^^^^
1. The failover endpoint allows to perform a manual failover when there are no healthy nodes, but at the same time it will not allow you to schedule a switchover.
``/switchover`` endpoint only works when the cluster is healthy (there is a leader). It also allows to schedule a switchover at a given time.
2. The switchover endpoint is the opposite. It works only when the cluster is healthy (there is a leader) and allows to schedule a switchover at a given time.
When calling ``/switchover`` endpoint a candidate can be specified but is not required, in contrast to ``/failover`` endpoint. If a candidate is not provided, all the eligible nodes of the cluster will participate in the leader race after the leader stepped down.
In the JSON body of the ``POST`` request you must specify the ``leader`` field. The ``candidate`` and the ``scheduled_at`` fields are optional and can be used to schedule a switchover at a specific time.
In the JSON body of the ``POST`` request you must specify at least the ``leader`` or ``candidate`` fields and optionally the ``scheduled_at`` field if you want to schedule a switchover at a specific time.
Depending on the situation, requests might return different HTTP status codes and bodies. Status code **200** is returned when the switchover or failover successfully completed. If the switchover was successfully scheduled, Patroni will return HTTP status code **202**. In case something went wrong, the error status code (one of **400**, **412**, or **503**) will be returned with some details in the response body.
``DELETE /switchover`` can be used to delete the currently scheduled switchover.
**Example:** perform a switchover to any healthy standby
Example: perform a failover to the specific node:
.. code-block:: bash
$ curl -s http://localhost:8008/switchover -XPOST -d '{"leader":"postgresql1"}'
Successfully switched over to "postgresql2"
$ curl -s http://localhost:8009/failover -XPOST -d '{"candidate":"postgresql1"}'
Successfully failed over to "postgresql1"
**Example:** perform a switchover to a specific node
Example: schedule a switchover from the leader to any other healthy replica in the cluster at a specific time:
.. code-block:: bash
$ curl -s http://localhost:8008/switchover -XPOST -d \
'{"leader":"postgresql1","candidate":"postgresql2"}'
Successfully switched over to "postgresql2"
$ curl -s http://localhost:8008/switchover -XPOST -d \
'{"leader":"postgresql0","scheduled_at":"2019-09-24T12:00+00"}'
Switchover scheduled
**Example:** schedule a switchover from the leader to any other healthy standby in the cluster at a specific time.
Depending on the situation the request might finish with a different HTTP status code and body. The status code **200** is returned when the switchover or failover successfully completed. If the switchover was successfully scheduled, Patroni will return HTTP status code **202**. In case something went wrong, the error status code (one of **400**, **412** or **503**) will be returned with some details in the response body. For more information please check the source code of ``patroni/api.py:do_POST_failover()`` method.
.. code-block:: bash
- ``DELETE /switchover``: delete the scheduled switchover
$ curl -s http://localhost:8008/switchover -XPOST -d \
'{"leader":"postgresql0","scheduled_at":"2019-09-24T12:00+00"}'
Switchover scheduled
Failover
^^^^^^^^
``/failover`` endpoint can be used to perform a manual failover when there are no healthy nodes (e.g. to an asynchronous standby if all synchronous standbys are not healthy enough to promote). However there is no requirement for a cluster not to have leader - failover can also be run on a healthy cluster.
In the JSON body of the ``POST`` request you must specify the ``candidate`` field. If the ``leader`` field is specified, a switchover is triggered instead.
**Example:**
.. code-block:: bash
$ curl -s http://localhost:8008/failover -XPOST -d '{"candidate":"postgresql1"}'
Successfully failed over to "postgresql1"
.. warning::
:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs.
``POST /switchover`` and ``POST /failover`` endpoints are used by :ref:`patronictl_switchover` and :ref:`patronictl_failover`, respectively.
``DELETE /switchover`` is used by :ref:`patronictl flush cluster-name switchover <patronictl_flush_parameters>`.
.. list-table:: Failover/Switchover comparison
:widths: 25 25 25
:header-rows: 1
* -
- Failover
- Switchover
* - Requires leader specified
- no
- yes
* - Requires candidate specified
- yes
- no
* - Can be run in pause
- yes
- yes (only to a specific candidate)
* - Can be scheduled
- no
- yes (if not in pause)
.. _failover_healthcheck:
Healthy standby
^^^^^^^^^^^^^^^
There are a couple of checks that a member of a cluster should pass to be able to participate in the leader race during a switchover or to become a leader as a failover/switchover candidate:
- be reachable via Patroni API;
- not have ``nofailover`` tag set to ``true``;
- have watchdog fully functional (if required by the configuration);
- in case of a switchover in a healthy cluster or an automatic failover, not exceed maximum replication lag (``maximum_lag_on_failover`` :ref:`configuration parameter <dynamic_configuration>`);
- in case of a switchover in a healthy cluster or an automatic failover, not have a timeline number smaller than the cluster timeline if ``check_timeline`` :ref:`configuration parameter <dynamic_configuration>` is set to ``true``;
- in :ref:`synchronous mode <synchronous_mode>`:
- In case of a switchover (both with and without a candidate): be listed in the ``/sync`` key members;
- For a failover in both healthy and unhealthy clusters, this check is omitted.
.. warning::
In case of a manual failover in a cluster without a leader, a candidate will be allowed to promote even if:
- it is not in the ``/sync`` key members when synchronous mode is enabled;
- its lag exceeds the maximum replication lag allowed;
- it has the timeline number smaller than the last known cluster timeline.
The ``POST /switchover`` and ``POST failover`` endpoints are used by ``patronictl switchover`` and ``patronictl failover``, respectively.
The ``DELETE /switchover`` is used by ``patronictl flush <cluster-name> switchover``.
Restart endpoint
@@ -680,15 +441,15 @@ Restart endpoint
- ``DELETE /restart``: delete the scheduled restart
``POST /restart`` and ``DELETE /restart`` endpoints are used by :ref:`patronictl_restart` and :ref:`patronictl flush cluster-name restart <patronictl_flush_parameters>` respectively.
``POST /restart`` and ``DELETE /restart`` endpoints are used by ``patronictl restart`` and ``patronictl flush <cluster-name> restart`` respectively.
Reload endpoint
---------------
The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of :ref:`patronictl_restart`.
The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of ``patronictl restart``.
The reload endpoint is used by :ref:`patronictl_reload`.
The reload endpoint is used by ``patronictl reload``.
Reinitialize endpoint
@@ -698,4 +459,4 @@ Reinitialize endpoint
The call might fail if Patroni is in a loop trying to recover (restart) a failed Postgres. In order to overcome this problem one can specify ``{"force":true}`` in the request body.
The reinitialize endpoint is used by :ref:`patronictl_reinit`.
The reinitialize endpoint is used by ``patronictl reinit``.
+3 -3
View File
@@ -9,7 +9,7 @@ A Patroni cluster has two interfaces to be protected from unauthorized access: t
Protecting DCS
==============
Patroni and :ref:`patronictl` both store and retrieve data to/from the DCS.
Patroni and patronictl both store and retrieve data to/from the DCS.
Despite DCS doesn't contain any sensitive information, it allows changing some of Patroni/Postgres configuration. Therefore the very first thing that should be protected is DCS itself.
@@ -22,7 +22,7 @@ Protecting the REST API
Protecting the REST API is a more complicated task.
The Patroni REST API is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
The Patroni REST API is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
From the point of view of security, REST API contains safe (``GET`` requests, only retrieve information) and unsafe (``PUT``, ``POST``, ``PATCH`` and ``DELETE`` requests, change the state of nodes) endpoints.
@@ -32,6 +32,6 @@ When TLS for the REST API is enabled and a PKI is established, mutual authentica
The ``restapi`` section parameters enable TLS client authentication to the server. Depending on the value of the ``verify_client`` parameter, the API server requires a successful client certificate verification for both safe and unsafe API calls (``verify_client: required``), or only for unsafe API calls (``verify_client: optional``), or for no API calls (``verify_client: none``).
The ``ctl`` section parameters enable TLS server authentication to the client (the :ref:`patronictl` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
The ``ctl`` section parameters enable TLS server authentication to the client (the ``patronictl`` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
Protecting the PostgreSQL database proper from unauthorized access is beyond the scope of this document and is covered in https://www.postgresql.org/docs/current/client-authentication.html
+22 -16
View File
@@ -30,15 +30,9 @@ Log
Bootstrap configuration
-----------------------
.. note::
Once Patroni has initialized the cluster for the first time and settings have been stored in the DCS, all future
changes to the ``bootstrap.dcs`` section of the YAML configuration will not take any effect! If you want to change
them please use either :ref:`patronictl_edit_config` or the Patroni :ref:`REST API <rest_api>`.
- **bootstrap**:
- **dcs**: This section will be written into `/<namespace>/<scope>/config` of the given configuration store after initializing the new cluster. The global dynamic configuration for the cluster. You can put any of the parameters described in the :ref:`Dynamic Configuration settings <dynamic_configuration>` under ``bootstrap.dcs`` and after Patroni has initialized (bootstrapped) the new cluster, it will write this section into `/<namespace>/<scope>/config` of the configuration store.
- **dcs**: This section will be written into `/<namespace>/<scope>/config` of the given configuration store after initializing of new cluster. The global dynamic configuration for the cluster. Under the ``bootstrap.dcs`` you can put any of the parameters described in the :ref:`Dynamic Configuration settings <dynamic_configuration>` and after Patroni initialized (bootstrapped) the new cluster, it will write this section into `/<namespace>/<scope>/config` of the configuration store. All later changes of ``bootstrap.dcs`` will not take any effect! If you want to change them please use either ``patronictl edit-config`` or Patroni :ref:`REST API <rest_api>`.
- **method**: custom script to use for bootstrapping this cluster.
See :ref:`custom bootstrap methods documentation <custom_bootstrap>` for details.
@@ -49,8 +43,24 @@ Bootstrap configuration
- **- data-checksums**: Must be enabled when pg_rewind is needed on 9.3.
- **- encoding: UTF8**: default encoding for new databases.
- **- locale: UTF8**: default locale for new databases.
- **users**: Some additional users which need to be created after initializing new cluster, see :ref:`Bootstrap users configuration <bootstrap_users_configuration>` below.
- **post\_bootstrap** or **post\_init**: An additional script that will be executed after initializing the cluster. The script receives a connection string URL (with the cluster superuser as a user name). The PGPASSFILE variable is set to the location of pgpass file.
.. _bootstrap_users_configuration:
Bootstrap users configuration
=============================
Users which need to be created after initializing the cluster:
- **admin**: the name of user
- **password**: (optional) password for the user
- **options**: list of options for CREATE USER statement
- **- createrole**
- **- createdb**
.. _citus_settings:
Citus
@@ -350,10 +360,10 @@ CTL
- **authentication**:
- **username**: Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
- **password**: Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
- **username**: Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
- **password**: Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
- **insecure**: Allow connections to REST API without verifying SSL certs.
- **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
- **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
- **certfile**: Specifies the file with the client certificate in the PEM format.
- **keyfile**: Specifies the file with the client secret key in the PEM format.
- **keyfile\_password**: Specifies a password for decrypting the client keyfile.
@@ -368,15 +378,11 @@ Watchdog
Tags
----
- **nofailover**: ``true`` or ``false``, controls whether this node is allowed to participate in the leader race and become a leader. Defaults to ``false``
- **clonefrom**: ``true`` or ``false``. If set to ``true`` other nodes might prefer to use this node for bootstrap (take ``pg_basebackup`` from). If there are several nodes with ``clonefrom`` tag set to ``true`` the node to bootstrap from will be chosen randomly. The default value is ``false``.
- **noloadbalance**: ``true`` or ``false``. If set to ``true`` the node will return HTTP Status Code 503 for the ``GET /replica`` REST API health-check and therefore will be excluded from the load-balancing. Defaults to ``false``.
- **replicatefrom**: The IP address/hostname of another replica. Used to support cascading replication.
- **nosync**: ``true`` or ``false``. If set to ``true`` the node will never be selected as a synchronous replica.
- **nofailover**: ``true`` or ``false``, controls whether this node is allowed to participate in the leader race and become a leader. Defaults to ``false``, meaning this node _can_ participate in leader races.
- **failover_priority**: integer, controls the priority that this node should have during failover. Nodes with higher priority will be preferred over lower priority nodes if they received/replayed the same amount of WAL. However, nodes with higher values of receive/replay LSN are preferred regardless of their priority. If the ``failover_priority`` is 0 or negative - such node is not allowed to participate in the leader race and to become a leader (similar to ``nofailover: true``).
.. warning::
Provide only one of ``nofailover`` or ``failover_priority``. Providing ``nofailover: true`` is the same as ``failover_priority: 0``, and providing ``nofailover: false`` will give the node priority 1.
In addition to these predefined tags, you can also add your own ones:
@@ -385,4 +391,4 @@ In addition to these predefined tags, you can also add your own ones:
- **key3**: ``1.4``
- **key4**: ``"RandomString"``
Tags are visible in the :ref:`REST API <rest_api>` and :ref:`patronictl_list` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
Tags are visible in the :ref:`REST API <rest_api>` and ``patronictl list`` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
+2 -5
View File
@@ -27,8 +27,8 @@ RUN set -ex \
&& apt-get update \
&& apt-get reinstall init-system-helpers \
&& apt-get install -y \
python3-pip \
python3-dev \
python3-venv \
rsync \
curl \
gcc \
@@ -40,9 +40,7 @@ RUN set -ex \
net-tools \
iputils-ping \
&& rm -rf /var/cache/apt \
\
&& python3 -m venv /tox \
&& /tox/bin/pip install --no-cache-dir tox>=4 \
&& python3 -m pip install --no-cache-dir tox \
\
&& mkdir -p "$PGHOME" \
&& sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
@@ -52,7 +50,6 @@ RUN set -ex \
&& curl -sL "$ETCDURL/etcd-v$ETCDVERSION-linux-$(dpkg --print-architecture).tar.gz" \
| tar xz -C /usr/local/bin --strip=1 --wildcards --no-anchored etcd etcdctl
ENV PATH="/tox/bin:$PATH"
# This Dockerfile syntax only works with docker buildx and the syntax
# line at the top of this file.
-1
View File
@@ -6,7 +6,6 @@ if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--datadir", required=True)
parser.add_argument("--sourcedir", required=True)
parser.add_argument("--test-argument", required=True)
args, _ = parser.parse_known_args()
shutil.copytree(args.sourcedir, args.datadir)
+1 -1
View File
@@ -82,4 +82,4 @@ Feature: basic replication
@reject-duplicate-name
Scenario: check graceful rejection when two nodes have the same name
Given I start duplicate postgres0 on port 8011
Then there is one of ["Can't start; there is already a node named 'postgres0' running"] CRITICAL in the dup-postgres0 patroni log after 5 seconds
Then there is a "Can't start; there is already a node named 'postgres0' running" CRITICAL in the dup-postgres0 patroni log
+1 -1
View File
@@ -68,6 +68,6 @@ Feature: citus
And I receive a response output "+ttl: 20"
Then postgres4 is registered in the postgres2 as the primary in group 2 after 5 seconds
When I shut down postgres4
Then there is a transaction in progress on postgres0 changing pg_dist_node after 5 seconds
Then There is a transaction in progress on postgres0 changing pg_dist_node
When I run patronictl.py restart batman postgres2 --group 1 --force
Then a transaction finishes in 20 seconds
+11 -42
View File
@@ -4,14 +4,14 @@ Feature: dcs failsafe mode
Scenario: check failsafe mode can be successfully enabled
Given I start postgres0
And postgres0 is a leader after 10 seconds
Then "config" key in DCS has ttl=30 after 10 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 3, "failsafe_mode": true}
And I sleep for 3 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 5, "failsafe_mode": true}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/failsafe
Then I receive a response code 200
And I receive a response postgres0 http://127.0.0.1:8008/patroni
When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}},"slots":{"dcs_slot_1": null,"postgres0":null}}
When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}}}
Then I receive a response code 200
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"dcs_slot_0": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
Then I receive a response code 200
@@ -28,6 +28,7 @@ Feature: dcs failsafe mode
When I do a backup of postgres0
And I shut down postgres0
When I start postgres1 in a cluster batman from backup with no_leader
And I sleep for 2 seconds
Then postgres1 role is the replica after 12 seconds
Scenario: check leader and replica are both in /failsafe key after leader is back
@@ -44,73 +45,41 @@ Feature: dcs failsafe mode
@dcs-failsafe
@slot-advance
Scenario: check leader and replica are functioning while DCS is down
Given I get all changes from physical slot dcs_slot_1 on postgres0
Then physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds
And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 10 seconds
Given logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 10 seconds
And DCS is down
Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
Then postgres0 role is the primary after 10 seconds
And postgres1 role is the replica after 2 seconds
And replication works from postgres0 to postgres1 after 10 seconds
When I get all changes from logical slot dcs_slot_0 on postgres0
And I get all changes from physical slot dcs_slot_1 on postgres0
Then logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds
And I get all changes from logical slot dcs_slot_0 on postgres0
And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds
@dcs-failsafe
Scenario: check primary is demoted when one replica is shut down and DCS is down
Given DCS is down
And I kill postgres1
And I kill postmaster on postgres1
And I sleep for 2 seconds
Then postgres0 role is the replica after 12 seconds
@dcs-failsafe
Scenario: check known replica is promoted when leader is down and DCS is up
Given I kill postgres0
And I shut down postmaster on postgres0
Given I shut down postgres0
And DCS is up
When I start postgres1
Then "members/postgres1" key in DCS has state=running after 10 seconds
And postgres1 role is the primary after 25 seconds
@dcs-failsafe
Scenario: scale to three-node cluster
Scenario: check three-node cluster is functioning while DCS is down
Given I start postgres0
And I start postgres2
Then "members/postgres2" key in DCS has state=running after 10 seconds
And "members/postgres0" key in DCS has state=running after 20 seconds
And Response on GET http://127.0.0.1:8008/failsafe contains postgres2 after 10 seconds
And replication works from postgres1 to postgres0 after 10 seconds
And replication works from postgres1 to postgres2 after 10 seconds
@dcs-failsafe
@slot-advance
Scenario: make sure permanent slots exist on replicas
Given I issue a PATCH request to http://127.0.0.1:8009/config with {"slots":{"dcs_slot_0":null,"dcs_slot_2":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds
When I get all changes from physical slot dcs_slot_1 on postgres1
Then physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds
@dcs-failsafe
Scenario: check three-node cluster is functioning while DCS is down
Given DCS is down
Then Response on GET http://127.0.0.1:8009/primary contains failsafe_mode_is_active after 12 seconds
Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
Then postgres1 role is the primary after 10 seconds
And postgres0 role is the replica after 2 seconds
And postgres2 role is the replica after 2 seconds
@dcs-failsafe
@slot-advance
Scenario: check that permanent slots are in sync between nodes while DCS is down
Given replication works from postgres1 to postgres0 after 10 seconds
And replication works from postgres1 to postgres2 after 10 seconds
When I get all changes from logical slot dcs_slot_2 on postgres1
And I get all changes from physical slot dcs_slot_1 on postgres1
Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds
+15 -23
View File
@@ -162,10 +162,9 @@ class PatroniController(AbstractController):
def stop(self, kill=False, timeout=15, postgres=False):
if postgres:
mode = 'i' if kill else 'f'
return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-m' + mode, '-w'])
return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-mi', '-w'])
super(PatroniController, self).stop(kill, timeout)
if isinstance(self._context.dcs_ctl, KubernetesController) and not kill:
if isinstance(self._context.dcs_ctl, KubernetesController):
self._context.dcs_ctl.delete_pod(self._name[8:])
if self.watchdog:
self.watchdog.stop()
@@ -245,10 +244,6 @@ class PatroniController(AbstractController):
self.recursive_update(config, custom_config)
self.recursive_update(config, {
'log': {
'format': '%(asctime)s %(levelname)s [%(pathname)s:%(lineno)d - %(funcName)s]: %(message)s',
'loggers': {'patroni.postgresql.callback_executor': 'DEBUG'}
},
'bootstrap': {
'dcs': {
'loop_wait': 2,
@@ -654,10 +649,9 @@ class KubernetesController(AbstractExternalDcsController):
try:
if group is not None:
scope = '{0}-{1}'.format(scope, group)
rkey = 'leader' if key in ('status', 'failsafe') else key
ep = scope + {'leader': '', 'history': '-config', 'initialize': '-config'}.get(rkey, '-' + rkey)
ep = scope + {'leader': '', 'history': '-config', 'initialize': '-config'}.get(key, '-' + key)
e = self._api.read_namespaced_endpoints(ep, self._namespace)
if key not in ('sync', 'status', 'failsafe'):
if key != 'sync':
return e.metadata.annotations[key]
else:
return json.dumps(e.metadata.annotations)
@@ -693,7 +687,7 @@ class ZooKeeperController(AbstractExternalDcsController):
self._client = kazoo.client.KazooClient()
def process_name(self):
return "java .*zookeeper"
return "zookeeper"
def query(self, key, scope='batman', group=None):
import kazoo.exceptions
@@ -892,28 +886,22 @@ class PatroniPoolController(object):
}
self.start(to_name, custom_config=custom_config)
def backup_restore_config(self, params=None):
return {
'command': (self.BACKUP_RESTORE_SCRIPT
+ ' --sourcedir=' + os.path.join(self.patroni_path, 'data', 'basebackup')).replace('\\', '/'),
'test-argument': 'test-value', # test config mapping approach on custom bootstrap/replica creation
**(params or {}),
}
def bootstrap_from_backup(self, name, cluster_name):
custom_config = {
'scope': cluster_name,
'bootstrap': {
'method': 'backup_restore',
'backup_restore': self.backup_restore_config({
'backup_restore': {
'command': (self.BACKUP_RESTORE_SCRIPT + ' --sourcedir='
+ os.path.join(self.patroni_path, 'data', 'basebackup').replace('\\', '/')),
'recovery_conf': {
'recovery_target_action': 'promote',
'recovery_target_timeline': 'latest',
'restore_command': (self.ARCHIVE_RESTORE_SCRIPT + ' --mode restore '
+ '--dirname {} --filename %f --pathname %p').format(
os.path.join(self.patroni_path, 'data', 'wal_archive_clone').replace('\\', '/'))
},
})
}
}
},
'postgresql': {
'authentication': {
@@ -934,7 +922,11 @@ class PatroniPoolController(object):
.format(os.path.join(self.patroni_path, 'data', 'wal_archive').replace('\\', '/'))
},
'create_replica_methods': ['no_leader_bootstrap'],
'no_leader_bootstrap': self.backup_restore_config({'no_leader': '1'})
'no_leader_bootstrap': {
'command': (self.BACKUP_RESTORE_SCRIPT + ' --sourcedir='
+ os.path.join(self.patroni_path, 'data', 'basebackup').replace('\\', '/')),
'no_leader': '1'
}
}
}
self.start(name, custom_config=custom_config)
+13 -13
View File
@@ -25,10 +25,10 @@ Feature: ignored slots
# but Patroni can actually end up dropping them almost immediately, so it's helpful
# to verify they exist before we begin testing whether they persist through failover
# cycles.
Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
When I start postgres0
Then "members/postgres0" key in DCS has role=replica after 10 seconds
@@ -46,16 +46,16 @@ Feature: ignored slots
And "members/postgres1" key in DCS has role=replica after 10 seconds
# give Patroni time to sync replication slots
And I sleep for 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
And postgres1 does not have a replication slot named dummy_slot
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
And postgres1 does not have a logical replication slot named dummy_slot
# 3. After a failover the server (now a primary) still has the slot.
When I shut down postgres0
Then "members/postgres1" key in DCS has role=master after 10 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
-75
View File
@@ -1,75 +0,0 @@
Feature: permanent slots
Scenario: check that physical permanent slots are created
Given I start postgres0
Then postgres0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres0":0,"postgres1":0,"postgres3":0},"postgresql":{"parameters":{"wal_level":"logical"}}}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/config contains slots after 10 seconds
When I start postgres1
And I start postgres2
And I configure and start postgres3 with a tag replicatefrom postgres2
Then postgres0 has a physical replication slot named test_physical after 10 seconds
And postgres0 has a physical replication slot named postgres1 after 10 seconds
And postgres0 has a physical replication slot named postgres2 after 10 seconds
And postgres2 has a physical replication slot named postgres3 after 10 seconds
@slot-advance
Scenario: check that logical permanent slots are created
Given I run patronictl.py restart batman postgres0 --force
And I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_logical":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
@slot-advance
Scenario: check that permanent slots are created on replicas
Given postgres1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds
And postgres1 has a physical replication slot named test_physical after 2 seconds
And postgres2 has a physical replication slot named test_physical after 2 seconds
And postgres3 has a physical replication slot named test_physical after 2 seconds
@slot-advance
Scenario: check permanent physical slots that match with member names
Given postgres0 has a physical replication slot named postgres3 after 2 seconds
And postgres1 has a physical replication slot named postgres0 after 2 seconds
And postgres1 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres0 after 2 seconds
And postgres2 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres1 after 2 seconds
And postgres1 does not have a replication slot named postgres2
And postgres3 does not have a replication slot named postgres2
@slot-advance
Scenario: check that permanent slots are advanced on replicas
Given I add the table replicate_me to postgres0
When I get all changes from logical slot test_logical on postgres0
And I get all changes from physical slot test_physical on postgres0
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres1 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres2 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot postgres1 is in sync between postgres0 and postgres2 after 10 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres0 after 20 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres1 after 10 seconds
And postgres1 does not have a replication slot named postgres2
And postgres3 does not have a replication slot named postgres2
@slot-advance
Scenario: check that only permanent slots are written to the /status key
Given "status" key in DCS has test_physical in slots
And "status" key in DCS has postgres0 in slots
And "status" key in DCS has postgres1 in slots
And "status" key in DCS does not have postgres2 in slots
And "status" key in DCS has postgres3 in slots
Scenario: check permanent physical replication slot after failover
Given I shut down postgres3
And I shut down postgres2
And I shut down postgres0
Then postgres1 has a physical replication slot named test_physical after 10 seconds
And postgres1 has a physical replication slot named postgres0 after 10 seconds
And postgres1 has a physical replication slot named postgres3 after 10 seconds
-23
View File
@@ -1,23 +0,0 @@
Feature: priority replication
We should check that we can give nodes priority during failover
Scenario: check failover priority 0 prevents leaderships
Given I configure and start postgres0 with a tag failover_priority 1
And I configure and start postgres1 with a tag failover_priority 0
Then replication works from postgres0 to postgres1 after 20 seconds
When I shut down postgres0
And I sleep for 5 seconds
Then postgres1 role is the secondary after 10 seconds
And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres1 patroni log after 5 seconds
Given I start postgres0
Then postgres0 role is the primary after 10 seconds
Scenario: check higher failover priority is respected
Given I configure and start postgres2 with a tag failover_priority 1
And I configure and start postgres3 with a tag failover_priority 2
Then replication works from postgres0 to postgres2 after 20 seconds
And replication works from postgres0 to postgres3 after 20 seconds
When I shut down postgres0
And I sleep for 5 seconds
Then postgres3 role is the primary after 10 seconds
And there is one of ["postgres3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres3 is ahead of my wal position"] INFO in the postgres2 patroni log after 5 seconds
-2
View File
@@ -14,8 +14,6 @@ Feature: recovery
Then I receive a response code 200
And I receive a response role master
And I receive a response timeline 1
And "members/postgres0" key in DCS has state=running after 12 seconds
And replication works from postgres0 to postgres1 after 15 seconds
Scenario: check immediate failover when master_start_timeout=0
Given I issue a PATCH request to http://127.0.0.1:8008/config with {"master_start_timeout": 0}
+13 -14
View File
@@ -22,6 +22,9 @@ Feature: standby cluster
Scenario: check permanent logical slots are synced to the replica
Given I run patronictl.py restart batman postgres1 --force
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
When I add the table replicate_me to postgres1
And I get all changes from logical slot test_logical on postgres1
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
Scenario: Detach exiting node from the cluster
When I shut down postgres1
@@ -51,21 +54,17 @@ Feature: standby cluster
When I issue a GET request to http://127.0.0.1:8010/patroni
Then I receive a response code 200
And I receive a response replication_state streaming
And postgres1 does not have a replication slot named test_logical
Scenario: check switchover
Given I run patronictl.py switchover batman1 --force
Then Status code on GET http://127.0.0.1:8010/standby_leader is 200 after 10 seconds
And postgres1 is replicating from postgres2 after 32 seconds
And there is a postgres2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres2 data directory
And postgres1 does not have a logical replication slot named test_logical
Scenario: check failover
When I kill postgres2
And I kill postmaster on postgres2
Then postgres1 is replicating from postgres0 after 32 seconds
And Status code on GET http://127.0.0.1:8009/standby_leader is 200 after 10 seconds
When I issue a GET request to http://127.0.0.1:8009/primary
When I kill postgres1
And I kill postmaster on postgres1
Then postgres2 is replicating from postgres0 after 32 seconds
When I issue a GET request to http://127.0.0.1:8010/primary
Then I receive a response code 503
And I sleep for 3 seconds
When I issue a GET request to http://127.0.0.1:8010/standby_leader
Then I receive a response code 200
And I receive a response role standby_leader
And replication works from postgres0 to postgres1 after 15 seconds
And there is a postgres1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres1 data directory
And replication works from postgres0 to postgres2 after 15 seconds
And there is a postgres2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres2 data directory
+7 -20
View File
@@ -1,4 +1,3 @@
import json
import patroni.psycopg as pg
from behave import step, then
@@ -36,16 +35,11 @@ def kill_patroni(context, name):
return context.pctl.stop(name, kill=True)
@step('I shut down postmaster on {name:w}')
@step('I kill postmaster on {name:w}')
def stop_postgres(context, name):
return context.pctl.stop(name, postgres=True)
@step('I kill postmaster on {name:w}')
def kill_postgres(context, name):
return context.pctl.stop(name, kill=True, postgres=True)
@step('I add the table {table_name:w} to {pg_name:w}')
def add_table(context, table_name, pg_name):
# parse the configuration file and get the port
@@ -111,18 +105,11 @@ def replication_works(context, primary, replica, time_limit):
context.execute_steps(u"""
When I add the table test_{0} to {1}
Then table test_{0} is present on {2} after {3} seconds
""".format(str(time()).replace('.', '_').replace(',', '_'), primary, replica, time_limit))
""".format(int(time()), primary, replica, time_limit))
@then('there is one of {message_list} {level:w} in the {node} patroni log after {timeout:d} seconds')
def check_patroni_log(context, message_list, level, node, timeout):
timeout *= context.timeout_multiplier
message_list = json.loads(message_list)
for _ in range(int(timeout)):
messsages_of_level = context.pctl.read_patroni_log(node, level)
if any(any(message in line for line in messsages_of_level) for message in message_list):
break
time.sleep(1)
else:
assert False, f"There were none of {message_list} {level} in the {node} patroni log after {timeout} seconds"
@then('there is a "{message}" {level:w} in the {node} patroni log')
def check_patroni_log(context, message, level, node):
messsages_of_level = context.pctl.read_patroni_log(node, level)
assert any(message in line for line in messsages_of_level), \
"There was no {0} {1} in the {2} patroni log".format(message, level, node)
+1 -1
View File
@@ -28,7 +28,7 @@ def check_member(context, name, key, value, time_limit):
while time.time() < max_time:
try:
response = json.loads(context.dcs_ctl.query(name))
dcs_value = str(response.get(key))
dcs_value = response.get(key)
if dcs_value == value:
return
except Exception:
+6 -12
View File
@@ -115,18 +115,12 @@ def count_rows(context, name):
assert rows == context.insert_counter, "Distributed table doesn't have expected amount of rows"
@step("there is a transaction in progress on {name:w} changing pg_dist_node after {time_limit:d} seconds")
def check_transaction(context, name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
while time.time() < max_time:
cur = context.pctl.query(name, "SELECT xact_start FROM pg_stat_activity WHERE pid <> pg_backend_pid()"
" AND state = 'idle in transaction' AND query ~ 'citus_update_node'")
if cur.rowcount == 1:
context.xact_start = cur.fetchone()[0]
return
time.sleep(1)
assert False, f"There is no idle in transaction on {name} updating pg_dist_node after {time_limit} seconds"
@step("There is a transaction in progress on {name:w} changing pg_dist_node")
def check_transaction(context, name):
cur = context.pctl.query(name, "SELECT xact_start FROM pg_stat_activity WHERE pid <> pg_backend_pid()"
" AND state = 'idle in transaction' AND query ~ 'citus_update_node'")
assert cur.rowcount == 1, "There is no idle in transaction updating pg_dist_node"
context.xact_start = cur.fetchone()[0]
@step("a transaction finishes in {timeout:d} seconds")
+17 -62
View File
@@ -1,4 +1,3 @@
import json
import time
from behave import step, then
@@ -16,30 +15,21 @@ def create_logical_replication_slot(context, slot_name, pg_name, plugin):
assert False, "Error creating slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin)
@step('{pg_name:w} has a logical replication slot named {slot_name}'
' with the {plugin:w} plugin after {time_limit:d} seconds')
@then('{pg_name:w} has a logical replication slot named {slot_name}'
' with the {plugin:w} plugin after {time_limit:d} seconds')
def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
while time.time() < max_time:
try:
row = context.pctl.query(pg_name, ("SELECT slot_type, plugin FROM pg_replication_slots"
f" WHERE slot_name = '{slot_name}'")).fetchone()
if row:
assert row[0] == "logical", f"Replication slot {slot_name} isn't a logical but {row[0]}"
assert row[1] == plugin, f"Replication slot {slot_name} using plugin {row[1]} rather than {plugin}"
return
except Exception:
pass
time.sleep(1)
assert False, f"Error looking for slot {slot_name} on {pg_name} with plugin {plugin}"
@then('{pg_name:w} has a logical replication slot named {slot_name} with the {plugin:w} plugin')
def has_logical_replication_slot(context, pg_name, slot_name, plugin):
try:
row = context.pctl.query(pg_name, ("SELECT slot_type, plugin FROM pg_replication_slots"
" WHERE slot_name = '{0}'").format(slot_name)).fetchone()
assert row, "Couldn't find replication slot named {0}".format(slot_name)
assert row[0] == "logical", "Found replication slot named {0} but wasn't a logical slot".format(slot_name)
assert row[1] == plugin, ("Found replication slot named {0} but was using plugin "
"{1} rather than {2}").format(slot_name, row[1], plugin)
except pg.Error:
assert False, "Error looking for slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin)
@step('{pg_name:w} does not have a replication slot named {slot_name:w}')
@then('{pg_name:w} does not have a replication slot named {slot_name:w}')
def does_not_have_replication_slot(context, pg_name, slot_name):
@then('{pg_name:w} does not have a logical replication slot named {slot_name}')
def does_not_have_logical_replication_slot(context, pg_name, slot_name):
try:
row = context.pctl.query(pg_name, ("SELECT 1 FROM pg_replication_slots"
" WHERE slot_name = '{0}'").format(slot_name)).fetchone()
@@ -48,14 +38,13 @@ def does_not_have_replication_slot(context, pg_name, slot_name):
assert False, "Error looking for slot {0} on {1}".format(slot_name, pg_name)
@step('{slot_type:w} slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds')
def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit):
@step('Logical slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds')
def logical_slots_in_sync(context, slot_name, pg_name1, pg_name2, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
column = 'confirmed_flush_lsn' if slot_type.lower() == 'logical' else 'restart_lsn'
query = f"SELECT {column} FROM pg_replication_slots WHERE slot_name = '{slot_name}'"
while time.time() < max_time:
try:
query = "SELECT confirmed_flush_lsn FROM pg_replication_slots WHERE slot_name = '{0}'".format(slot_name)
slot1 = context.pctl.query(pg_name1, query).fetchone()
slot2 = context.pctl.query(pg_name2, query).fetchone()
if slot1[0] == slot2[0]:
@@ -63,43 +52,9 @@ def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit)
except Exception:
pass
time.sleep(1)
assert False, \
f"{slot_type} slot {slot_name} is not in sync between {pg_name1} and {pg_name2} after {time_limit} seconds"
assert False, "Logical slot {0} is not in sync between {1} and {2}".format(slot_name, pg_name1, pg_name2)
@step('I get all changes from logical slot {slot_name:w} on {pg_name:w}')
def logical_slot_get_changes(context, slot_name, pg_name):
context.pctl.query(pg_name, "SELECT * FROM pg_logical_slot_get_changes('{0}', NULL, NULL)".format(slot_name))
@step('I get all changes from physical slot {slot_name:w} on {pg_name:w}')
def physical_slot_get_changes(context, slot_name, pg_name):
context.pctl.query(pg_name, f"SELECT * FROM pg_replication_slot_advance('{slot_name}', pg_current_wal_lsn())")
@step('{pg_name:w} has a physical replication slot named {slot_name} after {time_limit:d} seconds')
def has_physical_replication_slot(context, pg_name, slot_name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
query = f"SELECT * FROM pg_catalog.pg_replication_slots WHERE slot_type = 'physical' AND slot_name = '{slot_name}'"
while time.time() < max_time:
try:
row = context.pctl.query(pg_name, query).fetchone()
if row:
return
except Exception:
pass
time.sleep(1)
assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds"
@step('"{name}" key in DCS has {subkey:w} in {key:w}')
def dcs_key_contains(context, name, subkey, key):
response = json.loads(context.dcs_ctl.query(name))
assert key in response and subkey in response[key], f"{name} key in DCS doesn't have {subkey} in {key}"
@step('"{name}" key in DCS does not have {subkey:w} in {key:w}')
def dcs_key_does_not_contain(context, name, subkey, key):
response = json.loads(context.dcs_ctl.query(name))
assert key not in response or subkey not in response[key], f"{name} key in DCS has {subkey} in {key}"
+3 -2
View File
@@ -15,7 +15,9 @@ def start_patroni(context, name, cluster_name):
"scope": cluster_name,
"postgresql": {
"callbacks": callbacks(context, name),
"backup_restore": context.pctl.backup_restore_config()
"backup_restore": {
"command": (context.pctl.PYTHON + " features/backup_restore.py --sourcedir="
+ os.path.join(context.pctl.patroni_path, 'data', 'basebackup').replace('\\', '/'))}
}
})
@@ -32,7 +34,6 @@ def start_patroni_standby_cluster(context, name, cluster_name, name2):
"ttl": 20,
"loop_wait": 2,
"retry_timeout": 5,
"synchronous_mode": True, # should be completely ignored
"standby_cluster": {
"host": "localhost",
"port": port,
+2 -2
View File
@@ -9,8 +9,8 @@ RUN export DEBIAN_FRONTEND=noninteractive \
| xargs apt-get install -y vim-tiny curl jq locales git python3-pip python3-wheel \
## Make sure we have a en_US.UTF-8 locale available
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
&& pip3 install --break-system-packages setuptools \
&& pip3 install --break-system-packages 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
&& pip3 install setuptools \
&& pip3 install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
&& PGHOME=/home/postgres \
&& mkdir -p $PGHOME \
&& chown postgres $PGHOME \
+6 -21
View File
@@ -10,24 +10,12 @@ RUN export DEBIAN_FRONTEND=noninteractive \
| xargs apt-get install -y busybox vim-tiny curl jq less locales git python3-pip python3-wheel lsb-release \
## Make sure we have a en_US.UTF-8 locale available
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
&& if [ $(dpkg --print-architecture) = 'arm64' ]; then \
apt-get install -y postgresql-server-dev-15 \
gcc make autoconf \
libc6-dev flex libcurl4-gnutls-dev \
libicu-dev libkrb5-dev liblz4-dev \
libpam0g-dev libreadline-dev libselinux1-dev\
libssl-dev libxslt1-dev libzstd-dev uuid-dev \
&& git clone -b "main" https://github.com/citusdata/citus.git \
&& MAKEFLAGS="-j $(grep -c ^processor /proc/cpuinfo)" \
&& cd citus && ./configure && make install && cd ../ && rm -rf /citus; \
else \
echo "deb [signed-by=/etc/apt/trusted.gpg.d/citusdata_community.gpg] https://packagecloud.io/citusdata/community/debian/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/citusdata_community.list \
&& curl -sL https://packagecloud.io/citusdata/community/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/citusdata_community.gpg \
&& apt-get update -y \
&& apt-get -y install postgresql-15-citus-12.0; \
fi \
&& pip3 install --break-system-packages setuptools \
&& pip3 install --break-system-packages 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
&& echo "deb [signed-by=/etc/apt/trusted.gpg.d/citusdata_community.gpg] https://packagecloud.io/citusdata/community/debian/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/citusdata_community.list \
&& curl -sL https://packagecloud.io/citusdata/community/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/citusdata_community.gpg \
&& apt-get update -y \
&& apt-get -y install postgresql-$PG_MAJOR-citus-11.3 \
&& pip3 install setuptools \
&& pip3 install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
&& PGHOME=/home/postgres \
&& mkdir -p $PGHOME \
&& chown postgres $PGHOME \
@@ -38,9 +26,6 @@ RUN export DEBIAN_FRONTEND=noninteractive \
&& chmod 664 /etc/passwd \
# Clean up
&& apt-get remove -y git python3-pip python3-wheel \
postgresql-server-dev-15 gcc make autoconf \
libc6-dev flex libicu-dev libkrb5-dev liblz4-dev \
libpam0g-dev libreadline-dev libselinux1-dev libssl-dev libxslt1-dev libzstd-dev uuid-dev \
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/* /root/.cache
+53 -7
View File
@@ -3,14 +3,23 @@
:var PATRONI_ENV_PREFIX: prefix for Patroni related configuration environment variables.
:var KUBERNETES_ENV_PREFIX: prefix for Kubernetes related configuration environment variables.
:var MIN_PSYCOPG2: minimum version of :mod:`psycopg2` required by Patroni to work.
:var MIN_PSYCOPG3: minimum version of :mod:`psycopg` required by Patroni to work.
"""
from typing import Iterator, Tuple
import sys
from typing import Any, Callable, Iterator, Tuple
PATRONI_ENV_PREFIX = 'PATRONI_'
KUBERNETES_ENV_PREFIX = 'KUBERNETES_'
MIN_PSYCOPG2 = (2, 5, 4)
MIN_PSYCOPG3 = (3, 0, 0)
def fatal(string: str, *args: Any) -> None:
"""Write a fatal message to stderr and exit with code ``1``.
:param string: message to be written before exiting.
"""
sys.exit('FATAL: ' + string.format(*args))
def parse_version(version: str) -> Tuple[int, ...]:
@@ -19,25 +28,25 @@ def parse_version(version: str) -> Tuple[int, ...]:
.. note::
Designed for easy comparison of software versions in Python.
:param version: human-readable software version, e.g. ``2.5.4.dev1 (dt dec pq3 ext lo64)``.
:param version: human-readable software version, e.g. ``2.5.4``.
:returns: tuple of *version* parts, each part as an integer.
:Example:
>>> parse_version('2.5.4.dev1 (dt dec pq3 ext lo64)')
>>> parse_version('2.5.4')
(2, 5, 4)
"""
def _parse_version(version: str) -> Iterator[int]:
"""Yield each part of a human-readable version string as an integer.
:param version: human-readable software version, e.g. ``2.5.4.dev1``.
:param version: human-readable software version, e.g. ``2.5.4``.
:yields: each part of *version* as an integer.
:Example:
>>> tuple(_parse_version('2.5.4.dev1'))
>>> tuple(_parse_version('2.5.4'))
(2, 5, 4)
"""
for e in version.split('.'):
@@ -46,3 +55,40 @@ def parse_version(version: str) -> Tuple[int, ...]:
except ValueError:
break
return tuple(_parse_version(version.split(' ')[0]))
def check_psycopg(_min_psycopg2: Tuple[int, ...] = MIN_PSYCOPG2,
_parse_version: Callable[[str], Tuple[int, ...]] = parse_version) -> None:
"""Ensure at least one among :mod:`psycopg2` or :mod:`psycopg` libraries are available in the environment.
.. note::
We pass ``MIN_PSYCOPG2`` and :func:`parse_version` as arguments to simplify usage of :func:`check_psycopg` from
the ``setup.py``.
.. note::
Patroni chooses :mod:`psycopg2` over :mod:`psycopg`, if possible.
If nothing meeting the requirements is found, then exit with a fatal message.
:param _min_psycopg2: minimum required version in case :mod:`psycopg2` is chosen.
:param _parse_version: function used to parse :mod:`psycopg2`/:mod:`psycopg` version into a comparable object.
"""
min_psycopg2_str = '.'.join(map(str, _min_psycopg2))
# try psycopg2
try:
from psycopg2 import __version__
if _parse_version(__version__) >= _min_psycopg2:
return
version_str = __version__.split(' ')[0]
except ImportError:
version_str = None
# try psycopg3
try:
from psycopg import __version__
except ImportError:
error = 'Patroni requires psycopg2>={0}, psycopg2-binary, or psycopg>=3.0'.format(min_psycopg2_str)
if version_str is not None:
error += ', but only psycopg2=={0} is available'.format(version_str)
fatal(error)
+26 -186
View File
@@ -1,8 +1,3 @@
"""Patroni main entry point.
Implement ``patroni`` main daemon and expose its entry point.
"""
import logging
import os
import signal
@@ -10,11 +5,9 @@ import sys
import time
from argparse import Namespace
from typing import Any, Dict, List, Optional, TYPE_CHECKING
from typing import Any, Dict, Optional, TYPE_CHECKING
from patroni import MIN_PSYCOPG2, MIN_PSYCOPG3, parse_version
from patroni.daemon import AbstractPatroniDaemon, abstract_main, get_base_arg_parser
from patroni.tags import Tags
if TYPE_CHECKING: # pragma: no cover
from .config import Config
@@ -22,33 +15,9 @@ if TYPE_CHECKING: # pragma: no cover
logger = logging.getLogger(__name__)
class Patroni(AbstractPatroniDaemon, Tags):
"""Implement ``patroni`` command daemon.
:ivar version: Patroni version.
:ivar dcs: DCS object.
:ivar watchdog: watchdog handler, if configured to use watchdog.
:ivar postgresql: managed Postgres instance.
:ivar api: REST API server instance of this node.
:ivar request: wrapper for performing HTTP requests.
:ivar ha: HA handler.
:ivar next_run: time when to run the next HA loop cycle.
:ivar scheduled_restart: when a restart has been scheduled to occur, if any. In that case, should contain two keys:
* ``schedule``: timestamp when restart should occur;
* ``postmaster_start_time``: timestamp when Postgres was last started.
"""
class Patroni(AbstractPatroniDaemon):
def __init__(self, config: 'Config') -> None:
"""Create a :class:`Patroni` instance with the given *config*.
Get a connection to the DCS, configure watchdog (if required), set up Patroni interface with Postgres, configure
the HA loop and bring the REST API up.
.. note::
Expected to be instantiated and run through :func:`~patroni.daemon.abstract_main`.
:param config: Patroni configuration.
"""
from patroni.api import RestApiServer
from patroni.dcs import get_dcs
from patroni.ha import Ha
@@ -72,22 +41,11 @@ class Patroni(AbstractPatroniDaemon, Tags):
self.api = RestApiServer(self, self.config['restapi'])
self.ha = Ha(self)
self._tags = self._get_tags()
self.tags = self.get_tags()
self.next_run = time.time()
self.scheduled_restart: Dict[str, Any] = {}
def load_dynamic_configuration(self) -> None:
"""Load Patroni dynamic configuration.
Load dynamic configuration from the DCS, if `/config` key is available in the DCS, otherwise fall back to
``bootstrap.dcs`` section from the configuration file.
If the DCS connection fails returning the exception :class:`~patroni.exceptions.DCSError` an attempt will be
remade every 5 seconds.
.. note::
This method is called only once, at the time when Patroni is started.
"""
from patroni.exceptions import DCSError
while True:
try:
@@ -127,31 +85,23 @@ class Patroni(AbstractPatroniDaemon, Tags):
except Exception:
return
def _get_tags(self) -> Dict[str, Any]:
"""Get tags configured for this node, if any.
def get_tags(self) -> Dict[str, Any]:
return {tag: value for tag, value in self.config.get('tags', {}).items()
if tag not in ('clonefrom', 'nofailover', 'noloadbalance', 'nosync') or value}
:returns: a dictionary of tags set for this node.
"""
return self._filter_tags(self.config.get('tags', {}))
@property
def nofailover(self) -> bool:
return bool(self.tags.get('nofailover', False))
@property
def nosync(self) -> bool:
return bool(self.tags.get('nosync', False))
def reload_config(self, sighup: bool = False, local: Optional[bool] = False) -> None:
"""Apply new configuration values for ``patroni`` daemon.
Reload:
* Cached tags;
* Request wrapper configuration;
* REST API configuration;
* Watchdog configuration;
* Postgres configuration;
* DCS configuration.
:param sighup: if it is related to a SIGHUP signal.
:param local: if there has been changes to the local configuration file.
"""
try:
super(Patroni, self).reload_config(sighup, local)
if local:
self._tags = self._get_tags()
self.tags = self.get_tags()
self.request.reload_config(self.config)
if local or sighup and self.api.reload_local_certificate():
self.api.reload_config(self.config['restapi'])
@@ -162,16 +112,14 @@ class Patroni(AbstractPatroniDaemon, Tags):
logger.exception('Failed to reload config_file=%s', self.config.config_file)
@property
def tags(self) -> Dict[str, Any]:
"""Tags configured for this node, if any."""
return self._tags
def replicatefrom(self):
return self.tags.get('replicatefrom')
@property
def noloadbalance(self):
return bool(self.tags.get('noloadbalance', False))
def schedule_next_run(self) -> None:
"""Schedule the next run of the ``patroni`` daemon main loop.
Next run is scheduled based on previous run plus value of ``loop_wait`` configuration from DCS. If that has
already been exceeded, run the next cycle immediately.
"""
self.next_run += self.dcs.loop_wait
current_time = time.time()
nap_time = self.next_run - current_time
@@ -185,21 +133,11 @@ class Patroni(AbstractPatroniDaemon, Tags):
self.next_run = time.time()
def run(self) -> None:
"""Run ``patroni`` daemon process main loop.
Start the REST API and keep running HA cycles every ``loop_wait`` seconds.
"""
self.api.start()
self.next_run = time.time()
super(Patroni, self).run()
def _run_cycle(self) -> None:
"""Run a cycle of the ``patroni`` daemon main loop.
Run an HA cycle and schedule the next cycle run. If any dynamic configuration change request is detected, apply
the change and cache the new dynamic configuration values in ``patroni.dynamic.json`` file under Postgres data
directory.
"""
logger.info(self.ha.run_cycle())
if self.dcs.cluster and self.dcs.cluster.config and self.dcs.cluster.config.data \
@@ -212,10 +150,6 @@ class Patroni(AbstractPatroniDaemon, Tags):
self.schedule_next_run()
def _shutdown(self) -> None:
"""Perform shutdown of ``patroni`` daemon process.
Shut down the REST API and the HA handler.
"""
try:
self.api.shutdown()
except Exception:
@@ -227,54 +161,18 @@ class Patroni(AbstractPatroniDaemon, Tags):
def patroni_main(configfile: str) -> None:
"""Configure and start ``patroni`` main daemon process.
:param configfile: path to Patroni configuration file.
"""
from multiprocessing import freeze_support
# Windows executables created by PyInstaller are frozen, thus we need to enable frozen support for
# :mod:`multiprocessing` to avoid :class:`RuntimeError` exceptions.
freeze_support()
abstract_main(Patroni, configfile)
def process_arguments() -> Namespace:
"""Process command-line arguments.
Create a basic command-line parser through :func:`~patroni.daemon.get_base_arg_parser`, extend its capabilities by
adding these flags and parse command-line arguments.:
* ``--validate-config`` -- used to validate the Patroni configuration file
* ``--generate-config`` -- used to generate Patroni configuration from a running PostgreSQL instance
* ``--generate-sample-config`` -- used to generate a sample Patroni configuration
.. note::
If running with ``--generate-config``, ``--generate-sample-config`` or ``--validate-flag`` will exit
after generating or validating configuration.
:returns: parsed arguments, if not running with ``--validate-config`` flag.
"""
from patroni.config_generator import generate_config
parser = get_base_arg_parser()
group = parser.add_mutually_exclusive_group()
group.add_argument('--validate-config', action='store_true', help='Run config validator and exit')
group.add_argument('--generate-sample-config', action='store_true',
help='Generate a sample Patroni yaml configuration file')
group.add_argument('--generate-config', action='store_true',
help='Generate a Patroni yaml configuration file for a running instance')
parser.add_argument('--dsn', help='Optional DSN string of the instance to be used as a source \
for config generation. Superuser connection is required.')
parser.add_argument('--validate-config', action='store_true', help='Run config validator and exit')
args = parser.parse_args()
if args.generate_sample_config:
generate_config(args.configfile, True, None)
sys.exit(0)
elif args.generate_config:
generate_config(args.configfile, False, args.dsn)
sys.exit(0)
elif args.validate_config:
if args.validate_config:
from patroni.validator import schema
from patroni.config import Config, ConfigParseError
@@ -287,60 +185,13 @@ def process_arguments() -> Namespace:
return args
def check_psycopg() -> None:
"""Ensure at least one among :mod:`psycopg2` or :mod:`psycopg` libraries are available in the environment.
.. note::
Patroni chooses :mod:`psycopg2` over :mod:`psycopg`, if possible.
If nothing meeting the requirements is found, then exit with a fatal message.
"""
min_psycopg2_str = '.'.join(map(str, MIN_PSYCOPG2))
min_psycopg3_str = '.'.join(map(str, MIN_PSYCOPG3))
available_versions: List[str] = []
# try psycopg2
try:
from psycopg2 import __version__
if parse_version(__version__) >= MIN_PSYCOPG2:
return
available_versions.append('psycopg2=={0}'.format(__version__.split(' ')[0]))
except ImportError:
logger.debug('psycopg2 module is not available')
# try psycopg3
try:
from psycopg import __version__
if parse_version(__version__) >= MIN_PSYCOPG3:
return
available_versions.append('psycopg=={0}'.format(__version__.split(' ')[0]))
except ImportError:
logger.debug('psycopg module is not available')
error = f'FATAL: Patroni requires psycopg2>={min_psycopg2_str}, psycopg2-binary, or psycopg>={min_psycopg3_str}'
if available_versions:
error += ', but only {0} {1} available'.format(
' and '.join(available_versions),
'is' if len(available_versions) == 1 else 'are')
sys.exit(error)
def main() -> None:
"""Main entrypoint of :mod:`patroni.__main__`.
Process command-line arguments, ensure :mod:`psycopg2` (or :mod:`psycopg`) attendee the pre-requisites and start
``patroni`` daemon process.
.. note::
If running through a Docker container, make the main process take care of init process duties and run
``patroni`` daemon as another process. In that case relevant signals received by the main process and forwarded
to ``patroni`` daemon process.
"""
check_psycopg()
from patroni import check_psycopg
args = process_arguments()
check_psycopg()
if os.getpid() != 1:
return patroni_main(args.configfile)
@@ -350,13 +201,7 @@ def main() -> None:
# Looks like we are in a docker, so we will act like init
def sigchld_handler(signo: int, stack_frame: Optional[FrameType]) -> None:
"""Handle ``SIGCHLD`` received by main process from ``patroni`` daemon when the daemon terminates.
:param signo: signal number.
:param stack_frame: current stack frame.
"""
try:
# log exit code of all children processes, and break loop when there is none left
while True:
ret = os.waitpid(-1, os.WNOHANG)
if ret == (0, 0):
@@ -366,12 +211,7 @@ def main() -> None:
except OSError:
pass
def passtochild(signo: int, stack_frame: Optional[FrameType]) -> None:
"""Forward a signal *signo* from main process to child process.
:param signo: signal number.
:param stack_frame: current stack frame.
"""
def passtochild(signo: int, stack_frame: Optional[FrameType]):
if pid:
os.kill(pid, signo)
+56 -78
View File
@@ -103,7 +103,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
if TYPE_CHECKING: # pragma: no cover
assert isinstance(server, RestApiServer)
super(RestApiHandler, self).__init__(request, client_address, server)
self.server: 'RestApiServer' = server # pyright: ignore [reportIncompatibleVariableOverride]
self.server: 'RestApiServer' = server
self.__start_time: float = 0.0
self.path_query: Dict[str, List[str]] = {}
@@ -198,11 +198,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
response['database_system_identifier'] = patroni.postgresql.sysid
if patroni.postgresql.pending_restart:
response['pending_restart'] = True
response['patroni'] = {
'version': patroni.version,
'scope': patroni.postgresql.scope,
'name': patroni.postgresql.name
}
response['patroni'] = {'version': patroni.version, 'scope': patroni.postgresql.scope}
if patroni.scheduled_restart:
response['scheduled_restart'] = patroni.scheduled_restart.copy()
del response['scheduled_restart']['postmaster_start_time']
@@ -451,12 +447,9 @@ class RestApiHandler(BaseHTTPRequestHandler):
Write an HTTP response with JSON content based on the output of :func:`~patroni.utils.cluster_as_json`, with
HTTP status ``200`` and the JSON representation of the cluster topology.
"""
cluster = self.server.patroni.dcs.get_cluster()
cluster = self.server.patroni.dcs.get_cluster(True)
global_config = self.server.patroni.config.get_global_config(cluster)
response = cluster_as_json(cluster, global_config)
response['scope'] = self.server.patroni.postgresql.scope
self._write_json_response(200, response)
self._write_json_response(200, cluster_as_json(cluster, global_config))
def do_GET_history(self) -> None:
"""Handle a ``GET`` request to ``/history`` path.
@@ -533,113 +526,113 @@ class RestApiHandler(BaseHTTPRequestHandler):
metrics: List[str] = []
labels = f'{{scope="{patroni.postgresql.scope}",name="{patroni.postgresql.name}"}}'
scope_label = '{{scope="{0}"}}'.format(patroni.postgresql.scope)
metrics.append("# HELP patroni_version Patroni semver without periods.")
metrics.append("# TYPE patroni_version gauge")
padded_semver = ''.join([x.zfill(2) for x in patroni.version.split('.')]) # 2.0.2 => 020002
metrics.append("patroni_version{0} {1}".format(labels, padded_semver))
metrics.append("patroni_version{0} {1}".format(scope_label, padded_semver))
metrics.append("# HELP patroni_postgres_running Value is 1 if Postgres is running, 0 otherwise.")
metrics.append("# TYPE patroni_postgres_running gauge")
metrics.append("patroni_postgres_running{0} {1}".format(labels, int(postgres['state'] == 'running')))
metrics.append("patroni_postgres_running{0} {1}".format(scope_label, int(postgres['state'] == 'running')))
metrics.append("# HELP patroni_postmaster_start_time Epoch seconds since Postgres started.")
metrics.append("# TYPE patroni_postmaster_start_time gauge")
postmaster_start_time = postgres.get('postmaster_start_time')
postmaster_start_time = (postmaster_start_time - epoch).total_seconds() if postmaster_start_time else 0
metrics.append("patroni_postmaster_start_time{0} {1}".format(labels, postmaster_start_time))
metrics.append("patroni_postmaster_start_time{0} {1}".format(scope_label, postmaster_start_time))
metrics.append("# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.")
metrics.append("# TYPE patroni_master gauge")
metrics.append("patroni_master{0} {1}".format(labels, int(postgres['role'] in ('master', 'primary'))))
metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
metrics.append("# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.")
metrics.append("# TYPE patroni_primary gauge")
metrics.append("patroni_primary{0} {1}".format(labels, int(postgres['role'] in ('master', 'primary'))))
metrics.append("patroni_primary{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
metrics.append("# HELP patroni_xlog_location Current location of the Postgres"
" transaction log, 0 if this node is not the leader.")
metrics.append("# TYPE patroni_xlog_location counter")
metrics.append("patroni_xlog_location{0} {1}".format(labels, postgres.get('xlog', {}).get('location', 0)))
metrics.append("patroni_xlog_location{0} {1}".format(scope_label, postgres.get('xlog', {}).get('location', 0)))
metrics.append("# HELP patroni_standby_leader Value is 1 if this node is the standby_leader, 0 otherwise.")
metrics.append("# TYPE patroni_standby_leader gauge")
metrics.append("patroni_standby_leader{0} {1}".format(labels, int(postgres['role'] == 'standby_leader')))
metrics.append("patroni_standby_leader{0} {1}".format(scope_label, int(postgres['role'] == 'standby_leader')))
metrics.append("# HELP patroni_replica Value is 1 if this node is a replica, 0 otherwise.")
metrics.append("# TYPE patroni_replica gauge")
metrics.append("patroni_replica{0} {1}".format(labels, int(postgres['role'] == 'replica')))
metrics.append("patroni_replica{0} {1}".format(scope_label, int(postgres['role'] == 'replica')))
metrics.append("# HELP patroni_sync_standby Value is 1 if this node is a sync standby replica, 0 otherwise.")
metrics.append("# TYPE patroni_sync_standby gauge")
metrics.append("patroni_sync_standby{0} {1}".format(labels, int(postgres.get('sync_standby', False))))
metrics.append("patroni_sync_standby{0} {1}".format(scope_label, int(postgres.get('sync_standby', False))))
metrics.append("# HELP patroni_xlog_received_location Current location of the received"
" Postgres transaction log, 0 if this node is not a replica.")
metrics.append("# TYPE patroni_xlog_received_location counter")
metrics.append("patroni_xlog_received_location{0} {1}"
.format(labels, postgres.get('xlog', {}).get('received_location', 0)))
.format(scope_label, postgres.get('xlog', {}).get('received_location', 0)))
metrics.append("# HELP patroni_xlog_replayed_location Current location of the replayed"
" Postgres transaction log, 0 if this node is not a replica.")
metrics.append("# TYPE patroni_xlog_replayed_location counter")
metrics.append("patroni_xlog_replayed_location{0} {1}"
.format(labels, postgres.get('xlog', {}).get('replayed_location', 0)))
.format(scope_label, postgres.get('xlog', {}).get('replayed_location', 0)))
metrics.append("# HELP patroni_xlog_replayed_timestamp Current timestamp of the replayed"
" Postgres transaction log, 0 if null.")
metrics.append("# TYPE patroni_xlog_replayed_timestamp gauge")
replayed_timestamp = postgres.get('xlog', {}).get('replayed_timestamp')
replayed_timestamp = (replayed_timestamp - epoch).total_seconds() if replayed_timestamp else 0
metrics.append("patroni_xlog_replayed_timestamp{0} {1}".format(labels, replayed_timestamp))
metrics.append("patroni_xlog_replayed_timestamp{0} {1}".format(scope_label, replayed_timestamp))
metrics.append("# HELP patroni_xlog_paused Value is 1 if the Postgres xlog is paused, 0 otherwise.")
metrics.append("# TYPE patroni_xlog_paused gauge")
metrics.append("patroni_xlog_paused{0} {1}"
.format(labels, int(postgres.get('xlog', {}).get('paused', False) is True)))
.format(scope_label, int(postgres.get('xlog', {}).get('paused', False) is True)))
if postgres.get('server_version', 0) >= 90600:
metrics.append("# HELP patroni_postgres_streaming Value is 1 if Postgres is streaming, 0 otherwise.")
metrics.append("# TYPE patroni_postgres_streaming gauge")
metrics.append("patroni_postgres_streaming{0} {1}"
.format(labels, int(postgres.get('replication_state') == 'streaming')))
.format(scope_label, int(postgres.get('replication_state') == 'streaming')))
metrics.append("# HELP patroni_postgres_in_archive_recovery Value is 1"
" if Postgres is replicating from archive, 0 otherwise.")
metrics.append("# TYPE patroni_postgres_in_archive_recovery gauge")
metrics.append("patroni_postgres_in_archive_recovery{0} {1}"
.format(labels, int(postgres.get('replication_state') == 'in archive recovery')))
.format(scope_label, int(postgres.get('replication_state') == 'in archive recovery')))
metrics.append("# HELP patroni_postgres_server_version Version of Postgres (if running), 0 otherwise.")
metrics.append("# TYPE patroni_postgres_server_version gauge")
metrics.append("patroni_postgres_server_version {0} {1}".format(labels, postgres.get('server_version', 0)))
metrics.append("patroni_postgres_server_version {0} {1}".format(scope_label, postgres.get('server_version', 0)))
metrics.append("# HELP patroni_cluster_unlocked Value is 1 if the cluster is unlocked, 0 if locked.")
metrics.append("# TYPE patroni_cluster_unlocked gauge")
metrics.append("patroni_cluster_unlocked{0} {1}".format(labels, int(postgres.get('cluster_unlocked', 0))))
metrics.append("patroni_cluster_unlocked{0} {1}".format(scope_label, int(postgres.get('cluster_unlocked', 0))))
metrics.append("# HELP patroni_failsafe_mode_is_active Value is 1 if failsafe mode is active, 0 if inactive.")
metrics.append("# TYPE patroni_failsafe_mode_is_active gauge")
metrics.append("patroni_failsafe_mode_is_active{0} {1}"
.format(labels, int(postgres.get('failsafe_mode_is_active', 0))))
.format(scope_label, int(postgres.get('failsafe_mode_is_active', 0))))
metrics.append("# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.")
metrics.append("# TYPE patroni_postgres_timeline counter")
metrics.append("patroni_postgres_timeline{0} {1}".format(labels, postgres.get('timeline', 0)))
metrics.append("patroni_postgres_timeline{0} {1}".format(scope_label, postgres.get('timeline', 0)))
metrics.append("# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully"
" by Patroni.")
metrics.append("# TYPE patroni_dcs_last_seen gauge")
metrics.append("patroni_dcs_last_seen{0} {1}".format(labels, postgres.get('dcs_last_seen', 0)))
metrics.append("patroni_dcs_last_seen{0} {1}".format(scope_label, postgres.get('dcs_last_seen', 0)))
metrics.append("# HELP patroni_pending_restart Value is 1 if the node needs a restart, 0 otherwise.")
metrics.append("# TYPE patroni_pending_restart gauge")
metrics.append("patroni_pending_restart{0} {1}"
.format(labels, int(patroni.postgresql.pending_restart)))
.format(scope_label, int(patroni.postgresql.pending_restart)))
metrics.append("# HELP patroni_is_paused Value is 1 if auto failover is disabled, 0 otherwise.")
metrics.append("# TYPE patroni_is_paused gauge")
metrics.append("patroni_is_paused{0} {1}".format(labels, int(postgres.get('pause', 0))))
metrics.append("patroni_is_paused{0} {1}".format(scope_label, int(postgres.get('pause', 0))))
self.write_response(200, '\n'.join(metrics) + '\n', content_type='text/plain')
@@ -690,7 +683,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
"""
request = self._read_json_content()
if request:
cluster = self.server.patroni.dcs.get_cluster()
cluster = self.server.patroni.dcs.get_cluster(True)
if not (cluster.config and cluster.config.modify_version):
return self.send_error(503)
data = cluster.config.data.copy()
@@ -1074,8 +1067,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
* ``412``: if operation is not possible;
* ``503``: if unable to register the operation to the DCS;
* HTTP status returned by :func:`parse_schedule`, if any error was observed while parsing the schedule;
* HTTP status returned by :func:`poll_failover_result` if the operation has been processed immediately;
* ``400``: if none of the above applies.
* HTTP status returned by :func:`poll_failover_result` if the operation has been processed immediately.
.. note::
If unable to parse the request body, then the request is silently discarded.
@@ -1102,22 +1094,15 @@ class RestApiHandler(BaseHTTPRequestHandler):
data = 'Switchover could be performed only from a specific leader'
if not data and scheduled_at:
if action == 'failover':
data = "Failover can't be scheduled"
elif global_config.is_paused:
data = "Can't schedule switchover in the paused state"
else:
if not leader:
data = 'Scheduled {0} is possible only from a specific leader'.format(action)
if not data and global_config.is_paused:
data = "Can't schedule {0} in the paused state".format(action)
if not data:
(status_code, data, scheduled_at) = self.parse_schedule(scheduled_at, action)
if not data and global_config.is_paused and not candidate:
data = 'Switchover is possible only to a specific candidate in a paused state'
if action == 'failover' and leader:
logger.warning('received failover request with leader specifed - performing switchover instead')
action = 'switchover'
if not data and leader == candidate:
data = 'Switchover target and source are the same'
data = action.title() + ' is possible only to a specific candidate in a paused state'
if not data and not scheduled_at:
data = self.is_failover_possible(cluster, leader, candidate, action)
@@ -1134,7 +1119,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
status_code, data = self.poll_failover_result(cluster.leader and cluster.leader.name,
candidate, action)
else:
data = 'failed to write failover key into DCS'
data = 'failed to write {0} key into DCS'.format(action)
status_code = 503
# pyright thinks ``status_code`` can be ``None`` because ``parse_schedule`` call may return ``None``. However,
# if that's the case, ``status_code`` will be overwritten somewhere between ``parse_schedule`` and
@@ -1166,7 +1151,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
patroni = self.server.patroni
if patroni.postgresql.citus_handler.is_coordinator() and patroni.ha.is_leader():
cluster = patroni.dcs.get_cluster()
cluster = patroni.dcs.get_cluster(True)
patroni.postgresql.citus_handler.handle_event(cluster, request)
self.write_response(200, 'OK')
@@ -1198,18 +1183,20 @@ class RestApiHandler(BaseHTTPRequestHandler):
self.command = mname
return ret
def query(self, sql: str, *params: Any, retry: bool = False) -> List[Tuple[Any, ...]]:
"""Execute *sql* query with *params* and optionally return results.
def query(self, sql: str, *params: Any, **kwargs: Any) -> List[Tuple[Any, ...]]:
"""Execute *sql* query with *params*.
:param sql: the SQL statement to be run.
:param params: positional arguments to call :func:`RestApiServer.query` with.
:param retry: whether the query should be retried upon failure or given up immediately.
:param kwargs: can contain the key ``retry``. If the key is present its value should be a :class:`bool` which
indicates whether the query should be retried upon failure or given up immediately.
:returns: a list of rows that were fetched from the database.
"""
if not retry:
if not kwargs.get('retry', False):
return self.server.query(sql, *params)
return Retry(delay=1, retry_exceptions=PostgresConnectionException)(self.server.query, sql, *params)
retry = Retry(delay=1, retry_exceptions=PostgresConnectionException)
return retry(self.server.query, sql, *params)
def get_postgresql_status(self, retry: bool = False) -> Dict[str, Any]:
"""Builds an object representing a status of "postgres".
@@ -1275,8 +1262,8 @@ class RestApiHandler(BaseHTTPRequestHandler):
" application_name, client_addr, w.state, sync_state, sync_priority"
" FROM pg_catalog.pg_stat_get_wal_senders() w, pg_catalog.pg_stat_get_activity(pid)) AS ri")
row = self.query(stmt.format(postgresql.wal_name, postgresql.lsn_name,
postgresql.wal_flush), retry=retry)[0]
row = self.query(stmt.format(postgresql.wal_name, postgresql.lsn_name), retry=retry)[0]
result = {
'state': postgresql.state,
'postmaster_start_time': row[0],
@@ -1381,10 +1368,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
self.daemon = True
def query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
"""Execute *sql* query with *params* and optionally return results.
.. note::
Prefer to use own connection to postgres and fallback to ``heartbeat`` when own isn't available.
"""Execute *sql* query with *params*.
:param sql: the SQL statement to be run.
:param params: positional arguments to be used as parameters for *sql*.
@@ -1395,21 +1379,15 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
:class:`psycopg.Error`: if had issues while executing *sql*.
:class:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database.
"""
# We first try to get a heartbeat connection because it is always required for the main thread.
cursor = None
try:
heartbeat_connection = self.patroni.postgresql.connection_pool.get('heartbeat')
heartbeat_connection.get() # try to open psycopg connection to postgres
except psycopg.Error as exc:
raise PostgresConnectionException('connection problems') from exc
try:
connection = self.patroni.postgresql.connection_pool.get('restapi')
connection.get() # try to open psycopg connection to postgres
except psycopg.Error:
logger.debug('restapi connection to postgres is not available')
connection = heartbeat_connection
return connection.query(sql, *params)
with self.patroni.postgresql.connection().cursor() as cursor:
cursor.execute(sql.encode('utf-8'), params)
return [r for r in cursor]
except psycopg.Error as e:
if cursor and cursor.connection.closed == 0:
raise e
raise PostgresConnectionException('connection problems')
@staticmethod
def _set_fd_cloexec(fd: socket.socket) -> None:
@@ -1575,7 +1553,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
:param listen: IP and port to bind REST API to. It should be a string in the format ``host:port``, where
``host`` can be a hostname or IP address. It is the value of ``restapi.listen`` setting.
:param ssl_options: dictionary that may contain the following keys, depending on what has been configured in
``restapi`` section:
``restapi` section:
* ``certfile``: path to PEM certificate. If given, will start in HTTPS mode;
* ``keyfile``: path to key of ``certfile``;
+1 -8
View File
@@ -3,7 +3,7 @@
Provides a case insensitive :class:`dict` and :class:`set` object types.
"""
from collections import OrderedDict
from typing import Any, Collection, Dict, Iterator, KeysView, MutableMapping, MutableSet, Optional
from typing import Any, Collection, Dict, Iterator, MutableMapping, MutableSet, Optional
class CaseInsensitiveSet(MutableSet[str]):
@@ -187,13 +187,6 @@ class CaseInsensitiveDict(MutableMapping[str, Any]):
"""
return CaseInsensitiveDict({v[0]: v[1] for v in self._values.values()})
def keys(self) -> KeysView[str]:
"""Return a new view of the dict's keys.
:returns: a set-like object providing a view on the dict's keys
"""
return self._values.keys()
def __repr__(self) -> str:
"""Get a string representation of the dict.
+66 -429
View File
@@ -1,4 +1,3 @@
"""Facilities related to Patroni configuration."""
import json
import logging
import os
@@ -37,162 +36,121 @@ _AUTH_ALLOWED_PARAMETERS = (
def default_validator(conf: Dict[str, Any]) -> List[str]:
"""Ensure *conf* is not empty.
Designed to be used as default validator for :class:`Config` objects, if no specific validator is provided.
:param conf: configuration to be validated.
:returns: an empty list -- :class:`Config` expects the validator to return a list of 0 or more issues found while
validating the configuration.
:raises:
:class:`ConfigParseError`: if *conf* is empty.
"""
if not conf:
raise ConfigParseError("Config is empty.")
return []
class GlobalConfig(object):
"""A class that wraps global configuration and provides convenient methods to access/check values.
It is instantiated either by calling :func:`get_global_config` or :meth:`Config.get_global_config`, which picks
either a configuration from provided :class:`Cluster` object (the most up-to-date) or from the
local cache if :class:`ClusterConfig` is not initialized or doesn't have a valid config.
"""A class that wrapps global configuration and provides convinient methods to access/check values.
It is instantiated by calling :func:`Config.global_config` method which picks either a
configuration from provided :class:`Cluster` object (the most up-to-date) or from the
local cache if :class::`ClusterConfig` is not initialized or doesn't have a valid config.
"""
def __init__(self, config: Dict[str, Any]) -> None:
"""Initialize :class:`GlobalConfig` object with given *config*.
"""Initialize :class:`GlobalConfig` object.
:param config: current configuration either from
:class:`ClusterConfig` or from :func:`Config.dynamic_configuration`.
:class:`ClusterConfig` or from :class:`Config.dynamic_configuration`
"""
self.__config = config
def get(self, name: str) -> Any:
"""Gets global configuration value by *name*.
"""Gets global configuration value by name.
:param name: parameter name.
:returns: configuration value or ``None`` if it is missing.
:param name: parameter name
:returns: configuration value or `None` if it is missing
"""
return self.__config.get(name)
def check_mode(self, mode: str) -> bool:
"""Checks whether the certain parameter is enabled.
:param mode: parameter name, e.g. ``synchronous_mode``, ``failsafe_mode``, ``pause``, ``check_timeline``, and
so on.
:returns: ``True`` if parameter *mode* is enabled in the global configuration.
:param mode: parameter name could be: synchronous_mode, failsafe_mode, pause, check_timeline, and so on
:returns: `True` if *mode* is enabled in the global configuration.
"""
return bool(parse_bool(self.__config.get(mode)))
@property
def is_paused(self) -> bool:
"""``True`` if cluster is in maintenance mode."""
""":returns: `True` if cluster is in maintenance mode."""
return self.check_mode('pause')
@property
def is_synchronous_mode(self) -> bool:
"""``True`` if synchronous replication is requested and it is not a standby cluster config."""
return self.check_mode('synchronous_mode') and not self.is_standby_cluster
""":returns: `True` if synchronous replication is requested."""
return self.check_mode('synchronous_mode')
@property
def is_synchronous_mode_strict(self) -> bool:
"""``True`` if at least one synchronous node is required."""
""":returns: `True` if at least one synchronous node is required."""
return self.check_mode('synchronous_mode_strict')
def get_standby_cluster_config(self) -> Union[Dict[str, Any], Any]:
"""Get ``standby_cluster`` configuration.
:returns: a copy of ``standby_cluster`` configuration.
"""
""":returns: "standby_cluster" configuration."""
return deepcopy(self.get('standby_cluster'))
@property
def is_standby_cluster(self) -> bool:
"""``True`` if global configuration has a valid ``standby_cluster`` section."""
""":returns: `True` if global configuration has a valid "standby_cluster" section."""
config = self.get_standby_cluster_config()
return isinstance(config, dict) and\
bool(config.get('host') or config.get('port') or config.get('restore_command'))
def get_int(self, name: str, default: int = 0) -> int:
"""Gets current value of *name* from the global configuration and try to return it as :class:`int`.
"""Gets current value from the global configuration and trying to return it as int.
:param name: name of the parameter.
:param default: default value if *name* is not in the configuration or invalid.
:returns: currently configured value of *name* from the global configuration or *default* if it is not set or
invalid.
:param name: name of the parameter
:param default: default value if *name* is not in the configuration or invalid
:returns: currently configured value from the global configuration or *default* if it is not set or invalid.
"""
ret = parse_int(self.get(name))
return default if ret is None else ret
@property
def min_synchronous_nodes(self) -> int:
"""The minimal number of synchronous nodes based on whether ``synchronous_mode_strict`` is enabled or not."""
""":returns: the minimal number of synchronous nodes based on whether strict mode is requested or not."""
return 1 if self.is_synchronous_mode_strict else 0
@property
def synchronous_node_count(self) -> int:
"""Currently configured value of ``synchronous_node_count`` from the global configuration.
Assume ``1`` if it is not set or invalid.
"""
""":returns: currently configured value from the global configuration or 1 if it is not set or invalid."""
return max(self.get_int('synchronous_node_count', 1), self.min_synchronous_nodes)
@property
def maximum_lag_on_failover(self) -> int:
"""Currently configured value of ``maximum_lag_on_failover`` from the global configuration.
Assume ``1048576`` if it is not set or invalid.
"""
""":returns: currently configured value from the global configuration or 1048576 if it is not set or invalid."""
return self.get_int('maximum_lag_on_failover', 1048576)
@property
def maximum_lag_on_syncnode(self) -> int:
"""Currently configured value of ``maximum_lag_on_syncnode`` from the global configuration.
Assume ``-1`` if it is not set or invalid.
"""
""":returns: currently configured value from the global configuration or -1 if it is not set or invalid."""
return self.get_int('maximum_lag_on_syncnode', -1)
@property
def primary_start_timeout(self) -> int:
"""Currently configured value of ``primary_start_timeout`` from the global configuration.
Assume ``300`` if it is not set or invalid.
.. note::
``master_start_timeout`` is still supported to keep backward compatibility.
"""
""":returns: currently configured value from the global configuration or 300 if it is not set or invalid."""
default = 300
return self.get_int('primary_start_timeout', default)\
if 'primary_start_timeout' in self.__config else self.get_int('master_start_timeout', default)
@property
def primary_stop_timeout(self) -> int:
"""Currently configured value of ``primary_stop_timeout`` from the global configuration.
Assume ``0`` if it is not set or invalid.
.. note::
``master_stop_timeout`` is still supported to keep backward compatibility.
"""
""":returns: currently configured value from the global configuration or 300 if it is not set or invalid."""
default = 0
return self.get_int('primary_stop_timeout', default)\
if 'primary_stop_timeout' in self.__config else self.get_int('master_stop_timeout', default)
def get_global_config(cluster: Optional[Cluster], default: Optional[Dict[str, Any]] = None) -> GlobalConfig:
def get_global_config(cluster: Union[Cluster, None], default: Optional[Dict[str, Any]] = None) -> GlobalConfig:
"""Instantiates :class:`GlobalConfig` based on the input.
:param cluster: the currently known cluster state from DCS.
:param default: default configuration, which will be used if there is no valid *cluster.config*.
:returns: :class:`GlobalConfig` object.
:param cluster: the currently known cluster state from DCS
:param default: default configuration, which will be used if there is no valid *cluster.config*
:returns: :class:`GlobalConfig` object
"""
# Try to protect from the case when DCS was wiped out
if cluster and cluster.config and cluster.config.modify_version:
@@ -203,29 +161,23 @@ def get_global_config(cluster: Optional[Cluster], default: Optional[Dict[str, An
class Config(object):
"""Handle Patroni configuration.
"""
This class is responsible for:
1) Building and giving access to ``effective_configuration`` from:
1) Building and giving access to `effective_configuration` from:
* `Config.__DEFAULT_CONFIG` -- some sane default values
* `dynamic_configuration` -- configuration stored in DCS
* `local_configuration` -- configuration from `config.yml` or environment
* ``Config.__DEFAULT_CONFIG`` -- some sane default values;
* ``dynamic_configuration`` -- configuration stored in DCS;
* ``local_configuration`` -- configuration from `config.yml` or environment.
2) Saving and loading ``dynamic_configuration`` into 'patroni.dynamic.json' file
2) Saving and loading `dynamic_configuration` into 'patroni.dynamic.json' file
located in local_configuration['postgresql']['data_dir'] directory.
This is necessary to be able to restore ``dynamic_configuration``
if DCS was accidentally wiped.
This is necessary to be able to restore `dynamic_configuration`
if DCS was accidentally wiped
3) Loading of configuration file in the old format and converting it into new format.
3) Loading of configuration file in the old format and converting it into new format
4) Mimicking some ``dict`` interfaces to make it possible
to work with it as with the old ``config`` object.
:cvar PATRONI_CONFIG_VARIABLE: name of the environment variable that can be used to load Patroni configuration from.
:cvar __CACHE_FILENAME: name of the file used to cache dynamic configuration under Postgres data directory.
:cvar __DEFAULT_CONFIG: default configuration values for some Patroni settings.
4) Mimicking some of the `dict` interfaces to make it possible
to work with it as with the old `config` object.
"""
PATRONI_CONFIG_VARIABLE = PATRONI_ENV_PREFIX + 'CONFIGURATION'
@@ -243,38 +195,21 @@ class Config(object):
'recovery_min_apply_delay': ''
},
'postgresql': {
'bin_dir': '',
'use_slots': True,
'parameters': CaseInsensitiveDict({p: v[0] for p, v in ConfigHandler.CMDLINE_OPTIONS.items()
if v[0] is not None and p not in ('wal_keep_segments', 'wal_keep_size')})
if p not in ('wal_keep_segments', 'wal_keep_size')})
}
}
def __init__(self, configfile: str,
validator: Optional[Callable[[Dict[str, Any]], List[str]]] = default_validator) -> None:
"""Create a new instance of :class:`Config` and validate the loaded configuration using *validator*.
.. note::
Patroni will read configuration from these locations in this order:
* file or directory path passed as command-line argument (*configfile*), if it exists and the file or
files found in the directory can be parsed (see :meth:`~Config._load_config_path`), otherwise
* YAML file passed via the environment variable (see :attr:`PATRONI_CONFIG_VARIABLE`), if the referenced
file exists and can be parsed, otherwise
* from configuration values defined as environment variables, see
:meth:`~Config._build_environment_configuration`.
:param configfile: path to Patroni configuration file.
:param validator: function used to validate Patroni configuration. It should receive a dictionary which
represents Patroni configuration, and return a list of zero or more error messages based on validation.
:raises:
:class:`ConfigParseError`: if any issue is reported by *validator*.
"""
self._modify_version = -1
self._dynamic_configuration = {}
self.__environment_configuration = self._build_environment_configuration()
# Patroni reads the configuration from the command-line argument if it exists, otherwise from the environment
self._config_file = configfile if configfile and os.path.exists(configfile) else None
if self._config_file:
self._local_configuration = self._load_config_file()
@@ -293,46 +228,19 @@ class Config(object):
if validator: # patronictl uses validator=None and we don't want to load anything from local cache in this case
self._load_cache()
self._cache_needs_saving = False
self._validate_failover_tags()
@property
def config_file(self) -> Optional[str]:
"""Path to Patroni configuration file, if any, else ``None``."""
def config_file(self) -> Union[str, None]:
return self._config_file
@property
def dynamic_configuration(self) -> Dict[str, Any]:
"""Deep copy of cached Patroni dynamic configuration."""
return deepcopy(self._dynamic_configuration)
@property
def local_configuration(self) -> Dict[str, Any]:
"""Deep copy of cached Patroni local configuration.
:returns: copy of :attr:`~Config._local_configuration`
"""
return deepcopy(dict(self._local_configuration))
@classmethod
def get_default_config(cls) -> Dict[str, Any]:
"""Deep copy default configuration.
:returns: copy of :attr:`~Config.__DEFAULT_CONFIG`
"""
return deepcopy(cls.__DEFAULT_CONFIG)
def _load_config_path(self, path: str) -> Dict[str, Any]:
"""Load Patroni configuration file(s) from *path*.
If *path* is a file, load the yml file pointed to by *path*.
If *path* is a directory, load all yml files in that directory in alphabetical order.
:param path: path to either an YAML configuration file, or to a folder containing YAML configuration files.
:returns: configuration after reading the configuration file(s) from *path*.
:raises:
:class:`ConfigParseError`: if *path* is invalid.
"""
If path is a file, loads the yml file pointed to by path.
If path is a directory, loads all yml files in that directory in alphabetical order
"""
if os.path.isfile(path):
files = [path]
@@ -351,18 +259,14 @@ class Config(object):
return overall_config
def _load_config_file(self) -> Dict[str, Any]:
"""Load configuration file(s) from filesystem and apply values which were set via environment variables.
:returns: final configuration after merging configuration file(s) and environment variables.
"""
"""Loads config.yaml from filesystem and applies some values which were set via ENV"""
if TYPE_CHECKING: # pragma: no cover
assert self.config_file is not None
config = self._load_config_path(self.config_file)
assert self._config_file is not None
config = self._load_config_path(self._config_file)
patch_config(config, self.__environment_configuration)
return config
def _load_cache(self) -> None:
"""Load dynamic configuration from ``patroni.dynamic.json``."""
if os.path.isfile(self._cache_file):
try:
with open(self._cache_file) as f:
@@ -371,12 +275,6 @@ class Config(object):
logger.exception('Exception when loading file: %s', self._cache_file)
def save_cache(self) -> None:
"""Save dynamic configuration to ``patroni.dynamic.json`` under Postgres data directory.
.. note::
``patroni.dynamic.jsonXXXXXX`` is created as a temporary file and than renamed to ``patroni.dynamic.json``,
where ``XXXXXX`` is a random suffix.
"""
if self._cache_needs_saving:
tmpfile = fd = None
try:
@@ -401,84 +299,16 @@ class Config(object):
except Exception:
logger.error('Can not remove temporary file %s', tmpfile)
def __get_and_maybe_adjust_int_value(self, config: Dict[str, Any], param: str, min_value: int) -> int:
"""Get, validate and maybe adjust a *param* integer value from the *config* :class:`dict`.
.. note:
If the value is smaller than provided *min_value* we update the *config*.
This method may raise an exception if value isn't :class:`int` or cannot be casted to :class:`int`.
:param config: :class:`dict` object with new global configuration.
:param param: name of the configuration parameter we want to read/validate/adjust.
:param min_value: the minimum possible value that a given *param* could have.
:returns: an integer value which corresponds to a provided *param*.
"""
value = int(config.get(param, self.__DEFAULT_CONFIG[param]))
if value < min_value:
logger.warning("%s=%d can't be smaller than %d, adjusting...", param, value, min_value)
value = config[param] = min_value
return value
def _validate_and_adjust_timeouts(self, config: Dict[str, Any]) -> None:
"""Validate and adjust ``loop_wait``, ``retry_timeout``, and ``ttl`` values if necessary.
Minimum values:
* ``loop_wait``: 1 second;
* ``retry_timeout``: 3 seconds.
* ``ttl``: 20 seconds;
Maximum values:
In case if values don't fulfill the following rule, ``retry_timeout`` and ``loop_wait``
are reduced so that the rule is fulfilled:
.. code-block:: python
loop_wait + 2 * retry_timeout <= ttl
.. note:
We prefer to reduce ``loop_wait`` and will reduce ``retry_timeout`` only if ``loop_wait``
is already set to a minimal possible value.
:param config: :class:`dict` object with new global configuration.
"""
min_loop_wait = 1
loop_wait = self. __get_and_maybe_adjust_int_value(config, 'loop_wait', min_loop_wait)
retry_timeout = self. __get_and_maybe_adjust_int_value(config, 'retry_timeout', 3)
ttl = self. __get_and_maybe_adjust_int_value(config, 'ttl', 20)
if min_loop_wait + 2 * retry_timeout > ttl:
config['loop_wait'] = min_loop_wait
config['retry_timeout'] = (ttl - min_loop_wait) // 2
logger.warning('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d. '
'Adjusting loop_wait from %d to %d and retry_timeout from %d to %d',
ttl, loop_wait, min_loop_wait, retry_timeout, config['retry_timeout'])
elif loop_wait + 2 * retry_timeout > ttl:
config['loop_wait'] = ttl - 2 * retry_timeout
logger.warning('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d and retry_timeout=%d.'
' Adjusting loop_wait from %d to %d', ttl, retry_timeout, loop_wait, config['loop_wait'])
# configuration could be either ClusterConfig or dict
def set_dynamic_configuration(self, configuration: Union[ClusterConfig, Dict[str, Any]]) -> bool:
"""Set dynamic configuration values with given *configuration*.
:param configuration: new dynamic configuration values. Supports :class:`dict` for backward compatibility.
:returns: ``True`` if changes have been detected between current dynamic configuration and the new dynamic
*configuration*, ``False`` otherwise.
"""
if isinstance(configuration, ClusterConfig):
if self._modify_version == configuration.modify_version:
return False # If the version didn't change there is nothing to do
return False # If the version didn't changed there is nothing to do
self._modify_version = configuration.modify_version
configuration = configuration.data
if not deep_compare(self._dynamic_configuration, configuration):
try:
self._validate_and_adjust_timeouts(configuration)
self.__effective_configuration = self._build_effective_configuration(configuration,
self._local_configuration)
self._dynamic_configuration = configuration
@@ -489,14 +319,6 @@ class Config(object):
return False
def reload_local_configuration(self) -> Optional[bool]:
"""Reload configuration values from the configuration file(s).
.. note::
Designed to be used when user applies changes to configuration file(s), so Patroni can use the new values
with a reload instead of a restart.
:returns: ``True`` if changes have been detected between current local configuration
"""
if self.config_file:
try:
configuration = self._load_config_file()
@@ -512,38 +334,6 @@ class Config(object):
@staticmethod
def _process_postgresql_parameters(parameters: Dict[str, Any], is_local: bool = False) -> Dict[str, Any]:
"""Process Postgres *parameters*.
.. note::
If *is_local* configuration discard any setting from *parameters* that is listed under
:attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` as those are supposed to be set only
through dynamic configuration.
When setting parameters from :attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` through
dynamic configuration their value will be validated as per the validator defined in that very same
attribute entry. If the given value cannot be validated, a warning will be logged and the default value of
the GUC will be used instead.
Some parameters from :attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` cannot be set even if
not *is_local* configuration:
* ``listen_addresses``: inferred from ``postgresql.listen`` local configuration or from
``PATRONI_POSTGRESQL_LISTEN`` environment variable;
* ``port``: inferred from ``postgresql.listen`` local configuration or from
``PATRONI_POSTGRESQL_LISTEN`` environment variable;
* ``cluster_name``: set through ``scope`` local configuration or through ``PATRONI_SCOPE`` environment
variable;
* ``hot_standby``: always enabled;
* ``wal_log_hints``: always enabled.
:param parameters: Postgres parameters to be processed. Should be the parsed YAML value of
``postgresql.parameters`` configuration, either from local or from dynamic configuration.
:param is_local: should be ``True`` if *parameters* refers to local configuration, or ``False`` if *parameters*
refers to dynamic configuration.
:returns: new value for ``postgresql.parameters`` after processing and validating *parameters*.
"""
pg_params: Dict[str, Any] = {}
for name, value in (parameters or {}).items():
@@ -561,33 +351,7 @@ class Config(object):
return pg_params
def _safe_copy_dynamic_configuration(self, dynamic_configuration: Dict[str, Any]) -> Dict[str, Any]:
"""Create a copy of *dynamic_configuration*.
Merge *dynamic_configuration* with :attr:`__DEFAULT_CONFIG` (*dynamic_configuration* takes precedence), and
process ``postgresql.parameters`` from *dynamic_configuration* through :func:`_process_postgresql_parameters`,
if present.
.. note::
The following settings are not allowed in ``postgresql`` section as they are intended to be local
configuration, and are removed if present:
* ``connect_address``;
* ``proxy_address``;
* ``listen``;
* ``config_dir``;
* ``data_dir``;
* ``pgpass``;
* ``authentication``;
Besides that any setting present in *dynamic_configuration* but absent from :attr:`__DEFAULT_CONFIG` is
discarded.
:param dynamic_configuration: Patroni dynamic configuration.
:returns: copy of *dynamic_configuration*, merged with default dynamic configuration and with some sanity checks
performed over it.
"""
config = self.get_default_config()
config = deepcopy(self.__DEFAULT_CONFIG)
for name, value in dynamic_configuration.items():
if name == 'postgresql':
@@ -607,25 +371,9 @@ class Config(object):
@staticmethod
def _build_environment_configuration() -> Dict[str, Any]:
"""Get local configuration settings that were specified through environment variables.
:returns: dictionary containing the found environment variables and their values, respecting the expected
structure of Patroni configuration.
"""
ret: Dict[str, Any] = defaultdict(dict)
def _popenv(name: str) -> Optional[str]:
"""Get value of environment variable *name*.
.. note::
*name* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` when searching in the environment.
Also, the corresponding environment variable is removed from the environment upon reading its value.
:param name: name of the environment variable.
:returns: value of *name*, if present in the environment, otherwise ``None``.
"""
def _popenv(name: str) -> Union[str, None]:
return os.environ.pop(PATRONI_ENV_PREFIX + name.upper(), None)
for param in ('name', 'namespace', 'scope'):
@@ -634,23 +382,6 @@ class Config(object):
ret[param] = value
def _fix_log_env(name: str, oldname: str) -> None:
"""Normalize a log related environment variable.
.. note::
Patroni used to support different names for log related environment variables in the past. As the
environment variables were renamed, this function takes care of mapping and normalizing the environment.
*name* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` and ``LOG`` when searching in the
environment.
*oldname* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` when searching in the environment.
If both *name* and *oldname* are set in the environment, *name* takes precedence.
:param name: new name of a log related environment variable.
:param oldname: original name of a log related environment variable.
:type oldname: str
"""
value = _popenv(oldname)
name = PATRONI_ENV_PREFIX + 'LOG_' + name.upper()
if value and name not in os.environ:
@@ -660,15 +391,6 @@ class Config(object):
_fix_log_env(name, oldname)
def _set_section_values(section: str, params: List[str]) -> None:
"""Get value of *params* environment variables that are related with *section*.
.. note::
The values are retrieved from the environment and updated directly into the returning dictionary of
:func:`_build_environment_configuration`.
:param section: configuration section the *params* belong to.
:param params: name of the Patroni settings.
"""
for param in params:
value = _popenv(section + '_' + param)
if value:
@@ -690,7 +412,6 @@ class Config(object):
if value:
ret['postgresql'].setdefault('bin_name', {})[binary] = value
# parse all values retrieved from the environment as Python objects, according to the expected type
for first, second in (('restapi', 'allowlist_include_members'), ('ctl', 'insecure')):
value = ret.get(first, {}).pop(second, None)
if value:
@@ -707,13 +428,7 @@ class Config(object):
if value is not None:
ret[first][second] = value
def _parse_list(value: str) -> Optional[List[str]]:
"""Parse an YAML list *value* as a :class:`list`.
:param value: YAML list as a string.
:returns: *value* as :class:`list`.
"""
def _parse_list(value: str) -> Union[List[str], None]:
if not (value.strip().startswith('-') or '[' in value):
value = '[{0}]'.format(value)
try:
@@ -729,13 +444,7 @@ class Config(object):
if value:
ret[first][second] = value
def _parse_dict(value: str) -> Optional[Dict[str, Any]]:
"""Parse an YAML dictionary *value* as a :class:`dict`.
:param value: YAML dictionary as a string.
:returns: *value* as :class:`dict`.
"""
def _parse_dict(value: str) -> Union[Dict[str, Any], None]:
if not value.strip().startswith('{'):
value = '{{{0}}}'.format(value)
try:
@@ -752,16 +461,9 @@ class Config(object):
if value:
ret[first][second] = value
def _get_auth(name: str, params: Collection[str] = _AUTH_ALLOWED_PARAMETERS[:2]) -> Dict[str, str]:
"""Get authorization related environment variables *params* from section *name*.
:param name: name of a configuration section that may contain authorization *params*.
:param params: the authorization settings that may be set under section *name*.
:returns: dictionary containing environment values for authorization *params* of section *name*.
"""
def _get_auth(name: str, params: Optional[Collection[str]] = None) -> Dict[str, str]:
ret: Dict[str, str] = {}
for param in params:
for param in params or _AUTH_ALLOWED_PARAMETERS[:2]:
value = _popenv(name + '_' + param)
if value:
ret[param] = value
@@ -784,7 +486,7 @@ class Config(object):
for param in list(os.environ.keys()):
if param.startswith(PATRONI_ENV_PREFIX):
# PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
name, suffix = (param[len(PATRONI_ENV_PREFIX):].split('_', 1) + [''])[:2]
name, suffix = (param[8:].split('_', 1) + [''])[:2]
if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'SRV_SUFFIX', 'URL', 'PROXY',
'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY',
'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'SERVICE_CHECK_TLS_SERVER_NAME',
@@ -815,14 +517,14 @@ class Config(object):
users = {}
for param in list(os.environ.keys()):
if param.startswith(PATRONI_ENV_PREFIX):
name, suffix = (param[len(PATRONI_ENV_PREFIX):].rsplit('_', 1) + [''])[:2]
name, suffix = (param[8:].rsplit('_', 1) + [''])[:2]
# PATRONI_<username>_PASSWORD=<password>, PATRONI_<username>_OPTIONS=<option1,option2,...>
# CREATE USER "<username>" WITH <OPTIONS> PASSWORD '<password>'
if name and suffix == 'PASSWORD':
password = os.environ.pop(param)
if password:
users[name] = {'password': password}
options = os.environ.pop(param[:-9] + '_OPTIONS', None) # replace "_PASSWORD" with "_OPTIONS"
options = os.environ.pop(param[:-9] + '_OPTIONS', None)
options = options and _parse_list(options)
if options:
users[name]['options'] = options
@@ -833,16 +535,6 @@ class Config(object):
def _build_effective_configuration(self, dynamic_configuration: Dict[str, Any],
local_configuration: Dict[str, Union[Dict[str, Any], Any]]) -> Dict[str, Any]:
"""Build effective configuration by merging *dynamic_configuration* and *local_configuration*.
.. note::
*local_configuration* takes precedence over *dynamic_configuration* if a setting is defined in both.
:param dynamic_configuration: Patroni dynamic configuration.
:param local_configuration: Patroni local configuration.
:returns: _description_
"""
config = self._safe_copy_dynamic_configuration(dynamic_configuration)
for name, value in local_configuration.items():
if name == 'citus': # remove invalid citus configuration
@@ -906,78 +598,23 @@ class Config(object):
return config
def get(self, key: str, default: Optional[Any] = None) -> Any:
"""Get effective value of ``key`` setting from Patroni configuration root.
Designed to work the same way as :func:`dict.get`.
:param key: name of the setting.
:param default: default value if *key* is not present in the effective configuration.
:returns: value of *key*, if present in the effective configuration, otherwise *default*.
"""
return self.__effective_configuration.get(key, default)
def __contains__(self, key: str) -> bool:
"""Check if setting *key* is present in the effective configuration.
Designed to work the same way as :func:`dict.__contains__`.
:param key: name of the setting to be checked.
:returns: ``True`` if setting *key* exists in effective configuration, else ``False``.
"""
return key in self.__effective_configuration
def __getitem__(self, key: str) -> Any:
"""Get value of setting *key* from effective configuration.
Designed to work the same way as :func:`dict.__getitem__`.
:param key: name of the setting.
:returns: value of setting *key*.
:raises:
:class:`KeyError`: if *key* is not present in effective configuration.
"""
return self.__effective_configuration[key]
def copy(self) -> Dict[str, Any]:
"""Get a deep copy of effective Patroni configuration.
:returns: a deep copy of the Patroni configuration.
"""
return deepcopy(self.__effective_configuration)
def get_global_config(self, cluster: Optional[Cluster]) -> GlobalConfig:
def get_global_config(self, cluster: Union[Cluster, None]) -> GlobalConfig:
"""Instantiate :class:`GlobalConfig` based on input.
Use the configuration from provided *cluster* (the most up-to-date) or from the
local cache if *cluster.config* is not initialized or doesn't have a valid config.
:param cluster: the currently known cluster state from DCS.
:returns: :class:`GlobalConfig` object.
:param cluster: the currently known cluster state from DCS
:returns: :class:`GlobalConfig` object
"""
return get_global_config(cluster, self._dynamic_configuration)
def _validate_failover_tags(self) -> None:
"""Check ``nofailover``/``failover_priority`` config and warn user if it's contradictory.
.. note::
To preserve sanity (and backwards compatibility) the ``nofailover`` tag will still exist. A contradictory
configuration is one where ``nofailover`` is ``True`` but ``failover_priority > 0``, or where
``nofailover`` is ``False``, but ``failover_priority <= 0``. Essentially, ``nofailover`` and
``failover_priority`` are communicating different things.
This checks for this edge case (which is a misconfiguration on the part of the user) and warns them.
The behaviour is as if ``failover_priority`` were not provided (i.e ``nofailover`` is the
bedrock source of truth)
"""
tags = self.get('tags', {})
nofailover_tag = tags.get('nofailover')
failover_priority_tag = parse_int(tags.get('failover_priority'))
if failover_priority_tag is not None \
and (nofailover_tag is True and failover_priority_tag > 0
or nofailover_tag is False and failover_priority_tag <= 0):
logger.warning('Conflicting configuration between nofailover: %s and failover_priority: %s. '
'Defaulting to nofailover: %s', nofailover_tag, failover_priority_tag, nofailover_tag)
-510
View File
@@ -1,510 +0,0 @@
"""patroni ``--generate-config`` machinery."""
import abc
import logging
import os
import psutil
import socket
import sys
import yaml
from getpass import getuser, getpass
from contextlib import contextmanager
from typing import Any, Dict, Iterator, List, Optional, TextIO, Tuple, TYPE_CHECKING, Union
if TYPE_CHECKING: # pragma: no cover
from psycopg import Cursor
from psycopg2 import cursor
from . import psycopg
from .config import Config
from .exceptions import PatroniException
from .log import PatroniLogger
from .postgresql.config import ConfigHandler, parse_dsn
from .postgresql.misc import postgres_major_version_to_int
from .utils import get_major_version, parse_bool, patch_config, read_stripped
# Mapping between the libpq connection parameters and the environment variables.
# This dict should be kept in sync with `patroni.utils._AUTH_ALLOWED_PARAMETERS`
# (we use "username" in the Patroni config for some reason, other parameter names are the same).
_AUTH_ALLOWED_PARAMETERS_MAPPING = {
'user': 'PGUSER',
'password': 'PGPASSWORD',
'sslmode': 'PGSSLMODE',
'sslcert': 'PGSSLCERT',
'sslkey': 'PGSSLKEY',
'sslpassword': '',
'sslrootcert': 'PGSSLROOTCERT',
'sslcrl': 'PGSSLCRL',
'sslcrldir': 'PGSSLCRLDIR',
'gssencmode': 'PGGSSENCMODE',
'channel_binding': 'PGCHANNELBINDING'
}
NO_VALUE_MSG = '#FIXME'
def get_address() -> Tuple[str, str]:
"""Try to get hostname and the ip address for it returned by :func:`~socket.gethostname`.
.. note::
Can also return local ip.
:returns: tuple consisting of the hostname returned by :func:`~socket.gethostname`
and the first element in the sorted list of the addresses returned by :func:`~socket.getaddrinfo`.
Sorting guarantees it will prefer IPv4.
If an exception occured, hostname and ip values are equal to :data:`~patroni.config_generator.NO_VALUE_MSG`.
"""
hostname = None
try:
hostname = socket.gethostname()
return hostname, sorted(socket.getaddrinfo(hostname, 0, socket.AF_UNSPEC, socket.SOCK_STREAM, 0),
key=lambda x: x[0])[0][4][0]
except Exception as err:
logging.warning('Failed to obtain address: %r', err)
return NO_VALUE_MSG, NO_VALUE_MSG
class AbstractConfigGenerator(abc.ABC):
"""Object representing the generated Patroni config.
:ivar output_file: full path to the output file to be used.
:ivar pg_major: integer representation of the major PostgreSQL version.
:ivar config: dictionary used for the generated configuration storage.
"""
_HOSTNAME, _IP = get_address()
def __init__(self, output_file: Optional[str]) -> None:
"""Set up the output file (if passed), helper vars and the minimal config structure.
:param output_file: full path to the output file to be used.
"""
self.output_file = output_file
self.pg_major = 0
self.config = self.get_template_config()
self.generate()
@classmethod
def get_template_config(cls) -> Dict[str, Any]:
"""Generate a template config for further extension (e.g. in the inherited classes).
:returns: dictionary with the values gathered from Patroni env, hopefully defined hostname and ip address
(otherwise set to :data:`~patroni.config_generator.NO_VALUE_MSG`), and some sane defaults.
"""
template_config: Dict[str, Any] = {
'scope': NO_VALUE_MSG,
'name': cls._HOSTNAME,
'restapi': {
'connect_address': cls._IP + ':8008',
'listen': cls._IP + ':8008'
},
'log': {
'level': PatroniLogger.DEFAULT_LEVEL,
'traceback_level': PatroniLogger.DEFAULT_TRACEBACK_LEVEL,
'format': PatroniLogger.DEFAULT_FORMAT,
'max_queue_size': PatroniLogger.DEFAULT_MAX_QUEUE_SIZE
},
'postgresql': {
'data_dir': NO_VALUE_MSG,
'connect_address': cls._IP + ':5432',
'listen': cls._IP + ':5432',
'bin_dir': '',
'authentication': {
'superuser': {
'username': 'postgres',
'password': NO_VALUE_MSG
},
'replication': {
'username': 'replicator',
'password': NO_VALUE_MSG
}
}
},
'tags': {
'failover_priority': 1,
'noloadbalance': False,
'clonefrom': True,
'nosync': False,
}
}
dynamic_config = Config.get_default_config()
# to properly dump CaseInsensitiveDict as YAML later
dynamic_config['postgresql']['parameters'] = dict(dynamic_config['postgresql']['parameters'])
config = Config('', None).local_configuration # Get values from env
config.setdefault('bootstrap', {})['dcs'] = dynamic_config
config.setdefault('postgresql', {})
del config['bootstrap']['dcs']['standby_cluster']
patch_config(template_config, config)
return template_config
@abc.abstractmethod
def generate(self) -> None:
"""Generate config and store in :attr:`~AbstractConfigGenerator.config`."""
@staticmethod
def _format_block(block: Any, line_prefix: str = '') -> str:
"""Format a single YAML block.
.. note::
Optionally the formatted block could be indented with the *line_prefix*
:param block: the object that should be formatted to YAML.
:param line_prefix: is used for indentation.
:returns: a formatted and indented *block*.
"""
return line_prefix + yaml.safe_dump(block, default_flow_style=False, line_break='\n',
allow_unicode=True, indent=2).strip().replace('\n', '\n' + line_prefix)
def _format_config_section(self, section_name: str) -> Iterator[str]:
"""Format and yield as single section of the current :attr:`~AbstractConfigGenerator.config`.
.. note::
If the section is a :class:`dict` object we put an empty line before it.
:param section_name: a section name in the :attr:`~AbstractConfigGenerator.config`.
:yields: a formatted section in case if it exists in the :attr:`~AbstractConfigGenerator.config`.
"""
if section_name in self.config:
if isinstance(self.config[section_name], dict):
yield ''
yield self._format_block({section_name: self.config[section_name]})
def _format_config(self) -> Iterator[str]:
"""Format current :attr:`~AbstractConfigGenerator.config` and enrich it with some comments.
:yields: formatted lines or blocks that represent a text output of the YAML document.
"""
for name in ('scope', 'namespace', 'name', 'log', 'restapi', 'ctl' 'citus',
'consul', 'etcd', 'etcd3', 'exhibitor', 'kubernetes', 'raft', 'zookeeper'):
yield from self._format_config_section(name)
if 'bootstrap' in self.config:
yield '\n# The bootstrap configuration. Works only when the cluster is not yet initialized.'
yield '# If the cluster is already initialized, all changes in the `bootstrap` section are ignored!'
yield 'bootstrap:'
if 'dcs' in self.config['bootstrap']:
yield ' # This section will be written into <dcs>:/<namespace>/<scope>/config after initializing'
yield ' # new cluster and all other cluster members will use it as a `global configuration`.'
yield ' # WARNING! If you want to change any of the parameters that were set up'
yield ' # via `bootstrap.dcs` section, please use `patronictl edit-config`!'
yield ' dcs:'
for name in ('loop_wait', 'retry_timeout', 'ttl'):
if name in self.config['bootstrap']['dcs']:
yield self._format_block({name: self.config['bootstrap']['dcs'].pop(name)}, ' ')
for name, value in self.config['bootstrap']['dcs'].items():
yield self._format_block({name: value}, ' ')
for name in ('postgresql', 'watchdog', 'tags'):
yield from self._format_config_section(name)
def _write_config_to_fd(self, fd: TextIO) -> None:
"""Format and write current :attr:`~AbstractConfigGenerator.config` to provided file descriptor.
:param fd: where to write the config file. Could be ``sys.stdout`` or the real file.
"""
fd.write('\n'.join(self._format_config()))
def write_config(self) -> None:
"""Write current :attr:`~AbstractConfigGenerator.config` to the output file if provided, to stdout otherwise."""
if self.output_file:
dir_path = os.path.dirname(self.output_file)
if dir_path and not os.path.isdir(dir_path):
os.makedirs(dir_path)
with open(self.output_file, 'w', encoding='UTF-8') as output_file:
self._write_config_to_fd(output_file)
else:
self._write_config_to_fd(sys.stdout)
class SampleConfigGenerator(AbstractConfigGenerator):
"""Object representing the generated sample Patroni config.
Sane defults are used based on the gathered PG version.
"""
@property
def get_auth_method(self) -> str:
"""Return the preferred authentication method for a specific PG version if provided or the default ``md5``.
:returns: :class:`str` value for the preferred authentication method.
"""
return 'scram-sha-256' if self.pg_major and self.pg_major >= 100000 else 'md5'
def _get_int_major_version(self) -> int:
"""Get major PostgreSQL version from the binary as an integer.
:returns: an integer PostgreSQL major version representation gathered from the PostgreSQL binary.
See :func:`~patroni.postgresql.misc.postgres_major_version_to_int` and
:func:`~patroni.utils.get_major_version`.
"""
postgres_bin = ((self.config.get('postgresql') or {}).get('bin_name') or {}).get('postgres', 'postgres')
return postgres_major_version_to_int(get_major_version(self.config['postgresql'].get('bin_dir'), postgres_bin))
def generate(self) -> None:
"""Generate sample config using some sane defaults and update :attr:`~AbstractConfigGenerator.config`."""
self.pg_major = self._get_int_major_version()
self.config['postgresql']['parameters'] = {'password_encryption': self.get_auth_method}
username = self.config["postgresql"]["authentication"]["replication"]["username"]
self.config['postgresql']['pg_hba'] = [
f'host all all all {self.get_auth_method}',
f'host replication {username} all {self.get_auth_method}'
]
# add version-specific configuration
wal_keep_param = 'wal_keep_segments' if self.pg_major < 130000 else 'wal_keep_size'
self.config['bootstrap']['dcs']['postgresql']['parameters'][wal_keep_param] = \
ConfigHandler.CMDLINE_OPTIONS[wal_keep_param][0]
wal_level = 'hot_standby' if self.pg_major < 90600 else 'replica'
self.config['bootstrap']['dcs']['postgresql']['parameters']['wal_level'] = wal_level
self.config['bootstrap']['dcs']['postgresql']['use_pg_rewind'] = True
if self.pg_major >= 110000:
self.config['postgresql']['authentication'].setdefault(
'rewind', {'username': 'rewind_user'}).setdefault('password', NO_VALUE_MSG)
class RunningClusterConfigGenerator(AbstractConfigGenerator):
"""Object representing the Patroni config generated using information gathered from the running instance.
:ivar dsn: DSN string for the local instance to get GUC values from (if provided).
:ivar parsed_dsn: DSN string parsed into a dictionary (see :func:`~patroni.postgresql.config.parse_dsn`).
"""
def __init__(self, output_file: Optional[str] = None, dsn: Optional[str] = None) -> None:
"""Additionally store the passed dsn (if any) in both original and parsed version and run config generation.
:param output_file: full path to the output file to be used.
:param dsn: DSN string for the local instance to get GUC values from.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if DSN parsing failed.
"""
self.dsn = dsn
self.parsed_dsn = {}
super().__init__(output_file)
@property
def _get_hba_conn_types(self) -> Tuple[str, ...]:
"""Return the connection types allowed.
If :attr:`~RunningClusterConfigGenerator.pg_major` is defined, adds additional parameters
for PostgreSQL version >=16.
:returns: tuple of the connection methods allowed.
"""
allowed_types = ('local', 'host', 'hostssl', 'hostnossl', 'hostgssenc', 'hostnogssenc')
if self.pg_major and self.pg_major >= 160000:
allowed_types += ('include', 'include_if_exists', 'include_dir')
return allowed_types
@property
def _required_pg_params(self) -> List[str]:
"""PG configuration prameters that have to be always present in the generated config.
:returns: list of the parameter names.
"""
return ['hba_file', 'ident_file', 'config_file', 'data_directory'] + \
list(ConfigHandler.CMDLINE_OPTIONS.keys())
def _get_bin_dir_from_running_instance(self) -> str:
"""Define the directory postgres binaries reside using postmaster's pid executable.
:returns: path to the PostgreSQL binaries directory.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if:
* pid could not be obtained from the ``postmaster.pid`` file; or
* :exc:`OSError` occured during ``postmaster.pid`` file handling; or
* the obtained postmaster pid doesn't exist.
"""
postmaster_pid = None
data_dir = self.config['postgresql']['data_dir']
try:
with open(f"{data_dir}/postmaster.pid", 'r') as pid_file:
postmaster_pid = pid_file.readline()
if not postmaster_pid:
raise PatroniException('Failed to obtain postmaster pid from postmaster.pid file')
postmaster_pid = int(postmaster_pid.strip())
except OSError as err:
raise PatroniException(f'Error while reading postmaster.pid file: {err}')
try:
return os.path.dirname(psutil.Process(postmaster_pid).exe())
except psutil.NoSuchProcess:
raise PatroniException("Obtained postmaster pid doesn't exist.")
@contextmanager
def _get_connection_cursor(self) -> Iterator[Union['cursor', 'Cursor[Any]']]:
"""Get cursor for the PG connection established based on the stored information.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if :exc:`psycopg.Error` occured.
"""
try:
conn = psycopg.connect(dsn=self.dsn,
password=self.config['postgresql']['authentication']['superuser']['password'])
with conn.cursor() as cur:
yield cur
conn.close()
except psycopg.Error as e:
raise PatroniException(f'Failed to establish PostgreSQL connection: {e}')
def _set_pg_params(self, cur: Union['cursor', 'Cursor[Any]']) -> None:
"""Extend :attr:`~RunningClusterConfigGenerator.config` with the actual PG GUCs values.
THe following GUC values are set:
* Non-internal having configuration file, postmaster command line or environment variable
as a source.
* List of the always required parameters (see :meth:`~RunningClusterConfigGenerator._required_pg_params`).
:param cur: connection cursor to use.
"""
cur.execute("SELECT name, pg_catalog.current_setting(name) FROM pg_catalog.pg_settings "
"WHERE context <> 'internal' "
"AND source IN ('configuration file', 'command line', 'environment variable') "
"AND category <> 'Write-Ahead Log / Recovery Target' "
"AND setting <> '(disabled)' "
"OR name = ANY(%s)", (self._required_pg_params,))
helper_dict = dict.fromkeys(['port', 'listen_addresses'])
self.config['postgresql'].setdefault('parameters', {})
for param, value in cur.fetchall():
if param == 'data_directory':
self.config['postgresql']['data_dir'] = value
elif param == 'cluster_name' and value:
self.config['scope'] = value
elif param in ('archive_command', 'restore_command',
'archive_cleanup_command', 'recovery_end_command',
'ssl_passphrase_command', 'hba_file',
'ident_file', 'config_file'):
# write commands to the local config due to security implications
# write hba/ident/config_file to local config to ensure they are not removed later
self.config['postgresql']['parameters'][param] = value
elif param in helper_dict:
helper_dict[param] = value
else:
self.config['bootstrap']['dcs']['postgresql']['parameters'][param] = value
connect_port = self.parsed_dsn.get('port', os.getenv('PGPORT', helper_dict['port']))
self.config['postgresql']['connect_address'] = f'{self._IP}:{connect_port}'
self.config['postgresql']['listen'] = f'{helper_dict["listen_addresses"]}:{helper_dict["port"]}'
def _set_su_params(self) -> None:
"""Extend :attr:`~RunningClusterConfigGenerator.config` with the superuser auth information.
Information set is based on the options used for connection.
"""
su_params: Dict[str, str] = {}
for conn_param, env_var in _AUTH_ALLOWED_PARAMETERS_MAPPING.items():
val = self.parsed_dsn.get(conn_param, os.getenv(env_var))
if val:
su_params[conn_param] = val
patroni_env_su_username = ((self.config.get('authentication') or {}).get('superuser') or {}).get('username')
patroni_env_su_pwd = ((self.config.get('authentication') or {}).get('superuser') or {}).get('password')
# because we use "username" in the config for some reason
su_params['username'] = su_params.pop('user', patroni_env_su_username) or getuser()
su_params['password'] = su_params.get('password', patroni_env_su_pwd) or \
getpass('Please enter the user password:')
self.config['postgresql']['authentication'] = {
'superuser': su_params,
'replication': {'username': NO_VALUE_MSG, 'password': NO_VALUE_MSG}
}
def _set_conf_files(self) -> None:
"""Extend :attr:`~RunningClusterConfigGenerator.config` with ``pg_hba.conf`` and ``pg_ident.conf`` content.
.. note::
This function only defines ``postgresql.pg_hba`` and ``postgresql.pg_ident`` when
``hba_file`` and ``ident_file`` are set to the defaults. It may happen these files
are located outside of ``PGDATA`` and Patroni doesn't have write permissions for them.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if :exc:`OSError` occured during the conf files handling.
"""
default_hba_path = os.path.join(self.config['postgresql']['data_dir'], 'pg_hba.conf')
if self.config['postgresql']['parameters']['hba_file'] == default_hba_path:
try:
self.config['postgresql']['pg_hba'] = list(
filter(lambda i: i and i.split()[0] in self._get_hba_conn_types, read_stripped(default_hba_path)))
except OSError as err:
raise PatroniException(f'Failed to read pg_hba.conf: {err}')
default_ident_path = os.path.join(self.config['postgresql']['data_dir'], 'pg_ident.conf')
if self.config['postgresql']['parameters']['ident_file'] == default_ident_path:
try:
self.config['postgresql']['pg_ident'] = [i for i in read_stripped(default_ident_path)
if i and not i.startswith('#')]
except OSError as err:
raise PatroniException(f'Failed to read pg_ident.conf: {err}')
if not self.config['postgresql']['pg_ident']:
del self.config['postgresql']['pg_ident']
def _enrich_config_from_running_instance(self) -> None:
"""Extend :attr:`~RunningClusterConfigGenerator.config` with the values gathered from the running instance.
Retrieve the following information from the running PostgreSQL instance:
* superuser auth parameters (see :meth:`~RunningClusterConfigGenerator._set_su_params`);
* some GUC values (see :meth:`~RunningClusterConfigGenerator._set_pg_params`);
* ``postgresql.connect_address``, ``postgresql.listen``;
* ``postgresql.pg_hba`` and ``postgresql.pg_ident`` (see :meth:`~RunningClusterConfigGenerator._set_conf_files`)
And redefine ``scope`` with the ``cluster_name`` GUC value if set.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if the provided user doesn't have superuser privileges.
"""
self._set_su_params()
with self._get_connection_cursor() as cur:
self.pg_major = getattr(cur.connection, 'server_version', 0)
if not parse_bool(cur.connection.info.parameter_status('is_superuser')):
raise PatroniException('The provided user does not have superuser privilege')
self._set_pg_params(cur)
self._set_conf_files()
def generate(self) -> None:
"""Generate config using the info gathered from the specified running PG instance.
Result is written to :attr:`~RunningClusterConfigGenerator.config`.
"""
if self.dsn:
self.parsed_dsn = parse_dsn(self.dsn) or {}
if not self.parsed_dsn:
raise PatroniException('Failed to parse DSN string')
self._enrich_config_from_running_instance()
self.config['postgresql']['bin_dir'] = self._get_bin_dir_from_running_instance()
def generate_config(output_file: str, sample: bool, dsn: Optional[str]) -> None:
"""Generate Patroni configuration file.
:param output_file: Full path to the configuration file to be used. If not provided, result is sent to ``stdout``.
:param sample: Optional flag. If set, no source instance will be used - generate config with some sane defaults.
:param dsn: Optional DSN string for the local instance to get GUC values from.
"""
try:
if sample:
config_generator = SampleConfigGenerator(output_file)
else:
config_generator = RunningClusterConfigGenerator(output_file, dsn)
config_generator.write_config()
except PatroniException as e:
sys.exit(str(e))
except Exception as e:
sys.exit(f'Unexpected exception: {e}')
+33 -62
View File
@@ -46,7 +46,6 @@ try:
except ImportError: # pragma: no cover
from cdiff import markup_to_pager, PatchStream # pyright: ignore [reportMissingModuleSource]
from .config import Config, get_global_config
from .dcs import get_dcs as _get_dcs, AbstractDCS, Cluster, Member
from .exceptions import PatroniException
from .postgresql.misc import postgres_version_to_int
@@ -165,7 +164,7 @@ class PatronictlPrettyTable(PrettyTable):
def parse_dcs(dcs: Optional[str]) -> Optional[Dict[str, Any]]:
"""Parse a DCS URL.
:param dcs: the DCS URL in the format ``DCS://HOST:PORT/NAMESPACE``. ``DCS`` can be one among:
:param dcs: the DCS URL in the format ``DCS://HOST:PORT``. ``DCS`` can be one among:
* ``consul``
* ``etcd``
@@ -174,12 +173,10 @@ def parse_dcs(dcs: Optional[str]) -> Optional[Dict[str, Any]]:
* ``zookeeper``
If ``DCS`` is not specified, assume ``etcd`` by default. If ``HOST`` is not specified, assume ``localhost`` by
default. If ``PORT`` is not specified, assume the default port of the given ``DCS``. If ``NAMESPACE`` is not
specified, use whatever is in config.
default. If ``PORT`` is not specified, assume the default port of the given ``DCS``.
:returns: ``None`` if *dcs* is ``None``, otherwise a dictionary. The dictionary represents *dcs* as if it were
parsed from the Patroni configuration file. Additionally, if a namespace is specified in *dcs*, return a
``namespace`` key with the parsed value.
parsed from the Patroni configuration file.
:raises:
:class:`PatroniCtlException`: if the DCS name in *dcs* is not valid.
@@ -197,9 +194,6 @@ def parse_dcs(dcs: Optional[str]) -> Optional[Dict[str, Any]]:
>>> parse_dcs('etcd3://random.com:2399')
{'etcd3': {'host': 'random.com:2399'}}
>>> parse_dcs('etcd3://random.com:2399/customnamespace')
{'etcd3': {'host': 'random.com:2399'}, 'namespace': '/customnamespace'}
"""
if dcs is None:
return None
@@ -216,27 +210,23 @@ def parse_dcs(dcs: Optional[str]) -> Optional[Dict[str, Any]]:
raise PatroniCtlException('Unknown dcs scheme: {}'.format(scheme))
default = DCS_DEFAULTS[scheme]
ret = yaml.safe_load(default['template'].format(host=parsed.hostname or 'localhost', port=port or default['port']))
if parsed.path and parsed.path.strip() != '/':
ret['namespace'] = parsed.path.strip()
return ret
return yaml.safe_load(default['template'].format(host=parsed.hostname or 'localhost', port=port or default['port']))
def load_config(path: str, dcs_url: Optional[str]) -> Dict[str, Any]:
"""Load configuration file from *path* and optionally override its DCS configuration with *dcs_url*.
:param path: path to the configuration file.
:param dcs_url: the DCS URL in the format ``DCS://HOST:PORT/NAMESPACE``, e.g. ``etcd3://random.com:2399/service``.
If given, override whatever DCS and ``namespace`` that are set in the configuration file. See :func:`parse_dcs`
for more information.
:param dcs_url: the DCS URL in the format ``DCS://HOST:PORT``, e.g. ``etcd3://random.com:2399``. If given override
whatever DCS is set in the configuration file.
:returns: a dictionary representing the configuration.
:raises:
:class:`PatroniCtlException`: if *path* does not exist or is not readable.
"""
from patroni.config import Config
if not (os.path.exists(path) and os.access(path, os.R_OK)):
if path != CONFIG_FILE_PATH: # bail if non-default config location specified but file not found / readable
raise PatroniCtlException('Provided config file {0} not existing or no read rights.'
@@ -1026,6 +1016,7 @@ def reload(obj: Dict[str, Any], cluster_name: str, member_names: List[str],
if r.status == 200:
click.echo('No changes to apply on member {0}'.format(member.name))
elif r.status == 202:
from patroni.config import get_global_config
config = get_global_config(cluster)
click.echo('Reload request received for member {0} and will be processed within {1} seconds'.format(
member.name, config.get('loop_wait') or dcs.loop_wait)
@@ -1107,6 +1098,7 @@ def restart(obj: Dict[str, Any], cluster_name: str, group: Optional[int], member
content['postgres_version'] = version
if scheduled_at:
from patroni.config import get_global_config
if get_global_config(cluster).is_paused:
raise PatroniCtlException("Can't schedule restart in the paused state")
content['schedule'] = scheduled_at.isoformat()
@@ -1234,22 +1226,19 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
dcs = get_dcs(obj, cluster_name, group)
cluster = dcs.get_cluster()
global_config = get_global_config(cluster)
if action == 'switchover' and (cluster.leader is None or not cluster.leader.name):
raise PatroniCtlException('This cluster has no leader')
# leader has to be be defined for switchover only
if action == 'switchover':
if cluster.leader is None or not cluster.leader.name:
raise PatroniCtlException('This cluster has no leader')
if leader is None:
if force or action == 'failover':
leader = cluster.leader and cluster.leader.name
else:
from patroni.config import get_global_config
prompt = 'Standby Leader' if get_global_config(cluster).is_standby_cluster else 'Primary'
leader = click.prompt(prompt, type=str, default=(cluster.leader and cluster.leader.member.name))
if leader is None:
if force:
leader = cluster.leader.name
else:
prompt = 'Standby Leader' if global_config.is_standby_cluster else 'Primary'
leader = click.prompt(prompt, type=str, default=(cluster.leader and cluster.leader.name))
if cluster.leader.name != leader:
raise PatroniCtlException(f'Member {leader} is not the leader of cluster {cluster_name}')
if leader is not None and cluster.leader and cluster.leader.member.name != leader:
raise PatroniCtlException('Member {0} is not the leader of cluster {1}'.format(leader, cluster_name))
# excluding members with nofailover tag
candidate_names = [str(m.name) for m in cluster.members if m.name != leader and not m.nofailover]
@@ -1269,16 +1258,7 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
raise PatroniCtlException(action.title() + ' target and source are the same.')
if candidate and candidate not in candidate_names:
raise PatroniCtlException(
f'Member {candidate} does not exist in cluster {cluster_name} or is tagged as nofailover')
if all((not force,
action == 'failover',
global_config.is_synchronous_mode,
not cluster.sync.is_empty,
not cluster.sync.matches(candidate, True))):
if click.confirm(f'Are you sure you want to failover to the asynchronous node {candidate}'):
raise PatroniCtlException('Aborting ' + action)
raise PatroniCtlException('Member {0} does not exist in cluster {1}'.format(candidate, cluster_name))
scheduled_at_str = None
scheduled_at = None
@@ -1291,29 +1271,25 @@ def _do_failover_or_switchover(obj: Dict[str, Any], action: str, cluster_name: s
scheduled_at = parse_scheduled(scheduled)
if scheduled_at:
if global_config.is_paused:
from patroni.config import get_global_config
if get_global_config(cluster).is_paused:
raise PatroniCtlException("Can't schedule switchover in the paused state")
scheduled_at_str = scheduled_at.isoformat()
failover_value = {'candidate': candidate}
if action == 'switchover':
failover_value['leader'] = leader
if scheduled_at_str:
failover_value['scheduled_at'] = scheduled_at_str
failover_value = {'leader': leader, 'candidate': candidate, 'scheduled_at': scheduled_at_str}
logging.debug(failover_value)
# By now we have established that the leader exists and the candidate exists
if not force:
demote_msg = f', demoting current leader {cluster.leader.name}' if cluster.leader else ''
demote_msg = ', demoting current leader ' + leader if leader else ''
if scheduled_at_str:
# only switchover can be scheduled
if not click.confirm(f'Are you sure you want to schedule switchover of cluster '
f'{cluster_name} at {scheduled_at_str}{demote_msg}?'):
# action as a var to catch a regression in the tests
if not click.confirm('Are you sure you want to schedule {0} of cluster {1} at {2}{3}?'
.format(action, cluster_name, scheduled_at_str, demote_msg)):
raise PatroniCtlException('Aborting scheduled ' + action)
else:
if not click.confirm(f'Are you sure you want to {action} cluster {cluster_name}{demote_msg}?'):
if not click.confirm('Are you sure you want to {0} cluster {1}{2}?'
.format(action, cluster_name, demote_msg)):
raise PatroniCtlException('Aborting ' + action)
r = None
@@ -1359,8 +1335,6 @@ def failover(obj: Dict[str, Any], cluster_name: str, group: Optional[int],
.. note::
If *leader* is given perform a switchover instead of a failover.
This behavior is deprecated. ``--leader`` option support will be
removed in the next major release.
.. seealso::
Refer to :func:`_do_failover_or_switchover` for details.
@@ -1374,12 +1348,7 @@ def failover(obj: Dict[str, Any], cluster_name: str, group: Optional[int],
:param candidate: name of a standby member to be promoted. Nodes that are tagged with ``nofailover`` cannot be used.
:param force: perform the failover or switchover without asking for confirmations.
"""
action = 'failover'
if leader:
action = 'switchover'
click.echo(click.style(
'Supplying a leader name using this command is deprecated and will be removed in a future version of'
' Patroni, change your scripts to use `switchover` instead.\nExecuting switchover!', fg='red'))
action = 'switchover' if leader else 'failover'
_do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force)
@@ -1752,6 +1721,7 @@ def wait_until_pause_is_applied(dcs: AbstractDCS, paused: bool, old_cluster: Clu
:param old_cluster: original cluster information before pause or unpause has been requested. Used to report which
nodes are still pending to have ``pause`` equal *paused* at a given point in time.
"""
from patroni.config import get_global_config
config = get_global_config(old_cluster)
click.echo("'{0}' request sent, waiting until it is recognized by all nodes".format(paused and 'pause' or 'resume'))
@@ -1789,6 +1759,7 @@ def toggle_pause(config: Dict[str, Any], cluster_name: str, group: Optional[int]
* ``pause`` state is already *paused*; or
* cluster contains no accessible members.
"""
from patroni.config import get_global_config
dcs = get_dcs(config, cluster_name, group)
cluster = dcs.get_cluster()
if get_global_config(cluster).is_paused == paused:
+101 -238
View File
@@ -23,13 +23,10 @@ import dateutil.parser
from ..exceptions import PatroniFatalException
from ..utils import deep_compare, uri
from ..tags import Tags
from ..utils import parse_int
if TYPE_CHECKING: # pragma: no cover
from ..config import Config
SLOT_ADVANCE_AVAILABLE_VERSION = 110000
CITUS_COORDINATOR_GROUP_ID = 0
citus_group_re = re.compile('^(0|[1-9][0-9]*)$')
slot_name_re = re.compile('^[a-z0-9_]{1,63}$')
@@ -159,7 +156,7 @@ def get_dcs(config: Union['Config', Dict[str, Any]]) -> 'AbstractDCS':
"""Attempt to load a Distributed Configuration Store from known available implementations.
.. note::
Using the list of available DCS classes returned by :func:`iter_dcs_classes` attempt to dynamically
Using the list of available DCS modules returned by :func:`iter_dcs_modules` attempt to dynamically import and
instantiate the class that implements a DCS using the abstract class :class:`AbstractDCS`.
Basic top-level configuration parameters retrieved from *config* are propagated to the DCS specific config
@@ -194,11 +191,11 @@ _Version = Union[int, str]
_Session = Union[int, float, str, None]
class Member(Tags, NamedTuple('Member',
[('version', _Version),
('name', str),
('session', _Session),
('data', Dict[str, Any])])):
class Member(NamedTuple('Member',
[('version', _Version),
('name', str),
('session', _Session),
('data', Dict[str, Any])])):
"""Immutable object (namedtuple) which represents single member of PostgreSQL cluster.
.. note::
@@ -319,10 +316,20 @@ class Member(Tags, NamedTuple('Member',
"""The ``tags`` value from :attr:`~Member.data` if defined, otherwise an empty dictionary."""
return self.data.get('tags', {})
@property
def nofailover(self) -> bool:
"""The value for ``nofailover`` in :attr:`Member`.tags`` if defined, otherwise ``False``."""
return self.tags.get('nofailover', False)
@property
def replicatefrom(self) -> Optional[str]:
"""The value for ``replicatefrom`` in :attr:`Member`.tags`` if defined."""
return self.tags.get('replicatefrom')
@property
def clonefrom(self) -> bool:
"""``True`` if both ``clonefrom`` tag is ``True`` and a connection URL is defined."""
return super().clonefrom and bool(self.conn_url)
return self.tags.get('clonefrom', False) and bool(self.conn_url)
@property
def state(self) -> str:
@@ -352,11 +359,6 @@ class Member(Tags, NamedTuple('Member',
logger.debug('Failed to parse Patroni version %s', version)
return None
@property
def lsn(self) -> Optional[int]:
"""Current LSN (receive/flush/replay)."""
return parse_int(self.data.get('xlog_location'))
class RemoteMember(Member):
"""Represents a remote member (typically a primary) for a standby cluster.
@@ -785,71 +787,16 @@ class TimelineHistory(NamedTuple):
return TimelineHistory(version, value, lines)
class Status(NamedTuple):
"""Immutable object (namedtuple) which represents `/status` key.
Consists of the following fields:
:ivar last_lsn: :class:`int` object containing position of last known leader LSN.
:ivar slots: state of permanent replication slots on the primary in the format: ``{"slot_name": int}``.
"""
last_lsn: int
slots: Optional[Dict[str, int]]
@staticmethod
def empty() -> 'Status':
"""Construct an empty :class:`Status` instance.
:returns: empty :class:`Status` object.
"""
return Status(0, None)
@staticmethod
def from_node(value: Union[str, Dict[str, Any], None]) -> 'Status':
"""Factory method to parse *value* as :class:`Status` object.
:param value: JSON serialized string
:returns: constructed :class:`Status` object.
"""
try:
if isinstance(value, str):
value = json.loads(value)
except Exception:
return Status.empty()
if isinstance(value, int): # legacy
return Status(value, None)
if not isinstance(value, dict):
return Status.empty()
try:
last_lsn = int(value.get('optime', ''))
except Exception:
last_lsn = 0
slots: Union[str, Dict[str, int], None] = value.get('slots')
if isinstance(slots, str):
try:
slots = json.loads(slots)
except Exception:
slots = None
if not isinstance(slots, dict):
slots = None
return Status(last_lsn, slots)
class Cluster(NamedTuple('Cluster',
[('initialize', Optional[str]),
('config', Optional[ClusterConfig]),
('leader', Optional[Leader]),
('status', Status),
('last_lsn', int),
('members', List[Member]),
('failover', Optional[Failover]),
('sync', SyncState),
('history', Optional[TimelineHistory]),
('slots', Optional[Dict[str, int]]),
('failsafe', Optional[Dict[str, str]]),
('workers', Dict[int, 'Cluster'])])):
"""Immutable object (namedtuple) which represents PostgreSQL or Citus cluster.
@@ -863,11 +810,13 @@ class Cluster(NamedTuple('Cluster',
:ivar initialize: shows whether this cluster has initialization key stored in DC or not.
:ivar config: global dynamic configuration, reference to `ClusterConfig` object.
:ivar leader: :class:`Leader` object which represents current leader of the cluster.
:ivar status: :class:`Status` object which represents the `/status` key.
:ivar last_lsn: :class:int object containing position of last known leader LSN.
This value is stored in the `/status` key or `/optime/leader` (legacy) key.
:ivar members: list of:class:` Member` objects, all PostgreSQL cluster members including leader
:ivar failover: reference to :class:`Failover` object.
:ivar sync: reference to :class:`SyncState` object, last observed synchronous replication state.
:ivar history: reference to `TimelineHistory` object.
:ivar slots: state of permanent logical replication slots on the primary in the format: {"slot_name": int}.
:ivar failsafe: failsafe topology. Node is allowed to become the leader only if its name is found in this list.
:ivar workers: dictionary of workers of the Citus cluster, optional. Each key is an :class:`int` representing
the group, and the corresponding value is a :class:`Cluster` instance.
@@ -879,20 +828,10 @@ class Cluster(NamedTuple('Cluster',
kwargs['workers'] = {}
return super(Cluster, cls).__new__(cls, *args, **kwargs)
@property
def last_lsn(self) -> int:
"""Last known leader LSN."""
return self.status.last_lsn
@property
def slots(self) -> Optional[Dict[str, int]]:
"""State of permanent replication slots on the primary in the format: ``{"slot_name": int}``."""
return self.status.slots
@staticmethod
def empty() -> 'Cluster':
"""Produce an empty :class:`Cluster` instance."""
return Cluster(None, None, None, Status.empty(), [], None, SyncState.empty(), None, None, {})
return Cluster(None, None, None, 0, [], None, SyncState.empty(), None, None, None, {})
def is_empty(self):
"""Validate definition of all attributes of this :class:`Cluster` instance.
@@ -915,7 +854,7 @@ class Cluster(NamedTuple('Cluster',
>>> assert bool(cluster) is False
>>> cluster = Cluster(None, None, None, Status(0, None), [1, 2, 3], None, SyncState.empty(), None, None, {})
>>> cluster = Cluster(None, None, None, 0, [1, 2, 3], None, SyncState.empty(), None, None, None, {})
>>> len(cluster)
1
@@ -971,59 +910,22 @@ class Cluster(NamedTuple('Cluster',
candidates = [m for m in self.members if m.clonefrom and m.is_running and m.name not in exclude]
return candidates[randint(0, len(candidates) - 1)] if candidates else self.leader
@staticmethod
def is_physical_slot(value: Union[Any, Dict[str, Any]]) -> bool:
"""Check whether provided configuration is for permanent physical replication slot.
:param value: configuration of the permanent replication slot.
:returns: ``True`` if *value* is a physical replication slot, otherwise ``False``.
"""
return not value or isinstance(value, dict) and value.get('type', 'physical') == 'physical'
@staticmethod
def is_logical_slot(value: Union[Any, Dict[str, Any]]) -> bool:
"""Check whether provided configuration is for permanent logical replication slot.
:param value: configuration of the permanent replication slot.
:returns: ``True`` if *value* is a logical replication slot, otherwise ``False``.
"""
return isinstance(value, dict) \
and value.get('type', 'logical') == 'logical' \
and bool(value.get('database') and value.get('plugin'))
@property
def __permanent_slots(self) -> Dict[str, Union[Dict[str, Any], Any]]:
"""Dictionary of permanent replication slots with their known LSN."""
ret: Dict[str, Union[Dict[str, Any], Any]] = deepcopy(self.config.permanent_slots if self.config else {})
members: Dict[str, int] = {slot_name_from_member_name(m.name): m.lsn or 0 for m in self.members}
slots: Dict[str, int] = {k: parse_int(v) or 0 for k, v in (self.slots or {}).items()}
for name, value in list(ret.items()):
if not value:
value = ret[name] = {}
if isinstance(value, dict):
# for permanent physical slots we want to get MAX LSN from the `Cluster.slots` and from the
# member with the matching name. It is necessary because we may have the replication slot on
# the primary that is streaming from the other standby node using the `replicatefrom` tag.
lsn = max(members.get(name, 0) if self.is_physical_slot(value) else 0, slots.get(name, 0))
if lsn:
value['lsn'] = lsn
else:
# Don't let anyone set 'lsn' in the global configuration :)
value.pop('lsn', None)
return ret
"""Dictionary of permanent replication slots."""
return self.config and self.config.permanent_slots or {}
@property
def __permanent_physical_slots(self) -> Dict[str, Any]:
"""Dictionary of permanent ``physical`` replication slots."""
return {name: value for name, value in self.__permanent_slots.items() if self.is_physical_slot(value)}
return {name: value for name, value in self.__permanent_slots.items()
if not value or isinstance(value, dict) and value.get('type', 'physical') == 'physical'}
@property
def __permanent_logical_slots(self) -> Dict[str, Any]:
"""Dictionary of permanent ``logical`` replication slots."""
return {name: value for name, value in self.__permanent_slots.items() if self.is_logical_slot(value)}
return {name: value for name, value in self.__permanent_slots.items() if isinstance(value, dict)
and value.get('type', 'logical') == 'logical' and value.get('database') and value.get('plugin')}
@property
def use_slots(self) -> bool:
@@ -1036,6 +938,7 @@ class Cluster(NamedTuple('Cluster',
Will log an error if:
* Conflicting slot names between members are found
* Any logical slots are disabled, due to version compatibility, and *show_error* is ``True``.
:param my_name: name of this node.
@@ -1048,11 +951,21 @@ class Cluster(NamedTuple('Cluster',
:returns: final dictionary of slot names, after merging with permanent slots and performing sanity checks.
"""
slots: Dict[str, Dict[str, str]] = self._get_members_slots(my_name, role)
permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster=is_standby_cluster,
role=role, nofailover=nofailover,
major_version=major_version)
slot_members: List[str] = self._get_slot_members(my_name, role)
slots: Dict[str, Dict[str, str]] = {slot_name_from_member_name(name): {'type': 'physical'}
for name in slot_members}
if len(slots) < len(slot_members):
# Find which names are conflicting for a nicer error message
slot_conflicts: Dict[str, List[str]] = defaultdict(list)
for name in slot_members:
slot_conflicts[slot_name_from_member_name(name)].append(name)
logger.error("Following cluster members share a replication slot name: %s",
"; ".join(f"{', '.join(v)} map to {k}"
for k, v in slot_conflicts.items() if len(v) > 1))
permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster, role, nofailover)
disabled_permanent_logical_slots: List[str] = self._merge_permanent_slots(
slots, permanent_slots, my_name, major_version)
@@ -1062,7 +975,8 @@ class Cluster(NamedTuple('Cluster',
return slots
def _merge_permanent_slots(self, slots: Dict[str, Dict[str, str]], permanent_slots: Dict[str, Any], my_name: str,
@staticmethod
def _merge_permanent_slots(slots: Dict[str, Dict[str, str]], permanent_slots: Dict[str, Any], my_name: str,
major_version: int) -> List[str]:
"""Merge replication *slots* for members with *permanent_slots*.
@@ -1097,8 +1011,8 @@ class Cluster(NamedTuple('Cluster',
slots[name] = value
continue
if self.is_logical_slot(value):
if major_version < SLOT_ADVANCE_AVAILABLE_VERSION:
if value['type'] == 'logical' and value.get('database') and value.get('plugin'):
if major_version < 110000:
disabled_permanent_logical_slots.append(name)
elif name in slots:
logger.error("Permanent logical replication slot {'%s': %s} is conflicting with"
@@ -1110,9 +1024,8 @@ class Cluster(NamedTuple('Cluster',
logger.error("Bad value for slot '%s' in permanent_slots: %s", name, permanent_slots[name])
return disabled_permanent_logical_slots
def _get_permanent_slots(self, *, is_standby_cluster: bool, role: str,
nofailover: bool, major_version: int) -> Dict[str, Any]:
"""Get configured permanent replication slots.
def _get_permanent_slots(self, is_standby_cluster: bool, role: str, nofailover: bool) -> Dict[str, Any]:
"""Get configured permanent slot names.
.. note::
Permanent replication slots are only considered if ``use_slots`` configuration is enabled.
@@ -1127,7 +1040,6 @@ class Cluster(NamedTuple('Cluster',
the outside because we want to protect from the ``/config`` key removal.
:param role: role of this node -- ``primary``, ``standby_leader`` or ``replica``.
:param nofailover: ``True`` if this node is tagged to not be a failover candidate.
:param major_version: postgresql major version.
:returns: dictionary of permanent slot names mapped to attributes.
"""
@@ -1135,112 +1047,55 @@ class Cluster(NamedTuple('Cluster',
return {}
if is_standby_cluster:
return self.__permanent_physical_slots \
if major_version >= SLOT_ADVANCE_AVAILABLE_VERSION or role == 'standby_leader' else {}
return self.__permanent_physical_slots if role == 'standby_leader' else {}
return self.__permanent_slots if major_version >= SLOT_ADVANCE_AVAILABLE_VERSION\
or role in ('master', 'primary') else self.__permanent_logical_slots
return self.__permanent_slots if role in ('master', 'primary') else self.__permanent_logical_slots
def _get_members_slots(self, my_name: str, role: str) -> Dict[str, Dict[str, str]]:
"""Get physical replication slots configuration for members that sourcing from this node.
def _get_slot_members(self, my_name: str, role: str) -> List[str]:
"""Get a list of member names that have replication slots sourcing from this node.
If the ``replicatefrom`` tag is set on the member - we should not create the replication slot for it on
the current primary, because that member would replicate from elsewhere. We still create the slot if
the ``replicatefrom`` destination member is currently not a member of the cluster (fallback to the
primary), or if ``replicatefrom`` destination member happens to be the current primary.
Will log an error if:
* Conflicting slot names between members are found
:param my_name: name of this node.
:param role: role of this node, if this is a ``primary`` or ``standby_leader`` return list of members
replicating from this node. If not then return a list of members replicating as cascaded
replicas from this node.
:returns: dictionary of physical replication slots that should exist on a given node.
:returns: list of member names.
"""
if not self.use_slots:
return {}
# we always want to exclude the member with our name from the list
members = filter(lambda m: m.name != my_name, self.members)
return []
if role in ('master', 'primary', 'standby_leader'):
members = [m for m in members if m.replicatefrom is None
or m.replicatefrom == my_name or not self.has_member(m.replicatefrom)]
slot_members = [m.name for m in self.members
if m.name != my_name
and (m.replicatefrom is None
or m.replicatefrom == my_name
or not self.has_member(m.replicatefrom))]
else:
# only manage slots for replicas that replicate from this one, except for the leader among them
members = [m for m in members if m.replicatefrom == my_name and m.name != self.leader_name]
slot_members = [m.name for m in self.members
if m.replicatefrom == my_name and m.name != self.leader_name]
return slot_members
slots = {slot_name_from_member_name(m.name): {'type': 'physical'} for m in members}
if len(slots) < len(members):
# Find which names are conflicting for a nicer error message
slot_conflicts: Dict[str, List[str]] = defaultdict(list)
for member in members:
slot_conflicts[slot_name_from_member_name(member.name)].append(member.name)
logger.error("Following cluster members share a replication slot name: %s",
"; ".join(f"{', '.join(v)} map to {k}"
for k, v in slot_conflicts.items() if len(v) > 1))
return slots
def has_permanent_slots(self, my_name: str, *, is_standby_cluster: bool = False, nofailover: bool = False,
major_version: int = SLOT_ADVANCE_AVAILABLE_VERSION) -> bool:
"""Check if the given member node has permanent replication slots configured.
:param my_name: name of the member node to check.
:param is_standby_cluster: ``True`` if it is known that this is a standby cluster. We pass the value from
the outside because we want to protect from the ``/config`` key removal.
:param nofailover: ``True`` if this node is tagged to not be a failover candidate.
:param major_version: postgresql major version.
:returns: ``True`` if there are permanent replication slots configured, otherwise ``False``.
"""
role = 'replica'
members_slots: Dict[str, Dict[str, str]] = self._get_members_slots(my_name, role)
permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster=is_standby_cluster,
role=role, nofailover=nofailover,
major_version=major_version)
slots = deepcopy(members_slots)
self._merge_permanent_slots(slots, permanent_slots, my_name, major_version)
return len(slots) > len(members_slots) or any(self.is_physical_slot(v) for v in permanent_slots.values())
def filter_permanent_slots(self, slots: Dict[str, int], is_standby_cluster: bool,
major_version: int) -> Dict[str, int]:
"""Filter out all non-permanent slots from provided *slots* dict.
:param slots: slot names with LSN values
:param is_standby_cluster: ``True`` if it is known that this is a standby cluster. We pass the value from
the outside because we want to protect from the ``/config`` key removal.
:param major_version: postgresql major version.
:returns: a :class:`dict` object that contains only slots that are known to be permanent.
"""
if major_version < SLOT_ADVANCE_AVAILABLE_VERSION:
return {} # for legacy PostgreSQL we don't support permanent slots on standby nodes
permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster=is_standby_cluster,
role='replica',
nofailover=False,
major_version=major_version)
members_slots = {slot_name_from_member_name(m.name) for m in self.members}
return {name: value for name, value in slots.items() if name in permanent_slots
and (self.is_physical_slot(permanent_slots[name])
or self.is_logical_slot(permanent_slots[name]) and name not in members_slots)}
def _has_permanent_logical_slots(self, my_name: str, nofailover: bool) -> bool:
def has_permanent_logical_slots(self, my_name: str, nofailover: bool, major_version: int = 110000) -> bool:
"""Check if the given member node has permanent ``logical`` replication slots configured.
:param my_name: name of the member node to check.
:param nofailover: ``True`` if this node is tagged to not be a failover candidate.
:param major_version: the PostgreSQL major version number.
:returns: ``True`` if any detected replications slots are ``logical``, otherwise ``False``.
:returns: ``False`` if PostgreSQL is < 11, ``True`` if any detected replications slots are ``logical``.
"""
slots = self.get_replication_slots(my_name, 'replica', nofailover, SLOT_ADVANCE_AVAILABLE_VERSION).values()
if major_version < 110000:
return False
slots = self.get_replication_slots(my_name, 'replica', nofailover, major_version).values()
return any(v for v in slots if v.get("type") == "logical")
def should_enforce_hot_standby_feedback(self, my_name: str, nofailover: bool) -> bool:
def should_enforce_hot_standby_feedback(self, my_name: str, nofailover: bool, major_version: int) -> bool:
"""Determine whether ``hot_standby_feedback`` should be enabled for the given member.
The ``hot_standby_feedback`` must be enabled if the current replica has ``logical`` slots,
@@ -1248,16 +1103,20 @@ class Cluster(NamedTuple('Cluster',
:param my_name: name of the member node to check.
:param nofailover: ``True`` if this node is tagged to not be a failover candidate.
:param major_version: PostgreSQL major version number.
:returns: ``True`` if this node or any member replicating from this node has
permanent logical slots, otherwise ``False``.
:returns: ``True`` if this node or any member replicating from this node has permanent logical slots.
``False`` if PostgreSQL major version is < 11.
"""
if self._has_permanent_logical_slots(my_name, nofailover):
if major_version < 110000:
return False
if self.has_permanent_logical_slots(my_name, nofailover, major_version):
return True
if self.use_slots:
members = [m for m in self.members if m.replicatefrom == my_name and m.name != self.leader_name]
return any(self.should_enforce_hot_standby_feedback(m.name, m.nofailover) for m in members)
return any(self.should_enforce_hot_standby_feedback(m.name, m.nofailover, major_version) for m in members)
return False
def get_my_slot_name_on_primary(self, my_name: str, replicatefrom: Optional[str]) -> str:
@@ -1289,20 +1148,19 @@ class Cluster(NamedTuple('Cluster',
:Example:
No history provided:
>>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, 0, None, {}).timeline
>>> Cluster(0, 0, 0, 0, 0, 0, 0, 0, 0, None, {}).timeline
0
Empty history assume timeline is ``1``:
>>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, TimelineHistory.from_node(1, '[]'), None, {}).timeline
>>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[]'), 0, None, {}).timeline
1
Invalid history format, a string of ``a``, returns ``0``:
>>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, TimelineHistory.from_node(1, '[["a"]]'), None, {}).timeline
>>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[["a"]]'), 0, None, {}).timeline
0
History as a list of strings:
>>> history = TimelineHistory.from_node(1, '[["3", "2", "1"]]')
>>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, history, None, {}).timeline
>>> Cluster(0, 0, 0, 0, 0, 0, 0, TimelineHistory.from_node(1, '[["3", "2", "1"]]'), 0, None, {}).timeline
4
"""
if self.history:
@@ -1591,6 +1449,9 @@ class AbstractDCS(abc.ABC):
primary and exception raised, instance would be demoted.
"""
def _bypass_caches(self) -> None:
"""Used only in Zookeeper."""
def __get_patroni_cluster(self, path: Optional[str] = None) -> Cluster:
"""Low level method to load a :class:`Cluster` object from DCS.
@@ -1633,14 +1494,15 @@ class AbstractDCS(abc.ABC):
dict.
"""
groups = self._load_cluster(self._base_path + '/', self._citus_cluster_loader)
if TYPE_CHECKING: # pragma: no cover
assert isinstance(groups, dict)
cluster = groups.pop(CITUS_COORDINATOR_GROUP_ID, Cluster.empty())
cluster.workers.update(groups)
if isinstance(groups, Cluster): # Zookeeper could return a cached version
cluster = groups
else:
cluster = groups.pop(CITUS_COORDINATOR_GROUP_ID, Cluster.empty())
cluster.workers.update(groups)
return cluster
def get_cluster(self) -> Cluster:
"""Retrieve a fresh view of DCS.
def get_cluster(self, force: bool = False) -> Cluster:
"""Retrieve an appropriate cached or fresh view of DCS.
.. note::
Stores copy of time, status and failsafe values for comparison in DCS update decisions.
@@ -1648,8 +1510,12 @@ class AbstractDCS(abc.ABC):
Returns either a Citus or Patroni implementation of :class:`Cluster` depending on availability.
:param force: a value of ``True`` will override Zookeeper caching features.
:returns:
"""
if force:
self._bypass_caches()
try:
cluster = self._get_citus_cluster() if self.is_citus_coordinator() else self.__get_patroni_cluster()
except Exception:
@@ -1892,29 +1758,26 @@ class AbstractDCS(abc.ABC):
"""
@abc.abstractmethod
def _delete_leader(self, leader: Leader) -> bool:
def _delete_leader(self) -> bool:
"""Remove leader key from DCS.
This method should remove leader key if current instance is the leader.
:param leader: :class:`Leader` object with information about the leader.
:returns: ``True`` if successfully committed to DCS.
"""
def delete_leader(self, leader: Optional[Leader], last_lsn: Optional[int] = None) -> bool:
def delete_leader(self, last_lsn: Optional[int] = None) -> bool:
"""Update ``optime/leader`` and voluntarily remove leader key from DCS.
This method should remove leader key if current instance is the leader.
:param leader: :class:`Leader` object with information about the leader.
:param last_lsn: latest checkpoint location in bytes.
:returns: boolean result of called abstract :meth:`~AbstractDCS._delete_leader`.
"""
if last_lsn:
self.write_status({self._OPTIME: last_lsn})
return bool(leader) and self._delete_leader(leader)
return self._delete_leader()
@abc.abstractmethod
def cancel_initialization(self) -> bool:
+25 -36
View File
@@ -15,7 +15,7 @@ from urllib3.exceptions import HTTPError
from urllib.parse import urlencode, urlparse, quote
from typing import Any, Callable, Dict, List, Mapping, NamedTuple, Optional, Union, Tuple, TYPE_CHECKING
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, Status, SyncState, \
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState, \
TimelineHistory, ReturnFalseException, catch_return_false_exception, citus_group_re
from ..exceptions import DCSError
from ..utils import deep_compare, parse_bool, Retry, RetryFailedError, split_host_port, uri, USER_AGENT
@@ -141,36 +141,6 @@ class HTTPClient(object):
class ConsulClient(base.Consul):
def __init__(self, *args: Any, **kwargs: Any) -> None:
"""
Consul client with Patroni customisations.
.. note::
Parameters, *token*, *cert* and *ca_cert* are not passed to the parent class :class:`consul.base.Consul`.
Original class documentation,
*token* is an optional ``ACL token``. If supplied it will be used by
default for all requests made with this client session. It's still
possible to override this token by passing a token explicitly for a
request.
*consistency* sets the consistency mode to use by default for all reads
that support the consistency option. It's still possible to override
this by passing explicitly for a given request. *consistency* can be
either 'default', 'consistent' or 'stale'.
*dc* is the datacenter that this agent will communicate with.
By default, the datacenter of the host is used.
*verify* is whether to verify the SSL certificate for HTTPS requests
*cert* client side certificates for HTTPS requests
:param args: positional arguments to pass to :class:`consul.base.Consul`
:param kwargs: keyword arguments, with *cert*, *ca_cert* and *token* removed, passed to
:class:`consul.base.Consul`
"""
self._cert = kwargs.pop('cert', None)
self._ca_cert = kwargs.pop('ca_cert', None)
self.token = kwargs.get('token')
@@ -383,8 +353,23 @@ class Consul(AbstractDCS):
history = history and TimelineHistory.from_node(history['ModifyIndex'], history['Value'])
# get last known leader lsn and slots
status = nodes.get(self._STATUS) or nodes.get(self._LEADER_OPTIME)
status = Status.from_node(status and status['Value'])
status = nodes.get(self._STATUS)
if status:
try:
status = json.loads(status['Value'])
last_lsn = status.get(self._OPTIME)
slots = status.get('slots')
except Exception:
slots = last_lsn = None
else:
last_lsn = nodes.get(self._LEADER_OPTIME)
last_lsn = last_lsn and last_lsn['Value']
slots = None
try:
last_lsn = int(last_lsn or '')
except Exception:
last_lsn = 0
# get list of members
members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
@@ -413,7 +398,7 @@ class Consul(AbstractDCS):
except Exception:
failsafe = None
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
@property
def _consistency(self) -> str:
@@ -658,8 +643,12 @@ class Consul(AbstractDCS):
return self._client.kv.put(self.history_path, value)
@catch_consul_errors
def _delete_leader(self, leader: Leader) -> bool:
return self._client.kv.delete(self.leader_path, cas=int(leader.version))
def _delete_leader(self) -> bool:
cluster = self.cluster
if cluster and isinstance(cluster.leader, Leader) and\
cluster.leader.name == self._name and isinstance(cluster.leader.version, int):
return self._client.kv.delete(self.leader_path, cas=cluster.leader.version)
return True
@catch_consul_errors
def set_sync_state_value(self, value: str, version: Optional[int] = None) -> Union[int, bool]:
+20 -5
View File
@@ -21,7 +21,7 @@ from urllib.parse import urlparse
from urllib3 import Timeout
from urllib3.exceptions import HTTPError, ReadTimeoutError, ProtocolError
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, Status, SyncState, \
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState, \
TimelineHistory, ReturnFalseException, catch_return_false_exception, citus_group_re
from ..exceptions import DCSError
from ..request import get as requests_get
@@ -677,8 +677,23 @@ class Etcd(AbstractEtcd):
history = history and TimelineHistory.from_node(history.modifiedIndex, history.value)
# get last know leader lsn and slots
status = nodes.get(self._STATUS) or nodes.get(self._LEADER_OPTIME)
status = Status.from_node(status and status.value)
status = nodes.get(self._STATUS)
if status:
try:
status = json.loads(status.value)
last_lsn = status.get(self._OPTIME)
slots = status.get('slots')
except Exception:
slots = last_lsn = None
else:
last_lsn = nodes.get(self._LEADER_OPTIME)
last_lsn = last_lsn and last_lsn.value
slots = None
try:
last_lsn = int(last_lsn or '')
except Exception:
last_lsn = 0
# get list of members
members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
@@ -707,7 +722,7 @@ class Etcd(AbstractEtcd):
except Exception:
failsafe = None
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
def _cluster_loader(self, path: str) -> Cluster:
result = self.retry(self._client.read, path, recursive=True, quorum=self._ctl)
@@ -794,7 +809,7 @@ class Etcd(AbstractEtcd):
return bool(self.retry(self._client.write, self.initialize_path, sysid, prevExist=(not create_new)))
@catch_etcd_errors
def _delete_leader(self, leader: Leader) -> bool:
def _delete_leader(self) -> bool:
return bool(self._client.delete(self.leader_path, prevValue=self._name))
@catch_etcd_errors
+63 -75
View File
@@ -15,7 +15,7 @@ from urllib3.exceptions import ReadTimeoutError, ProtocolError
from threading import Condition, Lock, Thread
from typing import Any, Callable, Collection, Dict, Iterator, List, Optional, Tuple, Type, TYPE_CHECKING, Union
from . import ClusterConfig, Cluster, Failover, Leader, Member, Status, SyncState, \
from . import ClusterConfig, Cluster, Failover, Leader, Member, SyncState, \
TimelineHistory, catch_return_false_exception, citus_group_re
from .etcd import AbstractEtcdClientWithFailover, AbstractEtcd, catch_etcd_errors, DnsCachingResolver, Retry
from ..exceptions import DCSError, PatroniException
@@ -124,10 +124,6 @@ class AuthFailed(InvalidArgument):
error = "etcdserver: authentication failed, invalid user ID or password"
class AuthOldRevision(InvalidArgument):
error = "etcdserver: revision of auth store is old"
class PermissionDenied(Etcd3ClientError):
code = GRPCCode.PermissionDenied
error = "etcdserver: permission denied"
@@ -197,12 +193,6 @@ def build_range_request(key: str, range_end: Union[bytes, str, None] = None) ->
return fields
class ReAuthenticateMode(IntEnum):
NOT_REQUIRED = 0
REQUIRED = 1
WITHOUT_WATCHER_RESTART = 2
def _handle_auth_errors(func: Callable[..., Any]) -> Any:
def wrapper(self: 'Etcd3Client', *args: Any, **kwargs: Any) -> Any:
return self.handle_auth_errors(func, *args, **kwargs)
@@ -214,7 +204,6 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
ERROR_CLS = Etcd3Error
def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cache_ttl: int = 300) -> None:
self._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
self._token = None
self._cluster_version: Tuple[int, ...] = tuple()
super(Etcd3Client, self).__init__({**config, 'version_prefix': '/v3beta'}, dns_resolver, cache_ttl)
@@ -293,7 +282,7 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
fields['retry'] = retry
return self.api_execute(self.version_prefix + method, self._MPOST, fields)
def authenticate(self, *, restart_watcher: bool = True, retry: Optional[Retry] = None) -> bool:
def authenticate(self) -> bool:
if self._use_proxies and not self._cluster_version:
kwargs = self._prepare_common_parameters(1)
self._ensure_version_prefix(self._base_uri, **kwargs)
@@ -302,7 +291,7 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
logger.info('Trying to authenticate on Etcd...')
old_token, self._token = self._token, None
try:
response = self.call_rpc('/auth/authenticate', {'name': self.username, 'password': self.password}, retry)
response = self.call_rpc('/auth/authenticate', {'name': self.username, 'password': self.password})
except AuthNotEnabled:
logger.info('Etcd authentication is not enabled')
self._token = None
@@ -313,65 +302,48 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
self._token = response.get('token')
return old_token != self._token
def handle_auth_errors(self: 'Etcd3Client', func: Callable[..., Any], *args: Any,
retry: Optional[Retry] = None, **kwargs: Any) -> Any:
exc = None
while True:
if self._reauthenticate_reason:
if self.username and self.password:
self.authenticate(
restart_watcher=self._reauthenticate_reason != ReAuthenticateMode.WITHOUT_WATCHER_RESTART,
retry=retry)
self._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
if retry:
retry.ensure_deadline(0)
else:
msg = 'Username or password not set, authentication is not possible'
logger.fatal(msg)
raise exc or Etcd3Exception(msg)
def handle_auth_errors(self: 'Etcd3Client', func: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
def retry(ex: Exception) -> Any:
if self.username and self.password:
self.authenticate()
return func(self, *args, **kwargs)
else:
logger.fatal('Username or password not set, authentication is not possible')
raise ex
try:
return func(self, *args, retry=retry, **kwargs)
except (UserEmpty, PermissionDenied) as e: # no token provided
# PermissionDenied is raised on 3.0 and 3.1
if self._cluster_version < (3, 3) and (not isinstance(e, PermissionDenied)
or self._cluster_version < (3, 2)):
raise UnsupportedEtcdVersion('Authentication is required by Etcd cluster but not '
'supported on version lower than 3.3.0. Cluster version: '
'{0}'.format('.'.join(map(str, self._cluster_version))))
exc = e
except InvalidAuthToken as e:
logger.error('Invalid auth token: %s', self._token)
exc = e
except AuthOldRevision as e:
logger.error('Auth token is for old revision of auth store')
exc = e
self._reauthenticate_reason = ReAuthenticateMode.WITHOUT_WATCHER_RESTART \
if isinstance(exc, AuthOldRevision) else ReAuthenticateMode.REQUIRED
if not retry:
raise exc
retry.ensure_deadline(0.5, exc)
try:
return func(self, *args, **kwargs)
except (UserEmpty, PermissionDenied) as e: # no token provided
# PermissionDenied is raised on 3.0 and 3.1
if self._cluster_version < (3, 3) and (not isinstance(e, PermissionDenied)
or self._cluster_version < (3, 2)):
raise UnsupportedEtcdVersion('Authentication is required by Etcd cluster but not '
'supported on version lower than 3.3.0. Cluster version: '
'{0}'.format('.'.join(map(str, self._cluster_version))))
return retry(e)
except InvalidAuthToken as e:
logger.error('Invalid auth token: %s', self._token)
return retry(e)
@_handle_auth_errors
def range(self, key: str, range_end: Union[bytes, str, None] = None, serializable: bool = True,
*, retry: Optional[Retry] = None) -> Dict[str, Any]:
retry: Optional[Retry] = None) -> Dict[str, Any]:
params = build_range_request(key, range_end)
params['serializable'] = serializable # For better performance. We can tolerate stale reads
return self.call_rpc('/kv/range', params, retry)
def prefix(self, key: str, serializable: bool = True, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
return self.range(key, prefix_range_end(key), serializable, retry=retry)
def prefix(self, key: str, serializable: bool = True, retry: Optional[Retry] = None) -> Dict[str, Any]:
return self.range(key, prefix_range_end(key), serializable, retry)
@_handle_auth_errors
def lease_grant(self, ttl: int, *, retry: Optional[Retry] = None) -> str:
def lease_grant(self, ttl: int, retry: Optional[Retry] = None) -> str:
return self.call_rpc('/lease/grant', {'TTL': ttl}, retry)['ID']
def lease_keepalive(self, ID: str, *, retry: Optional[Retry] = None) -> Optional[str]:
def lease_keepalive(self, ID: str, retry: Optional[Retry] = None) -> Optional[str]:
return self.call_rpc('/lease/keepalive', {'ID': ID}, retry).get('result', {}).get('TTL')
@_handle_auth_errors
def txn(self, compare: Dict[str, Any], success: Dict[str, Any],
failure: Optional[Dict[str, Any]] = None, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
failure: Optional[Dict[str, Any]] = None, retry: Optional[Retry] = None) -> Dict[str, Any]:
fields = {'compare': [compare], 'success': [success]}
if failure:
fields['failure'] = [failure]
@@ -380,7 +352,7 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
@_handle_auth_errors
def put(self, key: str, value: str, lease: Optional[str] = None, create_revision: Optional[str] = None,
mod_revision: Optional[str] = None, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
mod_revision: Optional[str] = None, retry: Optional[Retry] = None) -> Dict[str, Any]:
fields = {'key': base64_encode(key), 'value': base64_encode(value)}
if lease:
fields['lease'] = lease
@@ -395,14 +367,14 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
@_handle_auth_errors
def deleterange(self, key: str, range_end: Union[bytes, str, None] = None,
mod_revision: Optional[str] = None, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
mod_revision: Optional[str] = None, retry: Optional[Retry] = None) -> Dict[str, Any]:
fields = build_range_request(key, range_end)
if mod_revision is None:
return self.call_rpc('/kv/deleterange', fields, retry)
compare = {'target': 'MOD', 'mod_revision': mod_revision, 'key': fields['key']}
return self.txn(compare, {'request_delete_range': fields}, retry=retry)
def deleteprefix(self, key: str, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
def deleteprefix(self, key: str, retry: Optional[Retry] = None) -> Dict[str, Any]:
return self.deleterange(key, prefix_range_end(key), retry=retry)
def watchrange(self, key: str, range_end: Union[bytes, str, None] = None,
@@ -602,9 +574,9 @@ class PatroniEtcd3Client(Etcd3Client):
super(PatroniEtcd3Client, self).set_base_uri(value)
self._restart_watcher()
def authenticate(self, *, restart_watcher: bool = True, retry: Optional[Retry] = None) -> bool:
ret = super(PatroniEtcd3Client, self).authenticate(restart_watcher=restart_watcher, retry=retry)
if ret and restart_watcher:
def authenticate(self) -> bool:
ret = super(PatroniEtcd3Client, self).authenticate()
if ret:
self._restart_watcher()
return ret
@@ -659,8 +631,8 @@ class PatroniEtcd3Client(Etcd3Client):
return ret
def txn(self, compare: Dict[str, Any], success: Dict[str, Any],
failure: Optional[Dict[str, Any]] = None, *, retry: Optional[Retry] = None) -> Dict[str, Any]:
ret = super(PatroniEtcd3Client, self).txn(compare, success, failure, retry=retry)
failure: Optional[Dict[str, Any]] = None, retry: Optional[Retry] = None) -> Dict[str, Any]:
ret = super(PatroniEtcd3Client, self).txn(compare, success, failure, retry)
# Here we abuse the fact that the `failure` is only set in the call from update_leader().
# In all other cases the txn() call failure may be an indicator of a stale cache,
# and therefore we want to restart watcher.
@@ -704,12 +676,12 @@ class Etcd3(AbstractEtcd):
if not force and self._lease and self._last_lease_refresh + self._loop_wait > time.time():
return False
if self._lease and not self._client.lease_keepalive(self._lease, retry=retry):
if self._lease and not self._client.lease_keepalive(self._lease, retry):
self._lease = None
ret = not self._lease
if ret:
self._lease = self._client.lease_grant(self._ttl, retry=retry)
self._lease = self._client.lease_grant(self._ttl, retry)
self._last_lease_refresh = time.time()
return ret
@@ -751,8 +723,23 @@ class Etcd3(AbstractEtcd):
history = history and TimelineHistory.from_node(history['mod_revision'], history['value'])
# get last know leader lsn and slots
status = nodes.get(self._STATUS) or nodes.get(self._LEADER_OPTIME)
status = Status.from_node(status and status['value'])
status = nodes.get(self._STATUS)
if status:
try:
status = json.loads(status['value'])
last_lsn = status.get(self._OPTIME)
slots = status.get('slots')
except Exception:
slots = last_lsn = None
else:
last_lsn = nodes.get(self._LEADER_OPTIME)
last_lsn = last_lsn and last_lsn['value']
slots = None
try:
last_lsn = int(last_lsn or '')
except Exception:
last_lsn = 0
# get list of members
members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
@@ -783,7 +770,7 @@ class Etcd3(AbstractEtcd):
except Exception:
failsafe = None
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
def _cluster_loader(self, path: str) -> Cluster:
nodes = {node['key'][len(path):]: node
@@ -925,10 +912,11 @@ class Etcd3(AbstractEtcd):
return self.retry(self._client.put, self.initialize_path, sysid, create_revision='0' if create_new else None)
@catch_etcd_errors
def _delete_leader(self, leader: Leader) -> bool:
fields = build_range_request(self.leader_path)
compare = {'key': fields['key'], 'target': 'VALUE', 'value': base64_encode(self._name)}
return bool(self._client.txn(compare, {'request_delete_range': fields}))
def _delete_leader(self) -> bool:
cluster = self.cluster
if cluster and isinstance(cluster.leader, Leader) and cluster.leader.name == self._name:
return self._client.deleterange(self.leader_path, mod_revision=cluster.leader.version)
return True
@catch_etcd_errors
def cancel_initialization(self) -> bool:
+22 -32
View File
@@ -19,7 +19,7 @@ from urllib3.exceptions import HTTPError
from threading import Condition, Lock, Thread
from typing import Any, Callable, Collection, Dict, List, Optional, Tuple, Type, Union, TYPE_CHECKING
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, Status, SyncState, \
from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState, \
TimelineHistory, CITUS_COORDINATOR_GROUP_ID, citus_group_re
from ..exceptions import DCSError
from ..utils import deep_compare, iter_response_objects, keepalive_socket_options, \
@@ -771,7 +771,8 @@ class Kubernetes(AbstractDCS):
except k8s_config.ConfigException:
k8s_config.load_kube_config(context=config.get('context', 'kind-kind'))
self.__ips: List[str] = [] if self._ctl else [config.get('pod_ip', '')]
pod_ip = config.get('pod_ip')
self.__ips: List[str] = [] if self._ctl or not isinstance(pod_ip, str) else [pod_ip]
self.__ports: List[K8sObject] = []
ports: List[Dict[str, Any]] = config.get('ports', [{}])
for p in ports:
@@ -887,8 +888,18 @@ class Kubernetes(AbstractDCS):
self._leader_resource_version = metadata.resource_version if metadata else None
annotations: Dict[str, str] = metadata and metadata.annotations or {}
# get last known leader lsn and slots
status = Status.from_node(annotations)
# get last known leader lsn
try:
last_lsn = int(annotations.get(self._OPTIME, ''))
except Exception:
last_lsn = 0
# get permanent slots state (confirmed_flush_lsn)
slots = annotations.get('slots')
try:
slots = json.loads(annotations.get('slots', ''))
except Exception:
slots = None
# get failsafe topology
try:
@@ -934,7 +945,7 @@ class Kubernetes(AbstractDCS):
metadata = sync and sync.metadata
sync = SyncState.from_node(metadata and metadata.resource_version, metadata and metadata.annotations)
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
def _cluster_loader(self, path: Dict[str, Any]) -> Cluster:
return self._cluster_from_nodes(path['group'], path['nodes'], path['pods'].values())
@@ -1058,27 +1069,6 @@ class Kubernetes(AbstractDCS):
def _patch_or_create(self, name: str, annotations: Dict[str, Any],
resource_version: Optional[str] = None, patch: bool = False,
retry: Optional[Callable[..., Any]] = None, ips: Optional[List[str]] = None) -> K8sObject:
"""Patch or create K8s object, Endpoint or ConfigMap.
:param name: the name of the object.
:param annotations: mapping of annotations that we want to create/update.
:param resource_version: object should be updated only if the ``resource_version`` matches provided value.
:param patch: ``True`` if we know in advance that the object already exists and we should patch it.
:param retry: a callable that will take care of retries
:param ips: IP address that we want to put to the subsets of the endpoint. Could have following values:
* ``None`` - when we don't need to touch subset;
* ``[]`` - to set subsets to the empty list, when :meth:`delete_leader` method is called;
* ``['ip.add.re.ss']`` - when we want to make sure that the subsets of the leader endpoint
contains the IP address of the leader, that we get from the ``kubernetes.pod_ip``;
* ``['']`` - when we want to make sure that the subsets of the leader endpoint contains the IP
address of the leader, but ``kubernetes.pod_ip`` configuration is missing. In this case we will
try to take the IP address of the Pod which name matches ``name`` from the config file.
:returns: the new :class:`V1Endpoints` or :class:`V1ConfigMap` object, that was created or updated.
"""
metadata = {'namespace': self._namespace, 'name': name, 'labels': self._labels, 'annotations': annotations}
if patch or resource_version:
if resource_version is not None:
@@ -1091,10 +1081,9 @@ class Kubernetes(AbstractDCS):
metadata['annotations'] = {k: v for k, v in annotations.items() if v is not None}
metadata = k8s_client.V1ObjectMeta(**metadata)
if self._api.use_endpoints:
if ips is not None and self._api.use_endpoints:
endpoints = {'metadata': metadata}
if ips is not None:
self._map_subsets(endpoints, ips)
self._map_subsets(endpoints, ips)
body = k8s_client.V1Endpoints(**endpoints)
else:
body = k8s_client.V1ConfigMap(metadata=metadata)
@@ -1243,10 +1232,11 @@ class Kubernetes(AbstractDCS):
else:
annotations['acquireTime'] = self._leader_observed_record.get('acquireTime') or now
annotations['transitions'] = str(transitions)
ips: Optional[List[str]] = [] if self._api.use_endpoints else None
try:
ret = bool(self._patch_or_create(self.leader_path, annotations,
self._leader_resource_version, retry=self.retry, ips=self.__ips))
self._leader_resource_version, retry=self.retry, ips=ips))
except k8s_client.rest.ApiException as e:
if e.status == 409 and self._leader_resource_version: # Conflict in resource_version
# Terminate watchers, it could be a sign that K8s API is in a failed state
@@ -1322,11 +1312,11 @@ class Kubernetes(AbstractDCS):
if cluster and cluster.config and cluster.config.version else None
return self.patch_or_create_config({self._INITIALIZE: sysid}, resource_version)
def _delete_leader(self, leader: Leader) -> bool:
def _delete_leader(self) -> bool:
"""Unused"""
raise NotImplementedError # pragma: no cover
def delete_leader(self, leader: Optional[Leader], last_lsn: Optional[int] = None) -> bool:
def delete_leader(self, last_lsn: Optional[int] = None) -> bool:
ret = False
kind = self._kinds.get(self.leader_path)
if kind and (kind.metadata.annotations or {}).get(self._LEADER) == self._name:
+20 -6
View File
@@ -12,8 +12,7 @@ from pysyncobj.transport import TCPTransport, CONNECTION_STATE
from pysyncobj.utility import TcpUtility
from typing import Any, Callable, Collection, Dict, List, Optional, Set, Union, TYPE_CHECKING
from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, Status, SyncState, \
TimelineHistory, citus_group_re
from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory, citus_group_re
from ..exceptions import DCSError
from ..utils import validate_directory
if TYPE_CHECKING: # pragma: no cover
@@ -344,8 +343,23 @@ class Raft(AbstractDCS):
history = history and TimelineHistory.from_node(history['index'], history['value'])
# get last know leader lsn and slots
status = nodes.get(self._STATUS) or nodes.get(self._LEADER_OPTIME)
status = Status.from_node(status and status['value'])
status = nodes.get(self._STATUS)
if status:
try:
status = json.loads(status['value'])
last_lsn = status.get(self._OPTIME)
slots = status.get('slots')
except Exception:
slots = last_lsn = None
else:
last_lsn = nodes.get(self._LEADER_OPTIME)
last_lsn = last_lsn and last_lsn['value']
slots = None
try:
last_lsn = int(last_lsn or '')
except Exception:
last_lsn = 0
# get list of members
members = [self.member(k, n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
@@ -373,7 +387,7 @@ class Raft(AbstractDCS):
except Exception:
failsafe = None
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
def _cluster_loader(self, path: str) -> Cluster:
response = self._sync_obj.get(path, recursive=True)
@@ -432,7 +446,7 @@ class Raft(AbstractDCS):
def initialize(self, create_new: bool = True, sysid: str = '') -> bool:
return self._sync_obj.set(self.initialize_path, sysid, prevExist=(not create_new)) is not False
def _delete_leader(self, leader: Leader) -> bool:
def _delete_leader(self) -> bool:
return self._sync_obj.delete(self.leader_path, prevValue=self._name, timeout=1)
def cancel_initialization(self) -> bool:
+107 -39
View File
@@ -12,8 +12,7 @@ from kazoo.retry import RetryFailedError
from kazoo.security import ACL, make_acl
from typing import Any, Callable, Dict, List, Optional, Union, Tuple, TYPE_CHECKING
from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, Status, SyncState, \
TimelineHistory, citus_group_re
from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory, citus_group_re
from ..exceptions import DCSError
from ..utils import deep_compare
if TYPE_CHECKING: # pragma: no cover
@@ -116,7 +115,10 @@ class ZooKeeper(AbstractDCS):
timeout=config['ttl'], connection_retry=KazooRetry(max_delay=1, max_tries=-1,
sleep_func=time.sleep), command_retry=KazooRetry(max_delay=1, max_tries=-1,
deadline=config['retry_timeout'], sleep_func=time.sleep), **kwargs)
self._client.add_listener(self.session_listener)
self._fetch_cluster: bool = True
self._fetch_status: bool = True
self.__last_member_data: Optional[Dict[str, Any]] = None
self._orig_kazoo_connect = self._client._connection._connect
@@ -139,9 +141,18 @@ class ZooKeeper(AbstractDCS):
ret = self._orig_kazoo_connect(*args)
return max(self.loop_wait - 2, 2) * 1000, ret[1]
def _watcher(self, event: WatchedEvent) -> None:
if event.state != KazooState.CONNECTED or event.path.startswith(self.client_path('')):
self.event.set()
def session_listener(self, state: str) -> None:
if state in [KazooState.SUSPENDED, KazooState.LOST]:
self.cluster_watcher(None)
def status_watcher(self, event: Optional[WatchedEvent]) -> None:
self._fetch_status = True
self.event.set()
def cluster_watcher(self, event: Optional[WatchedEvent]) -> None:
self._fetch_cluster = True
if not event or event.state != KazooState.CONNECTED or event.path.startswith(self.client_path('')):
self.status_watcher(event)
def reload_config(self, config: Union['Config', Dict[str, Any]]) -> None:
self.set_retry_timeout(config['retry_timeout'])
@@ -189,89 +200,138 @@ class ZooKeeper(AbstractDCS):
except NoNodeError:
return None
def get_status(self, path: str, leader: Optional[Leader]) -> Status:
status = self.get_node(path + self._STATUS)
if not status:
status = self.get_node(path + self._LEADER_OPTIME)
return Status.from_node(status and status[0])
def get_status(self, path: str, leader: Optional[Leader]) -> Tuple[int, Optional[Dict[str, int]]]:
watch = self.status_watcher if not leader or leader.name != self._name else None
status = self.get_node(path + self._STATUS, watch)
if status:
try:
status = json.loads(status[0])
last_lsn = status.get(self._OPTIME)
slots = status.get('slots')
except Exception:
slots = last_lsn = None
else:
last_lsn = self.get_node(path + self._LEADER_OPTIME, watch)
last_lsn = last_lsn and last_lsn[0]
slots = None
try:
last_lsn = int(last_lsn or '')
except Exception:
last_lsn = 0
self._fetch_status = False
return last_lsn, slots
@staticmethod
def member(name: str, value: str, znode: ZnodeStat) -> Member:
return Member.from_node(znode.version, name, znode.ephemeralOwner, value)
def get_children(self, key: str) -> List[str]:
def get_children(self, key: str, watch: Optional[Callable[[WatchedEvent], None]] = None) -> List[str]:
try:
return self._client.get_children(key)
return self._client.get_children(key, watch)
except NoNodeError:
return []
def load_members(self, path: str) -> List[Member]:
members: List[Member] = []
for member in self.get_children(path + self._MEMBERS):
for member in self.get_children(path + self._MEMBERS, self.cluster_watcher):
data = self.get_node(path + self._MEMBERS + member)
if data is not None:
members.append(self.member(member, *data))
return members
def _cluster_loader(self, path: str) -> Cluster:
nodes = set(self.get_children(path))
self._fetch_cluster = False
self.event.clear()
nodes = set(self.get_children(path, self.cluster_watcher))
if not nodes:
self._fetch_cluster = True
# get initialize flag
initialize = (self.get_node(path + self._INITIALIZE) or [None])[0] if self._INITIALIZE in nodes else None
# get global dynamic configuration
config = self.get_node(path + self._CONFIG, watch=self._watcher) if self._CONFIG in nodes else None
config = self.get_node(path + self._CONFIG, watch=self.cluster_watcher) if self._CONFIG in nodes else None
config = config and ClusterConfig.from_node(config[1].version, config[0], config[1].mzxid)
# get timeline history
history = self.get_node(path + self._HISTORY) if self._HISTORY in nodes else None
history = self.get_node(path + self._HISTORY, watch=self.cluster_watcher) if self._HISTORY in nodes else None
history = history and TimelineHistory.from_node(history[1].mzxid, history[0])
# get synchronization state
sync = self.get_node(path + self._SYNC) if self._SYNC in nodes else None
sync = self.get_node(path + self._SYNC, watch=self.cluster_watcher) if self._SYNC in nodes else None
sync = SyncState.from_node(sync and sync[1].version, sync and sync[0])
# get list of members
members = self.load_members(path) if self._MEMBERS[:-1] in nodes else []
# get leader
leader = self.get_node(path + self._LEADER, watch=self._watcher) if self._LEADER in nodes else None
leader = self.get_node(path + self._LEADER) if self._LEADER in nodes else None
if leader:
member = Member(-1, leader[0], None, {})
member = ([m for m in members if m.name == leader[0]] or [member])[0]
leader = Leader(leader[1].version, leader[1].ephemeralOwner, member)
self._fetch_cluster = member.version == -1
# get last known leader lsn and slots
status = self.get_status(path, leader)
last_lsn, slots = self.get_status(path, leader)
# failover key
failover = self.get_node(path + self._FAILOVER) if self._FAILOVER in nodes else None
failover = self.get_node(path + self._FAILOVER, watch=self.cluster_watcher) if self._FAILOVER in nodes else None
failover = failover and Failover.from_node(failover[1].version, failover[0])
# get failsafe topology
failsafe = self.get_node(path + self._FAILSAFE) if self._FAILSAFE in nodes else None
failsafe = self.get_node(path + self._FAILSAFE, watch=self.cluster_watcher) if self._FAILSAFE in nodes else None
try:
failsafe = json.loads(failsafe[0]) if failsafe else None
except Exception:
failsafe = None
return Cluster(initialize, config, leader, status, members, failover, sync, history, failsafe)
return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
def _citus_cluster_loader(self, path: str) -> Dict[int, Cluster]:
fetch_cluster = False
ret: Dict[int, Cluster] = {}
for node in self.get_children(path):
for node in self.get_children(path, self.cluster_watcher):
if citus_group_re.match(node):
ret[int(node)] = self._cluster_loader(path + node + '/')
fetch_cluster = fetch_cluster or self._fetch_cluster
self._fetch_cluster = fetch_cluster
return ret
def _load_cluster(
self, path: str, loader: Callable[[str], Union[Cluster, Dict[int, Cluster]]]
) -> Union[Cluster, Dict[int, Cluster]]:
try:
return self._client.retry(loader, path)
except Exception:
logger.exception('get_cluster')
raise ZooKeeperError('ZooKeeper in not responding properly')
cluster = self.cluster if path == self._base_path + '/' else None
if self._fetch_cluster or cluster is None:
try:
cluster = self._client.retry(loader, path)
except Exception:
logger.exception('get_cluster')
self.cluster_watcher(None)
raise ZooKeeperError('ZooKeeper in not responding properly')
# The /status ZNode was updated or doesn't exist
elif self._fetch_status and not self._fetch_cluster or not cluster.last_lsn \
or cluster.has_permanent_logical_slots(self._name, False) and not cluster.slots:
# If current node is the leader just clear the event without fetching anything (we are updating the /status)
if cluster.leader and cluster.leader.name == self._name:
self.event.clear()
else:
try:
last_lsn, slots = self.get_status(self.client_path(''), cluster.leader)
self.event.clear()
new_cluster: List[Any] = list(cluster)
new_cluster[3] = last_lsn
new_cluster[8] = slots
cluster = Cluster(*new_cluster)
except Exception:
pass
return cluster
def _bypass_caches(self) -> None:
self._fetch_cluster = True
def _create(self, path: str, value: bytes, retry: bool = False, ephemeral: bool = False) -> bool:
try:
@@ -334,9 +394,20 @@ class ZooKeeper(AbstractDCS):
member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
member_data = self.__last_member_data or member and member.data
if member and member_data:
# We want delete the member ZNode if our session doesn't match with session id on our member key
if self._client.client_id is not None and member.session != self._client.client_id[0]:
logger.warning('Recreating the member ZNode due to ownership mismatch')
is_leader = data.get('role') in ('master', 'primary', 'standby_leader')
checkpoint_after_promote_changed = member_data.get('checkpoint_after_promote') \
!= data.get('checkpoint_after_promote')
state_running_changed = member_data.get('state') != data.get('state') \
and 'running' in (member_data.get('state'), data.get('state'))
tags_changed = not deep_compare(member_data.get('tags', {}), data.get('tags', {}))
# We want delete the member ZNode if:
# - our session doesn't match with session id on our member key; or
# - we want to notify leader if some important fields in the member key changed; or
# - if we are the leader and want to notify replicas about checkpoint_after_promote;
if self._client.client_id is not None and member.session != self._client.client_id[0] \
or is_leader and checkpoint_after_promote_changed \
or not is_leader and (state_running_changed or tags_changed):
try:
self._client.delete_async(self.member_path).get(timeout=1)
except NoNodeError:
@@ -402,7 +473,7 @@ class ZooKeeper(AbstractDCS):
return False
return True
def _delete_leader(self, leader: Leader) -> bool:
def _delete_leader(self) -> bool:
self._client.restart()
return True
@@ -435,10 +506,7 @@ class ZooKeeper(AbstractDCS):
return self.set_sync_state_value("{}", version) is not False
def watch(self, leader_version: Optional[int], timeout: float) -> bool:
if leader_version:
timeout += 0.5
try:
return super(ZooKeeper, self).watch(leader_version, timeout)
finally:
self.event.clear()
ret = super(ZooKeeper, self).watch(leader_version, timeout + 0.5)
if ret and not self._fetch_status:
self._fetch_cluster = True
return ret or self._fetch_cluster
+144 -244
View File
@@ -14,34 +14,37 @@ from . import psycopg
from .__main__ import Patroni
from .async_executor import AsyncExecutor, CriticalTask
from .collections import CaseInsensitiveSet
from .dcs import AbstractDCS, Cluster, Leader, Member, RemoteMember, Status, slot_name_from_member_name
from .dcs import AbstractDCS, Cluster, Leader, Member, RemoteMember
from .exceptions import DCSError, PostgresConnectionException, PatroniFatalException
from .postgresql.callback_executor import CallbackAction
from .postgresql.misc import postgres_version_to_int
from .postgresql.postmaster import PostmasterProcess
from .postgresql.rewind import Rewind
from .tags import Tags
from .utils import polling_loop, tzutc
logger = logging.getLogger(__name__)
class _MemberStatus(Tags, NamedTuple('_MemberStatus',
[('member', Member),
('reachable', bool),
('in_recovery', Optional[bool]),
('wal_position', int),
('data', Dict[str, Any])])):
"""Node status distilled from API response.
class _MemberStatus(NamedTuple):
"""Node status distilled from API response:
Consists of the following fields:
:ivar member: :class:`~patroni.dcs.Member` object of the node.
:ivar reachable: ``False`` if the node is not reachable or is not responding with correct JSON.
:ivar in_recovery: ``False`` if the node is running as a primary (`if pg_is_in_recovery() == true`).
:ivar wal_position: maximum value of ``replayed_location`` or ``received_location`` from JSON.
:ivar data: the whole JSON response for future usage.
member - dcs.Member object of the node
reachable - `!False` if the node is not reachable or is not responding with correct JSON
in_recovery - `!True` if pg_is_in_recovery() == true
dcs_last_seen - timestamp from JSON of last succesful communication with DCS
timeline - timeline value from JSON
wal_position - maximum value of `replayed_location` or `received_location` from JSON
tags - dictionary with values of different tags (i.e. nofailover)
watchdog_failed - indicates that watchdog is required by configuration but not available or failed
"""
member: Member
reachable: bool
in_recovery: Optional[bool]
dcs_last_seen: int
timeline: int
wal_position: int
tags: Dict[str, Any]
watchdog_failed: bool
@classmethod
def from_api_response(cls, member: Member, json: Dict[str, Any]) -> '_MemberStatus':
@@ -54,34 +57,21 @@ class _MemberStatus(Tags, NamedTuple('_MemberStatus',
wal: Dict[str, Any] = json.get('wal') or json['xlog']
# abuse difference in primary/replica response format
in_recovery = not (bool(wal.get('location')) or json.get('role') in ('master', 'primary'))
timeline = json.get('timeline', 0)
dcs_last_seen = json.get('dcs_last_seen', 0)
lsn = int(in_recovery and max(wal.get('received_location', 0), wal.get('replayed_location', 0)))
return cls(member, True, in_recovery, lsn, json)
@property
def tags(self) -> Dict[str, Any]:
"""Dictionary with values of different tags (i.e. nofailover)."""
return self.data.get('tags', {})
@property
def timeline(self) -> int:
"""Timeline value from JSON."""
return self.data.get('timeline', 0)
@property
def watchdog_failed(self) -> bool:
"""Indicates that watchdog is required by configuration but not available or failed."""
return self.data.get('watchdog_failed', False)
return cls(member, True, in_recovery, dcs_last_seen, timeline, lsn,
json.get('tags', {}), json.get('watchdog_failed', False))
@classmethod
def unknown(cls, member: Member) -> '_MemberStatus':
"""Create a new class instance with empty or null values."""
return cls(member, False, None, 0, {})
return cls(member, False, None, 0, 0, 0, {}, False)
def failover_limitation(self) -> Optional[str]:
"""Returns reason why this node can't promote or None if everything is ok."""
if not self.reachable:
return 'not reachable'
if self.nofailover:
if self.tags.get('nofailover', False):
return 'not allowed to promote'
if self.watchdog_failed:
return 'not watchdog capable'
@@ -123,8 +113,7 @@ class Failsafe(object):
leader = self.leader
if leader:
# We rely on the strict order of fields in the namedtuple
status = Status(cluster.status.last_lsn, leader.member.data['slots'])
cluster = Cluster(*cluster[0:2], leader, status, *cluster[4:])
cluster = Cluster(*cluster[0:2], leader, *cluster[3:8], leader.member.data['slots'], *cluster[9:])
return cluster
def is_active(self) -> bool:
@@ -158,8 +147,8 @@ class Ha(object):
self.cluster = Cluster.empty()
self.global_config = self.patroni.config.get_global_config(None)
self.old_cluster = Cluster.empty()
self._leader_expiry = 0
self._leader_expiry_lock = RLock()
self._is_leader = False
self._is_leader_lock = RLock()
self._failsafe = Failsafe(patroni.dcs)
self._was_paused = False
self._leader_timeline = None
@@ -204,38 +193,12 @@ class Ha(object):
return self.global_config.is_standby_cluster
def is_leader(self) -> bool:
""":returns: `True` if the current node is the leader, based on expiration set when it last held the key."""
with self._leader_expiry_lock:
return self._leader_expiry > time.time()
with self._is_leader_lock:
return self._is_leader > time.time()
def set_is_leader(self, value: bool) -> None:
"""Update the current node's view of it's own leadership status.
Will update the expiry timestamp to match the dcs ttl if setting leadership to true,
otherwise will set the expiry to the past to immediately invalidate.
:param value: is the current node the leader.
"""
with self._leader_expiry_lock:
self._leader_expiry = time.time() + self.dcs.ttl if value else 0
def sync_mode_is_active(self) -> bool:
"""Check whether synchronous replication is requested and already active.
:returns: ``True`` if the primary already put its name into the ``/sync`` in DCS.
"""
return self.is_synchronous_mode() and not self.cluster.sync.is_empty
def _get_failover_action_name(self) -> str:
"""Return the currently requested manual failover action name or the default ``failover``.
:returns: :class:`str` representing the manually requested action (``manual failover`` if no leader
is specified in the ``/failover`` in DCS, ``switchover`` otherwise) or ``failover`` if
``/failover`` is empty.
"""
if not self.cluster.failover:
return 'failover'
return 'switchover' if self.cluster.failover.leader else 'manual failover'
with self._is_leader_lock:
self._is_leader = time.time() + self.dcs.ttl if value else 0
def load_cluster_from_dcs(self) -> None:
cluster = self.dcs.get_cluster()
@@ -272,32 +235,12 @@ class Ha(object):
ret[self.state_handler.name] = self.patroni.api.connection_string
return ret
def update_lock(self, update_status: bool = False) -> bool:
"""Update the leader lock in DCS.
.. note::
After successful update of the leader key the :meth:`AbstractDCS.update_leader` method could also
optionally update the ``/status`` and ``/failsafe`` keys.
The ``/status`` key contains the last known LSN on the leader node and the last known state
of permanent replication slots including permanent physical replication slot for the leader.
Last, but not least, this method calls a :meth:`Watchdog.keepalive` method after the leader key
was successfully updated.
:param update_status: ``True`` if we also need to update the ``/status`` key in DCS, otherwise ``False``.
:returns: ``True`` if the leader key was successfully updated and we can continue to run postgres
as a ``primary`` or as a ``standby_leader``, otherwise ``False``.
"""
def update_lock(self, write_leader_optime: bool = False) -> bool:
last_lsn = slots = None
if update_status:
if write_leader_optime:
try:
last_lsn = self.state_handler.last_operation()
slots = self.cluster.filter_permanent_slots(
{**self.state_handler.slots(), slot_name_from_member_name(self.state_handler.name): last_lsn},
self.is_standby_cluster(),
self.state_handler.major_version)
slots = self.state_handler.slots()
except Exception:
logger.exception('Exception when called state_handler.last_operation()')
if TYPE_CHECKING: # pragma: no cover
@@ -529,7 +472,7 @@ class Ha(object):
if timeout == 0:
# We are requested to prefer failing over to restarting primary. But see first if there
# is anyone to fail over to.
if self.is_failover_possible():
if self.is_failover_possible(self.cluster.members):
self.watchdog.disable()
logger.info("Primary crashed. Failing over.")
self.demote('immediate')
@@ -604,9 +547,7 @@ class Ha(object):
"""
# The standby leader or when there is no standby leader we want to follow
# the remote member, except when there is no standby leader in pause.
if self.is_standby_cluster() \
and (cluster.leader and cluster.leader.name and cluster.leader.name == self.state_handler.name
or cluster.is_unlocked() and not self.is_paused()):
if self.is_standby_cluster() and (self.has_lock(False) or self.cluster.is_unlocked() and not self.is_paused()):
node_to_follow = self.get_remote_member()
# If replicatefrom tag is set, try to follow the node mentioned there, otherwise, follow the leader.
elif self.patroni.replicatefrom and self.patroni.replicatefrom != self.state_handler.name:
@@ -631,7 +572,7 @@ class Ha(object):
if refresh:
self.load_cluster_from_dcs()
is_leader = self.state_handler.is_primary()
is_leader = self.state_handler.is_leader()
node_to_follow = self._get_node_to_follow(self.cluster)
@@ -827,7 +768,7 @@ class Ha(object):
"""
if not self.is_paused():
if not self.watchdog.is_running and not self.watchdog.activate():
if self.state_handler.is_primary():
if self.state_handler.is_leader():
self.demote('immediate')
return 'Demoting self because watchdog could not be activated'
else:
@@ -843,7 +784,7 @@ class Ha(object):
self._async_response.reset()
return 'Promotion cancelled because the pre-promote script failed'
if self.state_handler.is_primary():
if self.state_handler.is_leader():
# Inform the state handler about its primary role.
# It may be unaware of it if postgres is promoted manually.
self.state_handler.set_role('master')
@@ -893,8 +834,6 @@ class Ha(object):
return _MemberStatus.unknown(member)
def fetch_nodes_statuses(self, members: List[Member]) -> List[_MemberStatus]:
if not members:
return []
pool = ThreadPool(len(members))
results = pool.map(self.fetch_node_status, members) # Run API calls on members in parallel
pool.close()
@@ -920,26 +859,6 @@ class Ha(object):
return False
def check_failsafe_topology(self) -> bool:
"""Check whether we could continue to run as a primary by calling all members from the failsafe topology.
.. note::
If the ``/failsafe`` key contains invalid data or if the ``name`` of our node is missing in
the ``/failsafe`` key, we immediately give up and return ``False``.
We send the JSON document in the POST request with the following fields:
* ``name`` - the name of our node;
* ``conn_url`` - connection URL to the postgres, which is reachable from other nodes;
* ``api_url`` - connection URL to Patroni REST API on this node reachable from other nodes;
* ``slots`` - a :class:`dict` with replication slots that exist on the leader node, including the primary
itself with the last known LSN, because there could be a permanent physical slot on standby nodes.
Standby nodes are using information from the ``slots`` dict to advance position of permanent
replication slots while DCS is not accessible in order to avoid indefinite growth of ``pg_wal``.
:returns: ``True`` if all members from the ``/failsafe`` topology agree that this node could continue to
run as a ``primary``, or ``False`` if some of standby nodes are not accessible or don't agree.
"""
failsafe = self.dcs.failsafe
if not isinstance(failsafe, dict) or self.state_handler.name not in failsafe:
return False
@@ -949,10 +868,7 @@ class Ha(object):
'api_url': self.patroni.api.connection_string,
}
try:
data['slots'] = {
**self.state_handler.slots(),
slot_name_from_member_name(self.state_handler.name): self.state_handler.last_operation()
}
data['slots'] = self.state_handler.slots()
except Exception:
logger.exception('Exception when called state_handler.slots()')
members = [RemoteMember(name, {'api_url': url})
@@ -997,60 +913,51 @@ class Ha(object):
# Prepare list of nodes to run check against
members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
for st in self.fetch_nodes_statuses(members):
if st.failover_limitation() is None:
if st.in_recovery is False:
logger.warning('Primary (%s) is still alive', st.member.name)
return False
if my_wal_position < st.wal_position:
logger.info('Wal position of %s is ahead of my wal position', st.member.name)
# In synchronous mode the former leader might be still accessible and even be ahead of us.
# We should not disqualify himself from the leader race in such a situation.
if not self.sync_mode_is_active() or not self.cluster.sync.leader_matches(st.member.name):
if members:
for st in self.fetch_nodes_statuses(members):
if st.failover_limitation() is None:
if st.in_recovery is False:
logger.warning('Primary (%s) is still alive', st.member.name)
return False
logger.info('Ignoring the former leader being ahead of us')
if my_wal_position == st.wal_position and self.patroni.failover_priority < st.failover_priority:
# There's a higher priority non-lagging replica
logger.info(
'%s has equally tolerable WAL position and priority %s, while this node has priority %s',
st.member.name,
st.failover_priority,
self.patroni.failover_priority,
)
return False
if my_wal_position < st.wal_position:
logger.info('Wal position of %s is ahead of my wal position', st.member.name)
# In synchronous mode the former leader might be still accessible and even be ahead of us.
# We should not disqualify himself from the leader race in such a situation.
if not self.is_synchronous_mode() or self.cluster.sync.is_empty\
or not self.cluster.sync.leader_matches(st.member.name):
return False
logger.info('Ignoring the former leader being ahead of us')
return True
def is_failover_possible(self, *, cluster_lsn: int = 0, exclude_failover_candidate: bool = False) -> bool:
"""Checks whether any of the cluster members is allowed to promote and is healthy enough for that.
def is_failover_possible(self, members: List[Member], check_synchronous: Optional[bool] = True,
cluster_lsn: Optional[int] = 0) -> bool:
"""Checks whether one of the members from the list can possibly win the leader race.
:param cluster_lsn: to calculate replication lag and exclude member if it is lagging.
:param exclude_failover_candidate: if ``True``, exclude :attr:`failover.candidate` from the members
list against which the failover possibility checks are run.
:returns: `True` if there are members eligible to become the new leader.
:param members: list of members to check
:param check_synchronous: consider only members that are known to be listed in /sync key when sync replication.
:param cluster_lsn: to calculate replication lag and exclude member if it is laggin
:returns: `True` if there are members eligible to be the new leader
"""
candidates = self.get_failover_candidates(exclude_failover_candidate)
action = self._get_failover_action_name()
if self.is_synchronous_mode() and self.cluster.failover and self.cluster.failover.candidate and not candidates:
logger.warning('%s candidate=%s does not match with sync_standbys=%s',
action.title(), self.cluster.failover.candidate, self.cluster.sync.sync_standby)
elif not candidates:
logger.warning('%s: candidates list is empty', action)
ret = False
cluster_timeline = self.cluster.timeline
for st in self.fetch_nodes_statuses(candidates):
not_allowed_reason = st.failover_limitation()
if not_allowed_reason:
logger.info('Member %s is %s', st.member.name, not_allowed_reason)
elif cluster_lsn and st.wal_position < cluster_lsn or \
not cluster_lsn and self.is_lagging(st.wal_position):
logger.info('Member %s exceeds maximum replication lag', st.member.name)
elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline):
logger.info('Timeline %s of member %s is behind the cluster timeline %s',
st.timeline, st.member.name, cluster_timeline)
else:
ret = True
members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
if check_synchronous and self.is_synchronous_mode() and not self.cluster.sync.is_empty:
members = [m for m in members if self.cluster.sync.matches(m.name)]
if members:
for st in self.fetch_nodes_statuses(members):
not_allowed_reason = st.failover_limitation()
if not_allowed_reason:
logger.info('Member %s is %s', st.member.name, not_allowed_reason)
elif cluster_lsn and st.wal_position < cluster_lsn or\
not cluster_lsn and self.is_lagging(st.wal_position):
logger.info('Member %s exceeds maximum replication lag', st.member.name)
elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline):
logger.info('Timeline %s of member %s is behind the cluster timeline %s',
st.timeline, st.member.name, cluster_timeline)
else:
ret = True
else:
logger.warning('manual failover: members list is empty')
return ret
def manual_failover_process_no_leader(self) -> Optional[bool]:
@@ -1062,18 +969,15 @@ class Ha(object):
failover = self.cluster.failover
if TYPE_CHECKING: # pragma: no cover
assert failover is not None
action = self._get_failover_action_name()
if failover.candidate: # manual failover/switchover to specific member
if failover.candidate == self.state_handler.name: # manual failover/switchover to me
if failover.candidate: # manual failover to specific member
if failover.candidate == self.state_handler.name: # manual failover to me
return True
elif self.is_paused():
# Remove failover key if the node to failover has terminated to avoid waiting for it indefinitely
# In order to avoid attempts to delete this key from all nodes only the primary is allowed to do it.
if not self.cluster.get_member(failover.candidate, fallback_to_leader=False)\
and self.state_handler.is_primary():
logger.warning("%s: removing failover key because failover candidate is not running", action)
and self.state_handler.is_leader():
logger.warning("manual failover: removing failover key because failover candidate is not running")
self.dcs.manual_failover('', '', version=failover.version)
return None
return False
@@ -1089,21 +993,22 @@ class Ha(object):
st = self.fetch_node_status(member)
not_allowed_reason = st.failover_limitation()
if not_allowed_reason is None: # node is healthy
logger.info('%s: to %s, i am %s', action, st.member.name, self.state_handler.name)
logger.info('manual failover: to %s, i am %s', st.member.name, self.state_handler.name)
return False
# we wanted to failover/switchover to specific member but it is not healthy
logger.warning('%s: member %s is %s', action, st.member.name, not_allowed_reason)
# we wanted to failover to specific member but it is not healthy
logger.warning('manual failover: member %s is %s', st.member.name, not_allowed_reason)
# at this point we should consider all members as a candidates for failover/switchover
# at this point we should consider all members as a candidates for failover
# i.e. we assume that failover.candidate is None
elif self.is_paused():
return False
# try to pick some other members for switchover and check that they are healthy
# try to pick some other members to failover and check that they are healthy
if failover.leader:
if self.state_handler.name == failover.leader: # I was the leader
# exclude desired member which is unhealthy if it was specified
if self.is_failover_possible(exclude_failover_candidate=bool(failover.candidate)):
# exclude me and desired member which is unhealthy (failover.candidate can be None)
members = [m for m in self.cluster.members if m.name not in (failover.candidate, failover.leader)]
if self.is_failover_possible(members): # check that there are healthy members
return False
else: # I was the leader and it looks like currently I am the only healthy member
return True
@@ -1131,7 +1036,7 @@ class Ha(object):
if ret is not None: # continue if we just deleted the stale failover key as a leader
return ret
if self.state_handler.is_primary():
if self.state_handler.is_leader():
if self.is_paused():
# in pause leader is the healthiest only when no initialize or sysid matches with initialize!
return not self.cluster.initialize or self.state_handler.sysid == self.cluster.initialize
@@ -1157,8 +1062,8 @@ class Ha(object):
if self.cluster.failover:
# When doing a switchover in synchronous mode only synchronous nodes and former leader are allowed to race
if self.cluster.failover.leader and self.sync_mode_is_active() \
and not self.cluster.sync.matches(self.state_handler.name, True):
if self.is_synchronous_mode() and self.cluster.failover.leader and \
not self.cluster.sync.is_empty and not self.cluster.sync.matches(self.state_handler.name, True):
return False
return self.manual_failover_process_no_leader() or False
@@ -1179,7 +1084,7 @@ class Ha(object):
all_known_members += self.cluster.members
# When in sync mode, only last known primary and sync standby are allowed to promote automatically.
if self.sync_mode_is_active():
if self.is_synchronous_mode() and not self.cluster.sync.is_empty:
if not self.cluster.sync.matches(self.state_handler.name, True):
return False
# pick between synchronous candidates so we minimize unnecessary failovers/demotions
@@ -1192,7 +1097,7 @@ class Ha(object):
def _delete_leader(self, last_lsn: Optional[int] = None) -> None:
self.set_is_leader(False)
self.dcs.delete_leader(self.cluster.leader, last_lsn)
self.dcs.delete_leader(last_lsn)
self.dcs.reset_cluster()
def release_leader_key_voluntarily(self, last_lsn: Optional[int] = None) -> None:
@@ -1232,7 +1137,9 @@ class Ha(object):
# It could happen if Postgres is still archiving the backlog of WAL files.
# If we know that there are replicas that received the shutdown checkpoint
# location, we can remove the leader key and allow them to start leader race.
if self.is_failover_possible(cluster_lsn=checkpoint_location):
# for a manual failover/switchover with a candidate, we should check the requested candidate only
if self.is_failover_possible(self.get_failover_candidates(), cluster_lsn=checkpoint_location):
self.state_handler.set_role('demoted')
with self._async_executor:
self.release_leader_key_voluntarily(checkpoint_location)
@@ -1322,35 +1229,36 @@ class Ha(object):
:returns: action message if demote was initiated, None if no action was taken"""
failover = self.cluster.failover
# if there is no failover key or
# I am holding the lock but am not primary = I am the standby leader,
# then do nothing
if not failover or (self.is_paused() and not self.state_handler.is_primary()):
if not failover or (self.is_paused() and not self.state_handler.is_leader()):
return
action = self._get_failover_action_name()
bare_action = action.replace('manual ', '')
# it is not the time for the scheduled switchover yet, do nothing
if (failover.scheduled_at and not
self.should_run_scheduled_action(bare_action, failover.scheduled_at, lambda:
self.should_run_scheduled_action("failover", failover.scheduled_at, lambda:
self.dcs.manual_failover('', '', version=failover.version))):
return
if not failover.leader or failover.leader == self.state_handler.name:
if not failover.candidate or failover.candidate != self.state_handler.name:
if not failover.candidate and self.is_paused():
logger.warning('%s is possible only to a specific candidate in a paused state', action.title())
elif self.is_failover_possible():
ret = self._async_executor.try_run_async(f'{action}: demote', self.demote, ('graceful',))
return ret or f'{action}: demoting myself'
logger.warning('Failover is possible only to a specific candidate in a paused state')
else:
logger.warning('%s: no healthy members found, %s is not possible',
action, bare_action)
if self.is_synchronous_mode():
members = self.get_failover_candidates(check_sync=True)
if failover.candidate and not members:
logger.warning('Failover candidate=%s does not match with sync_standbys=%s',
failover.candidate, self.cluster.sync.sync_standby)
else:
members = self.get_failover_candidates()
if self.is_failover_possible(members, False): # check that there are healthy members
ret = self._async_executor.try_run_async('manual failover: demote', self.demote, ('graceful',))
return ret or 'manual failover: demoting myself'
else:
logger.warning('manual failover: no healthy members found, failover is not possible')
else:
logger.warning('%s: I am already the leader, no need to %s', action, bare_action)
logger.warning('manual failover: I am already the leader, no need to failover')
else:
logger.warning('%s: leader name does not match: %s != %s', action, failover.leader, self.state_handler.name)
logger.warning('manual failover: leader name does not match: %s != %s',
failover.leader, self.state_handler.name)
logger.info('Cleaning up failover key')
self.dcs.manual_failover('', '', version=failover.version)
@@ -1399,7 +1307,7 @@ class Ha(object):
def process_healthy_cluster(self) -> str:
if self.has_lock():
if self.is_paused() and not self.state_handler.is_primary():
if self.is_paused() and not self.state_handler.is_leader():
if self.cluster.failover and self.cluster.failover.candidate == self.state_handler.name:
return 'waiting to become primary after promote...'
@@ -1407,7 +1315,6 @@ class Ha(object):
self._delete_leader()
return 'removed leader lock because postgres is not running as primary'
# update lock to avoid split-brain
if self.update_lock(True):
msg = self.process_manual_failover_from_leader()
if msg is not None:
@@ -1432,7 +1339,7 @@ class Ha(object):
else:
# Either there is no connection to DCS or someone else acquired the lock
logger.error('failed to update leader lock')
if self.state_handler.is_primary():
if self.state_handler.is_leader():
if self.is_paused():
return 'continue to run as primary after failing to update leader lock in DCS'
self.demote('immediate-nolock')
@@ -1605,7 +1512,7 @@ class Ha(object):
if self.has_lock() and self.update_lock():
if self._async_executor.scheduled_action == 'doing crash recovery in a single user mode':
time_left = self.global_config.primary_start_timeout - (time.time() - self._crash_recovery_started)
if time_left <= 0 and self.is_failover_possible():
if time_left <= 0 and self.is_failover_possible(self.cluster.members):
logger.info("Demoting self because crash recovery is taking too long")
self.state_handler.cancellable.cancel(True)
self.demote('immediate')
@@ -1669,7 +1576,7 @@ class Ha(object):
self.cancel_initialization()
if result is None:
if not self.state_handler.is_primary():
if not self.state_handler.is_leader():
return 'waiting for end of recovery after bootstrap'
self.state_handler.set_role('master')
@@ -1716,7 +1623,7 @@ class Ha(object):
time_left = timeout - self.state_handler.time_in_state()
if time_left <= 0:
if self.is_failover_possible():
if self.is_failover_possible(self.cluster.members):
logger.info("Demoting self because primary startup is taking too long")
self.demote('immediate')
return 'stopped PostgreSQL because of startup timeout'
@@ -1853,7 +1760,7 @@ class Ha(object):
elif self.cluster.is_unlocked() and not self.is_paused():
# "bootstrap", but data directory is not empty
if not self.state_handler.cb_called and self.state_handler.is_running() \
and not self.state_handler.is_primary():
and not self.state_handler.is_leader():
self._join_aborted = True
logger.error('No initialize key in DCS and PostgreSQL is running as replica, aborting start')
logger.error('Please first start Patroni on the node running as primary')
@@ -1896,7 +1803,7 @@ class Ha(object):
create_slots = self._sync_replication_slots(False)
if not self.state_handler.cb_called:
if not is_promoting and not self.state_handler.is_primary():
if not is_promoting and not self.state_handler.is_leader():
self._rewind.trigger_check_diverged_lsn()
self.state_handler.call_nowait(CallbackAction.ON_START)
@@ -1921,7 +1828,7 @@ class Ha(object):
def _handle_dcs_error(self) -> str:
if not self.is_paused() and self.state_handler.is_running():
if self.state_handler.is_primary():
if self.state_handler.is_leader():
if self.is_failsafe_mode() and self.check_failsafe_topology():
self.set_is_leader(True)
self._failsafe.set_is_active(time.time())
@@ -1997,8 +1904,9 @@ class Ha(object):
# If we know that there are replicas that received the shutdown checkpoint
# location, we can remove the leader key and allow them to start leader race.
if self.is_failover_possible(cluster_lsn=checkpoint_location):
self.dcs.delete_leader(self.cluster.leader, checkpoint_location)
# for a manual failover/switchover with a candidate, we should check the requested candidate only
if self.is_failover_possible(self.get_failover_candidates(), cluster_lsn=checkpoint_location):
self.dcs.delete_leader(checkpoint_location)
status['deleted'] = True
else:
self.dcs.write_leader_optime(checkpoint_location)
@@ -2015,7 +1923,7 @@ class Ha(object):
if not self.state_handler.is_running():
if self.is_leader() and not status['deleted']:
checkpoint_location = self.state_handler.latest_checkpoint_location()
self.dcs.delete_leader(self.cluster.leader, checkpoint_location)
self.dcs.delete_leader(checkpoint_location)
self.touch_member()
else:
# XXX: what about when Patroni is started as the wrong user that has access to the watchdog device
@@ -2059,31 +1967,23 @@ class Ha(object):
name = member.name if member else 'remote_member:{}'.format(uuid.uuid1())
return RemoteMember(name, data)
def get_failover_candidates(self, exclude_failover_candidate: bool) -> List[Member]:
"""Return a list of candidates for either manual or automatic failover.
def get_failover_candidates(self, check_sync: bool = False) -> List[Member]:
"""Return list of candidates for either manual or automatic failover.
Exclude non-sync members when in synchronous mode, the current node (its checks are always performed earlier)
and the candidate if required. If failover candidate exclusion is not requested and a candidate is specified
in the /failover key, return the candidate only.
The result is further evaluated in the caller :func:`Ha.is_failover_possible` to check if any member is actually
healthy enough and is allowed to poromote.
Mainly used to later be passed to ``Ha.is_failover_possible()``.
:param exclude_failover_candidate: if ``True``, exclude :attr:`failover.candidate` from the candidates.
:param check_sync: if ``True``, also check against the sync key members
:returns: a list of :class:`Member` ojects or an empty list if there is no candidate available.
:returns: a list of ``Member`` ojects or an empty list if there is no candidate available
"""
failover = self.cluster.failover
exclude = [self.state_handler.name] + ([failover.candidate] if failover and exclude_failover_candidate else [])
def is_eligible(node: Member) -> bool:
# in synchronous mode we allow failover (not switchover!) to async node
if self.sync_mode_is_active() and not self.cluster.sync.matches(node.name)\
and not (failover and not failover.leader):
return False
# Don't spend time on "nofailover" nodes checking.
# We also don't need nodes which we can't query with the api in the list.
return node.name not in exclude and \
not node.nofailover and bool(node.api_url) and \
(not failover or not failover.candidate or node.name == failover.candidate)
return list(filter(is_eligible, self.cluster.members))
if check_sync:
# TODO: allow manual failover (=no leader specified) to async node
# every sync_standby or the candidate specified if is in sync_standbys
return [m for m in self.cluster.members
if self.cluster.sync.matches(m.name)
and (not failover or not failover.candidate or m.name == failover.candidate)]
else:
# every member or the candidate specified
return [m for m in self.cluster.members
if not failover or not failover.candidate or m.name == failover.candidate]
+79 -93
View File
@@ -18,7 +18,7 @@ from .bootstrap import Bootstrap
from .callback_executor import CallbackAction, CallbackExecutor
from .cancellable import CancellableSubprocess
from .config import ConfigHandler, mtime
from .connection import ConnectionPool, get_connection_cursor
from .connection import Connection, get_connection_cursor
from .citus import CitusHandler
from .misc import parse_history, parse_lsn, postgres_major_version_to_int
from .postmaster import PostmasterProcess
@@ -27,7 +27,7 @@ from .sync import SyncHandler
from .. import psycopg
from ..async_executor import CriticalTask
from ..collections import CaseInsensitiveSet
from ..dcs import Cluster, Leader, Member, SLOT_ADVANCE_AVAILABLE_VERSION
from ..dcs import Cluster, Leader, Member
from ..exceptions import PostgresConnectionException
from ..utils import Retry, RetryFailedError, polling_loop, data_directory_is_empty, parse_int
@@ -57,8 +57,8 @@ class Postgresql(object):
TL_LSN = ("CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
"ELSE ('x' || pg_catalog.substr(pg_catalog.pg_{0}file_name("
"pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, " # primary timeline
"CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 ELSE "
"pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}{2}_{1}(), '0/0')::bigint END, " # wal(_flush)?_lsn
"CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
"ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), '0/0')::bigint END, " # write_lsn
"pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), '0/0')::bigint, "
"pg_catalog.pg_{0}_{1}_diff(COALESCE(pg_catalog.pg_last_{0}_receive_{1}(), '0/0'), '0/0')::bigint, "
"pg_catalog.pg_is_in_recovery() AND pg_catalog.pg_is_{0}_replay_paused()")
@@ -79,8 +79,7 @@ class Postgresql(object):
self.set_state('stopped')
self._pending_restart = False
self.connection_pool = ConnectionPool()
self._connection = self.connection_pool.get('heartbeat')
self._connection = Connection()
self.citus_handler = CitusHandler(self, config.get('citus'))
self.config = ConfigHandler(self, config)
self.config.check_directories()
@@ -112,7 +111,7 @@ class Postgresql(object):
self._state_entry_timestamp = 0
self._cluster_info_state = {}
self._has_permanent_slots = True
self._has_permanent_logical_slots = True
self._enforce_hot_standby_feedback = False
self._cached_replica_timeline = None
@@ -128,7 +127,7 @@ class Postgresql(object):
# we know that PostgreSQL is accepting connections and can read some GUC's from pg_settings
self.config.load_current_server_parameters()
self.set_role('master' if self.is_primary() else 'replica')
self.set_role('master' if self.is_leader() else 'replica')
hba_saved = self.config.replace_pg_hba()
ident_saved = self.config.replace_pg_ident()
@@ -171,11 +170,6 @@ class Postgresql(object):
def wal_name(self) -> str:
return 'wal' if self._major_version >= 100000 else 'xlog'
@property
def wal_flush(self) -> str:
"""For PostgreSQL 9.6 onwards we want to use pg_current_wal_flush_lsn()/pg_current_xlog_flush_location()."""
return '_flush' if self._major_version >= 90600 else ''
@property
def lsn_name(self) -> str:
return 'lsn' if self._major_version >= 100000 else 'location'
@@ -185,11 +179,6 @@ class Postgresql(object):
""":returns: `True` if Postgres version supports more than one synchronous node."""
return self._major_version >= 90600
@property
def can_advance_slots(self) -> bool:
"""``True`` if :attr:``major_version`` is greater than 110000."""
return self.major_version >= SLOT_ADVANCE_AVAILABLE_VERSION
@property
def cluster_info_query(self) -> str:
"""Returns the monitoring query with a fixed number of fields.
@@ -224,9 +213,8 @@ class Postgresql(object):
extra = ("pg_catalog.current_setting('restore_command')" if self._major_version >= 120000 else "NULL") +\
", " + ("(SELECT pg_catalog.json_agg(s.*) FROM (SELECT slot_name, slot_type as type, datoid::bigint, "
"plugin, catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint"
" AS confirmed_flush_lsn, pg_catalog.pg_wal_lsn_diff(restart_lsn, '0/0')::bigint"
" AS restart_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"
if self._has_permanent_slots and self.can_advance_slots else "NULL") + extra
" AS confirmed_flush_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"
if self._has_permanent_logical_slots and self._major_version >= 110000 else "NULL") + extra
extra = (", CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END,"
" slot_name, conninfo, status, {0} FROM pg_catalog.pg_stat_get_wal_receiver()").format(extra)
if self.role == 'standby_leader':
@@ -236,7 +224,7 @@ class Postgresql(object):
else:
extra = "0, NULL, NULL, NULL, NULL, NULL, NULL" + extra
return ("SELECT " + self.TL_LSN + ", {3}").format(self.wal_name, self.lsn_name, self.wal_flush, extra)
return ("SELECT " + self.TL_LSN + ", {2}").format(self.wal_name, self.lsn_name, extra)
@property
def available_gucs(self) -> CaseInsensitiveSet:
@@ -295,7 +283,7 @@ class Postgresql(object):
:returns: 'ok' if PostgreSQL is up, 'reject' if starting up, 'no_resopnse' if not up."""
r = self.connection_pool.conn_kwargs
r = self.config.local_connect_kwargs
cmd = [self.pgcommand('pg_isready'), '-p', r['port'], '-d', self._database]
# Host is not set if we are connecting via default unix socket
@@ -346,50 +334,40 @@ class Postgresql(object):
def connection(self) -> Union['connection3', 'Connection3[Any]']:
return self._connection.get()
def _query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
"""Execute *sql* query with *params* and optionally return results.
def set_connection_kwargs(self, kwargs: Dict[str, Any]) -> None:
self._connection.set_conn_kwargs(kwargs.copy())
self.citus_handler.set_conn_kwargs(kwargs.copy())
:param sql: SQL statement to execute.
:param params: parameters to pass.
:returns: a query response as a list of tuples if there is any.
:raises:
:exc:`~psycopg.Error` if had issues while executing *sql*.
:exc:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database.
:exc:`~patroni.utils.RetryFailedError`: if it was detected that connection/query failed due to PostgreSQL
restart.
"""
def _query(self, sql: str, *params: Any) -> Union['Cursor[Any]', 'cursor']:
"""We are always using the same cursor, therefore this method is not thread-safe!!!
You can call it from different threads only if you are holding explicit `AsyncExecutor` lock,
because the main thread is always holding this lock when running HA cycle."""
cursor = None
try:
return self._connection.query(sql, *params)
except PostgresConnectionException as exc:
cursor = self._connection.cursor()
cursor.execute(sql.encode('utf-8'), params or None)
return cursor
except psycopg.Error as e:
if cursor and cursor.connection.closed == 0:
# When connected via unix socket, psycopg2 can't recoginze 'connection lost'
# and leaves `_cursor_holder.connection.closed == 0`, but psycopg2.OperationalError
# is still raised (what is correct). It doesn't make sense to continiue with existing
# connection and we will close it, to avoid its reuse by the `cursor` method.
if isinstance(e, psycopg.OperationalError):
self._connection.close()
else:
raise e
if self.state == 'restarting':
raise RetryFailedError('cluster is being restarted') from exc
raise
raise RetryFailedError('cluster is being restarted')
raise PostgresConnectionException('connection problems')
def query(self, sql: str, *params: Any, retry: bool = True) -> List[Tuple[Any, ...]]:
"""Execute *sql* query with *params* and optionally return results.
:param sql: SQL statement to execute.
:param params: parameters to pass.
:param retry: whether the query should be retried upon failure or given up immediately.
:returns: a query response as a list of tuples if there is any.
:raises:
:exc:`~psycopg.Error` if had issues while executing *sql*.
:exc:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database.
:exc:`~patroni.utils.RetryFailedError`: if it was detected that connection/query failed due to PostgreSQL
restart or if retry deadline was exceeded.
"""
if not retry:
return self._query(sql, *params)
def query(self, sql: str, *args: Any, **kwargs: Any) -> Union['Cursor[Any]', 'cursor']:
if not kwargs.get('retry', True):
return self._query(sql, *args)
try:
return self.retry(self._query, sql, *params)
except RetryFailedError as exc:
raise PostgresConnectionException(str(exc)) from exc
return self.retry(self._query, sql, *args)
except RetryFailedError as e:
raise PostgresConnectionException(str(e))
def pg_control_exists(self) -> bool:
return os.path.isfile(self._pg_control)
@@ -452,31 +430,28 @@ class Postgresql(object):
if self._global_config.is_standby_cluster:
# Standby cluster can't have logical replication slots, and we don't need to enforce hot_standby_feedback
self._has_permanent_logical_slots = False
self.set_enforce_hot_standby_feedback(False)
elif cluster and cluster.config and cluster.config.modify_version:
self._has_permanent_logical_slots =\
cluster.has_permanent_logical_slots(self.name, nofailover, self.major_version)
if cluster and cluster.config and cluster.config.modify_version:
# We want to enable hot_standby_feedback if the replica is supposed
# to have a logical slot or in case if it is the cascading replica.
self.set_enforce_hot_standby_feedback(not self._global_config.is_standby_cluster and self.can_advance_slots
and cluster.should_enforce_hot_standby_feedback(self.name,
nofailover))
self._has_permanent_slots = cluster.has_permanent_slots(
my_name=self.name,
is_standby_cluster=self._global_config.is_standby_cluster,
nofailover=nofailover,
major_version=self.major_version)
self.set_enforce_hot_standby_feedback(
self._has_permanent_logical_slots
or cluster.should_enforce_hot_standby_feedback(self.name, nofailover, self.major_version))
def _cluster_info_state_get(self, name: str) -> Optional[Any]:
if not self._cluster_info_state:
try:
result = self._is_leader_retry(self._query, self.cluster_info_query)[0]
result = self._is_leader_retry(self._query, self.cluster_info_query).fetchone()
cluster_info_state = dict(zip(['timeline', 'wal_position', 'replayed_location',
'received_location', 'replay_paused', 'pg_control_timeline',
'received_tli', 'slot_name', 'conninfo', 'receiver_state',
'restore_command', 'slots', 'synchronous_commit',
'synchronous_standby_names', 'pg_stat_replication'], result))
if self._has_permanent_slots and self.can_advance_slots:
if self._has_permanent_logical_slots:
cluster_info_state['slots'] =\
self.slots_handler.process_permanent_slots(cluster_info_state['slots'])
self._cluster_info_state = cluster_info_state
@@ -520,14 +495,14 @@ class Postgresql(object):
""":returns: a result set of 'SELECT * FROM pg_stat_replication'."""
return self._cluster_info_state_get('pg_stat_replication') or []
def replication_state_from_parameters(self, is_primary: bool, receiver_state: Optional[str],
def replication_state_from_parameters(self, is_leader: bool, receiver_state: Optional[str],
restore_command: Optional[str]) -> Optional[str]:
"""Figure out the replication state from input parameters.
.. note::
This method could be only called when Postgres is up, running and queries are successfuly executed.
:is_primary: `True` is postgres is not running in recovery
:is_leader: `True` is postgres is not running in recovery
:receiver_state: value from `pg_stat_get_wal_receiver.state` or None if Postgres is older than 9.6
:restore_command: value of ``restore_command`` GUC for PostgreSQL 12+ or
`postgresql.recovery_conf.restore_command` if it is set in Patroni configuration
@@ -536,7 +511,7 @@ class Postgresql(object):
- 'streaming' if replica is streaming according to the `pg_stat_wal_receiver` view;
- 'in archive recovery' if replica isn't streaming and there is a `restore_command`
"""
if self._major_version >= 90600 and not is_primary:
if self._major_version >= 90600 and not is_leader:
if receiver_state == 'streaming':
return 'streaming'
# For Postgres older than 12 we get `restore_command` from Patroni config, otherwise we check GUC
@@ -551,11 +526,11 @@ class Postgresql(object):
:returns: ``streaming``, ``in archive recovery``, or ``None``
"""
return self.replication_state_from_parameters(self.is_primary(),
return self.replication_state_from_parameters(self.is_leader(),
self._cluster_info_state_get('receiver_state'),
self._cluster_info_state_get('restore_command'))
def is_primary(self) -> bool:
def is_leader(self) -> bool:
try:
return bool(self._cluster_info_state_get('timeline'))
except PostgresConnectionException:
@@ -711,7 +686,7 @@ class Postgresql(object):
# the former node, otherwise, we might get a stalled one
# after kill -9, which would report incorrect data to
# patroni.
self.connection_pool.close()
self._connection.close()
if self.is_running():
logger.error('Cannot start PostgreSQL because one is already running.')
@@ -782,7 +757,7 @@ class Postgresql(object):
def checkpoint(self, connect_kwargs: Optional[Dict[str, Any]] = None,
timeout: Optional[float] = None) -> Optional[str]:
check_not_is_in_recovery = connect_kwargs is not None
connect_kwargs = connect_kwargs or self.connection_pool.conn_kwargs
connect_kwargs = connect_kwargs or self.config.local_connect_kwargs
for p in ['connect_timeout', 'options']:
connect_kwargs.pop(p, None)
if timeout:
@@ -912,10 +887,11 @@ class Postgresql(object):
def _wait_for_connection_close(self, postmaster: PostmasterProcess) -> None:
try:
while postmaster.is_running(): # Need a timeout here?
self._connection.query("SELECT 1")
time.sleep(STOP_POLLING_INTERVAL)
except (psycopg.Error, PostgresConnectionException):
with self.connection().cursor() as cur:
while postmaster.is_running(): # Need a timeout here?
cur.execute("SELECT 1")
time.sleep(STOP_POLLING_INTERVAL)
except psycopg.Error:
pass
def reload(self, block_callbacks: bool = False) -> bool:
@@ -1202,9 +1178,9 @@ class Postgresql(object):
return ret
@staticmethod
def _wal_position(is_primary: bool, wal_position: int,
def _wal_position(is_leader: bool, wal_position: int,
received_location: Optional[int], replayed_location: Optional[int]) -> int:
return wal_position if is_primary else max(received_location or 0, replayed_location or 0)
return wal_position if is_leader else max(received_location or 0, replayed_location or 0)
def timeline_wal_position(self) -> Tuple[int, int, Optional[int]]:
# This method could be called from different threads (simultaneously with some other `_query` calls).
@@ -1216,21 +1192,31 @@ class Postgresql(object):
received_location = self.received_location()
pg_control_timeline = self._cluster_info_state_get('pg_control_timeline')
else:
timeline, wal_position, replayed_location, received_location, _, pg_control_timeline = \
self._query(self.cluster_info_query)[0][:6]
with self.connection().cursor() as cursor:
cursor.execute(self.cluster_info_query.encode('utf-8'))
row = cursor.fetchone()
if TYPE_CHECKING: # pragma: no cover
assert row is not None
(timeline, wal_position, replayed_location, received_location, _, pg_control_timeline) = row[:6]
wal_position = self._wal_position(bool(timeline), wal_position, received_location, replayed_location)
return timeline, wal_position, pg_control_timeline
return (timeline, wal_position, pg_control_timeline)
def postmaster_start_time(self) -> Optional[str]:
try:
sql = "SELECT " + self.POSTMASTER_START_TIME
return self.query(sql, retry=current_thread().ident == self.__thread_ident)[0][0].isoformat(sep=' ')
query = "SELECT " + self.POSTMASTER_START_TIME
if current_thread().ident == self.__thread_ident:
row = self.query(query).fetchone()
else:
with self.connection().cursor() as cursor:
cursor.execute(query)
row = cursor.fetchone()
return row[0].isoformat(sep=' ') if row else None
except psycopg.Error:
return None
def last_operation(self) -> int:
return self._wal_position(self.is_primary(), self._cluster_info_state_get('wal_position') or 0,
return self._wal_position(self.is_leader(), self._cluster_info_state_get('wal_position') or 0,
self.received_location(), self.replayed_location())
def configure_server_parameters(self) -> None:
+1 -42
View File
@@ -151,47 +151,9 @@ class Bootstrap(object):
os.unlink(trigger_file)
def _custom_bootstrap(self, config: Any) -> bool:
"""Bootstrap a fresh Patroni cluster using a custom method provided by the user.
:param config: configuration used for running a custom bootstrap method. It comes from the Patroni YAML file,
so it is expected to be a :class:`dict`.
.. note::
*config* must contain a ``command`` key, which value is the command or script to perform the custom
bootstrap procedure. The exit code of the ``command`` dictates if the bootstrap succeeded or failed.
When calling ``command``, Patroni will pass the following arguments to the ``command`` call:
* ``--scope``: contains the value of ``scope`` configuration;
* ``--data_dir``: contains the value of the ``postgresql.data_dir`` configuration.
You can avoid that behavior by filling the optional key ``no_params`` with the value ``False`` in the
configuration file, which will instruct Patroni to not pass these parameters to the ``command`` call.
Besides that, a couple more keys are supported in *config*, but optional:
* ``keep_existing_recovery_conf``: if ``True``, instruct Patroni to not remove the existing
``recovery.conf`` (PostgreSQL <= 11), to not discard recovery parameters from the configuration
(PostgreSQL >= 12), and to not remove the files ``recovery.signal`` or ``standby.signal``
(PostgreSQL >= 12). This is specially useful when you are restoring backups through tools like
pgBackRest and Barman, in which case they generated the appropriate recovery settings for you;
* ``recovery_conf``: a section containing a map, where each key is the name of a recovery related
setting, and the value is the value of the corresponding setting.
Any key/value other than the ones that were described above will be interpreted as additional arguments for
the ``command`` call. They will all be added to the call in the format ``--key=value``.
:returns: ``True`` if the bootstrap was successful, i.e. the execution of the custom ``command`` from *config*
exited with code ``0``, ``False`` otherwise.
"""
self._postgresql.set_state('running custom bootstrap script')
params = [] if config.get('no_params') else ['--scope=' + self._postgresql.scope,
'--datadir=' + self._postgresql.data_dir]
# Add custom parameters specified by the user
reserved_args = {'no_params', 'keep_existing_recovery_conf', 'recovery_conf', 'scope', 'datadir'}
for arg, val in config.items():
if arg not in reserved_args:
params.append(f"--{arg}={val}")
try:
logger.info('Running custom bootstrap script: %s', config['command'])
if self._postgresql.cancellable.call(shlex.split(config['command']) + params) != 0:
@@ -214,7 +176,7 @@ class Bootstrap(object):
"""
cmd = config.get('post_bootstrap') or config.get('post_init')
if cmd:
r = self._postgresql.connection_pool.conn_kwargs
r = self._postgresql.config.local_connect_kwargs
connstring = self._postgresql.config.format_dsn(r, True)
if 'host' not in r:
# https://www.postgresql.org/docs/current/static/libpq-pgpass.html
@@ -438,9 +400,6 @@ BEGIN
END;$$""".format(f, quote_ident(rewind['username'], postgresql.connection()))
postgresql.query(sql)
if config.get('users'):
logger.warning('User creation via "bootstrap.users" will be removed in v4.0.0')
for name, value in (config.get('users') or {}).items():
if all(name != a.get('username') for a in (superuser, replication, rewind)):
self.create_or_update_role(name, value.get('password'), value.get('options', []))
+2 -7
View File
@@ -1,9 +1,8 @@
import logging
import sys
from enum import Enum
from threading import Condition, Thread
from typing import Any, Dict, List
from typing import List
from .cancellable import CancellableExecutor, CancellableSubprocess
@@ -31,9 +30,7 @@ class OnReloadExecutor(CancellableSubprocess):
self.cancel(kill=True)
self._kill_children()
with self._lock:
started = self._start_process(cmd, close_fds=True)
if started and self._process is not None:
Thread(target=self._process.wait).start()
self._start_process(cmd, close_fds=True)
class CallbackExecutor(CancellableExecutor, Thread):
@@ -54,8 +51,6 @@ class CallbackExecutor(CancellableExecutor, Thread):
If it couldn't be killed we wait until it finishes.
:param cmd: command to be executed"""
kwargs: Dict[str, Any] = {'stacklevel': 3} if sys.version_info >= (3, 8) else {}
logger.debug('CallbackExecutor.call(%s)', cmd, **kwargs)
if cmd[-3] == CallbackAction.ON_RELOAD:
return self._on_reload_executor.call_nowait(cmd)
+24 -17
View File
@@ -6,10 +6,13 @@ from threading import Condition, Event, Thread
from urllib.parse import urlparse
from typing import Any, Dict, List, Optional, Union, Tuple, TYPE_CHECKING
from .connection import Connection
from ..dcs import CITUS_COORDINATOR_GROUP_ID, Cluster
from ..psycopg import connect, quote_ident
if TYPE_CHECKING: # pragma: no cover
from psycopg import Cursor
from psycopg2 import cursor
from . import Postgresql
CITUS_SLOT_NAME_RE = re.compile(r'^citus_shard_(move|split)_slot(_[1-9][0-9]*){2,3}$')
@@ -70,10 +73,7 @@ class CitusHandler(Thread):
self.daemon = True
self._postgresql = postgresql
self._config = config
if config:
self._connection = postgresql.connection_pool.get(
'citus', {'dbname': config['database'],
'options': '-c statement_timeout=0 -c idle_in_transaction_session_timeout=0'})
self._connection = Connection()
self._pg_dist_node: Dict[int, PgDistNode] = {} # Cache of pg_dist_node: {groupid: PgDistNode()}
self._tasks: List[PgDistNode] = [] # Requests to change pg_dist_node, every task is a `PgDistNode`
self._in_flight: Optional[PgDistNode] = None # Reference to the `PgDistNode` being changed in a transaction
@@ -93,6 +93,12 @@ class CitusHandler(Thread):
def is_worker(self) -> bool:
return self.is_enabled() and not self.is_coordinator()
def set_conn_kwargs(self, kwargs: Dict[str, Any]) -> None:
if isinstance(self._config, dict): # self.is_enabled():
kwargs.update({'dbname': self._config['database'],
'options': '-c statement_timeout=0 -c idle_in_transaction_session_timeout=0'})
self._connection.set_conn_kwargs(kwargs)
def schedule_cache_rebuild(self) -> None:
with self._condition:
self._schedule_load_pg_dist_node = True
@@ -103,10 +109,12 @@ class CitusHandler(Thread):
self._tasks[:] = []
self._in_flight = None
def query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
def query(self, sql: str, *params: Any) -> Union['Cursor[Any]', 'cursor']:
try:
logger.debug('query(%s, %s)', sql, params)
return self._connection.query(sql, *params)
cursor = self._connection.cursor()
cursor.execute(sql.encode('utf-8'), params or None)
return cursor
except Exception as e:
logger.error('Exception when executing query "%s", (%s): %r', sql, params, e)
self._connection.close()
@@ -124,13 +132,13 @@ class CitusHandler(Thread):
self._schedule_load_pg_dist_node = False
try:
rows = self.query("SELECT nodeid, groupid, nodename, nodeport, noderole"
" FROM pg_catalog.pg_dist_node WHERE noderole = 'primary'")
cursor = self.query("SELECT nodeid, groupid, nodename, nodeport, noderole"
" FROM pg_catalog.pg_dist_node WHERE noderole = 'primary'")
except Exception:
return False
with self._condition:
self._pg_dist_node = {r[1]: PgDistNode(r[1], r[2], r[3], 'after_promote', r[0]) for r in rows}
self._pg_dist_node = {r[1]: PgDistNode(r[1], r[2], r[3], 'after_promote', r[0]) for r in cursor}
return True
def sync_pg_dist_node(self, cluster: Cluster) -> None:
@@ -203,8 +211,10 @@ class CitusHandler(Thread):
self.query('SELECT pg_catalog.citus_update_node(%s, %s, %s, true, %s)',
task.nodeid, task.host, task.port, task.cooldown)
elif task.event != 'before_demote':
task.nodeid = self.query("SELECT pg_catalog.citus_add_node(%s, %s, %s, 'primary', 'default')",
task.host, task.port, task.group)[0][0]
row = self.query("SELECT pg_catalog.citus_add_node(%s, %s, %s, 'primary', 'default')",
task.host, task.port, task.group).fetchone()
if row is not None:
task.nodeid = row[0]
def process_task(self, task: PgDistNode) -> bool:
"""Updates a single row in `pg_dist_node` table, optionally in a transaction.
@@ -355,8 +365,8 @@ class CitusHandler(Thread):
if not isinstance(self._config, dict): # self.is_enabled()
return
conn_kwargs = {**self._postgresql.connection_pool.conn_kwargs,
'options': '-c synchronous_commit=local -c statement_timeout=0'}
conn_kwargs = self._postgresql.config.local_connect_kwargs
conn_kwargs['options'] = '-c synchronous_commit=local -c statement_timeout=0'
if self._config['database'] != self._postgresql.database:
conn = connect(**conn_kwargs)
try:
@@ -403,11 +413,8 @@ class CitusHandler(Thread):
# Resharding in Citus implemented using logical replication
parameters['wal_level'] = 'logical'
# Sometimes Citus needs to connect to the local postgres. We will do it the same way as Patroni does.
parameters['citus.local_hostname'] = self._postgresql.connection_pool.conn_kwargs.get('host', 'localhost')
def ignore_replication_slot(self, slot: Dict[str, str]) -> bool:
if isinstance(self._config, dict) and self._postgresql.is_primary() and\
if isinstance(self._config, dict) and self._postgresql.is_leader() and\
slot['type'] == 'logical' and slot['database'] == self._config['database']:
m = CITUS_SLOT_NAME_RE.match(slot['name'])
return bool(m and {'move': 'pgoutput', 'split': 'citus'}.get(m.group(1)) == slot['plugin'])
+23 -45
View File
@@ -244,10 +244,9 @@ class ConfigWriter(object):
self._fd.write(line)
self._fd.write('\n')
def writelines(self, lines: List[Optional[str]]) -> None:
def writelines(self, lines: List[str]) -> None:
for line in lines:
if isinstance(line, str):
self.writeline(line)
self.writeline(line)
@staticmethod
def escape(value: Any) -> str: # Escape (by doubling) any single quotes or backslashes in given string
@@ -978,32 +977,24 @@ class ConfigHandler(object):
return 'localhost' # connection via localhost is preferred
return listen_addresses[0].strip() # can't use localhost, take first address from listen_addresses
@property
def local_connect_kwargs(self) -> Dict[str, Any]:
ret = self._local_address.copy()
# add all of the other connection settings that are available
ret.update(self._superuser)
# if the "username" parameter is present, it actually needs to be "user"
# for connecting to PostgreSQL
if 'username' in self._superuser:
ret['user'] = self._superuser['username']
del ret['username']
# ensure certain Patroni configurations are available
ret.update({'dbname': self._postgresql.database,
'fallback_application_name': 'Patroni',
'connect_timeout': 3,
'options': '-c statement_timeout=2000'})
return ret
def resolve_connection_addresses(self) -> None:
"""Calculates and sets local and remote connection urls and options.
This method sets:
* :attr:`Postgresql.connection_string <patroni.postgresql.Postgresql.connection_string>` attribute, which
is later written to the member key in DCS as ``conn_url``.
* :attr:`ConfigHandler.local_replication_address` attribute, which is used for replication connections to
local postgres.
* :attr:`ConnectionPool.conn_kwargs <patroni.postgresql.connection.ConnectionPool.conn_kwargs>` attribute,
which is used for superuser connections to local postgres.
.. note::
If there is a valid directory in ``postgresql.parameters.unix_socket_directories`` in the Patroni
configuration and ``postgresql.use_unix_socket`` and/or ``postgresql.use_unix_socket_repl``
are set to ``True``, we respectively use unix sockets for superuser and replication connections
to local postgres.
If there is a requirement to use unix sockets, but nothing is set in the
``postgresql.parameters.unix_socket_directories``, we omit a ``host`` in connection parameters relying
on the ability of ``libpq`` to connect via some default unix socket directory.
If unix sockets are not requested we "switch" to TCP, prefering to use ``localhost`` if it is possible
to deduce that Postgres is listening on a local interface address.
Otherwise we just used the first address specified in the ``listen_addresses`` GUC.
"""
port = self._server_parameters['port']
tcp_local_address = self._get_tcp_local_address()
netloc = self._config.get('connect_address') or tcp_local_address + ':' + port
@@ -1016,25 +1007,12 @@ class ConfigHandler(object):
tcp_local_address = {'host': tcp_local_address, 'port': port}
self._local_address = unix_local_address if self._config.get('use_unix_socket') else tcp_local_address
self.local_replication_address = unix_local_address\
if self._config.get('use_unix_socket_repl') else tcp_local_address
self._postgresql.connection_string = uri('postgres', netloc, self._postgresql.database)
local_address = unix_local_address if self._config.get('use_unix_socket') else tcp_local_address
local_conn_kwargs = {
**local_address,
**self._superuser,
'dbname': self._postgresql.database,
'fallback_application_name': 'Patroni',
'connect_timeout': 3,
'options': '-c statement_timeout=2000'
}
# if the "username" parameter is present, it actually needs to be "user" for connecting to PostgreSQL
if 'username' in local_conn_kwargs:
local_conn_kwargs['user'] = local_conn_kwargs.pop('username')
# "notify" connection_pool about the "new" local connection address
self._postgresql.connection_pool.conn_kwargs = local_conn_kwargs
self._postgresql.set_connection_kwargs(self.local_connect_kwargs)
def _get_pg_settings(self, names: Collection[str]) -> Dict[Any, Tuple[Any, ...]]:
return {r[0]: r for r in self._postgresql.query(('SELECT name, setting, unit, vartype, context, sourcefile'
@@ -1147,10 +1125,10 @@ class ConfigHandler(object):
if self._postgresql.major_version >= 90500:
time.sleep(1)
try:
pending_restart = self._postgresql.query(
pending_restart = (self._postgresql.query(
'SELECT COUNT(*) FROM pg_catalog.pg_settings'
' WHERE pg_catalog.lower(name) != ALL(%s) AND pending_restart',
[n.lower() for n in self._RECOVERY_PARAMETERS])[0][0] > 0
[n.lower() for n in self._RECOVERY_PARAMETERS]).fetchone() or (0,))[0] > 0
self._postgresql.set_pending_restart(pending_restart)
except Exception as e:
logger.warning('Exception %r when running query', e)
+18 -126
View File
@@ -2,153 +2,45 @@ import logging
from contextlib import contextmanager
from threading import Lock
from typing import Any, Dict, Iterator, List, Optional, Union, Tuple, TYPE_CHECKING
from typing import Any, Dict, Iterator, Union, TYPE_CHECKING
if TYPE_CHECKING: # pragma: no cover
from psycopg import Connection, Cursor
from psycopg import Connection as Connection3, Cursor
from psycopg2 import connection, cursor
from .. import psycopg
from ..exceptions import PostgresConnectionException
logger = logging.getLogger(__name__)
class NamedConnection:
"""Helper class to manage ``psycopg`` connections from Patroni to PostgreSQL.
:ivar server_version: PostgreSQL version in integer format where we are connected to.
"""
class Connection(object):
server_version: int
def __init__(self, pool: 'ConnectionPool', name: str, kwargs_override: Optional[Dict[str, Any]]) -> None:
"""Create an instance of :class:`NamedConnection` class.
:param pool: reference to a :class:`ConnectionPool` object.
:param name: name of the connection.
:param kwargs_override: :class:`dict` object with connection parameters that should be
different from default values provided by connection *pool*.
"""
self._pool = pool
self._name = name
self._kwargs_override = kwargs_override or {}
self._lock = Lock() # used to make sure that only one connection to postgres is established
def __init__(self) -> None:
self._lock = Lock()
self._connection = None
self._cursor_holder = None
@property
def _conn_kwargs(self) -> Dict[str, Any]:
"""Connection parameters for this :class:`NamedConnection`."""
return {**self._pool.conn_kwargs, **self._kwargs_override, 'application_name': f'Patroni {self._name}'}
def set_conn_kwargs(self, conn_kwargs: Dict[str, Any]) -> None:
self._conn_kwargs = conn_kwargs
def get(self) -> Union['connection', 'Connection[Any]']:
"""Get ``psycopg``/``psycopg2`` connection object.
.. note::
Opens a new connection if necessary.
:returns: ``psycopg`` or ``psycopg2`` connection object.
"""
def get(self) -> Union['connection', 'Connection3[Any]']:
with self._lock:
if not self._connection or self._connection.closed != 0:
logger.info("establishing a new patroni %s connection to postgres", self._name)
self._connection = psycopg.connect(**self._conn_kwargs)
self.server_version = getattr(self._connection, 'server_version', 0)
return self._connection
def query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
"""Execute a query with parameters and optionally returns a response.
:param sql: SQL statement to execute.
:param params: parameters to pass.
:returns: a query response as a list of tuples if there is any.
:raises:
:exc:`~psycopg.Error` if had issues while executing *sql*.
:exc:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database.
"""
cursor = None
try:
with self.get().cursor() as cursor:
cursor.execute(sql.encode('utf-8'), params or None)
return cursor.fetchall() if cursor.rowcount and cursor.rowcount > 0 else []
except psycopg.Error as exc:
if cursor and cursor.connection.closed == 0:
# When connected via unix socket, psycopg2 can't recoginze 'connection lost' and leaves
# `self._connection.closed == 0`, but the generic exception is raised. It doesn't make
# sense to continue with existing connection and we will close it, to avoid its reuse.
if type(exc) in (psycopg.DatabaseError, psycopg.OperationalError):
self.close()
else:
raise exc
raise PostgresConnectionException('connection problems') from exc
def close(self, silent: bool = False) -> bool:
"""Close the psycopg connection to postgres.
:param silent: whether the method should not write logs.
:returns: ``True`` if ``psycopg`` connection was closed, ``False`` otherwise.``
"""
ret = False
if self._connection and self._connection.closed == 0:
self._connection.close()
if not silent:
logger.info("closed patroni %s connection to postgres", self._name)
ret = True
self._connection = None
return ret
class ConnectionPool:
"""Helper class to manage named connections from Patroni to PostgreSQL.
The instance keeps named :class:`NamedConnection` objects and parameters that must be used for new connections.
"""
def __init__(self) -> None:
"""Create an instance of :class:`ConnectionPool` class."""
self._lock = Lock()
self._connections: Dict[str, NamedConnection] = {}
self._conn_kwargs: Dict[str, Any] = {}
@property
def conn_kwargs(self) -> Dict[str, Any]:
"""Connection parameters that must be used for new ``psycopg`` connections."""
with self._lock:
return self._conn_kwargs.copy()
@conn_kwargs.setter
def conn_kwargs(self, value: Dict[str, Any]) -> None:
"""Set new connection parameters.
:param value: :class:`dict` object with connection parameters.
"""
with self._lock:
self._conn_kwargs = value
def get(self, name: str, kwargs_override: Optional[Dict[str, Any]] = None) -> NamedConnection:
"""Get a new named :class:`NamedConnection` object from the pool.
.. note::
Creates a new :class:`NamedConnection` object if it doesn't yet exist in the pool.
:param name: name of the connection.
:param kwargs_override: :class:`dict` object with connection parameters that should be
different from default values provided by :attr:`conn_kwargs`.
:returns: :class:`NamedConnection` object.
"""
with self._lock:
if name not in self._connections:
self._connections[name] = NamedConnection(self, name, kwargs_override)
return self._connections[name]
def cursor(self) -> Union['cursor', 'Cursor[Any]']:
if not self._cursor_holder or self._cursor_holder.closed or self._cursor_holder.connection.closed != 0:
logger.info("establishing a new patroni connection to the postgres cluster")
self._cursor_holder = self.get().cursor()
return self._cursor_holder
def close(self) -> None:
"""Close all named connections from Patroni to PostgreSQL registered in the pool."""
with self._lock:
if any(conn.close(True) for conn in self._connections.values()):
logger.info("closed patroni connections to postgres")
if self._connection and self._connection.closed == 0:
self._connection.close()
logger.info("closed patroni connection to the postgresql cluster")
self._cursor_holder = self._connection = None
@contextmanager
+1 -1
View File
@@ -280,7 +280,7 @@ class Rewind(object):
"""After promote issue a CHECKPOINT from a new thread and asynchronously check the result.
In case if CHECKPOINT failed, just check that timeline in pg_control was updated."""
if self._state != REWIND_STATUS.CHECKPOINT and self._postgresql.is_primary():
if self._state != REWIND_STATUS.CHECKPOINT and self._postgresql.is_leader():
with self._checkpoint_task_lock:
if self._checkpoint_task:
with self._checkpoint_task:
+51 -55
View File
@@ -186,7 +186,7 @@ class SlotsHandler:
self.pg_replslot_dir = os.path.join(self._postgresql.data_dir, 'pg_replslot')
self.schedule()
def _query(self, sql: str, *params: Any) -> List[Tuple[Any, ...]]:
def _query(self, sql: str, *params: Any) -> Union['cursor', 'Cursor[Any]']:
"""Helper method for :meth:`Postgresql.query`.
:param sql: SQL statement to execute.
@@ -231,16 +231,15 @@ class SlotsHandler:
ret: Dict[str, int] = {}
slots_dict: Dict[str, Dict[str, Any]] = {slot['slot_name']: slot for slot in slots or []}
for name, value in slots_dict.items():
if name in self._replication_slots:
if compare_slots(value, self._replication_slots[name], 'datoid'):
if value['type'] == 'logical':
ret[name] = value['confirmed_flush_lsn']
self._copy_items(value, self._replication_slots[name])
if slots_dict:
for name, value in slots_dict.items():
if name in self._replication_slots:
if compare_slots(value, self._replication_slots[name], 'datoid'):
if value['type'] == 'logical':
ret[name] = value['confirmed_flush_lsn']
self._copy_items(value, self._replication_slots[name])
else:
self._replication_slots[name]['restart_lsn'] = ret[name] = value['restart_lsn']
else:
self._schedule_load_slots = True
self._schedule_load_slots = True
# It could happen that the slot was deleted in the background, we want to detect this case
if any(name not in slots_dict for name in self._replication_slots.keys()):
@@ -261,19 +260,17 @@ class SlotsHandler:
"""
if self._postgresql.major_version >= 90400 and self._schedule_load_slots:
replication_slots: Dict[str, Dict[str, Any]] = {}
pg_wal_lsn_diff = f"pg_catalog.pg_{self._postgresql.wal_name}_{self._postgresql.lsn_name}_diff"
extra = f", catalog_xmin, {pg_wal_lsn_diff}(confirmed_flush_lsn, '0/0')::bigint" \
extra = ", catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint" \
if self._postgresql.major_version >= 100000 else ""
skip_temp_slots = ' WHERE NOT temporary' if self._postgresql.major_version >= 100000 else ''
for r in self._query(f"SELECT slot_name, slot_type, {pg_wal_lsn_diff}(restart_lsn, '0/0')::bigint, plugin,"
f" database, datoid{extra} FROM pg_catalog.pg_replication_slots{skip_temp_slots}"):
cursor = self._query(f'SELECT slot_name, slot_type, plugin, database, datoid'
f'{extra} FROM pg_catalog.pg_replication_slots{skip_temp_slots}')
for r in cursor:
value = {'type': r[1]}
if r[1] == 'logical':
value.update(plugin=r[3], database=r[4], datoid=r[5])
value.update(plugin=r[2], database=r[3], datoid=r[4])
if self._postgresql.major_version >= 100000:
value.update(catalog_xmin=r[6], confirmed_flush_lsn=r[7])
else:
value['restart_lsn'] = r[2]
value.update(catalog_xmin=r[5], confirmed_flush_lsn=r[6])
replication_slots[r[0]] = value
self._replication_slots = replication_slots
self._schedule_load_slots = False
@@ -311,13 +308,16 @@ class SlotsHandler:
``dropped`` is ``True`` if the slot was successfully dropped. If the slot was not found return
``False`` for both.
"""
rows = self._query(('WITH slots AS (SELECT slot_name, active'
' FROM pg_catalog.pg_replication_slots WHERE slot_name = %s),'
' dropped AS (SELECT pg_catalog.pg_drop_replication_slot(slot_name),'
' true AS dropped FROM slots WHERE not active) '
'SELECT active, COALESCE(dropped, false) FROM slots'
' FULL OUTER JOIN dropped ON true'), name)
return (rows[0][0], rows[0][1]) if rows else (False, False)
cursor = self._query(('WITH slots AS (SELECT slot_name, active'
' FROM pg_catalog.pg_replication_slots WHERE slot_name = %s),'
' dropped AS (SELECT pg_catalog.pg_drop_replication_slot(slot_name),'
' true AS dropped FROM slots WHERE not active) '
'SELECT active, COALESCE(dropped, false) FROM slots'
' FULL OUTER JOIN dropped ON true'), name)
row = cursor.fetchone()
if not row:
row = (False, False)
return row[0], row[1]
def _drop_incorrect_slots(self, cluster: Cluster, slots: Dict[str, Any], paused: bool) -> None:
"""Compare required slots and configured as permanent slots with those found, dropping extraneous ones.
@@ -357,7 +357,7 @@ class SlotsHandler:
self._schedule_load_slots = True
def _ensure_physical_slots(self, slots: Dict[str, Any]) -> None:
"""Create or advance physical replication *slots*.
"""Create any missing physical replication *slots*.
Any failures are logged and do not interrupt creation of all *slots*.
@@ -366,9 +366,7 @@ class SlotsHandler:
"""
immediately_reserve = ', true' if self._postgresql.major_version >= 90600 else ''
for name, value in slots.items():
if value['type'] != 'physical':
continue
if name not in self._replication_slots:
if name not in self._replication_slots and value['type'] == 'physical':
try:
self._query(f"SELECT pg_catalog.pg_create_physical_replication_slot(%s{immediately_reserve})"
f" WHERE NOT EXISTS (SELECT 1 FROM pg_catalog.pg_replication_slots"
@@ -377,15 +375,6 @@ class SlotsHandler:
except Exception:
logger.exception("Failed to create physical replication slot '%s'", name)
self._schedule_load_slots = True
elif self._postgresql.can_advance_slots and self._replication_slots[name]['type'] == 'physical':
value['restart_lsn'] = self._replication_slots[name]['restart_lsn']
lsn = value.get('lsn')
if lsn and lsn > value['restart_lsn']: # The slot has feedback in DCS and needs to be advanced
try:
lsn = format_lsn(lsn)
self._query("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)", name, lsn)
except Exception as exc:
logger.error("Error while advancing replication slot %s to position '%s': %r", name, lsn, exc)
@contextmanager
def get_local_connection_cursor(self, **kwargs: Any) -> Iterator[Union['cursor', 'Cursor[Any]']]:
@@ -399,7 +388,8 @@ class SlotsHandler:
:yields: connection cursor object, note implementation varies depending on version of :mod:`psycopg`.
"""
conn_kwargs = {**self._postgresql.connection_pool.conn_kwargs, **kwargs}
conn_kwargs = self._postgresql.config.local_connect_kwargs
conn_kwargs.update(kwargs)
with get_connection_cursor(**conn_kwargs) as cur:
yield cur
@@ -449,7 +439,7 @@ class SlotsHandler:
self._advance = SlotsAdvanceThread(self)
return self._advance.schedule(slots)
def _ensure_logical_slots_replica(self, slots: Dict[str, Any]) -> List[str]:
def _ensure_logical_slots_replica(self, cluster: Cluster, slots: Dict[str, Any]) -> List[str]:
"""Update logical *slots* on replicas.
If the logical slot already exists, copy state information into the replication slots structure stored in the
@@ -459,6 +449,7 @@ class SlotsHandler:
As logical slots can only be created when the primary is available, pass the list of slots that need to be
copied back to the caller. They will be created on replicas with :meth:`SlotsHandler.copy_logical_slots`.
:param cluster: object containing stateful information for the cluster.
:param slots: A dictionary mapping slot name to slot attributes. This method only considers a slot
if the value is a dictionary with the key ``type`` and a value of ``logical``.
@@ -473,13 +464,15 @@ class SlotsHandler:
continue
# If the logical already exists, copy some information about it into the original structure
if name in self._replication_slots and compare_slots(value, self._replication_slots[name]):
if self._replication_slots.get(name, {}).get('datoid'):
self._copy_items(self._replication_slots[name], value)
if 'lsn' in value and value['confirmed_flush_lsn'] < value['lsn']: # The slot has feedback in DCS
# Skip slots that don't need to be advanced
advance_slots[value['database']][name] = value['lsn']
elif name not in self._replication_slots and 'lsn' in value:
# We want to copy only slots with feedback in a DCS
if cluster.slots and name in cluster.slots:
try: # Skip slots that don't need to be advanced
if value['confirmed_flush_lsn'] < int(cluster.slots[name]):
advance_slots[value['database']][name] = int(cluster.slots[name])
except Exception as e:
logger.error('Failed to parse "%s": %r', cluster.slots[name], e)
elif cluster.slots and name in cluster.slots: # We want to copy only slots with feedback in a DCS
create_slots.append(name)
# Slots to be copied from the primary should be removed from the *slots* structure,
@@ -496,11 +489,10 @@ class SlotsHandler:
replicatefrom: Optional[str] = None, paused: bool = False) -> List[str]:
"""During the HA loop read, check and alter replication slots found in the cluster.
Read physical and logical slots from ``pg_replication_slots``, then compare to those configured in the DCS.
Read physical and logical slots found on the primary, then compare to those configured in the DCS.
Drop any slots that do not match those required by configuration and are not configured as permanent.
Create any missing physical slots, or advance their position according to feedback stored in DCS.
If we are the primary then create logical slots, otherwise if logical slots are known and active create
them on replica nodes by copying slot files from the primary.
Create any missing physical slots. If we are the leader then logical slots too, otherwise if logical slots
are known and active create them on replica nodes.
:param cluster: object containing stateful information for the cluster.
:param nofailover: ``True`` if this node has been tagged to not be a failover candidate.
@@ -522,12 +514,13 @@ class SlotsHandler:
self._ensure_physical_slots(slots)
if self._postgresql.is_primary():
if self._postgresql.is_leader():
self._logical_slots_processing_queue.clear()
self._ensure_logical_slots_primary(slots)
else:
elif cluster.slots and slots:
self.check_logical_slots_readiness(cluster, replicatefrom)
ret = self._ensure_logical_slots_replica(slots)
ret = self._ensure_logical_slots_replica(cluster, slots)
self._replication_slots = slots
except Exception:
@@ -608,8 +601,11 @@ class SlotsHandler:
# Replica isn't streaming or the hot_standby_feedback isn't enabled
try:
if not self._query("SELECT pg_catalog.current_setting('hot_standby_feedback')::boolean")[0][0]:
logger.error('Logical slot failover requires "hot_standby_feedback". Please check postgresql.auto.conf')
cur = self._query("SELECT pg_catalog.current_setting('hot_standby_feedback')::boolean")
row = cur.fetchone()
if row and not row[0]:
logger.error('Logical slot failover requires "hot_standby_feedback".'
' Please check postgresql.auto.conf')
except Exception as e:
logger.error('Failed to check the hot_standby_feedback setting: %r', e)
return False
+3 -3
View File
@@ -182,7 +182,7 @@ class _ReplicaList(List[_Replica]):
swapping, but only if lag on this member is exceeding a threshold (``maximum_lag_on_syncnode``).
:ivar max_lsn: maximum value of ``_Replica.lsn`` among all values. In case if there is just one
element in the list we take value of ``pg_current_wal_flush_lsn()``.
element in the list we take value of ``pg_current_wal_lsn()``.
"""
def __init__(self, postgresql: 'Postgresql', cluster: Cluster) -> None:
@@ -209,7 +209,7 @@ class _ReplicaList(List[_Replica]):
# 2. can be mapped to a ``Member`` of the ``Cluster``:
# a. ``Member`` doesn't have ``nosync`` tag set;
# b. PostgreSQL on the member is known to be running and accepting client connections.
if member and row[sort_col] is not None and member.is_running and not member.nosync:
if member and row[sort_col] is not None and member.is_running and not member.tags.get('nosync', False):
self.append(_Replica(row['pid'], row['application_name'],
row['sync_state'], row[sort_col], bool(member.nofailover)))
@@ -339,7 +339,7 @@ END;$$""")
sync_param = next(iter(sync), None)
if not (self._postgresql.config.set_synchronous_standby_names(sync_param)
and self._postgresql.state == 'running' and self._postgresql.is_primary()) or has_asterisk:
and self._postgresql.state == 'running' and self._postgresql.is_leader()) or has_asterisk:
return
time.sleep(0.1) # Usualy it takes 1ms to reload postgresql.conf, but we will give it 100ms
+59 -65
View File
@@ -290,7 +290,7 @@ def _load_postgres_gucs_validators() -> None:
Any problem faced while reading or parsing files will be logged as a ``WARNING`` by the child function, and the
corresponding file or validator will be ignored.
By default, Patroni only ships the file ``0_postgres.yml``, which contains Community Postgres GUCs validators, but
By default Patroni only ships the file ``0_postgres.yml``, which contains Community Postgres GUCs validators, but
that behavior can be extended. For example: if a vendor wants to add GUC validators to Patroni for covering a custom
Postgres build, then they can create their custom YAML files under ``available_parameters`` directory.
@@ -300,10 +300,8 @@ def _load_postgres_gucs_validators() -> None:
writes them to ``postgresql.conf`` if running PG 12 and above).
Then, each of these sections, if specified, may contain one or more attributes with the following structure:
* key: the name of a GUC;
* value: a list of validators. Each item in the list must contain a ``type`` attribute, which must be one among:
* ``Bool``; or
* ``Integer``; or
* ``Real``; or
@@ -315,7 +313,6 @@ def _load_postgres_gucs_validators() -> None:
class in this module.
.. seealso::
* :class:`Bool`;
* :class:`Integer`;
* :class:`Real`;
@@ -328,62 +325,61 @@ def _load_postgres_gucs_validators() -> None:
This is a sample content for an YAML file based on Postgres GUCs, showing each of the supported types and
sections:
.. code-block:: yaml
parameters:
archive_command:
- type: String
version_from: 90300
version_till: null
archive_mode:
- type: Bool
version_from: 90300
version_till: 90500
- type: EnumBool
version_from: 90500
version_till: null
possible_values:
- always
archive_timeout:
- type: Integer
version_from: 90300
version_till: null
min_val: 0
max_val: 1073741823
unit: s
autovacuum_vacuum_cost_delay:
- type: Integer
version_from: 90300
version_till: 120000
min_val: -1
max_val: 100
unit: ms
- type: Real
version_from: 120000
version_till: null
min_val: -1
max_val: 100
unit: ms
client_min_messages:
- type: Enum
version_from: 90300
version_till: null
possible_values:
- debug5
- debug4
- debug3
- debug2
- debug1
- log
- notice
- warning
- error
recovery_parameters:
archive_cleanup_command:
- type: String
version_from: 90300
version_till: null
```yaml
parameters:
archive_command:
- type: String
version_from: 90300
version_till: null
archive_mode:
- type: Bool
version_from: 90300
version_till: 90500
- type: EnumBool
version_from: 90500
version_till: null
possible_values:
- always
archive_timeout:
- type: Integer
version_from: 90300
version_till: null
min_val: 0
max_val: 1073741823
unit: s
autovacuum_vacuum_cost_delay:
- type: Integer
version_from: 90300
version_till: 120000
min_val: -1
max_val: 100
unit: ms
- type: Real
version_from: 120000
version_till: null
min_val: -1
max_val: 100
unit: ms
client_min_messages:
- type: Enum
version_from: 90300
version_till: null
possible_values:
- debug5
- debug4
- debug3
- debug2
- debug1
- log
- notice
- warning
- error
recovery_parameters:
archive_cleanup_command:
- type: String
version_from: 90300
version_till: null
```
"""
conf_dir = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
@@ -438,15 +434,13 @@ def _transform_parameter_value(validators: MutableMapping[str, Tuple[_Transforma
:param value: value of the Postgres GUC.
:param available_gucs: a set of all GUCs available in Postgres *version*. Each item is the name of a Postgres
GUC. Used for a couple purposes:
* Disallow writing GUCs to ``postgresql.conf`` (or ``recovery.conf``) that does not exist in Postgres *version*;
* Avoid ignoring GUC *name* if it does not have a validator in *validators*, but is a valid GUC in Postgres
*version*.
*version*.
:returns: the return value may be one among:
* *value* transformed to the expected format for GUC *name* in Postgres *version*, if *name* is present
in *available_gucs* and has a validator in *validators* for the corresponding Postgres *version*; or
* *value* transformed to the expected format for GUC *name* in Postgres *version*, if *name* is present in
*available_gucs* and has a validator in *validators* for the corresponding Postgres *version*; or
* The own *value* if *name* is present in *available_gucs* but not in *validators*; or
* ``None`` if *name* is not present in *available_gucs*.
"""
-87
View File
@@ -1,87 +0,0 @@
"""Tags handling."""
import abc
from typing import Any, Dict, Optional
from patroni.utils import parse_int
class Tags(abc.ABC):
"""An abstract class that encapsulates all the ``tags`` logic.
Child classes that want to use provided facilities must implement ``tags`` abstract property.
"""
@staticmethod
def _filter_tags(tags: Dict[str, Any]) -> Dict[str, Any]:
"""Get tags configured for this node, if any.
Handle both predefined Patroni tags and custom defined tags.
.. note::
A custom tag is any tag added to the configuration ``tags`` section that is not one of ``clonefrom``,
``nofailover``, ``noloadbalance`` or ``nosync``.
For the Patroni predefined tags, the returning object will only contain them if they are enabled as they
all are boolean values that default to disabled.
:returns: a dictionary of tags set for this node. The key is the tag name, and the value is the corresponding
tag value.
"""
return {tag: value for tag, value in tags.items()
if tag not in ('clonefrom', 'nofailover', 'noloadbalance', 'nosync') or value}
@property
@abc.abstractmethod
def tags(self) -> Dict[str, Any]:
"""Configured tags.
Must be implemented in a child class.
"""
raise NotImplementedError # pragma: no cover
@property
def clonefrom(self) -> bool:
"""``True`` if ``clonefrom`` tag is ``True``, else ``False``."""
return self.tags.get('clonefrom', False)
@property
def nofailover(self) -> bool:
"""Common logic for obtaining the value of ``nofailover`` from ``tags`` if defined.
If ``nofailover`` is not defined, this methods returns ``True`` if ``failover_priority`` is non-positive,
``False`` otherwise.
"""
from_tags = self.tags.get('nofailover')
if from_tags is not None:
# Value of `nofailover` takes precedence over `failover_priority`
return bool(from_tags)
failover_priority = parse_int(self.tags.get('failover_priority'))
return failover_priority is not None and failover_priority <= 0
@property
def failover_priority(self) -> int:
"""Common logic for obtaining the value of ``failover_priority`` from ``tags`` if defined.
If ``nofailover`` is defined as ``True``, this will return ``0``. Otherwise, it will return the value of
``failover_priority``, defaulting to ``1`` if it's not defined or invalid.
"""
from_tags = self.tags.get('nofailover')
failover_priority = parse_int(self.tags.get('failover_priority'))
failover_priority = 1 if failover_priority is None else failover_priority
return 0 if from_tags else failover_priority
@property
def noloadbalance(self) -> bool:
"""``True`` if ``noloadbalance`` is ``True``, else ``False``."""
return bool(self.tags.get('noloadbalance', False))
@property
def nosync(self) -> bool:
"""``True`` if ``nosync`` is ``True``, else ``False``."""
return bool(self.tags.get('nosync', False))
@property
def replicatefrom(self) -> Optional[str]:
"""Value of ``replicatefrom`` tag, if any."""
return self.tags.get('replicatefrom')
+1 -47
View File
@@ -16,7 +16,6 @@ import platform
import random
import re
import socket
import subprocess
import sys
import tempfile
import time
@@ -471,18 +470,6 @@ def _sleep(interval: Union[int, float]) -> None:
time.sleep(interval)
def read_stripped(file_path: str) -> Iterator[str]:
"""Iterate over stripped lines in the given file.
:param file_path: path to the file to read from
:yields: each line from the given file stripped
"""
with open(file_path) as f:
for line in f:
yield line.strip()
class RetryFailedError(PatroniException):
"""Maximum number of attempts exhausted in retry operation."""
@@ -819,7 +806,7 @@ def cluster_as_json(cluster: 'Cluster', global_config: Optional['GlobalConfig']
member.update({n: m.data[n] for n in optional_attributes if n in m.data})
if m.name != leader_name:
lsn = m.lsn
lsn = m.data.get('xlog_location')
if lsn is None:
member['lag'] = 'unknown'
elif cluster_lsn >= lsn:
@@ -1028,36 +1015,3 @@ def unquote(string: str) -> str:
except ValueError:
ret = string
return ret
def get_major_version(bin_dir: Optional[str] = None, bin_name: str = 'postgres') -> str:
"""Get the major version of PostgreSQL.
It is based on the output of ``postgres --version``.
:param bin_dir: path to the PostgreSQL binaries directory. If ``None`` or an empty string, it will use the first
*bin_name* binary that is found by the subprocess in the ``PATH``.
:param bin_name: name of the postgres binary to call (``postgres`` by default)
:returns: the PostgreSQL major version.
:raises:
:exc:`~patroni.exceptions.PatroniException`: if the postgres binary call failed due to :exc:`OSError`.
:Example:
* Returns `9.6` for PostgreSQL 9.6.24
* Returns `15` for PostgreSQL 15.2
"""
if not bin_dir:
binary = bin_name
else:
binary = os.path.join(bin_dir, bin_name)
try:
version = subprocess.check_output([binary, '--version']).decode()
except OSError as e:
raise PatroniException(f'Failed to get postgres version: {e}')
version = re.match(r'^[^\s]+ [^\s]+ (\d+)(\.(\d+))?', version)
if TYPE_CHECKING: # pragma: no cover
assert version is not None
return '.'.join([version.group(1), version.group(3)]) if int(version.group(1)) < 10 else version.group(1)
+104 -130
View File
@@ -6,16 +6,17 @@ This module contains facilities for validating configuration of Patroni processe
:var schema: configuration schema of the daemon launched by ``patroni`` command.
"""
import os
import re
import shutil
import socket
import subprocess
from typing import Any, Dict, Union, Iterator, List, Optional as OptionalType, Tuple
from typing import Any, Dict, Union, Iterator, List, Optional as OptionalType, Tuple, TYPE_CHECKING
from .collections import CaseInsensitiveSet
from .dcs import dcs_modules
from .exceptions import ConfigParseError
from .utils import parse_int, split_host_port, data_directory_is_empty, get_major_version
from .utils import parse_int, split_host_port, data_directory_is_empty
def data_directory_empty(data_dir: str) -> bool:
@@ -203,6 +204,31 @@ def get_bin_name(bin_name: str) -> str:
return (schema.data.get('postgresql', {}).get('bin_name', {}) or {}).get(bin_name, bin_name)
def get_major_version(bin_dir: OptionalType[str] = None) -> str:
"""Get the major version of PostgreSQL.
It is based on the output of ``postgres --version``.
:param bin_dir: path to PostgreSQL binaries directory. If ``None`` it will use the first ``postgres`` binary that
is found by subprocess in the ``PATH``.
:returns: the PostgreSQL major version.
:Example:
* Returns `9.6` for PostgreSQL 9.6.24
* Returns `15` for PostgreSQL 15.2
"""
if not bin_dir:
binary = get_bin_name('postgres')
else:
binary = os.path.join(bin_dir, get_bin_name('postgres'))
version = subprocess.check_output([binary, '--version']).decode()
version = re.match(r'^[^\s]+ [^\s]+ (\d+)(\.(\d+))?', version)
if TYPE_CHECKING: # pragma: no cover
assert version is not None
return '.'.join([version.group(1), version.group(3)]) if int(version.group(1)) < 10 else version.group(1)
def validate_data_dir(data_dir: str) -> bool:
"""Validate the value of ``postgresql.data_dir`` configuration option.
@@ -240,7 +266,7 @@ def validate_data_dir(data_dir: str) -> bool:
raise ConfigParseError("data dir for the cluster is not empty, but doesn't contain"
" \"{}\" directory".format(waldir))
bin_dir = schema.data.get("postgresql", {}).get("bin_dir", None)
major_version = get_major_version(bin_dir, get_bin_name('postgres'))
major_version = get_major_version(bin_dir)
if pgversion != major_version:
raise ConfigParseError("data_dir directory postgresql version ({}) doesn't match with "
"'postgres --version' output ({})".format(pgversion, major_version))
@@ -333,17 +359,15 @@ class Case(object):
"""Create a :class:`Case` object.
:param schema: the schema for validating a set of attributes that may be available in the configuration.
Each key is the configuration that is available in a given scope and that should be validated,
and the related value is the validation function or expected type.
Each key is the configuration that is available in a given scope and that should be validated, and the
related value is the validation function or expected type.
:Example:
.. code-block:: python
Case({
"host": validate_host_port,
"url": str,
})
Case({
"host": validate_host_port,
"url": str,
})
That will check that ``host`` configuration, if given, is valid based on :func:`validate_host_port`, and will
also check that ``url`` configuration, if given, is a ``str`` instance.
@@ -365,47 +389,14 @@ class Or(object):
:Example:
.. code-block:: python
Or("host", "hosts"): Case({
"host": validate_host_port,
"hosts": Or(comma_separated_host_port, [validate_host_port]),
})
Or("host", "hosts"): Case({
"host": validate_host_port,
"hosts": Or(comma_separated_host_port, [validate_host_port]),
})
The outer :class:`Or` is used to define that ``host`` and ``hosts`` are possible options in this scope.
The inner :class`Or` in the ``hosts`` key value is used to define that ``hosts`` option is valid if either
of :func:`comma_separated_host_port` or :func:`validate_host_port` succeed to validate it.
"""
self.args = args
class AtMostOne(object):
"""Mark that at most one option from a :class:`Case` can be suplied.
Represents a list of possible configuration options in a given scope, where at most one can actually
be provided.
.. note::
It should be used together with a :class:`Case` object.
"""
def __init__(self, *args: str) -> None:
"""Create a :class`AtMostOne` object.
:param `*args`: any arguments that the caller wants to be stored in this :class:`Or` object.
:Example:
.. code-block:: python
AtMostOne("nofailover", "failover_priority"): Case({
"nofailover": bool,
"failover_priority": IntValidator(min=0, raise_assert=True),
})
The :class`AtMostOne` object is used to define that at most one of ``nofailover`` and
``failover_priority`` can be provided.
The outer :class:`Or` is used to define that ``host`` and ``hosts`` are possible options in this scope.
The inner :class`Or` in the ``hosts`` key value is used to define that ``hosts`` option is valid if either of
:func:`comma_separated_host_port` or :func:`validate_host_port` succeed to validate it.
"""
self.args = args
@@ -570,34 +561,32 @@ class Schema(object):
:Example:
.. code-block:: python
Schema({
"application_name": str,
"bind": {
"host": validate_host,
"port": int,
},
"aliases": [str],
Optional("data_directory"): "/var/lib/myapp",
Or("log_to_file", "log_to_db"): Case({
"log_to_file": bool,
"log_to_db": bool,
}),
"version": Or(int, float),
})
Schema({
"application_name": str,
"bind": {
"host": validate_host,
"port": int,
},
"aliases": [str],
Optional("data_directory"): "/var/lib/myapp",
Or("log_to_file", "log_to_db"): Case({
"log_to_file": bool,
"log_to_db": bool,
}),
"version": Or(int, float),
})
This sample schema defines that your YAML configuration follows these rules:
This sample schema defines that your YAML configuration follows these rules:
* It must contain an ``application_name`` entry which value should be a :class:`str` instance;
* It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``;
* It must contain a ``bind.port`` entry which value should be an :class:`int` instance;
* It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances;
* It may optionally contain a ``data_directory`` entry, with a value which should be a string;
* It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a
:class:`bool` instance;
* It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float`
instance.
* It must contain an ``application_name`` entry which value should be a :class:`str` instance;
* It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``;
* It must contain a ``bind.port`` entry which value should be an :class:`int` instance;
* It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances;
* It may optionally contain a ``data_directory`` entry, with a value which should be a string;
* It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a
:class:`bool` instance;
* It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float`
instance.
"""
self.validator = validator
@@ -702,9 +691,6 @@ class Schema(object):
# One key in `validator` attribute (`key` variable) can be mapped to one or more keys in `data` attribute (`d`
# variable), depending on the `key` type.
for key in self.validator.keys():
if isinstance(key, AtMostOne) and len(list(self._data_key(key))) > 1:
yield Result(False, f"Multiple of {key.args} provided")
continue
for d in self._data_key(key):
if d not in self.data and not isinstance(key, Optional):
yield Result(False, "is not defined.", path=d)
@@ -714,7 +700,7 @@ class Schema(object):
if d not in self.data and isinstance(key, Optional):
self.data[d] = key.default
validator = self.validator[key]
if isinstance(key, (Or, AtMostOne)) and isinstance(self.validator[key], Case):
if isinstance(key, Or) and isinstance(self.validator[key], Case):
validator = self.validator[key]._schema[d]
# In this loop we may be calling a new `Schema` either over an intermediate node in the tree, or
# over a leaf node. In the latter case the recursive calls in the given path will finish.
@@ -749,7 +735,7 @@ class Schema(object):
max_level = v.level
yield Result(v.status, v.error, path=v.path, level=v.level, data=v.data)
def _data_key(self, key: Union[str, Optional, Or, AtMostOne]) -> Iterator[str]:
def _data_key(self, key: Union[str, Optional, Or]) -> Iterator[str]:
"""Map a key from the ``validator`` dictionary to the corresponding key(s) in the ``data`` dictionary.
:param key: key from the ``validator`` attribute.
@@ -769,23 +755,15 @@ class Schema(object):
elif isinstance(key, Or):
# At least one of the `Or` entries should be available in the `data` dictionary. If we find at least one of
# them in `data`, then we return all found entries so the caller method can validate them all.
if any([item in self.data for item in key.args]):
for item in key.args:
if item in self.data:
yield item
if any([i in self.data for i in key.args]):
for i in key.args:
if i in self.data:
yield i
# If none of the `Or` entries is available in the `data` dictionary, then we return all entries so the
# caller method will issue errors that they are all absent.
else:
for item in key.args:
yield item
# If the key was defined as a `AtMostOne` object in `validator` attribute, then each of its values
# are the keys to access the `data` dictionary.
elif isinstance(key, AtMostOne):
# Yield back all of the entries from the `data` dictionary, each will be validated and then counted
# to inform us if we've provided too many
for item in key.args:
if item in self.data:
yield item
for i in key.args:
yield i
def _get_type_name(python_type: Any) -> str:
@@ -814,28 +792,27 @@ def assert_(condition: bool, message: str = "Wrong value") -> None:
class IntValidator(object):
"""Validate an integer setting.
:cvar expected_type: the expected Python type for an integer setting (:class:`int`).
:ivar min: minimum allowed value for the setting, if any.
:ivar max: maximum allowed value for the setting, if any.
:ivar base_unit: the base unit to convert the value to before checking if it's within *min* and *max* range.
:ivar expected_type: the expected Python type.
:ivar raise_assert: if an ``assert`` test should be performed regarding expected type and valid range.
"""
expected_type = int
def __init__(self, min: OptionalType[int] = None, max: OptionalType[int] = None,
base_unit: OptionalType[str] = None, expected_type: Any = None, raise_assert: bool = False) -> None:
base_unit: OptionalType[str] = None, raise_assert: bool = False) -> None:
"""Create an :class:`IntValidator` object with the given rules.
:param min: minimum allowed value for the setting, if any.
:param max: maximum allowed value for the setting, if any.
:param base_unit: the base unit to convert the value to before checking if it's within *min* and *max* range.
:param expected_type: the expected Python type.
:param raise_assert: if an ``assert`` test should be performed regarding expected type and valid range.
"""
self.min = min
self.max = max
self.base_unit = base_unit
if expected_type:
self.expected_type = expected_type
self.raise_assert = raise_assert
def __call__(self, value: Any) -> bool:
@@ -954,36 +931,36 @@ schema = Schema({
Optional("allowlist_include_members"): bool,
Optional("http_extra_headers"): dict,
Optional("https_extra_headers"): dict,
Optional("request_queue_size"): IntValidator(min=0, max=4096, expected_type=int, raise_assert=True)
Optional("request_queue_size"): IntValidator(min=0, max=4096, raise_assert=True)
},
Optional("bootstrap"): {
"dcs": {
Optional("ttl"): IntValidator(min=20, raise_assert=True),
Optional("loop_wait"): IntValidator(min=1, raise_assert=True),
Optional("retry_timeout"): IntValidator(min=3, raise_assert=True),
Optional("maximum_lag_on_failover"): IntValidator(min=0, raise_assert=True),
Optional("maximum_lag_on_syncnode"): IntValidator(min=-1, raise_assert=True),
Optional("ttl"): int,
Optional("loop_wait"): int,
Optional("retry_timeout"): int,
Optional("maximum_lag_on_failover"): int,
Optional("maximum_lag_on_syncnode"): int,
Optional("postgresql"): {
Optional("parameters"): {
Optional("max_connections"): IntValidator(1, 262143, raise_assert=True),
Optional("max_locks_per_transaction"): IntValidator(10, 2147483647, raise_assert=True),
Optional("max_prepared_transactions"): IntValidator(0, 262143, raise_assert=True),
Optional("max_replication_slots"): IntValidator(0, 262143, raise_assert=True),
Optional("max_wal_senders"): IntValidator(0, 262143, raise_assert=True),
Optional("max_worker_processes"): IntValidator(0, 262143, raise_assert=True),
Optional("max_connections"): int,
Optional("max_locks_per_transaction"): int,
Optional("max_prepared_transactions"): int,
Optional("max_replication_slots"): int,
Optional("max_wal_senders"): int,
Optional("max_worker_processes"): int
},
Optional("use_pg_rewind"): bool,
Optional("pg_hba"): [str],
Optional("pg_ident"): [str],
Optional("pg_ctl_timeout"): IntValidator(min=0, raise_assert=True),
Optional("pg_ctl_timeout"): int,
Optional("use_slots"): bool,
},
Optional("primary_start_timeout"): IntValidator(min=0, raise_assert=True),
Optional("primary_stop_timeout"): IntValidator(min=0, raise_assert=True),
Optional("primary_start_timeout"): int,
Optional("primary_stop_timeout"): int,
Optional("standby_cluster"): {
Or("host", "port", "restore_command"): Case({
"host": str,
"port": IntValidator(max=65535, expected_type=int, raise_assert=True),
"port": int,
"restore_command": str
}),
Optional("primary_slot_name"): str,
@@ -993,7 +970,7 @@ schema = Schema({
},
Optional("synchronous_mode"): bool,
Optional("synchronous_mode_strict"): bool,
Optional("synchronous_node_count"): IntValidator(min=1, raise_assert=True),
Optional("synchronous_node_count"): int
},
Optional("initdb"): [Or(str, dict)],
Optional("method"): str
@@ -1004,7 +981,7 @@ schema = Schema({
"host": validate_host_port,
"url": str
}),
Optional("port"): IntValidator(max=65535, expected_type=int, raise_assert=True),
Optional("port"): int,
Optional("scheme"): str,
Optional("token"): str,
Optional("verify"): bool,
@@ -1024,8 +1001,8 @@ schema = Schema({
"etcd3": validate_etcd,
"exhibitor": {
"hosts": [str],
"port": IntValidator(max=65535, expected_type=int, raise_assert=True),
Optional("poll_interval"): IntValidator(min=1, expected_type=int, raise_assert=True),
"port": IntValidator(max=65535, raise_assert=True),
Optional("pool_interval"): int
},
"raft": {
"self_addr": validate_connect_address,
@@ -1056,14 +1033,14 @@ schema = Schema({
Optional("tmp_role_label"): str,
Optional("use_endpoints"): bool,
Optional("pod_ip"): Or(is_ipv4_address, is_ipv6_address),
Optional("ports"): [{"name": str, "port": IntValidator(max=65535, expected_type=int, raise_assert=True)}],
Optional("ports"): [{"name": str, "port": int}],
Optional("cacert"): str,
Optional("retriable_http_codes"): Or(int, [int]),
},
}),
Optional("citus"): {
"database": str,
"group": IntValidator(min=0, expected_type=int, raise_assert=True),
"group": int
},
"postgresql": {
"listen": validate_host_port_listen_multiple_hosts,
@@ -1090,19 +1067,16 @@ schema = Schema({
},
Optional("pg_hba"): [str],
Optional("pg_ident"): [str],
Optional("pg_ctl_timeout"): IntValidator(min=0, raise_assert=True),
Optional("pg_ctl_timeout"): int,
Optional("use_pg_rewind"): bool
},
Optional("watchdog"): {
Optional("mode"): validate_watchdog_mode,
Optional("device"): str,
Optional("safety_margin"): IntValidator(min=-1, expected_type=int, raise_assert=True),
Optional("safety_margin"): int
},
Optional("tags"): {
AtMostOne("nofailover", "failover_priority"): Case({
"nofailover": bool,
"failover_priority": IntValidator(min=0, expected_type=int, raise_assert=True),
}),
Optional("nofailover"): bool,
Optional("clonefrom"): bool,
Optional("noloadbalance"): bool,
Optional("replicatefrom"): str,
+1 -1
View File
@@ -2,4 +2,4 @@
:var __version__: the current Patroni version.
"""
__version__ = '3.2.0'
__version__ = '3.1.2'
+10 -6
View File
@@ -43,13 +43,9 @@ etcd:
# - 127.0.0.1:2223
# - 127.0.0.1:2224
# The bootstrap configuration. Works only when the cluster is not yet initialized.
# If the cluster is already initialized, all changes in the `bootstrap` section are ignored!
bootstrap:
# This section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`.
# WARNING! If you want to change any of the parameters that were set up
# via `bootstrap.dcs` section, please use `patronictl edit-config`!
# this section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`
dcs:
ttl: 30
loop_wait: 10
@@ -97,6 +93,14 @@ bootstrap:
# Additional script to be launched after initial cluster creation (will be passed the connection URL as parameter)
# post_init: /usr/local/bin/setup_cluster.sh
# Some additional users which needs to be created after initializing new cluster
users:
admin:
password: admin%
options:
- createrole
- createdb
postgresql:
listen: 127.0.0.1:5432
connect_address: 127.0.0.1:5432
+10 -6
View File
@@ -43,13 +43,9 @@ etcd:
# - 127.0.0.1:2222
# - 127.0.0.1:2224
# The bootstrap configuration. Works only when the cluster is not yet initialized.
# If the cluster is already initialized, all changes in the `bootstrap` section are ignored!
bootstrap:
# This section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`.
# WARNING! If you want to change any of the parameters that were set up
# via `bootstrap.dcs` section, please use `patronictl edit-config`!
# this section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`
dcs:
ttl: 30
loop_wait: 10
@@ -91,6 +87,14 @@ bootstrap:
# Additional script to be launched after initial cluster creation (will be passed the connection URL as parameter)
# post_init: /usr/local/bin/setup_cluster.sh
# Some additional users which needs to be created after initializing new cluster
users:
admin:
password: admin%
options:
- createrole
- createdb
postgresql:
listen: 127.0.0.1:5433
connect_address: 127.0.0.1:5433
+10 -6
View File
@@ -43,13 +43,9 @@ etcd:
# - 127.0.0.1:2222
# - 127.0.0.1:2223
# The bootstrap configuration. Works only when the cluster is not yet initialized.
# If the cluster is already initialized, all changes in the `bootstrap` section are ignored!
bootstrap:
# This section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`.
# WARNING! If you want to change any of the parameters that were set up
# via `bootstrap.dcs` section, please use `patronictl edit-config`!
# this section will be written into Etcd:/<namespace>/<scope>/config after initializing new cluster
# and all other cluster members will use it as a `global configuration`
dcs:
ttl: 30
loop_wait: 10
@@ -88,6 +84,14 @@ bootstrap:
- encoding: UTF8
- data-checksums
# Some additional users which needs to be created after initializing new cluster
users:
admin:
password: admin%
options:
- createrole
- createdb
postgresql:
listen: 127.0.0.1:5434
connect_address: 127.0.0.1:5434
+2 -1
View File
@@ -1,4 +1,5 @@
sphinx>=4
sphinx_rtd_theme>1
sphinxcontrib-apidoc
sphinx-github-style<1.0.3
sphinx-github-style
pyyaml
+22 -25
View File
@@ -26,6 +26,7 @@ KEYWORDS = 'etcd governor patroni postgresql postgres ha haproxy confd' +\
EXTRAS_REQUIRE = {'aws': ['boto3'], 'etcd': ['python-etcd'], 'etcd3': ['python-etcd'],
'consul': ['python-consul'], 'exhibitor': ['kazoo'], 'zookeeper': ['kazoo'],
'kubernetes': [], 'raft': ['pysyncobj', 'cryptography']}
COVERAGE_XML = True
# Add here all kinds of additional classifiers as defined under
# https://pypi.python.org/pypi?%3Aaction=list_classifiers
@@ -119,21 +120,14 @@ def read(fname):
return fd.read()
def get_versions():
old_modules = sys.modules.copy()
try:
from patroni import MIN_PSYCOPG2, MIN_PSYCOPG3
from patroni.version import __version__
return __version__, MIN_PSYCOPG2, MIN_PSYCOPG3
finally:
sys.modules.clear()
sys.modules.update(old_modules)
def main():
def setup_package(version):
logging.basicConfig(format='%(message)s', level=os.getenv('LOGLEVEL', logging.WARNING))
# Assemble additional setup commands
cmdclass = {'test': PyTest, 'flake8': Flake8}
install_requires = []
for r in read('requirements.txt').split('\n'):
r = r.strip()
if r == '':
@@ -145,22 +139,15 @@ def main():
deps[i] = r
EXTRAS_REQUIRE[e] = deps
extra = True
break
if extra:
break
if not extra:
install_requires.append(r)
# Just for convenience, if someone wants to install dependencies for all extras
EXTRAS_REQUIRE['all'] = list({e for extras in EXTRAS_REQUIRE.values() for e in extras})
patroni_version, min_psycopg2, min_psycopg3 = get_versions()
# Make it possible to specify psycopg dependency as extra
for name, version in {'psycopg[binary]': min_psycopg3, 'psycopg2': min_psycopg2, 'psycopg2-binary': None}.items():
EXTRAS_REQUIRE[name] = [name + ('>=' + '.'.join(map(str, version)) if version else '')]
EXTRAS_REQUIRE['psycopg3'] = EXTRAS_REQUIRE.pop('psycopg[binary]')
setup(
name=NAME,
version=patroni_version,
version=version,
url=URL,
author=AUTHOR,
author_email=AUTHOR_EMAIL,
@@ -176,10 +163,20 @@ def main():
]},
install_requires=install_requires,
extras_require=EXTRAS_REQUIRE,
cmdclass={'test': PyTest, 'flake8': Flake8},
cmdclass=cmdclass,
entry_points={'console_scripts': CONSOLE_SCRIPTS},
)
if __name__ == '__main__':
main()
old_modules = sys.modules.copy()
try:
from patroni import check_psycopg
from patroni.version import __version__
finally:
sys.modules.clear()
sys.modules.update(old_modules)
check_psycopg()
setup_package(__version__)
+10 -21
View File
@@ -104,19 +104,20 @@ class MockCursor(object):
elif sql.startswith('SELECT slot_name, slot_type, datname, plugin, catalog_xmin'):
self.results = [('ls', 'logical', 'a', 'b', 100, 500, b'123456')]
elif sql.startswith('SELECT slot_name'):
self.results = [('blabla', 'physical', 12345),
('foobar', 'physical', 12345),
('ls', 'logical', 499, 'b', 'a', 5, 100, 500)]
self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'a', 'b', 5, 100, 500)]
elif sql.startswith('WITH slots AS (SELECT slot_name, active'):
self.results = [(False, True)] if self.rowcount == 1 else []
self.results = [(False, True)] if self.rowcount == 1 else [None]
elif sql.startswith('SELECT CASE WHEN pg_catalog.pg_is_in_recovery()'):
self.results = [(1, 2, 1, 0, False, 1, 1, None, None, 'streaming', '',
[{"slot_name": "ls", "confirmed_flush_lsn": 12345, "restart_lsn": 12344}],
[{"slot_name": "ls", "confirmed_flush_lsn": 12345}],
'on', 'n1', None)]
elif sql.startswith('SELECT pg_catalog.pg_is_in_recovery()'):
self.results = [(False, 2)]
elif sql.startswith('SELECT pg_catalog.pg_postmaster_start_time'):
self.results = [(datetime.datetime.now(tzutc),)]
replication_info = '[{"application_name":"walreceiver","client_addr":"1.2.3.4",' +\
'"state":"streaming","sync_state":"async","sync_priority":0}]'
now = datetime.datetime.now(tzutc)
self.results = [(now, 0, '', 0, '', False, now, 'streaming', None, replication_info)]
elif sql.startswith('SELECT name, pg_catalog.current_setting(name) FROM pg_catalog.pg_settings'):
self.results = [('data_directory', 'data'),
('hba_file', os.path.join('data', 'pg_hba.conf')),
@@ -147,8 +148,6 @@ class MockCursor(object):
('listen_addresses', '*', None, 'string', 'postmaster'),
('autovacuum', 'on', None, 'bool', 'sighup'),
('unix_socket_directories', '/tmp', None, 'string', 'postmaster')]
elif sql.startswith('SELECT COUNT(*) FROM pg_catalog.pg_settings'):
self.results = [(1,)]
elif sql.startswith('IDENTIFY_SYSTEM'):
self.results = [('1', 3, '0/402EEC0', '')]
elif sql.startswith('TIMELINE_HISTORY '):
@@ -162,7 +161,6 @@ class MockCursor(object):
self.results = [(1, 0, 'host1', 5432, 'primary'), (2, 1, 'host2', 5432, 'primary')]
else:
self.results = [(None, None, None, None, None, None, None, None, None, None)]
self.rowcount = len(self.results)
def fetchone(self):
return self.results[0]
@@ -181,20 +179,11 @@ class MockCursor(object):
pass
class MockConnectionInfo(object):
def parameter_status(self, param_name):
if param_name == 'is_superuser':
return 'on'
return '0'
class MockConnect(object):
server_version = 99999
autocommit = False
closed = 0
info = MockConnectionInfo()
def cursor(self):
return MockCursor(self)
@@ -241,7 +230,7 @@ class PostgresInit(unittest.TestCase):
'replication': {'username': '', 'password': 'rep-pass'},
'rewind': {'username': 'rewind', 'password': 'test'}},
'remove_data_directory_on_rewind_failure': True,
'use_pg_rewind': True, 'pg_ctl_timeout': 'bla', 'use_unix_socket': True,
'use_pg_rewind': True, 'pg_ctl_timeout': 'bla',
'parameters': self._PARAMETERS,
'recovery_conf': {'foo': 'bar'},
'pg_hba': ['host all all 0.0.0.0/0 md5'],
@@ -260,8 +249,8 @@ class BaseTestPostgresql(PostgresInit):
if not os.path.exists(self.p.data_dir):
os.makedirs(self.p.data_dir)
self.leadermem = Member(0, 'leader', 28, {'xlog_location': 100, 'state': 'running',
'conn_url': 'postgres://replicator:[email protected]:5435/postgres'})
self.leadermem = Member(0, 'leader', 28, {
'state': 'running', 'conn_url': 'postgres://replicator:[email protected]:5435/postgres'})
self.leader = Leader(-1, 28, self.leadermem)
self.other = Member(0, 'test-1', 28, {'conn_url': 'postgres://replicator:[email protected]:5433/postgres',
'state': 'running', 'tags': {'replicatefrom': 'leader'}})
+58 -157
View File
@@ -3,6 +3,8 @@ import json
import unittest
import socket
import patroni.psycopg as psycopg
from http.server import HTTPServer
from io import BytesIO as IO
from mock import Mock, PropertyMock, patch
@@ -11,12 +13,10 @@ from socketserver import ThreadingMixIn
from patroni.api import RestApiHandler, RestApiServer
from patroni.config import GlobalConfig
from patroni.dcs import ClusterConfig, Member
from patroni.exceptions import PostgresConnectionException
from patroni.ha import _MemberStatus
from patroni.psycopg import OperationalError
from patroni.utils import RetryFailedError, tzutc
from patroni.utils import tzutc
from . import MockConnect, psycopg_connect
from . import psycopg_connect, MockCursor
from .test_ha import get_cluster_initialized_without_leader
@@ -24,29 +24,8 @@ future_restart_time = datetime.datetime.now(tzutc) + datetime.timedelta(days=5)
postmaster_start_time = datetime.datetime.now(tzutc)
class MockConnection:
class MockPostgresql(object):
@staticmethod
def get(*args):
return psycopg_connect()
@staticmethod
def query(sql, *params):
return [(postmaster_start_time, 0, '', 0, '', False, postmaster_start_time, 'streaming', None,
'[{"application_name":"walreceiver","client_addr":"1.2.3.4",'
+ '"state":"streaming","sync_state":"async","sync_priority":0}]')]
class MockConnectionPool:
@staticmethod
def get(*args):
return MockConnection()
class MockPostgresql:
connection_pool = MockConnectionPool()
name = 'test'
state = 'running'
role = 'primary'
@@ -57,11 +36,14 @@ class MockPostgresql:
pending_restart = True
wal_name = 'wal'
lsn_name = 'lsn'
wal_flush = '_flush'
POSTMASTER_START_TIME = 'pg_catalog.pg_postmaster_start_time()'
TL_LSN = 'CASE WHEN pg_catalog.pg_is_in_recovery()'
citus_handler = Mock()
@staticmethod
def connection():
return psycopg_connect()
@staticmethod
def postmaster_start_time():
return postmaster_start_time
@@ -118,7 +100,7 @@ class MockHa(object):
@staticmethod
def fetch_nodes_statuses(members):
return [_MemberStatus(None, True, None, 0, {})]
return [_MemberStatus(None, True, None, 0, 0, None, {}, False)]
@staticmethod
def schedule_future_restart(data):
@@ -505,7 +487,9 @@ class TestRestApiHandler(unittest.TestCase):
@patch('time.sleep', Mock())
def test_RestApiServer_query(self):
with patch.object(MockConnection, 'query', Mock(side_effect=RetryFailedError('bla'))):
with patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)):
self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))
with patch.object(MockPostgresql, 'connection', Mock(side_effect=psycopg.OperationalError)):
self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))
@patch('time.sleep', Mock())
@@ -516,161 +500,86 @@ class TestRestApiHandler(unittest.TestCase):
post = 'POST /switchover HTTP/1.0' + self._authorization + '\nContent-Length: '
# Invalid content
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, post + '7\n\n{"1":2}')
response_mock.assert_called_with(400, 'Switchover could be performed only from a specific leader')
MockRestApiServer(RestApiHandler, post + '7\n\n{"1":2}')
# Empty content
request = post + '0\n\n'
MockRestApiServer(RestApiHandler, request)
# [Switchover without a candidate]
cluster.leader.name = 'postgresql1'
MockRestApiServer(RestApiHandler, request)
# Cluster with only a leader
with patch.object(RestApiHandler, 'write_response') as response_mock:
cluster.leader.name = 'postgresql1'
request = post + '25\n\n{"leader": "postgresql1"}'
request = post + '25\n\n{"leader": "postgresql1"}'
with patch.object(GlobalConfig, 'is_paused', PropertyMock(return_value=True)):
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(
412, 'switchover is not possible: cluster does not have members except leader')
# Switchover in pause mode
with patch.object(RestApiHandler, 'write_response') as response_mock, \
patch.object(GlobalConfig, 'is_paused', PropertyMock(return_value=True)):
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(
400, 'Switchover is possible only to a specific candidate in a paused state')
# No healthy nodes to promote in both sync and async mode
for is_synchronous_mode, response in (
(True, 'switchover is not possible: can not find sync_standby'),
(False, 'switchover is not possible: cluster does not have members except leader')):
with patch.object(GlobalConfig, 'is_synchronous_mode', PropertyMock(return_value=is_synchronous_mode)), \
patch.object(RestApiHandler, 'write_response') as response_mock:
for is_synchronous_mode in (True, False):
with patch.object(GlobalConfig, 'is_synchronous_mode', PropertyMock(return_value=is_synchronous_mode)):
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(412, response)
# [Switchover to the candidate specified]
cluster.leader.name = 'postgresql2'
request = post + '53\n\n{"leader": "postgresql1", "candidate": "postgresql2"}'
MockRestApiServer(RestApiHandler, request)
# Candidate to promote is the same as the leader specified
with patch.object(RestApiHandler, 'write_response') as response_mock:
request = post + '53\n\n{"leader": "postgresql2", "candidate": "postgresql2"}'
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(400, 'Switchover target and source are the same')
# Current leader is different from the one specified
with patch.object(RestApiHandler, 'write_response') as response_mock:
cluster.leader.name = 'postgresql2'
request = post + '53\n\n{"leader": "postgresql1", "candidate": "postgresql2"}'
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(412, 'leader name does not match')
# Candidate to promote is not a member of the cluster
cluster.leader.name = 'postgresql1'
cluster.sync.matches.return_value = False
for is_synchronous_mode, response in (
(True, 'candidate name does not match with sync_standby'), (False, 'candidate does not exists')):
with patch.object(GlobalConfig, 'is_synchronous_mode', PropertyMock(return_value=is_synchronous_mode)), \
patch.object(RestApiHandler, 'write_response') as response_mock:
for is_synchronous_mode in (True, False):
with patch.object(GlobalConfig, 'is_synchronous_mode', PropertyMock(return_value=is_synchronous_mode)):
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(412, response)
cluster.members = [Member(0, 'postgresql0', 30, {'api_url': 'http'}),
Member(0, 'postgresql2', 30, {'api_url': 'http'})]
MockRestApiServer(RestApiHandler, request)
# Failover key is empty in DCS
with patch.object(RestApiHandler, 'write_response') as response_mock:
cluster.failover = None
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(503, 'Switchover failed')
cluster.failover = None
MockRestApiServer(RestApiHandler, request)
# Result polling failed
with patch.object(RestApiHandler, 'write_response') as response_mock:
dcs.get_cluster.side_effect = [cluster]
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(503, 'Switchover status unknown')
dcs.get_cluster.side_effect = [cluster]
MockRestApiServer(RestApiHandler, request)
# Switchover to a node different from the candidate specified
with patch.object(RestApiHandler, 'write_response') as response_mock:
cluster2 = cluster.copy()
cluster2.leader.name = 'postgresql0'
cluster2.is_unlocked.return_value = False
dcs.get_cluster.side_effect = [cluster, cluster2]
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(200, 'Switched over to "postgresql0" instead of "postgresql2"')
cluster2 = cluster.copy()
cluster2.leader.name = 'postgresql0'
cluster2.is_unlocked.return_value = False
dcs.get_cluster.side_effect = [cluster, cluster2]
MockRestApiServer(RestApiHandler, request)
# Successful switchover to the candidate
with patch.object(RestApiHandler, 'write_response') as response_mock:
cluster2.leader.name = 'postgresql2'
dcs.get_cluster.side_effect = [cluster, cluster2]
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(200, 'Successfully switched over to "postgresql2"')
with patch.object(RestApiHandler, 'write_response') as response_mock:
dcs.manual_failover.return_value = False
dcs.get_cluster.side_effect = None
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(503, 'failed to write failover key into DCS')
cluster2.leader.name = 'postgresql2'
dcs.get_cluster.side_effect = [cluster, cluster2]
MockRestApiServer(RestApiHandler, request)
dcs.get_cluster.side_effect = None
dcs.manual_failover.return_value = False
MockRestApiServer(RestApiHandler, request)
dcs.manual_failover.return_value = True
# Candidate is not healthy to be promoted
with patch.object(MockHa, 'fetch_nodes_statuses', Mock(return_value=[])), \
patch.object(RestApiHandler, 'write_response') as response_mock:
with patch.object(MockHa, 'fetch_nodes_statuses', Mock(return_value=[])):
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(412, 'switchover is not possible: no good candidates have been found')
# [Scheduled switchover]
# Valid future date
with patch.object(RestApiHandler, 'write_response') as response_mock:
request = post + '103\n\n{"leader": "postgresql1", "member": "postgresql2",' + \
' "scheduled_at": "6016-02-15T18:13:30.568224+01:00"}'
request = post + '103\n\n{"leader": "postgresql1", "member": "postgresql2",' +\
' "scheduled_at": "6016-02-15T18:13:30.568224+01:00"}'
MockRestApiServer(RestApiHandler, request)
with patch.object(GlobalConfig, 'is_paused', PropertyMock(return_value=True)), \
patch.object(MockPatroni, 'dcs') as d:
d.manual_failover.return_value = False
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(202, 'Switchover scheduled')
# Schedule in paused mode
with patch.object(RestApiHandler, 'write_response') as response_mock, \
patch.object(GlobalConfig, 'is_paused', PropertyMock(return_value=True)):
dcs.manual_failover.return_value = False
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(400, "Can't schedule switchover in the paused state")
# No timezone specified
with patch.object(RestApiHandler, 'write_response') as response_mock:
request = post + '97\n\n{"leader": "postgresql1", "member": "postgresql2",' + \
' "scheduled_at": "6016-02-15T18:13:30.568224"}'
MockRestApiServer(RestApiHandler, request)
response_mock.assert_called_with(400, 'Timezone information is mandatory for the scheduled switchover')
# Exception: No timezone specified
request = post + '97\n\n{"leader": "postgresql1", "member": "postgresql2",' +\
' "scheduled_at": "6016-02-15T18:13:30.568224"}'
MockRestApiServer(RestApiHandler, request)
# Exception: Scheduled in the past
request = post + '103\n\n{"leader": "postgresql1", "member": "postgresql2", "scheduled_at": "'
# Scheduled in the past
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, request + '1016-02-15T18:13:30.568224+01:00"}')
response_mock.assert_called_with(422, 'Cannot schedule switchover in the past')
MockRestApiServer(RestApiHandler, request + '1016-02-15T18:13:30.568224+01:00"}')
# Invalid date
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, request + '2010-02-29T18:13:30.568224+01:00"}')
response_mock.assert_called_with(
422, 'Unable to parse scheduled timestamp. It should be in an unambiguous format, e.g. ISO 8601')
self.assertIsNotNone(MockRestApiServer(RestApiHandler, request + '2010-02-29T18:13:30.568224+01:00"}'))
def test_do_POST_failover(self):
post = 'POST /failover HTTP/1.0' + self._authorization + '\nContent-Length: '
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, post + '14\n\n{"leader":"1"}')
response_mock.assert_called_once_with(400, 'Failover could be performed only to a specific candidate')
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, post + '37\n\n{"candidate":"2","scheduled_at": "1"}')
response_mock.assert_called_once_with(400, "Failover can't be scheduled")
with patch.object(RestApiHandler, 'write_response') as response_mock:
MockRestApiServer(RestApiHandler, post + '30\n\n{"leader":"1","candidate":"2"}')
response_mock.assert_called_once_with(412, 'leader name does not match')
MockRestApiServer(RestApiHandler, post + '14\n\n{"leader":"1"}')
MockRestApiServer(RestApiHandler, post + '37\n\n{"candidate":"2","scheduled_at": "1"}')
@patch.object(MockHa, 'is_leader', Mock(return_value=True))
def test_do_POST_citus(self):
@@ -752,11 +661,3 @@ class TestRestApiServer(unittest.TestCase):
def test_get_certificate_serial_number(self):
self.assertIsNone(self.srv.get_certificate_serial_number())
def test_query(self):
with patch.object(MockConnection, 'get', Mock(side_effect=OperationalError)):
self.assertRaises(PostgresConnectionException, self.srv.query, 'SELECT 1')
with patch.object(MockConnection, 'get', Mock(side_effect=[MockConnect(), OperationalError])), \
patch.object(MockConnection, 'query') as mock_query:
self.srv.query('SELECT 1')
mock_query.assert_called_once_with('SELECT 1')
+3 -3
View File
@@ -251,15 +251,15 @@ class TestBootstrap(BaseTestPostgresql):
self.assertFalse(self.b.call_post_bootstrap({'post_init': '/bin/false'}))
mock_cancellable_subprocess_call.return_value = 0
self.p.connection_pool._conn_kwargs.pop('user')
self.p.config.superuser.pop('username')
self.assertTrue(self.b.call_post_bootstrap({'post_init': '/bin/false'}))
mock_cancellable_subprocess_call.assert_called()
args, kwargs = mock_cancellable_subprocess_call.call_args
self.assertTrue('PGPASSFILE' in kwargs['env'])
self.assertEqual(args[0], ['/bin/false', 'dbname=postgres host=/tmp port=5432'])
self.assertEqual(args[0], ['/bin/false', 'dbname=postgres host=127.0.0.2 port=5432'])
mock_cancellable_subprocess_call.reset_mock()
self.p.connection_pool._conn_kwargs.pop('host')
self.p.config._local_address.pop('host')
self.assertTrue(self.b.call_post_bootstrap({'post_init': '/bin/false'}))
mock_cancellable_subprocess_call.assert_called()
self.assertEqual(mock_cancellable_subprocess_call.call_args[0][0], ['/bin/false', 'dbname=postgres port=5432'])
-1
View File
@@ -35,6 +35,5 @@ class TestCallbackExecutor(unittest.TestCase):
ce._invoke_excepthook = Mock()
self.assertIsNone(ce.call(callback))
mock_popen.side_effect = [Mock()]
self.assertIsNone(ce.call(['test.sh', 'on_reload', 'replica', 'foo']))
ce.join()
+1 -1
View File
@@ -13,6 +13,7 @@ class TestCitus(BaseTestPostgresql):
def setUp(self):
super(TestCitus, self).setUp()
self.c = self.p.citus_handler
self.c.set_conn_kwargs({'host': 'localhost', 'dbname': 'postgres'})
self.cluster = get_cluster_initialized_with_leader()
self.cluster.workers[1] = self.cluster
@@ -138,7 +139,6 @@ class TestCitus(BaseTestPostgresql):
self.assertEqual(parameters['max_prepared_transactions'], 202)
self.assertEqual(parameters['shared_preload_libraries'], 'citus,foo,bar')
self.assertEqual(parameters['wal_level'], 'logical')
self.assertEqual(parameters['citus.local_hostname'], '/tmp')
def test_bootstrap(self):
self.c._config = None
+1 -77
View File
@@ -5,7 +5,7 @@ import io
from copy import deepcopy
from mock import MagicMock, Mock, patch
from patroni.config import Config, ConfigParseError, GlobalConfig
from patroni.config import Config, ConfigParseError
class TestConfig(unittest.TestCase):
@@ -151,53 +151,6 @@ class TestConfig(unittest.TestCase):
def test_invalid_path(self):
self.assertRaises(ConfigParseError, Config, 'postgres0')
@patch.object(Config, 'get')
@patch('patroni.config.logger')
def test__validate_failover_tags(self, mock_logger, mock_get):
"""Ensures that only one of `nofailover` or `failover_priority` can be provided"""
mock_logger.warning.reset_mock()
config = Config("postgres0.yml")
# Providing one of `nofailover` or `failover_priority` is fine
just_nofailover = {"nofailover": True}
mock_get.side_effect = [just_nofailover] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_not_called()
just_failover_priority = {"failover_priority": 1}
mock_get.side_effect = [just_failover_priority] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_not_called()
# Providing both `nofailover` and `failover_priority` is fine if consistent
consistent_false = {"nofailover": False, "failover_priority": 1}
mock_get.side_effect = [consistent_false] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_not_called()
consistent_true = {"nofailover": True, "failover_priority": 0}
mock_get.side_effect = [consistent_true] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_not_called()
# Providing both inconsistently should log a warning
inconsistent_false = {"nofailover": False, "failover_priority": 0}
mock_get.side_effect = [inconsistent_false] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_called_once_with(
'Conflicting configuration between nofailover: %s and failover_priority: %s.'
+ ' Defaulting to nofailover: %s',
False,
0,
False
)
mock_logger.warning.reset_mock()
inconsistent_true = {"nofailover": True, "failover_priority": 1}
mock_get.side_effect = [inconsistent_true] * 2
self.assertIsNone(config._validate_failover_tags())
mock_logger.warning.assert_called_once_with(
'Conflicting configuration between nofailover: %s and failover_priority: %s.'
+ ' Defaulting to nofailover: %s',
True,
1,
True
)
def test__process_postgresql_parameters(self):
expected_params = {
'f.oo': 'bar', # not in ConfigHandler.CMDLINE_OPTIONS
@@ -220,32 +173,3 @@ class TestConfig(unittest.TestCase):
input_params['max_connections'] = 10
expected_params.pop('max_connections')
self.assertEqual(self.config._process_postgresql_parameters(input_params), expected_params)
def test__validate_and_adjust_timeouts(self):
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 15})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'ttl', 15, 20))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'loop_wait': 0})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'loop_wait', 0, 1))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'retry_timeout': 1})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'retry_timeout', 1, 3))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 20, 'loop_wait': 11, 'retry_timeout': 5})
self.assertEqual(mock_logger.call_args_list[0][0],
('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d '
'and retry_timeout=%d. Adjusting loop_wait from %d to %d', 20, 5, 11, 10))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 20, 'loop_wait': 10, 'retry_timeout': 10})
self.assertEqual(mock_logger.call_args_list[0][0],
('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d. Adjusting'
' loop_wait from %d to %d and retry_timeout from %d to %d', 20, 10, 1, 10, 9))
def test_global_config_is_synchronous_mode(self):
# we should ignore synchronous_mode setting in a standby cluster
config = {'standby_cluster': {'host': 'some_host'}, 'synchronous_mode': True}
self.assertFalse(GlobalConfig(config).is_synchronous_mode)
-352
View File
@@ -1,352 +0,0 @@
import os
import psutil
import unittest
import yaml
from . import MockConnect, MockCursor, MockConnectionInfo
from copy import deepcopy
from mock import MagicMock, Mock, PropertyMock, mock_open as _mock_open, patch
from patroni.__main__ import main as _main
from patroni.config import Config
from patroni.config_generator import AbstractConfigGenerator, get_address, NO_VALUE_MSG
from patroni.log import PatroniLogger
from patroni.utils import patch_config
from . import psycopg_connect
HOSTNAME = 'test_hostname'
IP = '1.9.8.4'
def mock_open(*args, **kwargs):
ret = _mock_open(*args, **kwargs)
ret.return_value.__iter__ = lambda o: iter(o.readline, '')
if not kwargs.get('read_data'):
ret.return_value.readline = Mock(return_value=None)
return ret
@patch('patroni.psycopg.connect', psycopg_connect)
@patch('builtins.open', MagicMock())
@patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 16.2"))
@patch('psutil.Process.exe', Mock(return_value='/bin/dir/from/running/postgres'))
@patch('psutil.Process.__init__', Mock(return_value=None))
@patch.object(AbstractConfigGenerator, '_HOSTNAME', HOSTNAME)
@patch.object(AbstractConfigGenerator, '_IP', IP)
class TestGenerateConfig(unittest.TestCase):
def setUp(self):
os.environ['PATRONI_SCOPE'] = 'scope_from_env'
os.environ['PATRONI_POSTGRESQL_BIN_DIR'] = '/bin/from/env'
os.environ['PATRONI_SUPERUSER_USERNAME'] = 'su_user_from_env'
os.environ['PATRONI_SUPERUSER_PASSWORD'] = 'su_pwd_from_env'
os.environ['PATRONI_REPLICATION_USERNAME'] = 'repl_user_from_env'
os.environ['PATRONI_REPLICATION_PASSWORD'] = 'repl_pwd_from_env'
os.environ['PATRONI_REWIND_USERNAME'] = 'rewind_user_from_env'
os.environ['PGUSER'] = 'pguser_from_env'
os.environ['PGPASSWORD'] = 'pguser_pwd_from_env'
os.environ['PATRONI_RESTAPI_CONNECT_ADDRESS'] = 'localhost:8080'
os.environ['PATRONI_RESTAPI_LISTEN'] = 'localhost:8080'
os.environ['PATRONI_POSTGRESQL_BIN_POSTGRES'] = 'custom_postgres_bin_from_env'
self.environ = deepcopy(os.environ)
dynamic_config = Config.get_default_config()
dynamic_config['postgresql']['parameters'] = dict(dynamic_config['postgresql']['parameters'])
del dynamic_config['standby_cluster']
dynamic_config['postgresql']['parameters']['wal_keep_segments'] = 8
dynamic_config['postgresql']['use_pg_rewind'] = True
self.config = {
'scope': self.environ['PATRONI_SCOPE'],
'name': HOSTNAME,
'log': {
'level': PatroniLogger.DEFAULT_LEVEL,
'traceback_level': PatroniLogger.DEFAULT_TRACEBACK_LEVEL,
'format': PatroniLogger.DEFAULT_FORMAT,
'max_queue_size': PatroniLogger.DEFAULT_MAX_QUEUE_SIZE
},
'restapi': {
'connect_address': self.environ['PATRONI_RESTAPI_CONNECT_ADDRESS'],
'listen': self.environ['PATRONI_RESTAPI_LISTEN']
},
'bootstrap': {
'dcs': dynamic_config
},
'postgresql': {
'connect_address': IP + ':5432',
'data_dir': NO_VALUE_MSG,
'listen': IP + ':5432',
'pg_hba': ['host all all all md5',
f'host replication {self.environ["PATRONI_REPLICATION_USERNAME"]} all md5'],
'authentication': {'superuser': {'username': self.environ['PATRONI_SUPERUSER_USERNAME'],
'password': self.environ['PATRONI_SUPERUSER_PASSWORD']},
'replication': {'username': self.environ['PATRONI_REPLICATION_USERNAME'],
'password': self.environ['PATRONI_REPLICATION_PASSWORD']},
'rewind': {'username': self.environ['PATRONI_REWIND_USERNAME']}},
'bin_dir': self.environ['PATRONI_POSTGRESQL_BIN_DIR'],
'bin_name': {'postgres': self.environ['PATRONI_POSTGRESQL_BIN_POSTGRES']},
'parameters': {'password_encryption': 'md5'}
}
}
def _set_running_instance_config_vals(self):
# values are taken from tests/__init__.py
conf = {
'scope': 'my_cluster',
'bootstrap': {
'dcs': {
'postgresql': {
'parameters': {
'max_connections': 42,
'max_locks_per_transaction': 73,
'max_replication_slots': 21,
'max_wal_senders': 37,
'wal_level': 'replica',
'wal_keep_segments': None
},
'use_pg_rewind': None
}
}
},
'postgresql': {
'connect_address': f'{IP}:bar',
'listen': '6.6.6.6:1984',
'data_dir': 'data',
'bin_dir': '/bin/dir/from/running',
'parameters': {
'archive_command': 'my archive command',
'hba_file': os.path.join('data', 'pg_hba.conf'),
'ident_file': os.path.join('data', 'pg_ident.conf'),
'password_encryption': None
},
'authentication': {
'superuser': {
'username': 'foobar',
'password': 'qwerty',
'channel_binding': 'prefer',
'gssencmode': 'prefer',
'sslmode': 'prefer'
},
'replication': {
'username': NO_VALUE_MSG,
'password': NO_VALUE_MSG
},
'rewind': None
},
},
'tags': {
'failover_priority': 1,
'noloadbalance': False,
'clonefrom': True,
'nosync': False,
}
}
patch_config(self.config, conf)
def _get_running_instance_open_res(self):
hba_content = '\n'.join(self.config['postgresql']['pg_hba'] + ['#host all all all md5',
' host all all all md5',
'',
'hostall all all md5'])
ident_content = '\n'.join(['# something very interesting', ' '])
self.config['postgresql']['pg_hba'] += ['host all all all md5']
return [
mock_open(read_data=hba_content)(),
mock_open(read_data=ident_content)(),
mock_open(read_data='1984')(),
mock_open()()
]
@patch('os.makedirs')
def test_generate_sample_config_pre_13_dir_creation(self, mock_makedir):
with patch('sys.argv', ['patroni.py', '--generate-sample-config', '/foo/bar.yml']), \
patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 9.4.3")) as pg_bin_mock, \
patch('builtins.open', _mock_open()) as mocked_file, \
self.assertRaises(SystemExit) as e:
_main()
self.assertEqual(self.config, yaml.safe_load(mocked_file().write.call_args_list[0][0][0]))
self.assertEqual(e.exception.code, 0)
mock_makedir.assert_called_once()
pg_bin_mock.assert_called_once_with([os.path.join(self.environ['PATRONI_POSTGRESQL_BIN_DIR'],
self.environ['PATRONI_POSTGRESQL_BIN_POSTGRES']),
'--version'])
@patch('os.makedirs', Mock())
def test_generate_sample_config_16(self):
conf = {
'bootstrap': {
'dcs': {
'postgresql': {
'parameters': {
'wal_keep_size': '128MB',
'wal_keep_segments': None
},
}
}
},
'postgresql': {
'parameters': {
'password_encryption': 'scram-sha-256'
},
'pg_hba': ['host all all all scram-sha-256',
f'host replication {self.environ["PATRONI_REPLICATION_USERNAME"]} all scram-sha-256'],
'authentication': {
'rewind': {
'username': self.environ['PATRONI_REWIND_USERNAME'],
'password': NO_VALUE_MSG}
},
}
}
patch_config(self.config, conf)
with patch('sys.argv', ['patroni.py', '--generate-sample-config', '/foo/bar.yml']), \
patch('builtins.open', _mock_open()) as mocked_file, \
self.assertRaises(SystemExit) as e:
_main()
self.assertEqual(self.config, yaml.safe_load(mocked_file().write.call_args_list[0][0][0]))
self.assertEqual(e.exception.code, 0)
@patch('os.makedirs', Mock())
@patch('sys.stdout')
def test_generate_config_running_instance_16(self, mock_sys_stdout):
self._set_running_instance_config_vals()
with patch('builtins.open', Mock(side_effect=self._get_running_instance_open_res())), \
patch('sys.argv', ['patroni.py', '--generate-config',
'--dsn', 'host=foo port=bar user=foobar password=qwerty']), \
self.assertRaises(SystemExit) as e:
_main()
self.assertEqual(e.exception.code, 0)
self.assertEqual(self.config, yaml.safe_load(mock_sys_stdout.write.call_args_list[0][0][0]))
@patch('os.makedirs', Mock())
@patch('sys.stdout')
def test_generate_config_running_instance_16_connect_from_env(self, mock_sys_stdout):
self._set_running_instance_config_vals()
# su auth params and connect host from env
os.environ['PGCHANNELBINDING'] = \
self.config['postgresql']['authentication']['superuser']['channel_binding'] = 'disable'
conf = {
'scope': 'my_cluster',
'bootstrap': {
'dcs': {
'postgresql': {
'parameters': {
'max_connections': 42,
'max_locks_per_transaction': 73,
'max_replication_slots': 21,
'max_wal_senders': 37,
'wal_level': 'replica',
'wal_keep_segments': None
},
'use_pg_rewind': None
}
}
},
'postgresql': {
'connect_address': f'{IP}:1984',
'authentication': {
'superuser': {
'username': self.environ['PGUSER'],
'password': self.environ['PGPASSWORD'],
'gssencmode': None,
'sslmode': None
},
},
}
}
patch_config(self.config, conf)
with patch('builtins.open', Mock(side_effect=self._get_running_instance_open_res())), \
patch('sys.argv', ['patroni.py', '--generate-config']), \
patch.object(MockConnect, 'server_version', PropertyMock(return_value=160000)), \
self.assertRaises(SystemExit) as e:
_main()
self.assertEqual(e.exception.code, 0)
self.assertEqual(self.config, yaml.safe_load(mock_sys_stdout.write.call_args_list[0][0][0]))
def test_generate_config_running_instance_errors(self):
# 1. Wrong DSN format
with patch('sys.argv', ['patroni.py', '--generate-config', '--dsn', 'host:foo port:bar user:foobar']), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to parse DSN string', e.exception.code)
# 2. User is not a superuser
with patch('sys.argv', ['patroni.py',
'--generate-config', '--dsn', 'host=foo port=bar user=foobar password=pwd_from_dsn']), \
patch.object(MockCursor, 'rowcount', PropertyMock(return_value=0), create=True), \
patch.object(MockConnectionInfo, 'parameter_status', Mock(return_value='off')), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('The provided user does not have superuser privilege', e.exception.code)
# 3. Error while calling postgres --version
with patch('subprocess.check_output', Mock(side_effect=OSError)), \
patch('sys.argv', ['patroni.py', '--generate-sample-config']), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to get postgres version:', e.exception.code)
with patch('sys.argv', ['patroni.py', '--generate-config']):
# 4. empty postmaster.pid
with patch('builtins.open', Mock(side_effect=[mock_open(read_data='hba_content')(),
mock_open(read_data='ident_content')(),
mock_open(read_data='')()])), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to obtain postmaster pid from postmaster.pid file', e.exception.code)
# 5. Failed to open postmaster.pid
with patch('builtins.open', Mock(side_effect=[mock_open(read_data='hba_content')(),
mock_open(read_data='ident_content')(),
OSError])), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Error while reading postmaster.pid file', e.exception.code)
# 6. Invalid postmaster pid
with patch('builtins.open', Mock(side_effect=[mock_open(read_data='hba_content')(),
mock_open(read_data='ident_content')(),
mock_open(read_data='1984')()])), \
patch('psutil.Process.__init__', Mock(return_value=None)), \
patch('psutil.Process.exe', Mock(side_effect=psutil.NoSuchProcess(1984))), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn("Obtained postmaster pid doesn't exist", e.exception.code)
# 7. Failed to open pg_hba
with patch('builtins.open', Mock(side_effect=OSError)), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to read pg_hba.conf', e.exception.code)
# 8. Failed to open pg_ident
with patch('builtins.open', Mock(side_effect=[mock_open(read_data='hba_content')(), OSError])), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to read pg_ident.conf', e.exception.code)
# 9. Failed PG connecttion
from . import psycopg
with patch('patroni.psycopg.connect', side_effect=psycopg.Error), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Failed to establish PostgreSQL connection', e.exception.code)
# 10. An unexpected error
with patch.object(AbstractConfigGenerator, '__init__', side_effect=psycopg.Error), \
self.assertRaises(SystemExit) as e:
_main()
self.assertIn('Unexpected exception', e.exception.code)
def test_get_address(self):
with patch('socket.getaddrinfo', Mock(side_effect=Exception)), \
patch('logging.warning') as mock_warning:
self.assertEqual(get_address(), (NO_VALUE_MSG, NO_VALUE_MSG))
self.assertIn('Failed to obtain address: %r', mock_warning.call_args_list[0][0])
+2 -3
View File
@@ -197,10 +197,9 @@ class TestConsul(unittest.TestCase):
@patch.object(consul.Consul.KV, 'delete', Mock(return_value=True))
def test_delete_leader(self):
leader = self.c.get_cluster().leader
self.c.delete_leader(leader)
self.c.delete_leader()
self.c._name = 'other'
self.c.delete_leader(leader)
self.c.delete_leader()
@patch.object(consul.Consul.KV, 'put', Mock(return_value=True))
def test_initialize(self):
+44 -91
View File
@@ -21,17 +21,10 @@ from .test_ha import get_cluster_initialized_without_leader, get_cluster_initial
get_cluster_initialized_with_only_leader, get_cluster_not_initialized_without_leader, get_cluster, Member
DEFAULT_CONFIG = {
'scope': 'alpha',
'restapi': {'listen': '::', 'certfile': 'a'},
'ctl': {'certfile': 'a'},
'etcd': {'host': 'localhost:2379'},
'citus': {'database': 'citus', 'group': 0},
'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}
}
@patch('patroni.ctl.load_config', Mock(return_value=DEFAULT_CONFIG))
@patch('patroni.ctl.load_config', Mock(return_value={
'scope': 'alpha', 'restapi': {'listen': '::', 'certfile': 'a'}, 'ctl': {'certfile': 'a'},
'etcd': {'host': 'localhost:2379'}, 'citus': {'database': 'citus', 'group': 0},
'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}}))
class TestCtl(unittest.TestCase):
TEST_ROLES = ('master', 'primary', 'leader')
@@ -118,131 +111,91 @@ class TestCtl(unittest.TestCase):
mock_get_dcs.return_value = self.e
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
mock_get_dcs.return_value.set_failover_value = Mock()
# Confirm
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
self.assertEqual(result.exit_code, 0)
assert 'leader' in result.output
# Abort
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\nN')
self.assertEqual(result.exit_code, 1)
# Without a candidate with --force option
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0', '--force'])
self.assertEqual(result.exit_code, 0)
# Scheduled (confirm)
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'],
input='leader\nother\n2300-01-01T12:23:00\ny')
self.assertEqual(result.exit_code, 0)
assert result.exit_code == 0
# Scheduled (abort)
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--scheduled', '2015-01-01T12:00:00+01:00'], input='leader\nother\n\nN')
self.assertEqual(result.exit_code, 1)
# Scheduled with --force option
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--force', '--scheduled', '2015-01-01T12:00:00+01:00'])
self.assertEqual(result.exit_code, 0)
# Scheduled in pause mode
with patch('patroni.config.GlobalConfig.is_paused', PropertyMock(return_value=True)):
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--force', '--scheduled', '2015-01-01T12:00:00'])
self.assertEqual(result.exit_code, 1)
self.assertIn("Can't schedule switchover in the paused state", result.output)
assert result.exit_code == 1
# Aborting switchover, as we answer NO to the confirmation
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\nN')
assert result.exit_code == 1
# Aborting scheduled switchover, as we answer NO to the confirmation
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--scheduled', '2015-01-01T12:00:00+01:00'], input='leader\nother\n\nN')
assert result.exit_code == 1
# Target and source are equal
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nleader\n\ny')
self.assertEqual(result.exit_code, 1)
self.assertIn('Switchover target and source are the same', result.output)
assert result.exit_code == 1
# Candidate is not a member of the cluster
# Reality is not part of this cluster
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nReality\n\ny')
self.assertEqual(result.exit_code, 1)
self.assertIn('Member Reality does not exist in cluster dummy or is tagged as nofailover', result.output)
assert result.exit_code == 1
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0', '--force'])
assert 'Member' in result.output
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--force', '--scheduled', '2015-01-01T12:00:00+01:00'])
assert result.exit_code == 0
# Invalid timestamp
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0', '--force', '--scheduled', 'invalid'])
self.assertEqual(result.exit_code, 1)
self.assertIn('Unable to parse scheduled timestamp', result.output)
assert result.exit_code != 0
# Invalid timestamp
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
'--force', '--scheduled', '2115-02-30T12:00:00+01:00'])
self.assertEqual(result.exit_code, 1)
self.assertIn('Unable to parse scheduled timestamp', result.output)
assert result.exit_code != 0
# Specifying wrong leader
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='dummy')
self.assertEqual(result.exit_code, 1)
self.assertIn('Member dummy is not the leader of cluster dummy', result.output)
assert result.exit_code == 1
# Errors while sending Patroni REST API request
with patch.object(PoolManager, 'request', Mock(side_effect=Exception)):
# Non-responding patroni
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'],
input='leader\nother\n2300-01-01T12:23:00\ny')
self.assertIn('falling back to DCS', result.output)
assert 'falling back to DCS' in result.output
with patch.object(PoolManager, 'request') as mock_api_request:
mock_api_request.return_value.status = 500
with patch.object(PoolManager, 'request') as mocked:
mocked.return_value.status = 500
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
self.assertIn('Switchover failed', result.output)
assert 'Switchover failed' in result.output
mock_api_request.return_value.status = 501
mock_api_request.return_value.data = b'Server does not support this operation'
mocked.return_value.status = 501
mocked.return_value.data = b'Server does not support this operation'
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
self.assertIn('Switchover failed', result.output)
assert 'Switchover failed' in result.output
# No members available
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_only_leader
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
self.assertEqual(result.exit_code, 1)
self.assertIn('No candidates found to switchover to', result.output)
assert result.exit_code == 1
# No leader available
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_without_leader
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
self.assertEqual(result.exit_code, 1)
self.assertIn('This cluster has no leader', result.output)
# Citus cluster, no group number specified
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force'], input='\n')
self.assertEqual(result.exit_code, 1)
self.assertIn('For Citus clusters the --group must me specified', result.output)
assert result.exit_code == 1
@patch('patroni.ctl.get_dcs')
@patch.object(PoolManager, 'request', Mock(return_value=MockResponse()))
@patch('patroni.ctl.request_patroni', Mock(return_value=MockResponse()))
def test_failover(self, mock_get_dcs):
mock_get_dcs.return_value.set_failover_value = Mock()
# No candidate specified
mock_get_dcs.return_value = self.e
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
mock_get_dcs.return_value.set_failover_value = Mock()
result = self.runner.invoke(ctl, ['failover', 'dummy', '--force'], input='\n')
assert 'For Citus clusters the --group must me specified' in result.output
result = self.runner.invoke(ctl, ['failover', 'dummy'], input='0\n')
self.assertIn('Failover could be performed only to a specific candidate', result.output)
cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
# Temp test to check a fallback to switchover if leader is specified
with patch('patroni.ctl._do_failover_or_switchover') as failover_func_mock:
result = self.runner.invoke(ctl, ['failover', '--leader', 'leader', 'dummy'], input='0\n')
self.assertIn('Supplying a leader name using this command is deprecated', result.output)
failover_func_mock.assert_called_once_with(
DEFAULT_CONFIG, 'switchover', 'dummy', None, 'leader', None, False)
# Failover to an async member in sync mode (confirm)
cluster.members.append(Member(0, 'async', 28, {'api_url': 'http://127.0.0.1:8012/patroni'}))
cluster.config.data['synchronous_mode'] = True
mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
result = self.runner.invoke(ctl, ['failover', 'dummy', '--group', '0', '--candidate', 'async'], input='y\ny')
self.assertIn('Are you sure you want to failover to the asynchronous node async', result.output)
# Failover to an async member in sync mode (abort)
mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
result = self.runner.invoke(ctl, ['failover', 'dummy', '--group', '0', '--candidate', 'async'], input='N')
self.assertEqual(result.exit_code, 1)
assert 'Failover could be performed only to a specific candidate' in result.output
@patch('patroni.dcs.dcs_modules', Mock(return_value=['patroni.dcs.dummy', 'patroni.dcs.etcd']))
def test_get_dcs(self):
+1 -1
View File
@@ -313,7 +313,7 @@ class TestEtcd(unittest.TestCase):
self.assertFalse(self.etcd.cancel_initialization())
def test_delete_leader(self):
self.assertFalse(self.etcd.delete_leader(self.etcd.get_cluster().leader))
self.assertFalse(self.etcd.delete_leader())
def test_delete_cluster(self):
self.assertFalse(self.etcd.delete_cluster())
+7 -15
View File
@@ -6,8 +6,8 @@ import urllib3
from mock import Mock, PropertyMock, patch
from patroni.dcs.etcd import DnsCachingResolver
from patroni.dcs.etcd3 import PatroniEtcd3Client, Cluster, Etcd3, Etcd3Client, \
Etcd3Error, Etcd3ClientError, ReAuthenticateMode, RetryFailedError, InvalidAuthToken, Unavailable, \
Unknown, UnsupportedEtcdVersion, UserEmpty, AuthFailed, AuthOldRevision, base64_encode
Etcd3Error, Etcd3ClientError, RetryFailedError, InvalidAuthToken, Unavailable, \
Unknown, UnsupportedEtcdVersion, UserEmpty, AuthFailed, base64_encode
from threading import Thread
from . import SleepException, MockResponse
@@ -161,16 +161,9 @@ class TestPatroniEtcd3Client(BaseTestEtcd3):
mock_urlopen.return_value.content = '{"code":16,"error":"etcdserver: invalid auth token"}'
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
with patch.object(PatroniEtcd3Client, 'authenticate', Mock(return_value=True)):
retry = self.etcd3._retry.copy()
with patch('time.time', Mock(side_effect=[0, 10, 20, 30, 40])):
self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
self.client.username = None
self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
retry = self.etcd3._retry.copy()
self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
mock_urlopen.return_value.content = '{"code":3,"error":"etcdserver: revision of auth store is old"}'
self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
self.assertRaises(AuthOldRevision, retry, self.client.deleteprefix, 'foo', retry=retry)
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
def test__handle_server_response(self):
response = MockResponse()
@@ -305,10 +298,9 @@ class TestEtcd3(BaseTestEtcd3):
self.etcd3.cancel_initialization()
def test_delete_leader(self):
leader = self.etcd3.get_cluster().leader
self.etcd3.delete_leader(leader)
self.etcd3.delete_leader()
self.etcd3._name = 'other'
self.etcd3.delete_leader(leader)
self.etcd3.delete_leader()
def test_delete_cluster(self):
self.etcd3.delete_cluster()
@@ -320,7 +312,7 @@ class TestEtcd3(BaseTestEtcd3):
self.etcd3.set_sync_state_value('', 1)
def test_delete_sync_state(self):
self.etcd3.delete_sync_state('1')
self.etcd3.delete_sync_state()
def test_watch(self):
self.etcd3.set_ttl(10)
+144 -320
View File
@@ -6,7 +6,7 @@ import sys
from mock import Mock, MagicMock, PropertyMock, patch, mock_open
from patroni.collections import CaseInsensitiveSet
from patroni.config import Config
from patroni.dcs import Cluster, ClusterConfig, Failover, Leader, Member, get_dcs, Status, SyncState, TimelineHistory
from patroni.dcs import Cluster, ClusterConfig, Failover, Leader, Member, get_dcs, SyncState, TimelineHistory
from patroni.dcs.etcd import AbstractEtcdClientWithFailover
from patroni.exceptions import DCSError, PostgresConnectionException, PatroniFatalException
from patroni.ha import Ha, _MemberStatus
@@ -39,7 +39,7 @@ def get_cluster(initialize, leader, members, failover, sync, cluster_config=None
history = TimelineHistory(1, '[[1,67197376,"no recovery target specified","' + t + '","foo"]]',
[(1, 67197376, 'no recovery target specified', t, 'foo')])
cluster_config = cluster_config or ClusterConfig(1, {'check_timeline': True}, 1)
return Cluster(initialize, cluster_config, leader, Status(10, None), members, failover, sync, history, failsafe)
return Cluster(initialize, cluster_config, leader, 10, members, failover, sync, history, None, failsafe)
def get_cluster_not_initialized_without_leader(cluster_config=None):
@@ -94,15 +94,12 @@ def get_cluster_initialized_with_leader_and_failsafe():
def get_node_status(reachable=True, in_recovery=True, dcs_last_seen=0,
timeline=2, wal_position=10, nofailover=False,
watchdog_failed=False, failover_priority=1):
watchdog_failed=False):
def fetch_node_status(e):
tags = {}
if nofailover:
tags['nofailover'] = True
tags['failover_priority'] = failover_priority
return _MemberStatus(e, reachable, in_recovery, wal_position,
{'tags': tags, 'watchdog_failed': watchdog_failed,
'dcs_last_seen': dcs_last_seen, 'timeline': timeline})
return _MemberStatus(e, reachable, in_recovery, dcs_last_seen, timeline, wal_position, tags, watchdog_failed)
return fetch_node_status
@@ -154,7 +151,6 @@ zookeeper:
'postmaster_start_time': str(postmaster_start_time)}
self.watchdog = Watchdog(self.config)
self.request = lambda *args, **kwargs: requests_get(args[0].api_url, *args[1:], **kwargs)
self.failover_priority = 1
def run_async(self, func, args=()):
@@ -166,10 +162,9 @@ def run_async(self, func, args=()):
@patch.object(Postgresql, 'is_running', Mock(return_value=MockPostmaster()))
@patch.object(Postgresql, 'is_primary', Mock(return_value=True))
@patch.object(Postgresql, 'is_leader', Mock(return_value=True))
@patch.object(Postgresql, 'timeline_wal_position', Mock(return_value=(1, 10, 1)))
@patch.object(Postgresql, '_cluster_info_state_get', Mock(return_value=10))
@patch.object(Postgresql, 'slots', Mock(return_value={'l': 100}))
@patch.object(Postgresql, 'data_directory_empty', Mock(return_value=False))
@patch.object(Postgresql, 'controldata', Mock(return_value={
'Database system identifier': SYSID,
@@ -229,7 +224,7 @@ class TestHa(PostgresInit):
@patch.object(Postgresql, 'received_timeline', Mock(return_value=None))
def test_touch_member(self):
self.p._major_version = 110000
self.p.is_primary = false
self.p.is_leader = false
self.p.timeline_wal_position = Mock(return_value=(0, 1, 0))
self.p.replica_cached_timeline = Mock(side_effect=Exception)
with patch.object(Postgresql, '_cluster_info_state_get', Mock(return_value='streaming')):
@@ -325,7 +320,7 @@ class TestHa(PostgresInit):
@patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
@patch.object(Rewind, 'can_rewind', PropertyMock(return_value=True))
def test_crash_recovery_before_rewind(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.is_running = false
self.p.controldata = lambda: {'Database cluster state': 'in archive recovery',
'Database system identifier': SYSID}
@@ -370,7 +365,7 @@ class TestHa(PostgresInit):
@patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
def test_start_as_readonly(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.is_healthy = true
self.ha.has_lock = true
self.p.controldata = lambda: {'Database cluster state': 'in production', 'Database system identifier': SYSID}
@@ -388,11 +383,11 @@ class TestHa(PostgresInit):
def test_promoted_by_acquiring_lock(self):
self.ha.is_healthiest_node = true
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
def test_promotion_cancelled_after_pre_promote_failed(self):
self.p.is_primary = false
self.p.is_leader = false
self.p._pre_promote = false
self.ha._is_healthiest_node = true
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
@@ -407,7 +402,7 @@ class TestHa(PostgresInit):
@patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
def test_long_promote(self):
self.ha.has_lock = true
self.p.is_primary = false
self.p.is_leader = false
self.p.set_role('primary')
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
@@ -418,7 +413,7 @@ class TestHa(PostgresInit):
def test_follow_new_leader_after_failing_to_obtain_lock(self):
self.ha.is_healthiest_node = true
self.ha.acquire_lock = false
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'following new leader after trying and failing to obtain lock')
def test_demote_because_not_healthiest(self):
@@ -427,21 +422,21 @@ class TestHa(PostgresInit):
def test_follow_new_leader_because_not_healthiest(self):
self.ha.is_healthiest_node = false
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
@patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
def test_promote_because_have_lock(self):
self.ha.has_lock = true
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because I had the session lock')
def test_promote_without_watchdog(self):
self.ha.has_lock = true
self.p.is_primary = true
self.p.is_leader = true
with patch.object(Watchdog, 'activate', Mock(return_value=False)):
self.assertEqual(self.ha.run_cycle(), 'Demoting self because watchdog could not be activated')
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'Not promoting self because watchdog could not be activated')
def test_leader_with_lock(self):
@@ -467,12 +462,12 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'demoted self because failed to update leader lock in DCS')
with patch.object(Ha, '_get_node_to_follow', Mock(side_effect=DCSError('foo'))):
self.assertEqual(self.ha.run_cycle(), 'demoted self because failed to update leader lock in DCS')
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'not promoting because failed to update leader lock in DCS')
@patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
def test_follow(self):
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), a secondary, and following a leader ()')
self.ha.patroni.replicatefrom = "foo"
self.p.config.check_recovery_conf = Mock(return_value=(True, False))
@@ -489,13 +484,13 @@ class TestHa(PostgresInit):
def test_follow_in_pause(self):
self.ha.is_paused = true
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'PAUSE: no action. I am (postgresql0)')
@patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
@patch.object(Rewind, 'can_rewind', PropertyMock(return_value=True))
def test_follow_triggers_rewind(self):
self.p.is_primary = false
self.p.is_leader = false
self.ha._rewind.trigger_check_diverged_lsn()
self.ha.cluster = get_cluster_initialized_with_leader()
self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')
@@ -549,7 +544,7 @@ class TestHa(PostgresInit):
self.ha.global_config = self.ha.patroni.config.get_global_config(self.ha.cluster)
self.ha.update_failsafe({'name': 'leader', 'api_url': 'http://127.0.0.1:8008/patroni',
'conn_url': 'postgres://127.0.0.1:5432/postgres', 'slots': {'foo': 1000}})
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'DCS is not accessible')
def test_no_dcs_connection_replica_failsafe_not_enabled_but_active(self):
@@ -557,7 +552,7 @@ class TestHa(PostgresInit):
self.ha.cluster = get_cluster_initialized_with_leader()
self.ha.update_failsafe({'name': 'leader', 'api_url': 'http://127.0.0.1:8008/patroni',
'conn_url': 'postgres://127.0.0.1:5432/postgres', 'slots': {'foo': 1000}})
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'DCS is not accessible')
def test_update_failsafe(self):
@@ -596,9 +591,9 @@ class TestHa(PostgresInit):
self.ha.cluster = get_cluster_not_initialized_without_leader()
self.e.initialize = true
self.assertEqual(self.ha.bootstrap(), 'trying to bootstrap a new cluster')
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'waiting for end of recovery after bootstrap')
self.p.is_primary = true
self.p.is_leader = true
self.ha.is_synchronous_mode = true
self.assertEqual(self.ha.run_cycle(), 'running post_bootstrap')
self.assertEqual(self.ha.run_cycle(), 'initialized a new cluster')
@@ -618,7 +613,7 @@ class TestHa(PostgresInit):
self.ha.cluster = get_cluster_not_initialized_without_leader()
self.e.initialize = true
self.ha.bootstrap()
self.p.is_primary = true
self.p.is_leader = true
with patch.object(Watchdog, 'activate', Mock(return_value=False)), \
patch('patroni.ha.logger.error') as mock_logger:
self.assertEqual(self.ha.post_bootstrap(), 'running post_bootstrap')
@@ -692,289 +687,112 @@ class TestHa(PostgresInit):
@patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
def test_manual_failover_from_leader(self):
self.ha.has_lock = true # I am the leader
# to me
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', self.p.name, None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
mock_warning.assert_called_with('%s: I am already the leader, no need to %s', 'manual failover', 'failover')
# to a non-existent candidate
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', 'blabla', None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
mock_warning.assert_called_with(
'%s: no healthy members found, %s is not possible', 'manual failover', 'failover')
# to an existent candidate
self.ha.fetch_node_status = get_node_status()
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', 'b', None))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.ha.has_lock = true
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', '', None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', self.p.name, None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', 'blabla', None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
f = Failover(0, self.p.name, '', None)
self.ha.cluster = get_cluster_initialized_with_leader(f)
self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
# to a candidate on an older timeline
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(timeline=1)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0],
('Timeline %s of member %s is behind the cluster timeline %s', 1, 'b', 2))
# to a lagging candidate
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(wal_position=1)
self.ha.cluster.config.data.update({'maximum_lag_on_failover': 5})
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0],
('Member %s exceeds maximum replication lag', 'b'))
self.ha.cluster.members.pop()
@patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
def test_manual_switchover_from_leader(self):
self.ha.has_lock = true # I am the leader
self.ha.fetch_node_status = get_node_status()
# different leader specified in failover key, no candidate
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', '', None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
mock_warning.assert_called_with(
'%s: leader name does not match: %s != %s', 'switchover', 'blabla', 'postgresql0')
# no candidate
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, '', None))
self.assertEqual(self.ha.run_cycle(), 'switchover: demoting myself')
self.ha._rewind.rewind_or_reinitialize_needed_and_possible = true
self.assertEqual(self.ha.run_cycle(), 'switchover: demoting myself')
self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
self.ha.fetch_node_status = get_node_status(nofailover=True)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.ha.fetch_node_status = get_node_status(watchdog_failed=True)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.ha.fetch_node_status = get_node_status(timeline=1)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.ha.fetch_node_status = get_node_status(wal_position=1)
self.ha.cluster.config.data.update({'maximum_lag_on_failover': 5})
self.ha.global_config = self.ha.patroni.config.get_global_config(self.ha.cluster)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
# manual failover from the previous leader to us won't happen if we hold the nofailover flag
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, None))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
# other members with failover_limitation_s
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(nofailover=True)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0], ('Member %s is %s', 'leader', 'not allowed to promote'))
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(watchdog_failed=True)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0], ('Member %s is %s', 'leader', 'not watchdog capable'))
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(timeline=1)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0],
('Timeline %s of member %s is behind the cluster timeline %s', 1, 'leader', 2))
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(wal_position=1)
self.ha.cluster.config.data.update({'maximum_lag_on_failover': 5})
self.ha.global_config = self.ha.patroni.config.get_global_config(self.ha.cluster)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertEqual(mock_info.call_args_list[0][0], ('Member %s exceeds maximum replication lag', 'leader'))
# Failover scheduled time must include timezone
scheduled = datetime.datetime.now()
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.ha.run_cycle()
@patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
def test_scheduled_switchover_from_leader(self):
self.ha.has_lock = true # I am the leader
self.ha.fetch_node_status = get_node_status()
# switchover scheduled time must include timezone
with patch('patroni.ha.logger.warning') as mock_warning:
scheduled = datetime.datetime.now()
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'blabla', scheduled))
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
self.assertIn('Incorrect value of scheduled_at: %s', mock_warning.call_args_list[0][0])
# scheduled now
scheduled = datetime.datetime.utcnow().replace(tzinfo=tzutc)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'b', scheduled))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.assertEqual('switchover: demoting myself', self.ha.run_cycle())
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
# scheduled in the future
with patch('patroni.ha.logger.info') as mock_info:
scheduled = scheduled + datetime.timedelta(seconds=30)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'blabla', scheduled))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.assertIn('Awaiting %s at %s (in %.0f seconds)', mock_info.call_args_list[0][0])
scheduled = scheduled + datetime.timedelta(seconds=30)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
# stale value
with patch('patroni.ha.logger.warning') as mock_warning:
scheduled = scheduled + datetime.timedelta(seconds=-600)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'b', scheduled))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.assertIn('Found a stale %s value, cleaning up: %s', mock_warning.call_args_list[0][0])
scheduled = scheduled + datetime.timedelta(seconds=-600)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
def test_manual_switchover_from_leader_in_pause(self):
self.ha.has_lock = true # I am the leader
self.ha.is_paused = true
# no candidate
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, '', None))
with patch('patroni.ha.logger.warning') as mock_warning:
self.assertEqual('PAUSE: no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
mock_warning.assert_called_with(
'%s is possible only to a specific candidate in a paused state', 'Switchover')
scheduled = None
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
def test_manual_failover_from_leader_in_pause(self):
self.ha.has_lock = true
self.ha.fetch_node_status = get_node_status()
self.ha.is_paused = true
# failover from me, candidate is healthy
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, None, 'b', None))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.assertEqual('PAUSE: manual failover: demoting myself', self.ha.run_cycle())
self.ha.cluster.members.pop()
scheduled = datetime.datetime.now()
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, 'blabla', self.p.name, scheduled))
self.assertEqual('PAUSE: no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, '', None))
self.assertEqual('PAUSE: no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
def test_manual_failover_from_leader_in_synchronous_mode(self):
self.ha.is_synchronous_mode = true
self.ha.process_sync_replication = Mock()
self.ha.fetch_node_status = get_node_status()
# I am the leader
self.p.is_primary = true
self.p.is_leader = true
self.ha.has_lock = true
# the candidate is not in sync members but we allow failover to an async candidate
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, None, 'b', None), sync=(self.p.name, 'a'))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.ha.is_synchronous_mode = true
self.ha.is_failover_possible = false
self.ha.process_sync_replication = Mock()
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'a', None), (self.p.name, None))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'a', None), (self.p.name, 'a'))
self.ha.is_failover_possible = true
self.assertEqual('manual failover: demoting myself', self.ha.run_cycle())
self.ha.cluster.members.pop()
def test_manual_switchover_from_leader_in_synchronous_mode(self):
self.ha.is_synchronous_mode = true
self.ha.process_sync_replication = Mock()
# I am the leader
self.p.is_primary = true
self.ha.has_lock = true
# candidate specified is not in sync members
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'a', None),
sync=(self.p.name, 'blabla'))
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.assertEqual(mock_warning.call_args_list[0][0],
('%s candidate=%s does not match with sync_standbys=%s', 'Switchover', 'a', 'blabla'))
# the candidate is in sync members and is healthy
self.ha.fetch_node_status = get_node_status(wal_position=305419896)
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, self.p.name, 'a', None),
sync=(self.p.name, 'a'))
self.ha.cluster.members.append(Member(0, 'a', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.assertEqual('switchover: demoting myself', self.ha.run_cycle())
# the candidate is in sync members but is not healthy
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(nofailover=true)
self.assertEqual('no action. I am (postgresql0), the leader with the lock', self.ha.run_cycle())
self.assertEqual(mock_info.call_args_list[0][0], ('Member %s is %s', 'a', 'not allowed to promote'))
def test_manual_failover_process_no_leader(self):
self.p.is_primary = false
self.p.is_leader = false
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', self.p.name, None))
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'leader', None))
self.p.set_role('replica')
# failover to another member, fetch_node_status for candidate fails
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'leader', None))
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
self.assertEqual(mock_warning.call_args_list[1][0],
('%s: member %s is %s', 'manual failover', 'leader', 'not reachable'))
# failover to another member, candidate is accessible, in_recovery
self.p.set_role('replica')
self.ha.fetch_node_status = get_node_status()
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
self.ha.fetch_node_status = get_node_status() # accessible, in_recovery
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# set nofailover flag to True for all members of the cluster
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, self.p.name, '', None))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
self.ha.fetch_node_status = get_node_status(reachable=False) # inaccessible, in_recovery
self.p.set_role('replica')
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
# set failover flag to True for all members of the cluster
# this should elect the current member, as we are not going to call the API for it.
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None))
self.ha.fetch_node_status = get_node_status(nofailover=True)
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
# failover to me but I am set to nofailover. In no case I should be elected as a leader
self.ha.fetch_node_status = get_node_status(nofailover=True) # accessible, in_recovery
self.p.set_role('replica')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'postgresql0', None))
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
# same as previous, but set the current member to nofailover. In no case it should be elected as a leader
self.ha.patroni.nofailover = True
self.assertEqual(self.ha.run_cycle(), 'following a different leader because I am not allowed to promote')
self.ha.patroni.nofailover = False
# failover to another member that is on an older timeline (only failover_limitation() is checked)
with patch('patroni.ha.logger.info') as mock_info:
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'b', None))
self.ha.cluster.members.append(Member(0, 'b', 28, {'api_url': 'http://127.0.0.1:8011/patroni'}))
self.ha.fetch_node_status = get_node_status(timeline=1)
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
mock_info.assert_called_with('%s: to %s, i am %s', 'manual failover', 'b', 'postgresql0')
# failover to another member lagging behind the cluster_lsn (only failover_limitation() is checked)
with patch('patroni.ha.logger.info') as mock_info:
self.ha.cluster.config.data.update({'maximum_lag_on_failover': 5})
self.ha.fetch_node_status = get_node_status(wal_position=1)
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
mock_info.assert_called_with('%s: to %s, i am %s', 'manual failover', 'b', 'postgresql0')
def test_manual_switchover_process_no_leader(self):
self.p.is_primary = false
self.p.set_role('replica')
# I was the leader, other members are healthy
self.ha.fetch_node_status = get_node_status()
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, self.p.name, '', None))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# I was the leader, I am the only healthy member
with patch('patroni.ha.logger.info') as mock_info:
self.ha.fetch_node_status = get_node_status(reachable=False) # inaccessible, in_recovery
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
self.assertEqual(mock_info.call_args_list[0][0], ('Member %s is %s', 'leader', 'not reachable'))
self.assertEqual(mock_info.call_args_list[1][0], ('Member %s is %s', 'other', 'not reachable'))
def test_manual_failover_process_no_leader_in_synchronous_mode(self):
self.ha.is_synchronous_mode = true
self.p.is_primary = false
self.ha.fetch_node_status = get_node_status(nofailover=True) # other nodes are not healthy
self.p.is_leader = false
# manual failover when our name (postgresql0) isn't in the /sync key and the candidate node is not available
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None),
sync=('leader1', 'blabla'))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# manual failover when the candidate node isn't available but our name is in the /sync key
# while other sync node is nofailover
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None),
sync=('leader1', 'postgresql0'))
self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(), CaseInsensitiveSet()))
self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
self.assertEqual(mock_warning.call_args_list[0][0],
('%s: member %s is %s', 'manual failover', 'other', 'not allowed to promote'))
# manual failover to our node (postgresql0),
# which name is not in sync nodes list (some sync nodes are available)
self.p.set_role('replica')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'postgresql0', None),
sync=('leader1', 'other'))
self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(['leader1']),
CaseInsensitiveSet(['leader1'])))
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
def test_manual_switchover_process_no_leader_in_synchronous_mode(self):
self.ha.is_synchronous_mode = true
self.p.is_primary = false
# to a specific node, which name doesn't match our name (postgresql0)
# switchover to a specific node, which name doesn't match our name (postgresql0)
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'other', None))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# to our node (postgresql0), which name is not in sync nodes list
# switchover to our node (postgresql0), which name is not in sync nodes list
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'postgresql0', None),
sync=('leader1', 'blabla'))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# without candidate, our name (postgresql0) is not in the sync nodes list
# switchover from a specific leader, but our name (postgresql0) is not in the sync nodes list
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', '', None),
sync=('leader', 'blabla'))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
@@ -984,39 +802,53 @@ class TestHa(PostgresInit):
sync=('postgresql0'))
self.ha.patroni.nofailover = True
self.assertEqual(self.ha.run_cycle(), 'following a different leader because I am not allowed to promote')
self.ha.patroni.nofailover = False
# manual failover when our name (postgresql0) isn't in the /sync key and the `other` node is not available
self.ha.fetch_node_status = get_node_status(nofailover=True) # accessible, in_recovery
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None),
sync=('leader1', 'blabla'))
self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
# manual failover when the `other` node isn't available but our name is in the /sync key
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None),
sync=('leader1', 'postgresql0'))
self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(), CaseInsensitiveSet()))
self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
# manual failover to our node (postgresql0),
# which name is not in sync nodes list (the leader and all sync nodes are not available)
self.p.set_role('replica')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'postgresql0', None),
sync=('leader1', 'other'))
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
# manual failover to our node (postgresql0),
# which name is not in sync nodes list (some sync nodes are available)
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'postgresql0', None),
sync=('leader1', 'other'))
self.p.set_role('replica')
self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(['leader1']),
CaseInsensitiveSet(['leader1'])))
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
def test_manual_failover_process_no_leader_in_pause(self):
self.ha.is_paused = true
# I am running as primary, cluster is unlocked, the candidate is allowed to promote
# but we are in pause
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
def test_manual_switchover_process_no_leader_in_pause(self):
self.ha.is_paused = true
# I am running as primary, cluster is unlocked, no candidate specified
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', '', None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
# the candidate is not running
with patch('patroni.ha.logger.warning') as mock_warning:
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'blabla', None))
self.assertEqual('PAUSE: acquired session lock as a leader', self.ha.run_cycle())
self.assertEqual(
mock_warning.call_args_list[0][0],
('%s: removing failover key because failover candidate is not running', 'switchover'))
# switchover to me, I am not leader
self.p.is_primary = false
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'blabla', None))
self.assertEqual('PAUSE: acquired session lock as a leader', self.ha.run_cycle())
self.p.is_leader = false
self.p.set_role('replica')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', self.p.name, None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: promoted self to leader by acquiring session lock')
def test_is_healthiest_node(self):
self.ha.is_failsafe_mode = true
self.ha.state_handler.is_primary = false
self.ha.state_handler.is_leader = false
self.ha.patroni.nofailover = False
self.ha.fetch_node_status = get_node_status()
self.ha.dcs._last_failsafe = {'foo': ''}
@@ -1030,7 +862,7 @@ class TestHa(PostgresInit):
self.assertFalse(self.ha.is_healthiest_node())
def test__is_healthiest_node(self):
self.p.is_primary = false
self.p.is_leader = false
self.ha.cluster = get_cluster_initialized_without_leader(sync=('postgresql1', self.p.name))
self.ha.global_config = self.ha.patroni.config.get_global_config(self.ha.cluster)
self.assertTrue(self.ha._is_healthiest_node(self.ha.old_cluster.members))
@@ -1038,11 +870,6 @@ class TestHa(PostgresInit):
self.assertTrue(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.fetch_node_status = get_node_status(in_recovery=False) # accessible, not in_recovery
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.fetch_node_status = get_node_status(failover_priority=2) # accessible, in_recovery, higher priority
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
# if there is a higher-priority node but it has a lower WAL position then this node should race
self.ha.fetch_node_status = get_node_status(failover_priority=6, wal_position=9)
self.assertTrue(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.fetch_node_status = get_node_status(wal_position=11) # accessible, in_recovery, wal position ahead
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
# in synchronous_mode consider itself healthy if the former leader is accessible in read-only and ahead of us
@@ -1058,9 +885,7 @@ class TestHa(PostgresInit):
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.patroni.nofailover = True
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.patroni.nofailover = None
self.ha.patroni.failover_priority = 0
self.assertFalse(self.ha._is_healthiest_node(self.ha.old_cluster.members))
self.ha.patroni.nofailover = False
def test_fetch_node_status(self):
member = Member(0, 'test', 1, {'api_url': 'http://127.0.0.1:8011/patroni'})
@@ -1136,7 +961,7 @@ class TestHa(PostgresInit):
self.assertTrue(self.ha.restart_matches("replica", "9.5.2", False))
def test_process_healthy_cluster_in_pause(self):
self.p.is_primary = false
self.p.is_leader = false
self.ha.is_paused = true
self.p.name = 'leader'
self.ha.cluster = get_cluster_initialized_with_leader()
@@ -1147,7 +972,7 @@ class TestHa(PostgresInit):
@patch('patroni.postgresql.mtime', Mock(return_value=1588316884))
@patch('builtins.open', mock_open(read_data='1\t0/40159C0\tno recovery target specified\n'))
def test_process_healthy_standby_cluster_as_standby_leader(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'leader'
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.p.config.check_recovery_conf = Mock(return_value=(False, False))
@@ -1159,7 +984,7 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'promoted self to a standby leader because i had the session lock')
def test_process_healthy_standby_cluster_as_cascade_replica(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'replica'
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.assertEqual(self.ha.run_cycle(),
@@ -1169,7 +994,7 @@ class TestHa(PostgresInit):
@patch.object(Cluster, 'is_unlocked', Mock(return_value=True))
def test_process_unhealthy_standby_cluster_as_standby_leader(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'leader'
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.ha.sysid_valid = true
@@ -1179,13 +1004,13 @@ class TestHa(PostgresInit):
@patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
@patch.object(Rewind, 'can_rewind', PropertyMock(return_value=True))
def test_process_unhealthy_standby_cluster_as_cascade_replica(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'replica'
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_member:'))
def test_recover_unhealthy_leader_in_standby_cluster(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'leader'
self.p.is_running = false
self.p.follow = false
@@ -1194,7 +1019,7 @@ class TestHa(PostgresInit):
@patch.object(Cluster, 'is_unlocked', Mock(return_value=True))
def test_recover_unhealthy_unlocked_standby_cluster(self):
self.p.is_primary = false
self.p.is_leader = false
self.p.name = 'leader'
self.p.is_running = false
self.p.follow = false
@@ -1254,7 +1079,7 @@ class TestHa(PostgresInit):
check_calls([(update_lock, True), (demote, True)])
self.ha.has_lock = false
self.p.is_primary = false
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(),
'no action. I am (postgresql0), a secondary, and following a leader (leader)')
check_calls([(update_lock, False), (demote, False)])
@@ -1265,7 +1090,7 @@ class TestHa(PostgresInit):
f = Failover(0, self.p.name, '', None)
self.ha.cluster = get_cluster_initialized_with_leader(f)
self.ha.fetch_node_status = get_node_status() # accessible, in_recovery
self.assertEqual(self.ha.run_cycle(), 'switchover: demoting myself')
self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
@patch('patroni.ha.Ha.demote')
def test_failover_immediately_on_zero_primary_start_timeout(self, demote):
@@ -1388,7 +1213,7 @@ class TestHa(PostgresInit):
self.ha.is_synchronous_mode = true
mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
self.p.is_primary = false
self.p.is_leader = false
self.p.set_role('replica')
self.ha.has_lock = true
mock_write_sync = self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
@@ -1411,7 +1236,7 @@ class TestHa(PostgresInit):
def test_unhealthy_sync_mode(self):
self.ha.is_synchronous_mode = true
self.p.is_primary = false
self.p.is_leader = false
self.p.set_role('replica')
self.p.name = 'other'
self.ha.cluster = get_cluster_initialized_without_leader(sync=('leader', 'other2'))
@@ -1442,7 +1267,7 @@ class TestHa(PostgresInit):
self.ha.is_synchronous_mode = true
self.p.name = 'other'
self.p.is_primary = false
self.p.is_leader = false
self.p.set_role('replica')
mock_restart = self.p.restart = Mock(return_value=True)
self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
@@ -1578,7 +1403,7 @@ class TestHa(PostgresInit):
@patch('sys.exit', return_value=1)
def test_abort_join(self, exit_mock):
self.ha.cluster = get_cluster_not_initialized_without_leader()
self.p.is_primary = false
self.p.is_leader = false
self.ha.run_cycle()
exit_mock.assert_called_once_with(1)
@@ -1592,7 +1417,6 @@ class TestHa(PostgresInit):
@patch('patroni.psycopg.connect', psycopg_connect)
def test_permanent_logical_slots_after_promote(self):
self.p._major_version = 110000
config = ClusterConfig(1, {'slots': {'l': {'database': 'postgres', 'plugin': 'test_decoding'}}}, 1)
self.p.name = 'other'
self.ha.cluster = get_cluster_initialized_without_leader(cluster_config=config)
@@ -1639,7 +1463,7 @@ class TestHa(PostgresInit):
@patch.object(SlotsHandler, 'sync_replication_slots', Mock(return_value=['ls']))
def test_follow_copy(self):
self.ha.cluster.config.data['slots'] = {'ls': {'database': 'a', 'plugin': 'b'}}
self.p.is_primary = false
self.p.is_leader = false
self.assertTrue(self.ha.run_cycle().startswith('Copying logical slots'))
def test_acquire_lock(self):
+2 -17
View File
@@ -63,7 +63,7 @@ def mock_list_namespaced_pod(*args, **kwargs):
metadata = k8s_client.V1ObjectMeta(resource_version='1', labels={'f': 'b', Kubernetes._CITUS_LABEL: '1'},
name='p-0', annotations={'status': '{}'},
uid='964dfeae-e79b-4476-8a5a-1920b5c2a69d')
status = k8s_client.V1PodStatus(pod_ip='10.0.0.1')
status = k8s_client.V1PodStatus(pod_ip='10.0.0.0')
spec = k8s_client.V1PodSpec(hostname='p-0', node_name='kind-control-plane', containers=[])
items = [k8s_client.V1Pod(metadata=metadata, status=status, spec=spec)]
return k8s_client.V1PodList(items=items, kind='PodList')
@@ -326,7 +326,7 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
self.k.initialize()
def test_delete_leader(self):
self.k.delete_leader(self.k.get_cluster().leader, 1)
self.k.delete_leader(1)
def test_cancel_initialization(self):
self.k.cancel_initialization()
@@ -356,20 +356,6 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
mock_warning.assert_called_once()
class TestKubernetesEndpointsNoPodIP(BaseTestKubernetes):
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_endpoints', mock_list_namespaced_endpoints, create=True)
def setUp(self, config=None):
super(TestKubernetesEndpointsNoPodIP, self).setUp({'use_endpoints': True})
@patch.object(k8s_client.CoreV1Api, 'patch_namespaced_endpoints', create=True)
def test_update_leader(self, mock_patch_namespaced_endpoints):
leader = self.k.get_cluster().leader
self.assertIsNotNone(self.k.update_leader(leader, '123', failsafe={'foo': 'bar'}))
args = mock_patch_namespaced_endpoints.call_args[0]
self.assertEqual(args[2].subsets[0].addresses[0].target_ref.resource_version, '1')
self.assertEqual(args[2].subsets[0].addresses[0].ip, '10.0.0.1')
class TestKubernetesEndpoints(BaseTestKubernetes):
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_endpoints', mock_list_namespaced_endpoints, create=True)
@@ -382,7 +368,6 @@ class TestKubernetesEndpoints(BaseTestKubernetes):
self.assertIsNotNone(self.k.update_leader(leader, '123', failsafe={'foo': 'bar'}))
args = mock_patch_namespaced_endpoints.call_args[0]
self.assertEqual(args[2].subsets[0].addresses[0].target_ref.resource_version, '10')
self.assertEqual(args[2].subsets[0].addresses[0].ip, '10.0.0.0')
self.k._kinds._object_cache['test'].subsets[:] = []
self.assertIsNotNone(self.k.update_leader(leader, '123'))
self.k._kinds._object_cache['test'].metadata.annotations['leader'] = 'p-1'
+9 -48
View File
@@ -15,7 +15,8 @@ from patroni.dcs.etcd import AbstractEtcdClientWithFailover
from patroni.exceptions import DCSError
from patroni.postgresql import Postgresql
from patroni.postgresql.config import ConfigHandler
from patroni.__main__ import check_psycopg, Patroni, main as _main
from patroni import check_psycopg
from patroni.__main__ import Patroni, main as _main
from threading import Thread
from . import psycopg_connect, SleepException
@@ -24,16 +25,10 @@ from .test_postgresql import MockPostmaster
def mock_import(*args, **kwargs):
ret = Mock()
ret.__version__ = '2.5.3.dev1 a b c' if args[0] == 'psycopg2' else '3.1.0'
return ret
def mock_import2(*args, **kwargs):
if args[0] == 'psycopg2':
if args[0] == 'psycopg':
raise ImportError
ret = Mock()
ret.__version__ = '0.1.2'
ret.__version__ = '2.5.3.dev1 a b c'
return ret
@@ -179,42 +174,10 @@ class TestPatroni(unittest.TestCase):
self.assertTrue(self.p.noloadbalance)
def test_nofailover(self):
for (nofailover, failover_priority, expected) in [
# Without any tags, default is False
(None, None, False),
# Setting `nofailover: True` has precedence
(True, 0, True),
(True, 1, True),
# Similarly, setting `nofailover: False` has precedence
(False, 0, False),
(False, 1, False),
# Only when we have `nofailover: None` should we got based on priority
(None, 0, True),
(None, 1, False),
]:
with self.subTest(nofailover=nofailover, failover_priority=failover_priority, expected=expected):
self.p.tags['nofailover'] = nofailover
self.p.tags['failover_priority'] = failover_priority
self.assertEqual(self.p.nofailover, expected)
def test_failover_priority(self):
for (nofailover, failover_priority, expected) in [
# Without any tags, default is 1
(None, None, 1),
# Setting `nofailover: True` has precedence (value 0)
(True, 0, 0),
(True, 1, 0),
# Setting `nofailover: False` and `failover_priority: None` gives 1
(False, None, 1),
# Normal function of failover_priority
(None, 0, 0),
(None, 1, 1),
(None, 2, 2),
]:
with self.subTest(nofailover=nofailover, failover_priority=failover_priority, expected=expected):
self.p.tags['nofailover'] = nofailover
self.p.tags['failover_priority'] = failover_priority
self.assertEqual(self.p.failover_priority, expected)
self.p.tags['nofailover'] = True
self.assertTrue(self.p.nofailover)
self.p.tags['nofailover'] = None
self.assertFalse(self.p.nofailover)
def test_replicatefrom(self):
self.assertIsNone(self.p.replicatefrom)
@@ -223,7 +186,7 @@ class TestPatroni(unittest.TestCase):
def test_reload_config(self):
self.p.reload_config()
self.p._get_tags = Mock(side_effect=Exception)
self.p.get_tags = Mock(side_effect=Exception)
self.p.reload_config(local=True)
def test_nosync(self):
@@ -242,8 +205,6 @@ class TestPatroni(unittest.TestCase):
with patch('builtins.__import__', Mock(side_effect=ImportError)):
self.assertRaises(SystemExit, check_psycopg)
with patch('builtins.__import__', mock_import):
self.assertIsNone(check_psycopg())
with patch('builtins.__import__', mock_import2):
self.assertRaises(SystemExit, check_psycopg)
def test_ensure_unique_name(self):
+12 -25
View File
@@ -374,11 +374,11 @@ class TestPostgresql(BaseTestPostgresql):
self.assertRaises(psycopg.ProgrammingError, self.p.query, 'blabla')
@patch.object(Postgresql, 'pg_isready', Mock(return_value=STATE_REJECT))
def test_is_primary(self):
self.assertTrue(self.p.is_primary())
def test_is_leader(self):
self.assertTrue(self.p.is_leader())
self.p.reset_cluster_info_state(None)
with patch.object(Postgresql, '_query', Mock(side_effect=RetryFailedError(''))):
self.assertFalse(self.p.is_primary())
self.assertFalse(self.p.is_leader())
@patch.object(Postgresql, 'controldata', Mock(return_value={'Database cluster state': 'shut down',
'Latest checkpoint location': '0/1ADBC18',
@@ -472,7 +472,7 @@ class TestPostgresql(BaseTestPostgresql):
self.assertIsNone(self.p.call_nowait(CallbackAction.ON_START))
@patch.object(Postgresql, 'is_running', Mock(return_value=MockPostmaster()))
def test_is_primary_exception(self):
def test_is_leader_exception(self):
self.p.start()
self.p.query = Mock(side_effect=psycopg.OperationalError("not supported"))
self.assertTrue(self.p.stop())
@@ -556,7 +556,9 @@ class TestPostgresql(BaseTestPostgresql):
@patch('time.sleep', Mock())
@patch.object(Postgresql, 'is_running', Mock(return_value=True))
def test_reload_config(self):
@patch.object(MockCursor, 'fetchone')
def test_reload_config(self, mock_fetchone):
mock_fetchone.return_value = (1,)
parameters = self._PARAMETERS.copy()
parameters.pop('f.oo')
parameters['wal_buffers'] = '512'
@@ -564,14 +566,9 @@ class TestPostgresql(BaseTestPostgresql):
'authentication': {},
'retry_timeout': 10, 'listen': '*', 'krbsrvname': 'postgres', 'parameters': parameters}
self.p.reload_config(config)
mock_fetchone.side_effect = Exception
parameters['b.ar'] = 'bar'
with patch.object(MockCursor, 'fetchall',
Mock(side_effect=[[('wal_block_size', '8191', None, 'integer', 'internal'),
('wal_segment_size', '2048', '8kB', 'integer', 'internal'),
('shared_buffers', '16384', '8kB', 'integer', 'postmaster'),
('wal_buffers', '-1', '8kB', 'integer', 'postmaster'),
('port', '5433', None, 'integer', 'postmaster')], Exception])):
self.p.reload_config(config)
self.p.reload_config(config)
parameters['autovacuum'] = 'on'
self.p.reload_config(config)
parameters['autovacuum'] = 'off'
@@ -588,10 +585,7 @@ class TestPostgresql(BaseTestPostgresql):
self.assertEqual(self.p.config.local_replication_address, {'host': '/tmp', 'port': '5432'})
self.p.config._server_parameters.pop('unix_socket_directories')
self.p.config.resolve_connection_addresses()
self.assertEqual(self.p.connection_pool.conn_kwargs, {'connect_timeout': 3, 'dbname': 'postgres',
'fallback_application_name': 'Patroni',
'options': '-c statement_timeout=2000',
'password': 'test', 'port': '5432', 'user': 'foo'})
self.assertEqual(self.p.config._local_address, {'port': '5432'})
@patch.object(Postgresql, '_version_file_exists', Mock(return_value=True))
def test_get_major_version(self):
@@ -602,7 +596,7 @@ class TestPostgresql(BaseTestPostgresql):
def test_postmaster_start_time(self):
now = datetime.datetime.now()
with patch.object(MockCursor, "fetchall", Mock(return_value=[(now, True, '', '', '', '', False)])):
with patch.object(MockCursor, "fetchone", Mock(return_value=(now, True, '', '', '', '', False))):
self.assertEqual(self.p.postmaster_start_time(), now.isoformat(sep=' '))
t = Thread(target=self.p.postmaster_start_time)
t.start()
@@ -965,7 +959,7 @@ class TestPostgresql2(BaseTestPostgresql):
@patch('patroni.postgresql.CallbackExecutor', Mock())
@patch.object(Postgresql, 'get_major_version', Mock(return_value=140000))
@patch.object(Postgresql, 'is_running', Mock(return_value=True))
@patch.object(Postgresql, 'is_primary', Mock(return_value=False))
@patch.object(Postgresql, 'is_leader', Mock(return_value=False))
def setUp(self):
super(TestPostgresql2, self).setUp()
@@ -974,10 +968,3 @@ class TestPostgresql2(BaseTestPostgresql):
gucs = self.p.available_gucs
self.assertIsInstance(gucs, CaseInsensitiveSet)
self.assertEqual(gucs, mock_available_gucs.return_value)
def test_cluster_info_query(self):
self.assertIn('diff(pg_catalog.pg_current_wal_flush_lsn(', self.p.cluster_info_query)
self.p._major_version = 90600
self.assertIn('diff(pg_catalog.pg_current_xlog_flush_location(', self.p.cluster_info_query)
self.p._major_version = 90500
self.assertIn('diff(pg_catalog.pg_current_xlog_location(', self.p.cluster_info_query)
+4 -4
View File
@@ -142,25 +142,25 @@ class TestRaft(unittest.TestCase):
raft._citus_group = '1'
self.assertTrue(raft.manual_failover('foo', 'bar'))
raft._citus_group = '0'
self.assertTrue(raft.take_leader())
cluster = raft.get_cluster()
self.assertIsInstance(cluster, Cluster)
self.assertIsInstance(cluster.workers[1], Cluster)
leader = cluster.leader
self.assertTrue(raft.delete_leader(leader))
self.assertTrue(raft._sync_obj.set(raft.status_path, '{"optime":1234567,"slots":{"ls":12345}}'))
raft.get_cluster()
leader = raft.get_cluster().leader
self.assertTrue(raft.update_leader(leader, '1', failsafe={'foo': 'bat'}))
self.assertTrue(raft._sync_obj.set(raft.failsafe_path, '{"foo"}'))
self.assertTrue(raft._sync_obj.set(raft.status_path, '{'))
raft.get_citus_coordinator()
self.assertTrue(raft.delete_sync_state())
self.assertTrue(raft.delete_leader())
self.assertTrue(raft.set_history_value(''))
self.assertTrue(raft.delete_cluster())
raft._citus_group = '1'
self.assertTrue(raft.delete_cluster())
raft._citus_group = None
raft.get_cluster()
self.assertTrue(raft.take_leader())
raft.get_cluster()
raft.watch(None, 0.001)
raft._sync_obj.destroy()
+2 -3
View File
@@ -92,9 +92,8 @@ class TestRewind(BaseTestPostgresql):
self.r.rewind_or_reinitialize_needed_and_possible(self.leader)
with patch.object(Postgresql, 'is_running', Mock(return_value=True)), \
patch.object(MockCursor, 'fetchone', Mock(side_effect=Exception)), \
patch.object(MockCursor, 'fetchall',
Mock(return_value=[(0, 0, 1, 1, 0, 0, 0, 0, 0, None, None, None)])):
patch.object(MockCursor, 'fetchone',
Mock(side_effect=[Exception, (0, 0, 1, 1, 0, 0, 0, 0, 0, None, None, None)])):
self.r.rewind_or_reinitialize_needed_and_possible(self.leader)
@patch.object(CancellableSubprocess, 'call', mock_cancellable_call)
+27 -48
View File
@@ -8,7 +8,7 @@ from threading import Thread
from patroni import psycopg
from patroni.config import GlobalConfig
from patroni.dcs import Cluster, ClusterConfig, Member, Status, SyncState
from patroni.dcs import Cluster, ClusterConfig, Member, SyncState
from patroni.postgresql import Postgresql
from patroni.postgresql.misc import fsync_dir
from patroni.postgresql.slots import SlotsAdvanceThread, SlotsHandler
@@ -32,16 +32,16 @@ class TestSlotsHandler(BaseTestPostgresql):
self.p._global_config = GlobalConfig({})
self.s = self.p.slots_handler
self.p.start()
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}, 'ls2': None}}, 1)
self.cluster = Cluster(True, config, self.leader, Status(0, {'ls': 12345, 'ls2': 12345}),
[self.me, self.other, self.leadermem], None, SyncState.empty(), None, None)
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}}}, 1)
self.cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem],
None, SyncState.empty(), None, {'ls': 12345}, None)
def test_sync_replication_slots(self):
config = ClusterConfig(1, {'slots': {'test_3': {'database': 'a', 'plugin': 'b'},
'A': 0, 'ls': 0, 'b': {'type': 'logical', 'plugin': '1'}},
'ignore_slots': [{'name': 'blabla'}]}, 1)
cluster = Cluster(True, config, self.leader, Status(0, {'test_3': 10}),
[self.me, self.other, self.leadermem], None, SyncState.empty(), None, None)
cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem],
None, SyncState.empty(), None, {'test_3': 10}, None)
with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg.OperationalError)):
self.s.sync_replication_slots(cluster, False)
self.p.set_role('standby_leader')
@@ -51,9 +51,8 @@ class TestSlotsHandler(BaseTestPostgresql):
self.s.sync_replication_slots(cluster, False)
mock_debug.assert_called_once()
self.p.set_role('replica')
with patch.object(Postgresql, 'is_primary', Mock(return_value=False)), \
with patch.object(Postgresql, 'is_leader', Mock(return_value=False)), \
patch.object(SlotsHandler, 'drop_replication_slot') as mock_drop:
config.data['slots'].pop('ls')
self.s.sync_replication_slots(cluster, False, paused=True)
mock_drop.assert_not_called()
self.p.set_role('primary')
@@ -70,8 +69,6 @@ class TestSlotsHandler(BaseTestPostgresql):
self.assertTrue("test.3" in ca, "non matching {0}".format(ca))
with patch.object(Postgresql, 'major_version', PropertyMock(return_value=90618)):
self.s.sync_replication_slots(cluster, False)
self.p.set_role('replica')
self.s.sync_replication_slots(cluster, False)
def test_cascading_replica_sync_replication_slots(self):
"""Test sync with a cascading replica so physical slots are present on a replica."""
@@ -80,55 +77,52 @@ class TestSlotsHandler(BaseTestPostgresql):
'state': 'running', 'conn_url': 'postgres://replicator:[email protected]:5436/postgres',
'tags': {'replicatefrom': 'postgresql0'}
})
cluster = Cluster(True, config, self.leader, Status(0, {'ls': 10}),
[self.me, self.other, self.leadermem, cascading_replica], None, SyncState.empty(), None, None)
cluster = Cluster(True, config, self.leader, 0,
[self.me, self.other, self.leadermem, cascading_replica],
None, SyncState.empty(), None, {'ls': 10}, None)
self.p.set_role('replica')
with patch.object(Postgresql, '_query') as mock_query, \
patch.object(Postgresql, 'is_primary', Mock(return_value=False)):
mock_query.return_value = [('ls', 'logical', 104, 'b', 'a', 5, 12345, 105)]
patch.object(Postgresql, 'is_leader', Mock(return_value=False)):
mock_query.return_value = [('ls', 'logical', 'b', 'a', 5, 12345, 105)]
ret = self.s.sync_replication_slots(cluster, False)
self.assertEqual(ret, [])
def test_process_permanent_slots(self):
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}, 'blabla': {'type': 'physical'}},
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}},
'ignore_slots': [{'name': 'blabla'}]}, 1)
cluster = Cluster(True, config, self.leader, Status.empty(), [self.me, self.other, self.leadermem],
None, SyncState.empty(), None, None)
cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem],
None, SyncState.empty(), None, None, None)
self.s.sync_replication_slots(cluster, False)
with patch.object(Postgresql, '_query') as mock_query:
self.p.reset_cluster_info_state(None)
mock_query.return_value = [(
mock_query.return_value.fetchone.return_value = (
1, 0, 0, 0, 0, 0, 0, 0, 0, None, None,
[{"slot_name": "ls", "type": "logical", "datoid": 5, "plugin": "b",
"confirmed_flush_lsn": 12345, "catalog_xmin": 105, "restart_lsn": 12344},
{"slot_name": "blabla", "type": "physical", "datoid": None, "plugin": None,
"confirmed_flush_lsn": None, "catalog_xmin": 105, "restart_lsn": 12344}])]
self.assertEqual(self.p.slots(), {'ls': 12345, 'blabla': 12344})
"confirmed_flush_lsn": 12345, "catalog_xmin": 105}])
self.assertEqual(self.p.slots(), {'ls': 12345})
self.p.reset_cluster_info_state(None)
mock_query.return_value = [(
mock_query.return_value.fetchone.return_value = (
1, 0, 0, 0, 0, 0, 0, 0, 0, None, None,
[{"slot_name": "ls", "type": "logical", "datoid": 6, "plugin": "b",
"confirmed_flush_lsn": 12345, "catalog_xmin": 105}])]
"confirmed_flush_lsn": 12345, "catalog_xmin": 105}])
self.assertEqual(self.p.slots(), {})
@patch.object(Postgresql, 'is_primary', Mock(return_value=False))
@patch.object(Postgresql, 'is_leader', Mock(return_value=False))
def test__ensure_logical_slots_replica(self):
self.p.set_role('replica')
self.cluster.slots['ls'] = 12346
with patch.object(SlotsHandler, 'check_logical_slots_readiness', Mock(return_value=False)):
self.assertEqual(self.s.sync_replication_slots(self.cluster, False), [])
with patch.object(SlotsHandler, '_query', Mock(return_value=[('ls', 'logical', 499, 'b', 'a', 5, 100, 500)])), \
patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)), \
self.s._schedule_load_slots = False
with patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)), \
patch.object(SlotsAdvanceThread, 'schedule', Mock(return_value=(True, ['ls']))), \
patch.object(psycopg.OperationalError, 'diag') as mock_diag:
type(mock_diag).sqlstate = PropertyMock(return_value='58P01')
self.assertEqual(self.s.sync_replication_slots(self.cluster, False), ['ls'])
self.cluster.slots['ls'] = 'a'
self.assertEqual(self.s.sync_replication_slots(self.cluster, False), [])
self.cluster.config.data['slots']['ls']['database'] = 'b'
self.cluster.slots['ls'] = '500'
with patch.object(MockCursor, 'rowcount', PropertyMock(return_value=1), create=True):
self.assertEqual(self.s.sync_replication_slots(self.cluster, False), ['ls'])
@@ -142,21 +136,21 @@ class TestSlotsHandler(BaseTestPostgresql):
@patch.object(Postgresql, 'stop', Mock(return_value=True))
@patch.object(Postgresql, 'start', Mock(return_value=True))
@patch.object(Postgresql, 'is_primary', Mock(return_value=False))
@patch.object(Postgresql, 'is_leader', Mock(return_value=False))
def test_check_logical_slots_readiness(self):
self.s.copy_logical_slots(self.cluster, ['ls'])
with patch.object(MockCursor, '__iter__', Mock(return_value=iter([('postgresql0', None)]))), \
patch.object(MockCursor, 'fetchall', Mock(side_effect=Exception)):
patch.object(MockCursor, 'fetchone', Mock(side_effect=Exception)):
self.assertFalse(self.s.check_logical_slots_readiness(self.cluster, None))
with patch.object(MockCursor, '__iter__', Mock(return_value=iter([('postgresql0', None)]))), \
patch.object(MockCursor, 'fetchall', Mock(return_value=[(False,)])):
patch.object(MockCursor, 'fetchone', Mock(return_value=(False,))):
self.assertFalse(self.s.check_logical_slots_readiness(self.cluster, None))
with patch.object(MockCursor, '__iter__', Mock(return_value=iter([('ls', 100)]))):
self.s.check_logical_slots_readiness(self.cluster, None)
@patch.object(Postgresql, 'stop', Mock(return_value=True))
@patch.object(Postgresql, 'start', Mock(return_value=True))
@patch.object(Postgresql, 'is_primary', Mock(return_value=False))
@patch.object(Postgresql, 'is_leader', Mock(return_value=False))
def test_on_promote(self):
self.s.schedule_advance_slots({'foo': {'bar': 100}})
self.s.copy_logical_slots(self.cluster, ['ls'])
@@ -183,18 +177,3 @@ class TestSlotsHandler(BaseTestPostgresql):
with patch.object(SlotsHandler, 'get_local_connection_cursor', Mock(side_effect=Exception)):
self.s.schedule_advance_slots({'foo': {'bar': 100}})
self.s._advance.sync_slots()
@patch.object(Postgresql, 'is_primary', Mock(return_value=False))
def test_advance_physical_slots(self):
config = ClusterConfig(1, {'slots': {'blabla': {'type': 'physical'}, 'leader': None}}, 1)
cluster = Cluster(True, config, self.leader, Status(0, {'blabla': 12346}),
[self.me, self.other, self.leadermem], None, SyncState.empty(), None, None)
self.s.sync_replication_slots(cluster, False)
with patch.object(SlotsHandler, '_query', Mock(side_effect=[[('blabla', 'physical', 12345, None, None, None,
None, None)], Exception])) as mock_query, \
patch('patroni.postgresql.slots.logger.error') as mock_error:
self.s.sync_replication_slots(cluster, False)
self.assertEqual(mock_query.call_args[0],
("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)", "blabla", '0/303A'))
self.assertEqual(mock_error.call_args[0][0],
"Error while advancing replication slot %s to position '%s': %r")
-50
View File
@@ -134,23 +134,6 @@ def connect_side_effect(host_port):
raise socket.gaierror()
def mock_getaddrinfo(host, port, *args):
if port is None or port == "":
port = 0
port = int(port)
if port not in range(0, 65536):
raise socket.gaierror()
if host == "127.0.0.1" or host == "" or host is None:
return [(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, '', ('127.0.0.1', port))]
elif host == "127.0.0.2":
return [(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, '', ('127.0.0.2', port))]
elif host == "::1":
return [(socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_TCP, '', ('::1', port, 0, 0))]
else:
raise socket.gaierror()
def parse_output(output):
result = []
for s in output.split("\n"):
@@ -162,7 +145,6 @@ def parse_output(output):
@patch('socket.socket.connect_ex', Mock(side_effect=connect_side_effect))
@patch('socket.getaddrinfo', Mock(side_effect=mock_getaddrinfo))
@patch('os.path.exists', Mock(side_effect=exists_side_effect))
@patch('os.path.isdir', Mock(side_effect=isdir_side_effect))
@patch('os.path.isfile', Mock(side_effect=isfile_side_effect))
@@ -325,35 +307,3 @@ class TestValidator(unittest.TestCase):
output = "\n".join(errors)
self.assertEqual(['postgresql.bin_dir', 'postgresql.bin_name.postgres', 'raft.bind_addr', 'raft.self_addr'],
parse_output(output))
def test_one_of(self, _, __):
c = copy.deepcopy(config)
# Providing neither is fine
del c["tags"]["nofailover"]
errors = schema(c)
self.assertNotIn("tags Multiple of ('nofailover', 'failover_priority') provided", errors)
# Just nofailover is fine
c["tags"]["nofailover"] = False
errors = schema(c)
self.assertNotIn("tags Multiple of ('nofailover', 'failover_priority') provided", errors)
# Just failover_priority is fine
del c["tags"]["nofailover"]
c["tags"]["failover_priority"] = 1
errors = schema(c)
self.assertNotIn("tags Multiple of ('nofailover', 'failover_priority') provided", errors)
# Providing both is not fine
c["tags"]["nofailover"] = False
errors = schema(c)
self.assertIn("tags Multiple of ('nofailover', 'failover_priority') provided", errors)
def test_failover_priority_int(self, *args):
c = copy.deepcopy(config)
del c["tags"]["nofailover"]
c["tags"]["failover_priority"] = 'a string'
errors = schema(c)
self.assertIn('tags.failover_priority a string is not an integer', errors)
c = copy.deepcopy(config)
del c["tags"]["nofailover"]
c["tags"]["failover_priority"] = -6
errors = schema(c)
self.assertIn('tags.failover_priority -6 didn\'t pass validation: Wrong value', errors)
+16 -8
View File
@@ -1,13 +1,13 @@
import select
import unittest
from kazoo.client import KazooClient
from kazoo.client import KazooClient, KazooState
from kazoo.exceptions import NoNodeError, NodeExistsError
from kazoo.handlers.threading import SequentialThreadingHandler
from kazoo.protocol.states import KeeperState, WatchedEvent, ZnodeStat
from kazoo.protocol.states import KeeperState, ZnodeStat
from kazoo.retry import RetryFailedError
from mock import Mock, PropertyMock, patch
from patroni.dcs.zookeeper import Cluster, PatroniKazooClient, \
from patroni.dcs.zookeeper import Cluster, Leader, PatroniKazooClient, \
PatroniSequentialThreadingHandler, ZooKeeper, ZooKeeperError
@@ -152,6 +152,9 @@ class TestZooKeeper(unittest.TestCase):
'name': 'foo', 'ttl': 30, 'retry_timeout': 10, 'loop_wait': 10,
'set_acls': {'CN=principal2': ['ALL']}})
def test_session_listener(self):
self.zk.session_listener(KazooState.SUSPENDED)
def test_reload_config(self):
self.zk.reload_config({'ttl': 20, 'retry_timeout': 10, 'loop_wait': 10})
self.zk.reload_config({'ttl': 20, 'retry_timeout': 10, 'loop_wait': 5})
@@ -173,6 +176,15 @@ class TestZooKeeper(unittest.TestCase):
self.zk._cluster_loader(self.zk.client_path(''))
def test_get_cluster(self):
cluster = self.zk.get_cluster(True)
self.assertIsInstance(cluster.leader, Leader)
self.zk.status_watcher(None)
self.zk.get_cluster()
self.zk.touch_member({'foo': 'foo'})
self.zk._name = 'bar'
self.zk.status_watcher(None)
with patch.object(ZooKeeper, 'get_node', Mock(side_effect=Exception)):
self.zk.get_cluster()
cluster = self.zk.get_cluster()
self.assertEqual(cluster.last_lsn, 500)
@@ -190,7 +202,7 @@ class TestZooKeeper(unittest.TestCase):
mock_logger.assert_called_once()
def test_delete_leader(self):
self.assertTrue(self.zk.delete_leader(self.zk.get_cluster().leader))
self.assertTrue(self.zk.delete_leader())
def test_set_failover_value(self):
self.zk.set_failover_value('')
@@ -283,7 +295,3 @@ class TestZooKeeper(unittest.TestCase):
def test_set_history_value(self):
self.zk.set_history_value('{}')
def test_watcher(self):
self.zk._watcher(WatchedEvent('', '', ''))
self.assertTrue(self.zk.watch(1, 1))
+5 -10
View File
@@ -125,11 +125,10 @@ commands =
--file features/Dockerfile
allowlist_externals = docker
[testenv:pg{12,13,14,15,16}-docker-behave-{etcd,etcd3}-{lin,mac}]
[testenv:pg{12,13,14,15,16}-docker-behave-{etcd}-{lin,mac}]
description = Run behaviour tests in patroni-dev docker container
setenv =
etcd: DCS=etcd
etcd3: DCS=etcd3
{[common]postgres_matrix}
CONTAINER_NAME = tox-{env_name}-{env:PYTHONHASHSEED}
labels =
@@ -150,7 +149,7 @@ commands =
--tty \
{env:PATRONI_DEV_IMAGE:patroni-dev:{env:PG_MAJOR}} \
tox run -x 'tox.env_list=py{[common]python_matrix}-behave-{env:DCS}-lin' \
-- {posargs}
-- --format plain {posargs}
allowlist_externals =
docker
@@ -160,7 +159,7 @@ platform =
; win: win32
mac: darwin
[testenv:py{36,38,39,310,311}-behave-{etcd,etcd3}-{lin,win,mac}]
[testenv:py{36,38,39,310,311}-behave-{etcd}-{lin,win,mac}]
description = Run behaviour tests (locally with tox)
deps =
-r requirements.txt
@@ -168,15 +167,11 @@ deps =
coverage
{[common]psycopg_deps}
setenv =
etcd: DCS = {env:DCS:etcd}
etcd3: DCS = {env:DCS:etcd3}
DCS = {env:DCS:etcd}
passenv =
ETCD_UNSUPPORTED_ARCH
commands =
python3 -m behave --format json --format plain --outfile result.json {posargs}
mv result.json features/output
allowlist_externals =
mv
python3 -m behave {posargs}
platform =
{[common]platforms}