gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-26 15:40:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:zalando/patroni into feature/quorum-commit

Browse Source
This commit is contained in:
Alexander Kukushkin
2023-10-10 09:57:04 +02:00
parent 2f6678e4a7 9b8c40a6e1
commit 5c6b34a757
32 changed files with 2274 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENVIRONMENT.rst
+3 -3
View File
@@ -209,10 +209,10 @@ REST API
CTL
---
- **PATRONICTL\_CONFIG\_FILE**: (optional) location of the configuration file.
- **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
- **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
- **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
- **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
- **PATRONI\_CTL\_INSECURE**: (optional) Allow connections to REST API without verifying SSL certs.
- **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
- **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
- **PATRONI\_CTL\_CERTFILE**: (optional) Specifies the file with the client certificate in the PEM format.
- **PATRONI\_CTL\_KEYFILE**: (optional) Specifies the file with the client secret key in the PEM format.
- **PATRONI\_CTL\_KEYFILE\_PASSWORD**: (optional) Specifies a password for decrypting the client keyfile.
docs/citus.rst
+7 -7
View File
@@ -59,7 +59,7 @@ clusters that are just logically groupped together using the
PostgreSQL. Therefore in most cases it is not possible to manage them as a
single entity.
It results in two major differences in ``patronictl`` behaviour when
It results in two major differences in :ref:`patronictl` behaviour when
``patroni.yaml`` has the ``citus`` section comparing with the usual:
1. The ``list`` and the ``topology`` by default output all members of the Citus
@@ -67,12 +67,12 @@ It results in two major differences in ``patronictl`` behaviour when
which Citus group they belong to.
2. For all ``patronictl`` commands the new option is introduced, named
``--group``. For some commands the default value for the group might be
taken from the ``patroni.yaml``. For example, ``patronictl pause`` will
taken from the ``patroni.yaml``. For example, :ref:`patronictl_pause` will
enable the maintenance mode by default for the ``group`` that is set in the
``citus`` section, but for example for ``patronictl switchover`` or
``patronictl remove`` the group must be explicitly specified.
``citus`` section, but for example for :ref:`patronictl_switchover` or
:ref:`patronictl_remove` the group must be explicitly specified.
An example of ``patronictl list`` output for the Citus cluster::
An example of :ref:`patronictl_list` output for the Citus cluster::
postgres@coord1:~$ patronictl list demo
+ Citus cluster: demo ----------+----------------+---------+----+-----------+
@@ -117,7 +117,7 @@ the coordinator for the shards hosted on a worker node. The switchover then
happens while the traffic is kept on the coordinator, and resumes as soon as a
new primary worker node is ready to accept read-write queries.
An example of ``patronictl switchover`` on the worker cluster::
An example of :ref:`patronictl_switchover` on the worker cluster::
postgres@coord1:~$ patronictl switchover demo
+ Citus cluster: demo ----------+----------------+---------+----+-----------+
@@ -345,7 +345,7 @@ Citus upgrades and PostgreSQL major upgrades
First, please read about upgrading Citus version in the `documentation`__.
There is one minor change in the process. When executing upgrade, you have to
use ``patronictl restart`` instead of ``systemctl restart`` to restart
use :ref:`patronictl_restart` instead of ``systemctl restart`` to restart
PostgreSQL.
__ https://docs.citusdata.com/en/latest/admin_guide/upgrading_citus.html
docs/contributing_guidelines.rst
+13 -16
View File
@@ -3,21 +3,25 @@
Contributing guidelines
=======================
Wanna contribute to Patroni? Yay - here is how!
Chatting
--------
Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
Reporting bugs
--------------
Before reporting a bug please make sure to **reproduce it with the latest Patroni version**!
Also please double check if the issue already exists in our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Running tests
-------------
Requirements for running behave tests:
1. PostgreSQL packages need to be installed.
2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
#. PostgreSQL packages including `contrib <https://www.postgresql.org/docs/current/contrib.html>`__ modules need to be installed.
#. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
#. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
Install dependencies:
@@ -163,19 +167,12 @@ the watchdog behave feature test scenario with all versions of Postgres.
Of course you can combine the two.
Reporting issues
----------------
If you have a question about patroni or have a problem using it, please read the :ref:`README <readme>` before filing an issue.
Also double check with the current issues on our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Contributing a pull request
---------------------------
1) Submit a comment to the relevant issue or create a new issue describing your proposed change.
2) Do a fork, develop and test your code changes.
3) Include documentation
4) Submit a pull request.
#. Fork the repository, develop and test your code changes.
#. Reflect changes in the user documentation.
#. Submit a pull request with a clear description of the changes objective. Link an existing issue if necessary.
You'll get feedback about your pull request as soon as possible.
docs/dcs_failsafe_mode.rst
+1 -1
View File
@@ -60,4 +60,4 @@ F.A.Q.
- How to enable the Failsafe Mode?
Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or ``patronictl edit-config -s failsafe_mode=true``
Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or :ref:`patronictl edit-config -s failsafe_mode=true <patronictl_edit_config_parameters>`
docs/dynamic_configuration.rst
+13 -4
View File
@@ -6,11 +6,20 @@ Dynamic Configuration Settings
Dynamic configuration is stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
In order to change the dynamic configuration you can use either ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
In order to change the dynamic configuration you can use either :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
- **loop\_wait**: the number of seconds the loop will sleep. Default value: 10, minimum possible value: 1
- **ttl**: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30, minimum possible value: 20
- **retry\_timeout**: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10, minimum possible value: 3
.. warning::
when changing values of **loop_wait**, **retry_timeout**, or **ttl** you have to follow the rule:
.. code-block:: python
loop_wait + 2 * retry_timeout <= ttl
- **loop\_wait**: the number of seconds the loop will sleep. Default value: 10
- **ttl**: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30
- **retry\_timeout**: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10
- **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
- **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
- **max\_timelines\_history**: maximum number of timeline history items kept in DCS. Default value: 0. When set to 0, it keeps the full history in DCS.
docs/existing_data.rst
+3 -3
View File
@@ -42,12 +42,12 @@ You can find below an overview of steps for converting an existing Postgres clus
#. Start Patroni using the ``patroni`` systemd service unit. It automatically detects that Postgres is already running and starts monitoring the instance.
#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through ``patronictl restart cluster-name member-name`` command. For minimal downtime you might want to split this step into:
#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through :ref:`patronictl restart cluster-name member-name <patronictl_restart_parameters>` command. For minimal downtime you might want to split this step into:
#. Immediate restart of the standby nodes.
#. Scheduled restart of the primary node within a maintenance window.
#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through ``patronictl edit-config cluster-name member-name`` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through :ref:`patronictl edit-config cluster-name member-name <patronictl_edit_config_parameters>` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
.. code-block:: sql
@@ -73,7 +73,7 @@ The only possible way to do a major upgrade currently is:
#. Stop Patroni
#. Upgrade PostgreSQL binaries and perform `pg_upgrade <https://www.postgresql.org/docs/current/pgupgrade.html>`_ on the primary node
#. Update patroni.yml
#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running ``patronictl remove <cluster-name>``. It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running :ref:`patronictl remove cluster-name <patronictl_remove_parameters>` . It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
#. If you wiped the cluster state in the previous step, you may wish to copy patroni.dynamic.json from old data dir to the new one. It will help you to retain some PostgreSQL parameters you had set before.
#. Start Patroni on the primary node.
#. Upgrade PostgreSQL binaries, update patroni.yml and wipe the data_dir on standby nodes.
docs/index.rst
+1
View File
@@ -25,6 +25,7 @@ Currently supported PostgreSQL versions: 9.3 to 16.
installation
patroni_configuration
rest_api
patronictl
replica_bootstrap
replication_modes
watchdog
docs/patroni_configuration.rst
+5 -5
View File
@@ -15,7 +15,7 @@ There are 3 types of Patroni configuration:
- Global :ref:`dynamic configuration <dynamic_configuration>`.
These options are stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
Dynamic configuration can be set at any time using ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
Dynamic configuration can be set at any time using :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
If the options changed are not part of the startup configuration, they are applied asynchronously (upon the next wake up cycle)
to every node, which gets subsequently reloaded.
If the node requires a restart to apply the configuration (for `PostgreSQL parameters <https://www.postgresql.org/docs/current/view-pg-settings.html>`__ with context postmaster, if their values
@@ -24,7 +24,7 @@ There are 3 types of Patroni configuration:
- Local :ref:`configuration file <yaml_configuration>` (patroni.yml).
These options are defined in the configuration file and take precedence over dynamic configuration.
``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing ``patronictl reload``. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing :ref:`patronictl_reload`. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
- :ref:`Environment configuration <environment>`.
It is possible to set/override some of the "Local" configuration parameters with environment variables.
@@ -105,10 +105,10 @@ Changing these parameters require a PostgreSQL restart to take effect, and their
As explained before, Patroni restrict changing their values through :ref:`dynamic configuration <dynamic_configuration>`, which usually consists of:
1. Applying changes through ``patronictl edit-config`` (or via REST API ``/config`` endpoint)
2. Restarting nodes through ``patronictl restart`` (or via REST API ``/restart`` endpoint)
1. Applying changes through :ref:`patronictl_edit_config` (or via REST API ``/config`` endpoint)
2. Restarting nodes through :ref:`patronictl_restart` (or via REST API ``/restart`` endpoint)
**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through ``patronictl restart`` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through :ref:`patronictl_restart` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
However, as those settings manage shared memory, some extra care should be taken when restarting the nodes:
docs/patronictl.rst
+1975
View File
File diff suppressed because it is too large Load Diff
docs/pause.rst
+1 -1
View File
@@ -32,6 +32,6 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,
User guide
----------
``patronictl`` supports ``pause`` and ``resume`` commands.
``patronictl`` supports :ref:`pause <patronictl_pause>` and :ref:`resume <patronictl_resume>` commands.
One can also issue a ``PATCH`` request to the ``{namespace}/{cluster}/config`` key with ``{"pause": true/false/null}``
docs/releases.rst
+22
View File
@@ -3,6 +3,28 @@
Release notes
=============
Version 3.1.2
-------------
**Bugfixes**
- Fixed bug with ``wal_keep_size`` checks (Alexander Kukushkin)
The ``wal_keep_size`` is a GUC that normally has a unit and Patroni was failing to cast its value to ``int``. As a result the value of ``bootstrap.dcs`` was not written to the ``/config`` key afterwards.
- Detect and resolve inconsistencies between ``/sync`` key and ``synchronous_standby_names`` (Alexander Kukushkin)
Normally, Patroni updates ``/sync`` and ``synchronous_standby_names`` in a very specific order, but in case of a bug or when someone manually reset ``synchronous_standby_names``, Patroni was getting into an inconsistent state. As a result it was possible that the failover happens to an asynchronous node.
- Read GUC's values when joining running Postgres (Alexander Kukushkin)
When restarted in ``pause``, Patroni was discarding the ``synchronous_standby_names`` GUC from the ``postgresql.conf``. To solve it and avoid similar issues, Patroni will read GUC's value if it is joining an already running Postgres.
- Silenced annoying warnings when checking for node uniqueness (Alexander Kukushkin)
``WARNING`` messages are produced by ``urllib3`` if Patroni is quickly restarted.
Version 3.1.1
-------------
docs/replica_bootstrap.rst
+1 -1
View File
@@ -191,7 +191,7 @@ There is no further relationship between the standby cluster and the primary
cluster it replicates from, in particular, they must not share the same DCS
scope if they use the same DCS. They do not know anything else from each other
apart from replication information. Also, the standby cluster is not being
displayed in ``patronictl list`` or ``patronictl topology`` output on the
displayed in :ref:`patronictl_list` or :ref:`patronictl_topology` output on the
primary cluster.
For the sake of flexibility, you can specify methods of creating a replica and
docs/rest_api.rst
+7 -7
View File
@@ -3,7 +3,7 @@
Patroni REST API
================
Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
Health check endpoints
----------------------
@@ -626,9 +626,9 @@ In the JSON body of the ``POST`` request you must specify the ``candidate`` fiel
:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs.
``POST /switchover`` and ``POST /failover`` endpoints are used by ``patronictl switchover`` and ``patronictl failover``, respectively.
``POST /switchover`` and ``POST /failover`` endpoints are used by :ref:`patronictl_switchover` and :ref:`patronictl_failover`, respectively.
``DELETE /switchover`` is used by ``patronictl flush <cluster-name> switchover``.
``DELETE /switchover`` is used by :ref:`patronictl flush cluster-name switchover <patronictl_flush_parameters>`.
.. list-table:: Failover/Switchover comparison
:widths: 25 25 25
@@ -687,15 +687,15 @@ Restart endpoint
- ``DELETE /restart``: delete the scheduled restart
``POST /restart`` and ``DELETE /restart`` endpoints are used by ``patronictl restart`` and ``patronictl flush <cluster-name> restart`` respectively.
``POST /restart`` and ``DELETE /restart`` endpoints are used by :ref:`patronictl_restart` and :ref:`patronictl flush cluster-name restart <patronictl_flush_parameters>` respectively.
Reload endpoint
---------------
The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of ``patronictl restart``.
The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of :ref:`patronictl_restart`.
The reload endpoint is used by ``patronictl reload``.
The reload endpoint is used by :ref:`patronictl_reload`.
Reinitialize endpoint
@@ -705,4 +705,4 @@ Reinitialize endpoint
The call might fail if Patroni is in a loop trying to recover (restart) a failed Postgres. In order to overcome this problem one can specify ``{"force":true}`` in the request body.
The reinitialize endpoint is used by ``patronictl reinit``.
The reinitialize endpoint is used by :ref:`patronictl_reinit`.
docs/security.rst
+3 -3
View File
@@ -9,7 +9,7 @@ A Patroni cluster has two interfaces to be protected from unauthorized access: t
Protecting DCS
==============
Patroni and patronictl both store and retrieve data to/from the DCS.
Patroni and :ref:`patronictl` both store and retrieve data to/from the DCS.
Despite DCS doesn't contain any sensitive information, it allows changing some of Patroni/Postgres configuration. Therefore the very first thing that should be protected is DCS itself.
@@ -22,7 +22,7 @@ Protecting the REST API
Protecting the REST API is a more complicated task.
The Patroni REST API is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
The Patroni REST API is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
From the point of view of security, REST API contains safe (``GET`` requests, only retrieve information) and unsafe (``PUT``, ``POST``, ``PATCH`` and ``DELETE`` requests, change the state of nodes) endpoints.
@@ -32,6 +32,6 @@ When TLS for the REST API is enabled and a PKI is established, mutual authentica
The ``restapi`` section parameters enable TLS client authentication to the server. Depending on the value of the ``verify_client`` parameter, the API server requires a successful client certificate verification for both safe and unsafe API calls (``verify_client: required``), or only for unsafe API calls (``verify_client: optional``), or for no API calls (``verify_client: none``).
The ``ctl`` section parameters enable TLS server authentication to the client (the ``patronictl`` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
The ``ctl`` section parameters enable TLS server authentication to the client (the :ref:`patronictl` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
Protecting the PostgreSQL database proper from unauthorized access is beyond the scope of this document and is covered in https://www.postgresql.org/docs/current/client-authentication.html
docs/yaml_configuration.rst
+5 -5
View File
@@ -34,7 +34,7 @@ Bootstrap configuration
.. note::
Once Patroni has initialized the cluster for the first time and settings have been stored in the DCS, all future
changes to the ``bootstrap.dcs`` section of the YAML configuration will not take any effect! If you want to change
them please use either ``patronictl edit-config`` or the Patroni :ref:`REST API <rest_api>`.
them please use either :ref:`patronictl_edit_config` or the Patroni :ref:`REST API <rest_api>`.
- **bootstrap**:
@@ -366,10 +366,10 @@ CTL
- **authentication**:
- **username**: Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
- **password**: Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
- **username**: Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
- **password**: Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
- **insecure**: Allow connections to REST API without verifying SSL certs.
- **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
- **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
- **certfile**: Specifies the file with the client certificate in the PEM format.
- **keyfile**: Specifies the file with the client secret key in the PEM format.
- **keyfile\_password**: Specifies a password for decrypting the client keyfile.
@@ -397,4 +397,4 @@ In addition to these predefined tags, you can also add your own ones:
- **key3**: ``1.4``
- **key4**: ``"RandomString"``
Tags are visible in the :ref:`REST API <rest_api>` and ``patronictl list`` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
Tags are visible in the :ref:`REST API <rest_api>` and :ref:`patronictl_list` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
features/dcs_failsafe_mode.feature
+4 -5
View File
@@ -4,8 +4,8 @@ Feature: dcs failsafe mode
Scenario: check failsafe mode can be successfully enabled
Given I start postgres0
And postgres0 is a leader after 10 seconds
And I sleep for 3 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 5, "failsafe_mode": true}
Then "config" key in DCS has ttl=30 after 10 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 3, "failsafe_mode": true}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/failsafe
@@ -28,7 +28,6 @@ Feature: dcs failsafe mode
When I do a backup of postgres0
And I shut down postgres0
When I start postgres1 in a cluster batman from backup with no_leader
And I sleep for 2 seconds
Then postgres1 role is the replica after 12 seconds
Scenario: check leader and replica are both in /failsafe key after leader is back
@@ -59,12 +58,12 @@ Feature: dcs failsafe mode
Given DCS is down
And I kill postgres1
And I kill postmaster on postgres1
And I sleep for 2 seconds
Then postgres0 role is the replica after 12 seconds
@dcs-failsafe
Scenario: check known replica is promoted when leader is down and DCS is up
Given I shut down postgres0
Given I kill postgres0
And I shut down postmaster on postgres0
And DCS is up
When I start postgres1
Then "members/postgres1" key in DCS has state=running after 10 seconds
features/environment.py
+3 -2
View File
@@ -162,9 +162,10 @@ class PatroniController(AbstractController):
def stop(self, kill=False, timeout=15, postgres=False):
if postgres:
return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-mi', '-w'])
mode = 'i' if kill else 'f'
return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-m' + mode, '-w'])
super(PatroniController, self).stop(kill, timeout)
if isinstance(self._context.dcs_ctl, KubernetesController):
if isinstance(self._context.dcs_ctl, KubernetesController) and not kill:
self._context.dcs_ctl.delete_pod(self._name[8:])
if self.watchdog:
self.watchdog.stop()
features/standby_cluster.feature
+17 -10
View File
@@ -53,15 +53,22 @@ Feature: standby cluster
And I receive a response replication_state streaming
And postgres1 does not have a logical replication slot named test_logical
Scenario: check failover
When I kill postgres1
And I kill postmaster on postgres1
Then postgres2 is replicating from postgres0 after 32 seconds
When I issue a GET request to http://127.0.0.1:8010/primary
Then I receive a response code 503
And I sleep for 3 seconds
When I issue a GET request to http://127.0.0.1:8010/standby_leader
Scenario: check switchover
When I run patronictl.py switchover batman1 --force
And I issue a GET request to http://127.0.0.1:8010/standby_leader
Then I receive a response code 200
And I receive a response role standby_leader
And replication works from postgres0 to postgres2 after 15 seconds
And there is a postgres2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres2 data directory
And postgres1 is replicating from postgres2 after 32 seconds
Scenario: check failover
When I kill postgres2
And I kill postmaster on postgres2
Then postgres1 is replicating from postgres0 after 32 seconds
When I issue a GET request to http://127.0.0.1:8009/primary
Then I receive a response code 503
And I sleep for 3 seconds
When I issue a GET request to http://127.0.0.1:8009/standby_leader
Then I receive a response code 200
And I receive a response role standby_leader
And replication works from postgres0 to postgres1 after 15 seconds
And there is a postgres1_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres1 data directory
features/steps/basic_replication.py
+6 -1
View File
@@ -35,11 +35,16 @@ def kill_patroni(context, name):
return context.pctl.stop(name, kill=True)
@step('I kill postmaster on {name:w}')
@step('I shut down postmaster on {name:w}')
def stop_postgres(context, name):
return context.pctl.stop(name, postgres=True)
@step('I kill postmaster on {name:w}')
def kill_postgres(context, name):
return context.pctl.stop(name, kill=True, postgres=True)
@step('I add the table {table_name:w} to {pg_name:w}')
def add_table(context, table_name, pg_name):
# parse the configuration file and get the port
features/steps/cascading_replication.py
+1 -1
View File
@@ -28,7 +28,7 @@ def check_member(context, name, key, value, time_limit):
while time.time() < max_time:
try:
response = json.loads(context.dcs_ctl.query(name))
dcs_value = response.get(key)
dcs_value = str(response.get(key))
if dcs_value == value:
return
except Exception:
features/steps/standby_cluster.py
+1
View File
@@ -34,6 +34,7 @@ def start_patroni_standby_cluster(context, name, cluster_name, name2):
"ttl": 20,
"loop_wait": 2,
"retry_timeout": 5,
"synchronous_mode": True, # should be completely ignored
"standby_cluster": {
"host": "localhost",
"port": port,
patroni/config.py
+64 -2
View File
@@ -101,8 +101,9 @@ class GlobalConfig(object):
@property
def is_synchronous_mode(self) -> bool:
"""``True`` if synchronous replication is requested."""
return self.check_mode('synchronous_mode') is True or self.is_quorum_commit_mode
"""``True`` if synchronous replication is requested and it is not a standby cluster config."""
return (self.check_mode('synchronous_mode') is True or self.is_quorum_commit_mode) \
and not self.is_standby_cluster
@property
def is_synchronous_mode_strict(self) -> bool:
@@ -405,6 +406,66 @@ class Config(object):
except Exception:
logger.error('Can not remove temporary file %s', tmpfile)
def __get_and_maybe_adjust_int_value(self, config: Dict[str, Any], param: str, min_value: int) -> int:
"""Get, validate and maybe adjust a *param* integer value from the *config* :class:`dict`.
.. note:
If the value is smaller than provided *min_value* we update the *config*.
This method may raise an exception if value isn't :class:`int` or cannot be casted to :class:`int`.
:param config: :class:`dict` object with new global configuration.
:param param: name of the configuration parameter we want to read/validate/adjust.
:param min_value: the minimum possible value that a given *param* could have.
:returns: an integer value which corresponds to a provided *param*.
"""
value = int(config.get(param, self.__DEFAULT_CONFIG[param]))
if value < min_value:
logger.warning("%s=%d can't be smaller than %d, adjusting...", param, value, min_value)
value = config[param] = min_value
return value
def _validate_and_adjust_timeouts(self, config: Dict[str, Any]) -> None:
"""Validate and adjust ``loop_wait``, ``retry_timeout``, and ``ttl`` values if necessary.
Minimum values:
* ``loop_wait``: 1 second;
* ``retry_timeout``: 3 seconds.
* ``ttl``: 20 seconds;
Maximum values:
In case if values don't fulfill the following rule, ``retry_timeout`` and ``loop_wait``
are reduced so that the rule is fulfilled:
.. code-block:: python
loop_wait + 2 * retry_timeout <= ttl
.. note:
We prefer to reduce ``loop_wait`` and will reduce ``retry_timeout`` only if ``loop_wait``
is already set to a minimal possible value.
:param config: :class:`dict` object with new global configuration.
"""
min_loop_wait = 1
loop_wait = self. __get_and_maybe_adjust_int_value(config, 'loop_wait', min_loop_wait)
retry_timeout = self. __get_and_maybe_adjust_int_value(config, 'retry_timeout', 3)
ttl = self. __get_and_maybe_adjust_int_value(config, 'ttl', 20)
if min_loop_wait + 2 * retry_timeout > ttl:
config['loop_wait'] = min_loop_wait
config['retry_timeout'] = (ttl - min_loop_wait) // 2
logger.warning('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d. '
'Adjusting loop_wait from %d to %d and retry_timeout from %d to %d',
ttl, loop_wait, min_loop_wait, retry_timeout, config['retry_timeout'])
elif loop_wait + 2 * retry_timeout > ttl:
config['loop_wait'] = ttl - 2 * retry_timeout
logger.warning('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d and retry_timeout=%d.'
' Adjusting loop_wait from %d to %d', ttl, retry_timeout, loop_wait, config['loop_wait'])
# configuration could be either ClusterConfig or dict
def set_dynamic_configuration(self, configuration: Union[ClusterConfig, Dict[str, Any]]) -> bool:
"""Set dynamic configuration values with given *configuration*.
@@ -422,6 +483,7 @@ class Config(object):
if not deep_compare(self._dynamic_configuration, configuration):
try:
self._validate_and_adjust_timeouts(configuration)
self.__effective_configuration = self._build_effective_configuration(configuration,
self._local_configuration)
self._dynamic_configuration = configuration
patroni/config_generator.py
+12 -12
View File
@@ -38,7 +38,7 @@ _AUTH_ALLOWED_PARAMETERS_MAPPING = {
'gssencmode': 'PGGSSENCMODE',
'channel_binding': 'PGCHANNELBINDING'
}
_NO_VALUE_MSG = '#FIXME'
NO_VALUE_MSG = '#FIXME'
def get_address() -> Tuple[str, str]:
@@ -50,7 +50,7 @@ def get_address() -> Tuple[str, str]:
:returns: tuple consisting of the hostname returned by :func:`~socket.gethostname`
and the first element in the sorted list of the addresses returned by :func:`~socket.getaddrinfo`.
Sorting guarantees it will prefer IPv4.
If an exception occured, hostname and ip values are equal to :data:`~patroni.config_generator._NO_VALUE_MSG`.
If an exception occured, hostname and ip values are equal to :data:`~patroni.config_generator.NO_VALUE_MSG`.
"""
hostname = None
try:
@@ -59,7 +59,7 @@ def get_address() -> Tuple[str, str]:
key=lambda x: x[0])[0][4][0]
except Exception as err:
logging.warning('Failed to obtain address: %r', err)
return _NO_VALUE_MSG, _NO_VALUE_MSG
return NO_VALUE_MSG, NO_VALUE_MSG
class AbstractConfigGenerator(abc.ABC):
@@ -88,24 +88,24 @@ class AbstractConfigGenerator(abc.ABC):
"""Generate a template config for further extension (e.g. in the inherited classes).
:returns: dictionary with the values gathered from Patroni env, hopefully defined hostname and ip address
(otherwise set to :data:`~patroni.config_generator._NO_VALUE_MSG`), and some sane defaults.
(otherwise set to :data:`~patroni.config_generator.NO_VALUE_MSG`), and some sane defaults.
"""
template_config: Dict[str, Any] = {
'scope': _NO_VALUE_MSG,
'scope': NO_VALUE_MSG,
'name': cls._HOSTNAME,
'postgresql': {
'data_dir': _NO_VALUE_MSG,
'connect_address': _NO_VALUE_MSG + ':5432',
'listen': _NO_VALUE_MSG + ':5432',
'data_dir': NO_VALUE_MSG,
'connect_address': NO_VALUE_MSG + ':5432',
'listen': NO_VALUE_MSG + ':5432',
'bin_dir': '',
'authentication': {
'superuser': {
'username': 'postgres',
'password': _NO_VALUE_MSG
'password': NO_VALUE_MSG
},
'replication': {
'username': 'replicator',
'password': _NO_VALUE_MSG
'password': NO_VALUE_MSG
}
}
},
@@ -185,7 +185,7 @@ class SampleConfigGenerator(AbstractConfigGenerator):
self.config['bootstrap']['dcs']['postgresql']['use_pg_rewind'] = True
if self.pg_major >= 110000:
self.config['postgresql']['authentication'].setdefault(
'rewind', {'username': 'rewind_user'}).setdefault('password', _NO_VALUE_MSG)
'rewind', {'username': 'rewind_user'}).setdefault('password', NO_VALUE_MSG)
class RunningClusterConfigGenerator(AbstractConfigGenerator):
@@ -335,7 +335,7 @@ class RunningClusterConfigGenerator(AbstractConfigGenerator):
getpass('Please enter the user password:')
self.config['postgresql']['authentication'] = {
'superuser': su_params,
'replication': {'username': _NO_VALUE_MSG, 'password': _NO_VALUE_MSG}
'replication': {'username': NO_VALUE_MSG, 'password': NO_VALUE_MSG}
}
def _set_conf_files(self) -> None:
patroni/dcs/kubernetes.py
+26 -6
View File
@@ -771,8 +771,7 @@ class Kubernetes(AbstractDCS):
except k8s_config.ConfigException:
k8s_config.load_kube_config(context=config.get('context', 'kind-kind'))
pod_ip = config.get('pod_ip')
self.__ips: List[str] = [] if self._ctl or not isinstance(pod_ip, str) else [pod_ip]
self.__ips: List[str] = [] if self._ctl else [config.get('pod_ip', '')]
self.__ports: List[K8sObject] = []
ports: List[Dict[str, Any]] = config.get('ports', [{}])
for p in ports:
@@ -1059,6 +1058,27 @@ class Kubernetes(AbstractDCS):
def _patch_or_create(self, name: str, annotations: Dict[str, Any],
resource_version: Optional[str] = None, patch: bool = False,
retry: Optional[Callable[..., Any]] = None, ips: Optional[List[str]] = None) -> K8sObject:
"""Patch or create K8s object, Endpoint or ConfigMap.
:param name: the name of the object.
:param annotations: mapping of annotations that we want to create/update.
:param resource_version: object should be updated only if the ``resource_version`` matches provided value.
:param patch: ``True`` if we know in advance that the object already exists and we should patch it.
:param retry: a callable that will take care of retries
:param ips: IP address that we want to put to the subsets of the endpoint. Could have following values:
* ``None`` - when we don't need to touch subset;
* ``[]`` - to set subsets to the empty list, when :meth:`delete_leader` method is called;
* ``['ip.add.re.ss']`` - when we want to make sure that the subsets of the leader endpoint
contains the IP address of the leader, that we get from the ``kubernetes.pod_ip``;
* ``['']`` - when we want to make sure that the subsets of the leader endpoint contains the IP
address of the leader, but ``kubernetes.pod_ip`` configuration is missing. In this case we will
try to take the IP address of the Pod which name matches ``name`` from the config file.
:returns: the new :class:`V1Endpoints` or :class:`V1ConfigMap` object, that was created or updated.
"""
metadata = {'namespace': self._namespace, 'name': name, 'labels': self._labels, 'annotations': annotations}
if patch or resource_version:
if resource_version is not None:
@@ -1071,9 +1091,10 @@ class Kubernetes(AbstractDCS):
metadata['annotations'] = {k: v for k, v in annotations.items() if v is not None}
metadata = k8s_client.V1ObjectMeta(**metadata)
if ips is not None and self._api.use_endpoints:
if self._api.use_endpoints:
endpoints = {'metadata': metadata}
self._map_subsets(endpoints, ips)
if ips is not None:
self._map_subsets(endpoints, ips)
body = k8s_client.V1Endpoints(**endpoints)
else:
body = k8s_client.V1ConfigMap(metadata=metadata)
@@ -1222,11 +1243,10 @@ class Kubernetes(AbstractDCS):
else:
annotations['acquireTime'] = self._leader_observed_record.get('acquireTime') or now
annotations['transitions'] = str(transitions)
ips: Optional[List[str]] = [] if self._api.use_endpoints else None
try:
ret = bool(self._patch_or_create(self.leader_path, annotations,
self._leader_resource_version, retry=self.retry, ips=ips))
self._leader_resource_version, retry=self.retry, ips=self.__ips))
except k8s_client.rest.ApiException as e:
if e.status == 409 and self._leader_resource_version: # Conflict in resource_version
# Terminate watchers, it could be a sign that K8s API is in a failed state
patroni/postgresql/callback_executor.py
+3 -1
View File
@@ -30,7 +30,9 @@ class OnReloadExecutor(CancellableSubprocess):
self.cancel(kill=True)
self._kill_children()
with self._lock:
self._start_process(cmd, close_fds=True)
started = self._start_process(cmd, close_fds=True)
if started and self._process is not None:
Thread(target=self._process.wait).start()
class CallbackExecutor(CancellableExecutor, Thread):
patroni/postgresql/config.py
+3 -2
View File
@@ -244,9 +244,10 @@ class ConfigWriter(object):
self._fd.write(line)
self._fd.write('\n')
def writelines(self, lines: List[str]) -> None:
def writelines(self, lines: List[Optional[str]]) -> None:
for line in lines:
self.writeline(line)
if isinstance(line, str):
self.writeline(line)
@staticmethod
def escape(value: Any) -> str: # Escape (by doubling) any single quotes or backslashes in given string
patroni/version.py
+1 -1
View File
@@ -2,4 +2,4 @@
:var __version__: the current Patroni version.
"""
__version__ = '3.1.1'
__version__ = '3.1.2'
tests/test_callback_executor.py
+1
View File
@@ -35,5 +35,6 @@ class TestCallbackExecutor(unittest.TestCase):
ce._invoke_excepthook = Mock()
self.assertIsNone(ce.call(callback))
mock_popen.side_effect = [Mock()]
self.assertIsNone(ce.call(['test.sh', 'on_reload', 'replica', 'foo']))
ce.join()
tests/test_config.py
+30 -1
View File
@@ -5,7 +5,7 @@ import io
from copy import deepcopy
from mock import MagicMock, Mock, patch
from patroni.config import Config, ConfigParseError
from patroni.config import Config, ConfigParseError, GlobalConfig
class TestConfig(unittest.TestCase):
@@ -173,3 +173,32 @@ class TestConfig(unittest.TestCase):
input_params['max_connections'] = 10
expected_params.pop('max_connections')
self.assertEqual(self.config._process_postgresql_parameters(input_params), expected_params)
def test__validate_and_adjust_timeouts(self):
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 15})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'ttl', 15, 20))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'loop_wait': 0})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'loop_wait', 0, 1))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'retry_timeout': 1})
self.assertEqual(mock_logger.call_args_list[0][0],
("%s=%d can't be smaller than %d, adjusting...", 'retry_timeout', 1, 3))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 20, 'loop_wait': 11, 'retry_timeout': 5})
self.assertEqual(mock_logger.call_args_list[0][0],
('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d '
'and retry_timeout=%d. Adjusting loop_wait from %d to %d', 20, 5, 11, 10))
with patch('patroni.config.logger.warning') as mock_logger:
self.config._validate_and_adjust_timeouts({'ttl': 20, 'loop_wait': 10, 'retry_timeout': 10})
self.assertEqual(mock_logger.call_args_list[0][0],
('Violated the rule "loop_wait + 2*retry_timeout <= ttl", where ttl=%d. Adjusting'
' loop_wait from %d to %d and retry_timeout from %d to %d', 20, 10, 1, 10, 9))
def test_global_config_is_synchronous_mode(self):
# we should ignore synchronous_mode setting in a standby cluster
config = {'standby_cluster': {'host': 'some_host'}, 'synchronous_mode': True}
self.assertFalse(GlobalConfig(config).is_synchronous_mode)
tests/test_config_generator.py
+16 -20
View File
@@ -1,6 +1,5 @@
import os
import psutil
import socket
import unittest
from . import MockConnect, MockCursor, MockConnectionInfo
@@ -9,28 +8,25 @@ from mock import MagicMock, Mock, PropertyMock, mock_open, patch
from patroni.__main__ import main as _main
from patroni.config import Config
from patroni.config_generator import AbstractConfigGenerator, get_address
from patroni.config_generator import AbstractConfigGenerator, get_address, NO_VALUE_MSG
from patroni.utils import patch_config
from . import psycopg_connect
HOSTNAME = 'test_hostname'
IP = '1.9.8.4'
@patch('patroni.psycopg.connect', psycopg_connect)
@patch('socket.getaddrinfo', Mock(return_value=[(0, 0, 0, 0, ('1.9.8.4', 1984))]))
@patch('builtins.open', MagicMock())
@patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 16.2"))
@patch('psutil.Process.exe', Mock(return_value='/bin/dir/from/running/postgres'))
@patch('psutil.Process.__init__', Mock(return_value=None))
@patch.object(AbstractConfigGenerator, '_HOSTNAME', HOSTNAME)
@patch.object(AbstractConfigGenerator, '_IP', IP)
class TestGenerateConfig(unittest.TestCase):
no_value_msg = '#FIXME'
_HOSTNAME = socket.gethostname()
_IP = sorted(socket.getaddrinfo(_HOSTNAME, 0, socket.AF_UNSPEC, socket.SOCK_STREAM, 0), key=lambda x: x[0])[0][4][0]
def setUp(self):
self.maxDiff = None
os.environ['PATRONI_SCOPE'] = 'scope_from_env'
os.environ['PATRONI_POSTGRESQL_BIN_DIR'] = '/bin/from/env'
os.environ['PATRONI_SUPERUSER_USERNAME'] = 'su_user_from_env'
@@ -54,14 +50,14 @@ class TestGenerateConfig(unittest.TestCase):
self.config = {
'scope': self.environ['PATRONI_SCOPE'],
'name': self._HOSTNAME,
'name': HOSTNAME,
'bootstrap': {
'dcs': dynamic_config
},
'postgresql': {
'connect_address': self.no_value_msg + ':5432',
'data_dir': self.no_value_msg,
'listen': self.no_value_msg + ':5432',
'connect_address': NO_VALUE_MSG + ':5432',
'data_dir': NO_VALUE_MSG,
'listen': NO_VALUE_MSG + ':5432',
'pg_hba': ['host all all all md5',
f'host replication {self.environ["PATRONI_REPLICATION_USERNAME"]} all md5'],
'authentication': {'superuser': {'username': self.environ['PATRONI_SUPERUSER_USERNAME'],
@@ -99,7 +95,7 @@ class TestGenerateConfig(unittest.TestCase):
}
},
'postgresql': {
'connect_address': f'{self._IP}:bar',
'connect_address': f'{IP}:bar',
'listen': '6.6.6.6:1984',
'data_dir': 'data',
'bin_dir': '/bin/dir/from/running',
@@ -118,8 +114,8 @@ class TestGenerateConfig(unittest.TestCase):
'sslmode': 'prefer'
},
'replication': {
'username': self.no_value_msg,
'password': self.no_value_msg
'username': NO_VALUE_MSG,
'password': NO_VALUE_MSG
},
'rewind': None
},
@@ -179,7 +175,7 @@ class TestGenerateConfig(unittest.TestCase):
'authentication': {
'rewind': {
'username': self.environ['PATRONI_REWIND_USERNAME'],
'password': self.no_value_msg}
'password': NO_VALUE_MSG}
},
}
}
@@ -230,7 +226,7 @@ class TestGenerateConfig(unittest.TestCase):
}
},
'postgresql': {
'connect_address': f'{self._IP}:1984',
'connect_address': f'{IP}:1984',
'authentication': {
'superuser': {
'username': self.environ['PGUSER'],
@@ -330,5 +326,5 @@ class TestGenerateConfig(unittest.TestCase):
def test_get_address(self):
with patch('socket.getaddrinfo', Mock(side_effect=Exception)), \
patch('logging.warning') as mock_warning:
self.assertEqual(get_address(), (self.no_value_msg, self.no_value_msg))
self.assertEqual(get_address(), (NO_VALUE_MSG, NO_VALUE_MSG))
self.assertIn('Failed to obtain address: %r', mock_warning.call_args_list[0][0])
tests/test_kubernetes.py
+16 -1
View File
@@ -63,7 +63,7 @@ def mock_list_namespaced_pod(*args, **kwargs):
metadata = k8s_client.V1ObjectMeta(resource_version='1', labels={'f': 'b', Kubernetes._CITUS_LABEL: '1'},
name='p-0', annotations={'status': '{}'},
uid='964dfeae-e79b-4476-8a5a-1920b5c2a69d')
status = k8s_client.V1PodStatus(pod_ip='10.0.0.0')
status = k8s_client.V1PodStatus(pod_ip='10.0.0.1')
spec = k8s_client.V1PodSpec(hostname='p-0', node_name='kind-control-plane', containers=[])
items = [k8s_client.V1Pod(metadata=metadata, status=status, spec=spec)]
return k8s_client.V1PodList(items=items, kind='PodList')
@@ -356,6 +356,20 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
mock_warning.assert_called_once()
class TestKubernetesEndpointsNoPodIP(BaseTestKubernetes):
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_endpoints', mock_list_namespaced_endpoints, create=True)
def setUp(self, config=None):
super(TestKubernetesEndpointsNoPodIP, self).setUp({'use_endpoints': True})
@patch.object(k8s_client.CoreV1Api, 'patch_namespaced_endpoints', create=True)
def test_update_leader(self, mock_patch_namespaced_endpoints):
leader = self.k.get_cluster().leader
self.assertIsNotNone(self.k.update_leader(leader, '123', failsafe={'foo': 'bar'}))
args = mock_patch_namespaced_endpoints.call_args[0]
self.assertEqual(args[2].subsets[0].addresses[0].target_ref.resource_version, '1')
self.assertEqual(args[2].subsets[0].addresses[0].ip, '10.0.0.1')
class TestKubernetesEndpoints(BaseTestKubernetes):
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_endpoints', mock_list_namespaced_endpoints, create=True)
@@ -368,6 +382,7 @@ class TestKubernetesEndpoints(BaseTestKubernetes):
self.assertIsNotNone(self.k.update_leader(leader, '123', failsafe={'foo': 'bar'}))
args = mock_patch_namespaced_endpoints.call_args[0]
self.assertEqual(args[2].subsets[0].addresses[0].target_ref.resource_version, '10')
self.assertEqual(args[2].subsets[0].addresses[0].ip, '10.0.0.0')
self.k._kinds._object_cache['test'].subsets[:] = []
self.assertIsNotNone(self.k.update_leader(leader, '123'))
self.k._kinds._object_cache['test'].metadata.annotations['leader'] = 'p-1'
tox.ini
+10 -5
View File
@@ -125,10 +125,11 @@ commands =
--file features/Dockerfile
allowlist_externals = docker
[testenv:pg{12,13,14,15,16}-docker-behave-{etcd}-{lin,mac}]
[testenv:pg{12,13,14,15,16}-docker-behave-{etcd,etcd3}-{lin,mac}]
description = Run behaviour tests in patroni-dev docker container
setenv =
etcd: DCS=etcd
etcd3: DCS=etcd3
{[common]postgres_matrix}
CONTAINER_NAME = tox-{env_name}-{env:PYTHONHASHSEED}
labels =
@@ -149,7 +150,7 @@ commands =
--tty \
{env:PATRONI_DEV_IMAGE:patroni-dev:{env:PG_MAJOR}} \
tox run -x 'tox.env_list=py{[common]python_matrix}-behave-{env:DCS}-lin' \
-- --format plain {posargs}
-- {posargs}
allowlist_externals =
docker
@@ -159,7 +160,7 @@ platform =
; win: win32
mac: darwin
[testenv:py{36,38,39,310,311}-behave-{etcd}-{lin,win,mac}]
[testenv:py{36,38,39,310,311}-behave-{etcd,etcd3}-{lin,win,mac}]
description = Run behaviour tests (locally with tox)
deps =
-r requirements.txt
@@ -167,11 +168,15 @@ deps =
coverage
{[common]psycopg_deps}
setenv =
DCS = {env:DCS:etcd}
etcd: DCS = {env:DCS:etcd}
etcd3: DCS = {env:DCS:etcd3}
passenv =
ETCD_UNSUPPORTED_ARCH
commands =
python3 -m behave {posargs}
python3 -m behave --format json --format plain --outfile result.json {posargs}
mv result.json features/output
allowlist_externals =
mv
platform =
{[common]platforms}