Otherwise, the etcd (not etcd3) behave tests fail to connect:
```
Jan 02 09:56:18 HOOK-ERROR in before_all: AssertionError: etcd instance is not available for queries after 5 seconds
```
When doing the leader race we need to check that the former primary isn't alive anymore. For that we relied on non-inclusive terms. In order to simplify future work on getting rid from all non-inclusive words we change the check to rely on a difference in format of wal/xlog field. There is only "location" for the primary and "replayed_location" + "received_location" for standbys.
In addition to that we start supporting "wal" field as well as deprecated "xlog".
Co-authored-by: Polina Bungina <[email protected]>
In the `find_executable()` function we split the extension of the file name passed from the base of the file name. In the case of WIN32, patroni will append a `.exe` if the current extension is not `.exe`.
This brings the wrong behavior given that `more` in WIN32 is `more.com` and not `more.exe`. This also makes the pager setting from `ctl.py` fail as the code changes `more.com` (hardcoded in `ctl.py`) for `more.com.exe`.
This change adjusts the behavior only to add the `.exe` if the executable variable doesn't have an extension.
A better solution would be to *not* modify the name of the executable that is being passed, as that is what we are looking for. But that might have other consequences, so it would require further testing.
Signed-off-by: Martín Marqués <[email protected]>
If openssl binary is available use it to generate a self-signed certificate. Use it to protect Patroni REST API
(`verify_client: required`).
In case if Postgres is compiled with SSL support enable it in the configuration and configure pg_hba.conf to check client certificates (`verify-ca`) in addition to passwords. Also configure superuser/replication/rewind users to use client certificates and verify server certificate (`verify-ca`)
There is a known [vector of attact](https://pganalyze.com/blog/5mins-postgres-security-patch-releases-pgspot-pghostile) by creating functions and/or operators in a public scheme with the same name and signature as corresponding objects in `pg_catalog`.
Since Patroni is heavily relying on superuser connections we want to mitigate it by enforcing `search_path=pg_catalog` for all connections created by Patroni (except replication connections). It is achieved by introducing a new function, that wraps psycopg.connect() and appends ` -c search_path=pg_catalog` to `options` parameter.
In addition to that, we set connection.autocommit to True before returning it.
pg_stat_statements is enabled by many by default, but will by default also track utility commands including an
ALTER USER john WITH PASSWORD 's3cret'
We can prevent this leaking by ensuring that our session currently does not track utility commands when running a sensitive query.
Local testing shows that this command works fine, even for those that do not have `pg_stat_statements` configured in their
`shared_preload_libraries`.
Fix issue of patroni configuring watchdog with defaults when bootstrapping a new cluster rather than taking configuration used to bootstrap the DCS.
Also log changes to watchdog configuration based on calculated timeout value.
Close#2470
Previously such an exception was raised only from the `get_cluster()` method, and now we will to do the same from the `update_leader()` and `attempt_to_acquire_leader()` methods.
These methods influence Postgres promotion and demotion and we want to make a difference between different types of failures. Specifically, if calls have failed because DCS isn't accessible or due to a timeout.
This commit is extracted from the #2379
* Remove patronictl configure command
* Change name of the "secret" ENV variable (DCS->DCS_URL) and the corresponding patronictl option (to avoid mixing it up with the one from tests)
The only advantage of localkube was being a low weight. Anything else started creating only problems:
1. It is not properly maintained for many years.
2. It effectively worked only on Linux, but stopped on modern version due to changes in iptables.
Instead, we will use widely adoped tools like kind or k3s. The "kind-kind" is the default K8s context (see ~/.kube/config), but it could be overriden using `PATRONI_KUBERNETES_CONTEXT` environment variable. When executed from GH actions the context is set to k3d-k3s-default, because K3s is much faster to start.
The HAProxy is closing connections as soon as it got the HTTP Status code leaving no time for Patroni to properly shutdown SSL connection.
Close https://github.com/zalando/patroni/issues/2466
It's effectively non-required when used in patroni/postgresql/config.py. This breaks most configuration files in the wild, starting from the examples:
$ patroni --validate-config postgres0.yml
restapi.connect_address 127.0.0.1:8008 didn't pass validation: 'must not contain "127.0.0.1", "0.0.0.0", "*", "::1", "localhost"'
etcd.host 127.0.0.1:2379 didn't pass validation: '127.0.0.1:2379 is not reachable'
postgresql.connect_address 127.0.0.1:5432 didn't pass validation: 'must not contain "127.0.0.1", "0.0.0.0", "*", "::1", "localhost"'
postgresql.proxy_address is not defined.
* bump version
* update release notes
* run some behave tests on v15
* automate release process by building/pushing packages on tag creation and release publication
* Bump actions/checkout and actions/setup-python versions
* Install wheel to silence some warnings
Co-authored-by: Alexander Kukushkin <[email protected]>
If there is no `catalog_xmin` on physical slot on the primary the only further check that makes sense is whether the `hot_standby_feedback` is enabled.
Close https://github.com/zalando/patroni/issues/2438
Windows doesn't support `SIGTERM`, but our behave tests in majority of cases relying on Patroni graceful shutdown.
In order to emulate the behaviour we introduced the new REST API endpoint `POST /sigterm`. The endpoint works only on Windows and when `BEHAVE_DEBUG` environment variable is set.
Besides that some minor adjustments in behave tests were done. Mainly related to backslash-slash handling.
In addition to that improve test coverage on Windows by properly mocking access to filesystem and avoiding calling
`subprocess.call()`. Specifically, symlink creation on Windows requires Admin privileges and there is no `true.exe`.
If cluster has a create replica method that does not require a leader it can get triggered while bootstrap is running. If that method comes up with an accessible cluster faster than the bootstrap completes it will get promoted as the leader, only to lose leader lock when bootstrap completes. To fix this we only consider leaderless create replica methods if sysid is non-empty, i.e. there is no bootstrap running.
- We use `failover.leader` check as an indicator of switchover. The check on `failover.candidate` was unnecessary and incorrect.
- Add more unit-tests for switchover in sync mode
Close https://github.com/zalando/patroni/issues/2405
Co-authored-by: Polina Bungina <[email protected]>
On busy clusters with many logical replication slots the pg_replication_slot_advance () call affects the main HA loop and could result in the member key expiration.
The only way to solve it is a dedicated thread response for moving slots forward.
The thread is started only when there are logical slots to be advanced.
Will help to solve #2388Close#2239
There is currently a risk to lose some WAL segments entirely in case
archive_mode was set to 'on' before a promotion and there are some WALs
with .ready files on the former leader we are trying to rewind. It happens
because of the pg_rewind's modus operandi: it simply syncs the content of
pg_wal directory of the old leader with the new leader's one. Including
deletion of all WALs that are not present on the current leader, regardless
their archive status on the former one. Thus, in case the new leader has
already recycled such files, we just remove them entirely.
In case archive_mode was set to 'always' and the .ready WALs are acrually
present in archive, it is for end user who writes the archive_command to
avoid overwritting and to properly test it.
Co-authored-by: Alexander Kukushkin <[email protected]>
If replication slots are enabled Patroni automatically creates them for any cluster member that is supposed to stream from a given node and for any permanent slot defined in the global configuration. If the member disappears from the DCS Patroni automatically removes the replication slot for it. The same behavior was in the maintenance mode (pause).
This commit disables removal of any replication slots that don't match Patroni's expectations in pause.
Close https://github.com/zalando/patroni/issues/2314
