The old strategy was waiting for 1 second and hoping that we will get an update event from the WATCH connection.
Unfortunately, it didn't work well in practice. Instead, we will get the current value from the API by performing an explicit read request.
Close https://github.com/zalando/patroni/issues/1767
This commit makes it possible to configure the maximum lag (`maximum_lag_on_syncnode`) after which Patroni will "demote" the node from synchronous and replace it with another node.
The previous implementation always tried to stick to the same synchronous nodes (even if they are not optimal ones).
If an existing instance was configured with WAL residing outside of
PGDATA then currently a 'reinit' would lose such symlinks. So add some
bits of information on that to draw attention to this cornercase issue
and also add the --waldir option to the sample `postgresql.basebackup`
configuration sections to increase visibility.
Discussion: https://github.com/zalando/patroni/issues/1817
It could happen that the cluster role wither not configured or doesn't provide enough permissions. In this case bypass_api_service is ignored, but the warning is logged, which is rather annoying when patronictl is used.
Since the bypass_api_service is most useful for Patroni, we will simply ignore it when patronictl is used.
The Python SSL library allows for the inclusion of a password in its "load_cert_chain" function when setting up a SSLContext[1].
This allows for loading an encrypted key file in PEM representation to be loaded into the certificate chain.
This commit adds the optional "keyfile_password" parameter to the REST API block of configuration so that Patroni can load in encrypted private keys when establishing its TLS socket.
This also adds the corollary "PATRONI_RESTAPI_KEYFILE_PASSWORD" environmental variable, which has the same effect.
[1] https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain
It could happen that one of etcd servers is not accessible on Patroni start.
In this case Patroni was trying to perform authentication and exiting, while it should exit only if Etcd explicitly responded with the `AuthFailed` error.
Close https://github.com/zalando/patroni/issues/1805
the good old python-consul is not maintained for a few years in a row, therefore someone forked under a different name, but package files are installed into the same location as for the old.
The API of both modules is mostly compatible therefore it wasn't hard to add the support of both modules in Patroni.
Taking into account that python-consul is not a direct requirement for Patroni, but extra, now the end-user has a choice what to install.
Close https://github.com/zalando/patroni/issues/1810
1. Fix flaky behave tests with zookeeper. First, install/start binaries (zookeeper/localkube) and only after that continue with installing requirements and running behave. Previously zookeeper didn't had enough time to start and tests sometimes were failing.
2. Fix flaky raft tests. Despite observations of MacOS slowness, for some unknown reason the delete test with a very small timeout was not timing out, but succeeding, causing unit-tests to fail. The solution - do not rely on the actual timeout, but mock it.
1. If the superuser name is different from postgres, the pg_rewind in the standby cluster was failing because the connection string didn't contain the database name.
2. Provide output if the single-user mode recovery failed.
Close https://github.com/zalando/patroni/pull/1736
Python 3.8 changed the way how exceptions raised from the Thread.run() method are handled.
It resulted in unit-tests showing a couple of warnings. They are not important and we just silence them.
When the__del__() method is executed the python interpreter already unloaded some of the modules that are still used down the http.clear() method.
The only we could do in this case - silence some exceptions like ReferenceError, TypeError, AttributeError.
Close https://github.com/zalando/patroni/issues/1785
When there is no config key in DCS Patroni shouldn't try accessing ignore_slots, otherwise an exception is raised.
In addition to that implement missing unit-tests and fix linting issues in behave tests.
When the watchdog device is not writable or missing in required mode, the member cannot be promoted. It only keeps logging that it is not the healthiest member.
Add a warning to show the user where to search for this misconfiguration.
When doing a custom bootstrap the configured value of max_connections could be smaller than the actual value stored in the pg_contol. As a workaround Patroni already did an adjustment of such parameters. Unfortunately, it doesn't cover the case when the parameter value was increased even higher during the WAL replay. That was causing postgres to stop.
The hot_standby=off disable connections to the instance, but allows postgres to run with parameter values smaller than required.
The hot_standby=off doesn't affect the running primary. When the recovery finishes and postgres promotes it starts accepting connections as usual. If the node will be demoted, Patroni will stop postgres and start it up as replica with hot_standby=on.
Fixes#1780
Specifically fixes the calls to `os.symlink` within the `move_data_directory` function, as used to update the symlink(s) for any WAL or tablespace directories following an init failure. The new name for a WAL and/or tablespace directory is now passed in as in as the first argument when calling `os.symlink`, while the second argument is now the name of the symlink that is being recreated.
This aligns with the Python documentation for `os.symlink`, which states the following regarding the first two arguments for the `os.symlink` function (arguments `src` and `dst`):
> Create a symbolic link pointing to src named dst (https://docs.python.org/3/library/os.html#os.symlink)
With this change, all WAL and/or tablespaces directories will be properly renamed following a init failure, and the `FileExistsError` that would otherwise occur when attempting to recreate the associated symlinks is no longer thrown, e.g.:
```bash
2020-12-05 20:35:42.219 GMT [262] LOG: unrecognized configuration parameter "invalid" in file "/pgdata/mycluster1/postgresql.auto.conf" line 5
2020-12-05 20:35:42.219 GMT [262] FATAL: configuration file "/pgdata/mycluster1/postgresql.auto.conf" contains errors
2020-12-05 20:35:43,220 ERROR: postmaster is not running
2020-12-05 20:35:43,223 INFO: removing initialize key after failed attempt to bootstrap the cluster
2020-12-05 20:35:43,292 INFO: renaming user defined tablespace directory and updating symlink: /tablespaces/ts1/ts1
2020-12-05 20:35:43,314 INFO: renaming data directory to /pgdata/mycluster1_2020-12-05-20-35-43
Traceback (most recent call last):
```
Additionally, any/all WAL and/or tablespace symlinks are properly recreated, e.g.:
```bash
$ ls -l pg_tblspc
total 0
lrwxrwxrwx. 1 postgres postgres 40 Dec 5 20:35 16414 -> /tablespaces/ts1/ts1_2020-12-05-20-35-43
```
1. The `ttl` was incorrectly returned 1000 times higher then it should
2. The `watch()` method must return True if the parent method returned True. Not doing so resulted in the incorrect calculation of sleep time.
3. Move mock of exhibitor api to the features/environment.py. It simplifies testing with behave.
This patch fixes the error handling of cases where there are runtime errors in `socketserver`.
For example, when creating a new thread (to handle a request) fails.
`get_request` handles ssl connections by replacing the new client socket by a tuple containing `(server_socket, new_client_socket)` in order to later deal with handshakes in `process_request_thread`
During the processing of a request, the socketserver `BaseServer` calls `handle_request`, calling the `_handle_request_noblock`, which is calling the following functions (https://github.com/python/cpython/blob/3.8/Lib/socketserver.py#L303):
```
request, client_addr = get_request()
verify_request(request, client_address):
process_request(request, client_address)
handle_error(request, client_address)
shutdown_request(request)
```
- `get_request` is overloaded in patroni and returns `request` as a tuple in case of ssl calls
- `verify_request` defaults to `return True` and should be fixed if used but is fine in this case
- `process_request` just calls `process_request_thread` (which is overloaded in patroni and handles tuple-style requests)
- `handle_error` is overloaded in patroni and handles tuple-style requests)
- but `shutdown_request` is not overloaded and thus missing support for tuple-style requests
This patch adds support for tuple-style requests in patroni api
There are sometimes good reasons to manage replication slots externally
to Patroni. For example, a consumer may wish to manage its own slots (so
that it can more easily track when a failover has a occurred and whether
it is ahead of or behind the WAL position on the new primary).
Additionally tooling like pglogical actually replicates slots to all
replicas so that the current position can be maintained on failover
targets (this also aids consumers by supplying primitives so that they
can verify data hasn't been lost or a split brain occurred relative to
the physical cluster).
To support these use cases this new feature allows configuring Patroni
to entirely ignore sets of slots specified by any subset of name,
database, slot type, and plugin.
Previously the only documentation for how to run tests was the
implementation in the Travis configuration file. Here we add
instructions as well as move development dependencies to an easily used
and shared (with Travis config) separate requirements.dev.txt file.
It is not very common, but the master Postgres might "crash" due to different reasons, like OOM, or out of disk space. Of course, there are chances that the current node holds some unreplicated data and therefore Patroni by default prefers to start Postgres on the leader node rather than doing a failover.
In order to be on the safe side Patroni always starts Postgres in recovery no matter whether the current node owns the leader lock or not. If the Postgres wasn't shut down cleanly, starting in recovery might fail, therefore in some cases as a workaround Patroni is executing a crash recovery by starting the postgres up in the single-user mode.
A few times we end up in the situation:
1. Master postgres crashed due to the out of disk space
2. Patroni starts crash recovery in a single-user mode
3. While doing crash-recovery Patroni keeps updating the leader lock
It makes Patroni stuck on step 3 and the manual intervention is required for recovering the cluster.
Patroni already has the option `master_start_timeout`, which controls for how long we let postgres stay in the `starting` state and after that Patroni might decide to release the leader lock if there are healthy replicas available which could take it over.
This PR makes the `master_start_timeout` option also work for crash recovery.
1. Don't call bootstrap if PGDATA is missing/empty, because it might be for purpose, and someone/something working on it.
2. Consider postgres running as a leader in pause not healthy if pg_control sysid doesn't match with the /initialize key (empty initialize key will allow the "race" and the leader will "restore" initialize key).
3. Don't exit on sysid mismatch in pause, only log a warning.
4. Cover corner cases when Patroni started in pause with empty PGDATA and it was restored by somebody else
5. Empty string is a valid `recovery_target`.
Unhandled exception prevented demoting the primary.
In addition to that wrap the update_leader call in the HA loop into try..except block and implement a test case.
Fixes https://github.com/zalando/patroni/issues/1684
* update release notes
* bump version
* change the default alignment in patronictl table output to `left`
* add missing tests
* add missing pieces to the documentation
It is now also possible to point the configuration path to a directory instead of a file.
Patroni will find all yml files in the directory and apply them in sorted order
Close https://github.com/zalando/patroni/issues/1669
