* Bump actions/checkout and actions/setup-python versions
* Install wheel to silence some warnings
Co-authored-by: Alexander Kukushkin <[email protected]>
If there is no `catalog_xmin` on physical slot on the primary the only further check that makes sense is whether the `hot_standby_feedback` is enabled.
Close https://github.com/zalando/patroni/issues/2438
Windows doesn't support `SIGTERM`, but our behave tests in majority of cases relying on Patroni graceful shutdown.
In order to emulate the behaviour we introduced the new REST API endpoint `POST /sigterm`. The endpoint works only on Windows and when `BEHAVE_DEBUG` environment variable is set.
Besides that some minor adjustments in behave tests were done. Mainly related to backslash-slash handling.
In addition to that improve test coverage on Windows by properly mocking access to filesystem and avoiding calling
`subprocess.call()`. Specifically, symlink creation on Windows requires Admin privileges and there is no `true.exe`.
If cluster has a create replica method that does not require a leader it can get triggered while bootstrap is running. If that method comes up with an accessible cluster faster than the bootstrap completes it will get promoted as the leader, only to lose leader lock when bootstrap completes. To fix this we only consider leaderless create replica methods if sysid is non-empty, i.e. there is no bootstrap running.
- We use `failover.leader` check as an indicator of switchover. The check on `failover.candidate` was unnecessary and incorrect.
- Add more unit-tests for switchover in sync mode
Close https://github.com/zalando/patroni/issues/2405
Co-authored-by: Polina Bungina <[email protected]>
On busy clusters with many logical replication slots the pg_replication_slot_advance () call affects the main HA loop and could result in the member key expiration.
The only way to solve it is a dedicated thread response for moving slots forward.
The thread is started only when there are logical slots to be advanced.
Will help to solve #2388Close#2239
There is currently a risk to lose some WAL segments entirely in case
archive_mode was set to 'on' before a promotion and there are some WALs
with .ready files on the former leader we are trying to rewind. It happens
because of the pg_rewind's modus operandi: it simply syncs the content of
pg_wal directory of the old leader with the new leader's one. Including
deletion of all WALs that are not present on the current leader, regardless
their archive status on the former one. Thus, in case the new leader has
already recycled such files, we just remove them entirely.
In case archive_mode was set to 'always' and the .ready WALs are acrually
present in archive, it is for end user who writes the archive_command to
avoid overwritting and to properly test it.
Co-authored-by: Alexander Kukushkin <[email protected]>
If replication slots are enabled Patroni automatically creates them for any cluster member that is supposed to stream from a given node and for any permanent slot defined in the global configuration. If the member disappears from the DCS Patroni automatically removes the replication slot for it. The same behavior was in the maintenance mode (pause).
This commit disables removal of any replication slots that don't match Patroni's expectations in pause.
Close https://github.com/zalando/patroni/issues/2314
This adds a paragraph to the Standby Cluster section clarifying that the standby cluster is independent of the primary cluster and not visible from the primary cluster's Patroni interface.
Close#2090
It makes little sense to have ` for the Patroni service – if it goes down and we don't notice it (I doubt that patroni service uptime is well monitored in most cases), then we lose autofailover for Postgres, implying bigger risks of downtime.
`Restart=on-failure` makes more sense in this case.
High CPU load on Etcd nodes and K8s API servers created a very strange situation. A few clusters were running without a leader and the pod which is ahead of others was failing to take a leader lock because updates were failing with HTTP response code `409` (`resource_version` mismatch).
Effectively that means that TCP connections to K8s master nodes were alive (in the opposite case tcp keepalives would have resolved it), but no `UPDATE` events were arriving via these connections, resulting in the stale cache of the cluster in memory.
The only good way to prevent this situation is to intercept 409 HTTP responses and terminate existing TCP connections used for watches.
Now a few words about implementation. Unfortunately, watch threads are waiting in the read() call most of the time and there is no good way to interrupt them. But, the `socket.shutdown()` seems to do this job. We already used this trick in the Etcd3 implementation.
This approach will help to mitigate the issue of not having a leader, but at the same time replicas might still end up with the stale cluster state cached and in the worst case will not stream from the leader. Non-streaming replicas are less dangerous and could be covered by monitoring and partially mitigated by correctly configured `archive_command` and `restore_command`.
The current state of permanent logical replication slots on the primary is queried together with `pg_current_wal_lsn()` and hence they "fail" simultaneously if Postgres isn't yet ready for accepting connections and in this case we want to avoid updating the `/status` key altogether.
On K8s we don't use a dedicated object for the `/status` key, but use the same object (Endpoint or ConfigMap) as for the leader. If the `last_lsn` isn't set we avoid patching the corresponding annotation, but, the `slots` annotation was reset due to the oversight.
