During the shutdown Patroni is trying to update its status in the DCS.
If the DCS is inaccessible an exception might be raised. Lack of exception handling prevents logger thread from stopping.
Fixes https://github.com/zalando/patroni/issues/1344
It might be that they are defined by the extension and therefore the unit is not necessarily is the string.
It also could be that change of the value requires a restart (for example pg_stat_statements.max).
it serves two purposes:
1. We don't want accidentally break the thread
2. During the shutdown socket.gaierror become unresolvable and nasty exceptions are raised
Close: https://github.com/zalando/patroni/issues/1353
The standby_leader was already doing it from the beginning feature existed. Not doing the same on replicas might prevent them from catching up with standby leader due to WALs being recycled.
In addition to that apply the same strategy to archive_cleanup_command.
Upon the start of Patroni and Postgres make sure that unix_socket_directories and stats_temp_directory exist or try to create them. Patroni will exit if failed to create them.
Close https://github.com/zalando/patroni/issues/863
That required a refactoring of `Config` and `Patroni` classes. Now one has to explicitely create the instance of `Config` before creating `Patroni`.
The Config file can optionally call the validate function.
We will try to import only the module which has a configuration section.
I.e. if there is only zookeeper section in the config, Patroni will try to import only `patroni.dcs.zookeeper` and skip `etcd`, `consul`, and `kubernetes`.
This approach has two benefits:
1. When there are no dependencies installed Patroni was showing INFO messages `Failed to import smth`, which looks scary.
2. It reduces memory usage, because sometimes dependencies are heavy.
* Implement proper tests for `multiprocessing.set_start_method()`
* Exclude some watchdog code from coverage (it is used only for behave tests)
* properly use os.path.join for windows compatibility
* import DCS modules in `features/environment.py` on demand. It allows to run behave tests against chosen DCS without installing all dependencies.
* remove some unused behave code
* fix some minor issues in the dcs.kubernetes module
There is an opinion that LIST requests with labelSelector to K8s API are expensive and Patroni was doing two such requests per HA loop (LIST pods and LIST endpoints/configmaps).
To efficiently detect object changes we will switch to the LIST+WATCH approach.
The initial LIST request populates the ObjectCache and events from the WATCH request update it.
In addition to that, the ObjectCache will be updated after performing the UPDATE operations on the K8s objects. To avoid race conditions, all operations on ObjectCache are performed after comparing the resource_version of the old and the new objects and rejected if the new resource_version value is smaller than the old one.
The disadvantage of such an approach is that it will require keeping three connections to the K8s API from each Patroni Pod (previously it was two).
Yesterday I deployed this feature branch on our biggest K8s cluster, with ~300 Patroni pods.
The CPU Utilization on K8s master nodes immediately dropped from ~20% to ~10% (two times), and the incoming traffic on master nodes dropped ~7-8 times!
Last, but not least, we get more or less the same impact on etcd cluster behind K8s master nodes, the CPU Utilization dropped nearly twice and outgoing traffic ~7-8 times.
Starting from PostgreSQL 12 the following recovery parameters could be changed without restart, but Patroni didn't yet support it:
* archive_cleanup_command
* promote_trigger_file
* recovery_end_command
* recovery_min_apply_delay
In future postgres releases this list will be extended and Patroni will support it automatically.
The start of postgres happens in two stages:
1. First Patroni is waiting for postgres port to be open
2. After that, it is waiting for postgres starts to accept connections
There is a default timeout 60 seconds for both stages (in total).
When the port isn't open, pg_isready exits with code=2.
If postgres is rejecting connections due to recovery, exit code=1.
In most cases postgres quickly opens the port and pg_isready starts returning 1, but in rare cases the whole timeout could spend in `1.`
After that, the HA loop is still waiting for postgres to start, but executing only the check from `2.`. Since pg_isready exit code is still = 2, Patroni was falsely assuming that 'start failed' without taking into consideration the fact that the postmaster process is up and running.
Fixes https://github.com/zalando/patroni/issues/1160
The previous documentation was wrong and will throw the following error
when used:
Exception when parsing list {[{"name": "postgresql", "port": 5432}]}
When removing the surrounding braces, the error goes away and the
endpoint is updated with the correct Port name.
