We are doing number of attempts when trying to initialize replica using
different methods. Any of this attemp may create and put something into
data directory, what causes next attempts fail.
In addition to that improve logging when creating replica.
* reap children before and after running HA loop
When the Patroni is running in a docker container with the pid=1 it is
also responsible for reaping of all dead processes. We can't call
os.waitpid immediately after receiving SIGCHLD because it breaks
subprocess module. It simply stops receiving exit codes of the processes
it executes because these processes. That's why we just registering the
fact of receiving SIGCHLD and reaping children only after execution of
HA loop.
If the postmaster was dying for some reason, Patroni was able to detect
this fact only on the next iteration of HA loop, because zombie
processes where still there and it was possible to send 0 signal to it.
To avoid such situation we should also reap all dead processes before
executing HA loop.
* Don't rely on _cursor_holder when closing connection
it could happen that connection has been opened but not cursor...
* Don't "retry" when fetching current xlog location and it fails
On every iteration of HA loop we are updaing member key in DCS and among
other data there is current xlog location stored in the value.
If the postgres has died for some reason it is not possible to fetch
xlog position and we are just wasting retry_timeout/2 = 5 seconds there.
If this information will be missing from DCS during period of one HA
loop nothing should break. Patroni is not relying on this information
anyway. When it is doing manual or automatic failover it aways
communicates with other nodes directly to get the most fresh
infomation.
* Don't try to update leader optime when postgres is not 100% healthy
`update_lock` method is not only doing update of the leader lock but
also writes the most recent value of xlog position into optime/leader
key. If you know that postgres can be not 100% healthy because it is in
process of restart or recover we should not try to fetch current xlog
position and update 'optime/leader'. Previously we were using
`AsyncExecutor.busy` property for avoiding of such action, but I think
we should be more excpilicit and do the update only if we know that
postgres is 100% healty.
* Avoid retries when syncing replication slots.
Do not retry postgres queries that fetch, create and drop slots at the end of
the HA cycle. The complete run_cycle routine executes with the async_executor
lock. This lock is also used with scheduling operations like reinit or restart
in different threads. Looks like CPython threading class has fairness issues
when multiple threads try to acquire the same lock and one of them executes
long-running actions while holding it: the others have little chances of
acquiring the lock in order. To get around this issue, the long action (i.e.
retrying the query) is removed.
Investigation by Ants Aasma and Alexander Kukushkin.
When running this script using Python3, the output is bytestring instead of string.
We explicitly decode it to ensure checks further down are ok.
The trigger for this patch is:
ERROR: unable to get some of WALE backup parameters: 'expanded_size_bytes'
This error is send by etcd when Patroni is doing "watch" on leader key
which is never updated after creation and etcd cluster receives a lot of
updates, what cleans history of events.
Instead of doing watch on modifiedIndex + 1 we will do watch on X-Etcd-Index,
which is probably still available...
Previously, that was necessary in order to avoid repeating the rewind
after failure. Nowadays, depending on the failure, we either want to
retry (if PostgreSQL was not stopped on time or leader did not manage
to acquire a master role yet), or won't retry at all if the leader is
not available, assuming the replica role. In both cases, the hack with
setting the role to unknown seems to be unnecessary and actually stops
callbacks from running if rewind is done not from the first attempt.
loop_wait and ttl is configured cluster-wide via config key stored in
DCS. Depending on values of these parameters we are configuring
different kind of timeouts used in DCS controllers.
In order to fetch this configuration we first need to create DCS
controller and only after apply parameters...
Instead of empying the stale failover key as a master and bailing
out, continue with the healthiest node evaluation. This should make
the actual master acquire the leader key faster. Emit the warning
message as well and add unit tests.
- fix the false claim that Patroni in a paused mode may promote a master.
- merge all allowed API actions together.
- describe the payload of the API pause request as JSON.
When a node to promote dies before finishing the promote and the
cluster is in a standby mode, the failover key sticks indefinitely,
preventing any master to take over the leader role. Prevent it by
letting the node in a master role cleanup the failover key if the
node to failover is not present among the members.
The master check cannot be performed by the node role alone, since
the node will not change its cached role on a manual promote. We
need to check the DB state as well.
