* Avoid retries when syncing replication slots.
Do not retry postgres queries that fetch, create and drop slots at the end of
the HA cycle. The complete run_cycle routine executes with the async_executor
lock. This lock is also used with scheduling operations like reinit or restart
in different threads. Looks like CPython threading class has fairness issues
when multiple threads try to acquire the same lock and one of them executes
long-running actions while holding it: the others have little chances of
acquiring the lock in order. To get around this issue, the long action (i.e.
retrying the query) is removed.
Investigation by Ants Aasma and Alexander Kukushkin.
When running this script using Python3, the output is bytestring instead of string.
We explicitly decode it to ensure checks further down are ok.
The trigger for this patch is:
ERROR: unable to get some of WALE backup parameters: 'expanded_size_bytes'
This error is send by etcd when Patroni is doing "watch" on leader key
which is never updated after creation and etcd cluster receives a lot of
updates, what cleans history of events.
Instead of doing watch on modifiedIndex + 1 we will do watch on X-Etcd-Index,
which is probably still available...
Previously, that was necessary in order to avoid repeating the rewind
after failure. Nowadays, depending on the failure, we either want to
retry (if PostgreSQL was not stopped on time or leader did not manage
to acquire a master role yet), or won't retry at all if the leader is
not available, assuming the replica role. In both cases, the hack with
setting the role to unknown seems to be unnecessary and actually stops
callbacks from running if rewind is done not from the first attempt.
loop_wait and ttl is configured cluster-wide via config key stored in
DCS. Depending on values of these parameters we are configuring
different kind of timeouts used in DCS controllers.
In order to fetch this configuration we first need to create DCS
controller and only after apply parameters...
Instead of empying the stale failover key as a master and bailing
out, continue with the healthiest node evaluation. This should make
the actual master acquire the leader key faster. Emit the warning
message as well and add unit tests.
- fix the false claim that Patroni in a paused mode may promote a master.
- merge all allowed API actions together.
- describe the payload of the API pause request as JSON.
When a node to promote dies before finishing the promote and the
cluster is in a standby mode, the failover key sticks indefinitely,
preventing any master to take over the leader role. Prevent it by
letting the node in a master role cleanup the failover key if the
node to failover is not present among the members.
The master check cannot be performed by the node role alone, since
the node will not change its cached role on a manual promote. We
need to check the DB state as well.
PostgreSQL replication slot names only allow names consisting of [a-z0-9_].
Invalid characters cause replication slot creation and standby startup to fail.
This change substitutes the invalid characters with underscores or unicode
codepoints. In case multiple member names map to identical replication slots
master log will contain a corresponding error message.
Motivated by wanting to use hostnames as member names. Hostnames often
contain periods and dashes.
Alexander Kukushkin