Instead of empying the stale failover key as a master and bailing
out, continue with the healthiest node evaluation. This should make
the actual master acquire the leader key faster. Emit the warning
message as well and add unit tests.
Any node of the cluster will maintain it's member key until Patroni is
running there.
Master node will also maintain the leader key until postgres is running
as a master. If there is not postgres or it is running 'in_recovery',
Patroni will release leader lock.
Bootstrap of a new cluster will work (it is possible to specify
paused: true) in the `bootstrap.dcs`. Replicas also will be able to join
the cluster if the leader lock exist.
If the postgres is not running on the node it will not try to bring it
up. Also it disables reinitialize and all kind of scheduled actions, i.e.
scheduled restart and scheduled failover.
In case if DCS stops being reachable Patroni will not "demote" master if
the automatic failover was disabled.
Patroni will not stop postgres on exit.
There is no point to try to update topology until original request is
not performed. Also for us it is more important to execute original
request rather then keep topology of etcd cluster in sync.
In addition to that implement the same retry-timeout logic in the
`machines` property which already is used in `api_execute` method.
* Make different kazoo timeouts dependant on loop_wait
ping timeout ~ 1/2 * loop_wait
connect_timeout ~ 1/2 * loop_wait
Originally these values were calculated from negotiated session timeout
and didn't worked very well, because it was taking significant time to
figure out that connection is dead and reconnect (up to session timeout)
and not giving us time to retry.
* Address the code review
In particular, replace the fixed dates for the future actions
in the unit tests with those that depend on the current date,
avoiding the "timebomb" effect.
Make sure the scheduled restart flag is cleared when the
postmaster_start_time changes since the time restart was scheduled.
Additionally, separate the logic of checking the restart conditions
into the function in order to support conditions for the normal
restart as well.
The scheduled restart API extends the already existing restart
endpoint by processing the parameters in the request body.
Only one scheduled restart at a time is support. DELETE method
on the /restart endpoint is used to remove an existing restart.
where it is not necessary (test_ha, test_ctl, etc...)
It will simplyfy further refactoring and make it possible to install
implementations of AbstractDCS independant of each other.
According to https://github.com/zalando/patroni/issues/48 'clonefrom'
tag should be boolean and it should be used to mark node as a suitable
for creation of a new replica from. If there are more then one such node
in the cluster (with tag clonefrom=true), one of them will be chosed
randomly.
Instead of starting it up in "read-only", it wil wait 2 seconds, to give
a time to somebody to prompte and after it will execute normal `recover`
procedure.
In addition to that rename confusing `Etcd.client` and
`ZooKeeper.client` into `_client`. This attribute is available from
AbstractDCS and people had wrong impression that it provides the same
interface for different DCS implementations, which is obviously not the
case. For Etcd it has type etcd.Client and for ZooKeeper - KazooClient.
Previously, "without_leader" suffix was used in the name of methods
and functions that initialize a replica without an active replication
connection, and leader was part of the name for parameters and messages
that require an active replication conneciton. Since we support init
from the members other than the leader, those conventions have to be
changed.
At the moment we just replace the master with the
node at the 'clonefrom' tag if it's present. Master
should be available anyway, otherwise, it will not
even try to do cloning.
Acceptance tests:
https://github.com/zalando/patroni/pull/144/commits
The replication method should have no_master flag set and
support getting the base backup from some external storage
(i.e. S3). At the moment we only support initialization of
replicas in the already existing cluster when no master is
present, since there is no 'one fits all' way to decide whether
to run initidb or wait for the replica data storage to become
available when dealing with the new cluster.
Alexander Kukushkin