Previously, pg_rewind was called only if a crashed master tried
to rejoin the cluster. It didn't cover the important case of a
master shut down cleanly, but with a combination of a smart
shutdown and subsequently a fast shutdown. Since out pg_rewind
code does not depend on the "uncleanness" of the master's shutdown,
we can call it unconditionally in all cases where the former master
tries to rejoin as a replica.
This resolves #167.
Previously, "without_leader" suffix was used in the name of methods
and functions that initialize a replica without an active replication
connection, and leader was part of the name for parameters and messages
that require an active replication conneciton. Since we support init
from the members other than the leader, those conventions have to be
changed.
At the moment we just replace the master with the
node at the 'clonefrom' tag if it's present. Master
should be available anyway, otherwise, it will not
even try to do cloning.
Acceptance tests:
https://github.com/zalando/patroni/pull/144/commits
The actual amount of time to establish the master and the replication
after the scheduled failover seems sufficient (15 seconds with the
failover in 10 seconds), but occasionally leads to test failures.
This is unlikely the test issue and should be investigated inside
the patroni.
Add an ability to specify the origin and the destination for
the replication works clause. Use this ability in the API
promotion test to ensure the replication from the former
replica to the former master.
Toggle the etcd debug logging and write the log to the test dir.
Make sure etcd and patroni are terminated when the tests finish
by sending SIGKILL in case SIGTERM does not work.
Make sure before.all code does the proper cleanup when the exception
is thrown.
Move etcd code into a separate class.
Reduce the number of global interdependencies.
Clearly define private members of PatroniController and EtcdController.
It would not make the QuantifiedCode entirely happy, since lettuce
passes the step argument to the step definition, that is not used
in the client code, but internally (via the @steps decorator on
the steps class), but that's the issue of the tool used.
Do not try to create replication slots on the replica for the
member that wants to replicate from it if the member's currently
holds the master role.
Remove a debug message.
Master shouldn't keep a replication slot for the members that
replicate from other members instead of the master (replicatefrom).
Otherwise, the master will keep collecting WAL segments that won't
be requested ever. Of course, if the destination of replicatefrom
is not part of the cluster, master should create the slot.
The replication method should have no_master flag set and
support getting the base backup from some external storage
(i.e. S3). At the moment we only support initialization of
replicas in the already existing cluster when no master is
present, since there is no 'one fits all' way to decide whether
to run initidb or wait for the replica data storage to become
available when dealing with the new cluster.
Rename the follow_the_leader to just follow, since the node to
be followed is not necessary a leader anymore. Extend the code
that manages replication slots to the non-master nodes if they
are mentioned in at least one replicatefrom tag.
Add the 3rd configuration in order to be able to run cascading
replicas.
Call normal follow the leader method from HA even during recovery.
This provides a single place that changes recovery.conf, making
it easier to plug in a cascading replica in the future.
Remove an obsolete demote function from PostreSQL module, modified
the tests.
PostgreSQL does not run a checkpoint during promition.
Since pg_rewind relies on the last checkpoint to get the timeline,
there is a short race condition right after the promotion, when
it can get the timeline wrong and fail. We work around this by
calling the checkpoint manually.
Make sure our test configuration does both archive and recovery.
- command is deleted from method_config without checking whether
it was there in the first place.
- write_recovery_conf is called before the recovery file is restored
from the backup location.
Previously, patroni would die after receiving an exception
other than RetryFailedError, etcd.EtcdException from etcd.
We have observed an AttributeError raised by etcd on some
occasions. With this change, we demote ourselves, but not
terminate on such exceptions.
By default, haproxy sens an OPTION request, which we didn't
handle until now. In addition, all haproxy requests that doesn't
examine the request body close the connection as soon as the status
code is obtained. Such behavior breaks BaseHTTPRequestHandler,
namely handle_one_request, which doesn't check for connection reset
by peer and throw this error on a higher level, but since we don't
call this function directly, there is no place in the code to catch
it, therefore, we have to patch this function in the base class.
In addition, patch the StreamRequestHandler finish() function in
order to handle the connection reset error.
Re-read the cluster from DCS right after the failover to supply
the correct new values to the API thread. Fix a typo.
Oleksii Kliukin