Previously, that was necessary in order to avoid repeating the rewind
after failure. Nowadays, depending on the failure, we either want to
retry (if PostgreSQL was not stopped on time or leader did not manage
to acquire a master role yet), or won't retry at all if the leader is
not available, assuming the replica role. In both cases, the hack with
setting the role to unknown seems to be unnecessary and actually stops
callbacks from running if rewind is done not from the first attempt.
Instead of empying the stale failover key as a master and bailing
out, continue with the healthiest node evaluation. This should make
the actual master acquire the leader key faster. Emit the warning
message as well and add unit tests.
When a node to promote dies before finishing the promote and the
cluster is in a standby mode, the failover key sticks indefinitely,
preventing any master to take over the leader role. Prevent it by
letting the node in a master role cleanup the failover key if the
node to failover is not present among the members.
The master check cannot be performed by the node role alone, since
the node will not change its cached role on a manual promote. We
need to check the DB state as well.
When observing the leader running a master role, set the cached role
stored in the state_handler to master as well. Failure to do so
resulted in the manually promoted node to continue running with a
cached 'replica' role. This led to the failure to create replication
slots for the new replicas.
We could do it conditionally, but both reading and writing the role
require the same lock, and the unconditional approach makes the unit
tests simpler.
Master tried to delete all slots that did not correspond to the
replica registered in Patroni. That produced an error for the slots
that were active, potentially preventing drop and creation of other
slots.
Reported by Murat Kabilov.
- We don't want to export RestApi object, since it initializes the
socket and listens on it.
- Change get_dcs, so that the explicit scope passed to it will take
priority over the one in the configuration file.
- Add a new param to the abstract DCS attempt_to_take_leader
- Make sure the cluster is wiped-out properly if we created the
initialize key, but failed to populate it with leader and member.
This actually means that we may wipe out the running cluster without
the intialization key, but that is a very unlikely case in practice.
Creates the cluster structure in DCS as long as the initialize
key does not exist.
The structure consists of the leader and member keys. Both are
intentionally set to never expire, in order to support running the
cluster with the master that doesn't run the Patroni (external master).
Changes to the DCS code as well, in order to support non-expiring
leader and member keys.
Some silly default settings had to be applied if Patroni is unable
to find the configuration file. In particular, the connect address
will point to the localhost. Perhaps we should avoid running with
wihtout the valid configuration altogether, but currently there is
a valid use-case for this behavior, namely the replicas that are
running with the inaccessible master and getting up-to-date with
WAL segments only.
In particular, replace the fixed dates for the future actions
in the unit tests with those that depend on the current date,
avoiding the "timebomb" effect.
Initially, when the Patroni cluster key is not there, the etcd
and consul modules return an empty cluster value, with the
initialize flag set to False and not None. However, some checks
explictely verify that this flag is None, specifically, the one
that decides whether the new cluster should be bootstrapped.
This leads to the master waiting for the whole loop_wait before
running the initdb (and showing the "waiting for leader to bootstrap"
error): on the second loop_wait, the /service/cluster_name flag
is already there (because of the touch_member code).
Since I don't see where the initialize = False is explictely
differentiated from the None value, I think it makes sense to
set it to None uniformely; that's AFAIK always the case if you
use Zookeeper.
Fix return value in the should_run_scheduled_action and the comments.
Correct the json composition in the scheduled_restart test.
Fix the delete in case there is no scheduled restart.
Fix the usage of format in the logger output.
Fix the indentation in the evaluate_scheduled_restart.
Fix the condition related to the body_is_optional in the do_POST_restart.
Fix a few typos in the error messages.
Fix the _read_json_content
Make the scheduled restart unit-tests a bit less ugly
Make sure the scheduled restart flag is cleared when the
postmaster_start_time changes since the time restart was scheduled.
Additionally, separate the logic of checking the restart conditions
into the function in order to support conditions for the normal
restart as well.
The scheduled restart data structures are now independent of those
used by the normal restarts. This would be fixed in subsequent
commits.
Add the behave tests, that cover the POST /restart (but not DELETE).
The scheduled restart API extends the already existing restart
endpoint by processing the parameters in the request body.
Only one scheduled restart at a time is support. DELETE method
on the /restart endpoint is used to remove an existing restart.
Previously, pg_rewind was called only if a crashed master tried
to rejoin the cluster. It didn't cover the important case of a
master shut down cleanly, but with a combination of a smart
shutdown and subsequently a fast shutdown. Since out pg_rewind
code does not depend on the "uncleanness" of the master's shutdown,
we can call it unconditionally in all cases where the former master
tries to rejoin as a replica.
This resolves #167.
Previously, "without_leader" suffix was used in the name of methods
and functions that initialize a replica without an active replication
connection, and leader was part of the name for parameters and messages
that require an active replication conneciton. Since we support init
from the members other than the leader, those conventions have to be
changed.
At the moment we just replace the master with the
node at the 'clonefrom' tag if it's present. Master
should be available anyway, otherwise, it will not
even try to do cloning.
Acceptance tests:
https://github.com/zalando/patroni/pull/144/commits
The actual amount of time to establish the master and the replication
after the scheduled failover seems sufficient (15 seconds with the
failover in 10 seconds), but occasionally leads to test failures.
This is unlikely the test issue and should be investigated inside
the patroni.
Add an ability to specify the origin and the destination for
the replication works clause. Use this ability in the API
promotion test to ensure the replication from the former
replica to the former master.
Toggle the etcd debug logging and write the log to the test dir.
Make sure etcd and patroni are terminated when the tests finish
by sending SIGKILL in case SIGTERM does not work.
Make sure before.all code does the proper cleanup when the exception
is thrown.
Move etcd code into a separate class.
Reduce the number of global interdependencies.
Clearly define private members of PatroniController and EtcdController.
It would not make the QuantifiedCode entirely happy, since lettuce
passes the step argument to the step definition, that is not used
in the client code, but internally (via the @steps decorator on
the steps class), but that's the issue of the tool used.
