Creates the cluster structure in DCS as long as the initialize
key does not exist.
The structure consists of the leader and member keys. Both are
intentionally set to never expire, in order to support running the
cluster with the master that doesn't run the Patroni (external master).
Changes to the DCS code as well, in order to support non-expiring
leader and member keys.
Some silly default settings had to be applied if Patroni is unable
to find the configuration file. In particular, the connect address
will point to the localhost. Perhaps we should avoid running with
wihtout the valid configuration altogether, but currently there is
a valid use-case for this behavior, namely the replicas that are
running with the inaccessible master and getting up-to-date with
WAL segments only.
Initially, when the Patroni cluster key is not there, the etcd
and consul modules return an empty cluster value, with the
initialize flag set to False and not None. However, some checks
explictely verify that this flag is None, specifically, the one
that decides whether the new cluster should be bootstrapped.
This leads to the master waiting for the whole loop_wait before
running the initdb (and showing the "waiting for leader to bootstrap"
error): on the second loop_wait, the /service/cluster_name flag
is already there (because of the touch_member code).
Since I don't see where the initialize = False is explictely
differentiated from the None value, I think it makes sense to
set it to None uniformely; that's AFAIK always the case if you
use Zookeeper.
Although such situation should not happen in reality (follow method is
not supposed to be called when when the node is holding leader lock and
postgres is running), but to be on the safe side it is better to
implement as much checks as possible, because this method could
potentially remove data directory.
Client class takes care about retrying when connection to the etcd node
fails. It calculates amount of retries and timeout depending on etcd
cluster size.
Etcd class should not retry when EtcdConnectionFailed exception is
raised (this case is already handled in the Client).
Besides that adjust retry timeouts in the Client class.
rewind is not possible when:
1) trying to rewind from themself
2) leader is not reachable
3) leader is_in_recovery
All these cases were leading to removing of data directory...
In all cases except 1) it should "retry" when leader will became
available and not is_in_recovery.
In addition for that make pg_ctl --timeout option configurable.
If the stop or start didn't succeeded during given timeout when demoting
master, role will be forcibly changed to 'unknown' and all needed
callbacks executed.
For starting up cluster easy with docker-compose.
And unify Dockerfile and scripts to be able to work with docker-compose
and the old one dev_patroni_cluster.sh script
