when master node was being attached to patroni/zookeeper (no cluster in
zookeeper yet) patroni has never tried to "refetch" cluster from DCS.
It was leeding to demote...
1. run touch_member from the main loop
2. move code which takes care about long tasks into separate class
3. change format of data stored in a DCS: use json instead of url
4. change Member class: from now it deserialize everything into data property
5. rework API: from now it takes into account state of the current node in a dcs
Implementation is done on top of feature/is-healthiest-via-api and
feature/api branches.
In order to trigger manual failover one has to create 'failover' key in
a configuration store with the value in following format:
'leader_name:member_name'
leader_name can be empty or should match with the name of current leader
member_name can be empty or should match with the name one of cluster
nodes
Leader always checks that either desired member (if specified) or one of
the memners is accessible and healthy before demote.
After leader has deomted himself other nodes are performig checks that
desired node is healthy. If it is not they are participating in a leader
race. In some cases (when accidently there is no healthy nodes) former
leader can also participate in a leader race.
Current implementation does not provide REST API endpoint for a manual
failover.
POST /restart -- will restart postgres
You you are restartung leader node, lock would be maintained during
restart.
POST /reinitialize -- will reinitialize node from the leader.
It's not possible to reinitialize current leader.
Command will fail when the leader is unknown.
query method in an api.py also needs retry in some cases (for example
when we are running is_healthiest_node check).
In all cases we should retry only when connection is closed or broken.
BUT, the connection status must be checked via cursor.connection (old
implementation was using general connection object for that). For
multi-threaded applications this is not appropriate, because some other
thread might restore connection.
In addition to that I've changed most of the unit tests to use `Mock` and
`patch` where it is possible.
query method in an api.py also needs retry in some cases (for example
when we are running is_healthiest_node check).
In all cases We will retry only when connection is closed or broken, BUT
connection status must be checked via cursor.connection (old
implementation was using general connection object for that). For
multi-threaded applications this is not appropriate, because some other
thread might restore connection.
In appdition to that I've changed most of the unit tests to use `Mock` and
`patch` where it is possible.
First time is triggers some shutdown mechanisms in a patroni, but the
second one just terminates patroni. Solution is pretty simple, we need
to ignore all other sigterms which are coming after the first one.
Current etcd implementation does not yet support timeout option when
`wait=true`: https://github.com/coreos/etcd/issues/2468
Originaly I've implemented `watch` method for `Etcd` class in a
following manner: if the leader key was updated just because master
needs to update ttl and watch timeout is not yet expired, I was
recalculating timeout and starting `watch` call once again.
Usually after "restart" we were getting urllib3.exceptions.TimeoutError.
The only possible way to recover after such exception - close socket and
establish a new connection. With pure http it's relatively cheap, but
with https and some kind of authorization on etcd side it would became
rather expensive and should be avoided.
Alexander Kukushkin