query method in an api.py also needs retry in some cases (for example
when we are running is_healthiest_node check).
In all cases we should retry only when connection is closed or broken.
BUT, the connection status must be checked via cursor.connection (old
implementation was using general connection object for that). For
multi-threaded applications this is not appropriate, because some other
thread might restore connection.
In addition to that I've changed most of the unit tests to use `Mock` and
`patch` where it is possible.
First time is triggers some shutdown mechanisms in a patroni, but the
second one just terminates patroni. Solution is pretty simple, we need
to ignore all other sigterms which are coming after the first one.
Current etcd implementation does not yet support timeout option when
`wait=true`: https://github.com/coreos/etcd/issues/2468
Originaly I've implemented `watch` method for `Etcd` class in a
following manner: if the leader key was updated just because master
needs to update ttl and watch timeout is not yet expired, I was
recalculating timeout and starting `watch` call once again.
Usually after "restart" we were getting urllib3.exceptions.TimeoutError.
The only possible way to recover after such exception - close socket and
establish a new connection. With pure http it's relatively cheap, but
with https and some kind of authorization on etcd side it would became
rather expensive and should be avoided.
user:passwd pair should be configured in restapi section of main
configuration file in following format:
restapi:
auth: 'username:password'
Plus implemented some simple routing mechanisms:
GET /foo => do_GET_foo()
POST /bar => do_POST_bar()
Cleanup the initialize flag if the initializing node fails
to bootstrap its PostgreSQL database.
Rename dcs.race to initialize, since we only call it for the
initialize flag. Factored out PostgreSQL bootstrapping code
into a separate function.
`Cluster.leader` is not reference to `Member` anymore, but to `Leader`
`Leader` class contains field `index` (update index). This field is very
useful for watching for events which changing leader key. Also `Leader`
contains `member` field, which should reference real member.
Alexander Kukushkin