query method in an api.py also needs retry in some cases (for example
when we are running is_healthiest_node check).
In all cases we should retry only when connection is closed or broken.
BUT, the connection status must be checked via cursor.connection (old
implementation was using general connection object for that). For
multi-threaded applications this is not appropriate, because some other
thread might restore connection.
In addition to that I've changed most of the unit tests to use `Mock` and
`patch` where it is possible.
First time is triggers some shutdown mechanisms in a patroni, but the
second one just terminates patroni. Solution is pretty simple, we need
to ignore all other sigterms which are coming after the first one.
Current etcd implementation does not yet support timeout option when
`wait=true`: https://github.com/coreos/etcd/issues/2468
Originaly I've implemented `watch` method for `Etcd` class in a
following manner: if the leader key was updated just because master
needs to update ttl and watch timeout is not yet expired, I was
recalculating timeout and starting `watch` call once again.
Usually after "restart" we were getting urllib3.exceptions.TimeoutError.
The only possible way to recover after such exception - close socket and
establish a new connection. With pure http it's relatively cheap, but
with https and some kind of authorization on etcd side it would became
rather expensive and should be avoided.
Previous implementation was always setting Cluster.initialize to True.
Also it was throwing ZooKeeperError when there were no members in a
cluster.
Plus BUGFIX of a bug introduced with
https://github.com/zalando/patroni/pull/34 in a `load_members` method.
- data = self.get_node(self.member_path)
+ data = self.get_node(self.members_path + member)
It was always fetching the same node for all cluster members.
Fortunately Etcd doesn't have such problem because we are fetching the
whole cluster directory with one recursive API call.
user:passwd pair should be configured in restapi section of main
configuration file in following format:
restapi:
auth: 'username:password'
Plus implemented some simple routing mechanisms:
GET /foo => do_GET_foo()
POST /bar => do_POST_bar()
Also define following properties:
* initialize_path
* members_path
* member_path
* leader_path
* leader_optime_path
And replace any occurrences of these strings or client_path calls in a
etcd and zookeeper implementations with given constants and properties.
If config['restapi']['certfile'] is specified and not empty http server
would be wrapped into SSL and api connection string changed accordingly:
http:// => https://
If initializer node suddenly dies before the initialization is complete,
other nodes should try to take over.
Fix some unittests for etcd and zookeeper and add couple of new ones.
Cleanup the initialize flag if the initializing node fails
to bootstrap its PostgreSQL database.
Rename dcs.race to initialize, since we only call it for the
initialize flag. Factored out PostgreSQL bootstrapping code
into a separate function.
This method suppose to watch for changes of leader key if current node
is not leader and also it could watch for changes in a members list if
current conde is the leader.
`Cluster.leader` is not reference to `Member` anymore, but to `Leader`
`Leader` class contains field `index` (update index). This field is very
useful for watching for events which changing leader key. Also `Leader`
contains `member` field, which should reference real member.
Alexander Kukushkin