query method in an api.py also needs retry in some cases (for example
when we are running is_healthiest_node check).
In all cases we should retry only when connection is closed or broken.
BUT, the connection status must be checked via cursor.connection (old
implementation was using general connection object for that). For
multi-threaded applications this is not appropriate, because some other
thread might restore connection.
In addition to that I've changed most of the unit tests to use `Mock` and
`patch` where it is possible.
Also define following properties:
* initialize_path
* members_path
* member_path
* leader_path
* leader_optime_path
And replace any occurrences of these strings or client_path calls in a
etcd and zookeeper implementations with given constants and properties.
If initializer node suddenly dies before the initialization is complete,
other nodes should try to take over.
Fix some unittests for etcd and zookeeper and add couple of new ones.
Cleanup the initialize flag if the initializing node fails
to bootstrap its PostgreSQL database.
Rename dcs.race to initialize, since we only call it for the
initialize flag. Factored out PostgreSQL bootstrapping code
into a separate function.
This method suppose to watch for changes of leader key if current node
is not leader and also it could watch for changes in a members list if
current conde is the leader.
`Cluster.leader` is not reference to `Member` anymore, but to `Leader`
`Leader` class contains field `index` (update index). This field is very
useful for watching for events which changing leader key. Also `Leader`
contains `member` field, which should reference real member.
List of ZooKeeper nodes could be periodically updated from Exhibitor
Since we know that each Exhibitor accompanies one ZooKeeper node, list
of Exhibitor nodes also maintained. Exhibitor assumes that all ZooKeeper
nodes are using the same client port, 2181. The same assumption is valid
for Exhibitor, it should always listen on the same port on all nodes.
Original list of Exhibitor nodes is cached and used as a fallback when
it failed ito query information with using maintained list.
In case if one member of a cluster is not available it will retry with
another one and fetch the new cluster configuration. Default timeout for
all requests to etcd is 5 seconds.
Initial cluster configuration can be resolved through:
1) /v2/members call on one of the cluster members on a client port
2) when it is possible to resolve hostname into multiple ip's it will
iterate through list and try to perform action from 1)
3) If there is discovery_srv defined in etcd section of config file it
will resolve peer addresses of all cluster members and will fetch
cluster configuration with using peer protocol by doing /members call on
a peer port
Alexander Kukushkin