Any node of the cluster will maintain it's member key until Patroni is
running there.
Master node will also maintain the leader key until postgres is running
as a master. If there is not postgres or it is running 'in_recovery',
Patroni will release leader lock.
Bootstrap of a new cluster will work (it is possible to specify
paused: true) in the `bootstrap.dcs`. Replicas also will be able to join
the cluster if the leader lock exist.
If the postgres is not running on the node it will not try to bring it
up. Also it disables reinitialize and all kind of scheduled actions, i.e.
scheduled restart and scheduled failover.
In case if DCS stops being reachable Patroni will not "demote" master if
the automatic failover was disabled.
Patroni will not stop postgres on exit.
When Patroni was "joining" already running postgres it was not calling
callbacks, what in some cases causing issues (callback could be used to
change routing/load-balancer or assign/remove floating (service) ip.
In addition to that we should `start` postgres instead of `restart`-ing
it when doing recovery, because in this case 'on_start' callback should
be called, instead of 'on_restart'
In the original code we were parsing/deparsing url-style connection
strings back and forth. That was not really resource greedy but rather
annoying. Also it was not really obvious how to switch all local
connections to unix-sockets (preferably).
This commit isolates different use-cases of working with connection
strings and minimizes amount of code parsing and deparsing them. Also it
introduces one new helper method in the `Member` object - `conn_kwargs`.
This method can accept as a parameter dict object with credentials
(username and password). As a result it returns dict object which could
be used by `psycopg2.connect` or for building connection urls for
pg_rewind, pg_basebackup or some other replica creation methods.
Params for local connection are builded in the `_local_connect_kwargs`
method and could be changed to unix-socket later easily.
There is no point to try to update topology until original request is
not performed. Also for us it is more important to execute original
request rather then keep topology of etcd cluster in sync.
In addition to that implement the same retry-timeout logic in the
`machines` property which already is used in `api_execute` method.
* Make different kazoo timeouts dependant on loop_wait
ping timeout ~ 1/2 * loop_wait
connect_timeout ~ 1/2 * loop_wait
Originally these values were calculated from negotiated session timeout
and didn't worked very well, because it was taking significant time to
figure out that connection is dead and reconnect (up to session timeout)
and not giving us time to retry.
* Address the code review
- We don't want to export RestApi object, since it initializes the
socket and listens on it.
- Change get_dcs, so that the explicit scope passed to it will take
priority over the one in the configuration file.
- Add a new param to the abstract DCS attempt_to_take_leader
- Make sure the cluster is wiped-out properly if we created the
initialize key, but failed to populate it with leader and member.
This actually means that we may wipe out the running cluster without
the intialization key, but that is a very unlikely case in practice.
Creates the cluster structure in DCS as long as the initialize
key does not exist.
The structure consists of the leader and member keys. Both are
intentionally set to never expire, in order to support running the
cluster with the master that doesn't run the Patroni (external master).
Changes to the DCS code as well, in order to support non-expiring
leader and member keys.
Some silly default settings had to be applied if Patroni is unable
to find the configuration file. In particular, the connect address
will point to the localhost. Perhaps we should avoid running with
wihtout the valid configuration altogether, but currently there is
a valid use-case for this behavior, namely the replicas that are
running with the inaccessible master and getting up-to-date with
WAL segments only.
Implement scheduled restarts for Patroni via the API.
Even normal (immediate) restarts will take advantage of additional modifiers you can supply to the restart endpoint:
- restart_pending: restart only if the pending restart flag is set (because of the configuration change)
- role: restart if the Postgres role is set to a certain value
- postgres_version (x.y.z) - restart if the current Postgres version is less than the one specified.
For the scheduled restart, the schedule parameter can be used the same way as it is currently used for the scheduled failovers. Particularly, we don't allow restarts in the past, and always require the timezone to be present in the request in order to avoid client/server TZ difference issues.
Unify the code that evaluates the schedule for the scheduled restarts and scheduled failovers.
Use the RLock instead of Lock in the async_executor to avoid hanging if the thread takes the lock multiple times (mostly for the with blocks in the api).
In particular, replace the fixed dates for the future actions
in the unit tests with those that depend on the current date,
avoiding the "timebomb" effect.
Initially, when the Patroni cluster key is not there, the etcd
and consul modules return an empty cluster value, with the
initialize flag set to False and not None. However, some checks
explictely verify that this flag is None, specifically, the one
that decides whether the new cluster should be bootstrapped.
This leads to the master waiting for the whole loop_wait before
running the initdb (and showing the "waiting for leader to bootstrap"
error): on the second loop_wait, the /service/cluster_name flag
is already there (because of the touch_member code).
Since I don't see where the initialize = False is explictely
differentiated from the None value, I think it makes sense to
set it to None uniformely; that's AFAIK always the case if you
use Zookeeper.
Alexander Kukushkin