When Patroni was "joining" already running postgres it was not calling
callbacks, what in some cases causing issues (callback could be used to
change routing/load-balancer or assign/remove floating (service) ip.
In addition to that we should `start` postgres instead of `restart`-ing
it when doing recovery, because in this case 'on_start' callback should
be called, instead of 'on_restart'
* Make different kazoo timeouts dependant on loop_wait
ping timeout ~ 1/2 * loop_wait
connect_timeout ~ 1/2 * loop_wait
Originally these values were calculated from negotiated session timeout
and didn't worked very well, because it was taking significant time to
figure out that connection is dead and reconnect (up to session timeout)
and not giving us time to retry.
* Address the code review
Fix return value in the should_run_scheduled_action and the comments.
Correct the json composition in the scheduled_restart test.
Fix the delete in case there is no scheduled restart.
Fix the usage of format in the logger output.
Fix the indentation in the evaluate_scheduled_restart.
Fix the condition related to the body_is_optional in the do_POST_restart.
Fix a few typos in the error messages.
Fix the _read_json_content
Make the scheduled restart unit-tests a bit less ugly
Make sure the scheduled restart flag is cleared when the
postmaster_start_time changes since the time restart was scheduled.
Additionally, separate the logic of checking the restart conditions
into the function in order to support conditions for the normal
restart as well.
The scheduled restart data structures are now independent of those
used by the normal restarts. This would be fixed in subsequent
commits.
Add the behave tests, that cover the POST /restart (but not DELETE).
The scheduled restart API extends the already existing restart
endpoint by processing the parameters in the request body.
Only one scheduled restart at a time is support. DELETE method
on the /restart endpoint is used to remove an existing restart.
And some other tricks with overriding handle_one_request and finish
methods from the parent class which were necessary only to make OPTIONS
request from haproxy work with python2, but in fact it was still not
working with python3. Instead of doing all the magic we should simply
give to haproxy what it wants to get: HTTP response code and nothing
more.
Mostly this tag is necessary to give a hint to load balancer
auto-configuration tool that node should not be included into
LB configuration.
In addition to that Patroni also should not return status_code=200
for a health check if the tag is present and value is not `False`.
By default, haproxy sens an OPTION request, which we didn't
handle until now. In addition, all haproxy requests that doesn't
examine the request body close the connection as soon as the status
code is obtained. Such behavior breaks BaseHTTPRequestHandler,
namely handle_one_request, which doesn't check for connection reset
by peer and throw this error on a higher level, but since we don't
call this function directly, there is no place in the code to catch
it, therefore, we have to patch this function in the base class.
In addition, patch the StreamRequestHandler finish() function in
order to handle the connection reset error.
Re-read the cluster from DCS right after the failover to supply
the correct new values to the API thread. Fix a typo.
Tags are labels assigned to individual members in order
to alter its default behavior, i.e. exclude from the
leader election or indicate a possibility to create base
backups from the member.
This commit only adds support for setting tags in the
configuration file, exposes the tags to DCS /member subkey
and returns the tags in a response of the API request. At
the moment the tag names are not validated, nor they are
interpreted in any way.
Support for setting tags via the API is also in the scope
of further work.
If the value is empty and leader didn't changed, this probably means
that failover failed.
After 15 seconds timeout we will consider failover status = unknown
curl -XPOST --data '{"leader": "leader_name", "member": "member_name"}' http://127.0.0.1:8008/failover
It will execute some preliminary checks and write failover key into DCS.
Afterward it will wait until new leader key will appear in a DCS.
It's better to execute this request on the master node. It will send a
signal to the main HA loop which makes possible to release leader key
immidiately even if you are working with etcd.
1. run touch_member from the main loop
2. move code which takes care about long tasks into separate class
3. change format of data stored in a DCS: use json instead of url
4. change Member class: from now it deserialize everything into data property
5. rework API: from now it takes into account state of the current node in a dcs
POST /restart -- will restart postgres
You you are restartung leader node, lock would be maintained during
restart.
POST /reinitialize -- will reinitialize node from the leader.
It's not possible to reinitialize current leader.
Command will fail when the leader is unknown.
Alexander Kukushkin