Previously, patroni would die after receiving an exception
other than RetryFailedError, etcd.EtcdException from etcd.
We have observed an AttributeError raised by etcd on some
occasions. With this change, we demote ourselves, but not
terminate on such exceptions.
By default, haproxy sens an OPTION request, which we didn't
handle until now. In addition, all haproxy requests that doesn't
examine the request body close the connection as soon as the status
code is obtained. Such behavior breaks BaseHTTPRequestHandler,
namely handle_one_request, which doesn't check for connection reset
by peer and throw this error on a higher level, but since we don't
call this function directly, there is no place in the code to catch
it, therefore, we have to patch this function in the base class.
In addition, patch the StreamRequestHandler finish() function in
order to handle the connection reset error.
Re-read the cluster from DCS right after the failover to supply
the correct new values to the API thread. Fix a typo.
In the initial implementation we were using the only option
--encoding=UTF8. In order to have pg_rewind working with postgresql-9.3
we have to enable data-checksums. The naive approach was to enable it
globaly but taking into account some performance degradation it's better
not to do it but make it possible to configure it.
In addition to that fix all problems with setting up password of default
postgres user: execute CREATE ROLE | ALTER ROLE depending on content of
pg_authid
Previously, the leader key was watched for changes after a failover. This resulted in a delay
of up to 10 seconds to report a healthy failover back to the client.
With this patch, we are not relying on the role of a member registered in the dcs anymore.
For managing Patroni clusters, the Patroni api can be used. For many tasks, a command line interface for
this api would be a useful addition. This commit adds patroncli (The name is still under debate).
The command line interface needs access to the DCS; this is required for any operation. For some tasks it is required
to have access to the Patroni api.
A small summary of the additions to get the cli/ctl started:
* Updated Docker image to use 'true' as the archive_command, to ensure disk not filling up during failover
testing.
* The cli currently can list members, failover a master and remove a given cluster from DCS.
* The cli can be configured with a command, for repeated access to the same DCS
* Added some simple tests for the cli, code coverage is very low
Tags are labels assigned to individual members in order
to alter its default behavior, i.e. exclude from the
leader election or indicate a possibility to create base
backups from the member.
This commit only adds support for setting tags in the
configuration file, exposes the tags to DCS /member subkey
and returns the tags in a response of the API request. At
the moment the tag names are not validated, nor they are
interpreted in any way.
Support for setting tags via the API is also in the scope
of further work.
If the namespace is not specified in a config file /service/ would be
used.
Also it's possible to use just '/' as a namespace. It means we would
have following structure:
/scope1
/scope2
...
Such situation could happen if we replaced all etcd nodes except one
which was used by patroni. After replacing the last node patroni will
try to execute request on all other nodes from machines_cache but non of
them are available. Michines cache would became empty and patroni will
stick to the latest node which was available in the machines_cache and
will never try to refresh machines_cache from dns for example.
Currently machines cache is refreshed only when one request to the etcd
cluster has failed, but probably it should be done periodically, for
example every minute...
One can use pgpass configuration parameter in the postgres
subsection of Patroni. By default pgpass is written in ~/.
Mock actual writes to pgpass in the tests.
I accidentially removed the call when moving the backup functions
to the external script. It is intended to save the configuration,
so that at the restore phase one can just copy backup files.
Its primary intention was to save configuration files in the WAL-E
case (WAL-E just omits everything with .conf), but it is also
useful in the pg_basebackup case, which omits all symlinks, leaving
the cluster with .conf files symlinked in the broken state.
If the value is empty and leader didn't changed, this probably means
that failover failed.
After 15 seconds timeout we will consider failover status = unknown
curl -XPOST --data '{"leader": "leader_name", "member": "member_name"}' http://127.0.0.1:8008/failover
It will execute some preliminary checks and write failover key into DCS.
Afterward it will wait until new leader key will appear in a DCS.
It's better to execute this request on the master node. It will send a
signal to the main HA loop which makes possible to release leader key
immidiately even if you are working with etcd.
Feike Steenbergen