Task of restoring a cluster from backup or cloning existing cluster into a new one was floating around for some time. It was kind of possible to achieve it by doing a lot of manual actions and very error prone. So I come up with the idea of making the way how we bootstrap a new cluster configurable.
In short - we want to run a custom script instead of running initdb.
* Use more readable 'listen' section
* Explicitely defined check options
* Enable stats dashboard
* Close HAProxy connections to a backend server when it's marked as down.
as described here, https://github.com/postgres/postgres/blob/master/src/backend/replication/slot.c#L164-L210 the slot name should be truncated to 63 chars,
getting an error related to a slot_name
```
FATAL: replication slot name "c_formationid4_main_m_1_c_formationid4_main_m_nicksaccount_svc_c" is too long
```
the leader has the following slot names
```
postgres=# select * from pg_replication_slots;
slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_ls
n | confirmed_flush_lsn
-----------------------------------------------------------------+--------+-----------+--------+----------+--------+------------+------+--------------+-----------
--+---------------------
c_formationid4_main_m_1_c_formationid4_main_m_nicksaccount_svc_ | | physical | | | f | | | |
|
c_formationid4_main_m_2_c_formationid4_main_m_nicksaccount_svc_ | | physical | | | f | | | |
|
(2 rows)
```
On debian, the configuration files (postgresql.conf, pg_hba.conf, etc) are not stored in the data directory. It would be great to be able to configure the location of this separate directory. Patroni could override existing configuration files where they are used to be.
The default is to store configuration files in the data directory. This setting is targeting custom installations like debian and any others moving configuration files out of the data directory.
Fixes#465
This patch is adding the sleep function needed in the connection_retry and command_retry parameters. The KazooClient is trying to compare them with the ones included in the handler at instantiation.
So far Patroni was populating pg_hba.conf only when running bootstrap code and after that it was not very handy to manage it's content, because it was necessary to login to every node, change pg_hba.conf manually and run pg_ctl reload.
This commit intends to fix it and give Patroni control over pg_hba.conf. It is possible to define pg_hba.conf content via `postgresql.pg_hba` in the patroni configuration file or in the `DCS/config` (dynamic configuration).
If the `hba_file` is defined in the `postgresql.parameters`, Patroni will ignore `postgresql.pg_hba`.
For backward compatibility this feature is not enabled by default. To enable it you have to set `postgresql.use_unix_socket: true`.
If feature is enable, and `unix_socket_directories` is defined and non empty, Patroni will use the first suitable value from it to connect to the local postgres cluster.
If the `unix_socket_directories` is not defined, Patroni will assume that default value should be used and will not pass `host` to command line arguments and omit it from connection url.
Solves: https://github.com/zalando/patroni/issues/61
In addition to mentioned above, this commit solves couple of bugs:
* manual failover with pg_rewind in a pause state was broken
* psycopg2 (or libpq, I am not really sure what exactly) doesn't mark cursors connection as closed when we use unix socket and there is an `OperationalError` occurs. We will close such connection on our own.
because `boto.exception` is not an excpetion, but a python module.
+ increase retry timeout to 5 minutes
+ refactor unit-tests to cover the case with retries.
pg_controldata output depends on postgres major version and in some cases some of the parameters are prefixed by 'Current ' for old postgres versions.
Bug was introduced by commit 37c1552.
Fixes https://github.com/zalando/patroni/issues/455
Previously we were running pg_rewind only in limited amount of cases:
* when we knew postgres was a master (no recovery.conf in data dir)
* when we were doing a manual switchover to a specific node (no
guaranty that this node is the most up-to-date)
* when a given node has nofailover tag (it could be ahead of new master)
This approach was kind of working in most of the cases, but sometimes we
were executing pg_rewind when it was not necessary and in some other
cases we were not executing it although it was needed.
The main idea of this PR is first try to figure out that we really need
to run pg_rewind by analyzing timelineid, LSN and history file on master
and replica and run it only if it's needed.
Current UI to change cluster configuration is somewhat unfriendly, involving a curl command, knowing the REST API endpoint, knowing the specific syntax to call it with and writing a JSON document. I added two commands in this branch to make this a bit easier, `show-config` and `edit-config` (names are merely placeholders, any opinions on better ones?).
* `patronictl show-config clustername` fetches the config from DCS, formats it as YAML and outputs it.
* `patronictl edit-config clustername` fetches the config, formats it as YAML, invokes $EDITOR on it, then shows user the diff and after confirmation applies the changed config to DCS, guarding for concurrent modifications.
* `patronictl edit-config clustername --set synchronous_mode=true --set postgresql.use_slots=true` will set the specific key-value pairs.
There are also some UI capabilities I'm less sure of, but included them here as I already implemented them.
* If output is a tty then the diffs are colored. I'm not sure if this feature is cool enough to pull the weight of adding a dependency on cdiff. Or maybe someone knows of another more task focused diff coloring library?
* `patronictl edit-config clustername --pg work_mem=100MB` - Shorthand for `--set postgresql.parameters.work_mem=100MB`
* `patronictl edit-config clustername --apply changes.yaml` - apply changes from a yaml file.
* `patronictl edit-config clustername --replace new-config.yaml` - replace config with new version.
This was causing patroni to exit with following exception:
2017-04-27 08:54:25,000 CRITICAL: system ID mismatch, node foo-0 belongs to a different cluster: 6347967071236960319 !=
2017-04-27 08:54:25,221 ERROR:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/patroni/__init__.py", line 134, in patroni_main
patroni.run()
File "/usr/local/lib/python3.5/dist-packages/patroni/__init__.py", line 110, in run
logger.info(self.ha.run_cycle())
File "/usr/local/lib/python3.5/dist-packages/patroni/ha.py", line 946, in run_cycle
info = self._run_cycle()
File "/usr/local/lib/python3.5/dist-packages/patroni/ha.py", line 903, in _run_cycle
sys.exit(1)
SystemExit: 1
Avoid "postmaster became multithreaded during startup" error on OS X built with --enable-nls (default for petere/homebrew).
The issue is that on OS X the libintl that replaces setlocale() spawns a thread when it needs to detect the current locale. Setting the LC_ and LANG variables prevents this, but we don't propagate them to the fork.
See: https://www.postgresql.org/message-id/flat/20140902013458.GB906981%40tornado.leadboat.com#[email protected] for an additional explanation of the original Postgres behavior.
* Update the startup script shutdown / startup timeout to allow the current sync node time to release the role in synchronous_mode.
Without this change Patroni will be killed while waiting, and will leave
PostgreSQL up rather than stopping it.
* Updates to support strict sync mode.
If synchronous_mode_strict==true then 'patroni_dummy_host' will be written as synchronous_standby_names
when that last replication host dies.
* Oops, mixed two pull requests - backing out one
When all etcd servers refuse connections during watch the call will fail with an exception and will be immediately retried. This creates a huge amount of log spam potentially creating additional issues on top of losing the DCS. This patch takes note if etcd failures are repeating and starting from the second failure will sleep for a second before retrying. It additionally omits the stack trace after the first failure in a streak of failures.
Default value of wal_sender_timeout is 60 seconds while we are trying to remove replication slot after 30 seconds (ttl=30). That means postgres might think that slot is still active and does nothing. Patroni at the same time was thinking that it was removed successfully.
If the drop replication slot query didn't return any single row we must fetch list of existing physical replication slots from postgres on the next iteration of HA loop.
Fixes: issue #425
`Postgresql.connection` method could be called from different threads at the same time resulting in more than one connection open but only one used afterwards.
* Reassemble postgresql parameters when major version became known
Otherwise we were writing some "unknown" parameters into postgresql.conf
and postgres was refusing to start. Only 9.3 was affected.
In addition to that move rename of wal_level from hot_standby to replica
into get_server_parameters method. Now this rename is handled in a
single place.
* Bump etcd and consul versions
Change hostnames by ip addresses was causing certificate verification to
fail. Instead of doing it we will better monkey patch urllib3
functionality which does name resolution. It should work without
problems even for https connection.
