when Patroni is trying to figure out the necessity of pg_rewind it could write the content history file from the primary into the log. The history file is growing with every failover/switchover and eventually starts taking too many lines in the log, most of them are not so much useful.
Instead of showing the raw data, we will show only 3 lines before the current replica timeline and 2 lines after.
Replicas are waiting for checkpoint indication via member key of the leader in DCS. The key is normally updated only one time per HA loop.
Without waking the main thread up replicas will have to wait up to `loop_wait` seconds longer than necessary.
Without supplying the --enable-v2=true flag to etcd on startup, patroni cannot find etcd to run.
after running `etcd --data-dir=data/etcd` in one terminal and `patroni postgres0.yaml` in another terminal, etcd starts fine, but the postgres instance cannot find etcd.
```
patroni postgres0.yaml
2020-05-09 15:58:48,560 ERROR: Failed to get list of machines from http://127.0.0.1:2379/v2: EtcdException('Bad response : 404 page not found\n')
2020-05-09 15:58:48,560 INFO: waiting on etcd
```
If etcd is passed the flag `--enable-v2=true` on startup, everything works fine.
initial bootstrap by attaching Patroni to the running postgres was causing the following error.
```
File "/xx/lib/python3.8/site-packages/patroni/ha.py", line 529, in update_cluster_history
history = history[-self.cluster.config.max_timelines_history:]
AttributeError: 'NoneType' object has no attribute 'max_timelines_history'
```
Skip missing values from pg_controldata
When calling controldata(), it may return an empty dictionary, which in
turn caused the following error to occur:
effective_configuration
cvalue = parse_int(data[cname])
KeyError: 'max_wal_senders setting'
Instead of causing a crash of this part, we now log the error and
continue.
This is the full output of the error:
```
2020-04-17 14:31:54,791 ERROR: Exception during execution of long running task restarting after failure
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/patroni/async_executor.py", line 97, in run
wakeup = func(*args) if args else func()
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 707, in follow
self.start(timeout=timeout, block_callbacks=change_role, role=role)
File "/usr/lib/python3/dist-packages/patroni/postgresql/__init__.py", line 409, in start
configuration = self.config.effective_configuration
File "/usr/lib/python3/dist-packages/patroni/postgresql/config.py", line 983, in effective_configuration
cvalue = parse_int(data[cname])
KeyError: 'max_wal_senders setting'
```
In dynamic environments it is common that during the rolling upgrade etcd nodes are changing their IP addresses. If the etcd node where Patroni is currently connected to is upgraded last, it could happen that the cached topology doesn't contain any live node anymore and therefore request can't be retried and totally fails, usually resulting in demoting of the primary.
In order to partially overcome the problem, Patroni is already doing a periodic (every 5 minutes) rediscovery of the etcd cluster topology, but in case of very fast node rotation there was still a possibility to hit the issue.
This PR is an attempt to address the problem. If the list of nodes exhausted, Patroni will try to perform initial discovery via an external mechanism, like resolving A or SRV dns records and if the new list is different from the original, Patroni will use it as the new etcd cluster topology.
In order to deal with tcp issues the connect_timeout is set to max(read_timeout/2, 1). It will make list of members exhaust faster, but leaves the time to perform topology rediscovery and another attempt.
The third issue addressed by this PR - it could happen that dns names of etcd nodes didn't change, but ip addresses are new, therefore we clean up the internal dns cache when doing topology rediscovery.
Besides that, this commit makes `_machines_cache` property pretty much static, it will be updated only when the topology has changed and helps to avoid concurrency issues.
It is safe to call pg_rewind on the replica only when pg_control on the primary contains information about the latest timeline. Postgres is usually doing immediate checkpoint right after promote and in most cases it works just fine. Unfortunately we regularly receive complaints that it takes to long (minutes) until the checkpoint is done and replicas can't perform rewind. At the same time doing the checkpoint manually immediately helped. So Patroni starts doing the same. When the promotion happened and postgres is not running in recovery, we explicitly issue the checkpoint.
We are intentionally not using the AsyncExecutor here, because we want the HA loop continues doing its normal flow.
## Feature: Postgres stop timeout
Switchover/Failover operation hangs on signal_stop (or checkpoint) call when postmaster doesn't respond or hangs for some reason(Issue described in [1371](https://github.com/zalando/patroni/issues/1371)). This is leading to service loss for an extended period of time until the hung postmaster starts responding or it is killed by some other actor.
### master_stop_timeout
The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
$ python3 patronictl.py -c postgresql0.yml list
Error: Provided config file postgresql0.yml not existing or no read rights. Check the -c/--config-file parameter
Right now it's not really clear from --help what it does, flushing in software
context usually means persisting...so actually it's the opposite, so make
the intention more explicit.
The KUBERNETES_ environment variables are not required for PostgreSQL, yet having them exposed to the postmaster will also expose them to backends and to regular database users (using pl/perl for example).
Consul client uses urllib3 with a verify=True by default. When
SSL verification is disabled with verify=False, we can see
CERTIFICATE_VERIFY_FAILED exceptions. With urllib3 1.19.1-1 on
Debian Stretch, the "cert_reqs" argument must be explicitaly set
to ssl.CERT_NONE to effectively disable SSL verification.
It is a regular issue that primary is recycling WALs when one of the replicas is down for a long time. So far there were only two solutions for such a problem and both of them are not perfect:
1. Increase `wal_keep_segments`, but it is hard to guess the good value.
2. Use continuous archiving and PITR, but it is not always possible.
This PR is introducing the way to solve the problem for static clusters, with a fixed number of nodes and names that never change. You just need to list the names of all nodes in the `slots` so the primary will not remove the slot when the node is down (not registered in DCS).
Of course, the primary will not create the permanent slot which is matching its own name.
Usage example: let's assume you have a cluster with nodes named *abc1*, *abc2*, and *abc3*.
You have to run `patronictl edit-config` and put the following snippet into the configuration:
```yaml
slots:
abc1:
type: physical
abc2:
type: physical
abc3:
type: physical
```
If the node *abc2* is the primary, it will always create slots for *abc1* and *abc3* even if they are not running, but will not create slot *abc2*.
Other nodes will behave the same.
Close#280
OpenShift enforces securityContext.fsGroups for block devices and sets group stickybits for volumeMounts.
This leads to patroni pods failing to start after the first restart:
> 2020-01-13 14:46:13.695 UTC [143] FATAL: data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2020-01-13 14:46:13.695 UTC [143] DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
A initContainer which fixes the OpenShift tampering solves the issue. I stole the solution from the stable postgres helm chart:
https://github.com/helm/charts/pull/14540/files
Tested on OpenShift v3.11
Note: This error does not occur when using shared filesystems (like NFS)
It was used only to add the local timezone to the datetime specified in the patronictl for scheduled switchover or restart.
The `dateutil.tz.tzlocal()` does the same job equally well.
