This commit makes it possible to configure the maximum lag (`maximum_lag_on_syncnode`) after which Patroni will "demote" the node from synchronous and replace it with another node.
The previous implementation always tried to stick to the same synchronous nodes (even if they are not optimal ones).
Patroni fails If hba conf happens to be located in non pgdata dir after custom bootstrap.
```
2020-08-27 23:25:06,966 INFO: establishing a new patroni connection to the postgres cluster
2020-08-27 23:25:06,997 INFO: running post_bootstrap
2020-08-27 23:25:07,009 ERROR: post_bootstrap
Traceback (most recent call last):
File "../lib/python3.8/site-packages/patroni/postgresql/bootstrap.py", line 357, in post_bootstrap
os.unlink(postgresql.config.pg_hba_conf)
FileNotFoundError: [Errno 2] No such file or directory: '../pgrept1/sysdata/pg_hba.conf'
2020-08-27 23:25:07,016 INFO: removing initialize key after failed attempt to bootstrap the cluster
```
The new parameter `synchronous_node_count` is used by Patroni to manage number of synchronous standby databases. It is set to 1 by default. It has no effect when synchronous_mode is set to off. When enabled, Patroni manages precise number of synchronous standby databases based on parameter synchronous_node_count and adjusts the state in DCS & synchronous_standby_names as members join and leave.
This functionality can be further extended to support Priority (FIRST n) based synchronous replication & Quorum (ANY n) based synchronous replication in future.
Patroni appeared to be creating dormant slot (when `slots` defined) for leader node when the name contains special chars such as '-' (for e.g. "abc-us-1").
- ``GET /replica?lag=<max-lag>``: replica check endpoint.
- ``GET /asynchronous?lag=<max-lag>`` or ``GET /async&lag=<max-lag>``: asynchronous standby check endpoint.
Checks replication latency and returns status code **200** only when the latency is below a specified value. The key leader_optime from DCS is used for the leader WAL position and compute latency on the replica for performance reasons. Please note that the value in leader_optime might be a couple of seconds old (based on loop_wait).
Co-authored-by: Alexander Kukushkin <[email protected]>
initial bootstrap by attaching Patroni to the running postgres was causing the following error.
```
File "/xx/lib/python3.8/site-packages/patroni/ha.py", line 529, in update_cluster_history
history = history[-self.cluster.config.max_timelines_history:]
AttributeError: 'NoneType' object has no attribute 'max_timelines_history'
```
## Feature: Postgres stop timeout
Switchover/Failover operation hangs on signal_stop (or checkpoint) call when postmaster doesn't respond or hangs for some reason(Issue described in [1371](https://github.com/zalando/patroni/issues/1371)). This is leading to service loss for an extended period of time until the hung postmaster starts responding or it is killed by some other actor.
### master_stop_timeout
The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
