gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 14:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:zalando/patroni into feature/terminaltables

Browse Source
This commit is contained in:
Alexander Kukushkin
2020-04-15 12:29:12 +02:00
parent 7c409f59d7 337f9efc9e
commit 52761ac46c
26 changed files with 929 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockerfile
+1 -1
View File
@@ -30,7 +30,7 @@ RUN set -ex \
\
# Cleanup all locales but en_US.UTF-8
&& find /usr/share/i18n/charmaps/ -type f ! -name UTF-8.gz -delete \
&& find /usr/share/i18n/locales/ -type f ! -name en_US ! -name en_GB ! -name i18n ! -name iso14651_t1 ! -name iso14651_t1_common ! -name 'translit_*' -delete \
&& find /usr/share/i18n/locales/ -type f ! -name en_US ! -name en_GB ! -name i18n* ! -name iso14651_t1 ! -name iso14651_t1_common ! -name 'translit_*' -delete \
&& echo 'en_US.UTF-8 UTF-8' > /usr/share/i18n/SUPPORTED \
\
# Make sure we have a en_US.UTF-8 locale available
docs/ENVIRONMENT.rst
+3 -2
View File
@@ -35,8 +35,8 @@ Example: defining ``PATRONI_admin_PASSWORD=strongpasswd`` and ``PATRONI_admin_OP
Consul
------
- **PATRONI\_CONSUL\_HOST**: the host:port for the Consul endpoint.
- **PATRONI\_CONSUL\_URL**: url for the Consul, in format: http(s)://host:port
- **PATRONI\_CONSUL\_HOST**: the host:port for the Consul local agent.
- **PATRONI\_CONSUL\_URL**: url for the Consul local agent, in format: http(s)://host:port
- **PATRONI\_CONSUL\_PORT**: (optional) Consul port
- **PATRONI\_CONSUL\_SCHEME**: (optional) **http** or **https**, defaults to **http**
- **PATRONI\_CONSUL\_TOKEN**: (optional) ACL token
@@ -130,6 +130,7 @@ REST API
CTL
---
- **PATRONICTL\_CONFIG\_FILE**: location of the configuration file.
- **PATRONI\_CTL\_INSECURE**: Allow connections to REST API without verifying SSL certs.
- **PATRONI\_CTL\_CACERT**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
- **PATRONI\_CTL\_CERTFILE**: Specifies the file with the client certificate in the PEM format. If not provided patronictl will use the value provided for REST API "certfile" parameter.
docs/README.rst
+1 -1
View File
@@ -107,7 +107,7 @@ obtain those files from the git repository and replace `./patroni.py` below with
To get started, do the following from different terminals:
::
> etcd --data-dir=data/etcd
> etcd --data-dir=data/etcd --enable-v2=true
> ./patroni.py postgres0.yml
> ./patroni.py postgres1.yml
docs/SETTINGS.rst
+15 -14
View File
@@ -16,11 +16,12 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
- **retry\_timeout**: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10
- **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
- **master\_start\_timeout**: the amount of time a master is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for master failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
- **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
- **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that successfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See :ref:`replication modes documentation <replication_modes>` for details.
- **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the master. See :ref:`replication modes documentation <replication_modes>` for details.
- **postgresql**:
- **use\_pg\_rewind**: whether or not to use pg_rewind. Defaults to `false`.
- **use\_slots**: whether or not to use replication_slots. Defaults to `true` on PostgreSQL 9.4+.
- **use\_slots**: whether or not to use replication slots. Defaults to `true` on PostgreSQL 9.4+.
- **recovery\_conf**: additional configuration settings written to recovery.conf when configuring follower. There is no recovery.conf anymore in PostgreSQL 12, but you may continue using this section, because Patroni handles it transparently.
- **parameters**: list of configuration settings for Postgres.
- **standby\_cluster**: if this section is defined, we want to bootstrap a standby cluster.
@@ -32,7 +33,7 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
- **archive\_cleanup\_command**: cleanup command for standby leader
- **recovery\_min\_apply\_delay**: how long to wait before actually apply WAL records on a standby leader
- **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Patroni will try to create slots before opening connections to the cluster.
- **my_slot_name**: the name of replication slot. It is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots.
- **my_slot_name**: the name of replication slot. If the permanent slot name matches with the name of the current primary it will not be created. Everything else is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots.
- **type**: slot type. Could be ``physical`` or ``logical``. If the slot is logical, you have to additionally define ``database`` and ``plugin``.
- **database**: the database name where logical slots should be created.
- **plugin**: the plugin name for the logical slot.
@@ -87,20 +88,20 @@ Consul
------
Most of the parameters are optional, but you have to specify one of the **host** or **url**
- **host**: the host:port for the Consul endpoint, in format: http(s)://host:port
- **url**: url for the Consul endpoint
- **port**: (optional) Consul port
- **scheme**: (optional) **http** or **https**, defaults to **http**
- **token**: (optional) ACL token
- **verify**: (optional) whether to verify the SSL certificate for HTTPS requests
- **host**: the host:port for the Consul local agent.
- **url**: url for the Consul local agent, in format: http(s)://host:port.
- **port**: (optional) Consul port.
- **scheme**: (optional) **http** or **https**, defaults to **http**.
- **token**: (optional) ACL token.
- **verify**: (optional) whether to verify the SSL certificate for HTTPS requests.
- **cacert**: (optional) The ca certificate. If present it will enable validation.
- **cert**: (optional) file with the client certificate
- **cert**: (optional) file with the client certificate.
- **key**: (optional) file with the client key. Can be empty if the key is part of **cert**.
- **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
- **consistency**: (optional) Select consul consistency mode. Possible values are ``default``, ``consistent``, or ``stale`` (more details in `consul API reference <https://www.consul.io/api/features/consistency.html/>`__)
- **checks**: (optional) list of Consul health checks used for the session. By default an empty list is used.
- **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
- **service\_check\_interval**: (optional) how often to perform health check against registered url
- **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**.
- **service\_check\_interval**: (optional) how often to perform health check against registered url.
Etcd
----
@@ -109,8 +110,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
- **host**: the host:port for the etcd endpoint.
- **hosts**: list of etcd endpoint in format host1:port1,host2:port2,etc... Could be a comma separated string or an actual yaml list.
- **use\_proxies**: If this parameter is set to true, Patroni will consider **hosts** as a list of proxies and will not perform a topology discovery of etcd cluster.
- **url**: url for the etcd
- **proxy**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **url**
- **url**: url for the etcd.
- **proxy**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **url**.
- **srv**: Domain to search the SRV record(s) for cluster autodiscovery.
- **protocol**: (optional) http or https, if not specified http is used. If the **url** or **proxy** is specified - will take protocol from them.
- **username**: (optional) username for etcd authentication.
@@ -126,7 +127,7 @@ ZooKeeper
Exhibitor
---------
- **hosts**: initial list of Exhibitor (ZooKeeper) nodes in format: 'host1,host2,etc...'. This list updates automatically whenever the Exhibitor (ZooKeeper) cluster topology changes.
- **poll\_interval**: how often the list of ZooKeeper and Exhibitor nodes should be updated from Exhibitor
- **poll\_interval**: how often the list of ZooKeeper and Exhibitor nodes should be updated from Exhibitor.
- **port**: Exhibitor port.
.. _kubernetes_settings:
docs/replication_modes.rst
+2
View File
@@ -57,6 +57,8 @@ You can ensure that a standby never becomes the synchronous standby by setting `
Synchronous mode can be switched on and off via Patroni REST interface. See :ref:`dynamic configuration <dynamic_configuration>` for instructions.
Note: Because of the way synchronous replication is implemented in PostgreSQL it is still possible to lose transactions even when using ``synchronous_mode_strict``. If the PostgreSQL backend is cancelled while waiting to acknowledge replication (as a result of packet cancellation due to client timeout or backend failure) transaction changes become visible for other backends. Such changes are not yet replicated and may be lost in case of standby promotion.
Synchronous mode implementation
-------------------------------
patroni/__init__.py
+12 -1
View File
@@ -9,6 +9,7 @@ from patroni.version import __version__
logger = logging.getLogger(__name__)
PATRONI_ENV_PREFIX = 'PATRONI_'
KUBERNETES_ENV_PREFIX = 'KUBERNETES_'
class Patroni(object):
@@ -168,16 +169,26 @@ class Patroni(object):
def patroni_main():
import argparse
from multiprocessing import freeze_support
from patroni.config import Config, ConfigParseError
from patroni.validator import schema
freeze_support()
parser = argparse.ArgumentParser()
parser.add_argument('--version', action='version', version='%(prog)s {0}'.format(__version__))
parser.add_argument('--validate-config', action='store_true', help='Run config validator and exit')
parser.add_argument('configfile', nargs='?', default='',
help='Patroni may also read the configuration from the {0} environment variable'
.format(Config.PATRONI_CONFIG_VARIABLE))
args = parser.parse_args()
try:
conf = Config(args.configfile)
if args.validate_config:
conf = Config(args.configfile, validator=schema)
sys.exit()
else:
conf = Config(args.configfile)
except ConfigParseError as e:
if e.value:
print(e.value)
patroni/config.py
+1
View File
@@ -59,6 +59,7 @@ class Config(object):
'maximum_lag_on_failover': 1048576,
'check_timeline': False,
'master_start_timeout': 300,
'master_stop_timeout': 0,
'synchronous_mode': False,
'synchronous_mode_strict': False,
'standby_cluster': {
patroni/ctl.py
+20 -14
View File
@@ -47,7 +47,7 @@ class PatroniCtlException(ClickException):
pass
class PatronictlTexttable(Texttable):
class PatronictlPrettyTable(Texttable):
def __init__(self, header=None):
Texttable.__init__(self, 0)
@@ -108,7 +108,11 @@ def load_config(path, dcs):
from patroni.config import Config
if not (os.path.exists(path) and os.access(path, os.R_OK)):
logging.debug('Ignoring configuration file "%s". It does not exists or is not readable.', path)
if path != CONFIG_FILE_PATH: # bail if non-default config location specified but file not found / readable
raise PatroniCtlException('Provided config file {0} not existing or no read rights.'
' Check the -c/--config-file parameter'.format(path))
else:
logging.debug('Ignoring configuration file "%s". It does not exists or is not readable.', path)
else:
logging.debug('Loading configuration from file %s', path)
config = Config(path, validator=None).copy()
@@ -129,7 +133,7 @@ def store_config(config, path):
yaml.dump(config, fd)
option_format = click.option('--format', '-f', 'fmt', help='Output format (pretty, json, yaml)', default='pretty')
option_format = click.option('--format', '-f', 'fmt', help='Output format (pretty, tsv, json, yaml)', default='pretty')
option_watchrefresh = click.option('-w', '--watch', type=float, help='Auto update the screen every X seconds')
option_watch = click.option('-W', is_flag=True, help='Auto update the screen every 2 seconds')
option_force = click.option('--force', is_flag=True, help='Do not ask for confirmation at any point')
@@ -173,11 +177,11 @@ def request_patroni(member, method='GET', endpoint=None, data=None):
def print_output(columns, rows, alignment=None, fmt='pretty', header=None, delimiter='\t'):
if fmt in ['json', 'yaml', 'yml']:
if fmt in {'json', 'yaml', 'yml'}:
elements = [{k: v for k, v in zip(columns, r) if not header or str(v)} for r in rows]
func = json.dumps if fmt == 'json' else format_config_for_editing
click.echo(func(elements))
elif fmt in ('pretty', 'tsv'):
elif fmt in {'pretty', 'tsv'}:
list_cluster = bool(header and columns and columns[0] == 'Cluster')
if list_cluster and 'Tags' in columns: # we want to format member tags as YAML
i = columns.index('Tags')
@@ -189,10 +193,10 @@ def print_output(columns, rows, alignment=None, fmt='pretty', header=None, delim
rows = [row[1:] for row in rows]
if fmt == 'tsv':
for r in [columns] + rows:
for r in ([columns] if columns else []) + rows:
click.echo(delimiter.join(map(str, r)))
else:
table = PatronictlTexttable(header)
table = PatronictlPrettyTable(header)
if not any(any(isinstance(c, six.string_types) and '\n' in c for c in r) for r in rows):
table.set_deco(Texttable.VLINES | Texttable.BORDER | Texttable.HEADER)
if rows:
@@ -351,7 +355,7 @@ def dsn(obj, cluster_name, role, member):
@ctl.command('query', help='Query a Patroni PostgreSQL member')
@arg_cluster_name
@option_format
@click.option('--format', 'fmt', help='Output format (pretty, tsv, json, yaml)', default='tsv')
@click.option('--file', '-f', 'p_file', help='Execute the SQL commands from this file', type=click.File('rb'))
@click.option('--password', help='force password prompt', is_flag=True)
@click.option('-U', '--username', help='database user name', type=str)
@@ -769,19 +773,20 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
columns.append(c)
# Show Host as 'host:port' if somebody is running on non-standard port or two nodes are running on the same host
append_port = any(m['port'] != 5432 for m in cluster['members']) or\
len(set(m['host'] for m in cluster['members'])) < len(cluster['members'])
members = [m for m in cluster['members'] if 'host' in m]
append_port = any('port' in m and m['port'] != 5432 for m in members) or\
len(set(m['host'] for m in cluster['members'])) < len(members)
for m in cluster['members']:
logging.debug(m)
lag = m.get('lag', '')
m.update(cluster=name, member=m['name'], tl=m.get('timeline', ''),
m.update(cluster=name, member=m['name'], host=m.get('host'), tl=m.get('timeline', ''),
role='' if m['role'] == 'replica' else m['role'].replace('_', ' ').title(),
lag_in_mb=round(lag/1024/1024) if isinstance(lag, six.integer_types) else lag,
pending_restart='*' if m.get('pending_restart') else '')
if append_port:
if append_port and m['host'] and m.get('port'):
m['host'] = ':'.join([m['host'], str(m['port'])])
if 'scheduled_restart' in m:
@@ -792,7 +797,8 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
rows.append([m.get(n.lower().replace(' ', '_'), '') for n in columns])
print_output(columns, rows, {'Lag in MB': 'r', 'TL': 'r'}, fmt, ' Cluster: {0} ({1}) '.format(name, initialize))
print_output(columns, rows, {'Lag in MB': 'r', 'TL': 'r', 'Tags': 'l'},
fmt, ' Cluster: {0} ({1}) '.format(name, initialize))
if fmt != 'pretty': # Omit service info when using machine-readable formats
return
@@ -905,7 +911,7 @@ def scaffold(obj, cluster_name, sysid):
click.echo("Cluster {0} has been created successfully".format(cluster_name))
@ctl.command('flush', help='Flush scheduled events')
@ctl.command('flush', help='Discard scheduled events (restarts only currently)')
@click.argument('cluster_name')
@click.argument('member_names', nargs=-1)
@click.argument('target', type=click.Choice(['restart']))
patroni/dcs/__init__.py
+28 -23
View File
@@ -66,8 +66,11 @@ def dcs_modules():
module_prefix = __package__ + '.'
if getattr(sys, 'frozen', False):
importer = pkgutil.get_importer(dcs_dirname)
return [module for module in list(importer.toc) if module.startswith(module_prefix) and module.count('.') == 2]
toc = set()
for importer in pkgutil.iter_importers(dcs_dirname):
if hasattr(importer, 'toc'):
toc |= importer.toc
return [module for module in toc if module.startswith(module_prefix) and module.count('.') == 2]
else:
return [module_prefix + name for _, name, is_pkg in pkgutil.iter_modules([dcs_dirname]) if not is_pkg]
@@ -137,10 +140,10 @@ class Member(namedtuple('Member', 'index,name,session,data')):
@property
def conn_url(self):
conn_url = self.data.get('conn_url')
conn_kwargs = self.data.get('conn_kwargs')
if conn_url:
return conn_url
conn_kwargs = self.data.get('conn_kwargs')
if conn_kwargs:
conn_url = uri('postgresql', (conn_kwargs.get('host'), conn_kwargs.get('port', 5432)))
self.data['conn_url'] = conn_url
@@ -148,16 +151,19 @@ class Member(namedtuple('Member', 'index,name,session,data')):
def conn_kwargs(self, auth=None):
defaults = {
"host": "",
"port": "",
"database": ""
"host": None,
"port": None,
"database": None
}
ret = self.data.get('conn_kwargs')
if ret:
defaults.update(ret)
ret = defaults
else:
r = urlparse(self.conn_url)
conn_url = self.conn_url
if not conn_url:
return {} # due to the invalid conn_url we don't care about authentication parameters
r = urlparse(conn_url)
ret = {
'host': r.hostname,
'port': r.port or 5432,
@@ -449,21 +455,21 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
def is_synchronous_mode(self):
return self.check_mode('synchronous_mode')
def get_replication_slots(self, name, role):
def get_replication_slots(self, my_name, role):
# if the replicatefrom tag is set on the member - we should not create the replication slot for it on
# the current master, because that member would replicate from elsewhere. We still create the slot if
# the replicatefrom destination member is currently not a member of the cluster (fallback to the
# master), or if replicatefrom destination member happens to be the current master
use_slots = self.config and self.config.data.get('postgresql', {}).get('use_slots', True)
if role in ('master', 'standby_leader'):
slot_members = [m.name for m in self.members if use_slots and m.name != name and
(m.replicatefrom is None or m.replicatefrom == name or
slot_members = [m.name for m in self.members if use_slots and m.name != my_name and
(m.replicatefrom is None or m.replicatefrom == my_name or
not self.has_member(m.replicatefrom))]
permanent_slots = (self.config and self.config.permanent_slots or {}).copy()
else:
# only manage slots for replicas that replicate from this one, except for the leader among them
slot_members = [m.name for m in self.members if use_slots and
m.replicatefrom == name and m.name != self.leader.name]
m.replicatefrom == my_name and m.name != self.leader.name]
permanent_slots = {}
slots = {slot_name_from_member_name(name): {'type': 'physical'} for name in slot_members}
@@ -484,22 +490,21 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
logger.error("Slot name may only contain lower case letters, numbers, and the underscore chars")
continue
if name in slots:
logger.error("Permanent replication slot {'%s': %s} is conflicting with" +
" physical replication slot for cluster member", name, value)
continue
value = deepcopy(value)
if not value:
value = {'type': 'physical'}
value = deepcopy(value) if value else {'type': 'physical'}
if isinstance(value, dict):
if 'type' not in value:
value['type'] = 'logical' if value.get('database') and value.get('plugin') else 'physical'
if value['type'] == 'physical' or value['type'] == 'logical' \
and value.get('database') and value.get('plugin'):
slots[name] = value
if value['type'] == 'physical':
if name != my_name: # Don't try to create permanent physical replication slot for yourself
slots[name] = value
continue
elif value['type'] == 'logical' and value.get('database') and value.get('plugin'):
if name in slots:
logger.error("Permanent logical replication slot {'%s': %s} is conflicting with" +
" physical replication slot for cluster member", name, value)
else:
slots[name] = value
continue
logger.error("Bad value for slot '%s' in permanent_slots: %s", name, permanent_slots[name])
patroni/dcs/consul.py
+1 -2
View File
@@ -53,8 +53,7 @@ class HTTPClient(object):
kwargs['cert_file'] = cert
if ca_cert:
kwargs['ca_certs'] = ca_cert
if verify or ca_cert:
kwargs['cert_reqs'] = ssl.CERT_REQUIRED
kwargs['cert_reqs'] = ssl.CERT_REQUIRED if verify or ca_cert else ssl.CERT_NONE
self.http = urllib3.PoolManager(num_pools=10, **kwargs)
self._ttl = None
patroni/dcs/kubernetes.py
+13 -2
View File
@@ -31,6 +31,13 @@ class KubernetesRetriableException(k8s_client.rest.ApiException):
self.body = orig.body
self.headers = orig.headers
@property
def sleeptime(self):
try:
return int(self.headers['retry-after'])
except Exception:
return None
class CoreV1ApiProxy(object):
@@ -68,7 +75,7 @@ class CoreV1ApiProxy(object):
try:
return getattr(self._api, func)(*args, **kwargs)
except k8s_client.rest.ApiException as e:
if e.status in (502, 503, 504): # XXX
if e.status in (502, 503, 504) or e.headers and 'retry-after' in e.headers: # XXX
raise KubernetesRetriableException(e)
raise
return wrapper
@@ -106,7 +113,11 @@ class ObjectCache(Thread):
self.start()
def _list(self):
return self._func(_request_timeout=(self._retry.deadline, Timeout.DEFAULT_TIMEOUT))
try:
return self._func(_request_timeout=(self._retry.deadline, Timeout.DEFAULT_TIMEOUT))
except Exception:
time.sleep(1)
raise
def _watch(self, resource_version):
return self._func(_request_timeout=(self._retry.deadline, Timeout.DEFAULT_TIMEOUT),
patroni/dcs/zookeeper.py
+2 -2
View File
@@ -76,7 +76,7 @@ class ZooKeeper(AbstractDCS):
self._client.start()
def _kazoo_connect(self, host, port):
def _kazoo_connect(self, *args):
"""Kazoo is using Ping's to determine health of connection to zookeeper. If there is no
response on Ping after Ping interval (1/2 from read_timeout) it will consider current
connection dead and try to connect to another node. Without this "magic" it was taking
@@ -88,7 +88,7 @@ class ZooKeeper(AbstractDCS):
than loop_wait, because we can spend up to 2 seconds when calling `touch_member()` and
`write_leader_optime()` methods, which also may hang..."""
ret = self._orig_kazoo_connect(host, port)
ret = self._orig_kazoo_connect(*args)
return max(self.loop_wait - 2, 2)*1000, ret[1]
def session_listener(self, state):
patroni/ha.py
+22 -11
View File
@@ -14,7 +14,7 @@ from patroni.exceptions import DCSError, PostgresConnectionException, PatroniExc
from patroni.postgresql import ACTION_ON_START, ACTION_ON_ROLE_CHANGE
from patroni.postgresql.misc import postgres_version_to_int
from patroni.postgresql.rewind import Rewind
from patroni.utils import polling_loop, tzutc, is_standby_cluster as _is_standby_cluster
from patroni.utils import polling_loop, tzutc, is_standby_cluster as _is_standby_cluster, parse_int
from patroni.dcs import RemoteMember
from threading import RLock
@@ -94,6 +94,11 @@ class Ha(object):
else:
return self.patroni.config.check_mode(mode)
def master_stop_timeout(self):
""" Master stop timeout """
ret = parse_int(self.patroni.config['master_stop_timeout'])
return ret if ret and ret > 0 and self.is_synchronous_mode() else None
def is_paused(self):
return self.check_mode('pause')
@@ -204,7 +209,7 @@ class Ha(object):
if self.state_handler.role == 'standby_leader':
timeline = pg_control_timeline or self.state_handler.pg_control_timeline()
else:
timeline = self.state_handler.replica_cached_timeline(timeline)
timeline = self.state_handler.replica_cached_timeline(self._leader_timeline)
if timeline:
data['timeline'] = timeline
except Exception:
@@ -772,7 +777,8 @@ class Ha(object):
self._rewind.trigger_check_diverged_lsn()
self.state_handler.stop(mode_control['stop'], checkpoint=mode_control['checkpoint'],
on_safepoint=self.watchdog.disable if self.watchdog.is_running else None)
on_safepoint=self.watchdog.disable if self.watchdog.is_running else None,
stop_timeout=self.master_stop_timeout())
self.state_handler.set_role('demoted')
self.set_is_leader(False)
@@ -1083,7 +1089,7 @@ class Ha(object):
return (False, 'restart failed')
def _do_reinitialize(self, cluster):
self.state_handler.stop('immediate')
self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
# Commented redundant data directory cleanup here
# self.state_handler.remove_data_directory()
@@ -1143,17 +1149,20 @@ class Ha(object):
if not self.state_handler.is_running():
self.watchdog.disable()
if self.has_lock():
self.state_handler.set_role('demoted')
if self.state_handler.role in ('master', 'standby_leader'):
self.state_handler.set_role('demoted')
self._delete_leader()
return 'removed leader key after trying and failing to start postgres'
return 'failed to start postgres'
self._crash_recovery_executed = False
if self._rewind.executed and not self._rewind.failed:
self._rewind.reset_state()
return None
def cancel_initialization(self):
logger.info('removing initialize key after failed attempt to bootstrap the cluster')
self.dcs.cancel_initialization()
self.state_handler.stop('immediate')
self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
self.state_handler.move_data_directory()
raise PatroniException('Failed to bootstrap cluster')
@@ -1172,10 +1181,11 @@ class Ha(object):
return ret or 'running post_bootstrap'
self.state_handler.bootstrapping = False
self.dcs.set_config_value(json.dumps(self.patroni.config.dynamic_configuration, separators=(',', ':')))
if not self.watchdog.activate():
logger.error('Cancelling bootstrap because watchdog activation failed')
self.cancel_initialization()
self.dcs.initialize(create_new=(self.cluster.initialize is None), sysid=self.state_handler.sysid)
self.dcs.set_config_value(json.dumps(self.patroni.config.dynamic_configuration, separators=(',', ':')))
self.state_handler.slots_handler.sync_replication_slots(self.cluster)
self.dcs.take_leader()
self.set_is_leader(True)
@@ -1275,7 +1285,7 @@ class Ha(object):
# is data directory empty?
if self.state_handler.data_directory_empty():
self.state_handler.set_role('uninitialized')
self.state_handler.stop('immediate')
self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
# In case datadir went away while we were master.
self.watchdog.disable()
@@ -1290,8 +1300,8 @@ class Ha(object):
data_sysid = self.state_handler.sysid
if not self.sysid_valid(data_sysid):
# data directory is not empty, but no valid sysid, cluster must be broken, suggest reinit
return ("data dir for the cluster is not empty, but system ID is invalid; consider doing"
"reinitialize")
return ("data dir for the cluster is not empty, "
"but system ID is invalid; consider doing reinitialize")
if self.sysid_valid(self.cluster.initialize):
if self.cluster.initialize != data_sysid:
@@ -1369,7 +1379,8 @@ class Ha(object):
# This might not be the desired behavior of users, as a graceful shutdown of the host can mean lost data.
# We probably need to something smarter here.
disable_wd = self.watchdog.disable if self.watchdog.is_running else None
self.while_not_sync_standby(lambda: self.state_handler.stop(checkpoint=False, on_safepoint=disable_wd))
self.while_not_sync_standby(lambda: self.state_handler.stop(checkpoint=False, on_safepoint=disable_wd,
stop_timeout=self.master_stop_timeout()))
if not self.state_handler.is_running():
if self.has_lock():
self.dcs.delete_leader()
patroni/postgresql/__init__.py
+37 -14
View File
@@ -17,8 +17,9 @@ from patroni.postgresql.misc import parse_history, postgres_major_version_to_int
from patroni.postgresql.postmaster import PostmasterProcess
from patroni.postgresql.slots import SlotsHandler
from patroni.exceptions import PostgresConnectionException
from patroni.utils import Retry, RetryFailedError, polling_loop
from patroni.utils import Retry, RetryFailedError, polling_loop, data_directory_is_empty
from threading import current_thread, Lock
from psutil import TimeoutExpired
logger = logging.getLogger(__name__)
@@ -266,9 +267,7 @@ class Postgresql(object):
def data_directory_empty(self):
if self.pg_control_exists():
return False
if not os.path.exists(self._data_dir):
return True
return all(os.name != 'nt' and (n.startswith('.') or n == 'lost+found') for n in os.listdir(self._data_dir))
return data_directory_is_empty(self._data_dir)
def replica_method_options(self, method):
return deepcopy(self.config.get(method, {}))
@@ -415,7 +414,11 @@ class Postgresql(object):
self.set_state('starting')
self._pending_restart = False
configuration = self.config.effective_configuration
try:
configuration = self.config.effective_configuration
except Exception:
return None
self.config.check_directories()
self.config.write_postgresql_conf(configuration)
self.config.resolve_connection_addresses()
@@ -460,11 +463,13 @@ class Postgresql(object):
else:
return None
def checkpoint(self, connect_kwargs=None):
def checkpoint(self, connect_kwargs=None, timeout=None):
check_not_is_in_recovery = connect_kwargs is not None
connect_kwargs = connect_kwargs or self.config.local_connect_kwargs
for p in ['connect_timeout', 'options']:
connect_kwargs.pop(p, None)
if timeout:
connect_kwargs['connect_timeout'] = timeout
try:
with get_connection_cursor(**connect_kwargs) as cur:
cur.execute("SET statement_timeout = 0")
@@ -477,7 +482,7 @@ class Postgresql(object):
logger.exception('Exception during CHECKPOINT')
return 'not accessible or not healty'
def stop(self, mode='fast', block_callbacks=False, checkpoint=None, on_safepoint=None):
def stop(self, mode='fast', block_callbacks=False, checkpoint=None, on_safepoint=None, stop_timeout=None):
"""Stop PostgreSQL
Supports a callback when a safepoint is reached. A safepoint is when no user backend can return a successful
@@ -489,7 +494,7 @@ class Postgresql(object):
if checkpoint is None:
checkpoint = False if mode == 'immediate' else True
success, pg_signaled = self._do_stop(mode, block_callbacks, checkpoint, on_safepoint)
success, pg_signaled = self._do_stop(mode, block_callbacks, checkpoint, on_safepoint, stop_timeout)
if success:
# block_callbacks is used during restart to avoid
# running start/stop callbacks in addition to restart ones
@@ -502,7 +507,7 @@ class Postgresql(object):
self.set_state('stop failed')
return success
def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint):
def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint, stop_timeout):
postmaster = self.is_running()
if not postmaster:
if on_safepoint:
@@ -510,7 +515,7 @@ class Postgresql(object):
return True, False
if checkpoint and not self.is_starting():
self.checkpoint()
self.checkpoint(timeout=stop_timeout)
if not block_callbacks:
self.set_state('stopping')
@@ -529,10 +534,28 @@ class Postgresql(object):
postmaster.wait_for_user_backends_to_close()
on_safepoint()
postmaster.wait()
try:
postmaster.wait(timeout=stop_timeout)
except TimeoutExpired:
logger.warning("Timeout during postmaster stop, aborting Postgres.")
if not self.terminate_postmaster(postmaster, mode, stop_timeout):
postmaster.wait()
return True, True
def terminate_postmaster(self, postmaster, mode, stop_timeout):
if mode in ['fast', 'smart']:
try:
success = postmaster.signal_stop('immediate', self.pgcommand('pg_ctl'))
if success:
return True
postmaster.wait(timeout=stop_timeout)
return True
except TimeoutExpired:
pass
logger.warning("Sending SIGKILL to Postmaster and its children")
return postmaster.signal_kill()
def terminate_starting_postmaster(self, postmaster):
"""Terminates a postmaster that has not yet opened ports or possibly even written a pid file. Blocks
until the process goes away."""
@@ -649,10 +672,10 @@ class Postgresql(object):
return result
@contextmanager
def get_replication_connection_cursor(self, host='localhost', port=5432, database=None, **kwargs):
def get_replication_connection_cursor(self, host='localhost', port=5432, **kwargs):
conn_kwargs = self.config.replication.copy()
conn_kwargs.update(host=host, port=int(port), database=database or self._database, connect_timeout=3,
user=conn_kwargs.pop('username'), replication=1, options='-c statement_timeout=2000')
conn_kwargs.update(host=host, port=int(port) if port else None, user=conn_kwargs.pop('username'),
connect_timeout=3, replication=1, options='-c statement_timeout=2000')
with get_connection_cursor(**conn_kwargs) as cur:
yield cur
patroni/postgresql/config.py
+2 -1
View File
@@ -614,6 +614,7 @@ class ConfigHandler(object):
values[match.group(1)] = [value, True]
self._recovery_conf_mtime = recovery_conf_mtime
values.setdefault('recovery_min_apply_delay', ['0', True])
values['recovery_min_apply_delay'][0] = parse_int(values['recovery_min_apply_delay'][0], 'ms')
values.update({param: ['', True] for param in self._recovery_parameters_to_compare if param not in values})
return values, True
@@ -651,7 +652,7 @@ class ConfigHandler(object):
else:
return False
return all(primary_conninfo.get(p) == str(v) for p, v in wanted_primary_conninfo.items())
return all(primary_conninfo.get(p) == str(v) for p, v in wanted_primary_conninfo.items() if v is not None)
def check_recovery_conf(self, member):
"""Returns a tuple. The first boolean element indicates that recovery params don't match
patroni/postgresql/postmaster.py
+39 -2
View File
@@ -7,7 +7,7 @@ import signal
import subprocess
import sys
from patroni import PATRONI_ENV_PREFIX
from patroni import PATRONI_ENV_PREFIX, KUBERNETES_ENV_PREFIX
# avoid spawning the resource tracker process
if sys.version_info >= (3, 8): # pragma: no cover
@@ -105,6 +105,42 @@ class PostmasterProcess(psutil.Process):
except psutil.NoSuchProcess:
return None
def signal_kill(self):
"""to suspend and kill postmaster and all children
:returns True if postmaster and children are killed, False if error
"""
try:
self.suspend()
except psutil.NoSuchProcess:
return True
except psutil.Error as e:
logger.warning('Failed to suspend postmaster: %s', e)
try:
children = self.children(recursive=True)
except psutil.NoSuchProcess:
return True
except psutil.Error as e:
logger.warning('Failed to get a list of postmaster children: %s', e)
children = []
try:
self.kill()
except psutil.NoSuchProcess:
return True
except psutil.Error as e:
logger.warning('Could not kill postmaster: %s', e)
return False
for child in children:
try:
child.kill()
except psutil.Error:
pass
psutil.wait_procs(children + [self])
return True
def signal_stop(self, mode, pg_ctl='pg_ctl'):
"""Signal postmaster process to stop
@@ -176,7 +212,8 @@ class PostmasterProcess(psutil.Process):
# In order to make everything portable we can't use fork&exec approach here, so we will call
# ourselves and pass list of arguments which must be used to start postgres.
# On Windows, in order to run a side-by-side assembly the specified env must include a valid SYSTEMROOT.
env = {p: os.environ[p] for p in os.environ if not p.startswith(PATRONI_ENV_PREFIX)}
env = {p: os.environ[p] for p in os.environ if not p.startswith(
PATRONI_ENV_PREFIX) and not p.startswith(KUBERNETES_ENV_PREFIX)}
try:
proc = PostmasterProcess._from_pidfile(data_dir)
if proc and not proc._is_postmaster_process():
patroni/utils.py
+12 -3
View File
@@ -334,7 +334,7 @@ class Retry(object):
logger.warning('Retry got exception: %s', e)
raise RetryFailedError("Too many retry attempts")
self._attempts += 1
sleeptime = self.sleeptime
sleeptime = hasattr(e, 'sleeptime') and e.sleeptime or self.sleeptime
if self._cur_stoptime is not None and time.time() + sleeptime >= self._cur_stoptime:
logger.warning('Retry got exception: %s', e)
@@ -390,9 +390,12 @@ def cluster_as_json(cluster):
else:
role = 'replica'
member = {'name': m.name, 'role': role, 'state': m.data.get('state', ''), 'api_url': m.api_url}
conn_kwargs = m.conn_kwargs()
member = {'name': m.name, 'host': conn_kwargs['host'], 'port': int(conn_kwargs['port']),
'role': role, 'state': m.data.get('state', ''), 'api_url': m.api_url}
if conn_kwargs.get('host'):
member['host'] = conn_kwargs['host']
if conn_kwargs.get('port'):
member['port'] = int(conn_kwargs['port'])
optional_attributes = ('timeline', 'pending_restart', 'scheduled_restart', 'tags')
member.update({n: m.data[n] for n in optional_attributes if n in m.data})
@@ -442,3 +445,9 @@ def validate_directory(d, msg="{} {}"):
raise PatroniException(msg.format(d, "the directory is not writable"))
else:
raise PatroniException(msg.format(d, "is not a directory"))
def data_directory_is_empty(data_dir):
if not os.path.exists(data_dir):
return True
return all(os.name != 'nt' and (n.startswith('.') or n == 'lost+found') for n in os.listdir(data_dir))
patroni/validator.py
+379
View File
@@ -0,0 +1,379 @@
#!/usr/bin/env python3
import os
import socket
import re
import subprocess
from patroni.utils import split_host_port, data_directory_is_empty
from patroni.ctl import find_executable
from patroni.dcs import dcs_modules
from patroni.exceptions import ConfigParseError
from six import string_types
def data_directory_empty(data_dir):
if os.path.isfile(os.path.join(data_dir, "global", "pg_control")):
return False
return data_directory_is_empty(data_dir)
def validate_connect_address(address):
try:
host, _ = split_host_port(address, 1)
except (AttributeError, TypeError, ValueError):
raise ConfigParseError("contains a wrong value")
if host in ["127.0.0.1", "0.0.0.0", "*", "::1", "localhost"]:
raise ConfigParseError('must not contain "127.0.0.1", "0.0.0.0", "*", "::1", "localhost"')
return True
def validate_host_port(host_port, listen=False, multiple_hosts=False):
try:
hosts, port = split_host_port(host_port, None)
except (ValueError, TypeError):
raise ConfigParseError("contains a wrong value")
else:
if multiple_hosts:
hosts = hosts.split(",")
else:
hosts = [hosts]
for host in hosts:
proto = socket.getaddrinfo(host, "", 0, socket.SOCK_STREAM, 0, socket.AI_PASSIVE)
s = socket.socket(proto[0][0], socket.SOCK_STREAM)
try:
if s.connect_ex((host, port)) == 0:
if listen:
raise ConfigParseError("Port {} is already in use.".format(port))
elif not listen:
raise ConfigParseError("{} is not reachable".format(host_port))
except socket.gaierror as e:
raise ConfigParseError(e)
finally:
s.close()
return True
def comma_separated_host_port(string):
assert all([validate_host_port(s.strip()) for s in string.split(",")]), "didn't pass the validation"
return True
def validate_host_port_listen(host_port):
return validate_host_port(host_port, listen=True)
def validate_host_port_listen_multiple_hosts(host_port):
return validate_host_port(host_port, listen=True, multiple_hosts=True)
def is_ipv4_address(ip):
try:
socket.inet_aton(ip)
except Exception:
raise ConfigParseError("Is not a valid ipv4 address")
return True
def is_ipv6_address(ip):
try:
socket.inet_pton(socket.AF_INET6, ip)
except Exception:
raise ConfigParseError("Is not a valid ipv6 address")
return True
def get_major_version(bin_dir=None):
if not bin_dir:
binary = 'postgres'
else:
binary = os.path.join(bin_dir, 'postgres')
version = subprocess.check_output([binary, '--version']).decode()
version = re.match(r'^[^\s]+ [^\s]+ (\d+)(\.(\d+))?', version)
return '.'.join([version.group(1), version.group(3)]) if int(version.group(1)) < 10 else version.group(1)
def validate_data_dir(data_dir):
if not data_dir:
raise ConfigParseError("is an empty string")
elif os.path.exists(data_dir) and not os.path.isdir(data_dir):
raise ConfigParseError("is not a directory")
elif not data_directory_empty(data_dir):
if not os.path.exists(os.path.join(data_dir, "PG_VERSION")):
raise ConfigParseError("doesn't look like a valid data directory")
else:
with open(os.path.join(data_dir, "PG_VERSION"), "r") as version:
pgversion = version.read().strip()
waldir = ("pg_wal" if float(pgversion) >= 10 else "pg_xlog")
if not os.path.isdir(os.path.join(data_dir, waldir)):
raise ConfigParseError("data dir for the cluster is not empty, but doesn't contain"
" \"{}\" directory".format(waldir))
bin_dir = schema.data.get("postgresql", {}).get("bin_dir", None)
major_version = get_major_version(bin_dir)
if pgversion != major_version:
raise ConfigParseError("data_dir directory postgresql version ({}) doesn't match with "
"'postgres --version' output ({})".format(pgversion, major_version))
return True
class Result(object):
def __init__(self, status, error="didn't pass validation", level=0, path="", data=""):
self.status = status
self.path = path
self.data = data
self.level = level
self._error = error
if not self.status:
self.error = error
else:
self.error = None
def __repr__(self):
return self.path + (" " + str(self.data) + " " + self._error if self.error else "")
class Case(object):
def __init__(self, schema):
self._schema = schema
class Or(object):
def __init__(self, *args):
self.args = args
class Optional(object):
def __init__(self, name):
self.name = name
class Directory(object):
def __init__(self, contains=None, contains_executable=None):
self.contains = contains
self.contains_executable = contains_executable
def validate(self, name):
if not name:
yield Result(False, "is an empty string")
elif not os.path.exists(name):
yield Result(False, "Directory '{}' does not exist.".format(name))
elif not os.path.isdir(name):
yield Result(False, "'{}' is not a directory.".format(name))
else:
if self.contains:
for path in self.contains:
if not os.path.exists(os.path.join(name, path)):
yield Result(False, "'{}' does not contain '{}'".format(name, path))
if self.contains_executable:
for program in self.contains_executable:
if not find_executable(program, name):
yield Result(False, "'{}' does not contain '{}'".format(name, program))
class Schema(object):
def __init__(self, validator):
self.validator = validator
def __call__(self, data):
for i in self.validate(data):
if not i.status:
print(i)
def validate(self, data):
self.data = data
if isinstance(self.validator, string_types):
yield Result(isinstance(self.data, string_types), "is not a string", level=1, data=self.data)
elif issubclass(type(self.validator), type):
validator = self.validator
if self.validator == str:
validator = string_types
yield Result(isinstance(self.data, validator),
"is not {}".format(_get_type_name(self.validator)), level=1, data=self.data)
elif callable(self.validator):
if hasattr(self.validator, "expected_type"):
if not isinstance(data, self.validator.expected_type):
yield Result(False, "is not {}"
.format(_get_type_name(self.validator.expected_type)), level=1, data=self.data)
return
try:
self.validator(data)
yield Result(True, data=self.data)
except Exception as e:
yield Result(False, "didn't pass validation: {}".format(e), data=self.data)
elif isinstance(self.validator, dict):
if not len(self.validator):
yield Result(isinstance(self.data, dict), "is not a dictionary", level=1, data=self.data)
elif isinstance(self.validator, list):
if not isinstance(self.data, list):
yield Result(isinstance(self.data, list), "is not a list", level=1, data=self.data)
return
for i in self.iter():
yield i
def iter(self):
if isinstance(self.validator, dict):
if not isinstance(self.data, dict):
yield Result(False, "is not a dictionary.", level=1)
else:
for i in self.iter_dict():
yield i
elif isinstance(self.validator, list):
if len(self.data) == 0:
yield Result(False, "is an empty list", data=self.data)
if len(self.validator) > 0:
for key, value in enumerate(self.data):
for v in Schema(self.validator[0]).validate(value):
yield Result(v.status, v.error,
path=(str(key) + ("." + v.path if v.path else "")), level=v.level, data=value)
elif isinstance(self.validator, Directory):
for v in self.validator.validate(self.data):
yield v
elif isinstance(self.validator, Or):
for i in self.iter_or():
yield i
def iter_dict(self):
for key in self.validator.keys():
for d in self._data_key(key):
if d not in self.data and not isinstance(key, Optional):
yield Result(False, "is not defined.", path=d)
elif d not in self.data and isinstance(key, Optional):
continue
else:
validator = self.validator[key]
if isinstance(key, Or) and isinstance(self.validator[key], Case):
validator = self.validator[key]._schema[d]
for v in Schema(validator).validate(self.data[d]):
yield Result(v.status, v.error,
path=(d + ("." + v.path if v.path else "")), level=v.level, data=v.data)
def iter_or(self):
results = []
for a in self.validator.args:
r = []
for v in Schema(a).validate(self.data):
r.append(v)
if any([x.status for x in r]) and not all([x.status for x in r]):
results += filter(lambda x: not x.status, r)
else:
results += r
if not any([x.status for x in results]):
max_level = 3
for v in sorted(results, key=lambda x: x.level):
if v.level > max_level:
break
max_level = v.level
yield Result(v.status, v.error, path=v.path, level=v.level, data=v.data)
def _data_key(self, key):
if isinstance(self.data, dict) and isinstance(key, str):
yield key
elif isinstance(key, Optional):
yield key.name
elif isinstance(key, Or):
if any([i in self.data for i in key.args]):
for i in key.args:
if i in self.data:
yield i
else:
for i in key.args:
yield i
def _get_type_name(python_type):
return {str: 'a string', int: 'and integer', float: 'a number', bool: 'a boolean',
list: 'an array', dict: 'a dictionary', string_types: "a string"}.get(
python_type, getattr(python_type, __name__, "unknown type"))
def assert_(condition, message="Wrong value"):
assert condition, message
userattributes = {"username": "", Optional("password"): ""}
available_dcs = [m.split(".")[-1] for m in dcs_modules()]
comma_separated_host_port.expected_type = string_types
validate_connect_address.expected_type = string_types
validate_host_port_listen.expected_type = string_types
validate_host_port_listen_multiple_hosts.expected_type = string_types
validate_data_dir.expected_type = string_types
schema = Schema({
"name": str,
"scope": str,
"restapi": {
"listen": validate_host_port_listen,
"connect_address": validate_connect_address
},
Optional("bootstrap"): {
"dcs": {
Optional("ttl"): int,
Optional("loop_wait"): int,
Optional("retry_timeout"): int,
Optional("maximum_lag_on_failover"): int
},
"pg_hba": [str],
"initdb": [Or(str, dict)]
},
Or(*available_dcs): Case({
"consul": {
Or("host", "url"): Case({
"host": validate_host_port,
"url": str})
},
"etcd": {
Or("host", "hosts", "srv", "url", "proxy"): Case({
"host": validate_host_port,
"hosts": Or(comma_separated_host_port, [validate_host_port]),
"srv": str,
"url": str,
"proxy": str})
},
"exhibitor": {
"hosts": [str],
"port": lambda i: assert_(int(i) <= 65535),
Optional("pool_interval"): int
},
"zookeeper": {
"hosts": Or(comma_separated_host_port, [validate_host_port]),
},
"kubernetes": {
"labels": {},
Optional("namespace"): str,
Optional("scope_label"): str,
Optional("role_label"): str,
Optional("use_endpoints"): bool,
Optional("pod_ip"): Or(is_ipv4_address, is_ipv6_address),
Optional("ports"): [{"name": str, "port": int}],
},
}),
"postgresql": {
"listen": validate_host_port_listen_multiple_hosts,
"connect_address": validate_connect_address,
"authentication": {
"replication": userattributes,
"superuser": userattributes,
"rewind": userattributes
},
"data_dir": validate_data_dir,
Optional("bin_dir"): Directory(contains_executable=["pg_ctl", "initdb", "pg_controldata", "pg_basebackup",
"postgres", "pg_isready"]),
Optional("parameters"): {
Optional("unix_socket_directories"): lambda s: assert_(all([isinstance(s, string_types), len(s)]))
},
Optional("pg_hba"): [str],
Optional("pg_ident"): [str],
Optional("pg_ctl_timeout"): int,
Optional("use_pg_rewind"): bool
},
Optional("watchdog"): {
Optional("mode"): lambda m: assert_(m in ["off", "automatic", "required"]),
Optional("device"): str
},
Optional("tags"): {
Optional("nofailover"): bool,
Optional("clonefrom"): bool,
Optional("noloadbalance"): bool,
Optional("replicatefrom"): str,
Optional("nosync"): bool
}
})
tests/__init__.py
+2 -1
View File
@@ -66,6 +66,7 @@ class MockPostmaster(object):
self.wait_for_user_backends_to_close = Mock()
self.signal_stop = Mock(return_value=None)
self.wait = Mock()
self.signal_kill = Mock(return_value=False)
class MockCursor(object):
@@ -75,7 +76,7 @@ class MockCursor(object):
self.closed = False
self.rowcount = 0
self.results = []
self.description = []
self.description = [Mock()]
def execute(self, sql, *params):
if sql.startswith('blabla'):
tests/test_ctl.py
+19 -13
View File
@@ -5,9 +5,9 @@ import unittest
from click.testing import CliRunner
from datetime import datetime, timedelta
from mock import patch, Mock
from patroni.ctl import ctl, store_config, load_config, output_members, get_dcs, parse_dcs, \
get_all_members, get_any_member, get_cursor, query_member, configure, PatroniCtlException, apply_config_changes, \
format_config_for_editing, show_diff, invoke_editor, format_pg_version, find_executable, print_output
from patroni.ctl import ctl, store_config, load_config, output_members, get_dcs, parse_dcs, get_all_members, \
get_any_member, get_cursor, query_member, configure, PatroniCtlException, apply_config_changes, show_diff, \
format_config_for_editing, invoke_editor, format_pg_version, find_executable, print_output, CONFIG_FILE_PATH
from patroni.dcs.etcd import Client, Failover
from patroni.utils import tzutc
from psycopg2 import OperationalError
@@ -18,17 +18,18 @@ from .test_etcd import etcd_read, socket_getaddrinfo
from .test_ha import get_cluster_initialized_without_leader, get_cluster_initialized_with_leader, \
get_cluster_initialized_with_only_leader, get_cluster_not_initialized_without_leader, get_cluster, Member
CONFIG_FILE_PATH = './test-ctl.yaml'
def test_rw_config():
global CONFIG_FILE_PATH
runner = CliRunner()
with runner.isolated_filesystem():
load_config(CONFIG_FILE_PATH + '/dummy', None)
store_config({'etcd': {'host': 'localhost:2379'}}, CONFIG_FILE_PATH + '/dummy')
load_config(CONFIG_FILE_PATH + '/dummy', '0.0.0.0')
os.remove(CONFIG_FILE_PATH + '/dummy')
os.rmdir(CONFIG_FILE_PATH)
load_config(CONFIG_FILE_PATH, None)
CONFIG_PATH = './test-ctl.yaml'
store_config({'etcd': {'host': 'localhost:2379'}}, CONFIG_PATH + '/dummy')
load_config(CONFIG_PATH + '/dummy', '0.0.0.0')
os.remove(CONFIG_PATH + '/dummy')
os.rmdir(CONFIG_PATH)
@patch('patroni.ctl.load_config',
@@ -44,6 +45,12 @@ class TestCtl(unittest.TestCase):
self.runner = CliRunner()
self.e = get_dcs({'etcd': {'ttl': 30, 'host': 'ok:2379', 'retry_timeout': 10}}, 'foo')
def test_load_config(self):
runner = CliRunner()
with runner.isolated_filesystem():
self.assertRaises(PatroniCtlException, load_config, './non-existing-config-file', None)
self.assertRaises(PatroniCtlException, load_config, './non-existing-config-file', None)
@patch('psycopg2.connect', psycopg2_connect)
def test_get_cursor(self):
self.assertIsNone(get_cursor(get_cluster_initialized_without_leader(), {}, role='master'))
@@ -68,10 +75,9 @@ class TestCtl(unittest.TestCase):
def test_output_members(self):
scheduled_at = datetime.now(tzutc) + timedelta(seconds=600)
cluster = get_cluster_initialized_with_leader(Failover(1, 'foo', 'bar', scheduled_at))
self.assertIsNone(output_members(cluster, name='abc', fmt='pretty'))
self.assertIsNone(output_members(cluster, name='abc', fmt='json'))
self.assertIsNone(output_members(cluster, name='abc', fmt='yaml'))
self.assertIsNone(output_members(cluster, name='abc', fmt='tsv'))
del cluster.members[1].data['conn_url']
for fmt in ('pretty', 'json', 'yaml', 'tsv'):
self.assertIsNone(output_members(cluster, name='abc', fmt=fmt))
@patch('patroni.ctl.get_dcs')
@patch.object(PoolManager, 'request', Mock(return_value=MockResponse()))
tests/test_ha.py
+16
View File
@@ -618,12 +618,17 @@ class TestHa(PostgresInit):
member = Member(0, 'test', 1, {'api_url': 'http://localhost:8011/patroni'})
self.ha.fetch_node_status(member)
@patch.object(Rewind, 'pg_rewind', true)
@patch.object(Rewind, 'check_leader_is_not_in_recovery', true)
def test_post_recover(self):
self.p.is_running = false
self.ha.has_lock = true
self.p.set_role('master')
self.assertEqual(self.ha.post_recover(), 'removed leader key after trying and failing to start postgres')
self.ha.has_lock = false
self.assertEqual(self.ha.post_recover(), 'failed to start postgres')
leader = Leader(0, 0, Member(0, 'l', 2, {"version": "1.6", "conn_url": "postgres://a", "role": "master"}))
self.ha._rewind.execute(leader)
self.p.is_running = true
self.assertIsNone(self.ha.post_recover())
@@ -812,6 +817,17 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'stopped PostgreSQL to fail over after a crash')
demote.assert_called_once()
def test_master_stop_timeout(self):
self.assertEqual(self.ha.master_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=True)):
self.assertEqual(self.ha.master_stop_timeout(), 30)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=False)):
self.assertEqual(self.ha.master_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': None})
self.assertEqual(self.ha.master_stop_timeout(), None)
@patch('patroni.postgresql.Postgresql.follow')
def test_demote_immediate(self, follow):
self.ha.has_lock = true
tests/test_kubernetes.py
+14 -1
View File
@@ -33,13 +33,18 @@ def mock_config_map(*args, **kwargs):
mock.metadata.resource_version = '2'
return mock
@patch('socket.TCP_KEEPIDLE', 4, create=True)
@patch('socket.TCP_KEEPINTVL', 5, create=True)
@patch('socket.TCP_KEEPCNT', 6, create=True)
@patch.object(k8s_client.CoreV1Api, 'patch_namespaced_config_map', mock_config_map)
@patch.object(k8s_client.CoreV1Api, 'create_namespaced_config_map', mock_config_map)
@patch('kubernetes.client.api_client.ThreadPool', Mock(), create=True)
@patch.object(Thread, 'start', Mock())
class TestKubernetes(unittest.TestCase):
@patch('socket.TCP_KEEPIDLE', 4, create=True)
@patch('socket.TCP_KEEPINTVL', 5, create=True)
@patch('socket.TCP_KEEPCNT', 6, create=True)
@patch('kubernetes.config.load_kube_config', Mock())
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_config_map', mock_list_namespaced_config_map)
@patch.object(k8s_client.CoreV1Api, 'list_namespaced_pod', mock_list_namespaced_pod)
@@ -149,6 +154,9 @@ class TestKubernetes(unittest.TestCase):
class TestCacheBuilder(unittest.TestCase):
@patch('socket.TCP_KEEPIDLE', 4, create=True)
@patch('socket.TCP_KEEPINTVL', 5, create=True)
@patch('socket.TCP_KEEPCNT', 6, create=True)
@patch('kubernetes.config.load_kube_config', Mock())
@patch('kubernetes.client.api_client.ThreadPool', Mock(), create=True)
@patch.object(Thread, 'start', Mock())
@@ -174,3 +182,8 @@ class TestCacheBuilder(unittest.TestCase):
@patch('patroni.dcs.kubernetes.ObjectCache._build_cache', Mock(side_effect=Exception))
def test_run(self):
self.assertRaises(SleepException, self.k._pods.run)
@patch('time.sleep', Mock())
def test__list(self):
self.k._pods._func = Mock(side_effect=Exception)
self.assertRaises(Exception, self.k._pods._list)
tests/test_patroni.py
+5 -1
View File
@@ -44,7 +44,11 @@ class TestPatroni(unittest.TestCase):
def test_no_config(self):
self.assertRaises(SystemExit, patroni_main)
@patch('pkgutil.get_importer', Mock(return_value=MockFrozenImporter()))
@patch('sys.argv', ['patroni.py', '--validate-config', 'postgres0.yml'])
def test_validate_config(self):
self.assertRaises(SystemExit, patroni_main)
@patch('pkgutil.iter_importers', Mock(return_value=[MockFrozenImporter()]))
@patch('sys.frozen', Mock(return_value=True), create=True)
@patch.object(BaseHTTPServer.HTTPServer, '__init__', Mock())
@patch.object(etcd.Client, 'read', etcd_read)
tests/test_postgresql.py
+23 -3
View File
@@ -1,5 +1,6 @@
import mock # for the mock.call method, importing it without a namespace breaks python3
import os
import psutil
import psycopg2
import re
import subprocess
@@ -134,6 +135,9 @@ class TestPostgresql(BaseTestPostgresql):
self.p.cancellable.cancel()
self.assertFalse(self.p.start())
with patch('patroni.postgresql.config.ConfigHandler.effective_configuration',
PropertyMock(side_effect=Exception)):
self.assertIsNone(self.p.start())
@patch.object(Postgresql, 'pg_isready')
@patch('patroni.postgresql.polling_loop', Mock(return_value=range(1)))
@@ -174,6 +178,17 @@ class TestPostgresql(BaseTestPostgresql):
mock_callback.assert_called()
mock_postmaster.signal_stop.assert_called()
# Timed out waiting for fast shutdown triggers immediate shutdown
mock_postmaster.wait.side_effect = [psutil.TimeoutExpired(30), psutil.TimeoutExpired(30), Mock()]
mock_callback.reset_mock()
self.assertTrue(self.p.stop(on_safepoint=mock_callback, stop_timeout=30))
mock_callback.assert_called()
mock_postmaster.signal_stop.assert_called()
# Immediate shutdown succeeded
mock_postmaster.wait.side_effect = [psutil.TimeoutExpired(30), Mock()]
self.assertTrue(self.p.stop(on_safepoint=mock_callback, stop_timeout=30))
# Stop signal failed
mock_postmaster.signal_stop.return_value = False
self.assertFalse(self.p.stop())
@@ -184,6 +199,11 @@ class TestPostgresql(BaseTestPostgresql):
self.assertTrue(self.p.stop(on_safepoint=mock_callback))
mock_callback.assert_called()
# Fast shutdown is timed out but when immediate postmaster is already gone
mock_postmaster.wait.side_effect = [psutil.TimeoutExpired(30), Mock()]
mock_postmaster.signal_stop.side_effect = [None, True]
self.assertTrue(self.p.stop(on_safepoint=mock_callback, stop_timeout=30))
def test_restart(self):
self.p.start = Mock(return_value=False)
self.assertFalse(self.p.restart())
@@ -200,7 +220,7 @@ class TestPostgresql(BaseTestPostgresql):
self.assertEqual(self.p.checkpoint({'user': 'postgres'}), 'is_in_recovery=true')
with patch.object(MockCursor, 'execute', Mock(return_value=None)):
self.assertIsNone(self.p.checkpoint())
self.assertEqual(self.p.checkpoint(), 'not accessible or not healty')
self.assertEqual(self.p.checkpoint(timeout=10), 'not accessible or not healty')
@patch('patroni.postgresql.config.mtime', mock_mtime)
@patch('patroni.postgresql.config.ConfigHandler._get_pg_settings')
@@ -281,8 +301,8 @@ class TestPostgresql(BaseTestPostgresql):
@patch.object(Postgresql, 'is_running', Mock(return_value=True))
def test_sync_replication_slots(self):
self.p.start()
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'},
'A': 0, 'test_3': 0, 'b': {'type': 'logical', 'plugin': '1'}}}, 1)
config = ClusterConfig(1, {'slots': {'test_3': {'database': 'a', 'plugin': 'b'},
'A': 0, 'ls': 0, 'b': {'type': 'logical', 'plugin': '1'}}}, 1)
cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem], None, None, None)
with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg2.OperationalError)):
self.p.slots_handler.sync_replication_slots(cluster)
tests/test_postmaster.py
+31
View File
@@ -63,6 +63,37 @@ class TestPostmasterProcess(unittest.TestCase):
mock_init.side_effect = None
self.assertNotEqual(PostmasterProcess.from_pid(123), None)
@patch('psutil.Process.__init__', Mock())
@patch('psutil.wait_procs', Mock())
@patch('psutil.Process.suspend')
@patch('psutil.Process.children')
@patch('psutil.Process.kill')
def test_signal_kill(self, mock_kill, mock_children, mock_suspend):
proc = PostmasterProcess(123)
# all processes successfully stopped
mock_children.return_value = [Mock()]
mock_children.return_value[0].kill.side_effect = psutil.Error
self.assertTrue(proc.signal_kill())
# postmaster has gone before suspend
mock_suspend.side_effect = psutil.NoSuchProcess(123)
self.assertTrue(proc.signal_kill())
# postmaster has gone before we got a list of children
mock_suspend.side_effect = psutil.Error()
mock_children.side_effect = psutil.NoSuchProcess(123)
self.assertTrue(proc.signal_kill())
# postmaster has gone after we got a list of children
mock_children.side_effect = psutil.Error()
mock_kill.side_effect = psutil.NoSuchProcess(123)
self.assertTrue(proc.signal_kill())
# failed to kill postmaster
mock_kill.side_effect = psutil.AccessDenied(123)
self.assertFalse(proc.signal_kill())
@patch('psutil.Process.__init__', Mock())
@patch('psutil.Process.send_signal')
@patch('psutil.Process.pid', Mock(return_value=123))
tests/test_validator.py
+229
View File
@@ -0,0 +1,229 @@
import copy
import os
import socket
import tempfile
import unittest
from mock import Mock, patch, mock_open
from patroni.dcs import dcs_modules
from patroni.validator import schema
from six import StringIO
available_dcs = [m.split(".")[-1] for m in dcs_modules()]
config = {
"name": "string",
"scope": "string",
"restapi": {
"listen": "127.0.0.2:800",
"connect_address": "127.0.0.2:800"
},
"bootstrap": {
"dcs": {
"ttl": 1000,
"loop_wait": 1000,
"retry_timeout": 1000,
"maximum_lag_on_failover": 1000
},
"pg_hba": ["string"],
"initdb": ["string", {"key": "value"}]
},
"consul": {
"host": "127.0.0.1:5000"
},
"etcd": {
"hosts": "127.0.0.1:2379,127.0.0.1:2380"
},
"exhibitor": {
"hosts": ["string"],
"port": 4000,
"pool_interval": 1000
},
"zookeeper": {
"hosts": "127.0.0.1:3379,127.0.0.1:3380"
},
"kubernetes": {
"namespace": "string",
"labels": {},
"scope_label": "string",
"role_label": "string",
"use_endpoints": False,
"pod_ip": "127.0.0.1",
"ports": [{"name": "string", "port": 1000}],
},
"postgresql": {
"listen": "127.0.0.2,::1:543",
"connect_address": "127.0.0.2:543",
"authentication": {
"replication": {"username": "user"},
"superuser": {"username": "user"},
"rewind": {"username": "user"},
},
"data_dir": os.path.join(tempfile.gettempdir(), "data_dir"),
"bin_dir": os.path.join(tempfile.gettempdir(), "bin_dir"),
"parameters": {
"unix_socket_directories": "."
},
"pg_hba": [u"string"],
"pg_ident": ["string"],
"pg_ctl_timeout": 1000,
"use_pg_rewind": False
},
"watchdog": {
"mode": "off",
"device": "string"
},
"tags": {
"nofailover": False,
"clonefrom": False,
"noloadbalance": False,
"nosync": False
}
}
directories = []
files = []
def isfile_side_effect(arg):
if arg.endswith('.exe'):
arg = arg[:-4]
return arg in files
def isdir_side_effect(arg):
return arg in directories
def exists_side_effect(arg):
return isfile_side_effect(arg) or isdir_side_effect(arg)
def connect_side_effect(host_port):
_, port = host_port
if port < 1000:
return 1
elif port < 10000:
return 0
else:
raise socket.gaierror()
def parse_output(output):
result = []
for s in output.split("\n"):
x = s.split(" ")[0]
if x and x not in result:
result.append(x)
result.sort()
return result
@patch('socket.socket.connect_ex', Mock(side_effect=connect_side_effect))
@patch('os.path.exists', Mock(side_effect=exists_side_effect))
@patch('os.path.isdir', Mock(side_effect=isdir_side_effect))
@patch('os.path.isfile', Mock(side_effect=isfile_side_effect))
@patch('sys.stderr', new_callable=StringIO)
@patch('sys.stdout', new_callable=StringIO)
class TestValidator(unittest.TestCase):
def setUp(self):
del files[:]
del directories[:]
def test_empty_config(self, mock_out, mock_err):
schema({})
output = mock_out.getvalue()
expected = list(sorted(['name', 'postgresql', 'restapi', 'scope'] + available_dcs))
self.assertEqual(expected, parse_output(output))
def test_complete_config(self, mock_out, mock_err):
schema(config)
output = mock_out.getvalue()
self.assertEqual(['postgresql.bin_dir'], parse_output(output))
def test_bin_dir_is_file(self, mock_out, mock_err):
files.append(config["postgresql"]["data_dir"])
files.append(config["postgresql"]["bin_dir"])
c = copy.deepcopy(config)
c["restapi"]["connect_address"] = 'False:blabla'
c["etcd"]["hosts"] = ["127.0.0.1:2379", "1244.0.0.1:2379", "127.0.0.1:invalidport"]
c["kubernetes"]["pod_ip"] = "127.0.0.1111"
schema(c)
output = mock_out.getvalue()
self.assertEqual(['etcd.hosts.1', 'etcd.hosts.2', 'kubernetes.pod_ip', 'postgresql.bin_dir',
'postgresql.data_dir', 'restapi.connect_address'], parse_output(output))
def test_bin_dir_is_empty(self, mock_out, mock_err):
directories.append(config["postgresql"]["data_dir"])
directories.append(config["postgresql"]["bin_dir"])
files.append(os.path.join(config["postgresql"]["data_dir"], "global", "pg_control"))
c = copy.deepcopy(config)
c["restapi"]["connect_address"] = "127.0.0.1:8008"
c["kubernetes"]["pod_ip"] = "::1"
c["consul"]["host"] = "127.0.0.1:50000"
c["etcd"]["host"] = "127.0.0.1:237"
c["postgresql"]["listen"] = "127.0.0.1:5432"
with patch('patroni.validator.open', mock_open(read_data='9')):
schema(c)
output = mock_out.getvalue()
self.assertEqual(['consul.host', 'etcd.host', 'postgresql.bin_dir', 'postgresql.data_dir',
'postgresql.listen', 'restapi.connect_address'], parse_output(output))
@patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 12.1"))
def test_data_dir_contains_pg_version(self, mock_out, mock_err):
directories.append(config["postgresql"]["data_dir"])
directories.append(config["postgresql"]["bin_dir"])
directories.append(os.path.join(config["postgresql"]["data_dir"], "pg_wal"))
files.append(os.path.join(config["postgresql"]["data_dir"], "global", "pg_control"))
files.append(os.path.join(config["postgresql"]["data_dir"], "PG_VERSION"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "pg_ctl"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "initdb"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "pg_controldata"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "pg_basebackup"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "postgres"))
files.append(os.path.join(config["postgresql"]["bin_dir"], "pg_isready"))
with patch('patroni.validator.open', mock_open(read_data='12')):
schema(config)
output = mock_out.getvalue()
self.assertEqual([], parse_output(output))
@patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 12.1"))
def test_pg_version_missmatch(self, mock_out, mock_err):
directories.append(config["postgresql"]["data_dir"])
directories.append(config["postgresql"]["bin_dir"])
directories.append(os.path.join(config["postgresql"]["data_dir"], "pg_wal"))
files.append(os.path.join(config["postgresql"]["data_dir"], "global", "pg_control"))
files.append(os.path.join(config["postgresql"]["data_dir"], "PG_VERSION"))
c = copy.deepcopy(config)
c["etcd"]["hosts"] = []
del c["postgresql"]["bin_dir"]
with patch('patroni.validator.open', mock_open(read_data='11')):
schema(c)
output = mock_out.getvalue()
self.assertEqual(['etcd.hosts', 'postgresql.data_dir'], parse_output(output))
@patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 12.1"))
def test_pg_wal_doesnt_exist(self, mock_out, mock_err):
directories.append(config["postgresql"]["data_dir"])
directories.append(config["postgresql"]["bin_dir"])
files.append(os.path.join(config["postgresql"]["data_dir"], "global", "pg_control"))
files.append(os.path.join(config["postgresql"]["data_dir"], "PG_VERSION"))
c = copy.deepcopy(config)
del c["postgresql"]["bin_dir"]
with patch('patroni.validator.open', mock_open(read_data='11')):
schema(c)
output = mock_out.getvalue()
self.assertEqual(['postgresql.data_dir'], parse_output(output))
def test_data_dir_is_empty_string(self, mock_out, mock_err):
directories.append(config["postgresql"]["data_dir"])
directories.append(config["postgresql"]["bin_dir"])
c = copy.deepcopy(config)
c["kubernetes"] = False
c["postgresql"]["pg_hba"] = ""
c["postgresql"]["data_dir"] = ""
c["postgresql"]["bin_dir"] = ""
schema(c)
output = mock_out.getvalue()
self.assertEqual(['kubernetes', 'postgresql.bin_dir',
'postgresql.data_dir', 'postgresql.pg_hba'], parse_output(output))