gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 07:20:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Change master->primary/leader/member (#2541)

Browse Source 
keep as much backward compatibility as possible.

Following changes were made:
1. All internal checks are performed as `role in ('master', 'primary')`
2. All internal variables/functions/methods are renamed
3. `GET /metrics` endpoint returns `patroni_primary` in addition to `patroni_master`.
4. Logs are changed to use leader/primary/member/remote depending on the context
5. Unit-tests are using only role = 'primary' instead of 'master' to verify that 1 works.
6. patronictl still supports old syntax, but also accepts `--leader` and `--primary`.
7. `master_(start|stop)_timeout` is automatically translated to `primary_(start|stop)_timeout` if the last one is not set.
8. updated the documentation and some examples

Future plan: in the next major release switch role name from `master` to `primary` and maybe drop `master` altogether.
The Kubernetes implementation will require more work and keep two labels in parallel. Label values should probably be configurable as described in https://github.com/zalando/patroni/issues/2495.
This commit is contained in:
Alexander Kukushkin
2023-01-27 07:40:24 +01:00
committed by GitHub
parent 0273eac15e
commit 4c3af2d1a0
44 changed files with 380 additions and 357 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ENVIRONMENT.rst
+2 -1
View File
@@ -54,7 +54,8 @@ Consul
- **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
- **PATRONI\_CONSUL\_CONSISTENCY**: (optional) Select consul consistency mode. Possible values are ``default``, ``consistent``, or ``stale`` (more details in `consul API reference <https://www.consul.io/api/features/consistency.html/>`__)
- **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. By default an empty list is used.
- **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
- **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, primary, replica, or standby-leader depending on the node's role. Defaults to **false**
- **PATRONI\_CONSUL\_SERVICE\_TAGS**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``primary``/``replica``/``standby-leader``). By default an empty list is used.
- **PATRONI\_CONSUL\_SERVICE\_CHECK\_INTERVAL**: (optional) how often to perform health check against registered url
- **PATRONI\_CONSUL\_SERVICE\_CHECK\_TLS\_SERVER\_NAME**: (optional) overide SNI host when connecting via TLS, see also `consul agent check API reference <https://www.consul.io/api-docs/agent/check#tlsservername>`__.
docs/SETTINGS.rst
+4 -4
View File
@@ -17,8 +17,8 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
- **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
- **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
- **max\_timelines\_history**: maximum number of timeline history items kept in DCS. Default value: 0. When set to 0, it keeps the full history in DCS.
- **master\_start\_timeout**: the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for primary failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
- **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
- **primary\_start\_timeout**: the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for primary failure is: loop\_wait + primary\_start\_timeout + loop\_wait, unless primary\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
- **primary\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by primary\_stop\_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, primary\_stop\_timeout does not apply.
- **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that successfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See :ref:`replication modes documentation <replication_modes>` for details.
- **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the primary. See :ref:`replication modes documentation <replication_modes>` for details.
- **failsafe\_mode**: Enables :ref:`DCS Failsafe Mode <dcs_failsafe_mode>`. Defaults to `false`.
@@ -139,8 +139,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
- **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
- **consistency**: (optional) Select consul consistency mode. Possible values are ``default``, ``consistent``, or ``stale`` (more details in `consul API reference <https://www.consul.io/api/features/consistency.html/>`__)
- **checks**: (optional) list of Consul health checks used for the session. By default an empty list is used.
- **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**.
- **service\_tags**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``replica``/``standby-leader``). By default an empty list is used.
- **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, primary, replica, or standby-leader depending on the node's role. Defaults to **false**.
- **service\_tags**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``primary``/``replica``/``standby-leader``). By default an empty list is used.
- **service\_check\_interval**: (optional) how often to perform health check against registered url. Defaults to '5s'.
- **service\_check\_tls\_server\_name**: (optional) overide SNI host when connecting via TLS, see also `consul agent check API reference <https://www.consul.io/api-docs/agent/check#tlsservername>`__.
docs/citus.rst
+2 -2
View File
@@ -127,7 +127,7 @@ An example of ``patronictl switchover`` on the worker cluster::
| 2 | work2-2 | 172.27.0.7 | Leader | running | 1 | |
+-------+---------+-------------+--------------+---------+----+-----------+
Citus group: 2
Master [work2-2]:
Primary [work2-2]:
Candidate ['work2-1'] []:
When should the switchover take place (e.g. 2022-12-22T08:02 ) [now]:
Current cluster topology
@@ -137,7 +137,7 @@ An example of ``patronictl switchover`` on the worker cluster::
| work2-1 | 172.27.0.5 | Sync Standby | running | 1 | 0 |
| work2-2 | 172.27.0.7 | Leader | running | 1 | |
+---------+------------+--------------+---------+----+-----------+
Are you sure you want to switchover cluster demo, demoting current master work2-2? [y/N]: y
Are you sure you want to switchover cluster demo, demoting current primary work2-2? [y/N]: y
2022-12-22 07:02:40.33003 Successfully switched over to "work2-1"
+ Citus cluster: demo (group: 2, 7179854924063375386) ------+
| Member | Host | Role | State | TL | Lag in MB |
docs/kubernetes.rst
+2 -5
View File
@@ -23,10 +23,7 @@ Use ConfigMaps
In this mode, Patroni will create ConfigMaps instead of Endpoints and store keys inside meta-data of those ConfigMaps.
Changing the leader takes at least two updates, one to the leader ConfigMap and another to the respective Endpoint.
There are two ways to direct the traffic to the Postgres leader:
- use the `callback script <https://github.com/zalando/patroni/blob/master/kubernetes/callback.py>`_ provided by Patroni
- configure the Kubernetes Postgres service to use the label selector with the `role_label` (configured in patroni configuration).
To direct the traffic to the Postgres leader you need to configure the Kubernetes Postgres service to use the label selector with the `role_label` (configured in patroni configuration).
Note that in some cases, for instance, when running on OpenShift, there is no alternative to using ConfigMaps.
@@ -39,7 +36,7 @@ Examples
--------
- The `kubernetes <https://github.com/zalando/patroni/tree/master/kubernetes>`__ folder of the Patroni repository contains
examples of the Docker image, the Kubernetes manifest and the callback script in order to test Patroni Kubernetes setup.
examples of the Docker image, and the Kubernetes manifest to test Patroni Kubernetes setup.
Note that in the current state it will not be able to use PersistentVolumes because of permission issues.
- You can find the full-featured Docker image that can use Persistent Volumes in the
docs/replica_bootstrap.rst
+3 -3
View File
@@ -77,7 +77,7 @@ scripts to clone a new replica. Those are configured in the ``postgresql`` confi
command: <command name>
keep_data: True
no_params: True
no_master: 1
no_leader: 1
example: wal_e
@@ -89,7 +89,7 @@ example: wal_e
- basebackup
wal_e:
command: patroni_wale_restore
no_master: 1
no_leader: 1
envdir: {{WALE_ENV_DIR}}
use_iam: 1
basebackup:
@@ -126,7 +126,7 @@ to execute and any custom parameters that should be passed to that command. All
Connection string to connect to the cluster member to clone from (primary or other replica). The user in the
connection string can execute SQL and replication protocol commands.
A special ``no_master`` parameter, if defined, allows Patroni to call the replica creation method even if there is no
A special ``no_leader`` parameter, if defined, allows Patroni to call the replica creation method even if there is no
running leader or replicas. In that case, an empty string will be passed in a connection string. This is useful for
restoring the formerly running cluster from the binary backup.
docs/rest_api.rst
+1 -3
View File
@@ -12,7 +12,6 @@ For all health check ``GET`` requests Patroni returns a JSON document with the s
- The following requests to Patroni REST API will return HTTP status code **200** only when the Patroni node is running as the primary with leader lock:
- ``GET /``
- ``GET /master``
- ``GET /primary``
- ``GET /read-write``
@@ -33,7 +32,6 @@ For all health check ``GET`` requests Patroni returns a JSON document with the s
In the following requests, since we are checking for the leader or standby-leader status, Patroni doesn't apply any of the user defined tags and they will be ignored.
- ``GET /?tag_key1=value1&tag_key2=value2``
- ``GET /master?tag_key1=value1&tag_key2=value2``
- ``GET /leader?tag_key1=value1&tag_key2=value2``
- ``GET /primary?tag_key1=value1&tag_key2=value2``
- ``GET /read-write?tag_key1=value1&tag_key2=value2``
@@ -368,7 +366,7 @@ Restart endpoint
- **restart_pending**: boolean, if set to ``true`` Patroni will restart PostgreSQL only when restart is pending in order to apply some changes in the PostgreSQL config.
- **role**: perform restart only if the current role of the node matches with the role from the POST request.
- **postgres_version**: perform restart only if the current version of postgres is smaller than specified in the POST request.
- **timeout**: how long we should wait before PostgreSQL starts accepting connections. Overrides ``master_start_timeout``.
- **timeout**: how long we should wait before PostgreSQL starts accepting connections. Overrides ``primary_start_timeout``.
- **schedule**: timestamp with time zone, schedule the restart somewhere in the future.
- ``DELETE /restart``: delete the scheduled restart
extras/confd/templates/haproxy.tmpl
+2 -2
View File
@@ -16,9 +16,9 @@ listen stats
stats enable
stats uri /
listen master
listen primary
bind *:5000
option httpchk HEAD /master
option httpchk HEAD /primary
http-check expect status 200
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
{{range gets "/members/*"}} server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
features/basic_replication.feature
+1 -1
View File
@@ -76,7 +76,7 @@ Feature: basic replication
Then postgres1 is a leader after 10 seconds
And postgres1 role is the primary after 10 seconds
Scenario: check rejoin of the former master with pg_rewind
Scenario: check rejoin of the former primary with pg_rewind
Given I add the table splitbrain to postgres0
And I start postgres0
Then postgres0 role is the secondary after 20 seconds
features/dcs_failsafe_mode.feature
+2 -2
View File
@@ -27,7 +27,7 @@ Feature: dcs failsafe mode
Given DCS is up
When I do a backup of postgres0
And I shut down postgres0
When I start postgres1 in a cluster batman from backup with no_master
When I start postgres1 in a cluster batman from backup with no_leader
And I sleep for 2 seconds
Then postgres1 role is the replica after 12 seconds
@@ -55,7 +55,7 @@ Feature: dcs failsafe mode
And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds
@dcs-failsafe
Scenario: check master is demoted when one replica is shut down and DCS is down
Scenario: check primary is demoted when one replica is shut down and DCS is down
Given DCS is down
And I kill postgres1
And I kill postmaster on postgres1
features/environment.py
+4 -4
View File
@@ -905,7 +905,7 @@ class PatroniPoolController(object):
}
self.start(name, custom_config=custom_config)
def bootstrap_from_backup_no_master(self, name, cluster_name):
def bootstrap_from_backup_no_leader(self, name, cluster_name):
custom_config = {
'scope': cluster_name,
'postgresql': {
@@ -914,11 +914,11 @@ class PatroniPoolController(object):
'--dirname {} --filename %f --pathname %p').format(
os.path.join(self.patroni_path, 'data', 'wal_archive').replace('\\', '/'))
},
'create_replica_methods': ['no_master_bootstrap'],
'no_master_bootstrap': {
'create_replica_methods': ['no_leader_bootstrap'],
'no_leader_bootstrap': {
'command': (self.BACKUP_RESTORE_SCRIPT + ' --sourcedir=' +
os.path.join(self.patroni_path, 'data', 'basebackup').replace('\\', '/')),
'no_master': '1'
'no_leader': '1'
}
}
}
features/ignored_slots.feature
+1 -1
View File
@@ -52,7 +52,7 @@ Feature: ignored slots
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
And postgres1 does not have a logical replication slot named dummy_slot
# 3. After a failover the server (now a master) still has the slot.
# 3. After a failover the server (now a primary) still has the slot.
When I shut down postgres0
Then "members/postgres1" key in DCS has role=master after 3 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
features/patroni_api.feature
+4 -4
View File
@@ -94,11 +94,11 @@ Scenario: check the switchover via the API in the pause mode
And postgres0 role is the secondary after 10 seconds
And replication works from postgres1 to postgres0 after 20 seconds
And "members/postgres0" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/master
When I issue a GET request to http://127.0.0.1:8008/primary
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8008/replica
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8009/master
When I issue a GET request to http://127.0.0.1:8009/primary
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 503
@@ -116,11 +116,11 @@ Scenario: check the scheduled switchover
And postgres1 role is the secondary after 10 seconds
And replication works from postgres0 to postgres1 after 25 seconds
And "members/postgres1" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/master
When I issue a GET request to http://127.0.0.1:8008/primary
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8008/replica
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8009/master
When I issue a GET request to http://127.0.0.1:8009/primary
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 200
features/standby_cluster.feature
+2 -2
View File
@@ -35,7 +35,7 @@ Feature: standby cluster
When I add the table foo to postgres0
Then table foo is present on postgres1 after 20 seconds
And I sleep for 3 seconds
When I issue a GET request to http://127.0.0.1:8009/master
When I issue a GET request to http://127.0.0.1:8009/primary
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8009/standby_leader
Then I receive a response code 200
@@ -50,7 +50,7 @@ Feature: standby cluster
When I kill postgres1
And I kill postmaster on postgres1
Then postgres2 is replicating from postgres0 after 32 seconds
When I issue a GET request to http://127.0.0.1:8010/master
When I issue a GET request to http://127.0.0.1:8010/primary
Then I receive a response code 503
And I sleep for 3 seconds
When I issue a GET request to http://127.0.0.1:8010/standby_leader
features/steps/basic_replication.py
+4 -4
View File
@@ -83,10 +83,10 @@ def check_role(context, pg_name, pg_role, max_promotion_timeout):
"{0} role didn't change to {1} after {2} seconds".format(pg_name, pg_role, max_promotion_timeout)
@step('replication works from {master:w} to {replica:w} after {time_limit:d} seconds')
@then('replication works from {master:w} to {replica:w} after {time_limit:d} seconds')
def replication_works(context, master, replica, time_limit):
@step('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
@then('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
def replication_works(context, primary, replica, time_limit):
context.execute_steps(u"""
When I add the table test_{0} to {1}
Then table test_{0} is present on {2} after {3} seconds
""".format(int(time()), master, replica, time_limit))
""".format(int(time()), primary, replica, time_limit))
features/steps/dcs_failsafe_mode.py
+3 -3
View File
@@ -11,6 +11,6 @@ def stop_dcs_outage(context):
context.dcs_ctl.stop_outage()
@step('I start {name:w} in a cluster {cluster_name:w} from backup with no_master')
def start_cluster_from_backup_no_master(context, name, cluster_name):
context.pctl.bootstrap_from_backup_no_master(name, cluster_name)
@step('I start {name:w} in a cluster {cluster_name:w} from backup with no_leader')
def start_cluster_from_backup_no_leader(context, name, cluster_name):
context.pctl.bootstrap_from_backup_no_leader(name, cluster_name)
patroni/api.py
+11 -7
View File
@@ -97,7 +97,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
def do_GET(self, write_status_code_only=False):
"""Default method for processing all GET requests which can not be routed to other methods"""
path = '/master' if self.path == '/' else self.path
path = '/primary' if self.path == '/' else self.path
response = self.get_postgresql_status()
patroni = self.server.patroni
@@ -114,8 +114,8 @@ class RestApiHandler(BaseHTTPRequestHandler):
response.get('role') == 'replica' and response.get('state') == 'running' else 503
if not cluster and patroni.ha.is_paused():
leader_status_code = 200 if response.get('role') in ('master', 'standby_leader') else 503
primary_status_code = 200 if response.get('role') == 'master' else 503
leader_status_code = 200 if response.get('role') in ('master', 'primary', 'standby_leader') else 503
primary_status_code = 200 if response.get('role') in ('master', 'primary') else 503
standby_leader_status_code = 200 if response.get('role') == 'standby_leader' else 503
elif patroni.ha.is_leader():
leader_status_code = 200
@@ -191,7 +191,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
def do_GET_liveness(self):
patroni = self.server.patroni
is_primary = patroni.postgresql.role == 'master' and patroni.postgresql.is_running()
is_primary = patroni.postgresql.role in ('master', 'primary') and patroni.postgresql.is_running()
# We can tolerate Patroni problems longer on the replica.
# On the primary the liveness probe most likely will start failing only after the leader key expired.
# It should not be a big problem because replicas will see that the primary is still alive via REST API call.
@@ -255,7 +255,11 @@ class RestApiHandler(BaseHTTPRequestHandler):
metrics.append("# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.")
metrics.append("# TYPE patroni_master gauge")
metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] == 'master')))
metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
metrics.append("# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.")
metrics.append("# TYPE patroni_primary gauge")
metrics.append("patroni_primary{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
metrics.append("# HELP patroni_xlog_location Current location of the Postgres"
" transaction log, 0 if this node is not the leader.")
@@ -443,9 +447,9 @@ class RestApiHandler(BaseHTTPRequestHandler):
status_code = _
break
elif k == 'role':
if request[k] not in ('master', 'replica'):
if request[k] not in ('master', 'primary', 'replica'):
status_code = 400
data = "PostgreSQL role should be either master or replica"
data = "PostgreSQL role should be either primary or replica"
break
elif k == 'postgres_version':
try:
patroni/config.py
+12 -5
View File
@@ -59,13 +59,17 @@ class Config(object):
PATRONI_CONFIG_VARIABLE = PATRONI_ENV_PREFIX + 'CONFIGURATION'
__CACHE_FILENAME = 'patroni.dynamic.json'
__REMAP_KEYS = {
'master_start_timeout': 'primary_start_timeout',
'master_stop_timeout': 'primary_stop_timeout'
}
__DEFAULT_CONFIG = {
'ttl': 30, 'loop_wait': 10, 'retry_timeout': 10,
'maximum_lag_on_failover': 1048576,
'maximum_lag_on_syncnode': -1,
'check_timeline': False,
'master_start_timeout': 300,
'master_stop_timeout': 0,
'primary_start_timeout': 300,
'primary_stop_timeout': 0,
'synchronous_mode': False,
'synchronous_mode_strict': False,
'synchronous_node_count': 1,
@@ -225,6 +229,9 @@ class Config(object):
config = deepcopy(self.__DEFAULT_CONFIG)
for name, value in dynamic_configuration.items():
# allow copying master_start_timeout->primary_start_timeout when the latter isn't in dynamic_configuration
if name in self.__REMAP_KEYS and self.__REMAP_KEYS[name] not in dynamic_configuration:
name = self.__REMAP_KEYS[name]
if name == 'postgresql':
for name, value in (value or {}).items():
if name == 'parameters':
@@ -355,8 +362,8 @@ class Config(object):
if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'SRV_SUFFIX', 'URL', 'PROXY',
'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY',
'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'SERVICE_CHECK_TLS_SERVER_NAME',
'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP',
'PORTS', 'LABELS', 'BYPASS_API_SERVICE', 'KEY_PASSWORD', 'USE_SSL', 'SET_ACLS',
'SERVICE_TAGS', 'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL',
'POD_IP', 'PORTS', 'LABELS', 'BYPASS_API_SERVICE', 'KEY_PASSWORD', 'USE_SSL', 'SET_ACLS',
'GROUP', 'DATABASE') and name:
value = os.environ.pop(param)
if name == 'CITUS':
@@ -366,7 +373,7 @@ class Config(object):
continue
elif suffix == 'PORT':
value = value and parse_int(value)
elif suffix in ('HOSTS', 'PORTS', 'CHECKS'):
elif suffix in ('HOSTS', 'PORTS', 'CHECKS', 'SERVICE_TAGS'):
value = value and _parse_list(value)
elif suffix in ('LABELS', 'SET_ACLS'):
value = _parse_dict(value)
patroni/ctl.py
+46 -44
View File
@@ -36,7 +36,7 @@ from .dcs import get_dcs as _get_dcs
from .exceptions import PatroniException
from .postgresql import Postgresql
from .postgresql.misc import postgres_version_to_int
from .utils import cluster_as_json, find_executable, patch_config, polling_loop
from .utils import cluster_as_json, find_executable, patch_config, polling_loop, is_standby_cluster
from .request import PatroniRequest
from .version import __version__
@@ -141,6 +141,7 @@ option_default_citus_group = click.option('--group', required=False, type=int, h
default=lambda: click.get_current_context().obj.get('citus', {}).get('group'))
option_citus_group = click.option('--group', required=False, type=int, help='Citus group')
option_insecure = click.option('-k', '--insecure', is_flag=True, help='Allow connections to SSL sites without certs')
role_choice = click.Choice(['leader', 'primary', 'standby-leader', 'replica', 'standby', 'any', 'master'])
@click.group()
@@ -241,24 +242,28 @@ def watching(w, watch, max_count=None, clear=True):
yield 0
def get_all_members(obj, cluster, group, role='master'):
def get_all_members(obj, cluster, group, role='leader'):
clusters = {0: cluster}
if obj.get('citus') and group is None:
clusters.update(cluster.workers)
if role == 'master':
if role in ('leader', 'master', 'primary', 'standby-leader'):
role = {'primary': 'master', 'standby-leader': 'standby_leader'}.get(role, role)
for cluster in clusters.values():
if cluster.leader is not None and cluster.leader.name:
if cluster.leader is not None and cluster.leader.name and\
(role == 'leader' or
cluster.leader.data.get('role') != 'master' and role == 'standby_leader' or
cluster.leader.data.get('role') != 'standby_leader' and role == 'master'):
yield cluster.leader.member
return
for cluster in clusters.values():
leader_name = (cluster.leader.member.name if cluster.leader else None)
for m in cluster.members:
if role == 'any' or role == 'replica' and m.name != leader_name:
if role == 'any' or role in ('replica', 'standby') and m.name != leader_name:
yield m
def get_any_member(obj, cluster, group, role='master', member=None):
def get_any_member(obj, cluster, group, role='leader', member=None):
for m in get_all_members(obj, cluster, group, role):
if member is None or m.name == member:
return m
@@ -273,7 +278,7 @@ def get_all_members_leader_first(cluster):
yield member
def get_cursor(obj, cluster, group, connect_parameters, role='master', member=None):
def get_cursor(obj, cluster, group, connect_parameters, role='leader', member=None):
member = get_any_member(obj, cluster, group, role=role, member=member)
if member is None:
return None
@@ -288,13 +293,14 @@ def get_cursor(obj, cluster, group, connect_parameters, role='master', member=No
from . import psycopg
conn = psycopg.connect(**params)
cursor = conn.cursor()
if role == 'any':
if role in ('any', 'leader'):
return cursor
cursor.execute('SELECT pg_catalog.pg_is_in_recovery()')
in_recovery = cursor.fetchone()[0]
if in_recovery and role == 'replica' or not in_recovery and role == 'master':
if in_recovery and role in ('replica', 'standby', 'standby-leader')\
or not in_recovery and role in ('master', 'primary'):
return cursor
conn.close()
@@ -347,9 +353,8 @@ def confirm_members_action(members, force, action, scheduled_at=None):
raise PatroniCtlException('Aborted {0}'.format(action))
@ctl.command('dsn', help='Generate a dsn for the provided member, defaults to a dsn of the master')
@click.option('--role', '-r', help='Give a dsn of any member with this role', type=click.Choice(['master', 'replica',
'any']), default=None)
@ctl.command('dsn', help='Generate a dsn for the provided member, defaults to a dsn of the leader')
@click.option('--role', '-r', help='Give a dsn of any member with this role', type=role_choice, default=None)
@click.option('--member', '-m', help='Generate a dsn for this member', type=str)
@arg_cluster_name
@option_citus_group
@@ -360,7 +365,7 @@ def dsn(obj, cluster_name, group, role, member):
raise PatroniCtlException('--role and --member are mutually exclusive options')
role = 'any'
if member is None and role is None:
role = 'master'
role = 'leader'
cluster = get_dcs(obj, cluster_name, group).get_cluster()
m = get_any_member(obj, cluster, group, role=role, member=member)
@@ -380,8 +385,7 @@ def dsn(obj, cluster_name, group, role, member):
@click.option('-U', '--username', help='database user name', type=str)
@option_watch
@option_watchrefresh
@click.option('--role', '-r', help='The role of the query', type=click.Choice(['master', 'replica', 'any']),
default=None)
@click.option('--role', '-r', help='The role of the query', type=role_choice, default=None)
@click.option('--member', '-m', help='Query a specific member', type=str)
@click.option('--delimiter', help='The column delimiter', default='\t')
@click.option('--command', '-c', help='The SQL commands to execute')
@@ -408,7 +412,7 @@ def query(
raise PatroniCtlException('--role and --member are mutually exclusive options')
role = 'any'
if member is None and role is None:
role = 'master'
role = 'leader'
if p_file is not None and command is not None:
raise PatroniCtlException('--file and --command are mutually exclusive options')
@@ -488,9 +492,9 @@ def remove(obj, cluster_name, group, fmt):
raise PatroniCtlException('You did not exactly type "{0}"'.format(message))
if cluster.leader and cluster.leader.name:
confirm = click.prompt('This cluster currently is healthy. Please specify the master name to continue')
confirm = click.prompt('This cluster currently is healthy. Please specify the leader name to continue')
if confirm != cluster.leader.name:
raise PatroniCtlException('You did not specify the current master of the cluster')
raise PatroniCtlException('You did not specify the current leader of the cluster')
dcs.delete_cluster()
@@ -524,8 +528,7 @@ def parse_scheduled(scheduled):
@click.argument('cluster_name')
@click.argument('member_names', nargs=-1)
@option_citus_group
@click.option('--role', '-r', help='Reload only members with this role', default='any',
type=click.Choice(['master', 'replica', 'any']))
@click.option('--role', '-r', help='Reload only members with this role', type=role_choice, default='any')
@option_force
@click.pass_obj
def reload(obj, cluster_name, member_names, group, force, role):
@@ -551,8 +554,7 @@ def reload(obj, cluster_name, member_names, group, force, role):
@click.argument('cluster_name')
@click.argument('member_names', nargs=-1)
@option_citus_group
@click.option('--role', '-r', help='Restart only members with this role', default='any',
type=click.Choice(['master', 'replica', 'any']))
@click.option('--role', '-r', help='Restart only members with this role', type=role_choice, default='any')
@click.option('--any', 'p_any', help='Restart a single member only', is_flag=True)
@click.option('--scheduled', help='Timestamp of a scheduled restart in unambiguous format (e.g. ISO 8601)',
default=None)
@@ -662,11 +664,11 @@ def reinit(obj, cluster_name, group, member_names, force, wait):
wait_on_members.remove(member)
def _do_failover_or_switchover(obj, action, cluster_name, group, master, candidate, force, scheduled=None):
def _do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force, scheduled=None):
"""
We want to trigger a failover or switchover for the specified cluster name.
We verify that the cluster name, master name and candidate name are correct.
We verify that the cluster name, leader name and candidate name are correct.
If so, we trigger an action and keep the client up to date.
"""
@@ -684,19 +686,20 @@ def _do_failover_or_switchover(obj, action, cluster_name, group, master, candida
cluster = dcs.get_cluster()
if action == 'switchover' and (cluster.leader is None or not cluster.leader.name):
raise PatroniCtlException('This cluster has no master')
raise PatroniCtlException('This cluster has no leader')
if master is None:
if leader is None:
if force or action == 'failover':
master = cluster.leader and cluster.leader.name
leader = cluster.leader and cluster.leader.name
else:
master = click.prompt('Master', type=str, default=cluster.leader.member.name)
prompt = 'Standby Leader' if is_standby_cluster(cluster.config) else 'Primary'
leader = click.prompt(prompt, type=str, default=cluster.leader.member.name)
if master is not None and cluster.leader and cluster.leader.member.name != master:
raise PatroniCtlException('Member {0} is not the leader of cluster {1}'.format(master, cluster_name))
if leader is not None and cluster.leader and cluster.leader.member.name != leader:
raise PatroniCtlException('Member {0} is not the leader of cluster {1}'.format(leader, cluster_name))
# excluding members with nofailover tag
candidate_names = [str(m.name) for m in cluster.members if m.name != master and not m.nofailover]
candidate_names = [str(m.name) for m in cluster.members if m.name != leader and not m.nofailover]
# We sort the names for consistent output to the client
candidate_names.sort()
@@ -709,7 +712,7 @@ def _do_failover_or_switchover(obj, action, cluster_name, group, master, candida
if action == 'failover' and not candidate:
raise PatroniCtlException('Failover could be performed only to a specific candidate')
if candidate == master:
if candidate == leader:
raise PatroniCtlException(action.title() + ' target and source are the same.')
if candidate and candidate not in candidate_names:
@@ -730,13 +733,13 @@ def _do_failover_or_switchover(obj, action, cluster_name, group, master, candida
raise PatroniCtlException("Can't schedule switchover in the paused state")
scheduled_at_str = scheduled_at.isoformat()
failover_value = {'leader': master, 'candidate': candidate, 'scheduled_at': scheduled_at_str}
failover_value = {'leader': leader, 'candidate': candidate, 'scheduled_at': scheduled_at_str}
logging.debug(failover_value)
# By now we have established that the leader exists and the candidate exists
if not force:
demote_msg = ', demoting current master ' + master if master else ''
demote_msg = ', demoting current leader ' + leader if leader else ''
if scheduled_at_str:
if not click.confirm('Are you sure you want to schedule {0} of cluster {1} at {2}{3}?'
.format(action, cluster_name, scheduled_at_str, demote_msg)):
@@ -768,7 +771,7 @@ def _do_failover_or_switchover(obj, action, cluster_name, group, master, candida
logging.exception(r)
logging.warning('Failing over to DCS')
click.echo('{0} Could not {1} using Patroni api, falling back to DCS'.format(timestamp(), action))
dcs.manual_failover(master, candidate, scheduled_at=scheduled_at)
dcs.manual_failover(leader, candidate, scheduled_at=scheduled_at)
output_members(obj, cluster, cluster_name, group=group)
@@ -776,26 +779,26 @@ def _do_failover_or_switchover(obj, action, cluster_name, group, master, candida
@ctl.command('failover', help='Failover to a replica')
@arg_cluster_name
@option_citus_group
@click.option('--master', help='The name of the current master', default=None)
@click.option('--leader', '--primary', '--master', 'leader', help='The name of the current leader', default=None)
@click.option('--candidate', help='The name of the candidate', default=None)
@option_force
@click.pass_obj
def failover(obj, cluster_name, group, master, candidate, force):
action = 'switchover' if master else 'failover'
_do_failover_or_switchover(obj, action, cluster_name, group, master, candidate, force)
def failover(obj, cluster_name, group, leader, candidate, force):
action = 'switchover' if leader else 'failover'
_do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force)
@ctl.command('switchover', help='Switchover to a replica')
@arg_cluster_name
@option_citus_group
@click.option('--master', help='The name of the current master', default=None)
@click.option('--leader', '--primary', '--master', 'leader', help='The name of the current leader', default=None)
@click.option('--candidate', help='The name of the candidate', default=None)
@click.option('--scheduled', help='Timestamp of a scheduled switchover in unambiguous format (e.g. ISO 8601)',
default=None)
@option_force
@click.pass_obj
def switchover(obj, cluster_name, group, master, candidate, force, scheduled):
_do_failover_or_switchover(obj, 'switchover', cluster_name, group, master, candidate, force, scheduled)
def switchover(obj, cluster_name, group, leader, candidate, force, scheduled):
_do_failover_or_switchover(obj, 'switchover', cluster_name, group, leader, candidate, force, scheduled)
def generate_topology(level, member, topology):
@@ -1007,8 +1010,7 @@ def scaffold(obj, cluster_name, group, sysid):
@option_citus_group
@click.argument('member_names', nargs=-1)
@click.argument('target', type=click.Choice(['restart', 'switchover']))
@click.option('--role', '-r', help='Flush only members with this role', default='any',
type=click.Choice(['master', 'replica', 'any']))
@click.option('--role', '-r', help='Flush only members with this role', type=role_choice, default='any')
@option_force
@click.pass_obj
def flush(obj, cluster_name, group, member_names, force, role, target):
patroni/dcs/__init__.py
+10 -11
View File
@@ -229,8 +229,7 @@ class Member(namedtuple('Member', 'index,name,session,data')):
class RemoteMember(Member):
""" Represents a remote master for a standby cluster
"""
"""Represents a remote member (typically a primary) for a standby cluster"""
def __new__(cls, name, data):
return super(RemoteMember, cls).__new__(cls, None, name, None, data)
@@ -283,7 +282,7 @@ class Leader(namedtuple('Leader', 'index,session,member')):
version = self.member.version
# 1.5.6 is the last version which doesn't expose checkpoint_after_promote: false
if version and version > (1, 5, 6):
return self.data.get('role') == 'master' and 'checkpoint_after_promote' not in self.data
return self.data.get('role') in ('master', 'primary') and 'checkpoint_after_promote' not in self.data
class Failover(namedtuple('Failover', 'index,leader,candidate,scheduled_at')):
@@ -520,16 +519,16 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_lsn,members,'
def get_replication_slots(self, my_name, role, nofailover, major_version, show_error=False):
# if the replicatefrom tag is set on the member - we should not create the replication slot for it on
# the current master, because that member would replicate from elsewhere. We still create the slot if
# the current primary, because that member would replicate from elsewhere. We still create the slot if
# the replicatefrom destination member is currently not a member of the cluster (fallback to the
# master), or if replicatefrom destination member happens to be the current master
# primary), or if replicatefrom destination member happens to be the current primary
use_slots = self.use_slots
if role in ('master', 'standby_leader'):
if role in ('master', 'primary', 'standby_leader'):
slot_members = [m.name for m in self.members if use_slots and m.name != my_name and
(m.replicatefrom is None or m.replicatefrom == my_name or
not self.has_member(m.replicatefrom))]
permanent_slots = self.__permanent_slots if use_slots and \
role == 'master' else self.__permanent_physical_slots
role in ('master', 'primary') else self.__permanent_physical_slots
else:
# only manage slots for replicas that replicate from this one, except for the leader among them
slot_members = [m.name for m in self.members if use_slots and
@@ -794,7 +793,7 @@ class AbstractDCS(object):
:param path: the path in DCS where to load Cluster(s) from.
:param loader: one of `_cluster_loader` or `_citus_cluster_loader`
:raise: `~DCSError` in case of communication problems with DCS.
If the current node was running as a master and exception
If the current node was running as a primary and exception
raised, instance would be demoted."""
def _bypass_caches(self):
@@ -926,7 +925,7 @@ class AbstractDCS(object):
"""Attempt to acquire leader lock
This method should create `/leader` key with value=`~self._name`
:param permanent: if set to `!True`, the leader key will never expire.
Used in patronictl for the external master
Used in patronictl for the external primary
:returns: `!True` if key has been created successfully.
Key must be created atomically. In case if key already exists it should not be
@@ -964,7 +963,7 @@ class AbstractDCS(object):
:param data: information about instance (including connection strings)
:param ttl: ttl for member key, optional parameter. If it is None `~self.member_ttl will be used`
:param permanent: if set to `!True`, the member key will never expire.
Used in patronictl for the external master.
Used in patronictl for the external primary
:returns: `!True` on success otherwise `!False`
"""
@@ -1031,7 +1030,7 @@ class AbstractDCS(object):
""""""
def watch(self, leader_index, timeout):
"""If the current node is a master it should just sleep.
"""If the current node is a leader it should just sleep.
Any other node should watch for changes of leader key with a given timeout
:param leader_index: index of a leader key
patroni/dcs/consul.py
+5 -1
View File
@@ -478,6 +478,10 @@ class Consul(AbstractDCS):
check['TLSServerName'] = self._service_check_tls_server_name
tags = self._service_tags[:]
tags.append(role)
if role == 'master':
tags.append('primary')
elif role == 'primary':
tags.append('master')
self._previous_loop_service_tags = self._service_tags
self._previous_loop_token = self._client.token
@@ -495,7 +499,7 @@ class Consul(AbstractDCS):
return self.deregister_service(params['service_id'])
self._previous_loop_register_service = self._register_service
if role in ['master', 'replica', 'standby-leader']:
if role in ['master', 'primary', 'replica', 'standby-leader']:
if state != 'running':
return
return self.register_service(service_name, **params)
patroni/dcs/etcd3.py
+1 -1
View File
@@ -92,7 +92,7 @@ class Unavailable(Etcd3ClientError):
code = GRPCCode.Unavailable
# https://github.com/etcd-io/etcd/blob/master/etcdserver/api/v3rpc/rpctypes/error.go
# https://github.com/etcd-io/etcd/commits/main/api/v3rpc/rpctypes/error.go
class LeaseNotFound(NotFound):
error = "etcdserver: requested lease not found"
patroni/dcs/exhibitor.py
+2 -2
View File
@@ -19,7 +19,7 @@ class ExhibitorEnsembleProvider(object):
self._uri_path = uri_path
self._poll_interval = poll_interval
self._exhibitors = hosts
self._master_exhibitors = hosts
self._boot_exhibitors = hosts
self._zookeeper_hosts = ''
self._next_poll = None
while not self.poll():
@@ -32,7 +32,7 @@ class ExhibitorEnsembleProvider(object):
json = self._query_exhibitors(self._exhibitors)
if not json:
json = self._query_exhibitors(self._master_exhibitors)
json = self._query_exhibitors(self._boot_exhibitors)
if isinstance(json, dict) and 'servers' in json and 'port' in json:
self._next_poll = time.time() + self._poll_interval
patroni/dcs/kubernetes.py
+2 -2
View File
@@ -242,7 +242,7 @@ class K8sClient(object):
def set_base_uri(self, value):
logger.info('Selected new K8s API server endpoint %s', value)
# We will connect by IP of the master node which is not listed as alternative name
# We will connect by IP of the K8s master node which is not listed as alternative name
self.pool_manager.connection_pool_kw['assert_hostname'] = False
self._base_uri = value
@@ -1190,7 +1190,7 @@ class Kubernetes(AbstractDCS):
cluster = self.cluster
if cluster and cluster.leader and cluster.leader.name == self._name:
role = 'master'
elif data['state'] == 'running' and data['role'] != 'master':
elif data['state'] == 'running' and data['role'] not in ('master', 'primary'):
role = data['role']
else:
role = None
patroni/ha.py
+71 -74
View File
@@ -173,9 +173,9 @@ class Ha(object):
else:
return self.patroni.config.check_mode(mode)
def master_stop_timeout(self):
""" Master stop timeout """
ret = parse_int(self.patroni.config['master_stop_timeout'])
def primary_stop_timeout(self):
""" Primary stop timeout """
ret = parse_int(self.patroni.config['primary_stop_timeout'])
return ret if ret and ret > 0 and self.is_synchronous_mode() else None
def is_paused(self):
@@ -341,10 +341,10 @@ class Ha(object):
ret = self.dcs.touch_member(data)
if ret:
if self._last_state != (data['state'], data['role'])\
and (data['state'], data['role']) == ('running', 'master'):
new_state = (data['state'], {'master': 'primary'}.get(data['role'], data['role']))
if self._last_state != new_state and new_state == ('running', 'primary'):
self.notify_citus_coordinator('after_promote')
self._last_state = (data['state'], data['role'])
self._last_state = new_state
return ret
def clone(self, clone_member=None, msg='(without leader)'):
@@ -369,7 +369,7 @@ class Ha(object):
ret = self._async_executor.try_run_async('bootstrap {0}'.format(msg), self.clone, args=(clone_member, msg))
return ret or 'trying to bootstrap {0}'.format(msg)
# no initialize key and node is allowed to be master and has 'bootstrap' section in a configuration file
# no initialize key and node is allowed to be primary and has 'bootstrap' section in a configuration file
elif self.cluster.initialize is None and not self.patroni.nofailover and 'bootstrap' in self.patroni.config:
if self.dcs.initialize(create_new=True): # race for initialization
self.state_handler.bootstrapping = True
@@ -397,11 +397,11 @@ class Ha(object):
def bootstrap_standby_leader(self):
""" If we found 'standby' key in the configuration, we need to bootstrap
not a real master, but a 'standby leader', that will take base backup
from a remote master and start follow it.
not a real primary, but a 'standby leader', that will take base backup
from a remote member and start follow it.
"""
clone_source = self.get_remote_master()
msg = 'clone from remote master {0}'.format(clone_source.conn_url)
clone_source = self.get_remote_member()
msg = 'clone from remote member {0}'.format(clone_source.conn_url)
result = self.clone(clone_source, msg)
with self._async_response: # pretend that post_bootstrap was already executed
self._async_response.complete(result)
@@ -418,7 +418,7 @@ class Ha(object):
return self._async_executor.try_run_async(msg, self._rewind.ensure_clean_shutdown) or msg
def _handle_rewind_or_reinitialize(self):
leader = self.get_remote_master() if self.is_standby_cluster() else self.cluster.leader
leader = self.get_remote_member() if self.is_standby_cluster() else self.cluster.leader
if not self._rewind.rewind_or_reinitialize_needed_and_possible(leader):
return None
@@ -442,12 +442,12 @@ class Ha(object):
self.watchdog.disable()
if self.has_lock() and self.update_lock():
timeout = self.patroni.config['master_start_timeout']
timeout = self.patroni.config['primary_start_timeout']
if timeout == 0:
# We are requested to prefer failing over to restarting master. But see first if there
# We are requested to prefer failing over to restarting primary. But see first if there
# is anyone to fail over to.
if self.is_failover_possible(self.cluster.members):
logger.info("Master crashed. Failing over.")
logger.info("Primary crashed. Failing over.")
self.demote('immediate')
return 'stopped PostgreSQL to fail over after a crash'
else:
@@ -475,8 +475,8 @@ class Ha(object):
role = 'standby_leader'
node_to_follow = self._get_node_to_follow(self.cluster)
elif self.is_standby_cluster() and self.cluster.is_unlocked():
msg = "trying to follow a remote master because standby cluster is unhealthy"
node_to_follow = self.get_remote_master()
msg = "trying to follow a remote member because standby cluster is unhealthy"
node_to_follow = self.get_remote_member()
else:
msg = "starting as a secondary"
node_to_follow = self._get_node_to_follow(self.cluster)
@@ -498,7 +498,7 @@ class Ha(object):
standby_config = self.get_standby_cluster_config()
is_standby_cluster = _is_standby_cluster(standby_config)
if is_standby_cluster and (self.cluster.is_unlocked() or self.has_lock(False)):
node_to_follow = self.get_remote_master()
node_to_follow = self.get_remote_member()
elif self.patroni.replicatefrom and self.patroni.replicatefrom != self.state_handler.name:
node_to_follow = cluster.get_member(self.patroni.replicatefrom)
else:
@@ -529,7 +529,7 @@ class Ha(object):
or self.cluster.is_unlocked():
if is_leader:
self.state_handler.set_role('master')
return 'continue to run as master without lock'
return 'continue to run as primary without lock'
elif self.state_handler.role != 'standby_leader':
self.state_handler.set_role('replica')
@@ -584,7 +584,7 @@ class Ha(object):
"""Process synchronous standby beahvior.
Synchronous standbys are registered in two places postgresql.conf and DCS. The order of updating them must
be right. The invariant that should be kept is that if a node is master and sync_standby is set in DCS,
be right. The invariant that should be kept is that if a node is primary and sync_standby is set in DCS,
then that node must have synchronous_standby set to that value. Or more simple, first set in postgresql.conf
and then in DCS. When removing, first remove in DCS, then in postgresql.conf. This is so we only consider
promoting standbys that were guaranteed to be replicating synchronously.
@@ -650,7 +650,7 @@ class Ha(object):
If the connection to DCS fails we run the action anyway, as this is only a hint.
There is a small race window where this function runs between a master picking us the sync standby and
There is a small race window where this function runs between a primary picking us the sync standby and
publishing it to the DCS. As the window is rather tiny consequences are holding up commits for one cycle
period we don't worry about it here."""
@@ -661,7 +661,7 @@ class Ha(object):
self._disable_sync += 1
try:
if self.touch_member():
# Master should notice the updated value during the next cycle. We will wait double that, if master
# Primary should notice the updated value during the next cycle. We will wait double that, if primary
# hasn't noticed the value by then not disabling sync replication is not likely to matter.
for _ in polling_loop(timeout=self.dcs.loop_wait*2, interval=2):
try:
@@ -670,7 +670,7 @@ class Ha(object):
except DCSError:
logger.warning("Could not get cluster state, skipping synchronous standby disable")
break
logger.info("Waiting for master to release us from synchronous standby")
logger.info("Waiting for primary to release us from synchronous standby")
else:
logger.warning("Updating member state failed, skipping synchronous standby disable")
@@ -680,14 +680,14 @@ class Ha(object):
self._disable_sync -= 1
def update_cluster_history(self):
master_timeline = self.state_handler.get_master_timeline()
primary_timeline = self.state_handler.get_primary_timeline()
cluster_history = self.cluster.history and self.cluster.history.lines
if master_timeline == 1:
if primary_timeline == 1:
if cluster_history:
self.dcs.set_history_value('[]')
elif not cluster_history or cluster_history[-1][0] != master_timeline - 1 or len(cluster_history[-1]) != 5:
elif not cluster_history or cluster_history[-1][0] != primary_timeline - 1 or len(cluster_history[-1]) != 5:
cluster_history = {line[0]: line for line in cluster_history or []}
history = self.state_handler.get_history(master_timeline)
history = self.state_handler.get_history(primary_timeline)
if history and self.cluster.config:
history = history[-self.cluster.config.max_timelines_history:]
for line in history:
@@ -700,14 +700,14 @@ class Ha(object):
line.append(cluster_history[line[0]][4])
self.dcs.set_history_value(json.dumps(history, separators=(',', ':')))
def enforce_follow_remote_master(self, message):
demote_reason = 'cannot be a real master in standby cluster'
def enforce_follow_remote_member(self, message):
demote_reason = 'cannot be a real primary in standby cluster'
return self.follow(demote_reason, message)
def enforce_master_role(self, message, promote_message):
def enforce_primary_role(self, message, promote_message):
"""
Ensure the node that has won the race for the leader key meets criteria
for promoting its PG server to the 'master' role.
for promoting its PG server to the 'primary' role.
"""
if not self.is_paused():
if not self.watchdog.is_running and not self.watchdog.activate():
@@ -728,14 +728,14 @@ class Ha(object):
return 'Promotion cancelled because the pre-promote script failed'
if self.state_handler.is_leader():
# Inform the state handler about its master role.
# Inform the state handler about its primary role.
# It may be unaware of it if postgres is promoted manually.
self.state_handler.set_role('master')
self.process_sync_replication()
self.update_cluster_history()
self.state_handler.citus_handler.sync_pg_dist_node(self.cluster)
return message
elif self.state_handler.role in ('master', 'promoted'):
elif self.state_handler.role in ('master', 'promoted', 'primary'):
self.process_sync_replication()
return message
else:
@@ -748,7 +748,7 @@ class Ha(object):
return 'Postponing promotion because synchronous replication state was updated by somebody else'
self.state_handler.sync_handler.set_synchronous_standby_names(
['*'] if self.is_synchronous_mode_strict() else [])
if self.state_handler.role not in ('master', 'promoted'):
if self.state_handler.role not in ('master', 'promoted', 'primary'):
def on_success():
self._rewind.reset_state()
logger.info("cleared rewind state after becoming the leader")
@@ -785,7 +785,7 @@ class Ha(object):
return results
def update_failsafe(self, data):
if self.state_handler.state == 'running' and self.state_handler.role == 'master':
if self.state_handler.state == 'running' and self.state_handler.role in ('master', 'primary'):
return 'Running as a leader'
self._failsafe.update(data)
@@ -842,7 +842,7 @@ class Ha(object):
my_wal_position = self.state_handler.last_operation()
if check_replication_lag and self.is_lagging(my_wal_position):
logger.info('My wal position exceeds maximum replication lag')
return False # Too far behind last reported wal position on master
return False # Too far behind last reported wal position on primary
if not self.is_standby_cluster() and self.check_timeline():
cluster_timeline = self.cluster.timeline
@@ -858,7 +858,7 @@ class Ha(object):
for st in self.fetch_nodes_statuses(members):
if st.failover_limitation() is None:
if not st.in_recovery:
logger.warning('Master (%s) is still alive', st.member.name)
logger.warning('Primary (%s) is still alive', st.member.name)
return False
if my_wal_position < st.wal_position:
logger.info('Wal position of %s is ahead of my wal position', st.member.name)
@@ -901,7 +901,7 @@ class Ha(object):
return True
elif self.is_paused():
# Remove failover key if the node to failover has terminated to avoid waiting for it indefinitely
# In order to avoid attempts to delete this key from all nodes only the master is allowed to do it.
# In order to avoid attempts to delete this key from all nodes only the primary is allowed to do it.
if (not self.cluster.get_member(failover.candidate, fallback_to_leader=False) and
self.state_handler.is_leader()):
logger.warning("manual failover: removing failover key because failover candidate is not running")
@@ -954,7 +954,7 @@ class Ha(object):
if self.is_paused() and not self.patroni.nofailover and \
self.cluster.failover and not self.cluster.failover.scheduled_at:
ret = self.manual_failover_process_no_leader()
if ret is not None: # continue if we just deleted the stale failover key as a master
if ret is not None: # continue if we just deleted the stale failover key as a leader
return ret
if self.state_handler.is_starting(): # postgresql still starting up is unhealthy
@@ -994,7 +994,7 @@ class Ha(object):
all_known_members += [RemoteMember(name, {'api_url': url}) for name, url in failsafe_members.items()]
all_known_members += self.cluster.members
# When in sync mode, only last known master and sync standby are allowed to promote automatically.
# When in sync mode, only last known primary and sync standby are allowed to promote automatically.
if self.is_synchronous_mode() and self.cluster.sync and self.cluster.sync.leader:
if not self.cluster.sync.matches(self.state_handler.name):
return False
@@ -1017,14 +1017,14 @@ class Ha(object):
logger.info("Leader key released")
def demote(self, mode):
"""Demote PostgreSQL running as master.
"""Demote PostgreSQL running as primary.
:param mode: One of offline, graceful or immediate.
offline is used when connection to DCS is not available.
graceful is used when failing over to another node due to user request. May only be called running async.
immediate is used when we determine that we are not suitable for master and want to failover quickly
immediate is used when we determine that we are not suitable for primary and want to failover quickly
without regard for data durability. May only be called synchronously.
immediate-nolock is used when find out that we have lost the lock to be master. Need to bring down
immediate-nolock is used when find out that we have lost the lock to be primary. Need to bring down
PostgreSQL as quickly as possible without regard for data durability. May only be called synchronously.
"""
mode_control = {
@@ -1061,7 +1061,7 @@ class Ha(object):
on_safepoint=self.watchdog.disable if self.watchdog.is_running else None,
on_shutdown=on_shutdown if mode_control['release'] else None,
before_shutdown=before_shutdown if mode == 'graceful' else None,
stop_timeout=self.master_stop_timeout())
stop_timeout=self.primary_stop_timeout())
self.state_handler.set_role('demoted')
self.set_is_leader(False)
@@ -1189,11 +1189,11 @@ class Ha(object):
if self.is_standby_cluster():
# standby leader disappeared, and this is the healthiest
# replica, so it should become a new standby leader.
# This implies we need to start following a remote master
# This implies we need to start following a remote member
msg = 'promoted self to a standby leader by acquiring session lock'
return self.enforce_follow_remote_master(msg)
return self.enforce_follow_remote_member(msg)
else:
return self.enforce_master_role(
return self.enforce_primary_role(
'acquired session lock as a leader',
'promoted self to leader by acquiring session lock'
)
@@ -1208,7 +1208,7 @@ class Ha(object):
time.sleep(2) # Give a time to somebody to take the leader lock
if self.patroni.nofailover:
return self.follow('demoting self because I am not allowed to become master',
return self.follow('demoting self because I am not allowed to become primary',
'following a different leader because I am not allowed to promote')
return self.follow('demoting self because i am not the healthiest node',
'following a different leader because i am not the healthiest node')
@@ -1217,11 +1217,11 @@ class Ha(object):
if self.has_lock():
if self.is_paused() and not self.state_handler.is_leader():
if self.cluster.failover and self.cluster.failover.candidate == self.state_handler.name:
return 'waiting to become master after promote...'
return 'waiting to become primary after promote...'
if not self.is_standby_cluster():
self._delete_leader()
return 'removed leader lock because postgres is not running as master'
return 'removed leader lock because postgres is not running as primary'
if self.update_lock(True):
msg = self.process_manual_failover_from_leader()
@@ -1233,14 +1233,14 @@ class Ha(object):
if self.is_standby_cluster():
# in case of standby cluster we don't really need to
# enforce anything, since the leader is not a master.
# enforce anything, since the leader is not a primary
# So just remind the role.
msg = 'no action. I am ({0}), the standby leader with the lock'.format(self.state_handler.name) \
if self.state_handler.role == 'standby_leader' else \
'promoted self to a standby leader because i had the session lock'
return self.enforce_follow_remote_master(msg)
return self.enforce_follow_remote_member(msg)
else:
return self.enforce_master_role(
return self.enforce_primary_role(
'no action. I am ({0}), the leader with the lock'.format(self.state_handler.name),
'promoted self to leader because I had the session lock'
)
@@ -1249,7 +1249,7 @@ class Ha(object):
logger.error('failed to update leader lock')
if self.state_handler.is_leader():
if self.is_paused():
return 'continue to run as master after failing to update leader lock in DCS'
return 'continue to run as primary after failing to update leader lock in DCS'
self.demote('immediate-nolock')
return 'demoted self because failed to update leader lock in DCS'
else:
@@ -1356,7 +1356,7 @@ class Ha(object):
# Now that restart is scheduled we can set timeout for startup, it will get reset
# once async executor runs and main loop notices PostgreSQL as up.
timeout = restart_data.get('timeout', self.patroni.config['master_start_timeout'])
timeout = restart_data.get('timeout', self.patroni.config['primary_start_timeout'])
self.set_start_timeout(timeout)
def before_shutdown():
@@ -1419,7 +1419,7 @@ class Ha(object):
"""
if self.has_lock() and self.update_lock():
if self._async_executor.scheduled_action == 'doing crash recovery in a single user mode':
time_left = self.patroni.config['master_start_timeout'] - (time.time() - self._crash_recovery_started)
time_left = self.patroni.config['primary_start_timeout'] - (time.time() - self._crash_recovery_started)
if time_left <= 0 and self.is_failover_possible(self.cluster.members):
logger.info("Demoting self because crash recovery is taking too long")
self.state_handler.cancellable.cancel(True)
@@ -1428,7 +1428,7 @@ class Ha(object):
return 'updated leader lock during ' + self._async_executor.scheduled_action
elif not self.state_handler.bootstrapping and not self.is_paused():
# Don't have lock, make sure we are not promoting or starting up a master in the background
# Don't have lock, make sure we are not promoting or starting up a primary in the background
if self._async_executor.scheduled_action == 'promote':
with self._async_response:
cancel = self._async_response.cancel()
@@ -1436,8 +1436,8 @@ class Ha(object):
self.state_handler.cancellable.cancel()
return 'lost leader before promote'
if self.state_handler.role == 'master':
logger.info("Demoting master during " + self._async_executor.scheduled_action)
if self.state_handler.role in ('master', 'primary'):
logger.info("Demoting primary during " + self._async_executor.scheduled_action)
if self._async_executor.scheduled_action == 'restart':
# Restart needs a special interlocking cancel because postmaster may be just started in a
# background thread and has not even written a pid file yet.
@@ -1462,7 +1462,7 @@ class Ha(object):
if not self.state_handler.is_running():
self.watchdog.disable()
if self.has_lock():
if self.state_handler.role in ('master', 'standby_leader'):
if self.state_handler.role in ('master', 'primary', 'standby_leader'):
self.state_handler.set_role('demoted')
self._delete_leader()
return 'removed leader key after trying and failing to start postgres'
@@ -1525,16 +1525,16 @@ class Ha(object):
self.demote('immediate-nolock')
return 'stopped PostgreSQL while starting up because leader key was lost'
timeout = self._start_timeout or self.patroni.config['master_start_timeout']
timeout = self._start_timeout or self.patroni.config['primary_start_timeout']
time_left = timeout - self.state_handler.time_in_state()
if time_left <= 0:
if self.is_failover_possible(self.cluster.members):
logger.info("Demoting self because master startup is taking too long")
logger.info("Demoting self because primary startup is taking too long")
self.demote('immediate')
return 'stopped PostgreSQL because of startup timeout'
else:
return 'master start has timed out, but continuing to wait because failover is not possible'
return 'primary start has timed out, but continuing to wait because failover is not possible'
else:
msg = self.process_manual_failover_from_leader()
if msg is not None:
@@ -1547,7 +1547,7 @@ class Ha(object):
return None
def set_start_timeout(self, value):
"""Sets timeout for starting as master before eligible for failover.
"""Sets timeout for starting as primary before eligible for failover.
Must be called when async_executor is busy or in the main thread."""
self._start_timeout = value
@@ -1622,7 +1622,7 @@ class Ha(object):
if not data_directory_is_accessible or data_directory_is_empty:
self.state_handler.set_role('uninitialized')
self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
# In case datadir went away while we were master.
# In case datadir went away while we were primary
self.watchdog.disable()
# is this instance the leader?
@@ -1662,7 +1662,7 @@ class Ha(object):
and not self.state_handler.is_leader():
self._join_aborted = True
logger.error('No initialize key in DCS and PostgreSQL is running as replica, aborting start')
logger.error('Please first start Patroni on the node running as master')
logger.error('Please first start Patroni on the node running as primary')
sys.exit(1)
self.dcs.initialize(create_new=(self.cluster.initialize is None), sysid=data_sysid)
@@ -1694,7 +1694,7 @@ class Ha(object):
# we might not have a valid PostgreSQL connection here if another thread
# stops PostgreSQL, therefore, we only reload replication slots if no
# asynchronous processes are running (should be always the case for the master)
# asynchronous processes are running (should be always the case for the primary)
if not self._async_executor.busy and not self.state_handler.is_starting():
create_slots = self._sync_replication_slots(False)
if not self.state_handler.cb_called:
@@ -1809,7 +1809,7 @@ class Ha(object):
self.while_not_sync_standby(lambda: self.state_handler.stop(checkpoint=False, on_safepoint=disable_wd,
on_shutdown=on_shutdown,
before_shutdown=before_shutdown,
stop_timeout=self.master_stop_timeout()))
stop_timeout=self.primary_stop_timeout()))
if not self.state_handler.is_running():
if self.is_leader() and not status['deleted']:
checkpoint_location = self.state_handler.latest_checkpoint_location()
@@ -1834,18 +1834,18 @@ class Ha(object):
def wakeup(self):
"""Call of this method will trigger the next run of HA loop if there is
no "active" leader watch request in progress.
This usually happens on the master or if the node is running async action"""
This usually happens on the leader or if the node is running async action"""
self.dcs.event.set()
def get_remote_member(self, member=None):
""" In case of standby cluster this will tel us from which remote
master to stream. Config can be both patroni config or
member to stream. Config can be both patroni config or
cluster.config.data
"""
cluster_params = self.get_standby_cluster_config()
if cluster_params:
name = member.name if member else 'remote_master:{}'.format(uuid.uuid1())
name = member.name if member else 'remote_member:{}'.format(uuid.uuid1())
data = {k: v for k, v in cluster_params.items() if k in RemoteMember.allowed_keys()}
data['no_replication_slot'] = 'primary_slot_name' not in cluster_params
@@ -1855,6 +1855,3 @@ class Ha(object):
data['conn_kwargs'] = conn_kwargs
return RemoteMember(name, data)
def get_remote_master(self):
return self.get_remote_member()
patroni/postgresql/__init__.py
+13 -11
View File
@@ -56,7 +56,7 @@ class Postgresql(object):
POSTMASTER_START_TIME = "pg_catalog.pg_postmaster_start_time()"
TL_LSN = ("CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
"ELSE ('x' || pg_catalog.substr(pg_catalog.pg_{0}file_name("
"pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, " # master timeline
"pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, " # primary timeline
"CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
"ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), '0/0')::bigint END, " # write_lsn
"pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), '0/0')::bigint, "
@@ -126,7 +126,7 @@ class Postgresql(object):
ident_saved = self.config.replace_pg_ident()
if hba_saved or ident_saved:
self.reload()
elif self.role == 'master':
elif self.role in ('master', 'primary'):
self.set_role('demoted')
@property
@@ -186,7 +186,7 @@ class Postgresql(object):
"FROM pg_catalog.pg_stat_get_wal_senders() w," +
" pg_catalog.pg_stat_get_activity(w.pid)" +
" WHERE w.state = 'streaming') r)").format(self.wal_name, self.lsn_name)
if self._is_synchronous_mode and self.role == 'master' else "'on', '', NULL")
if self._is_synchronous_mode and self.role in ('master', 'primary') else "'on', '', NULL")
if self._major_version >= 90600:
extra = ("(SELECT pg_catalog.json_agg(s.*) FROM (SELECT slot_name, slot_type as type, datoid::bigint, " +
@@ -331,7 +331,8 @@ class Postgresql(object):
return deepcopy(self.config.get(method, {}))
def replica_method_can_work_without_replication_connection(self, method):
return method != 'basebackup' and self.replica_method_options(method).get('no_master')
return method != 'basebackup' and (self.replica_method_options(method).get('no_master') or
self.replica_method_options(method).get('no_leader'))
def can_create_replica_without_replication_connection(self, replica_methods=None):
""" go through the replication methods to see if there are ones
@@ -421,7 +422,7 @@ class Postgresql(object):
return bool(self._cluster_info_state_get('timeline'))
except PostgresConnectionException:
logger.warning('Failed to determine PostgreSQL state from the connection, falling back to cached role')
return bool(self.is_running() and self.role == 'master')
return bool(self.is_running() and self.role in ('master', 'primary'))
def replay_paused(self):
return self._cluster_info_state_get('replay_paused')
@@ -905,12 +906,13 @@ class Postgresql(object):
except Exception:
logger.exception('Can not fetch local timeline and lsn from replication connection')
def replica_cached_timeline(self, master_timeline):
if not self._cached_replica_timeline or not master_timeline or self._cached_replica_timeline != master_timeline:
def replica_cached_timeline(self, primary_timeline):
if not self._cached_replica_timeline or not primary_timeline\
or self._cached_replica_timeline != primary_timeline:
self._cached_replica_timeline = self.get_replica_timeline()
return self._cached_replica_timeline
def get_master_timeline(self):
def get_primary_timeline(self):
return self._cluster_info_state_get('timeline')
def get_history(self, timeline):
@@ -933,11 +935,11 @@ class Postgresql(object):
recovery_params = self.config.build_recovery_params(member)
self.config.write_recovery_conf(recovery_params)
# When we demoting the master or standby_leader to replica or promoting replica to a standby_leader
# When we demoting the primary or standby_leader to replica or promoting replica to a standby_leader
# and we know for sure that postgres was already running before, we will only execute on_role_change
# callback and prevent execution of on_restart/on_start callback.
# If the role remains the same (replica or standby_leader), we will execute on_start or on_restart
change_role = self.cb_called and (self.role in ('master', 'demoted') or
change_role = self.cb_called and (self.role in ('master', 'primary', 'demoted') or
not {'standby_leader', 'replica'} - {self.role, role})
if change_role:
self.__cb_pending = ACTION_NOOP
@@ -982,7 +984,7 @@ class Postgresql(object):
return ret == 0
def promote(self, wait_seconds, task, before_promote=None, on_success=None):
if self.role in ('promoted', 'master'):
if self.role in ('promoted', 'master', 'primary'):
return True
ret = self._pre_promote()
patroni/postgresql/bootstrap.py
+4 -4
View File
@@ -155,12 +155,12 @@ class Bootstrap(object):
self._postgresql.set_state('creating replica')
self._postgresql.schedule_sanity_checks_after_pause()
is_remote_master = isinstance(clone_member, RemoteMember)
is_remote_member = isinstance(clone_member, RemoteMember)
# get list of replica methods either from clone member or from
# the config. If there is no configuration key, or no value is
# specified, use basebackup
replica_methods = (clone_member.create_replica_methods if is_remote_master
replica_methods = (clone_member.create_replica_methods if is_remote_member
else self._postgresql.create_replica_methods) or ['basebackup']
if clone_member and clone_member.conn_url:
@@ -212,7 +212,7 @@ class Bootstrap(object):
"datadir": self._postgresql.data_dir,
"connstring": connstring})
else:
for param in ('no_params', 'no_master', 'keep_data'):
for param in ('no_params', 'no_master', 'no_leader', 'keep_data'):
method_config.pop(param, None)
params = ["--{0}={1}".format(arg, val) for arg, val in method_config.items()]
try:
@@ -269,7 +269,7 @@ class Bootstrap(object):
def clone(self, clone_member):
"""
- initialize the replica from an existing member (master or replica)
- initialize the replica from an existing member (primary or replica)
- initialize the replica using the replica creation method that
works without the replication connection (i.e. restore from on-disk
base backup)
patroni/postgresql/citus.py
+1 -1
View File
@@ -153,7 +153,7 @@ class CitusHandler(Thread):
for group, worker in cluster.workers.items():
leader = worker.leader
if leader and leader.conn_url\
and leader.data.get('role') == 'master' and leader.data.get('state') == 'running':
and leader.data.get('role') in ('master', 'primary') and leader.data.get('state') == 'running':
self.add_task('after_promote', group, leader.conn_url)
def find_task_by_group(self, group):
patroni/postgresql/config.py
+6 -6
View File
@@ -529,24 +529,24 @@ class ConfigHandler(object):
recovery_params.update({'recovery_target': '', 'recovery_target_name': '', 'recovery_target_time': '',
'recovery_target_xid': '', 'recovery_target_lsn': ''})
is_remote_master = isinstance(member, RemoteMember)
is_remote_member = isinstance(member, RemoteMember)
primary_conninfo = self.primary_conninfo_params(member)
if primary_conninfo:
use_slots = self.get('use_slots', True) and self._postgresql.major_version >= 90400
if use_slots and not (is_remote_master and member.no_replication_slot):
primary_slot_name = member.primary_slot_name if is_remote_master else self._postgresql.name
if use_slots and not (is_remote_member and member.no_replication_slot):
primary_slot_name = member.primary_slot_name if is_remote_member else self._postgresql.name
recovery_params['primary_slot_name'] = slot_name_from_member_name(primary_slot_name)
# We are a standby leader and are using a replication slot. Make sure we connect to
# the leader of the main cluster (in case more than one host is specified in the
# connstr) by adding 'target_session_attrs=read-write' to primary_conninfo.
if is_remote_master and 'target_sesions_attrs' not in primary_conninfo and\
if is_remote_member and 'target_sesions_attrs' not in primary_conninfo and\
self._postgresql.major_version >= 100000:
primary_conninfo['target_session_attrs'] = 'read-write'
recovery_params['primary_conninfo'] = primary_conninfo
# standby_cluster config might have different parameters, we want to override them
standby_cluster_params = ['restore_command', 'archive_cleanup_command']\
+ (['recovery_min_apply_delay'] if is_remote_master else [])
+ (['recovery_min_apply_delay'] if is_remote_member else [])
recovery_params.update({p: member.data.get(p) for p in standby_cluster_params if member and member.data.get(p)})
return recovery_params
@@ -1065,7 +1065,7 @@ class ConfigHandler(object):
As a workaround we will start it with the values from controldata and set `pending_restart`
to true as an indicator that current values of parameters are not matching expectations."""
if self._postgresql.role == 'master':
if self._postgresql.role in ('master', 'primary'):
return self._server_parameters
options_mapping = {
patroni/postgresql/rewind.py
+19 -19
View File
@@ -126,7 +126,7 @@ class Rewind(object):
in_recovery = True
lsn = data.get('Minimum recovery ending location')
timeline = int(data.get("Min recovery ending loc's timeline"))
if lsn == '0/0' or timeline == 0: # it was a master when it crashed
if lsn == '0/0' or timeline == 0: # it was a primary when it crashed
data['Database cluster state'] = 'shut down'
if data.get('Database cluster state') == 'shut down':
in_recovery = False
@@ -157,7 +157,7 @@ class Rewind(object):
return in_recovery, timeline, lsn
@staticmethod
def _log_master_history(history, i):
def _log_primary_history(history, i):
start = max(0, i - 3)
end = None if i + 4 >= len(history) else i + 2
history_show = []
@@ -172,7 +172,7 @@ class Rewind(object):
history_show.append('...')
history_show.append(format_history_line(history[-1]))
logger.info('master: history=%s', '\n'.join(history_show))
logger.info('primary: history=%s', '\n'.join(history_show))
def _conn_kwargs(self, member, auth):
ret = member.conn_kwargs(auth)
@@ -189,7 +189,7 @@ class Rewind(object):
if local_timeline is None or local_lsn is None:
return
if isinstance(leader, Leader) and leader.member.data.get('role') != 'master':
if isinstance(leader, Leader) and leader.member.data.get('role') not in ('master', 'primary'):
return
# We want to use replication credentials when connecting to the "postgres" database in case if
@@ -206,20 +206,20 @@ class Rewind(object):
try:
with self._postgresql.get_replication_connection_cursor(**leader.conn_kwargs()) as cur:
cur.execute('IDENTIFY_SYSTEM')
master_timeline = cur.fetchone()[1]
logger.info('master_timeline=%s', master_timeline)
if local_timeline > master_timeline: # Not always supported by pg_rewind
primary_timeline = cur.fetchone()[1]
logger.info('primary_timeline=%s', primary_timeline)
if local_timeline > primary_timeline: # Not always supported by pg_rewind
need_rewind = True
elif local_timeline == master_timeline:
elif local_timeline == primary_timeline:
need_rewind = False
elif master_timeline > 1:
cur.execute('TIMELINE_HISTORY {0}'.format(master_timeline))
elif primary_timeline > 1:
cur.execute('TIMELINE_HISTORY {0}'.format(primary_timeline))
history = cur.fetchone()[1]
if not isinstance(history, six.string_types):
history = bytes(history).decode('utf-8')
logger.debug('master: history=%s', history)
logger.debug('primary: history=%s', history)
except Exception:
return logger.exception('Exception when working with master via replication connection')
return logger.exception('Exception when working with primary via replication connection')
if history is not None:
history = list(parse_history(history))
@@ -240,7 +240,7 @@ class Rewind(object):
break
else:
need_rewind = True
self._log_master_history(history, i)
self._log_primary_history(history, i)
self._state = need_rewind and REWIND_STATUS.NEED or REWIND_STATUS.NOT_NEED
@@ -270,7 +270,7 @@ class Rewind(object):
if self._checkpoint_task.result is not None:
self._state = REWIND_STATUS.CHECKPOINT
self._checkpoint_task = None
elif self._postgresql.get_master_timeline() == self._postgresql.pg_control_timeline():
elif self._postgresql.get_primary_timeline() == self._postgresql.pg_control_timeline():
self._state = REWIND_STATUS.CHECKPOINT
else:
self._checkpoint_task = CriticalTask()
@@ -431,12 +431,12 @@ class Rewind(object):
# prepare pg_rewind connection
r = self._conn_kwargs(leader, self._postgresql.config.rewind_credentials)
# 1. make sure that we are really trying to rewind from the master
# 1. make sure that we are really trying to rewind from the primary
# 2. make sure that pg_control contains the new timeline by:
# running a checkpoint or
# waiting until Patroni on the master will expose checkpoint_after_promote=True
# waiting until Patroni on the primary will expose checkpoint_after_promote=True
checkpoint_status = leader.checkpoint_after_promote if isinstance(leader, Leader) else None
if checkpoint_status is None: # we are the standby-cluster leader or master still runs the old Patroni
if checkpoint_status is None: # we are the standby-cluster leader or primary still runs the old Patroni
# superuser credentials match rewind_credentials if the latter are not provided or we run 10 or older
if self._postgresql.config.superuser == self._postgresql.config.rewind_credentials:
leader_status = self._postgresql.checkpoint(
@@ -455,11 +455,11 @@ class Rewind(object):
self._state = REWIND_STATUS.SUCCESS
else:
if not self.check_leader_is_not_in_recovery(r):
logger.warning('Failed to rewind because master %s become unreachable', leader.name)
logger.warning('Failed to rewind because primary %s become unreachable', leader.name)
if not self.can_rewind: # It is possible that the previous attempt damaged pg_control file!
self._state = REWIND_STATUS.FAILED
else:
logger.error('Failed to rewind from healty master: %s', leader.name)
logger.error('Failed to rewind from healty primary: %s', leader.name)
self._state = REWIND_STATUS.FAILED
if self.failed:
patroni/postgresql/sync.py
+1 -1
View File
@@ -254,5 +254,5 @@ class SyncHandler(object):
self._postgresql.reset_cluster_info_state(None)
# timeline == 0 -- indicates that this is the replica, shoudn't ever happen
if self._postgresql.get_master_timeline() > 0:
if self._postgresql.get_primary_timeline() > 0:
self._handle_synchronous_standby_names_change()
patroni/scripts/wale_restore.py
+14 -14
View File
@@ -11,7 +11,7 @@
# arguments are:
# - cluster scope
# - cluster role
# - master connection string
# - leader connection string
# - number of retries
# - envdir for the WALE env
# - WALE_BACKUP_THRESHOLD_MEGABYTES if WAL amount is above that - use pg_basebackup
@@ -104,11 +104,11 @@ WALEConfig = namedtuple(
class WALERestore(object):
def __init__(self, scope, datadir, connstring, env_dir, threshold_mb,
threshold_pct, use_iam, no_master, retries):
threshold_pct, use_iam, no_leader, retries):
self.scope = scope
self.master_connection = connstring
self.leader_connection = connstring
self.data_dir = datadir
self.no_master = no_master
self.no_leader = no_leader
wale_cmd = [
'envdir',
@@ -213,11 +213,11 @@ class WALERestore(object):
diff_in_bytes = backup_size
attempts_no = 0
while True:
if self.master_connection:
if self.leader_connection:
con = None
try:
# get the difference in bytes between the current WAL location and the backup start offset
con = psycopg.connect(self.master_connection)
con = psycopg.connect(self.leader_connection)
if con.server_version >= 100000:
wal_name = 'wal'
lsn_name = 'lsn'
@@ -235,22 +235,22 @@ class WALERestore(object):
diff_in_bytes = int(cur.fetchone()[0])
except psycopg.Error:
logger.exception('could not determine difference with the master location')
logger.exception('could not determine difference with the leader location')
if attempts_no < self.retries: # retry in case of a temporarily connection issue
attempts_no = attempts_no + 1
time.sleep(RETRY_SLEEP_INTERVAL)
continue
else:
if not self.no_master:
if not self.no_leader:
return False # do no more retries on the outer level
logger.info("continue with base backup from S3 since master is not available")
logger.info("continue with base backup from S3 since leader is not available")
diff_in_bytes = 0
break
finally:
if con:
con.close()
else:
# always try to use WAL-E if master connection string is not available
# always try to use WAL-E if leader connection string is not available
diff_in_bytes = 0
break
@@ -346,22 +346,22 @@ def main():
parser.add_argument('--threshold_megabytes', type=int, default=10240)
parser.add_argument('--threshold_backup_size_percentage', type=int, default=30)
parser.add_argument('--use_iam', type=int, default=0)
parser.add_argument('--no_master', type=int, default=0)
parser.add_argument('--no_leader', '--no_master', type=int, default=0)
args = parser.parse_args()
exit_code = None
assert args.retries >= 0
# Retry cloning in a loop. We do separate retries for the master
# Retry cloning in a loop. We do separate retries for the leader
# connection attempt inside should_use_s3_to_create_replica,
# because we need to differentiate between the last attempt and
# the rest and make a decision when the last attempt fails on
# whether to use WAL-E or not depending on the no_master flag.
# whether to use WAL-E or not depending on the no_leader flag.
for _ in range(0, args.retries + 1):
restore = WALERestore(scope=args.scope, datadir=args.datadir, connstring=args.connstring,
env_dir=args.envdir, threshold_mb=args.threshold_megabytes,
threshold_pct=args.threshold_backup_size_percentage, use_iam=args.use_iam,
no_master=args.no_master, retries=args.retries)
no_leader=args.no_leader, retries=args.retries)
exit_code = restore.run()
if not exit_code == ExitCode.RETRY_LATER: # only WAL-E failures lead to the retry
logger.debug('exit_code is %r, not retrying', exit_code)
postgres0.yml
+1 -1
View File
@@ -51,7 +51,7 @@ bootstrap:
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
# master_start_timeout: 300
# primary_start_timeout: 300
# synchronous_mode: false
#standby_cluster:
#host: 127.0.0.1
tests/__init__.py
+3 -4
View File
@@ -67,9 +67,8 @@ def requests_get(url, method='GET', endpoint=None, data='', **kwargs):
class MockPostmaster(object):
def __init__(self, is_running=True, is_single_master=False):
self.is_running = Mock(return_value=is_running)
self.is_single_master = Mock(return_value=is_single_master)
def __init__(self, pid=1):
self.is_running = Mock(return_value=self)
self.wait_for_user_backends_to_close = Mock()
self.signal_stop = Mock(return_value=None)
self.wait = Mock()
@@ -191,7 +190,7 @@ class PostgresInit(unittest.TestCase):
@patch.object(ConfigHandler, 'write_postgresql_conf', Mock())
@patch.object(ConfigHandler, 'replace_pg_hba', Mock())
@patch.object(ConfigHandler, 'replace_pg_ident', Mock())
@patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='master'))
@patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='primary'))
def setUp(self):
data_dir = os.path.join('data', 'test0')
self.p = Postgresql({'name': 'postgresql0', 'scope': 'batman', 'data_dir': data_dir,
tests/test_api.py
+21 -21
View File
@@ -25,7 +25,7 @@ class MockPostgresql(object):
name = 'test'
state = 'running'
role = 'master'
role = 'primary'
server_version = '999999'
sysid = 'dummysysid'
scope = 'dummy'
@@ -189,7 +189,7 @@ class TestRestApiHandler(unittest.TestCase):
MockRestApiServer(RestApiHandler, 'GET /read-only')
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={})):
MockRestApiServer(RestApiHandler, 'GET /replica')
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
MockRestApiServer(RestApiHandler, 'GET /replica')
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'state': 'running'})):
MockRestApiServer(RestApiHandler, 'GET /health')
@@ -208,11 +208,11 @@ class TestRestApiHandler(unittest.TestCase):
with patch.object(MockHa, 'is_standby_cluster', Mock(return_value=True)):
MockRestApiServer(RestApiHandler, 'GET /standby_leader')
MockPatroni.dcs.cluster = None
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
MockRestApiServer(RestApiHandler, 'GET /master')
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
MockRestApiServer(RestApiHandler, 'GET /primary')
with patch.object(MockHa, 'restart_scheduled', Mock(return_value=True)):
MockRestApiServer(RestApiHandler, 'GET /master')
self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /master'))
MockRestApiServer(RestApiHandler, 'GET /primary')
self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /primary'))
with patch.object(RestApiServer, 'query', Mock(return_value=[('', 1, '', '', '', '', False, '')])):
self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))
with patch.object(MockHa, 'is_standby_cluster', Mock(return_value=True)):
@@ -220,30 +220,30 @@ class TestRestApiHandler(unittest.TestCase):
# test tags
#
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=False&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1.0&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
#
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=False&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1.0&tag_key4=1.4&tag_key5=RandomTag')
MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
#
@@ -274,7 +274,7 @@ class TestRestApiHandler(unittest.TestCase):
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
#
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
MockRestApiServer(RestApiHandler, 'GET /replica?lag=1M&'
'tag_key1=true&tag_key2=false&'
'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
@@ -428,27 +428,27 @@ class TestRestApiHandler(unittest.TestCase):
request = make_request(schedule=future_restart_time.isoformat(), role='unknown', postgres_version='9.5.3')
MockRestApiServer(RestApiHandler, request)
# wrong version
request = make_request(schedule=future_restart_time.isoformat(), role='master', postgres_version='9.5.3.1')
request = make_request(schedule=future_restart_time.isoformat(), role='primary', postgres_version='9.5.3.1')
MockRestApiServer(RestApiHandler, request)
# unknown filter
request = make_request(schedule=future_restart_time.isoformat(), batman='lives')
MockRestApiServer(RestApiHandler, request)
# incorrect schedule
request = make_request(schedule='2016-08-42 12:45TZ+1', role='master')
request = make_request(schedule='2016-08-42 12:45TZ+1', role='primary')
MockRestApiServer(RestApiHandler, request)
# everything fine, but the schedule is missing
request = make_request(role='master', postgres_version='9.5.2')
request = make_request(role='primary', postgres_version='9.5.2')
MockRestApiServer(RestApiHandler, request)
for retval in (True, False):
with patch.object(MockHa, 'schedule_future_restart', Mock(return_value=retval)):
request = make_request(schedule=future_restart_time.isoformat())
MockRestApiServer(RestApiHandler, request)
with patch.object(MockHa, 'restart', Mock(return_value=(retval, "foo"))):
request = make_request(role='master', postgres_version='9.5.2')
request = make_request(role='primary', postgres_version='9.5.2')
MockRestApiServer(RestApiHandler, request)
mock_dcs.get_cluster.return_value.is_paused.return_value = True
MockRestApiServer(RestApiHandler, make_request(schedule='2016-08-42 12:45TZ+1', role='master'))
MockRestApiServer(RestApiHandler, make_request(schedule='2016-08-42 12:45TZ+1', role='primary'))
# Valid timeout
MockRestApiServer(RestApiHandler, make_request(timeout='60s'))
# Invalid timeout
tests/test_aws.py
+3 -3
View File
@@ -37,15 +37,15 @@ class TestAWSConnection(unittest.TestCase):
self.conn = AWSConnection('test')
def test_on_role_change(self):
self.assertTrue(self.conn.on_role_change('master'))
self.assertTrue(self.conn.on_role_change('primary'))
with patch.object(MockVolumes, 'filter', Mock(return_value=[])):
self.conn._retry.max_tries = 1
self.assertFalse(self.conn.on_role_change('master'))
self.assertFalse(self.conn.on_role_change('primary'))
@patch('patroni.scripts.aws.requests_get', Mock(side_effect=Exception('foo')))
def test_non_aws(self):
conn = AWSConnection('test')
self.assertFalse(conn.on_role_change("master"))
self.assertFalse(conn.on_role_change("primary"))
@patch('patroni.scripts.aws.requests_get', Mock(return_value=urllib3.HTTPResponse(status=200, body=b'foo')))
def test_aws_bizare_response(self):
tests/test_config.py
+3 -1
View File
@@ -21,7 +21,9 @@ class TestConfig(unittest.TestCase):
def test_set_dynamic_configuration(self):
with patch.object(Config, '_build_effective_configuration', Mock(side_effect=Exception)):
self.assertIsNone(self.config.set_dynamic_configuration({'foo': 'bar'}))
self.assertTrue(self.config.set_dynamic_configuration({'synchronous_mode': True, 'standby_cluster': {}}))
self.assertTrue(self.config.set_dynamic_configuration({'synchronous_mode': True,
'standby_cluster': {}, 'master_start_timeout': 1}))
self.assertEqual(self.config.get('primary_start_timeout'), 1)
def test_reload_local_configuration(self):
os.environ.update({
tests/test_consul.py
+4 -1
View File
@@ -229,7 +229,7 @@ class TestConsul(unittest.TestCase):
def test_set_history_value(self):
self.assertTrue(self.c.set_history_value('{}'))
@patch.object(consul.Consul.Agent.Service, 'register', Mock(side_effect=(False, True)))
@patch.object(consul.Consul.Agent.Service, 'register', Mock(side_effect=(False, True, True, True)))
@patch.object(consul.Consul.Agent.Service, 'deregister', Mock(return_value=True))
def test_update_service(self):
d = {'role': 'replica', 'api_url': 'http://a/t', 'conn_url': 'pg://c:1', 'state': 'running'}
@@ -244,6 +244,9 @@ class TestConsul(unittest.TestCase):
d['state'] = 'running'
d['role'] = 'bla'
self.assertIsNone(self.c.update_service({}, d))
for role in ('master', 'primary'):
d['role'] = role
self.assertTrue(self.c.update_service({}, d))
@patch.object(consul.Consul.KV, 'put', Mock(side_effect=ConsulException))
def test_reload_config(self):
tests/test_ctl.py
+29 -22
View File
@@ -25,6 +25,7 @@ from .test_ha import get_cluster_initialized_without_leader, get_cluster_initial
'etcd': {'host': 'localhost:2379'}, 'citus': {'database': 'citus', 'group': 0},
'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}}))
class TestCtl(unittest.TestCase):
TEST_ROLES = ('master', 'primary', 'leader')
@patch('socket.getaddrinfo', socket_getaddrinfo)
def setUp(self):
@@ -58,9 +59,9 @@ class TestCtl(unittest.TestCase):
@patch('patroni.psycopg.connect', psycopg_connect)
def test_get_cursor(self):
self.assertIsNone(get_cursor({}, get_cluster_initialized_without_leader(), None, {}, role='master'))
self.assertIsNotNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {}, role='master'))
for role in self.TEST_ROLES:
self.assertIsNone(get_cursor({}, get_cluster_initialized_without_leader(), None, {}, role=role))
self.assertIsNotNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {}, role=role))
# MockCursor returns pg_is_in_recovery as false
self.assertIsNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {}, role='replica'))
@@ -160,7 +161,7 @@ class TestCtl(unittest.TestCase):
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
assert result.exit_code == 1
# No master available
# No leader available
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_without_leader
result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
assert result.exit_code == 1
@@ -187,8 +188,9 @@ class TestCtl(unittest.TestCase):
def test_query(self, mock_get_dcs):
mock_get_dcs.return_value = self.e
# Mutually exclusive
result = self.runner.invoke(ctl, ['query', 'alpha', '--member', 'abc', '--role', 'master'])
assert result.exit_code == 1
for role in self.TEST_ROLES:
result = self.runner.invoke(ctl, ['query', 'alpha', '--member', 'abc', '--role', role])
assert result.exit_code == 1
with self.runner.isolated_filesystem():
with open('dummy', 'w') as dummy_file:
@@ -216,8 +218,9 @@ class TestCtl(unittest.TestCase):
def test_query_member(self):
with patch('patroni.ctl.get_cursor', Mock(return_value=MockConnect().cursor())):
rows = query_member({}, None, None, None, None, 'master', 'SELECT pg_catalog.pg_is_in_recovery()', {})
self.assertTrue('False' in str(rows))
for role in self.TEST_ROLES:
rows = query_member({}, None, None, None, None, role, 'SELECT pg_catalog.pg_is_in_recovery()', {})
self.assertTrue('False' in str(rows))
with patch.object(MockCursor, 'execute', Mock(side_effect=OperationalError('bla'))):
rows = query_member({}, None, None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
@@ -239,8 +242,9 @@ class TestCtl(unittest.TestCase):
assert 'host=127.0.0.1 port=5435' in result.output
# Mutually exclusive options
result = self.runner.invoke(ctl, ['dsn', 'alpha', '--role', 'master', '--member', 'dummy'])
assert result.exit_code == 1
for role in self.TEST_ROLES:
result = self.runner.invoke(ctl, ['dsn', 'alpha', '--role', role, '--member', 'dummy'])
assert result.exit_code == 1
# Non-existing member
result = self.runner.invoke(ctl, ['dsn', 'alpha', '--member', 'dummy'])
@@ -345,14 +349,14 @@ class TestCtl(unittest.TestCase):
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
result = self.runner.invoke(ctl, ['remove', 'dummy'], input='\n')
assert 'For Citus clusters the --group must me specified' in result.output
result = self.runner.invoke(ctl, ['-k', 'remove', 'alpha', '--group', '0'], input='alpha\nslave')
result = self.runner.invoke(ctl, ['-k', 'remove', 'alpha', '--group', '0'], input='alpha\nstandby')
assert 'Please confirm' in result.output
assert 'You are about to remove all' in result.output
# Not typing an exact confirmation
assert result.exit_code == 1
# master specified does not match master of cluster
result = self.runner.invoke(ctl, ['remove', 'alpha', '--group', '0'], input='alpha\nYes I am aware\nslave')
# leader specified does not match leader of cluster
result = self.runner.invoke(ctl, ['remove', 'alpha', '--group', '0'], input='alpha\nYes I am aware\nstandby')
assert result.exit_code == 1
# cluster specified on cmdline does not match verification prompt
@@ -369,17 +373,19 @@ class TestCtl(unittest.TestCase):
assert 'Usage:' in result.output
def test_get_any_member(self):
self.assertIsNone(get_any_member({}, get_cluster_initialized_without_leader(), None, role='master'))
for role in self.TEST_ROLES:
self.assertIsNone(get_any_member({}, get_cluster_initialized_without_leader(), None, role=role))
m = get_any_member({}, get_cluster_initialized_with_leader(), None, role='master')
self.assertEqual(m.name, 'leader')
m = get_any_member({}, get_cluster_initialized_with_leader(), None, role=role)
self.assertEqual(m.name, 'leader')
def test_get_all_members(self):
self.assertEqual(list(get_all_members({}, get_cluster_initialized_without_leader(), None, role='master')), [])
for role in self.TEST_ROLES:
self.assertEqual(list(get_all_members({}, get_cluster_initialized_without_leader(), None, role=role)), [])
r = list(get_all_members({}, get_cluster_initialized_with_leader(), None, role='master'))
self.assertEqual(len(r), 1)
self.assertEqual(r[0].name, 'leader')
r = list(get_all_members({}, get_cluster_initialized_with_leader(), None, role=role))
self.assertEqual(len(r), 1)
self.assertEqual(r[0].name, 'leader')
r = list(get_all_members({}, get_cluster_initialized_with_leader(), None, role='replica'))
self.assertEqual(len(r), 1)
@@ -467,8 +473,9 @@ class TestCtl(unittest.TestCase):
mock_get_dcs.return_value = self.e
mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '-r', 'master'], input='y')
assert 'No scheduled restart' in result.output
for role in self.TEST_ROLES:
result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '-r', role], input='y')
assert 'No scheduled restart' in result.output
result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '--force'])
assert 'Success: flush scheduled restart' in result.output
tests/test_ha.py
+40 -40
View File
@@ -53,7 +53,7 @@ def get_cluster_bootstrapping_without_leader(cluster_config=None):
def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None, cluster_config=None, failsafe=False):
m1 = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres',
'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4,
'role': 'master', 'state': 'running'})
'role': 'primary', 'state': 'running'})
leader = Leader(0, 0, m1 if leader else Member(0, '', 28, {}))
m2 = Member(0, 'other', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres',
'api_url': 'http://127.0.0.1:8011/patroni',
@@ -181,7 +181,7 @@ def run_async(self, func, args=()):
@patch.object(Postgresql, 'checkpoint', Mock())
@patch.object(CancellableSubprocess, 'call', Mock(return_value=0))
@patch.object(Postgresql, 'get_replica_timeline', Mock(return_value=2))
@patch.object(Postgresql, 'get_master_timeline', Mock(return_value=2))
@patch.object(Postgresql, 'get_primary_timeline', Mock(return_value=2))
@patch.object(ConfigHandler, 'restore_configuration_files', Mock())
@patch.object(etcd.Client, 'write', etcd_write)
@patch.object(etcd.Client, 'read', etcd_read)
@@ -229,7 +229,7 @@ class TestHa(PostgresInit):
self.p.timeline_wal_position = Mock(return_value=(0, 1, 1))
self.p.set_role('standby_leader')
self.ha.touch_member()
self.p.set_role('master')
self.p.set_role('primary')
self.ha.dcs.touch_member = true
self.ha.touch_member()
@@ -280,11 +280,11 @@ class TestHa(PostgresInit):
self.ha.dcs.__class__.__name__ = 'Raft'
self.assertEqual(self.ha.run_cycle(), 'started as a secondary')
def test_recover_former_master(self):
def test_recover_former_primary(self):
self.p.follow = false
self.p.is_running = false
self.p.name = 'leader'
self.p.set_role('master')
self.p.set_role('primary')
self.p.controldata = lambda: {'Database cluster state': 'shut down', 'Database system identifier': SYSID}
self.ha.cluster = get_cluster_initialized_with_leader()
self.assertEqual(self.ha.run_cycle(), 'starting as readonly because i had the session lock')
@@ -322,7 +322,7 @@ class TestHa(PostgresInit):
def test_recover_with_rewind(self):
self.p.is_running = false
self.ha.cluster = get_cluster_initialized_with_leader()
self.ha.cluster.leader.member.data.update(version='2.0.2', role='master')
self.ha.cluster.leader.member.data.update(version='2.0.2', role='primary')
self.ha._rewind.pg_rewind = true
self.ha._rewind.check_leader_is_not_in_recovery = true
with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True)):
@@ -359,7 +359,7 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because I had the session lock')
@patch('patroni.psycopg.connect', psycopg_connect)
def test_acquire_lock_as_master(self):
def test_acquire_lock_as_primary(self):
self.assertEqual(self.ha.run_cycle(), 'acquired session lock as a leader')
def test_promoted_by_acquiring_lock(self):
@@ -384,7 +384,7 @@ class TestHa(PostgresInit):
self.ha.cluster.is_unlocked = false
self.ha.has_lock = true
self.p.is_leader = false
self.p.set_role('master')
self.p.set_role('primary')
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
def test_demote_after_failing_to_obtain_lock(self):
@@ -468,7 +468,7 @@ class TestHa(PostgresInit):
def test_follow_in_pause(self):
self.ha.cluster.is_unlocked = false
self.ha.is_paused = true
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
self.p.is_leader = false
self.assertEqual(self.ha.run_cycle(), 'PAUSE: no action. I am (postgresql0)')
@@ -480,7 +480,7 @@ class TestHa(PostgresInit):
self.ha.cluster = get_cluster_initialized_with_leader()
self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')
def test_no_dcs_connection_master_demote(self):
def test_no_dcs_connection_primary_demote(self):
self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
self.assertEqual(self.ha.run_cycle(), 'demoting self because DCS is not accessible and I was a leader')
self.ha._async_executor.schedule('dummy')
@@ -539,7 +539,7 @@ class TestHa(PostgresInit):
def test_update_failsafe(self):
self.assertRaises(Exception, self.ha.update_failsafe, {})
self.p.set_role('master')
self.p.set_role('primary')
self.assertEqual(self.ha.update_failsafe({}), 'Running as a leader')
@patch('time.sleep', Mock())
@@ -655,7 +655,7 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'updated leader lock during restart')
self.ha.update_lock = false
self.p.set_role('master')
self.p.set_role('primary')
with patch('patroni.async_executor.CriticalTask.cancel', Mock(return_value=False)):
with patch('patroni.postgresql.Postgresql.terminate_starting_postmaster') as mock_terminate:
self.assertEqual(self.ha.run_cycle(), 'lost leader lock during restart')
@@ -813,9 +813,9 @@ class TestHa(PostgresInit):
def test_manual_failover_process_no_leader_in_pause(self):
self.ha.is_paused = true
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', '', None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'blabla', None))
self.assertEqual('PAUSE: acquired session lock as a leader', self.ha.run_cycle())
self.p.is_leader = false
@@ -874,11 +874,11 @@ class TestHa(PostgresInit):
def test_post_recover(self):
self.p.is_running = false
self.ha.has_lock = true
self.p.set_role('master')
self.p.set_role('primary')
self.assertEqual(self.ha.post_recover(), 'removed leader key after trying and failing to start postgres')
self.ha.has_lock = false
self.assertEqual(self.ha.post_recover(), 'failed to start postgres')
leader = Leader(0, 0, Member(0, 'l', 2, {"version": "1.6", "conn_url": "postgres://a", "role": "master"}))
leader = Leader(0, 0, Member(0, 'l', 2, {"version": "1.6", "conn_url": "postgres://a", "role": "primary"}))
self.ha._rewind.execute(leader)
self.p.is_running = true
self.assertIsNone(self.ha.post_recover())
@@ -925,7 +925,7 @@ class TestHa(PostgresInit):
self.p._role = 'replica'
self.p._connection.server_version = 90500
self.p._pending_restart = True
self.assertFalse(self.ha.restart_matches("master", "9.5.0", True))
self.assertFalse(self.ha.restart_matches("primary", "9.5.0", True))
self.assertFalse(self.ha.restart_matches("replica", "9.4.3", True))
self.p._pending_restart = False
self.assertFalse(self.ha.restart_matches("replica", "9.5.2", True))
@@ -936,9 +936,9 @@ class TestHa(PostgresInit):
self.ha.is_paused = true
self.p.name = 'leader'
self.ha.cluster = get_cluster_initialized_with_leader()
self.assertEqual(self.ha.run_cycle(), 'PAUSE: removed leader lock because postgres is not running as master')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: removed leader lock because postgres is not running as primary')
self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', self.p.name, None))
self.assertEqual(self.ha.run_cycle(), 'PAUSE: waiting to become master after promote...')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: waiting to become primary after promote...')
@patch('patroni.postgresql.mtime', Mock(return_value=1588316884))
@patch.object(builtins, 'open', mock_open(read_data='1\t0/40159C0\tno recovery target specified\n'))
@@ -979,7 +979,7 @@ class TestHa(PostgresInit):
self.p.name = 'replica'
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.ha.is_unlocked = true
self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_master:'))
self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_member:'))
def test_recover_unhealthy_leader_in_standby_cluster(self):
self.p.is_leader = false
@@ -997,7 +997,7 @@ class TestHa(PostgresInit):
self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
self.ha.cluster.is_unlocked = true
self.ha.has_lock = false
self.assertEqual(self.ha.run_cycle(), 'trying to follow a remote master because standby cluster is unhealthy')
self.assertEqual(self.ha.run_cycle(), 'trying to follow a remote member because standby cluster is unhealthy')
def test_failed_to_update_lock_in_pause(self):
self.ha.update_lock = false
@@ -1005,7 +1005,7 @@ class TestHa(PostgresInit):
self.p.name = 'leader'
self.ha.cluster = get_cluster_initialized_with_leader()
self.assertEqual(self.ha.run_cycle(),
'PAUSE: continue to run as master after failing to update leader lock in DCS')
'PAUSE: continue to run as primary after failing to update leader lock in DCS')
def test_postgres_unhealthy_in_pause(self):
self.ha.is_paused = true
@@ -1039,7 +1039,7 @@ class TestHa(PostgresInit):
self.p.time_in_state = lambda: 350
self.ha.fetch_node_status = get_node_status(reachable=False) # inaccessible, in_recovery
self.assertEqual(self.ha.run_cycle(),
'master start has timed out, but continuing to wait because failover is not possible')
'primary start has timed out, but continuing to wait because failover is not possible')
check_calls([(update_lock, True), (demote, False)])
self.ha.fetch_node_status = get_node_status() # accessible, in_recovery
@@ -1065,27 +1065,27 @@ class TestHa(PostgresInit):
self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')
@patch('patroni.ha.Ha.demote')
def test_failover_immediately_on_zero_master_start_timeout(self, demote):
def test_failover_immediately_on_zero_primary_start_timeout(self, demote):
self.p.is_running = false
self.ha.cluster = get_cluster_initialized_with_leader(sync=(self.p.name, 'other'))
self.ha.cluster.config.data['synchronous_mode'] = True
self.ha.patroni.config.set_dynamic_configuration({'master_start_timeout': 0})
self.ha.patroni.config.set_dynamic_configuration({'primary_start_timeout': 0})
self.ha.has_lock = true
self.ha.update_lock = true
self.ha.fetch_node_status = get_node_status() # accessible, in_recovery
self.assertEqual(self.ha.run_cycle(), 'stopped PostgreSQL to fail over after a crash')
demote.assert_called_once()
def test_master_stop_timeout(self):
self.assertEqual(self.ha.master_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
def test_primary_stop_timeout(self):
self.assertEqual(self.ha.primary_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': 30})
with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=True)):
self.assertEqual(self.ha.master_stop_timeout(), 30)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
self.assertEqual(self.ha.primary_stop_timeout(), 30)
self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': 30})
with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=False)):
self.assertEqual(self.ha.master_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': None})
self.assertEqual(self.ha.master_stop_timeout(), None)
self.assertEqual(self.ha.primary_stop_timeout(), None)
self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': None})
self.assertEqual(self.ha.primary_stop_timeout(), None)
@patch('patroni.postgresql.Postgresql.follow')
def test_demote_immediate(self, follow):
@@ -1177,7 +1177,7 @@ class TestHa(PostgresInit):
self.ha.run_cycle()
mock_set_sync.assert_called_once_with(['*'])
def test_sync_replication_become_master(self):
def test_sync_replication_become_primary(self):
self.ha.is_synchronous_mode = true
mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
@@ -1188,17 +1188,17 @@ class TestHa(PostgresInit):
self.p.name = 'leader'
self.ha.cluster = get_cluster_initialized_with_leader(sync=('other', None))
# When we just became master nobody is sync
self.assertEqual(self.ha.enforce_master_role('msg', 'promote msg'), 'promote msg')
# When we just became primary nobody is sync
self.assertEqual(self.ha.enforce_primary_role('msg', 'promote msg'), 'promote msg')
mock_set_sync.assert_called_once_with([])
mock_write_sync.assert_called_once_with('leader', None, index=0)
mock_set_sync.reset_mock()
# When we just became master nobody is sync
# When we just became primary nobody is sync
self.p.set_role('replica')
mock_write_sync.return_value = False
self.assertTrue(self.ha.enforce_master_role('msg', 'promote msg') != 'promote msg')
self.assertTrue(self.ha.enforce_primary_role('msg', 'promote msg') != 'promote msg')
mock_set_sync.assert_not_called()
def test_unhealthy_sync_mode(self):
@@ -1332,7 +1332,7 @@ class TestHa(PostgresInit):
self.ha.has_lock = true
self.ha.cluster.is_unlocked = false
for tl in (1, 3):
self.p.get_master_timeline = Mock(return_value=tl)
self.p.get_primary_timeline = Mock(return_value=tl)
self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
@patch('sys.exit', return_value=1)
@@ -1377,7 +1377,7 @@ class TestHa(PostgresInit):
def test_sysid_no_match_in_pause(self):
self.ha.is_paused = true
self.p.controldata = lambda: {'Database cluster state': 'in recovery', 'Database system identifier': '123'}
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
self.ha.has_lock = true
self.assertEqual(self.ha.run_cycle(), 'PAUSE: released leader key voluntarily due to the system ID mismatch')
tests/test_kubernetes.py
+1 -1
View File
@@ -293,7 +293,7 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
self.k.touch_member({'role': 'replica'})
self.k._name = 'p-1'
self.k.touch_member({'state': 'running', 'role': 'replica'})
self.k.touch_member({'state': 'stopped', 'role': 'master'})
self.k.touch_member({'state': 'stopped', 'role': 'primary'})
def test_initialize(self):
self.k.initialize()
tests/test_postgresql.py
+6 -5
View File
@@ -144,7 +144,8 @@ class TestPostgresql(BaseTestPostgresql):
@patch('patroni.postgresql.polling_loop', Mock(return_value=range(1)))
def test_wait_for_port_open(self, mock_pg_isready):
mock_pg_isready.return_value = STATE_NO_RESPONSE
mock_postmaster = MockPostmaster(is_running=False)
mock_postmaster = MockPostmaster()
mock_postmaster.is_running.return_value = None
# No pid file and postmaster death
self.assertFalse(self.p.wait_for_port_open(mock_postmaster, 1))
@@ -502,13 +503,13 @@ class TestPostgresql(BaseTestPostgresql):
self.p.config._config['create_replica_method'] = []
self.assertFalse(self.p.can_create_replica_without_replication_connection())
self.p.config._config['create_replica_method'] = ['wale', 'basebackup']
self.p.config._config['wale'] = {'command': 'foo', 'no_master': 1}
self.p.config._config['wale'] = {'command': 'foo', 'no_leader': 1}
self.assertTrue(self.p.can_create_replica_without_replication_connection())
def test_replica_method_can_work_without_replication_connection(self):
self.assertFalse(self.p.replica_method_can_work_without_replication_connection('basebackup'))
self.assertFalse(self.p.replica_method_can_work_without_replication_connection('foobar'))
self.p.config._config['foo'] = {'command': 'bar', 'no_master': 1}
self.p.config._config['foo'] = {'command': 'bar', 'no_leader': 1}
self.assertTrue(self.p.replica_method_can_work_without_replication_connection('foo'))
self.p.config._config['foo'] = {'command': 'bar'}
self.assertFalse(self.p.replica_method_can_work_without_replication_connection('foo'))
@@ -669,8 +670,8 @@ class TestPostgresql(BaseTestPostgresql):
def test_replica_cached_timeline(self):
self.assertEqual(self.p.replica_cached_timeline(2), 3)
def test_get_master_timeline(self):
self.assertEqual(self.p.get_master_timeline(), 1)
def test_get_primary_timeline(self):
self.assertEqual(self.p.get_primary_timeline(), 1)
@patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='replica'))
@patch.object(Bootstrap, 'running_custom_bootstrap', PropertyMock(return_value=True))
tests/test_rewind.py
+7 -7
View File
@@ -117,7 +117,7 @@ class TestRewind(BaseTestPostgresql):
self.r.trigger_check_diverged_lsn()
self.r.execute(self.leader)
self.leader.member.data.update(version='1.5.7', checkpoint_after_promote=False, role='master')
self.leader.member.data.update(version='1.5.7', checkpoint_after_promote=False, role='primary')
self.assertIsNone(self.r.execute(self.leader))
del self.leader.member.data['checkpoint_after_promote']
@@ -128,9 +128,9 @@ class TestRewind(BaseTestPostgresql):
self.r.execute(self.leader)
@patch('patroni.postgresql.rewind.logger.info')
def test__log_master_history(self, mock_logger):
def test__log_primary_history(self, mock_logger):
history = [[n, n, ''] for n in range(1, 10)]
self.r._log_master_history(history, 1)
self.r._log_primary_history(history, 1)
expected = '\n'.join(['{0}\t0/{0}\t'.format(n) for n in range(1, 4)] + ['...', '9\t0/9\t'])
self.assertEqual(mock_logger.call_args[0][1], expected)
@@ -298,14 +298,14 @@ class TestRewind(BaseTestPostgresql):
@patch('patroni.postgresql.rewind.Thread', MockThread)
@patch.object(Postgresql, 'controldata')
@patch.object(Postgresql, 'checkpoint')
@patch.object(Postgresql, 'get_master_timeline')
def test_ensure_checkpoint_after_promote(self, mock_get_master_timeline, mock_checkpoint, mock_controldata):
@patch.object(Postgresql, 'get_primary_timeline')
def test_ensure_checkpoint_after_promote(self, mock_get_primary_timeline, mock_checkpoint, mock_controldata):
mock_controldata.return_value = {"Latest checkpoint's TimeLineID": 1}
mock_get_master_timeline.return_value = 1
mock_get_primary_timeline.return_value = 1
self.r.ensure_checkpoint_after_promote(Mock())
self.r.reset_state()
mock_get_master_timeline.return_value = 2
mock_get_primary_timeline.return_value = 2
mock_checkpoint.return_value = 0
self.r.ensure_checkpoint_after_promote(Mock())
self.r.ensure_checkpoint_after_promote(Mock())
tests/test_slots.py
+1 -1
View File
@@ -52,7 +52,7 @@ class TestSlotsHandler(BaseTestPostgresql):
patch.object(SlotsHandler, 'drop_replication_slot') as mock_drop:
self.s.sync_replication_slots(cluster, False, paused=True)
mock_drop.assert_not_called()
self.p.set_role('master')
self.p.set_role('primary')
with mock.patch('patroni.postgresql.Postgresql.role', new_callable=PropertyMock(return_value='replica')):
self.s.sync_replication_slots(cluster, False)
with patch('patroni.dcs.logger.error', new_callable=Mock()) as errorlog_mock:
tests/test_wale_restore.py
+6 -6
View File
@@ -59,22 +59,22 @@ class TestWALERestore(unittest.TestCase):
self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())
with patch('patroni.psycopg.connect', Mock(side_effect=psycopg.Error("foo"))):
save_no_master = self.wale_restore.no_master
save_master_connection = self.wale_restore.master_connection
save_no_leader = self.wale_restore.no_leader
save_leader_connection = self.wale_restore.leader_connection
self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())
with patch('time.sleep', mock_sleep):
self.wale_restore.no_master = 1
self.wale_restore.no_leader = 1
self.assertTrue(self.wale_restore.should_use_s3_to_create_replica())
# verify retries
self.assertEqual(sleeps[0], WALE_TEST_RETRIES)
self.wale_restore.master_connection = ''
self.wale_restore.leader_connection = ''
self.assertTrue(self.wale_restore.should_use_s3_to_create_replica())
self.wale_restore.no_master = save_no_master
self.wale_restore.master_connection = save_master_connection
self.wale_restore.no_leader = save_no_leader
self.wale_restore.leader_connection = save_leader_connection
with patch('subprocess.check_output', Mock(side_effect=subprocess.CalledProcessError(1, "cmd", "foo"))):
self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())