mirror of
https://github.com/outbackdingo/patroni.git
synced 2026-08-25 14:53:37 +00:00
Don't create permanent physical slot with name of the primary (#1392)
It is a regular issue that primary is recycling WALs when one of the replicas is down for a long time. So far there were only two solutions for such a problem and both of them are not perfect:
1. Increase `wal_keep_segments`, but it is hard to guess the good value.
2. Use continuous archiving and PITR, but it is not always possible.
This PR is introducing the way to solve the problem for static clusters, with a fixed number of nodes and names that never change. You just need to list the names of all nodes in the `slots` so the primary will not remove the slot when the node is down (not registered in DCS).
Of course, the primary will not create the permanent slot which is matching its own name.
Usage example: let's assume you have a cluster with nodes named *abc1*, *abc2*, and *abc3*.
You have to run `patronictl edit-config` and put the following snippet into the configuration:
```yaml
slots:
abc1:
type: physical
abc2:
type: physical
abc3:
type: physical
```
If the node *abc2* is the primary, it will always create slots for *abc1* and *abc3* even if they are not running, but will not create slot *abc2*.
Other nodes will behave the same.
Close #280
This commit is contained in:
Alexander Kukushkin
+2
-2
@@ -20,7 +20,7 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
|
||||
- **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the master. See :ref:`replication modes documentation <replication_modes>` for details.
|
||||
- **postgresql**:
|
||||
- **use\_pg\_rewind**: whether or not to use pg_rewind. Defaults to `false`.
|
||||
- **use\_slots**: whether or not to use replication_slots. Defaults to `true` on PostgreSQL 9.4+.
|
||||
- **use\_slots**: whether or not to use replication slots. Defaults to `true` on PostgreSQL 9.4+.
|
||||
- **recovery\_conf**: additional configuration settings written to recovery.conf when configuring follower. There is no recovery.conf anymore in PostgreSQL 12, but you may continue using this section, because Patroni handles it transparently.
|
||||
- **parameters**: list of configuration settings for Postgres.
|
||||
- **standby\_cluster**: if this section is defined, we want to bootstrap a standby cluster.
|
||||
@@ -32,7 +32,7 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
|
||||
- **archive\_cleanup\_command**: cleanup command for standby leader
|
||||
- **recovery\_min\_apply\_delay**: how long to wait before actually apply WAL records on a standby leader
|
||||
- **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Patroni will try to create slots before opening connections to the cluster.
|
||||
- **my_slot_name**: the name of replication slot. It is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots.
|
||||
- **my_slot_name**: the name of replication slot. If the permanent slot name matches with the name of the current primary it will not be created. Everything else is the responsibility of the operator to make sure that there are no clashes in names between replication slots automatically created by Patroni for members and permanent replication slots.
|
||||
- **type**: slot type. Could be ``physical`` or ``logical``. If the slot is logical, you have to additionally define ``database`` and ``plugin``.
|
||||
- **database**: the database name where logical slots should be created.
|
||||
- **plugin**: the plugin name for the logical slot.
|
||||
|
||||
+15
-16
@@ -449,21 +449,21 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
|
||||
def is_synchronous_mode(self):
|
||||
return self.check_mode('synchronous_mode')
|
||||
|
||||
def get_replication_slots(self, name, role):
|
||||
def get_replication_slots(self, my_name, role):
|
||||
# if the replicatefrom tag is set on the member - we should not create the replication slot for it on
|
||||
# the current master, because that member would replicate from elsewhere. We still create the slot if
|
||||
# the replicatefrom destination member is currently not a member of the cluster (fallback to the
|
||||
# master), or if replicatefrom destination member happens to be the current master
|
||||
use_slots = self.config and self.config.data.get('postgresql', {}).get('use_slots', True)
|
||||
if role in ('master', 'standby_leader'):
|
||||
slot_members = [m.name for m in self.members if use_slots and m.name != name and
|
||||
(m.replicatefrom is None or m.replicatefrom == name or
|
||||
slot_members = [m.name for m in self.members if use_slots and m.name != my_name and
|
||||
(m.replicatefrom is None or m.replicatefrom == my_name or
|
||||
not self.has_member(m.replicatefrom))]
|
||||
permanent_slots = (self.config and self.config.permanent_slots or {}).copy()
|
||||
else:
|
||||
# only manage slots for replicas that replicate from this one, except for the leader among them
|
||||
slot_members = [m.name for m in self.members if use_slots and
|
||||
m.replicatefrom == name and m.name != self.leader.name]
|
||||
m.replicatefrom == my_name and m.name != self.leader.name]
|
||||
permanent_slots = {}
|
||||
|
||||
slots = {slot_name_from_member_name(name): {'type': 'physical'} for name in slot_members}
|
||||
@@ -484,22 +484,21 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_leader_operat
|
||||
logger.error("Slot name may only contain lower case letters, numbers, and the underscore chars")
|
||||
continue
|
||||
|
||||
if name in slots:
|
||||
logger.error("Permanent replication slot {'%s': %s} is conflicting with" +
|
||||
" physical replication slot for cluster member", name, value)
|
||||
continue
|
||||
|
||||
value = deepcopy(value)
|
||||
if not value:
|
||||
value = {'type': 'physical'}
|
||||
|
||||
value = deepcopy(value) if value else {'type': 'physical'}
|
||||
if isinstance(value, dict):
|
||||
if 'type' not in value:
|
||||
value['type'] = 'logical' if value.get('database') and value.get('plugin') else 'physical'
|
||||
|
||||
if value['type'] == 'physical' or value['type'] == 'logical' \
|
||||
and value.get('database') and value.get('plugin'):
|
||||
slots[name] = value
|
||||
if value['type'] == 'physical':
|
||||
if name != my_name: # Don't try to create permanent physical replication slot for yourself
|
||||
slots[name] = value
|
||||
continue
|
||||
elif value['type'] == 'logical' and value.get('database') and value.get('plugin'):
|
||||
if name in slots:
|
||||
logger.error("Permanent logical replication slot {'%s': %s} is conflicting with" +
|
||||
" physical replication slot for cluster member", name, value)
|
||||
else:
|
||||
slots[name] = value
|
||||
continue
|
||||
|
||||
logger.error("Bad value for slot '%s' in permanent_slots: %s", name, permanent_slots[name])
|
||||
|
||||
@@ -281,8 +281,8 @@ class TestPostgresql(BaseTestPostgresql):
|
||||
@patch.object(Postgresql, 'is_running', Mock(return_value=True))
|
||||
def test_sync_replication_slots(self):
|
||||
self.p.start()
|
||||
config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'},
|
||||
'A': 0, 'test_3': 0, 'b': {'type': 'logical', 'plugin': '1'}}}, 1)
|
||||
config = ClusterConfig(1, {'slots': {'test_3': {'database': 'a', 'plugin': 'b'},
|
||||
'A': 0, 'ls': 0, 'b': {'type': 'logical', 'plugin': '1'}}}, 1)
|
||||
cluster = Cluster(True, config, self.leader, 0, [self.me, self.other, self.leadermem], None, None, None)
|
||||
with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg2.OperationalError)):
|
||||
self.p.slots_handler.sync_replication_slots(cluster)
|
||||
|
||||
Reference in New Issue
Block a user