gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 14:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Configurable retention of members replication slots (#3108)

Browse Source 
Current problem of Patroni that strikes many people is that it removes replication slot for member which key is expired from DCS. As a result, when the replica comes back from a scheduled maintenance WAL segments could be already absent, and it can't continue streaming without pulling files from archive.
With PostgreSQL 16 and newer we get another problem: logical slot on a standby node could be invalidated if physical replication slot on the primary was removed (and `pg_catalog` vacuumed).
The most problematic environment is Kubernetes, where slot is removed nearly instantly when member Pod is deleted.

So far, one of the recommended solutions was to configure permanent physical slots with names that match member names to avoid removal of replication slots. It works, but depending on environment might be non-trivial to implement (when for example members may change their names).

This PR implements support of `member_slots_ttl` global configuration parameter, that controls for how long member replication slots should be kept when the member key is absent. Default value is set to `30min`.
The feature is supported only starting from PostgreSQL 11 and newer, because we want to retain slots not only on the leader node, but on all nodes that could potentially become the new leader, and they should be moved forward using `pg_replication_slot_advance()` function.

One could disable feature and get back to the old behavior by setting `member_slots_ttl` to `0`.
This commit is contained in:
Alexander Kukushkin
2024-08-23 14:50:36 +02:00
committed by GitHub
parent 8c5ab4c07d
commit 6d65aa311a
19 changed files with 227 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
features/dcs_failsafe_mode.feature
+1 -1
View File
@@ -86,7 +86,7 @@ Feature: dcs failsafe mode
@dcs-failsafe
@slot-advance
Scenario: make sure permanent slots exist on replicas
Given I issue a PATCH request to http://127.0.0.1:8009/config with {"slots":{"postgres2":0,"dcs_slot_0":null,"dcs_slot_2":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Given I issue a PATCH request to http://127.0.0.1:8009/config with {"slots":{"dcs_slot_0":null,"dcs_slot_2":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds
When I get all changes from physical slot dcs_slot_1 on postgres1
features/nostream_node.feature
+8
View File
@@ -16,3 +16,11 @@ Scenario: check permanent logical replication slots are not copied
Then "members/postgres2" key in DCS has replication_state=streaming after 10 seconds
And postgres1 does not have a replication slot named test_logical
And postgres2 does not have a replication slot named test_logical
@slot-advance
Scenario: check that slots are written to the /status key
Given "status" key in DCS has postgres0 in slots
And "status" key in DCS has postgres2 in slots
And "status" key in DCS has test_logical in slots
And "status" key in DCS has test_logical in slots
And "status" key in DCS does not have postgres1 in slots
features/permanent_slots.feature
+15 -7
View File
@@ -3,7 +3,7 @@ Feature: permanent slots
Given I start postgres0
Then postgres0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres0":0,"postgres1":0,"postgres3":0},"postgresql":{"parameters":{"wal_level":"logical"}}}
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres3":0},"postgresql":{"parameters":{"wal_level":"logical"}}}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/config contains slots after 10 seconds
When I start postgres1
@@ -34,12 +34,14 @@ Feature: permanent slots
Scenario: check permanent physical slots that match with member names
Given postgres0 has a physical replication slot named postgres3 after 2 seconds
And postgres1 has a physical replication slot named postgres0 after 2 seconds
And postgres1 has a physical replication slot named postgres2 after 2 seconds
And postgres1 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres0 after 2 seconds
And postgres2 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres1 after 2 seconds
And postgres1 does not have a replication slot named postgres2
And postgres3 does not have a replication slot named postgres2
And postgres3 has a physical replication slot named postgres0 after 2 seconds
And postgres3 has a physical replication slot named postgres1 after 2 seconds
And postgres3 has a physical replication slot named postgres2 after 2 seconds
@slot-advance
Scenario: check that permanent slots are advanced on replicas
@@ -53,19 +55,25 @@ Feature: permanent slots
And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot postgres1 is in sync between postgres0 and postgres2 after 10 seconds
And Physical slot postgres1 is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres0 after 20 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres1 after 10 seconds
And postgres1 does not have a replication slot named postgres2
And postgres3 does not have a replication slot named postgres2
@slot-advance
Scenario: check that only permanent slots are written to the /status key
Scenario: check that permanent slots and member slots are written to the /status key
Given "status" key in DCS has test_physical in slots
And "status" key in DCS has postgres0 in slots
And "status" key in DCS has postgres1 in slots
And "status" key in DCS does not have postgres2 in slots
And "status" key in DCS has postgres2 in slots
And "status" key in DCS has postgres3 in slots
@slot-advance
Scenario: check that only non-permanent member slots are written to the retain_slots in /status key
And "status" key in DCS has postgres0 in retain_slots
And "status" key in DCS has postgres1 in retain_slots
And "status" key in DCS has postgres2 in retain_slots
And "status" key in DCS does not have postgres3 in retain_slots
Scenario: check permanent physical replication slot after failover
Given I shut down postgres3
And I shut down postgres2