gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 14:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Release the leader lock when pg_controldata reports "shut down" (#2067)

Browse Source 
Due to different reasons, it could happen that WAL archiving on the primary stuck or significantly delayed. If we try to do a switchover or shut it down, the shutdown will take forever and will not finish until the whole backlog of WALs is processed.
In the meantime, Patroni keeps updating the leader lock, which prevents other nodes from starting the leader race even if it is known that they received/applied all changes.

The `Database cluster state:` is changed to `"shut down"` after:
- all data is fsynced to disk and the latest checkpoint is written to WAL
- all streaming replicas confirmed that they received all changes (including the latest checkpoint)
- at the same time, the archiver process continues to do its job and the postmaster process is still running.

In order to solve this problem and make the switchover more reliable/fast in a case when `archive_command` is slow/failing, Patroni will remove the leader key immediately after `pg_controldata` started reporting PGDATA as `"shut down"` cleanly and it verified that there is at least one replica that received all changes. If there are no replicas that fulfill the condition the leader key isn't removed and the old behavior is retained, i.e. Patroni will keep updating it.
This commit is contained in:
Alexander Kukushkin
2021-10-05 10:55:35 +02:00
committed by GitHub
parent 1c2bf258d6
commit d394b63c9f
4 changed files with 92 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_postgresql.py
+15
View File
@@ -203,6 +203,21 @@ class TestPostgresql(BaseTestPostgresql):
mock_postmaster.signal_stop.side_effect = [None, True]
self.assertTrue(self.p.stop(on_safepoint=mock_callback, stop_timeout=30))
@patch('time.sleep', Mock())
@patch.object(Postgresql, 'is_running', MockPostmaster)
@patch.object(Postgresql, '_wait_for_connection_close', Mock())
@patch.object(Postgresql, 'latest_checkpoint_location', Mock(return_value='7'))
def test__do_stop(self):
mock_callback = Mock()
with patch.object(Postgresql, 'controldata', Mock(return_value={'Database cluster state': 'shut down'})):
self.assertTrue(self.p.stop(on_shutdown=mock_callback, stop_timeout=3))
mock_callback.assert_called()
with patch.object(Postgresql, 'controldata',
Mock(return_value={'Database cluster state': 'shut down in recovery'})):
self.assertTrue(self.p.stop(on_shutdown=mock_callback, stop_timeout=3))
with patch.object(Postgresql, 'controldata', Mock(return_value={'Database cluster state': 'shutting down'})):
self.assertTrue(self.p.stop(on_shutdown=mock_callback, stop_timeout=3))
def test_restart(self):
self.p.start = Mock(return_value=False)
self.assertFalse(self.p.restart())