Invalidate cache if txn failed due to revision mismatch (#2783)

It was reported in #2779 that the primary was constantly logging messages like `Synchronous replication key updated by someone else`.

It happened after Patroni was stuck due to resource starvation. Key updates are performed using create_revision/mod_revision field, which value is taken from the internal cached. Hence, it is a clear symptom of stale cache.

Similar issues in K8s implementation were addressed by invalidating the cache and restarting watcher connections every time when update failed due to resource_version mismatch, so we do the same for Etcd3.
This commit is contained in:
Alexander Kukushkin
2023-07-31 10:16:19 +02:00
committed by GitHub
parent 7e89583ec7
commit 8f3ed00886
2 changed files with 17 additions and 1 deletions
+7 -1
View File
@@ -127,10 +127,16 @@ class TestPatroniEtcd3Client(BaseTestEtcd3):
request = {'key': base64_encode('/patroni/test/leader')}
mock_urlopen.return_value = MockResponse()
mock_urlopen.return_value.content = '{"succeeded":true,"header":{"revision":"1"}}'
self.client.call_rpc('/kv/txn', {'success': [{'request_delete_range': request}]})
self.client.call_rpc('/kv/put', request)
self.client.call_rpc('/kv/deleterange', request)
@patch.object(urllib3.PoolManager, 'urlopen')
def test_txn(self, mock_urlopen):
mock_urlopen.return_value = MockResponse()
mock_urlopen.return_value.content = '{"header":{"revision":"1"}}'
self.client.txn({'target': 'MOD', 'mod_revision': '1'},
{'request_delete_range': {'key': base64_encode('/patroni/test/leader')}})
@patch('time.time', Mock(side_effect=[1, 10.9, 100]))
def test__wait_cache(self):
with self.kv_cache.condition: