gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 14:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Handle AuthOldRevision error (#2913)

Browse Source 
The error is raised if Etcd is configured to use JWT auth tokens and when the user database in Etcd is updated, because the update invalidates all tokens.

If retries are requested - try to get a new new token and repeat the request. Repeat it in a loop until request is successfully executed or until `retry_timeout` is exhausted. This is the only way of solving a race condition, because between authentication and executing the request yet another modification of the user database in Etcd might happen.

In case if the request doesn't have to be immediately retried - set a flag that the next API request should perform the authentication first and let Patroni to naturally repeat the request on the next heartbeat loop.

Co-authored-by: Kenny Do <[email protected]>
Ref: https://github.com/zalando/patroni/pull/2911
This commit is contained in:
Alexander Kukushkin
2023-10-23 14:00:37 +02:00
committed by GitHub
co-authored by Kenny Do
parent 6d98944e73
commit d471f1156d
2 changed files with 78 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_etcd3.py
+11 -4
View File
@@ -6,8 +6,8 @@ import urllib3
from mock import Mock, PropertyMock, patch
from patroni.dcs.etcd import DnsCachingResolver
from patroni.dcs.etcd3 import PatroniEtcd3Client, Cluster, Etcd3, Etcd3Client, \
Etcd3Error, Etcd3ClientError, RetryFailedError, InvalidAuthToken, Unavailable, \
Unknown, UnsupportedEtcdVersion, UserEmpty, AuthFailed, base64_encode
Etcd3Error, Etcd3ClientError, ReAuthenticateMode, RetryFailedError, InvalidAuthToken, Unavailable, \
Unknown, UnsupportedEtcdVersion, UserEmpty, AuthFailed, AuthOldRevision, base64_encode
from threading import Thread
from . import SleepException, MockResponse
@@ -161,9 +161,16 @@ class TestPatroniEtcd3Client(BaseTestEtcd3):
mock_urlopen.return_value.content = '{"code":16,"error":"etcdserver: invalid auth token"}'
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
with patch.object(PatroniEtcd3Client, 'authenticate', Mock(return_value=True)):
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
retry = self.etcd3._retry.copy()
with patch('time.time', Mock(side_effect=[0, 10, 20, 30, 40])):
self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
self.client.username = None
self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
retry = self.etcd3._retry.copy()
self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
mock_urlopen.return_value.content = '{"code":3,"error":"etcdserver: revision of auth store is old"}'
self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
self.assertRaises(AuthOldRevision, retry, self.client.deleteprefix, 'foo', retry=retry)
def test__handle_server_response(self):
response = MockResponse()