Citus integration (#2504)

Citus cluster (coordinator and workers) will be stored in DCS as a fleet of Patroni logically grouped together: ``` /service/batman/ /service/batman/0/ /service/batman/0/initialize /service/batman/0/leader /service/batman/0/members/ /service/batman/0/members/m1 /service/batman/0/members/m2 /service/batman/ /service/batman/1/ /service/batman/1/initialize /service/batman/1/leader /service/batman/1/members/ /service/batman/1/members/m1 /service/batman/1/members/m2 ... ``` Where 0 is a Citus group for coordinator and 1, 2, etc are worker groups. Such hierarchy allows reading the entire Citus cluster with a single call to DCS (except Zookeeper). The get_cluster() method will be reading the entire Citus cluster on the coordinator because it needs to discover workers. For the worker cluster it will be reading the subtree of its own group. Besides that we introduce a new method get_citus_coordinator(). It will be used only by worker clusters. Since there is no hierarchical structures on K8s we will use the citus group suffix on all objects that Patroni creates. E.g. ``` batman-0-leader # the leader config map for the coordinator batman-0-config # the config map holding initialize, config, and history "keys" ... batman-1-leader # the leader config map for worker group 1 batman-1-config ... ``` Citus integration is enabled from patroni.yaml: ```yaml citus: database: citus group: 0 # 0 is for coordinator, 1, 2, etc are for workers ``` If enabled, Patroni will create the database, citus extension in it, and INSERTs INTO `pg_dist_authinfo` information required for Citus nodes to communicate between each other, i.e. 'password', 'sslcert', 'sslkey' for superuser if they are defined in the Patroni configuration file. When the new Citus coordinator/worker is bootstrapped, Patroni adds `synchronous_mode: on` to the `bootstrap.dcs` section. Besides that, Patroni takes over management of some Postgres GUCs: - `shared_preload_libraries` - Patroni ensures that the "citus" is added to the first place - `max_prepared_transactions` - if not set or set to 0, Patroni changes the value to `max_connections*2` - wal_level - automatically set to logical. It is used by Citus to move/split shards. Under the hood Citus is creating/removing replication slots and they are automatically added by Patroni to the `ignore_slots` configuration to avoid accidental removal. The coordinator primary actively discovers worker primary nodes and registers/updates them in the `pg_dist_node` table using citus_add_node() and citus_update_node() functions. Patroni running on the coordinator provides the new REST API endpoint: `POST /citus`. It is used by workers to facilitate controlled switchovers and restarts of worker primaries. When the worker primary needs to shut down Postgres because of restart or switchover, it calls the `POST /citus` endpoint on the coordinator and the Patroni on the coordinator starts a transaction and calls `citus_update_node(nodeid, 'host-demoted', port)` in order to pause client connections that work with the given worker. Once the new leader is elected or postgres started back, they perform another call to the `POST/citus` endpoint, that does another `citus_update_node()` call with actual hostname and port and commits a transaction. After transaction is committed, coordinator reestablishes connections to the worker node and client connections are unblocked. If clients don't run long transaction the operation finishes without client visible errors, but only a short latency spike. All operations on the `pg_dist_node` are serialized by Patroni on the coordinator. It allows to have more control and ROLLBACK transaction in progress if its lifetime exceeding a certain threshold and there are other worker nodes should be updated.
2026-08-25 14:53:37 +00:00 · 2023-01-24 16:14:58 +01:00
parent 3161f31088
commit 4872ac51e0
54 changed files with 3611 additions and 624 deletions
@@ -0,0 +1,72 @@
+Feature: citus
+  We should check that coordinator discovers and registers workers and clients don't have errors when worker cluster switches over
+
+  Scenario: check that worker cluster is registered in the coordinator
+    Given I start postgres0 in citus group 0
+    And I start postgres2 in citus group 1
+    Then postgres0 is a leader in a group 0 after 10 seconds
+    And postgres2 is a leader in a group 1 after 10 seconds
+    When I start postgres1 in citus group 0
+    And I start postgres3 in citus group 1
+    Then replication works from postgres0 to postgres1 after 15 seconds
+    Then replication works from postgres2 to postgres3 after 15 seconds
+    And postgres0 is registered in the coordinator postgres0 as the worker in group 0
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+
+  Scenario: coordinator failover updates pg_dist_node
+    Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force
+    Then postgres1 role is the primary after 10 seconds
+    And replication works from postgres1 to postgres0 after 15 seconds
+    And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds
+    And postgres1 is registered in the coordinator postgres1 as the worker in group 0
+    When I run patronictl.py failover batman --group 0 --candidate postgres0 --force
+    Then postgres0 role is the primary after 10 seconds
+    And replication works from postgres0 to postgres1 after 15 seconds
+    And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds
+    And postgres0 is registered in the coordinator postgres0 as the worker in group 0
+
+  Scenario: worker switchover doesn't break client queries on the coordinator
+    Given I create a distributed table on postgres0
+    And I start a thread inserting data on postgres0
+    When I run patronictl.py switchover batman --group 1 --force
+    Then I receive a response returncode 0
+    And postgres3 role is the primary after 10 seconds
+    And replication works from postgres3 to postgres2 after 15 seconds
+    And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds
+    And postgres3 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I run patronictl.py switchover batman --group 1 --force
+    Then I receive a response returncode 0
+    And postgres2 role is the primary after 10 seconds
+    And replication works from postgres2 to postgres3 after 15 seconds
+    And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I stop a thread
+    Then a distributed table on postgres0 has expected rows
+
+  Scenario: worker primary restart doesn't break client queries on the coordinator
+    Given I cleanup a distributed table on postgres0
+    And I start a thread inserting data on postgres0
+    When I run patronictl.py restart batman postgres2 --group 1 --force
+    Then I receive a response returncode 0
+    And postgres2 role is the primary after 10 seconds
+    And replication works from postgres2 to postgres3 after 15 seconds
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I stop a thread
+    Then a distributed table on postgres0 has expected rows
+
+  Scenario: check that in-flight transaction is rolled back after timeout when other workers need to change pg_dist_node
+    Given I start postgres4 in citus group 2
+    Then postgres4 is a leader in a group 2 after 10 seconds
+    And "members/postgres4" key in a group 2 in DCS has role=master after 3 seconds
+    When I run patronictl.py edit-config batman --group 2 -s ttl=20 --force
+    Then I receive a response returncode 0
+    And I receive a response output "+ttl: 20"
+    When I sleep for 2 seconds
+    Then postgres4 is registered in the coordinator postgres0 as the worker in group 2
+    When I shut down postgres4
+    Then There is a transaction in progress on postgres0 changing pg_dist_node
+    When I run patronictl.py restart batman postgres2 --group 1 --force
+    Then a transaction finishes in 20 seconds
@@ -74,8 +74,8 @@ Feature: dcs failsafe mode
  Scenario: check three-node cluster is functioning while DCS is down
    Given I start postgres0
    And I start postgres2
-    Then "members/postgres0" key in DCS has state=running after 10 seconds
-    And "members/postgres2" key in DCS has state=running after 10 seconds
+    Then "members/postgres2" key in DCS has state=running after 10 seconds
+    And "members/postgres0" key in DCS has state=running after 20 seconds
    And Response on GET http://127.0.0.1:8008/failsafe contains postgres2 after 10 seconds
    And replication works from postgres1 to postgres0 after 10 seconds
    Given DCS is down
@@ -102,6 +102,7 @@ class PatroniController(AbstractController):
            self.watchdog = None

        self._scope = (custom_config or {}).get('scope', 'batman')
+        self._citus_group = (custom_config or {}).get('citus', {}).get('group')
        self._config = self._make_patroni_test_config(name, custom_config)
        self._closables = []

@@ -143,7 +144,7 @@ class PatroniController(AbstractController):
            self.watchdog.start()
        env = os.environ.copy()
        if isinstance(self._context.dcs_ctl, KubernetesController):
-            self._context.dcs_ctl.create_pod(self._name[8:], self._scope)
+            self._context.dcs_ctl.create_pod(self._name[8:], self._scope, self._citus_group)
            env['PATRONI_KUBERNETES_POD_IP'] = '10.0.0.' + self._name[-1]
        if os.name == 'nt':
            env['BEHAVE_DEBUG'] = 'true'
@@ -385,6 +386,10 @@ class AbstractDcsController(AbstractController):
        if self._work_directory:
            shutil.rmtree(self._work_directory)

+    def path(self, key=None, scope='batman', group=None):
+        citus_group = '/{0}'.format(group) if group is not None else ''
+        return self._CLUSTER_NODE.format(scope) + citus_group + (key and '/' + key or '')
+
    def start_outage(self):
        if not self._paused and self._handle:
            self._handle.suspend()
@@ -395,11 +400,8 @@ class AbstractDcsController(AbstractController):
            self._handle.resume()
            self._paused = False

-    def path(self, key=None, scope='batman'):
-        return self._CLUSTER_NODE.format(scope) + (key and '/' + key or '')
-
    @abc.abstractmethod
-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        """ query for a value of a given key """

    @abc.abstractmethod
@@ -447,11 +449,11 @@ class ConsulController(AbstractDcsController):
        except Exception:
            return False

-    def path(self, key=None, scope='batman'):
-        return super(ConsulController, self).path(key, scope)[1:]
+    def path(self, key=None, scope='batman', group=None):
+        return super(ConsulController, self).path(key, scope, group)[1:]

-    def query(self, key, scope='batman'):
-        _, value = self._client.kv.get(self.path(key, scope))
+    def query(self, key, scope='batman', group=None):
+        _, value = self._client.kv.get(self.path(key, scope, group))
        return value and value['Value'].decode('utf-8')

    def cleanup_service_tree(self):
@@ -491,10 +493,10 @@ class EtcdController(AbstractEtcdController):
        super(EtcdController, self).__init__(context, EtcdClient)
        os.environ['PATRONI_ETCD_HOST'] = 'localhost:2379'

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import etcd
        try:
-            return self._client.get(self.path(key, scope)).value
+            return self._client.get(self.path(key, scope, group)).value
        except etcd.EtcdKeyNotFound:
            return None

@@ -515,9 +517,9 @@ class Etcd3Controller(AbstractEtcdController):
        super(Etcd3Controller, self).__init__(context, Etcd3Client)
        os.environ['PATRONI_ETCD3_HOST'] = 'localhost:2379'

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import base64
-        response = self._client.range(self.path(key, scope))
+        response = self._client.range(self.path(key, scope, group))
        for k in response.get('kvs', []):
            return base64.b64decode(k['value']).decode('utf-8') if 'value' in k else None

@@ -609,10 +611,12 @@ class KubernetesController(AbstractExternalDcsController):
            return False
        return True

-    def create_pod(self, name, scope):
+    def create_pod(self, name, scope, group=None):
        self.delete_pod(name)
        labels = self._labels.copy()
        labels['cluster-name'] = scope
+        if group is not None:
+            labels['citus-group'] = str(group)
        metadata = self._client.V1ObjectMeta(namespace=self._namespace, name=name, labels=labels)
        spec = self._client.V1PodSpec(containers=[self._client.V1Container(name=name, image='empty')])
        body = self._client.V1Pod(metadata=metadata, spec=spec)
@@ -629,12 +633,14 @@ class KubernetesController(AbstractExternalDcsController):
            except Exception:
                break

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        if key.startswith('members/'):
            pod = self._api.read_namespaced_pod(key[8:], self._namespace)
            return (pod.metadata.annotations or {}).get('status', '')
        else:
            try:
+                if group is not None:
+                    scope = '{0}-{1}'.format(scope, group)
                ep = scope + {'leader': '', 'history': '-config', 'initialize': '-config'}.get(key, '-' + key)
                e = self._api.read_namespaced_endpoints(ep, self._namespace)
                if key != 'sync':
@@ -675,10 +681,10 @@ class ZooKeeperController(AbstractExternalDcsController):
    def process_name(self):
        return "zookeeper"

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import kazoo.exceptions
        try:
-            return self._client.get(self.path(key, scope))[0].decode('utf-8')
+            return self._client.get(self.path(key, scope, group))[0].decode('utf-8')
        except kazoo.exceptions.NoNodeError:
            return None

@@ -748,8 +754,8 @@ class RaftController(AbstractDcsController):
                             '--source=patroni', '-p', 'patroni_raft_controller.py'],
                            stdout=self._log, stderr=subprocess.STDOUT, env=env)

-    def query(self, key, scope='batman'):
-        ret = self._raft.get(self.path(key, scope))
+    def query(self, key, scope='batman', group=None):
+        ret = self._raft.get(self.path(key, scope, group))
        return ret and ret['value']

    def set(self, key, value):
@@ -1087,9 +1093,12 @@ def after_all(context):
 def before_feature(context, feature):
    """ create per-feature output directory to collect Patroni and PostgreSQL logs """
    if feature.name == 'watchdog' and os.name == 'nt':
-        feature.skip("Watchdog isn't supported on Windows")
-    else:
-        context.pctl.create_and_set_output_directory(feature.name)
+        return feature.skip("Watchdog isn't supported on Windows")
+    elif feature.name == 'citus':
+        lib = subprocess.check_output(['pg_config', '--pkglibdir']).decode('utf-8').strip()
+        if not os.path.exists(os.path.join(lib, 'citus.so')):
+            return feature.skip("Citus extenstion isn't available")
+    context.pctl.create_and_set_output_directory(feature.name)


 def after_feature(context, feature):
@@ -14,9 +14,9 @@ Scenario: check API requests on a stand-alone server
 	Then I receive a response code 200
 	When I issue a GET request to http://127.0.0.1:8008/replica
 	Then I receive a response code 503
-	When I run patronictl.py reinit batman postgres0 --force
-	Then I receive a response returncode 0
-	And I receive a response output "Failed: reinitialize for member postgres0, status code=503, (I am the leader, can not reinitialize)"
+	When I issue a POST request to http://127.0.0.1:8008/reinitialize with {"force": true}
+	Then I receive a response code 503
+	And I receive a response text I am the leader, can not reinitialize
 	When I run patronictl.py switchover batman --master postgres0 --force
 	Then I receive a response returncode 1
 	And I receive a response output "Error: No candidates found to switchover to"
@@ -0,0 +1,117 @@
+import json
+import time
+
+from behave import step, then
+from dateutil import tz
+from datetime import datetime
+from functools import partial
+from threading import Thread, Event
+
+tzutc = tz.tzutc()
+
+
+@step('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
+@then('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
+def is_a_group_leader(context, name, group, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    while (context.dcs_ctl.query("leader", group=group) != name):
+        time.sleep(1)
+        assert time.time() < max_time, "{0} is not a leader in dcs after {1} seconds".format(name, time_limit)
+
+
+@step('"{name}" key in a group {group:d} in DCS has {key:w}={value} after {time_limit:d} seconds')
+def check_group_member(context, name, group, key, value, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    dcs_value = None
+    response = None
+    while time.time() < max_time:
+        try:
+            response = json.loads(context.dcs_ctl.query(name, group=group))
+            dcs_value = response.get(key)
+            if dcs_value == value:
+                return
+        except Exception:
+            pass
+        time.sleep(1)
+    assert False, ("{0} in a group {1} does not have {2}={3} (found {4}) in dcs" +
+                   " after {5} seconds").format(name, group, key, value, response, time_limit)
+
+
+@step('I start {name:w} in citus group {group:d}')
+def start_citus(context, name, group):
+    return context.pctl.start(name, custom_config={"citus": {"database": "postgres", "group": int(group)}})
+
+
+@step('{name1:w} is registered in the coordinator {name2:w} as the worker in group {group:d}')
+def check_registration(context, name1, name2, group):
+    worker_port = int(context.pctl.query(name1, "SHOW port").fetchone()[0])
+    r = context.pctl.query(name2, "SELECT nodeport FROM pg_catalog.pg_dist_node WHERE groupid = {0}".format(group))
+    assert worker_port == r.fetchone()[0],\
+        "Worker {0} is not registered in pg_dist_node on the coordinator {1}".format(name1, name2)
+
+
+@step('I create a distributed table on {name:w}')
+def create_distributed_table(context, name):
+    context.pctl.query(name, 'CREATE TABLE public.d(id int not null)')
+    context.pctl.query(name, "SELECT create_distributed_table('public.d', 'id')")
+
+
+@step('I cleanup a distributed table on {name:w}')
+def cleanup_distributed_table(context, name):
+    context.pctl.query(name, 'TRUNCATE public.d')
+
+
+def insert_thread(query_func, context):
+    while True:
+        if context.thread_stop_event.is_set():
+            break
+
+        context.insert_counter += 1
+        query_func('INSERT INTO public.d VALUES({0})'.format(context.insert_counter))
+
+        context.thread_stop_event.wait(0.01)
+
+
+@step('I start a thread inserting data on {name:w}')
+def start_insert_thread(context, name):
+    context.thread_stop_event = Event()
+    context.insert_counter = 0
+    query_func = partial(context.pctl.query, name)
+    thread_func = partial(insert_thread, query_func, context)
+    context.thread = Thread(target=thread_func)
+    context.thread.daemon = True
+    context.thread.start()
+
+
+@then('a thread is still alive')
+def thread_is_alive(context):
+    assert context.thread.is_alive(), "Thread is not alive"
+
+
+@step("I stop a thread")
+def stop_insert_thread(context):
+    context.thread_stop_event.set()
+    context.thread.join(1*context.timeout_multiplier)
+    assert not context.thread.is_alive(), "Thread is still alive"
+
+
+@step("a distributed table on {name:w} has expected rows")
+def count_rows(context, name):
+    rows = context.pctl.query(name, "SELECT COUNT(*) FROM public.d").fetchone()[0]
+    assert rows == context.insert_counter, "Distributed table doesn't have expected amount of rows"
+
+
+@step("There is a transaction in progress on {name:w} changing pg_dist_node")
+def check_transaction(context, name):
+    cur = context.pctl.query(name, "SELECT xact_start FROM pg_stat_activity WHERE pid <> pg_backend_pid() AND state"
+                                   " = 'idle in transaction' AND query ~ 'citus_update_node' AND query ~ 'demoted'")
+    assert cur.rowcount == 1, "There is no idle in transaction updating pg_dist_node"
+    context.xact_start = cur.fetchone()[0]
+
+
+@step("a transaction finishes in {timeout:d} seconds")
+def check_transaction_timeout(context, timeout):
+    assert (datetime.now(tzutc) - context.xact_start).seconds > timeout,\
+            "a transaction finished earlier than in {0} seconds".format(timeout)