gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 14:53:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
4872ac51e00abefc98678f1771dd93aead59f4ce
patroni/features/patroni_api.feature
T
Alexander KukushkinandGitHub 4872ac51e0 Citus integration (#2504) 
Citus cluster (coordinator and workers) will be stored in DCS as a fleet of Patroni logically grouped together:
```
/service/batman/
/service/batman/0/
/service/batman/0/initialize
/service/batman/0/leader
/service/batman/0/members/
/service/batman/0/members/m1
/service/batman/0/members/m2
/service/batman/
/service/batman/1/
/service/batman/1/initialize
/service/batman/1/leader
/service/batman/1/members/
/service/batman/1/members/m1
/service/batman/1/members/m2
...
```

Where 0 is a Citus group for coordinator and 1, 2, etc are worker groups.

Such hierarchy allows reading the entire Citus cluster with a single call to DCS (except Zookeeper).

The get_cluster() method will be reading the entire Citus cluster on the coordinator because it needs to discover workers. For the worker cluster it will be reading the subtree of its own group.

Besides that we introduce a new method  get_citus_coordinator(). It will be used only by worker clusters.

Since there is no hierarchical structures on K8s we will use the citus group suffix on all objects that Patroni creates.
E.g.
```
batman-0-leader  # the leader config map for the coordinator
batman-0-config  # the config map holding initialize, config, and history "keys"
...
batman-1-leader  # the leader config map for worker group 1
batman-1-config
...
```

Citus integration is enabled from patroni.yaml:
```yaml
citus:
  database: citus
  group: 0  # 0 is for coordinator, 1, 2, etc are for workers
```

If enabled, Patroni will create the database, citus extension in it, and INSERTs INTO `pg_dist_authinfo` information required for Citus nodes to communicate between each other, i.e. 'password', 'sslcert', 'sslkey' for superuser if they are defined in the Patroni configuration file.

When the new Citus coordinator/worker is bootstrapped, Patroni adds `synchronous_mode: on` to the `bootstrap.dcs` section.

Besides that, Patroni takes over management of some Postgres GUCs:
- `shared_preload_libraries` - Patroni ensures that the "citus" is added to the first place
- `max_prepared_transactions` - if not set or set to 0, Patroni changes the value to `max_connections*2`
- wal_level - automatically set to logical. It is used by Citus to move/split shards. Under the hood Citus is creating/removing replication slots and they are automatically added by Patroni to the `ignore_slots` configuration to avoid accidental removal.

The coordinator primary actively discovers worker primary nodes and registers/updates them in the `pg_dist_node` table using
citus_add_node() and citus_update_node() functions.

Patroni running on the coordinator provides the new REST API endpoint: `POST /citus`. It is used by workers to facilitate controlled switchovers and restarts of worker primaries.
When the worker primary needs to shut down Postgres because of restart or switchover, it calls the `POST /citus` endpoint on the coordinator and the Patroni on the coordinator starts a transaction and calls `citus_update_node(nodeid, 'host-demoted', port)` in order to pause client connections that work with the given worker.
Once the new leader is elected or postgres started back, they perform another call to the `POST/citus` endpoint, that does another `citus_update_node()` call with actual hostname and port and commits a transaction. After transaction is committed, coordinator reestablishes connections to the worker node and client connections are unblocked.
If clients don't run long transaction the operation finishes without client visible errors, but only a short latency spike.

All operations on the `pg_dist_node` are serialized by Patroni on the coordinator. It allows to have more control and ROLLBACK transaction in progress if its lifetime exceeding a certain threshold and there are other worker nodes should be updated.
2023-01-24 16:14:58 +01:00

127 lines
6.4 KiB
Gherkin
Raw Blame History

Feature: patroni api
We should check that patroni correctly responds to valid and not-valid API requests.
Scenario: check API requests on a stand-alone server
Given I start postgres0
And postgres0 is a leader after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/
Then I receive a response code 200
And I receive a response state running
And I receive a response role master
When I issue a GET request to http://127.0.0.1:8008/standby_leader
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8008/health
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8008/replica
Then I receive a response code 503
When I issue a POST request to http://127.0.0.1:8008/reinitialize with {"force": true}
Then I receive a response code 503
And I receive a response text I am the leader, can not reinitialize
When I run patronictl.py switchover batman --master postgres0 --force
Then I receive a response returncode 1
And I receive a response output "Error: No candidates found to switchover to"
When I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0"}
Then I receive a response code 412
And I receive a response text switchover is not possible: cluster does not have members except leader
When I issue an empty POST request to http://127.0.0.1:8008/failover
Then I receive a response code 400
When I issue a POST request to http://127.0.0.1:8008/failover with {"foo": "bar"}
Then I receive a response code 400
And I receive a response text "Failover could be performed only to a specific candidate"
Scenario: check local configuration reload
Given I add tag new_tag new_value to postgres0 config
And I issue an empty POST request to http://127.0.0.1:8008/reload
Then I receive a response code 202
Scenario: check dynamic configuration change via DCS
Given I run patronictl.py edit-config -s 'ttl=10' -p 'max_connections=101' --force batman
Then I receive a response returncode 0
And I receive a response output "+ttl: 10"
And Response on GET http://127.0.0.1:8008/patroni contains pending_restart after 11 seconds
When I issue a GET request to http://127.0.0.1:8008/config
Then I receive a response code 200
And I receive a response ttl 10
When I issue a GET request to http://127.0.0.1:8008/patroni
Then I receive a response code 200
And I receive a response tags {'new_tag': 'new_value'}
And I sleep for 4 seconds
Scenario: check the scheduled restart
Given I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"superuser_reserved_connections": "6"}}}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/patroni contains pending_restart after 5 seconds
Given I issue a scheduled restart at http://127.0.0.1:8008 in 5 seconds with {"role": "replica"}
Then I receive a response code 202
And I sleep for 8 seconds
And Response on GET http://127.0.0.1:8008/patroni contains pending_restart after 10 seconds
Given I issue a scheduled restart at http://127.0.0.1:8008 in 5 seconds with {"restart_pending": "True"}
Then I receive a response code 202
And Response on GET http://127.0.0.1:8008/patroni does not contain pending_restart after 10 seconds
And postgres0 role is the primary after 10 seconds
Scenario: check API requests for the primary-replica pair in the pause mode
Given I start postgres1
Then replication works from postgres0 to postgres1 after 20 seconds
When I run patronictl.py pause batman
Then I receive a response returncode 0
When I kill postmaster on postgres1
And I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 503
When I run patronictl.py restart batman postgres1 --force
Then I receive a response returncode 0
Then replication works from postgres0 to postgres1 after 20 seconds
And I sleep for 2 seconds
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 200
And I receive a response state running
And I receive a response role replica
When I run patronictl.py reinit batman postgres1 --force
Then I receive a response returncode 0
And I receive a response output "Success: reinitialize for member postgres1"
When I run patronictl.py restart batman postgres0 --force
Then I receive a response returncode 0
And I receive a response output "Success: restart on member postgres0"
And postgres0 role is the primary after 5 seconds
When I sleep for 10 seconds
Then postgres1 role is the secondary after 15 seconds
Scenario: check the switchover via the API in the pause mode
Given I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0", "candidate": "postgres1"}
Then I receive a response code 200
And postgres1 is a leader after 5 seconds
And postgres1 role is the primary after 10 seconds
And postgres0 role is the secondary after 10 seconds
And replication works from postgres1 to postgres0 after 20 seconds
And "members/postgres0" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/master
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8008/replica
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8009/master
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 503
Scenario: check the scheduled switchover
Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds
Then I receive a response returncode 1
And I receive a response output "Can't schedule switchover in the paused state"
When I run patronictl.py resume batman
Then I receive a response returncode 0
Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds
Then I receive a response returncode 0
And postgres0 is a leader after 20 seconds
And postgres0 role is the primary after 10 seconds
And postgres1 role is the secondary after 10 seconds
And replication works from postgres0 to postgres1 after 25 seconds
And "members/postgres1" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/master
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8008/replica
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8009/master
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 200
Reference in New Issue View Git Blame Copy Permalink