gitops/patroni
1
0
Fork 0
mirror of https://github.com/outbackdingo/patroni.git synced 2026-08-25 07:20:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Use names with "unusual" symbols in behave tests (#3162)

Browse Source 
It'll hopefully prevent problems like #3142 in future.
This commit is contained in:
Alexander Kukushkin
2024-09-16 09:35:22 +02:00
committed by GitHub
parent 94a592d275
commit 416a0f7c8b
24 changed files with 503 additions and 492 deletions
Download Patch File Download Diff File Expand all files Collapse all files
features/basic_replication.feature
+40 -40
View File
@@ -2,57 +2,57 @@ Feature: basic replication
We should check that the basic bootstrapping, replication and failover works.
Scenario: check replication of a single table
Given I start postgres0
Then postgres0 is a leader after 10 seconds
Given I start postgres-0
Then postgres-0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "synchronous_mode": true}
Then I receive a response code 200
When I start postgres1
And I configure and start postgres2 with a tag replicatefrom postgres0
And "sync" key in DCS has leader=postgres0 after 20 seconds
And I add the table foo to postgres0
Then table foo is present on postgres1 after 20 seconds
Then table foo is present on postgres2 after 20 seconds
When I start postgres-1
And I configure and start postgres-2 with a tag replicatefrom postgres-0
And "sync" key in DCS has leader=postgres-0 after 20 seconds
And I add the table foo to postgres-0
Then table foo is present on postgres-1 after 20 seconds
Then table foo is present on postgres-2 after 20 seconds
Scenario: check restart of sync replica
Given I shut down postgres2
Then "sync" key in DCS has sync_standby=postgres1 after 5 seconds
When I start postgres2
And I shut down postgres1
Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds
When I start postgres1
Then "members/postgres1" key in DCS has state=running after 10 seconds
Given I shut down postgres-2
Then "sync" key in DCS has sync_standby=postgres-1 after 5 seconds
When I start postgres-2
And I shut down postgres-1
Then "sync" key in DCS has sync_standby=postgres-2 after 10 seconds
When I start postgres-1
Then "members/postgres-1" key in DCS has state=running after 10 seconds
And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds
Scenario: check stuck sync replica
Given I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": true, "maximum_lag_on_syncnode": 15000000, "postgresql": {"parameters": {"synchronous_commit": "remote_apply"}}}
Then I receive a response code 200
And I create table on postgres0
And table mytest is present on postgres1 after 2 seconds
And table mytest is present on postgres2 after 2 seconds
When I pause wal replay on postgres2
And I load data on postgres0
Then "sync" key in DCS has sync_standby=postgres1 after 15 seconds
And I resume wal replay on postgres2
And I create table on postgres-0
And table mytest is present on postgres-1 after 2 seconds
And table mytest is present on postgres-2 after 2 seconds
When I pause wal replay on postgres-2
And I load data on postgres-0
Then "sync" key in DCS has sync_standby=postgres-1 after 15 seconds
And I resume wal replay on postgres-2
And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds
And Status code on GET http://127.0.0.1:8010/async is 200 after 3 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": null, "maximum_lag_on_syncnode": -1, "postgresql": {"parameters": {"synchronous_commit": "on"}}}
Then I receive a response code 200
And I drop table on postgres0
And I drop table on postgres-0
Scenario: check multi sync replication
Given I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 2}
Then I receive a response code 200
Then "sync" key in DCS has sync_standby=postgres1,postgres2 after 10 seconds
Then "sync" key in DCS has sync_standby=postgres-1,postgres-2 after 10 seconds
And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 1}
Then I receive a response code 200
And I shut down postgres1
Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds
When I start postgres1
Then "members/postgres1" key in DCS has state=running after 10 seconds
And I shut down postgres-1
Then "sync" key in DCS has sync_standby=postgres-2 after 10 seconds
When I start postgres-1
Then "members/postgres-1" key in DCS has state=running after 10 seconds
And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds
@@ -60,26 +60,26 @@ Feature: basic replication
Given I run patronictl.py pause batman
Then I receive a response returncode 0
When I sleep for 2 seconds
And I shut down postgres0
And I shut down postgres-0
And I run patronictl.py resume batman
Then I receive a response returncode 0
And postgres2 role is the primary after 24 seconds
And postgres-2 role is the primary after 24 seconds
And Response on GET http://127.0.0.1:8010/history contains recovery after 10 seconds
And there is a postgres2_cb.log with "on_role_change primary batman" in postgres2 data directory
And there is a postgres-2_cb.log with "on_role_change primary batman" in postgres-2 data directory
When I issue a PATCH request to http://127.0.0.1:8010/config with {"synchronous_mode": null, "master_start_timeout": 0}
Then I receive a response code 200
When I add the table bar to postgres2
Then table bar is present on postgres1 after 20 seconds
When I add the table bar to postgres-2
Then table bar is present on postgres-1 after 20 seconds
And Response on GET http://127.0.0.1:8010/config contains master_start_timeout after 10 seconds
Scenario: check rejoin of the former primary with pg_rewind
Given I add the table splitbrain to postgres0
And I start postgres0
Then postgres0 role is the secondary after 20 seconds
When I add the table buz to postgres2
Then table buz is present on postgres0 after 20 seconds
Given I add the table splitbrain to postgres-0
And I start postgres-0
Then postgres-0 role is the secondary after 20 seconds
When I add the table buz to postgres-2
Then table buz is present on postgres-0 after 20 seconds
@reject-duplicate-name
Scenario: check graceful rejection when two nodes have the same name
Given I start duplicate postgres0 on port 8011
Then there is one of ["Can't start; there is already a node named 'postgres0' running"] CRITICAL in the dup-postgres0 patroni log after 5 seconds
Given I start duplicate postgres-0 on port 8011
Then there is one of ["Can't start; there is already a node named 'postgres-0' running"] CRITICAL in the dup-postgres-0 patroni log after 5 seconds
features/cascading_replication.feature
+10 -10
View File
@@ -2,13 +2,13 @@ Feature: cascading replication
We should check that patroni can do base backup and streaming from the replica
Scenario: check a base backup and streaming replication from a replica
Given I start postgres0
And postgres0 is a leader after 10 seconds
And I configure and start postgres1 with a tag clonefrom true
And replication works from postgres0 to postgres1 after 20 seconds
And I create label with "postgres0" in postgres0 data directory
And I create label with "postgres1" in postgres1 data directory
And "members/postgres1" key in DCS has state=running after 12 seconds
And I configure and start postgres2 with a tag replicatefrom postgres1
Then replication works from postgres0 to postgres2 after 30 seconds
And there is a label with "postgres1" in postgres2 data directory
Given I start postgres-0
And postgres-0 is a leader after 10 seconds
And I configure and start postgres-1 with a tag clonefrom true
And replication works from postgres-0 to postgres-1 after 20 seconds
And I create label with "postgres-0" in postgres-0 data directory
And I create label with "postgres-1" in postgres-1 data directory
And "members/postgres-1" key in DCS has state=running after 12 seconds
And I configure and start postgres-2 with a tag replicatefrom postgres-1
Then replication works from postgres-0 to postgres-2 after 30 seconds
And there is a label with "postgres-1" in postgres-2 data directory
features/citus.feature
+54 -54
View File
@@ -2,79 +2,79 @@ Feature: citus
We should check that coordinator discovers and registers workers and clients don't have errors when worker cluster switches over
Scenario: check that worker cluster is registered in the coordinator
Given I start postgres0 in citus group 0
And I start postgres2 in citus group 1
Then postgres0 is a leader in a group 0 after 10 seconds
And postgres2 is a leader in a group 1 after 10 seconds
When I start postgres1 in citus group 0
And I start postgres3 in citus group 1
Then replication works from postgres0 to postgres1 after 15 seconds
Then replication works from postgres2 to postgres3 after 15 seconds
And postgres0 is registered in the postgres0 as the primary in group 0 after 5 seconds
And postgres1 is registered in the postgres0 as the secondary in group 0 after 5 seconds
And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds
And postgres3 is registered in the postgres0 as the secondary in group 1 after 5 seconds
Given I start postgres-0 in citus group 0
And I start postgres-2 in citus group 1
Then postgres-0 is a leader in a group 0 after 10 seconds
And postgres-2 is a leader in a group 1 after 10 seconds
When I start postgres-1 in citus group 0
And I start postgres-3 in citus group 1
Then replication works from postgres-0 to postgres-1 after 15 seconds
Then replication works from postgres-2 to postgres-3 after 15 seconds
And postgres-0 is registered in the postgres-0 as the primary in group 0 after 5 seconds
And postgres-1 is registered in the postgres-0 as the secondary in group 0 after 5 seconds
And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds
And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 5 seconds
Scenario: coordinator failover updates pg_dist_node
Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force
Then postgres1 role is the primary after 10 seconds
And "members/postgres0" key in a group 0 in DCS has state=running after 15 seconds
And replication works from postgres1 to postgres0 after 15 seconds
And postgres1 is registered in the postgres2 as the primary in group 0 after 5 seconds
And postgres0 is registered in the postgres2 as the secondary in group 0 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds
When I run patronictl.py switchover batman --group 0 --candidate postgres0 --force
Then postgres0 role is the primary after 10 seconds
And replication works from postgres0 to postgres1 after 15 seconds
And postgres0 is registered in the postgres2 as the primary in group 0 after 5 seconds
And postgres1 is registered in the postgres2 as the secondary in group 0 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds
Given I run patronictl.py failover batman --group 0 --candidate postgres-1 --force
Then postgres-1 role is the primary after 10 seconds
And "members/postgres-0" key in a group 0 in DCS has state=running after 15 seconds
And replication works from postgres-1 to postgres-0 after 15 seconds
And postgres-1 is registered in the postgres-2 as the primary in group 0 after 5 seconds
And postgres-0 is registered in the postgres-2 as the secondary in group 0 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres-0 after 15 seconds
When I run patronictl.py switchover batman --group 0 --candidate postgres-0 --force
Then postgres-0 role is the primary after 10 seconds
And replication works from postgres-0 to postgres-1 after 15 seconds
And postgres-0 is registered in the postgres-2 as the primary in group 0 after 5 seconds
And postgres-1 is registered in the postgres-2 as the secondary in group 0 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres-1 after 15 seconds
Scenario: worker switchover doesn't break client queries on the coordinator
Given I create a distributed table on postgres0
And I start a thread inserting data on postgres0
Given I create a distributed table on postgres-0
And I start a thread inserting data on postgres-0
When I run patronictl.py switchover batman --group 1 --force
Then I receive a response returncode 0
And postgres3 role is the primary after 10 seconds
And "members/postgres2" key in a group 1 in DCS has state=running after 15 seconds
And replication works from postgres3 to postgres2 after 15 seconds
And postgres3 is registered in the postgres0 as the primary in group 1 after 5 seconds
And postgres2 is registered in the postgres0 as the secondary in group 1 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds
And postgres-3 role is the primary after 10 seconds
And "members/postgres-2" key in a group 1 in DCS has state=running after 15 seconds
And replication works from postgres-3 to postgres-2 after 15 seconds
And postgres-3 is registered in the postgres-0 as the primary in group 1 after 5 seconds
And postgres-2 is registered in the postgres-0 as the secondary in group 1 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres-2 after 15 seconds
And a thread is still alive
When I run patronictl.py switchover batman --group 1 --force
Then I receive a response returncode 0
And postgres2 role is the primary after 10 seconds
And replication works from postgres2 to postgres3 after 15 seconds
And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds
And postgres3 is registered in the postgres0 as the secondary in group 1 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds
And postgres-2 role is the primary after 10 seconds
And replication works from postgres-2 to postgres-3 after 15 seconds
And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds
And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres-3 after 15 seconds
And a thread is still alive
When I stop a thread
Then a distributed table on postgres0 has expected rows
Then a distributed table on postgres-0 has expected rows
Scenario: worker primary restart doesn't break client queries on the coordinator
Given I cleanup a distributed table on postgres0
And I start a thread inserting data on postgres0
When I run patronictl.py restart batman postgres2 --group 1 --force
Given I cleanup a distributed table on postgres-0
And I start a thread inserting data on postgres-0
When I run patronictl.py restart batman postgres-2 --group 1 --force
Then I receive a response returncode 0
And postgres2 role is the primary after 10 seconds
And replication works from postgres2 to postgres3 after 15 seconds
And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds
And postgres3 is registered in the postgres0 as the secondary in group 1 after 15 seconds
And postgres-2 role is the primary after 10 seconds
And replication works from postgres-2 to postgres-3 after 15 seconds
And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds
And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 15 seconds
And a thread is still alive
When I stop a thread
Then a distributed table on postgres0 has expected rows
Then a distributed table on postgres-0 has expected rows
Scenario: check that in-flight transaction is rolled back after timeout when other workers need to change pg_dist_node
Given I start postgres4 in citus group 2
Then postgres4 is a leader in a group 2 after 10 seconds
And "members/postgres4" key in a group 2 in DCS has role=primary after 3 seconds
Given I start postgres-4 in citus group 2
Then postgres-4 is a leader in a group 2 after 10 seconds
And "members/postgres-4" key in a group 2 in DCS has role=primary after 3 seconds
When I run patronictl.py edit-config batman --group 2 -s ttl=20 --force
Then I receive a response returncode 0
And I receive a response output "+ttl: 20"
Then postgres4 is registered in the postgres2 as the primary in group 2 after 5 seconds
When I shut down postgres4
Then there is a transaction in progress on postgres0 changing pg_dist_node after 5 seconds
When I run patronictl.py restart batman postgres2 --group 1 --force
Then postgres-4 is registered in the postgres-2 as the primary in group 2 after 5 seconds
When I shut down postgres-4
Then there is a transaction in progress on postgres-0 changing pg_dist_node after 5 seconds
When I run patronictl.py restart batman postgres-2 --group 1 --force
Then a transaction finishes in 20 seconds
features/custom_bootstrap.feature
+11 -11
View File
@@ -2,16 +2,16 @@ Feature: custom bootstrap
We should check that patroni can bootstrap a new cluster from a backup
Scenario: clone existing cluster using pg_basebackup
Given I start postgres0
Then postgres0 is a leader after 10 seconds
When I add the table foo to postgres0
And I start postgres1 in a cluster batman1 as a clone of postgres0
Then postgres1 is a leader of batman1 after 10 seconds
Then table foo is present on postgres1 after 10 seconds
Given I start postgres-0
Then postgres-0 is a leader after 10 seconds
When I add the table foo to postgres-0
And I start postgres-1 in a cluster batman1 as a clone of postgres-0
Then postgres-1 is a leader of batman1 after 10 seconds
Then table foo is present on postgres-1 after 10 seconds
Scenario: make a backup and do a restore into a new cluster
Given I add the table bar to postgres1
And I do a backup of postgres1
When I start postgres2 in a cluster batman2 from backup
Then postgres2 is a leader of batman2 after 30 seconds
And table bar is present on postgres2 after 10 seconds
Given I add the table bar to postgres-1
And I do a backup of postgres-1
When I start postgres-2 in a cluster batman2 from backup
Then postgres-2 is a leader of batman2 after 30 seconds
And table bar is present on postgres-2 after 10 seconds
features/dcs_failsafe_mode.feature
+62 -62
View File
@@ -2,16 +2,16 @@ Feature: dcs failsafe mode
We should check the basic dcs failsafe mode functioning
Scenario: check failsafe mode can be successfully enabled
Given I start postgres0
And postgres0 is a leader after 10 seconds
Given I start postgres-0
And postgres-0 is a leader after 10 seconds
Then "config" key in DCS has ttl=30 after 10 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 3, "failsafe_mode": true}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds
And Response on GET http://127.0.0.1:8008/failsafe contains postgres-0 after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/failsafe
Then I receive a response code 200
And I receive a response postgres0 http://127.0.0.1:8008/patroni
When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}},"slots":{"dcs_slot_1": null,"postgres0":null}}
And I receive a response postgres-0 http://127.0.0.1:8008/patroni
When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}},"slots":{"dcs_slot_1": null,"postgres_0":null}}
Then I receive a response code 200
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"dcs_slot_0": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
Then I receive a response code 200
@@ -20,99 +20,99 @@ Feature: dcs failsafe mode
Scenario: check one-node cluster is functioning while DCS is down
Given DCS is down
Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
And postgres0 role is the primary after 10 seconds
And postgres-0 role is the primary after 10 seconds
@dcs-failsafe
Scenario: check new replica isn't promoted when leader is down and DCS is up
Given DCS is up
When I do a backup of postgres0
And I shut down postgres0
When I start postgres1 in a cluster batman from backup with no_leader
Then postgres1 role is the replica after 12 seconds
When I do a backup of postgres-0
And I shut down postgres-0
When I start postgres-1 in a cluster batman from backup with no_leader
Then postgres-1 role is the replica after 12 seconds
Scenario: check leader and replica are both in /failsafe key after leader is back
Given I start postgres0
And I start postgres1
Then "members/postgres0" key in DCS has state=running after 10 seconds
And "members/postgres1" key in DCS has state=running after 2 seconds
And Response on GET http://127.0.0.1:8009/failsafe contains postgres1 after 10 seconds
Given I start postgres-0
And I start postgres-1
Then "members/postgres-0" key in DCS has state=running after 10 seconds
And "members/postgres-1" key in DCS has state=running after 2 seconds
And Response on GET http://127.0.0.1:8009/failsafe contains postgres-1 after 10 seconds
When I issue a GET request to http://127.0.0.1:8009/failsafe
Then I receive a response code 200
And I receive a response postgres0 http://127.0.0.1:8008/patroni
And I receive a response postgres1 http://127.0.0.1:8009/patroni
And I receive a response postgres-0 http://127.0.0.1:8008/patroni
And I receive a response postgres-1 http://127.0.0.1:8009/patroni
@dcs-failsafe
@slot-advance
Scenario: check leader and replica are functioning while DCS is down
Given I get all changes from physical slot dcs_slot_1 on postgres0
Then physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds
And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 10 seconds
Given I get all changes from physical slot dcs_slot_1 on postgres-0
Then physical slot dcs_slot_1 is in sync between postgres-0 and postgres-1 after 10 seconds
And logical slot dcs_slot_0 is in sync between postgres-0 and postgres-1 after 10 seconds
And DCS is down
Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
Then postgres0 role is the primary after 10 seconds
And postgres1 role is the replica after 2 seconds
And replication works from postgres0 to postgres1 after 10 seconds
When I get all changes from logical slot dcs_slot_0 on postgres0
And I get all changes from physical slot dcs_slot_1 on postgres0
Then logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds
Then postgres-0 role is the primary after 10 seconds
And postgres-1 role is the replica after 2 seconds
And replication works from postgres-0 to postgres-1 after 10 seconds
When I get all changes from logical slot dcs_slot_0 on postgres-0
And I get all changes from physical slot dcs_slot_1 on postgres-0
Then logical slot dcs_slot_0 is in sync between postgres-0 and postgres-1 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres-0 and postgres-1 after 10 seconds
@dcs-failsafe
Scenario: check primary is demoted when one replica is shut down and DCS is down
Given DCS is down
And I kill postgres1
And I kill postmaster on postgres1
Then postgres0 role is the replica after 12 seconds
And I kill postgres-1
And I kill postmaster on postgres-1
Then postgres-0 role is the replica after 12 seconds
@dcs-failsafe
Scenario: check known replica is promoted when leader is down and DCS is up
Given I kill postgres0
And I shut down postmaster on postgres0
Given I kill postgres-0
And I shut down postmaster on postgres-0
And DCS is up
When I start postgres1
Then "members/postgres1" key in DCS has state=running after 10 seconds
And postgres1 role is the primary after 25 seconds
When I start postgres-1
Then "members/postgres-1" key in DCS has state=running after 10 seconds
And postgres-1 role is the primary after 25 seconds
@dcs-failsafe
Scenario: scale to three-node cluster
Given I start postgres0
And I configure and start postgres2 with a tag replicatefrom postgres0
Then "members/postgres2" key in DCS has state=running after 10 seconds
And "members/postgres0" key in DCS has state=running after 20 seconds
And Response on GET http://127.0.0.1:8008/failsafe contains postgres2 after 10 seconds
And replication works from postgres1 to postgres0 after 10 seconds
And replication works from postgres1 to postgres2 after 10 seconds
Given I start postgres-0
And I configure and start postgres-2 with a tag replicatefrom postgres-0
Then "members/postgres-2" key in DCS has state=running after 10 seconds
And "members/postgres-0" key in DCS has state=running after 20 seconds
And Response on GET http://127.0.0.1:8008/failsafe contains postgres-2 after 10 seconds
And replication works from postgres-1 to postgres-0 after 10 seconds
And replication works from postgres-1 to postgres-2 after 10 seconds
@dcs-failsafe
@slot-advance
Scenario: make sure permanent slots exist on replicas
Given I issue a PATCH request to http://127.0.0.1:8009/config with {"slots":{"dcs_slot_0":null,"dcs_slot_2":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds
When I get all changes from physical slot dcs_slot_1 on postgres1
Then physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres2 is in sync between postgres0 and postgres1 after 10 seconds
Then logical slot dcs_slot_2 is in sync between postgres-1 and postgres-0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres-1 and postgres-2 after 20 seconds
When I get all changes from physical slot dcs_slot_1 on postgres-1
Then physical slot dcs_slot_1 is in sync between postgres-1 and postgres-0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-2 after 10 seconds
And physical slot postgres_0 is in sync between postgres-1 and postgres-2 after 10 seconds
And physical slot postgres_2 is in sync between postgres-0 and postgres-1 after 10 seconds
@dcs-failsafe
Scenario: check three-node cluster is functioning while DCS is down
Given DCS is down
Then Response on GET http://127.0.0.1:8009/primary contains failsafe_mode_is_active after 12 seconds
Then postgres1 role is the primary after 10 seconds
And postgres0 role is the replica after 2 seconds
And postgres2 role is the replica after 2 seconds
Then postgres-1 role is the primary after 10 seconds
And postgres-0 role is the replica after 2 seconds
And postgres-2 role is the replica after 2 seconds
@dcs-failsafe
@slot-advance
Scenario: check that permanent slots are in sync between nodes while DCS is down
Given replication works from postgres1 to postgres0 after 10 seconds
And replication works from postgres1 to postgres2 after 10 seconds
When I get all changes from logical slot dcs_slot_2 on postgres1
And I get all changes from physical slot dcs_slot_1 on postgres1
Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds
And physical slot postgres2 is in sync between postgres0 and postgres1 after 10 seconds
Given replication works from postgres-1 to postgres-0 after 10 seconds
And replication works from postgres-1 to postgres-2 after 10 seconds
When I get all changes from logical slot dcs_slot_2 on postgres-1
And I get all changes from physical slot dcs_slot_1 on postgres-1
Then logical slot dcs_slot_2 is in sync between postgres-1 and postgres-0 after 20 seconds
And logical slot dcs_slot_2 is in sync between postgres-1 and postgres-2 after 20 seconds
And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-0 after 10 seconds
And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-2 after 10 seconds
And physical slot postgres_0 is in sync between postgres-1 and postgres-2 after 10 seconds
And physical slot postgres_2 is in sync between postgres-0 and postgres-1 after 10 seconds
features/ignored_slots.feature
+36 -36
View File
@@ -1,61 +1,61 @@
Feature: ignored slots
Scenario: check ignored slots aren't removed on failover/switchover
Given I start postgres1
Then postgres1 is a leader after 10 seconds
Given I start postgres-1
Then postgres-1 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8009/config with {"ignore_slots": [{"name": "unmanaged_slot_0", "database": "postgres", "plugin": "test_decoding", "type": "logical"}, {"name": "unmanaged_slot_1", "database": "postgres", "plugin": "test_decoding"}, {"name": "unmanaged_slot_2", "database": "postgres"}, {"name": "unmanaged_slot_3"}], "postgresql": {"parameters": {"wal_level": "logical"}}}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8009/config contains ignore_slots after 10 seconds
# Make sure the wal_level has been changed.
When I shut down postgres1
And I start postgres1
Then postgres1 is a leader after 10 seconds
And "members/postgres1" key in DCS has role=primary after 10 seconds
When I shut down postgres-1
And I start postgres-1
Then postgres-1 is a leader after 10 seconds
And "members/postgres-1" key in DCS has role=primary after 10 seconds
# Make sure Patroni has finished telling Postgres it should be accepting writes.
And postgres1 role is the primary after 20 seconds
And postgres-1 role is the primary after 20 seconds
# 1. Create our test logical replication slot.
# Test that ny subset of attributes in the ignore slots matcher is enough to match a slot
# by using 3 different slots.
When I create a logical replication slot unmanaged_slot_0 on postgres1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_1 on postgres1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_2 on postgres1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_3 on postgres1 with the test_decoding plugin
And I create a logical replication slot dummy_slot on postgres1 with the test_decoding plugin
When I create a logical replication slot unmanaged_slot_0 on postgres-1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_1 on postgres-1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_2 on postgres-1 with the test_decoding plugin
And I create a logical replication slot unmanaged_slot_3 on postgres-1 with the test_decoding plugin
And I create a logical replication slot dummy_slot on postgres-1 with the test_decoding plugin
# It seems like it'd be obvious that these slots exist since we just created them,
# but Patroni can actually end up dropping them almost immediately, so it's helpful
# to verify they exist before we begin testing whether they persist through failover
# cycles.
Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
Then postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
When I start postgres0
Then "members/postgres0" key in DCS has role=replica after 10 seconds
And postgres0 role is the secondary after 20 seconds
When I start postgres-0
Then "members/postgres-0" key in DCS has role=replica after 10 seconds
And postgres-0 role is the secondary after 20 seconds
# Verify that the replica has advanced beyond the point in the WAL
# where we created the replication slot so that on the next failover
# cycle we don't accidentally rewind to before the slot creation.
And replication works from postgres1 to postgres0 after 20 seconds
When I shut down postgres1
Then "members/postgres0" key in DCS has role=primary after 10 seconds
And replication works from postgres-1 to postgres-0 after 20 seconds
When I shut down postgres-1
Then "members/postgres-0" key in DCS has role=primary after 10 seconds
# 2. After a failover the server (now a replica) still has the slot.
When I start postgres1
Then postgres1 role is the secondary after 20 seconds
And "members/postgres1" key in DCS has role=replica after 10 seconds
When I start postgres-1
Then postgres-1 role is the secondary after 20 seconds
And "members/postgres-1" key in DCS has role=replica after 10 seconds
# give Patroni time to sync replication slots
And I sleep for 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
And postgres1 does not have a replication slot named dummy_slot
And postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
And postgres-1 does not have a replication slot named dummy_slot
# 3. After a failover the server (now a primary) still has the slot.
When I shut down postgres0
Then "members/postgres1" key in DCS has role=primary after 10 seconds
And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
When I shut down postgres-0
Then "members/postgres-1" key in DCS has role=primary after 10 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
features/nostream_node.feature
+13 -13
View File
@@ -1,26 +1,26 @@
Feature: nostream node
Scenario: check nostream node is recovering from archive
When I start postgres0
And I configure and start postgres1 with a tag nostream true
Then "members/postgres1" key in DCS has replication_state=in archive recovery after 10 seconds
And replication works from postgres0 to postgres1 after 30 seconds
When I start postgres-0
And I configure and start postgres-1 with a tag nostream true
Then "members/postgres-1" key in DCS has replication_state=in archive recovery after 10 seconds
And replication works from postgres-0 to postgres-1 after 30 seconds
@slot-advance
Scenario: check permanent logical replication slots are not copied
When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}}, "slots":{"test_logical":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then I receive a response code 200
When I run patronictl.py restart batman postgres0 --force
Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
When I configure and start postgres2 with a tag replicatefrom postgres1
Then "members/postgres2" key in DCS has replication_state=streaming after 10 seconds
And postgres1 does not have a replication slot named test_logical
And postgres2 does not have a replication slot named test_logical
When I run patronictl.py restart batman postgres-0 --force
Then postgres-0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
When I configure and start postgres-2 with a tag replicatefrom postgres-1
Then "members/postgres-2" key in DCS has replication_state=streaming after 10 seconds
And postgres-1 does not have a replication slot named test_logical
And postgres-2 does not have a replication slot named test_logical
@slot-advance
Scenario: check that slots are written to the /status key
Given "status" key in DCS has postgres0 in slots
And "status" key in DCS has postgres2 in slots
Given "status" key in DCS has postgres_0 in slots
And "status" key in DCS has postgres_2 in slots
And "status" key in DCS has test_logical in slots
And "status" key in DCS has test_logical in slots
And "status" key in DCS does not have postgres1 in slots
And "status" key in DCS does not have postgres_1 in slots
features/patroni_api.feature
+32 -32
View File
@@ -2,8 +2,8 @@ Feature: patroni api
We should check that patroni correctly responds to valid and not-valid API requests.
Scenario: check API requests on a stand-alone server
Given I start postgres0
And postgres0 is a leader after 10 seconds
Given I start postgres-0
And postgres-0 is a leader after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/
Then I receive a response code 200
And I receive a response state running
@@ -17,10 +17,10 @@ Scenario: check API requests on a stand-alone server
When I issue a POST request to http://127.0.0.1:8008/reinitialize with {"force": true}
Then I receive a response code 503
And I receive a response text I am the leader, can not reinitialize
When I run patronictl.py switchover batman --primary postgres0 --force
When I run patronictl.py switchover batman --primary postgres-0 --force
Then I receive a response returncode 1
And I receive a response output "Error: No candidates found to switchover to"
When I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0"}
When I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres-0"}
Then I receive a response code 412
And I receive a response text switchover is not possible: cluster does not have members except leader
When I issue an empty POST request to http://127.0.0.1:8008/failover
@@ -30,7 +30,7 @@ Scenario: check API requests on a stand-alone server
And I receive a response text "Failover could be performed only to a specific candidate"
Scenario: check local configuration reload
Given I add tag new_tag new_value to postgres0 config
Given I add tag new_tag new_value to postgres-0 config
And I issue an empty POST request to http://127.0.0.1:8008/reload
Then I receive a response code 202
@@ -58,43 +58,43 @@ Scenario: check the scheduled restart
Given I issue a scheduled restart at http://127.0.0.1:8008 in 5 seconds with {"restart_pending": "True"}
Then I receive a response code 202
And Response on GET http://127.0.0.1:8008/patroni does not contain pending_restart after 10 seconds
And postgres0 role is the primary after 10 seconds
And postgres-0 role is the primary after 10 seconds
Scenario: check API requests for the primary-replica pair in the pause mode
Given I start postgres1
Then replication works from postgres0 to postgres1 after 20 seconds
Given I start postgres-1
Then replication works from postgres-0 to postgres-1 after 20 seconds
When I run patronictl.py pause batman
Then I receive a response returncode 0
When I kill postmaster on postgres1
When I kill postmaster on postgres-1
And I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 503
And "members/postgres1" key in DCS has state=stopped after 10 seconds
When I run patronictl.py restart batman postgres1 --force
And "members/postgres-1" key in DCS has state=stopped after 10 seconds
When I run patronictl.py restart batman postgres-1 --force
Then I receive a response returncode 0
Then replication works from postgres0 to postgres1 after 20 seconds
Then replication works from postgres-0 to postgres-1 after 20 seconds
And I sleep for 2 seconds
When I issue a GET request to http://127.0.0.1:8009/replica
Then I receive a response code 200
And I receive a response state running
And I receive a response role replica
When I run patronictl.py reinit batman postgres1 --force --wait
When I run patronictl.py reinit batman postgres-1 --force --wait
Then I receive a response returncode 0
And I receive a response output "Success: reinitialize for member postgres1"
And postgres1 role is the secondary after 30 seconds
And replication works from postgres0 to postgres1 after 20 seconds
When I run patronictl.py restart batman postgres0 --force
And I receive a response output "Success: reinitialize for member postgres-1"
And postgres-1 role is the secondary after 30 seconds
And replication works from postgres-0 to postgres-1 after 20 seconds
When I run patronictl.py restart batman postgres-0 --force
Then I receive a response returncode 0
And I receive a response output "Success: restart on member postgres0"
And postgres0 role is the primary after 5 seconds
And I receive a response output "Success: restart on member postgres-0"
And postgres-0 role is the primary after 5 seconds
Scenario: check the switchover via the API in the pause mode
Given I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0", "candidate": "postgres1"}
Given I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres-0", "candidate": "postgres-1"}
Then I receive a response code 200
And postgres1 is a leader after 5 seconds
And postgres1 role is the primary after 10 seconds
And postgres0 role is the secondary after 10 seconds
And replication works from postgres1 to postgres0 after 20 seconds
And "members/postgres0" key in DCS has state=running after 10 seconds
And postgres-1 is a leader after 5 seconds
And postgres-1 role is the primary after 10 seconds
And postgres-0 role is the secondary after 10 seconds
And replication works from postgres-1 to postgres-0 after 20 seconds
And "members/postgres-0" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/primary
Then I receive a response code 503
When I issue a GET request to http://127.0.0.1:8008/replica
@@ -105,18 +105,18 @@ Scenario: check the switchover via the API in the pause mode
Then I receive a response code 503
Scenario: check the scheduled switchover
Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds
Given I issue a scheduled switchover from postgres-1 to postgres-0 in 10 seconds
Then I receive a response returncode 1
And I receive a response output "Can't schedule switchover in the paused state"
When I run patronictl.py resume batman
Then I receive a response returncode 0
Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds
Given I issue a scheduled switchover from postgres-1 to postgres-0 in 10 seconds
Then I receive a response returncode 0
And postgres0 is a leader after 20 seconds
And postgres0 role is the primary after 10 seconds
And postgres1 role is the secondary after 10 seconds
And replication works from postgres0 to postgres1 after 25 seconds
And "members/postgres1" key in DCS has state=running after 10 seconds
And postgres-0 is a leader after 20 seconds
And postgres-0 role is the primary after 10 seconds
And postgres-1 role is the secondary after 10 seconds
And replication works from postgres-0 to postgres-1 after 25 seconds
And "members/postgres-1" key in DCS has state=running after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/primary
Then I receive a response code 200
When I issue a GET request to http://127.0.0.1:8008/replica
features/permanent_slots.feature
+60 -60
View File
@@ -1,87 +1,87 @@
Feature: permanent slots
Scenario: check that physical permanent slots are created
Given I start postgres0
Then postgres0 is a leader after 10 seconds
Given I start postgres-0
Then postgres-0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres3":0},"postgresql":{"parameters":{"wal_level":"logical"}}}
When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres_3":0},"postgresql":{"parameters":{"wal_level":"logical"}}}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/config contains slots after 10 seconds
When I start postgres1
And I start postgres2
And I configure and start postgres3 with a tag replicatefrom postgres2
Then postgres0 has a physical replication slot named test_physical after 10 seconds
And postgres0 has a physical replication slot named postgres1 after 10 seconds
And postgres0 has a physical replication slot named postgres2 after 10 seconds
And postgres2 has a physical replication slot named postgres3 after 10 seconds
When I start postgres-1
And I start postgres-2
And I configure and start postgres-3 with a tag replicatefrom postgres-2
Then postgres-0 has a physical replication slot named test_physical after 10 seconds
And postgres-0 has a physical replication slot named postgres_1 after 10 seconds
And postgres-0 has a physical replication slot named postgres_2 after 10 seconds
And postgres-2 has a physical replication slot named postgres_3 after 10 seconds
@slot-advance
Scenario: check that logical permanent slots are created
Given I run patronictl.py restart batman postgres0 --force
Given I run patronictl.py restart batman postgres-0 --force
And I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_logical":{"type":"logical","database":"postgres","plugin":"test_decoding"}}}
Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
Then postgres-0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
@slot-advance
Scenario: check that permanent slots are created on replicas
Given postgres1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds
And postgres1 has a physical replication slot named test_physical after 2 seconds
And postgres2 has a physical replication slot named test_physical after 2 seconds
And postgres3 has a physical replication slot named test_physical after 2 seconds
Given postgres-1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds
And Logical slot test_logical is in sync between postgres-0 and postgres-2 after 10 seconds
And Logical slot test_logical is in sync between postgres-0 and postgres-3 after 10 seconds
And postgres-1 has a physical replication slot named test_physical after 2 seconds
And postgres-2 has a physical replication slot named test_physical after 2 seconds
And postgres-3 has a physical replication slot named test_physical after 2 seconds
@slot-advance
Scenario: check permanent physical slots that match with member names
Given postgres0 has a physical replication slot named postgres3 after 2 seconds
And postgres1 has a physical replication slot named postgres0 after 2 seconds
And postgres1 has a physical replication slot named postgres2 after 2 seconds
And postgres1 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres0 after 2 seconds
And postgres2 has a physical replication slot named postgres3 after 2 seconds
And postgres2 has a physical replication slot named postgres1 after 2 seconds
And postgres3 has a physical replication slot named postgres0 after 2 seconds
And postgres3 has a physical replication slot named postgres1 after 2 seconds
And postgres3 has a physical replication slot named postgres2 after 2 seconds
Given postgres-0 has a physical replication slot named postgres_3 after 2 seconds
And postgres-1 has a physical replication slot named postgres_0 after 2 seconds
And postgres-1 has a physical replication slot named postgres_2 after 2 seconds
And postgres-1 has a physical replication slot named postgres_3 after 2 seconds
And postgres-2 has a physical replication slot named postgres_0 after 2 seconds
And postgres-2 has a physical replication slot named postgres_3 after 2 seconds
And postgres-2 has a physical replication slot named postgres_1 after 2 seconds
And postgres-3 has a physical replication slot named postgres_0 after 2 seconds
And postgres-3 has a physical replication slot named postgres_1 after 2 seconds
And postgres-3 has a physical replication slot named postgres_2 after 2 seconds
@slot-advance
Scenario: check that permanent slots are advanced on replicas
Given I add the table replicate_me to postgres0
When I get all changes from logical slot test_logical on postgres0
And I get all changes from physical slot test_physical on postgres0
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres1 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres2 after 10 seconds
And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot test_physical is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot postgres1 is in sync between postgres0 and postgres2 after 10 seconds
And Physical slot postgres1 is in sync between postgres0 and postgres3 after 10 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres0 after 20 seconds
And Physical slot postgres3 is in sync between postgres2 and postgres1 after 10 seconds
Given I add the table replicate_me to postgres-0
When I get all changes from logical slot test_logical on postgres-0
And I get all changes from physical slot test_physical on postgres-0
Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds
And Physical slot test_physical is in sync between postgres-0 and postgres-1 after 10 seconds
And Logical slot test_logical is in sync between postgres-0 and postgres-2 after 10 seconds
And Physical slot test_physical is in sync between postgres-0 and postgres-2 after 10 seconds
And Logical slot test_logical is in sync between postgres-0 and postgres-3 after 10 seconds
And Physical slot test_physical is in sync between postgres-0 and postgres-3 after 10 seconds
And Physical slot postgres_1 is in sync between postgres-0 and postgres-2 after 10 seconds
And Physical slot postgres_1 is in sync between postgres-0 and postgres-3 after 10 seconds
And Physical slot postgres_3 is in sync between postgres-2 and postgres-0 after 20 seconds
And Physical slot postgres_3 is in sync between postgres-2 and postgres-1 after 10 seconds
@slot-advance
Scenario: check that permanent slots and member slots are written to the /status key
Given "status" key in DCS has test_physical in slots
And "status" key in DCS has postgres0 in slots
And "status" key in DCS has postgres1 in slots
And "status" key in DCS has postgres2 in slots
And "status" key in DCS has postgres3 in slots
And "status" key in DCS has postgres_0 in slots
And "status" key in DCS has postgres_1 in slots
And "status" key in DCS has postgres_2 in slots
And "status" key in DCS has postgres_3 in slots
@slot-advance
Scenario: check that only non-permanent member slots are written to the retain_slots in /status key
And "status" key in DCS has postgres0 in retain_slots
And "status" key in DCS has postgres1 in retain_slots
And "status" key in DCS has postgres2 in retain_slots
And "status" key in DCS does not have postgres3 in retain_slots
And "status" key in DCS has postgres_0 in retain_slots
And "status" key in DCS has postgres_1 in retain_slots
And "status" key in DCS has postgres_2 in retain_slots
And "status" key in DCS does not have postgres_3 in retain_slots
Scenario: check permanent physical replication slot after failover
Given I shut down postgres3
And I shut down postgres2
And I shut down postgres0
Then postgres1 has a physical replication slot named test_physical after 10 seconds
And postgres1 has a physical replication slot named postgres0 after 10 seconds
And postgres1 has a physical replication slot named postgres3 after 10 seconds
When I start postgres0
Then postgres0 role is the replica after 20 seconds
And physical replication slot named postgres1 on postgres0 has no xmin value after 10 seconds
And physical replication slot named postgres2 on postgres0 has no xmin value after 10 seconds
Given I shut down postgres-3
And I shut down postgres-2
And I shut down postgres-0
Then postgres-1 has a physical replication slot named test_physical after 10 seconds
And postgres-1 has a physical replication slot named postgres_0 after 10 seconds
And postgres-1 has a physical replication slot named postgres_3 after 10 seconds
When I start postgres-0
Then postgres-0 role is the replica after 20 seconds
And physical replication slot named postgres_1 on postgres-0 has no xmin value after 10 seconds
And physical replication slot named postgres_2 on postgres-0 has no xmin value after 10 seconds
features/priority_failover.feature
+24 -24
View File
@@ -2,38 +2,38 @@ Feature: priority replication
We should check that we can give nodes priority during failover
Scenario: check failover priority 0 prevents leaderships
Given I configure and start postgres0 with a tag failover_priority 1
And I configure and start postgres1 with a tag failover_priority 0
Then replication works from postgres0 to postgres1 after 20 seconds
When I shut down postgres0
And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres1 patroni log after 5 seconds
Then postgres1 role is the secondary after 10 seconds
When I start postgres0
Then postgres0 role is the primary after 10 seconds
Given I configure and start postgres-0 with a tag failover_priority 1
And I configure and start postgres-1 with a tag failover_priority 0
Then replication works from postgres-0 to postgres-1 after 20 seconds
When I shut down postgres-0
And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres-1 patroni log after 5 seconds
Then postgres-1 role is the secondary after 10 seconds
When I start postgres-0
Then postgres-0 role is the primary after 10 seconds
Scenario: check higher failover priority is respected
Given I configure and start postgres2 with a tag failover_priority 1
And I configure and start postgres3 with a tag failover_priority 2
Then replication works from postgres0 to postgres2 after 20 seconds
And replication works from postgres0 to postgres3 after 20 seconds
When I shut down postgres0
Then postgres3 role is the primary after 10 seconds
And there is one of ["postgres3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres3 is ahead of my wal position"] INFO in the postgres2 patroni log after 5 seconds
Given I configure and start postgres-2 with a tag failover_priority 1
And I configure and start postgres-3 with a tag failover_priority 2
Then replication works from postgres-0 to postgres-2 after 20 seconds
And replication works from postgres-0 to postgres-3 after 20 seconds
When I shut down postgres-0
Then postgres-3 role is the primary after 10 seconds
And there is one of ["postgres-3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres-3 is ahead of my wal position"] INFO in the postgres-2 patroni log after 5 seconds
Scenario: check conflicting configuration handling
When I set nofailover tag in postgres2 config
When I set nofailover tag in postgres-2 config
And I issue an empty POST request to http://127.0.0.1:8010/reload
Then I receive a response code 202
And there is one of ["Conflicting configuration between nofailover: True and failover_priority: 1. Defaulting to nofailover: True"] WARNING in the postgres2 patroni log after 5 seconds
And "members/postgres2" key in DCS has tags={'failover_priority': '1', 'nofailover': True} after 10 seconds
When I issue a POST request to http://127.0.0.1:8010/failover with {"candidate": "postgres2"}
And there is one of ["Conflicting configuration between nofailover: True and failover_priority: 1. Defaulting to nofailover: True"] WARNING in the postgres-2 patroni log after 5 seconds
And "members/postgres-2" key in DCS has tags={'failover_priority': '1', 'nofailover': True} after 10 seconds
When I issue a POST request to http://127.0.0.1:8010/failover with {"candidate": "postgres-2"}
Then I receive a response code 412
And I receive a response text "failover is not possible: no good candidates have been found"
When I reset nofailover tag in postgres1 config
When I reset nofailover tag in postgres-1 config
And I issue an empty POST request to http://127.0.0.1:8009/reload
Then I receive a response code 202
And there is one of ["Conflicting configuration between nofailover: False and failover_priority: 0. Defaulting to nofailover: False"] WARNING in the postgres1 patroni log after 5 seconds
And "members/postgres1" key in DCS has tags={'failover_priority': '0', 'nofailover': False} after 10 seconds
And I issue a POST request to http://127.0.0.1:8009/failover with {"candidate": "postgres1"}
And there is one of ["Conflicting configuration between nofailover: False and failover_priority: 0. Defaulting to nofailover: False"] WARNING in the postgres-1 patroni log after 5 seconds
And "members/postgres-1" key in DCS has tags={'failover_priority': '0', 'nofailover': False} after 10 seconds
And I issue a POST request to http://127.0.0.1:8009/failover with {"candidate": "postgres-1"}
Then I receive a response code 200
And postgres1 role is the primary after 10 seconds
And postgres-1 role is the primary after 10 seconds
features/quorum_commit.feature
+32 -32
View File
@@ -2,53 +2,53 @@ Feature: quorum commit
Check basic workfrlows when quorum commit is enabled
Scenario: check enable quorum commit and that the only leader promotes after restart
Given I start postgres0
Then postgres0 is a leader after 10 seconds
Given I start postgres-0
Then postgres-0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "synchronous_mode": "quorum"}
Then I receive a response code 200
And sync key in DCS has leader=postgres0 after 20 seconds
And sync key in DCS has leader=postgres-0 after 20 seconds
And sync key in DCS has quorum=0 after 2 seconds
And synchronous_standby_names on postgres0 is set to "_empty_str_" after 2 seconds
When I shut down postgres0
And sync key in DCS has leader=postgres0 after 2 seconds
When I start postgres0
Then postgres0 role is the primary after 10 seconds
And synchronous_standby_names on postgres-0 is set to '_empty_str_' after 2 seconds
When I shut down postgres-0
And sync key in DCS has leader=postgres-0 after 2 seconds
When I start postgres-0
Then postgres-0 role is the primary after 10 seconds
When I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_mode_strict": true}
Then synchronous_standby_names on postgres0 is set to "ANY 1 (*)" after 10 seconds
Then synchronous_standby_names on postgres-0 is set to 'ANY 1 (*)' after 10 seconds
Scenario: check failover with one quorum standby
Given I start postgres1
Then sync key in DCS has sync_standby=postgres1 after 10 seconds
And synchronous_standby_names on postgres0 is set to "ANY 1 (postgres1)" after 2 seconds
When I shut down postgres0
Then postgres1 role is the primary after 10 seconds
Given I start postgres-1
Then sync key in DCS has sync_standby=postgres-1 after 10 seconds
And synchronous_standby_names on postgres-0 is set to 'ANY 1 ("postgres-1")' after 2 seconds
When I shut down postgres-0
Then postgres-1 role is the primary after 10 seconds
And sync key in DCS has quorum=0 after 10 seconds
Then synchronous_standby_names on postgres1 is set to "ANY 1 (*)" after 10 seconds
When I start postgres0
Then sync key in DCS has leader=postgres1 after 10 seconds
Then sync key in DCS has sync_standby=postgres0 after 10 seconds
And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0)" after 2 seconds
Then synchronous_standby_names on postgres-1 is set to 'ANY 1 (*)' after 10 seconds
When I start postgres-0
Then sync key in DCS has leader=postgres-1 after 10 seconds
Then sync key in DCS has sync_standby=postgres-0 after 10 seconds
And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0")' after 2 seconds
Scenario: check behavior with three nodes and different replication factor
Given I start postgres2
Then sync key in DCS has sync_standby=postgres0,postgres2 after 10 seconds
Given I start postgres-2
Then sync key in DCS has sync_standby=postgres-0,postgres-2 after 10 seconds
And sync key in DCS has quorum=1 after 2 seconds
And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0,postgres2)" after 2 seconds
And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0","postgres-2")' after 2 seconds
When I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_node_count": 2}
Then sync key in DCS has quorum=0 after 10 seconds
And synchronous_standby_names on postgres1 is set to "ANY 2 (postgres0,postgres2)" after 2 seconds
And synchronous_standby_names on postgres-1 is set to 'ANY 2 ("postgres-0","postgres-2")' after 2 seconds
Scenario: switch from quorum replication to good old multisync and back
Given I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_mode": true, "synchronous_node_count": 1}
And I shut down postgres0
Then synchronous_standby_names on postgres1 is set to "postgres2" after 10 seconds
And sync key in DCS has sync_standby=postgres2 after 10 seconds
And I shut down postgres-0
Then synchronous_standby_names on postgres-1 is set to '"postgres-2"' after 10 seconds
And sync key in DCS has sync_standby=postgres-2 after 10 seconds
Then sync key in DCS has quorum=0 after 2 seconds
When I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_mode": "quorum"}
And I start postgres0
Then synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0,postgres2)" after 10 seconds
And sync key in DCS has sync_standby=postgres0,postgres2 after 10 seconds
And I start postgres-0
Then synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0","postgres-2")' after 10 seconds
And sync key in DCS has sync_standby=postgres-0,postgres-2 after 10 seconds
Then sync key in DCS has quorum=1 after 2 seconds
Scenario: REST API and patronictl
@@ -59,10 +59,10 @@ Feature: quorum commit
And Status code on GET http://127.0.0.1:8010/quorum is 200 after 3 seconds
Scenario: nosync node is removed from voters and synchronous_standby_names
Given I add tag nosync true to postgres2 config
Given I add tag nosync true to postgres-2 config
When I issue an empty POST request to http://127.0.0.1:8010/reload
Then I receive a response code 202
And sync key in DCS has quorum=0 after 10 seconds
And sync key in DCS has sync_standby=postgres0 after 10 seconds
And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0)" after 2 seconds
And sync key in DCS has sync_standby=postgres-0 after 10 seconds
And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0")' after 2 seconds
And Status code on GET http://127.0.0.1:8010/quorum is 503 after 10 seconds
features/recovery.feature
+12 -12
View File
@@ -2,25 +2,25 @@ Feature: recovery
We want to check that crashed postgres is started back
Scenario: check that timeline is not incremented when primary is started after crash
Given I start postgres0
Then postgres0 is a leader after 10 seconds
Given I start postgres-0
Then postgres-0 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I start postgres1
And I add the table foo to postgres0
Then table foo is present on postgres1 after 20 seconds
When I kill postmaster on postgres0
Then postgres0 role is the primary after 10 seconds
When I start postgres-1
And I add the table foo to postgres-0
Then table foo is present on postgres-1 after 20 seconds
When I kill postmaster on postgres-0
Then postgres-0 role is the primary after 10 seconds
When I issue a GET request to http://127.0.0.1:8008/
Then I receive a response code 200
And I receive a response role primary
And I receive a response timeline 1
And "members/postgres0" key in DCS has state=running after 12 seconds
And replication works from postgres0 to postgres1 after 15 seconds
And "members/postgres-0" key in DCS has state=running after 12 seconds
And replication works from postgres-0 to postgres-1 after 15 seconds
Scenario: check immediate failover when master_start_timeout=0
Given I issue a PATCH request to http://127.0.0.1:8008/config with {"master_start_timeout": 0}
Then I receive a response code 200
And Response on GET http://127.0.0.1:8008/config contains master_start_timeout after 10 seconds
When I kill postmaster on postgres0
Then postgres1 is a leader after 10 seconds
And postgres1 role is the primary after 10 seconds
When I kill postmaster on postgres-0
Then postgres-1 is a leader after 10 seconds
And postgres-1 role is the primary after 10 seconds
features/standby_cluster.feature
+29 -29
View File
@@ -1,7 +1,7 @@
Feature: standby cluster
Scenario: prepare the cluster with logical slots
Given I start postgres1
Then postgres1 is a leader after 10 seconds
Given I start postgres-1
Then postgres-1 is a leader after 10 seconds
And there is a non empty initialize key in DCS after 15 seconds
When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
Then I receive a response code 200
@@ -9,32 +9,32 @@ Feature: standby cluster
And I sleep for 3 seconds
When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"test_logical": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
Then I receive a response code 200
And I do a backup of postgres1
When I start postgres0
Then "members/postgres0" key in DCS has state=running after 10 seconds
And replication works from postgres1 to postgres0 after 15 seconds
And I do a backup of postgres-1
When I start postgres-0
Then "members/postgres-0" key in DCS has state=running after 10 seconds
And replication works from postgres-1 to postgres-0 after 15 seconds
When I issue a GET request to http://127.0.0.1:8008/patroni
Then I receive a response code 200
And I receive a response replication_state streaming
And "members/postgres0" key in DCS has replication_state=streaming after 10 seconds
And "members/postgres-0" key in DCS has replication_state=streaming after 10 seconds
@slot-advance
Scenario: check permanent logical slots are synced to the replica
Given I run patronictl.py restart batman postgres1 --force
Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
Given I run patronictl.py restart batman postgres-1 --force
Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds
Scenario: Detach exiting node from the cluster
When I shut down postgres1
Then postgres0 is a leader after 10 seconds
And "members/postgres0" key in DCS has role=primary after 5 seconds
When I shut down postgres-1
Then postgres-0 is a leader after 10 seconds
And "members/postgres-0" key in DCS has role=primary after 5 seconds
When I issue a GET request to http://127.0.0.1:8008/
Then I receive a response code 200
Scenario: check replication of a single table in a standby cluster
Given I start postgres1 in a standby cluster batman1 as a clone of postgres0
Then postgres1 is a leader of batman1 after 10 seconds
When I add the table foo to postgres0
Then table foo is present on postgres1 after 20 seconds
Given I start postgres-1 in a standby cluster batman1 as a clone of postgres-0
Then postgres-1 is a leader of batman1 after 10 seconds
When I add the table foo to postgres-0
Then table foo is present on postgres-1 after 20 seconds
When I issue a GET request to http://127.0.0.1:8009/patroni
Then I receive a response code 200
And I receive a response replication_state streaming
@@ -44,29 +44,29 @@ Feature: standby cluster
When I issue a GET request to http://127.0.0.1:8009/standby_leader
Then I receive a response code 200
And I receive a response role standby_leader
And there is a postgres1_cb.log with "on_role_change standby_leader batman1" in postgres1 data directory
When I start postgres2 in a cluster batman1
Then postgres2 role is the replica after 24 seconds
And postgres2 is replicating from postgres1 after 10 seconds
And table foo is present on postgres2 after 20 seconds
And there is a postgres-1_cb.log with "on_role_change standby_leader batman1" in postgres-1 data directory
When I start postgres-2 in a cluster batman1
Then postgres-2 role is the replica after 24 seconds
And postgres-2 is replicating from postgres-1 after 10 seconds
And table foo is present on postgres-2 after 20 seconds
When I issue a GET request to http://127.0.0.1:8010/patroni
Then I receive a response code 200
And I receive a response replication_state streaming
And postgres1 does not have a replication slot named test_logical
And postgres-1 does not have a replication slot named test_logical
Scenario: check switchover
Given I run patronictl.py switchover batman1 --force
Then Status code on GET http://127.0.0.1:8010/standby_leader is 200 after 10 seconds
And postgres1 is replicating from postgres2 after 32 seconds
And there is a postgres2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres2 data directory
And postgres-1 is replicating from postgres-2 after 32 seconds
And there is a postgres-2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres-2 data directory
Scenario: check failover
When I kill postgres2
And I kill postmaster on postgres2
Then postgres1 is replicating from postgres0 after 32 seconds
When I kill postgres-2
And I kill postmaster on postgres-2
Then postgres-1 is replicating from postgres-0 after 32 seconds
And Status code on GET http://127.0.0.1:8009/standby_leader is 200 after 10 seconds
When I issue a GET request to http://127.0.0.1:8009/primary
Then I receive a response code 503
And I receive a response role standby_leader
And replication works from postgres0 to postgres1 after 15 seconds
And there is a postgres1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres1 data directory
And replication works from postgres-0 to postgres-1 after 15 seconds
And there is a postgres-1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres-1 data directory
features/steps/basic_replication.py
+25 -15
View File
@@ -2,17 +2,27 @@ import json
from time import sleep, time
from behave import step, then
import parse
from behave import register_type, step, then
import patroni.psycopg as pg
@step('I start {name:w}')
@parse.with_pattern(r'[a-z][a-z0-9_\-]*[a-z0-9]')
def parse_name(text):
return text
register_type(name=parse_name)
@step('I start {name:name}')
def start_patroni(context, name):
return context.pctl.start(name)
@step('I start duplicate {name:w} on port {port:d}')
@step('I start duplicate {name:name} on port {port:d}')
def start_duplicate_patroni(context, name, port):
config = {
"name": name,
@@ -28,22 +38,22 @@ def start_duplicate_patroni(context, name, port):
"No error was raised by duplicate start of {0} ".format(name)
@step('I shut down {name:w}')
@step('I shut down {name:name}')
def stop_patroni(context, name):
return context.pctl.stop(name, timeout=60)
@step('I kill {name:w}')
@step('I kill {name:name}')
def kill_patroni(context, name):
return context.pctl.stop(name, kill=True)
@step('I shut down postmaster on {name:w}')
@step('I shut down postmaster on {name:name}')
def stop_postgres(context, name):
return context.pctl.stop(name, postgres=True)
@step('I kill postmaster on {name:w}')
@step('I kill postmaster on {name:name}')
def kill_postgres(context, name):
return context.pctl.stop(name, kill=True, postgres=True)
@@ -53,7 +63,7 @@ def get_wal_name(context, pg_name):
return 'xlog' if int(version) / 10000 < 10 else 'wal'
@step('I add the table {table_name:w} to {pg_name:w}')
@step('I add the table {table_name:w} to {pg_name:name}')
def add_table(context, table_name, pg_name):
# parse the configuration file and get the port
try:
@@ -63,7 +73,7 @@ def add_table(context, table_name, pg_name):
assert False, "Error creating table {0} on {1}: {2}".format(table_name, pg_name, e)
@step('I {action:w} wal replay on {pg_name:w}')
@step('I {action:w} wal replay on {pg_name:name}')
def toggle_wal_replay(context, action, pg_name):
# pause or resume the wal replay process
try:
@@ -72,7 +82,7 @@ def toggle_wal_replay(context, action, pg_name):
assert False, "Error during {0} wal recovery on {1}: {2}".format(action, pg_name, e)
@step('I {action:w} table on {pg_name:w}')
@step('I {action:w} table on {pg_name:name}')
def crdr_mytest(context, action, pg_name):
try:
if (action == "create"):
@@ -83,7 +93,7 @@ def crdr_mytest(context, action, pg_name):
assert False, "Error {0} table mytest on {1}: {2}".format(action, pg_name, e)
@step('I load data on {pg_name:w}')
@step('I load data on {pg_name:name}')
def initiate_load(context, pg_name):
# perform dummy load
try:
@@ -92,7 +102,7 @@ def initiate_load(context, pg_name):
assert False, "Error loading test data on {0}: {1}".format(pg_name, e)
@then('Table {table_name:w} is present on {pg_name:w} after {max_replication_delay:d} seconds')
@then('Table {table_name:w} is present on {pg_name:name} after {max_replication_delay:d} seconds')
def table_is_present_on(context, table_name, pg_name, max_replication_delay):
max_replication_delay *= context.timeout_multiplier
for _ in range(int(max_replication_delay)):
@@ -104,15 +114,15 @@ def table_is_present_on(context, table_name, pg_name, max_replication_delay):
"Table {0} is not present on {1} after {2} seconds".format(table_name, pg_name, max_replication_delay)
@then('{pg_name:w} role is the {pg_role:w} after {max_promotion_timeout:d} seconds')
@then('{pg_name:name} role is the {pg_role:w} after {max_promotion_timeout:d} seconds')
def check_role(context, pg_name, pg_role, max_promotion_timeout):
max_promotion_timeout *= context.timeout_multiplier
assert context.pctl.check_role_has_changed_to(pg_name, pg_role, timeout=int(max_promotion_timeout)), \
"{0} role didn't change to {1} after {2} seconds".format(pg_name, pg_role, max_promotion_timeout)
@step('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
@then('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
@step('replication works from {primary:name} to {replica:name} after {time_limit:d} seconds')
@then('replication works from {primary:name} to {replica:name} after {time_limit:d} seconds')
def replication_works(context, primary, replica, time_limit):
context.execute_steps(u"""
When I add the table test_{0} to {1}
features/steps/cascading_replication.py
+3 -3
View File
@@ -4,18 +4,18 @@ import time
from behave import step, then
@step('I configure and start {name:w} with a tag {tag_name:w} {tag_value:w}')
@step('I configure and start {name:name} with a tag {tag_name:w} {tag_value}')
def start_patroni_with_a_name_value_tag(context, name, tag_name, tag_value):
return context.pctl.start(name, custom_config={'tags': {tag_name: tag_value}})
@then('There is a {label} with "{content}" in {name:w} data directory')
@then('There is a {label} with "{content}" in {name:name} data directory')
def check_label(context, label, content, name):
value = (context.pctl.read_label(name, label) or '').replace('\n', '\\n')
assert content in value, "\"{0}\" in {1} doesn't contain {2}".format(value, label, content)
@step('I create label with "{content:w}" in {name:w} data directory')
@step('I create label with "{content}" in {name:name} data directory')
def write_label(context, content, name):
context.pctl.write_label(name, content)
features/steps/citus.py
+9 -9
View File
@@ -11,8 +11,8 @@ from dateutil import tz
tzutc = tz.tzutc()
@step('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
@then('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
@step('{name:name} is a leader in a group {group:d} after {time_limit:d} seconds')
@then('{name:name} is a leader in a group {group:d} after {time_limit:d} seconds')
def is_a_group_leader(context, name, group, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -40,12 +40,12 @@ def check_group_member(context, name, group, key, value, time_limit):
" after {5} seconds").format(name, group, key, value, response, time_limit)
@step('I start {name:w} in citus group {group:d}')
@step('I start {name:name} in citus group {group:d}')
def start_citus(context, name, group):
return context.pctl.start(name, custom_config={"citus": {"database": "postgres", "group": int(group)}})
@step('{name1:w} is registered in the {name2:w} as the {role:w} in group {group:d} after {time_limit:d} seconds')
@step('{name1:name} is registered in the {name2:name} as the {role:w} in group {group:d} after {time_limit:d} seconds')
def check_registration(context, name1, name2, role, group, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -65,13 +65,13 @@ def check_registration(context, name1, name2, role, group, time_limit):
assert False, "Node {0} is not registered in pg_dist_node on the node {1}".format(name1, name2)
@step('I create a distributed table on {name:w}')
@step('I create a distributed table on {name:name}')
def create_distributed_table(context, name):
context.pctl.query(name, 'CREATE TABLE public.d(id int not null)')
context.pctl.query(name, "SELECT create_distributed_table('public.d', 'id')")
@step('I cleanup a distributed table on {name:w}')
@step('I cleanup a distributed table on {name:name}')
def cleanup_distributed_table(context, name):
context.pctl.query(name, 'TRUNCATE public.d')
@@ -87,7 +87,7 @@ def insert_thread(query_func, context):
context.thread_stop_event.wait(0.01)
@step('I start a thread inserting data on {name:w}')
@step('I start a thread inserting data on {name:name}')
def start_insert_thread(context, name):
context.thread_stop_event = Event()
context.insert_counter = 0
@@ -110,13 +110,13 @@ def stop_insert_thread(context):
assert not context.thread.is_alive(), "Thread is still alive"
@step("a distributed table on {name:w} has expected rows")
@step("a distributed table on {name:name} has expected rows")
def count_rows(context, name):
rows = context.pctl.query(name, "SELECT COUNT(*) FROM public.d").fetchone()[0]
assert rows == context.insert_counter, "Distributed table doesn't have expected amount of rows"
@step("there is a transaction in progress on {name:w} changing pg_dist_node after {time_limit:d} seconds")
@step("there is a transaction in progress on {name:name} changing pg_dist_node after {time_limit:d} seconds")
def check_transaction(context, name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
features/steps/custom_bootstrap.py
+4 -4
View File
@@ -3,17 +3,17 @@ import time
from behave import step, then
@step('I start {name:w} in a cluster {cluster_name:w} as a clone of {name2:w}')
@step('I start {name:name} in a cluster {cluster_name:w} as a clone of {name2:name}')
def start_cluster_clone(context, name, cluster_name, name2):
context.pctl.clone(name2, cluster_name, name)
@step('I start {name:w} in a cluster {cluster_name:w} from backup')
@step('I start {name:name} in a cluster {cluster_name:w} from backup')
def start_cluster_from_backup(context, name, cluster_name):
context.pctl.bootstrap_from_backup(name, cluster_name)
@then('{name:w} is a leader of {cluster_name:w} after {time_limit:d} seconds')
@then('{name:name} is a leader of {cluster_name:w} after {time_limit:d} seconds')
def is_a_leader(context, name, cluster_name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -22,6 +22,6 @@ def is_a_leader(context, name, cluster_name, time_limit):
assert time.time() < max_time, "{0} is not a leader in dcs after {1} seconds".format(name, time_limit)
@step('I do a backup of {name:w}')
@step('I do a backup of {name:name}')
def do_backup(context, name):
context.pctl.backup(name)
features/steps/dcs_failsafe_mode.py
+1 -1
View File
@@ -11,6 +11,6 @@ def stop_dcs_outage(context):
context.dcs_ctl.stop_outage()
@step('I start {name:w} in a cluster {cluster_name:w} from backup with no_leader')
@step('I start {name:name} in a cluster {cluster_name:w} from backup with no_leader')
def start_cluster_from_backup_no_leader(context, name, cluster_name):
context.pctl.bootstrap_from_backup_no_leader(name, cluster_name)
features/steps/patroni_api.py
+6 -6
View File
@@ -28,8 +28,8 @@ register_type(url=parse_url)
# just rely on the database availability, since there is
# a short gap between the time PostgreSQL becomes available
# and Patroni assuming the leader role.
@step('{name:w} is a leader after {time_limit:d} seconds')
@then('{name:w} is a leader after {time_limit:d} seconds')
@step('{name:name} is a leader after {time_limit:d} seconds')
@then('{name:name} is a leader after {time_limit:d} seconds')
def is_a_leader(context, name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -97,7 +97,7 @@ def do_run(context, cmd):
context.response = response.decode('utf-8').strip()
@then('I receive a response {component:w} {data}')
@then('I receive a response {component:name} {data}')
def check_response(context, component, data):
if component == 'code':
assert context.status_code == int(data), \
@@ -116,7 +116,7 @@ def check_response(context, component, data):
assert str(context.response[component]) == str(data), "{0} does not contain {1}".format(component, data)
@step('I issue a scheduled switchover from {from_host:w} to {to_host:w} in {in_seconds:d} seconds')
@step('I issue a scheduled switchover from {from_host:name} to {to_host:name} in {in_seconds:d} seconds')
def scheduled_switchover(context, from_host, to_host, in_seconds):
context.execute_steps(u"""
Given I run patronictl.py switchover batman --primary {0} --candidate {1} --scheduled "{2}" --force
@@ -130,13 +130,13 @@ def scheduled_restart(context, url, in_seconds, data):
context.execute_steps(u"""Given I issue a POST request to {0}/restart with {1}""".format(url, json.dumps(data)))
@step('I {action:w} {tag:w} tag in {pg_name:w} config')
@step('I {action:w} {tag:w} tag in {pg_name:name} config')
def add_bool_tag_to_config(context, action, tag, pg_name):
value = action == 'set'
context.pctl.add_tag_to_config(pg_name, tag, value)
@step('I add tag {tag:w} {value:w} to {pg_name:w} config')
@step('I add tag {tag:w} {value:w} to {pg_name:name} config')
def add_tag_to_config(context, tag, value, pg_name):
context.pctl.add_tag_to_config(pg_name, tag, value)
features/steps/quorum_commit.py
+1 -1
View File
@@ -36,7 +36,7 @@ def _parse_synchronous_standby_names(value):
return expected_num, expected_value
@then('synchronous_standby_names on {name:2} is set to "{value}" after {time_limit:d} seconds')
@then("synchronous_standby_names on {name:2} is set to '{value}' after {time_limit:d} seconds")
def check_synchronous_standby_names(context, name, value, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
features/steps/slots.py
+13 -12
View File
@@ -6,7 +6,7 @@ from behave import step, then
import patroni.psycopg as pg
@step('I create a logical replication slot {slot_name} on {pg_name:w} with the {plugin:w} plugin')
@step('I create a logical replication slot {slot_name} on {pg_name:name} with the {plugin:w} plugin')
def create_logical_replication_slot(context, slot_name, pg_name, plugin):
try:
output = context.pctl.query(pg_name, ("SELECT pg_create_logical_replication_slot('{0}', '{1}'),"
@@ -17,9 +17,9 @@ def create_logical_replication_slot(context, slot_name, pg_name, plugin):
assert False, "Error creating slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin)
@step('{pg_name:w} has a logical replication slot named {slot_name}'
@step('{pg_name:name} has a logical replication slot named {slot_name}'
' with the {plugin:w} plugin after {time_limit:d} seconds')
@then('{pg_name:w} has a logical replication slot named {slot_name}'
@then('{pg_name:name} has a logical replication slot named {slot_name}'
' with the {plugin:w} plugin after {time_limit:d} seconds')
def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit):
time_limit *= context.timeout_multiplier
@@ -38,8 +38,8 @@ def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit
assert False, f"Error looking for slot {slot_name} on {pg_name} with plugin {plugin}"
@step('{pg_name:w} does not have a replication slot named {slot_name:w}')
@then('{pg_name:w} does not have a replication slot named {slot_name:w}')
@step('{pg_name:name} does not have a replication slot named {slot_name:w}')
@then('{pg_name:name} does not have a replication slot named {slot_name:w}')
def does_not_have_replication_slot(context, pg_name, slot_name):
try:
row = context.pctl.query(pg_name, ("SELECT 1 FROM pg_replication_slots"
@@ -49,7 +49,8 @@ def does_not_have_replication_slot(context, pg_name, slot_name):
assert False, "Error looking for slot {0} on {1}".format(slot_name, pg_name)
@step('{slot_type:w} slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds')
@step('{slot_type:w} slot {slot_name:w} is in sync between '
'{pg_name1:name} and {pg_name2:name} after {time_limit:d} seconds')
def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -68,17 +69,17 @@ def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit)
f"{slot_type} slot {slot_name} is not in sync between {pg_name1} and {pg_name2} after {time_limit} seconds"
@step('I get all changes from logical slot {slot_name:w} on {pg_name:w}')
@step('I get all changes from logical slot {slot_name:w} on {pg_name:name}')
def logical_slot_get_changes(context, slot_name, pg_name):
context.pctl.query(pg_name, "SELECT * FROM pg_logical_slot_get_changes('{0}', NULL, NULL)".format(slot_name))
@step('I get all changes from physical slot {slot_name:w} on {pg_name:w}')
@step('I get all changes from physical slot {slot_name:w} on {pg_name:name}')
def physical_slot_get_changes(context, slot_name, pg_name):
context.pctl.query(pg_name, f"SELECT * FROM pg_replication_slot_advance('{slot_name}', pg_current_wal_lsn())")
@step('{pg_name:w} has a physical replication slot named {slot_name} after {time_limit:d} seconds')
@step('{pg_name:name} has a physical replication slot named {slot_name} after {time_limit:d} seconds')
def has_physical_replication_slot(context, pg_name, slot_name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -94,7 +95,7 @@ def has_physical_replication_slot(context, pg_name, slot_name, time_limit):
assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds"
@step('physical replication slot named {slot_name} on {pg_name:w} has no xmin value after {time_limit:d} seconds')
@step('physical replication slot named {slot_name} on {pg_name:name} has no xmin value after {time_limit:d} seconds')
def physical_slot_no_xmin(context, pg_name, slot_name, time_limit):
time_limit *= context.timeout_multiplier
max_time = time.time() + int(time_limit)
@@ -114,13 +115,13 @@ def physical_slot_no_xmin(context, pg_name, slot_name, time_limit):
else f"Physical slot {slot_name} has xmin value after {time_limit} seconds"
@step('"{name}" key in DCS has {subkey:w} in {key:w}')
@step('"{name}" key in DCS has {subkey} in {key:w}')
def dcs_key_contains(context, name, subkey, key):
response = json.loads(context.dcs_ctl.query(name))
assert key in response and subkey in response[key], f"{name} key in DCS doesn't have {subkey} in {key}"
@step('"{name}" key in DCS does not have {subkey:w} in {key:w}')
@step('"{name}" key in DCS does not have {subkey} in {key:w}')
def dcs_key_does_not_contain(context, name, subkey, key):
response = json.loads(context.dcs_ctl.query(name))
assert key not in response or subkey not in response[key], f"{name} key in DCS has {subkey} in {key}"
features/steps/standby_cluster.py
+3 -3
View File
@@ -9,7 +9,7 @@ def callbacks(context, name):
for c in ('on_start', 'on_stop', 'on_restart', 'on_role_change')}
@step('I start {name:w} in a cluster {cluster_name:w}')
@step('I start {name:name} in a cluster {cluster_name:w}')
def start_patroni(context, name, cluster_name):
return context.pctl.start(name, custom_config={
"scope": cluster_name,
@@ -20,7 +20,7 @@ def start_patroni(context, name, cluster_name):
})
@step('I start {name:w} in a standby cluster {cluster_name:w} as a clone of {name2:w}')
@step('I start {name:name} in a standby cluster {cluster_name:w} as a clone of {name2:name}')
def start_patroni_standby_cluster(context, name, cluster_name, name2):
# we need to remove patroni.dynamic.json in order to "bootstrap" standby cluster with existing PGDATA
os.unlink(os.path.join(context.pctl._processes[name]._data_dir, 'patroni.dynamic.json'))
@@ -49,7 +49,7 @@ def start_patroni_standby_cluster(context, name, cluster_name, name2):
return context.pctl.start(name)
@step('{pg_name1:w} is replicating from {pg_name2:w} after {timeout:d} seconds')
@step('{pg_name1:name} is replicating from {pg_name2:name} after {timeout:d} seconds')
def check_replication_status(context, pg_name1, pg_name2, timeout):
bound_time = time.time() + timeout * context.timeout_multiplier
features/steps/watchdog.py
+7 -7
View File
@@ -14,12 +14,12 @@ def polling_loop(timeout, interval=1):
time.sleep(interval)
@step('I start {name:w} with watchdog')
@step('I start {name:name} with watchdog')
def start_patroni_with_watchdog(context, name):
return context.pctl.start(name, custom_config={'watchdog': True, 'bootstrap': {'dcs': {'ttl': 20}}})
@step('{name:w} watchdog has been pinged after {timeout:d} seconds')
@step('{name:name} watchdog has been pinged after {timeout:d} seconds')
def watchdog_was_pinged(context, name, timeout):
for _ in polling_loop(timeout):
if context.pctl.get_watchdog(name).was_pinged:
@@ -27,22 +27,22 @@ def watchdog_was_pinged(context, name, timeout):
return False
@then('{name:w} watchdog has been closed')
@then('{name:name} watchdog has been closed')
def watchdog_was_closed(context, name):
assert context.pctl.get_watchdog(name).was_closed
@step('{name:w} watchdog has a {timeout:d} second timeout')
@step('{name:name} watchdog has a {timeout:d} second timeout')
def watchdog_has_timeout(context, name, timeout):
assert context.pctl.get_watchdog(name).timeout == timeout
@step('I reset {name:w} watchdog state')
@step('I reset {name:name} watchdog state')
def watchdog_reset_pinged(context, name):
context.pctl.get_watchdog(name).reset()
@then('{name:w} watchdog is triggered after {timeout:d} seconds')
@then('{name:name} watchdog is triggered after {timeout:d} seconds')
def watchdog_was_triggered(context, name, timeout):
for _ in polling_loop(timeout):
if context.pctl.get_watchdog(name).was_triggered:
@@ -50,6 +50,6 @@ def watchdog_was_triggered(context, name, timeout):
assert False
@step('{name:w} hangs for {timeout:d} seconds')
@step('{name:name} hangs for {timeout:d} seconds')
def patroni_hang(context, name, timeout):
return context.pctl.patroni_hang(name, timeout)
features/watchdog.feature
+16 -16
View File
@@ -2,38 +2,38 @@ Feature: watchdog
Verify that watchdog gets pinged and triggered under appropriate circumstances.
Scenario: watchdog is opened and pinged
Given I start postgres0 with watchdog
Then postgres0 is a leader after 10 seconds
And postgres0 role is the primary after 10 seconds
And postgres0 watchdog has been pinged after 10 seconds
And postgres0 watchdog has a 15 second timeout
Given I start postgres-0 with watchdog
Then postgres-0 is a leader after 10 seconds
And postgres-0 role is the primary after 10 seconds
And postgres-0 watchdog has been pinged after 10 seconds
And postgres-0 watchdog has a 15 second timeout
Scenario: watchdog is reconfigured after global ttl changed
Given I run patronictl.py edit-config batman -s ttl=30 --force
Then I receive a response returncode 0
And I receive a response output "+ttl: 30"
When I sleep for 4 seconds
Then postgres0 watchdog has a 25 second timeout
Then postgres-0 watchdog has a 25 second timeout
Scenario: watchdog is disabled during pause
Given I run patronictl.py pause batman
Then I receive a response returncode 0
When I sleep for 2 seconds
Then postgres0 watchdog has been closed
Then postgres-0 watchdog has been closed
Scenario: watchdog is opened and pinged after resume
Given I reset postgres0 watchdog state
Given I reset postgres-0 watchdog state
And I run patronictl.py resume batman
Then I receive a response returncode 0
And postgres0 watchdog has been pinged after 10 seconds
And postgres-0 watchdog has been pinged after 10 seconds
Scenario: watchdog is disabled when shutting down
Given I shut down postgres0
Then postgres0 watchdog has been closed
Given I shut down postgres-0
Then postgres-0 watchdog has been closed
Scenario: watchdog is triggered if patroni stops responding
Given I reset postgres0 watchdog state
And I start postgres0 with watchdog
Then postgres0 role is the primary after 10 seconds
When postgres0 hangs for 30 seconds
Then postgres0 watchdog is triggered after 30 seconds
Given I reset postgres-0 watchdog state
And I start postgres-0 with watchdog
Then postgres-0 role is the primary after 10 seconds
When postgres-0 hangs for 30 seconds
Then postgres-0 watchdog is triggered after 30 seconds