diff --git a/features/basic_replication.feature b/features/basic_replication.feature index e5d11ffc..c8a05453 100644 --- a/features/basic_replication.feature +++ b/features/basic_replication.feature @@ -2,57 +2,57 @@ Feature: basic replication We should check that the basic bootstrapping, replication and failover works. Scenario: check replication of a single table - Given I start postgres0 - Then postgres0 is a leader after 10 seconds + Given I start postgres-0 + Then postgres-0 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "synchronous_mode": true} Then I receive a response code 200 - When I start postgres1 - And I configure and start postgres2 with a tag replicatefrom postgres0 - And "sync" key in DCS has leader=postgres0 after 20 seconds - And I add the table foo to postgres0 - Then table foo is present on postgres1 after 20 seconds - Then table foo is present on postgres2 after 20 seconds + When I start postgres-1 + And I configure and start postgres-2 with a tag replicatefrom postgres-0 + And "sync" key in DCS has leader=postgres-0 after 20 seconds + And I add the table foo to postgres-0 + Then table foo is present on postgres-1 after 20 seconds + Then table foo is present on postgres-2 after 20 seconds Scenario: check restart of sync replica - Given I shut down postgres2 - Then "sync" key in DCS has sync_standby=postgres1 after 5 seconds - When I start postgres2 - And I shut down postgres1 - Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds - When I start postgres1 - Then "members/postgres1" key in DCS has state=running after 10 seconds + Given I shut down postgres-2 + Then "sync" key in DCS has sync_standby=postgres-1 after 5 seconds + When I start postgres-2 + And I shut down postgres-1 + Then "sync" key in DCS has sync_standby=postgres-2 after 10 seconds + When I start postgres-1 + Then "members/postgres-1" key in DCS has state=running after 10 seconds And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds Scenario: check stuck sync replica Given I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": true, "maximum_lag_on_syncnode": 15000000, "postgresql": {"parameters": {"synchronous_commit": "remote_apply"}}} Then I receive a response code 200 - And I create table on postgres0 - And table mytest is present on postgres1 after 2 seconds - And table mytest is present on postgres2 after 2 seconds - When I pause wal replay on postgres2 - And I load data on postgres0 - Then "sync" key in DCS has sync_standby=postgres1 after 15 seconds - And I resume wal replay on postgres2 + And I create table on postgres-0 + And table mytest is present on postgres-1 after 2 seconds + And table mytest is present on postgres-2 after 2 seconds + When I pause wal replay on postgres-2 + And I load data on postgres-0 + Then "sync" key in DCS has sync_standby=postgres-1 after 15 seconds + And I resume wal replay on postgres-2 And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds And Status code on GET http://127.0.0.1:8010/async is 200 after 3 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": null, "maximum_lag_on_syncnode": -1, "postgresql": {"parameters": {"synchronous_commit": "on"}}} Then I receive a response code 200 - And I drop table on postgres0 + And I drop table on postgres-0 Scenario: check multi sync replication Given I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 2} Then I receive a response code 200 - Then "sync" key in DCS has sync_standby=postgres1,postgres2 after 10 seconds + Then "sync" key in DCS has sync_standby=postgres-1,postgres-2 after 10 seconds And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 1} Then I receive a response code 200 - And I shut down postgres1 - Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds - When I start postgres1 - Then "members/postgres1" key in DCS has state=running after 10 seconds + And I shut down postgres-1 + Then "sync" key in DCS has sync_standby=postgres-2 after 10 seconds + When I start postgres-1 + Then "members/postgres-1" key in DCS has state=running after 10 seconds And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds @@ -60,26 +60,26 @@ Feature: basic replication Given I run patronictl.py pause batman Then I receive a response returncode 0 When I sleep for 2 seconds - And I shut down postgres0 + And I shut down postgres-0 And I run patronictl.py resume batman Then I receive a response returncode 0 - And postgres2 role is the primary after 24 seconds + And postgres-2 role is the primary after 24 seconds And Response on GET http://127.0.0.1:8010/history contains recovery after 10 seconds - And there is a postgres2_cb.log with "on_role_change primary batman" in postgres2 data directory + And there is a postgres-2_cb.log with "on_role_change primary batman" in postgres-2 data directory When I issue a PATCH request to http://127.0.0.1:8010/config with {"synchronous_mode": null, "master_start_timeout": 0} Then I receive a response code 200 - When I add the table bar to postgres2 - Then table bar is present on postgres1 after 20 seconds + When I add the table bar to postgres-2 + Then table bar is present on postgres-1 after 20 seconds And Response on GET http://127.0.0.1:8010/config contains master_start_timeout after 10 seconds Scenario: check rejoin of the former primary with pg_rewind - Given I add the table splitbrain to postgres0 - And I start postgres0 - Then postgres0 role is the secondary after 20 seconds - When I add the table buz to postgres2 - Then table buz is present on postgres0 after 20 seconds + Given I add the table splitbrain to postgres-0 + And I start postgres-0 + Then postgres-0 role is the secondary after 20 seconds + When I add the table buz to postgres-2 + Then table buz is present on postgres-0 after 20 seconds @reject-duplicate-name Scenario: check graceful rejection when two nodes have the same name - Given I start duplicate postgres0 on port 8011 - Then there is one of ["Can't start; there is already a node named 'postgres0' running"] CRITICAL in the dup-postgres0 patroni log after 5 seconds + Given I start duplicate postgres-0 on port 8011 + Then there is one of ["Can't start; there is already a node named 'postgres-0' running"] CRITICAL in the dup-postgres-0 patroni log after 5 seconds diff --git a/features/cascading_replication.feature b/features/cascading_replication.feature index c6774af3..38c2fdff 100644 --- a/features/cascading_replication.feature +++ b/features/cascading_replication.feature @@ -2,13 +2,13 @@ Feature: cascading replication We should check that patroni can do base backup and streaming from the replica Scenario: check a base backup and streaming replication from a replica - Given I start postgres0 - And postgres0 is a leader after 10 seconds - And I configure and start postgres1 with a tag clonefrom true - And replication works from postgres0 to postgres1 after 20 seconds - And I create label with "postgres0" in postgres0 data directory - And I create label with "postgres1" in postgres1 data directory - And "members/postgres1" key in DCS has state=running after 12 seconds - And I configure and start postgres2 with a tag replicatefrom postgres1 - Then replication works from postgres0 to postgres2 after 30 seconds - And there is a label with "postgres1" in postgres2 data directory + Given I start postgres-0 + And postgres-0 is a leader after 10 seconds + And I configure and start postgres-1 with a tag clonefrom true + And replication works from postgres-0 to postgres-1 after 20 seconds + And I create label with "postgres-0" in postgres-0 data directory + And I create label with "postgres-1" in postgres-1 data directory + And "members/postgres-1" key in DCS has state=running after 12 seconds + And I configure and start postgres-2 with a tag replicatefrom postgres-1 + Then replication works from postgres-0 to postgres-2 after 30 seconds + And there is a label with "postgres-1" in postgres-2 data directory diff --git a/features/citus.feature b/features/citus.feature index 6fb3d75a..e1a4260c 100644 --- a/features/citus.feature +++ b/features/citus.feature @@ -2,79 +2,79 @@ Feature: citus We should check that coordinator discovers and registers workers and clients don't have errors when worker cluster switches over Scenario: check that worker cluster is registered in the coordinator - Given I start postgres0 in citus group 0 - And I start postgres2 in citus group 1 - Then postgres0 is a leader in a group 0 after 10 seconds - And postgres2 is a leader in a group 1 after 10 seconds - When I start postgres1 in citus group 0 - And I start postgres3 in citus group 1 - Then replication works from postgres0 to postgres1 after 15 seconds - Then replication works from postgres2 to postgres3 after 15 seconds - And postgres0 is registered in the postgres0 as the primary in group 0 after 5 seconds - And postgres1 is registered in the postgres0 as the secondary in group 0 after 5 seconds - And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds - And postgres3 is registered in the postgres0 as the secondary in group 1 after 5 seconds + Given I start postgres-0 in citus group 0 + And I start postgres-2 in citus group 1 + Then postgres-0 is a leader in a group 0 after 10 seconds + And postgres-2 is a leader in a group 1 after 10 seconds + When I start postgres-1 in citus group 0 + And I start postgres-3 in citus group 1 + Then replication works from postgres-0 to postgres-1 after 15 seconds + Then replication works from postgres-2 to postgres-3 after 15 seconds + And postgres-0 is registered in the postgres-0 as the primary in group 0 after 5 seconds + And postgres-1 is registered in the postgres-0 as the secondary in group 0 after 5 seconds + And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds + And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 5 seconds Scenario: coordinator failover updates pg_dist_node - Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force - Then postgres1 role is the primary after 10 seconds - And "members/postgres0" key in a group 0 in DCS has state=running after 15 seconds - And replication works from postgres1 to postgres0 after 15 seconds - And postgres1 is registered in the postgres2 as the primary in group 0 after 5 seconds - And postgres0 is registered in the postgres2 as the secondary in group 0 after 15 seconds - And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds - When I run patronictl.py switchover batman --group 0 --candidate postgres0 --force - Then postgres0 role is the primary after 10 seconds - And replication works from postgres0 to postgres1 after 15 seconds - And postgres0 is registered in the postgres2 as the primary in group 0 after 5 seconds - And postgres1 is registered in the postgres2 as the secondary in group 0 after 15 seconds - And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds + Given I run patronictl.py failover batman --group 0 --candidate postgres-1 --force + Then postgres-1 role is the primary after 10 seconds + And "members/postgres-0" key in a group 0 in DCS has state=running after 15 seconds + And replication works from postgres-1 to postgres-0 after 15 seconds + And postgres-1 is registered in the postgres-2 as the primary in group 0 after 5 seconds + And postgres-0 is registered in the postgres-2 as the secondary in group 0 after 15 seconds + And "sync" key in a group 0 in DCS has sync_standby=postgres-0 after 15 seconds + When I run patronictl.py switchover batman --group 0 --candidate postgres-0 --force + Then postgres-0 role is the primary after 10 seconds + And replication works from postgres-0 to postgres-1 after 15 seconds + And postgres-0 is registered in the postgres-2 as the primary in group 0 after 5 seconds + And postgres-1 is registered in the postgres-2 as the secondary in group 0 after 15 seconds + And "sync" key in a group 0 in DCS has sync_standby=postgres-1 after 15 seconds Scenario: worker switchover doesn't break client queries on the coordinator - Given I create a distributed table on postgres0 - And I start a thread inserting data on postgres0 + Given I create a distributed table on postgres-0 + And I start a thread inserting data on postgres-0 When I run patronictl.py switchover batman --group 1 --force Then I receive a response returncode 0 - And postgres3 role is the primary after 10 seconds - And "members/postgres2" key in a group 1 in DCS has state=running after 15 seconds - And replication works from postgres3 to postgres2 after 15 seconds - And postgres3 is registered in the postgres0 as the primary in group 1 after 5 seconds - And postgres2 is registered in the postgres0 as the secondary in group 1 after 15 seconds - And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds + And postgres-3 role is the primary after 10 seconds + And "members/postgres-2" key in a group 1 in DCS has state=running after 15 seconds + And replication works from postgres-3 to postgres-2 after 15 seconds + And postgres-3 is registered in the postgres-0 as the primary in group 1 after 5 seconds + And postgres-2 is registered in the postgres-0 as the secondary in group 1 after 15 seconds + And "sync" key in a group 1 in DCS has sync_standby=postgres-2 after 15 seconds And a thread is still alive When I run patronictl.py switchover batman --group 1 --force Then I receive a response returncode 0 - And postgres2 role is the primary after 10 seconds - And replication works from postgres2 to postgres3 after 15 seconds - And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds - And postgres3 is registered in the postgres0 as the secondary in group 1 after 15 seconds - And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds + And postgres-2 role is the primary after 10 seconds + And replication works from postgres-2 to postgres-3 after 15 seconds + And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds + And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 15 seconds + And "sync" key in a group 1 in DCS has sync_standby=postgres-3 after 15 seconds And a thread is still alive When I stop a thread - Then a distributed table on postgres0 has expected rows + Then a distributed table on postgres-0 has expected rows Scenario: worker primary restart doesn't break client queries on the coordinator - Given I cleanup a distributed table on postgres0 - And I start a thread inserting data on postgres0 - When I run patronictl.py restart batman postgres2 --group 1 --force + Given I cleanup a distributed table on postgres-0 + And I start a thread inserting data on postgres-0 + When I run patronictl.py restart batman postgres-2 --group 1 --force Then I receive a response returncode 0 - And postgres2 role is the primary after 10 seconds - And replication works from postgres2 to postgres3 after 15 seconds - And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds - And postgres3 is registered in the postgres0 as the secondary in group 1 after 15 seconds + And postgres-2 role is the primary after 10 seconds + And replication works from postgres-2 to postgres-3 after 15 seconds + And postgres-2 is registered in the postgres-0 as the primary in group 1 after 5 seconds + And postgres-3 is registered in the postgres-0 as the secondary in group 1 after 15 seconds And a thread is still alive When I stop a thread - Then a distributed table on postgres0 has expected rows + Then a distributed table on postgres-0 has expected rows Scenario: check that in-flight transaction is rolled back after timeout when other workers need to change pg_dist_node - Given I start postgres4 in citus group 2 - Then postgres4 is a leader in a group 2 after 10 seconds - And "members/postgres4" key in a group 2 in DCS has role=primary after 3 seconds + Given I start postgres-4 in citus group 2 + Then postgres-4 is a leader in a group 2 after 10 seconds + And "members/postgres-4" key in a group 2 in DCS has role=primary after 3 seconds When I run patronictl.py edit-config batman --group 2 -s ttl=20 --force Then I receive a response returncode 0 And I receive a response output "+ttl: 20" - Then postgres4 is registered in the postgres2 as the primary in group 2 after 5 seconds - When I shut down postgres4 - Then there is a transaction in progress on postgres0 changing pg_dist_node after 5 seconds - When I run patronictl.py restart batman postgres2 --group 1 --force + Then postgres-4 is registered in the postgres-2 as the primary in group 2 after 5 seconds + When I shut down postgres-4 + Then there is a transaction in progress on postgres-0 changing pg_dist_node after 5 seconds + When I run patronictl.py restart batman postgres-2 --group 1 --force Then a transaction finishes in 20 seconds diff --git a/features/custom_bootstrap.feature b/features/custom_bootstrap.feature index 2c8671f7..8f24b862 100644 --- a/features/custom_bootstrap.feature +++ b/features/custom_bootstrap.feature @@ -2,16 +2,16 @@ Feature: custom bootstrap We should check that patroni can bootstrap a new cluster from a backup Scenario: clone existing cluster using pg_basebackup - Given I start postgres0 - Then postgres0 is a leader after 10 seconds - When I add the table foo to postgres0 - And I start postgres1 in a cluster batman1 as a clone of postgres0 - Then postgres1 is a leader of batman1 after 10 seconds - Then table foo is present on postgres1 after 10 seconds + Given I start postgres-0 + Then postgres-0 is a leader after 10 seconds + When I add the table foo to postgres-0 + And I start postgres-1 in a cluster batman1 as a clone of postgres-0 + Then postgres-1 is a leader of batman1 after 10 seconds + Then table foo is present on postgres-1 after 10 seconds Scenario: make a backup and do a restore into a new cluster - Given I add the table bar to postgres1 - And I do a backup of postgres1 - When I start postgres2 in a cluster batman2 from backup - Then postgres2 is a leader of batman2 after 30 seconds - And table bar is present on postgres2 after 10 seconds + Given I add the table bar to postgres-1 + And I do a backup of postgres-1 + When I start postgres-2 in a cluster batman2 from backup + Then postgres-2 is a leader of batman2 after 30 seconds + And table bar is present on postgres-2 after 10 seconds diff --git a/features/dcs_failsafe_mode.feature b/features/dcs_failsafe_mode.feature index 21feb655..ca36e899 100644 --- a/features/dcs_failsafe_mode.feature +++ b/features/dcs_failsafe_mode.feature @@ -2,16 +2,16 @@ Feature: dcs failsafe mode We should check the basic dcs failsafe mode functioning Scenario: check failsafe mode can be successfully enabled - Given I start postgres0 - And postgres0 is a leader after 10 seconds + Given I start postgres-0 + And postgres-0 is a leader after 10 seconds Then "config" key in DCS has ttl=30 after 10 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 3, "failsafe_mode": true} Then I receive a response code 200 - And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds + And Response on GET http://127.0.0.1:8008/failsafe contains postgres-0 after 10 seconds When I issue a GET request to http://127.0.0.1:8008/failsafe Then I receive a response code 200 - And I receive a response postgres0 http://127.0.0.1:8008/patroni - When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}},"slots":{"dcs_slot_1": null,"postgres0":null}} + And I receive a response postgres-0 http://127.0.0.1:8008/patroni + When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}},"slots":{"dcs_slot_1": null,"postgres_0":null}} Then I receive a response code 200 When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"dcs_slot_0": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}} Then I receive a response code 200 @@ -20,99 +20,99 @@ Feature: dcs failsafe mode Scenario: check one-node cluster is functioning while DCS is down Given DCS is down Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds - And postgres0 role is the primary after 10 seconds + And postgres-0 role is the primary after 10 seconds @dcs-failsafe Scenario: check new replica isn't promoted when leader is down and DCS is up Given DCS is up - When I do a backup of postgres0 - And I shut down postgres0 - When I start postgres1 in a cluster batman from backup with no_leader - Then postgres1 role is the replica after 12 seconds + When I do a backup of postgres-0 + And I shut down postgres-0 + When I start postgres-1 in a cluster batman from backup with no_leader + Then postgres-1 role is the replica after 12 seconds Scenario: check leader and replica are both in /failsafe key after leader is back - Given I start postgres0 - And I start postgres1 - Then "members/postgres0" key in DCS has state=running after 10 seconds - And "members/postgres1" key in DCS has state=running after 2 seconds - And Response on GET http://127.0.0.1:8009/failsafe contains postgres1 after 10 seconds + Given I start postgres-0 + And I start postgres-1 + Then "members/postgres-0" key in DCS has state=running after 10 seconds + And "members/postgres-1" key in DCS has state=running after 2 seconds + And Response on GET http://127.0.0.1:8009/failsafe contains postgres-1 after 10 seconds When I issue a GET request to http://127.0.0.1:8009/failsafe Then I receive a response code 200 - And I receive a response postgres0 http://127.0.0.1:8008/patroni - And I receive a response postgres1 http://127.0.0.1:8009/patroni + And I receive a response postgres-0 http://127.0.0.1:8008/patroni + And I receive a response postgres-1 http://127.0.0.1:8009/patroni @dcs-failsafe @slot-advance Scenario: check leader and replica are functioning while DCS is down - Given I get all changes from physical slot dcs_slot_1 on postgres0 - Then physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds - And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 10 seconds + Given I get all changes from physical slot dcs_slot_1 on postgres-0 + Then physical slot dcs_slot_1 is in sync between postgres-0 and postgres-1 after 10 seconds + And logical slot dcs_slot_0 is in sync between postgres-0 and postgres-1 after 10 seconds And DCS is down Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds - Then postgres0 role is the primary after 10 seconds - And postgres1 role is the replica after 2 seconds - And replication works from postgres0 to postgres1 after 10 seconds - When I get all changes from logical slot dcs_slot_0 on postgres0 - And I get all changes from physical slot dcs_slot_1 on postgres0 - Then logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds - And physical slot dcs_slot_1 is in sync between postgres0 and postgres1 after 10 seconds + Then postgres-0 role is the primary after 10 seconds + And postgres-1 role is the replica after 2 seconds + And replication works from postgres-0 to postgres-1 after 10 seconds + When I get all changes from logical slot dcs_slot_0 on postgres-0 + And I get all changes from physical slot dcs_slot_1 on postgres-0 + Then logical slot dcs_slot_0 is in sync between postgres-0 and postgres-1 after 20 seconds + And physical slot dcs_slot_1 is in sync between postgres-0 and postgres-1 after 10 seconds @dcs-failsafe Scenario: check primary is demoted when one replica is shut down and DCS is down Given DCS is down - And I kill postgres1 - And I kill postmaster on postgres1 - Then postgres0 role is the replica after 12 seconds + And I kill postgres-1 + And I kill postmaster on postgres-1 + Then postgres-0 role is the replica after 12 seconds @dcs-failsafe Scenario: check known replica is promoted when leader is down and DCS is up - Given I kill postgres0 - And I shut down postmaster on postgres0 + Given I kill postgres-0 + And I shut down postmaster on postgres-0 And DCS is up - When I start postgres1 - Then "members/postgres1" key in DCS has state=running after 10 seconds - And postgres1 role is the primary after 25 seconds + When I start postgres-1 + Then "members/postgres-1" key in DCS has state=running after 10 seconds + And postgres-1 role is the primary after 25 seconds @dcs-failsafe Scenario: scale to three-node cluster - Given I start postgres0 - And I configure and start postgres2 with a tag replicatefrom postgres0 - Then "members/postgres2" key in DCS has state=running after 10 seconds - And "members/postgres0" key in DCS has state=running after 20 seconds - And Response on GET http://127.0.0.1:8008/failsafe contains postgres2 after 10 seconds - And replication works from postgres1 to postgres0 after 10 seconds - And replication works from postgres1 to postgres2 after 10 seconds + Given I start postgres-0 + And I configure and start postgres-2 with a tag replicatefrom postgres-0 + Then "members/postgres-2" key in DCS has state=running after 10 seconds + And "members/postgres-0" key in DCS has state=running after 20 seconds + And Response on GET http://127.0.0.1:8008/failsafe contains postgres-2 after 10 seconds + And replication works from postgres-1 to postgres-0 after 10 seconds + And replication works from postgres-1 to postgres-2 after 10 seconds @dcs-failsafe @slot-advance Scenario: make sure permanent slots exist on replicas Given I issue a PATCH request to http://127.0.0.1:8009/config with {"slots":{"dcs_slot_0":null,"dcs_slot_2":{"type":"logical","database":"postgres","plugin":"test_decoding"}}} - Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds - And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds - When I get all changes from physical slot dcs_slot_1 on postgres1 - Then physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds - And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds - And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds - And physical slot postgres2 is in sync between postgres0 and postgres1 after 10 seconds + Then logical slot dcs_slot_2 is in sync between postgres-1 and postgres-0 after 20 seconds + And logical slot dcs_slot_2 is in sync between postgres-1 and postgres-2 after 20 seconds + When I get all changes from physical slot dcs_slot_1 on postgres-1 + Then physical slot dcs_slot_1 is in sync between postgres-1 and postgres-0 after 10 seconds + And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-2 after 10 seconds + And physical slot postgres_0 is in sync between postgres-1 and postgres-2 after 10 seconds + And physical slot postgres_2 is in sync between postgres-0 and postgres-1 after 10 seconds @dcs-failsafe Scenario: check three-node cluster is functioning while DCS is down Given DCS is down Then Response on GET http://127.0.0.1:8009/primary contains failsafe_mode_is_active after 12 seconds - Then postgres1 role is the primary after 10 seconds - And postgres0 role is the replica after 2 seconds - And postgres2 role is the replica after 2 seconds + Then postgres-1 role is the primary after 10 seconds + And postgres-0 role is the replica after 2 seconds + And postgres-2 role is the replica after 2 seconds @dcs-failsafe @slot-advance Scenario: check that permanent slots are in sync between nodes while DCS is down - Given replication works from postgres1 to postgres0 after 10 seconds - And replication works from postgres1 to postgres2 after 10 seconds - When I get all changes from logical slot dcs_slot_2 on postgres1 - And I get all changes from physical slot dcs_slot_1 on postgres1 - Then logical slot dcs_slot_2 is in sync between postgres1 and postgres0 after 20 seconds - And logical slot dcs_slot_2 is in sync between postgres1 and postgres2 after 20 seconds - And physical slot dcs_slot_1 is in sync between postgres1 and postgres0 after 10 seconds - And physical slot dcs_slot_1 is in sync between postgres1 and postgres2 after 10 seconds - And physical slot postgres0 is in sync between postgres1 and postgres2 after 10 seconds - And physical slot postgres2 is in sync between postgres0 and postgres1 after 10 seconds + Given replication works from postgres-1 to postgres-0 after 10 seconds + And replication works from postgres-1 to postgres-2 after 10 seconds + When I get all changes from logical slot dcs_slot_2 on postgres-1 + And I get all changes from physical slot dcs_slot_1 on postgres-1 + Then logical slot dcs_slot_2 is in sync between postgres-1 and postgres-0 after 20 seconds + And logical slot dcs_slot_2 is in sync between postgres-1 and postgres-2 after 20 seconds + And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-0 after 10 seconds + And physical slot dcs_slot_1 is in sync between postgres-1 and postgres-2 after 10 seconds + And physical slot postgres_0 is in sync between postgres-1 and postgres-2 after 10 seconds + And physical slot postgres_2 is in sync between postgres-0 and postgres-1 after 10 seconds diff --git a/features/ignored_slots.feature b/features/ignored_slots.feature index 470fc349..422c650b 100644 --- a/features/ignored_slots.feature +++ b/features/ignored_slots.feature @@ -1,61 +1,61 @@ Feature: ignored slots Scenario: check ignored slots aren't removed on failover/switchover - Given I start postgres1 - Then postgres1 is a leader after 10 seconds + Given I start postgres-1 + Then postgres-1 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds When I issue a PATCH request to http://127.0.0.1:8009/config with {"ignore_slots": [{"name": "unmanaged_slot_0", "database": "postgres", "plugin": "test_decoding", "type": "logical"}, {"name": "unmanaged_slot_1", "database": "postgres", "plugin": "test_decoding"}, {"name": "unmanaged_slot_2", "database": "postgres"}, {"name": "unmanaged_slot_3"}], "postgresql": {"parameters": {"wal_level": "logical"}}} Then I receive a response code 200 And Response on GET http://127.0.0.1:8009/config contains ignore_slots after 10 seconds # Make sure the wal_level has been changed. - When I shut down postgres1 - And I start postgres1 - Then postgres1 is a leader after 10 seconds - And "members/postgres1" key in DCS has role=primary after 10 seconds + When I shut down postgres-1 + And I start postgres-1 + Then postgres-1 is a leader after 10 seconds + And "members/postgres-1" key in DCS has role=primary after 10 seconds # Make sure Patroni has finished telling Postgres it should be accepting writes. - And postgres1 role is the primary after 20 seconds + And postgres-1 role is the primary after 20 seconds # 1. Create our test logical replication slot. # Test that ny subset of attributes in the ignore slots matcher is enough to match a slot # by using 3 different slots. - When I create a logical replication slot unmanaged_slot_0 on postgres1 with the test_decoding plugin - And I create a logical replication slot unmanaged_slot_1 on postgres1 with the test_decoding plugin - And I create a logical replication slot unmanaged_slot_2 on postgres1 with the test_decoding plugin - And I create a logical replication slot unmanaged_slot_3 on postgres1 with the test_decoding plugin - And I create a logical replication slot dummy_slot on postgres1 with the test_decoding plugin + When I create a logical replication slot unmanaged_slot_0 on postgres-1 with the test_decoding plugin + And I create a logical replication slot unmanaged_slot_1 on postgres-1 with the test_decoding plugin + And I create a logical replication slot unmanaged_slot_2 on postgres-1 with the test_decoding plugin + And I create a logical replication slot unmanaged_slot_3 on postgres-1 with the test_decoding plugin + And I create a logical replication slot dummy_slot on postgres-1 with the test_decoding plugin # It seems like it'd be obvious that these slots exist since we just created them, # but Patroni can actually end up dropping them almost immediately, so it's helpful # to verify they exist before we begin testing whether they persist through failover # cycles. - Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds + Then postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds - When I start postgres0 - Then "members/postgres0" key in DCS has role=replica after 10 seconds - And postgres0 role is the secondary after 20 seconds + When I start postgres-0 + Then "members/postgres-0" key in DCS has role=replica after 10 seconds + And postgres-0 role is the secondary after 20 seconds # Verify that the replica has advanced beyond the point in the WAL # where we created the replication slot so that on the next failover # cycle we don't accidentally rewind to before the slot creation. - And replication works from postgres1 to postgres0 after 20 seconds - When I shut down postgres1 - Then "members/postgres0" key in DCS has role=primary after 10 seconds + And replication works from postgres-1 to postgres-0 after 20 seconds + When I shut down postgres-1 + Then "members/postgres-0" key in DCS has role=primary after 10 seconds # 2. After a failover the server (now a replica) still has the slot. - When I start postgres1 - Then postgres1 role is the secondary after 20 seconds - And "members/postgres1" key in DCS has role=replica after 10 seconds + When I start postgres-1 + Then postgres-1 role is the secondary after 20 seconds + And "members/postgres-1" key in DCS has role=replica after 10 seconds # give Patroni time to sync replication slots And I sleep for 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds - And postgres1 does not have a replication slot named dummy_slot + And postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds + And postgres-1 does not have a replication slot named dummy_slot # 3. After a failover the server (now a primary) still has the slot. - When I shut down postgres0 - Then "members/postgres1" key in DCS has role=primary after 10 seconds - And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds - And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds + When I shut down postgres-0 + Then "members/postgres-1" key in DCS has role=primary after 10 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds + And postgres-1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds diff --git a/features/nostream_node.feature b/features/nostream_node.feature index d9d2b87b..da9d4163 100644 --- a/features/nostream_node.feature +++ b/features/nostream_node.feature @@ -1,26 +1,26 @@ Feature: nostream node Scenario: check nostream node is recovering from archive - When I start postgres0 - And I configure and start postgres1 with a tag nostream true - Then "members/postgres1" key in DCS has replication_state=in archive recovery after 10 seconds - And replication works from postgres0 to postgres1 after 30 seconds + When I start postgres-0 + And I configure and start postgres-1 with a tag nostream true + Then "members/postgres-1" key in DCS has replication_state=in archive recovery after 10 seconds + And replication works from postgres-0 to postgres-1 after 30 seconds @slot-advance Scenario: check permanent logical replication slots are not copied When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}}, "slots":{"test_logical":{"type":"logical","database":"postgres","plugin":"test_decoding"}}} Then I receive a response code 200 - When I run patronictl.py restart batman postgres0 --force - Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds - When I configure and start postgres2 with a tag replicatefrom postgres1 - Then "members/postgres2" key in DCS has replication_state=streaming after 10 seconds - And postgres1 does not have a replication slot named test_logical - And postgres2 does not have a replication slot named test_logical + When I run patronictl.py restart batman postgres-0 --force + Then postgres-0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds + When I configure and start postgres-2 with a tag replicatefrom postgres-1 + Then "members/postgres-2" key in DCS has replication_state=streaming after 10 seconds + And postgres-1 does not have a replication slot named test_logical + And postgres-2 does not have a replication slot named test_logical @slot-advance Scenario: check that slots are written to the /status key - Given "status" key in DCS has postgres0 in slots - And "status" key in DCS has postgres2 in slots + Given "status" key in DCS has postgres_0 in slots + And "status" key in DCS has postgres_2 in slots And "status" key in DCS has test_logical in slots And "status" key in DCS has test_logical in slots - And "status" key in DCS does not have postgres1 in slots + And "status" key in DCS does not have postgres_1 in slots diff --git a/features/patroni_api.feature b/features/patroni_api.feature index cb47e689..6b523237 100644 --- a/features/patroni_api.feature +++ b/features/patroni_api.feature @@ -2,8 +2,8 @@ Feature: patroni api We should check that patroni correctly responds to valid and not-valid API requests. Scenario: check API requests on a stand-alone server - Given I start postgres0 - And postgres0 is a leader after 10 seconds + Given I start postgres-0 + And postgres-0 is a leader after 10 seconds When I issue a GET request to http://127.0.0.1:8008/ Then I receive a response code 200 And I receive a response state running @@ -17,10 +17,10 @@ Scenario: check API requests on a stand-alone server When I issue a POST request to http://127.0.0.1:8008/reinitialize with {"force": true} Then I receive a response code 503 And I receive a response text I am the leader, can not reinitialize - When I run patronictl.py switchover batman --primary postgres0 --force + When I run patronictl.py switchover batman --primary postgres-0 --force Then I receive a response returncode 1 And I receive a response output "Error: No candidates found to switchover to" - When I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0"} + When I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres-0"} Then I receive a response code 412 And I receive a response text switchover is not possible: cluster does not have members except leader When I issue an empty POST request to http://127.0.0.1:8008/failover @@ -30,7 +30,7 @@ Scenario: check API requests on a stand-alone server And I receive a response text "Failover could be performed only to a specific candidate" Scenario: check local configuration reload - Given I add tag new_tag new_value to postgres0 config + Given I add tag new_tag new_value to postgres-0 config And I issue an empty POST request to http://127.0.0.1:8008/reload Then I receive a response code 202 @@ -58,43 +58,43 @@ Scenario: check the scheduled restart Given I issue a scheduled restart at http://127.0.0.1:8008 in 5 seconds with {"restart_pending": "True"} Then I receive a response code 202 And Response on GET http://127.0.0.1:8008/patroni does not contain pending_restart after 10 seconds - And postgres0 role is the primary after 10 seconds + And postgres-0 role is the primary after 10 seconds Scenario: check API requests for the primary-replica pair in the pause mode - Given I start postgres1 - Then replication works from postgres0 to postgres1 after 20 seconds + Given I start postgres-1 + Then replication works from postgres-0 to postgres-1 after 20 seconds When I run patronictl.py pause batman Then I receive a response returncode 0 - When I kill postmaster on postgres1 + When I kill postmaster on postgres-1 And I issue a GET request to http://127.0.0.1:8009/replica Then I receive a response code 503 - And "members/postgres1" key in DCS has state=stopped after 10 seconds - When I run patronictl.py restart batman postgres1 --force + And "members/postgres-1" key in DCS has state=stopped after 10 seconds + When I run patronictl.py restart batman postgres-1 --force Then I receive a response returncode 0 - Then replication works from postgres0 to postgres1 after 20 seconds + Then replication works from postgres-0 to postgres-1 after 20 seconds And I sleep for 2 seconds When I issue a GET request to http://127.0.0.1:8009/replica Then I receive a response code 200 And I receive a response state running And I receive a response role replica - When I run patronictl.py reinit batman postgres1 --force --wait + When I run patronictl.py reinit batman postgres-1 --force --wait Then I receive a response returncode 0 - And I receive a response output "Success: reinitialize for member postgres1" - And postgres1 role is the secondary after 30 seconds - And replication works from postgres0 to postgres1 after 20 seconds - When I run patronictl.py restart batman postgres0 --force + And I receive a response output "Success: reinitialize for member postgres-1" + And postgres-1 role is the secondary after 30 seconds + And replication works from postgres-0 to postgres-1 after 20 seconds + When I run patronictl.py restart batman postgres-0 --force Then I receive a response returncode 0 - And I receive a response output "Success: restart on member postgres0" - And postgres0 role is the primary after 5 seconds + And I receive a response output "Success: restart on member postgres-0" + And postgres-0 role is the primary after 5 seconds Scenario: check the switchover via the API in the pause mode - Given I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres0", "candidate": "postgres1"} + Given I issue a POST request to http://127.0.0.1:8008/switchover with {"leader": "postgres-0", "candidate": "postgres-1"} Then I receive a response code 200 - And postgres1 is a leader after 5 seconds - And postgres1 role is the primary after 10 seconds - And postgres0 role is the secondary after 10 seconds - And replication works from postgres1 to postgres0 after 20 seconds - And "members/postgres0" key in DCS has state=running after 10 seconds + And postgres-1 is a leader after 5 seconds + And postgres-1 role is the primary after 10 seconds + And postgres-0 role is the secondary after 10 seconds + And replication works from postgres-1 to postgres-0 after 20 seconds + And "members/postgres-0" key in DCS has state=running after 10 seconds When I issue a GET request to http://127.0.0.1:8008/primary Then I receive a response code 503 When I issue a GET request to http://127.0.0.1:8008/replica @@ -105,18 +105,18 @@ Scenario: check the switchover via the API in the pause mode Then I receive a response code 503 Scenario: check the scheduled switchover - Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds + Given I issue a scheduled switchover from postgres-1 to postgres-0 in 10 seconds Then I receive a response returncode 1 And I receive a response output "Can't schedule switchover in the paused state" When I run patronictl.py resume batman Then I receive a response returncode 0 - Given I issue a scheduled switchover from postgres1 to postgres0 in 10 seconds + Given I issue a scheduled switchover from postgres-1 to postgres-0 in 10 seconds Then I receive a response returncode 0 - And postgres0 is a leader after 20 seconds - And postgres0 role is the primary after 10 seconds - And postgres1 role is the secondary after 10 seconds - And replication works from postgres0 to postgres1 after 25 seconds - And "members/postgres1" key in DCS has state=running after 10 seconds + And postgres-0 is a leader after 20 seconds + And postgres-0 role is the primary after 10 seconds + And postgres-1 role is the secondary after 10 seconds + And replication works from postgres-0 to postgres-1 after 25 seconds + And "members/postgres-1" key in DCS has state=running after 10 seconds When I issue a GET request to http://127.0.0.1:8008/primary Then I receive a response code 200 When I issue a GET request to http://127.0.0.1:8008/replica diff --git a/features/permanent_slots.feature b/features/permanent_slots.feature index 216ff01a..36ae36e9 100644 --- a/features/permanent_slots.feature +++ b/features/permanent_slots.feature @@ -1,87 +1,87 @@ Feature: permanent slots Scenario: check that physical permanent slots are created - Given I start postgres0 - Then postgres0 is a leader after 10 seconds + Given I start postgres-0 + Then postgres-0 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds - When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres3":0},"postgresql":{"parameters":{"wal_level":"logical"}}} + When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_physical":0,"postgres_3":0},"postgresql":{"parameters":{"wal_level":"logical"}}} Then I receive a response code 200 And Response on GET http://127.0.0.1:8008/config contains slots after 10 seconds - When I start postgres1 - And I start postgres2 - And I configure and start postgres3 with a tag replicatefrom postgres2 - Then postgres0 has a physical replication slot named test_physical after 10 seconds - And postgres0 has a physical replication slot named postgres1 after 10 seconds - And postgres0 has a physical replication slot named postgres2 after 10 seconds - And postgres2 has a physical replication slot named postgres3 after 10 seconds + When I start postgres-1 + And I start postgres-2 + And I configure and start postgres-3 with a tag replicatefrom postgres-2 + Then postgres-0 has a physical replication slot named test_physical after 10 seconds + And postgres-0 has a physical replication slot named postgres_1 after 10 seconds + And postgres-0 has a physical replication slot named postgres_2 after 10 seconds + And postgres-2 has a physical replication slot named postgres_3 after 10 seconds @slot-advance Scenario: check that logical permanent slots are created - Given I run patronictl.py restart batman postgres0 --force + Given I run patronictl.py restart batman postgres-0 --force And I issue a PATCH request to http://127.0.0.1:8008/config with {"slots":{"test_logical":{"type":"logical","database":"postgres","plugin":"test_decoding"}}} - Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds + Then postgres-0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds @slot-advance Scenario: check that permanent slots are created on replicas - Given postgres1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds - Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds - And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds - And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds - And postgres1 has a physical replication slot named test_physical after 2 seconds - And postgres2 has a physical replication slot named test_physical after 2 seconds - And postgres3 has a physical replication slot named test_physical after 2 seconds + Given postgres-1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds + Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds + And Logical slot test_logical is in sync between postgres-0 and postgres-2 after 10 seconds + And Logical slot test_logical is in sync between postgres-0 and postgres-3 after 10 seconds + And postgres-1 has a physical replication slot named test_physical after 2 seconds + And postgres-2 has a physical replication slot named test_physical after 2 seconds + And postgres-3 has a physical replication slot named test_physical after 2 seconds @slot-advance Scenario: check permanent physical slots that match with member names - Given postgres0 has a physical replication slot named postgres3 after 2 seconds - And postgres1 has a physical replication slot named postgres0 after 2 seconds - And postgres1 has a physical replication slot named postgres2 after 2 seconds - And postgres1 has a physical replication slot named postgres3 after 2 seconds - And postgres2 has a physical replication slot named postgres0 after 2 seconds - And postgres2 has a physical replication slot named postgres3 after 2 seconds - And postgres2 has a physical replication slot named postgres1 after 2 seconds - And postgres3 has a physical replication slot named postgres0 after 2 seconds - And postgres3 has a physical replication slot named postgres1 after 2 seconds - And postgres3 has a physical replication slot named postgres2 after 2 seconds + Given postgres-0 has a physical replication slot named postgres_3 after 2 seconds + And postgres-1 has a physical replication slot named postgres_0 after 2 seconds + And postgres-1 has a physical replication slot named postgres_2 after 2 seconds + And postgres-1 has a physical replication slot named postgres_3 after 2 seconds + And postgres-2 has a physical replication slot named postgres_0 after 2 seconds + And postgres-2 has a physical replication slot named postgres_3 after 2 seconds + And postgres-2 has a physical replication slot named postgres_1 after 2 seconds + And postgres-3 has a physical replication slot named postgres_0 after 2 seconds + And postgres-3 has a physical replication slot named postgres_1 after 2 seconds + And postgres-3 has a physical replication slot named postgres_2 after 2 seconds @slot-advance Scenario: check that permanent slots are advanced on replicas - Given I add the table replicate_me to postgres0 - When I get all changes from logical slot test_logical on postgres0 - And I get all changes from physical slot test_physical on postgres0 - Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds - And Physical slot test_physical is in sync between postgres0 and postgres1 after 10 seconds - And Logical slot test_logical is in sync between postgres0 and postgres2 after 10 seconds - And Physical slot test_physical is in sync between postgres0 and postgres2 after 10 seconds - And Logical slot test_logical is in sync between postgres0 and postgres3 after 10 seconds - And Physical slot test_physical is in sync between postgres0 and postgres3 after 10 seconds - And Physical slot postgres1 is in sync between postgres0 and postgres2 after 10 seconds - And Physical slot postgres1 is in sync between postgres0 and postgres3 after 10 seconds - And Physical slot postgres3 is in sync between postgres2 and postgres0 after 20 seconds - And Physical slot postgres3 is in sync between postgres2 and postgres1 after 10 seconds + Given I add the table replicate_me to postgres-0 + When I get all changes from logical slot test_logical on postgres-0 + And I get all changes from physical slot test_physical on postgres-0 + Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds + And Physical slot test_physical is in sync between postgres-0 and postgres-1 after 10 seconds + And Logical slot test_logical is in sync between postgres-0 and postgres-2 after 10 seconds + And Physical slot test_physical is in sync between postgres-0 and postgres-2 after 10 seconds + And Logical slot test_logical is in sync between postgres-0 and postgres-3 after 10 seconds + And Physical slot test_physical is in sync between postgres-0 and postgres-3 after 10 seconds + And Physical slot postgres_1 is in sync between postgres-0 and postgres-2 after 10 seconds + And Physical slot postgres_1 is in sync between postgres-0 and postgres-3 after 10 seconds + And Physical slot postgres_3 is in sync between postgres-2 and postgres-0 after 20 seconds + And Physical slot postgres_3 is in sync between postgres-2 and postgres-1 after 10 seconds @slot-advance Scenario: check that permanent slots and member slots are written to the /status key Given "status" key in DCS has test_physical in slots - And "status" key in DCS has postgres0 in slots - And "status" key in DCS has postgres1 in slots - And "status" key in DCS has postgres2 in slots - And "status" key in DCS has postgres3 in slots + And "status" key in DCS has postgres_0 in slots + And "status" key in DCS has postgres_1 in slots + And "status" key in DCS has postgres_2 in slots + And "status" key in DCS has postgres_3 in slots @slot-advance Scenario: check that only non-permanent member slots are written to the retain_slots in /status key - And "status" key in DCS has postgres0 in retain_slots - And "status" key in DCS has postgres1 in retain_slots - And "status" key in DCS has postgres2 in retain_slots - And "status" key in DCS does not have postgres3 in retain_slots + And "status" key in DCS has postgres_0 in retain_slots + And "status" key in DCS has postgres_1 in retain_slots + And "status" key in DCS has postgres_2 in retain_slots + And "status" key in DCS does not have postgres_3 in retain_slots Scenario: check permanent physical replication slot after failover - Given I shut down postgres3 - And I shut down postgres2 - And I shut down postgres0 - Then postgres1 has a physical replication slot named test_physical after 10 seconds - And postgres1 has a physical replication slot named postgres0 after 10 seconds - And postgres1 has a physical replication slot named postgres3 after 10 seconds - When I start postgres0 - Then postgres0 role is the replica after 20 seconds - And physical replication slot named postgres1 on postgres0 has no xmin value after 10 seconds - And physical replication slot named postgres2 on postgres0 has no xmin value after 10 seconds + Given I shut down postgres-3 + And I shut down postgres-2 + And I shut down postgres-0 + Then postgres-1 has a physical replication slot named test_physical after 10 seconds + And postgres-1 has a physical replication slot named postgres_0 after 10 seconds + And postgres-1 has a physical replication slot named postgres_3 after 10 seconds + When I start postgres-0 + Then postgres-0 role is the replica after 20 seconds + And physical replication slot named postgres_1 on postgres-0 has no xmin value after 10 seconds + And physical replication slot named postgres_2 on postgres-0 has no xmin value after 10 seconds diff --git a/features/priority_failover.feature b/features/priority_failover.feature index 1737ae39..f11371e5 100644 --- a/features/priority_failover.feature +++ b/features/priority_failover.feature @@ -2,38 +2,38 @@ Feature: priority replication We should check that we can give nodes priority during failover Scenario: check failover priority 0 prevents leaderships - Given I configure and start postgres0 with a tag failover_priority 1 - And I configure and start postgres1 with a tag failover_priority 0 - Then replication works from postgres0 to postgres1 after 20 seconds - When I shut down postgres0 - And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres1 patroni log after 5 seconds - Then postgres1 role is the secondary after 10 seconds - When I start postgres0 - Then postgres0 role is the primary after 10 seconds + Given I configure and start postgres-0 with a tag failover_priority 1 + And I configure and start postgres-1 with a tag failover_priority 0 + Then replication works from postgres-0 to postgres-1 after 20 seconds + When I shut down postgres-0 + And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres-1 patroni log after 5 seconds + Then postgres-1 role is the secondary after 10 seconds + When I start postgres-0 + Then postgres-0 role is the primary after 10 seconds Scenario: check higher failover priority is respected - Given I configure and start postgres2 with a tag failover_priority 1 - And I configure and start postgres3 with a tag failover_priority 2 - Then replication works from postgres0 to postgres2 after 20 seconds - And replication works from postgres0 to postgres3 after 20 seconds - When I shut down postgres0 - Then postgres3 role is the primary after 10 seconds - And there is one of ["postgres3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres3 is ahead of my wal position"] INFO in the postgres2 patroni log after 5 seconds + Given I configure and start postgres-2 with a tag failover_priority 1 + And I configure and start postgres-3 with a tag failover_priority 2 + Then replication works from postgres-0 to postgres-2 after 20 seconds + And replication works from postgres-0 to postgres-3 after 20 seconds + When I shut down postgres-0 + Then postgres-3 role is the primary after 10 seconds + And there is one of ["postgres-3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres-3 is ahead of my wal position"] INFO in the postgres-2 patroni log after 5 seconds Scenario: check conflicting configuration handling - When I set nofailover tag in postgres2 config + When I set nofailover tag in postgres-2 config And I issue an empty POST request to http://127.0.0.1:8010/reload Then I receive a response code 202 - And there is one of ["Conflicting configuration between nofailover: True and failover_priority: 1. Defaulting to nofailover: True"] WARNING in the postgres2 patroni log after 5 seconds - And "members/postgres2" key in DCS has tags={'failover_priority': '1', 'nofailover': True} after 10 seconds - When I issue a POST request to http://127.0.0.1:8010/failover with {"candidate": "postgres2"} + And there is one of ["Conflicting configuration between nofailover: True and failover_priority: 1. Defaulting to nofailover: True"] WARNING in the postgres-2 patroni log after 5 seconds + And "members/postgres-2" key in DCS has tags={'failover_priority': '1', 'nofailover': True} after 10 seconds + When I issue a POST request to http://127.0.0.1:8010/failover with {"candidate": "postgres-2"} Then I receive a response code 412 And I receive a response text "failover is not possible: no good candidates have been found" - When I reset nofailover tag in postgres1 config + When I reset nofailover tag in postgres-1 config And I issue an empty POST request to http://127.0.0.1:8009/reload Then I receive a response code 202 - And there is one of ["Conflicting configuration between nofailover: False and failover_priority: 0. Defaulting to nofailover: False"] WARNING in the postgres1 patroni log after 5 seconds - And "members/postgres1" key in DCS has tags={'failover_priority': '0', 'nofailover': False} after 10 seconds - And I issue a POST request to http://127.0.0.1:8009/failover with {"candidate": "postgres1"} + And there is one of ["Conflicting configuration between nofailover: False and failover_priority: 0. Defaulting to nofailover: False"] WARNING in the postgres-1 patroni log after 5 seconds + And "members/postgres-1" key in DCS has tags={'failover_priority': '0', 'nofailover': False} after 10 seconds + And I issue a POST request to http://127.0.0.1:8009/failover with {"candidate": "postgres-1"} Then I receive a response code 200 - And postgres1 role is the primary after 10 seconds + And postgres-1 role is the primary after 10 seconds diff --git a/features/quorum_commit.feature b/features/quorum_commit.feature index 2204ab59..ccfe80bf 100644 --- a/features/quorum_commit.feature +++ b/features/quorum_commit.feature @@ -2,53 +2,53 @@ Feature: quorum commit Check basic workfrlows when quorum commit is enabled Scenario: check enable quorum commit and that the only leader promotes after restart - Given I start postgres0 - Then postgres0 is a leader after 10 seconds + Given I start postgres-0 + Then postgres-0 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "synchronous_mode": "quorum"} Then I receive a response code 200 - And sync key in DCS has leader=postgres0 after 20 seconds + And sync key in DCS has leader=postgres-0 after 20 seconds And sync key in DCS has quorum=0 after 2 seconds - And synchronous_standby_names on postgres0 is set to "_empty_str_" after 2 seconds - When I shut down postgres0 - And sync key in DCS has leader=postgres0 after 2 seconds - When I start postgres0 - Then postgres0 role is the primary after 10 seconds + And synchronous_standby_names on postgres-0 is set to '_empty_str_' after 2 seconds + When I shut down postgres-0 + And sync key in DCS has leader=postgres-0 after 2 seconds + When I start postgres-0 + Then postgres-0 role is the primary after 10 seconds When I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_mode_strict": true} - Then synchronous_standby_names on postgres0 is set to "ANY 1 (*)" after 10 seconds + Then synchronous_standby_names on postgres-0 is set to 'ANY 1 (*)' after 10 seconds Scenario: check failover with one quorum standby - Given I start postgres1 - Then sync key in DCS has sync_standby=postgres1 after 10 seconds - And synchronous_standby_names on postgres0 is set to "ANY 1 (postgres1)" after 2 seconds - When I shut down postgres0 - Then postgres1 role is the primary after 10 seconds + Given I start postgres-1 + Then sync key in DCS has sync_standby=postgres-1 after 10 seconds + And synchronous_standby_names on postgres-0 is set to 'ANY 1 ("postgres-1")' after 2 seconds + When I shut down postgres-0 + Then postgres-1 role is the primary after 10 seconds And sync key in DCS has quorum=0 after 10 seconds - Then synchronous_standby_names on postgres1 is set to "ANY 1 (*)" after 10 seconds - When I start postgres0 - Then sync key in DCS has leader=postgres1 after 10 seconds - Then sync key in DCS has sync_standby=postgres0 after 10 seconds - And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0)" after 2 seconds + Then synchronous_standby_names on postgres-1 is set to 'ANY 1 (*)' after 10 seconds + When I start postgres-0 + Then sync key in DCS has leader=postgres-1 after 10 seconds + Then sync key in DCS has sync_standby=postgres-0 after 10 seconds + And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0")' after 2 seconds Scenario: check behavior with three nodes and different replication factor - Given I start postgres2 - Then sync key in DCS has sync_standby=postgres0,postgres2 after 10 seconds + Given I start postgres-2 + Then sync key in DCS has sync_standby=postgres-0,postgres-2 after 10 seconds And sync key in DCS has quorum=1 after 2 seconds - And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0,postgres2)" after 2 seconds + And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0","postgres-2")' after 2 seconds When I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_node_count": 2} Then sync key in DCS has quorum=0 after 10 seconds - And synchronous_standby_names on postgres1 is set to "ANY 2 (postgres0,postgres2)" after 2 seconds + And synchronous_standby_names on postgres-1 is set to 'ANY 2 ("postgres-0","postgres-2")' after 2 seconds Scenario: switch from quorum replication to good old multisync and back Given I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_mode": true, "synchronous_node_count": 1} - And I shut down postgres0 - Then synchronous_standby_names on postgres1 is set to "postgres2" after 10 seconds - And sync key in DCS has sync_standby=postgres2 after 10 seconds + And I shut down postgres-0 + Then synchronous_standby_names on postgres-1 is set to '"postgres-2"' after 10 seconds + And sync key in DCS has sync_standby=postgres-2 after 10 seconds Then sync key in DCS has quorum=0 after 2 seconds When I issue a PATCH request to http://127.0.0.1:8009/config with {"synchronous_mode": "quorum"} - And I start postgres0 - Then synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0,postgres2)" after 10 seconds - And sync key in DCS has sync_standby=postgres0,postgres2 after 10 seconds + And I start postgres-0 + Then synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0","postgres-2")' after 10 seconds + And sync key in DCS has sync_standby=postgres-0,postgres-2 after 10 seconds Then sync key in DCS has quorum=1 after 2 seconds Scenario: REST API and patronictl @@ -59,10 +59,10 @@ Feature: quorum commit And Status code on GET http://127.0.0.1:8010/quorum is 200 after 3 seconds Scenario: nosync node is removed from voters and synchronous_standby_names - Given I add tag nosync true to postgres2 config + Given I add tag nosync true to postgres-2 config When I issue an empty POST request to http://127.0.0.1:8010/reload Then I receive a response code 202 And sync key in DCS has quorum=0 after 10 seconds - And sync key in DCS has sync_standby=postgres0 after 10 seconds - And synchronous_standby_names on postgres1 is set to "ANY 1 (postgres0)" after 2 seconds + And sync key in DCS has sync_standby=postgres-0 after 10 seconds + And synchronous_standby_names on postgres-1 is set to 'ANY 1 ("postgres-0")' after 2 seconds And Status code on GET http://127.0.0.1:8010/quorum is 503 after 10 seconds diff --git a/features/recovery.feature b/features/recovery.feature index 75ba5bc4..7e9774b8 100644 --- a/features/recovery.feature +++ b/features/recovery.feature @@ -2,25 +2,25 @@ Feature: recovery We want to check that crashed postgres is started back Scenario: check that timeline is not incremented when primary is started after crash - Given I start postgres0 - Then postgres0 is a leader after 10 seconds + Given I start postgres-0 + Then postgres-0 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds - When I start postgres1 - And I add the table foo to postgres0 - Then table foo is present on postgres1 after 20 seconds - When I kill postmaster on postgres0 - Then postgres0 role is the primary after 10 seconds + When I start postgres-1 + And I add the table foo to postgres-0 + Then table foo is present on postgres-1 after 20 seconds + When I kill postmaster on postgres-0 + Then postgres-0 role is the primary after 10 seconds When I issue a GET request to http://127.0.0.1:8008/ Then I receive a response code 200 And I receive a response role primary And I receive a response timeline 1 - And "members/postgres0" key in DCS has state=running after 12 seconds - And replication works from postgres0 to postgres1 after 15 seconds + And "members/postgres-0" key in DCS has state=running after 12 seconds + And replication works from postgres-0 to postgres-1 after 15 seconds Scenario: check immediate failover when master_start_timeout=0 Given I issue a PATCH request to http://127.0.0.1:8008/config with {"master_start_timeout": 0} Then I receive a response code 200 And Response on GET http://127.0.0.1:8008/config contains master_start_timeout after 10 seconds - When I kill postmaster on postgres0 - Then postgres1 is a leader after 10 seconds - And postgres1 role is the primary after 10 seconds + When I kill postmaster on postgres-0 + Then postgres-1 is a leader after 10 seconds + And postgres-1 role is the primary after 10 seconds diff --git a/features/standby_cluster.feature b/features/standby_cluster.feature index abc8e733..160ddd48 100644 --- a/features/standby_cluster.feature +++ b/features/standby_cluster.feature @@ -1,7 +1,7 @@ Feature: standby cluster Scenario: prepare the cluster with logical slots - Given I start postgres1 - Then postgres1 is a leader after 10 seconds + Given I start postgres-1 + Then postgres-1 is a leader after 10 seconds And there is a non empty initialize key in DCS after 15 seconds When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}} Then I receive a response code 200 @@ -9,32 +9,32 @@ Feature: standby cluster And I sleep for 3 seconds When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"test_logical": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}} Then I receive a response code 200 - And I do a backup of postgres1 - When I start postgres0 - Then "members/postgres0" key in DCS has state=running after 10 seconds - And replication works from postgres1 to postgres0 after 15 seconds + And I do a backup of postgres-1 + When I start postgres-0 + Then "members/postgres-0" key in DCS has state=running after 10 seconds + And replication works from postgres-1 to postgres-0 after 15 seconds When I issue a GET request to http://127.0.0.1:8008/patroni Then I receive a response code 200 And I receive a response replication_state streaming - And "members/postgres0" key in DCS has replication_state=streaming after 10 seconds + And "members/postgres-0" key in DCS has replication_state=streaming after 10 seconds @slot-advance Scenario: check permanent logical slots are synced to the replica - Given I run patronictl.py restart batman postgres1 --force - Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds + Given I run patronictl.py restart batman postgres-1 --force + Then Logical slot test_logical is in sync between postgres-0 and postgres-1 after 10 seconds Scenario: Detach exiting node from the cluster - When I shut down postgres1 - Then postgres0 is a leader after 10 seconds - And "members/postgres0" key in DCS has role=primary after 5 seconds + When I shut down postgres-1 + Then postgres-0 is a leader after 10 seconds + And "members/postgres-0" key in DCS has role=primary after 5 seconds When I issue a GET request to http://127.0.0.1:8008/ Then I receive a response code 200 Scenario: check replication of a single table in a standby cluster - Given I start postgres1 in a standby cluster batman1 as a clone of postgres0 - Then postgres1 is a leader of batman1 after 10 seconds - When I add the table foo to postgres0 - Then table foo is present on postgres1 after 20 seconds + Given I start postgres-1 in a standby cluster batman1 as a clone of postgres-0 + Then postgres-1 is a leader of batman1 after 10 seconds + When I add the table foo to postgres-0 + Then table foo is present on postgres-1 after 20 seconds When I issue a GET request to http://127.0.0.1:8009/patroni Then I receive a response code 200 And I receive a response replication_state streaming @@ -44,29 +44,29 @@ Feature: standby cluster When I issue a GET request to http://127.0.0.1:8009/standby_leader Then I receive a response code 200 And I receive a response role standby_leader - And there is a postgres1_cb.log with "on_role_change standby_leader batman1" in postgres1 data directory - When I start postgres2 in a cluster batman1 - Then postgres2 role is the replica after 24 seconds - And postgres2 is replicating from postgres1 after 10 seconds - And table foo is present on postgres2 after 20 seconds + And there is a postgres-1_cb.log with "on_role_change standby_leader batman1" in postgres-1 data directory + When I start postgres-2 in a cluster batman1 + Then postgres-2 role is the replica after 24 seconds + And postgres-2 is replicating from postgres-1 after 10 seconds + And table foo is present on postgres-2 after 20 seconds When I issue a GET request to http://127.0.0.1:8010/patroni Then I receive a response code 200 And I receive a response replication_state streaming - And postgres1 does not have a replication slot named test_logical + And postgres-1 does not have a replication slot named test_logical Scenario: check switchover Given I run patronictl.py switchover batman1 --force Then Status code on GET http://127.0.0.1:8010/standby_leader is 200 after 10 seconds - And postgres1 is replicating from postgres2 after 32 seconds - And there is a postgres2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres2 data directory + And postgres-1 is replicating from postgres-2 after 32 seconds + And there is a postgres-2_cb.log with "on_start replica batman1\non_role_change standby_leader batman1" in postgres-2 data directory Scenario: check failover - When I kill postgres2 - And I kill postmaster on postgres2 - Then postgres1 is replicating from postgres0 after 32 seconds + When I kill postgres-2 + And I kill postmaster on postgres-2 + Then postgres-1 is replicating from postgres-0 after 32 seconds And Status code on GET http://127.0.0.1:8009/standby_leader is 200 after 10 seconds When I issue a GET request to http://127.0.0.1:8009/primary Then I receive a response code 503 And I receive a response role standby_leader - And replication works from postgres0 to postgres1 after 15 seconds - And there is a postgres1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres1 data directory + And replication works from postgres-0 to postgres-1 after 15 seconds + And there is a postgres-1_cb.log with "on_role_change replica batman1\non_role_change standby_leader batman1" in postgres-1 data directory diff --git a/features/steps/basic_replication.py b/features/steps/basic_replication.py index 3ec61ca6..c6529150 100644 --- a/features/steps/basic_replication.py +++ b/features/steps/basic_replication.py @@ -2,17 +2,27 @@ import json from time import sleep, time -from behave import step, then +import parse + +from behave import register_type, step, then import patroni.psycopg as pg -@step('I start {name:w}') +@parse.with_pattern(r'[a-z][a-z0-9_\-]*[a-z0-9]') +def parse_name(text): + return text + + +register_type(name=parse_name) + + +@step('I start {name:name}') def start_patroni(context, name): return context.pctl.start(name) -@step('I start duplicate {name:w} on port {port:d}') +@step('I start duplicate {name:name} on port {port:d}') def start_duplicate_patroni(context, name, port): config = { "name": name, @@ -28,22 +38,22 @@ def start_duplicate_patroni(context, name, port): "No error was raised by duplicate start of {0} ".format(name) -@step('I shut down {name:w}') +@step('I shut down {name:name}') def stop_patroni(context, name): return context.pctl.stop(name, timeout=60) -@step('I kill {name:w}') +@step('I kill {name:name}') def kill_patroni(context, name): return context.pctl.stop(name, kill=True) -@step('I shut down postmaster on {name:w}') +@step('I shut down postmaster on {name:name}') def stop_postgres(context, name): return context.pctl.stop(name, postgres=True) -@step('I kill postmaster on {name:w}') +@step('I kill postmaster on {name:name}') def kill_postgres(context, name): return context.pctl.stop(name, kill=True, postgres=True) @@ -53,7 +63,7 @@ def get_wal_name(context, pg_name): return 'xlog' if int(version) / 10000 < 10 else 'wal' -@step('I add the table {table_name:w} to {pg_name:w}') +@step('I add the table {table_name:w} to {pg_name:name}') def add_table(context, table_name, pg_name): # parse the configuration file and get the port try: @@ -63,7 +73,7 @@ def add_table(context, table_name, pg_name): assert False, "Error creating table {0} on {1}: {2}".format(table_name, pg_name, e) -@step('I {action:w} wal replay on {pg_name:w}') +@step('I {action:w} wal replay on {pg_name:name}') def toggle_wal_replay(context, action, pg_name): # pause or resume the wal replay process try: @@ -72,7 +82,7 @@ def toggle_wal_replay(context, action, pg_name): assert False, "Error during {0} wal recovery on {1}: {2}".format(action, pg_name, e) -@step('I {action:w} table on {pg_name:w}') +@step('I {action:w} table on {pg_name:name}') def crdr_mytest(context, action, pg_name): try: if (action == "create"): @@ -83,7 +93,7 @@ def crdr_mytest(context, action, pg_name): assert False, "Error {0} table mytest on {1}: {2}".format(action, pg_name, e) -@step('I load data on {pg_name:w}') +@step('I load data on {pg_name:name}') def initiate_load(context, pg_name): # perform dummy load try: @@ -92,7 +102,7 @@ def initiate_load(context, pg_name): assert False, "Error loading test data on {0}: {1}".format(pg_name, e) -@then('Table {table_name:w} is present on {pg_name:w} after {max_replication_delay:d} seconds') +@then('Table {table_name:w} is present on {pg_name:name} after {max_replication_delay:d} seconds') def table_is_present_on(context, table_name, pg_name, max_replication_delay): max_replication_delay *= context.timeout_multiplier for _ in range(int(max_replication_delay)): @@ -104,15 +114,15 @@ def table_is_present_on(context, table_name, pg_name, max_replication_delay): "Table {0} is not present on {1} after {2} seconds".format(table_name, pg_name, max_replication_delay) -@then('{pg_name:w} role is the {pg_role:w} after {max_promotion_timeout:d} seconds') +@then('{pg_name:name} role is the {pg_role:w} after {max_promotion_timeout:d} seconds') def check_role(context, pg_name, pg_role, max_promotion_timeout): max_promotion_timeout *= context.timeout_multiplier assert context.pctl.check_role_has_changed_to(pg_name, pg_role, timeout=int(max_promotion_timeout)), \ "{0} role didn't change to {1} after {2} seconds".format(pg_name, pg_role, max_promotion_timeout) -@step('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds') -@then('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds') +@step('replication works from {primary:name} to {replica:name} after {time_limit:d} seconds') +@then('replication works from {primary:name} to {replica:name} after {time_limit:d} seconds') def replication_works(context, primary, replica, time_limit): context.execute_steps(u""" When I add the table test_{0} to {1} diff --git a/features/steps/cascading_replication.py b/features/steps/cascading_replication.py index c6b43f31..1f09a33b 100644 --- a/features/steps/cascading_replication.py +++ b/features/steps/cascading_replication.py @@ -4,18 +4,18 @@ import time from behave import step, then -@step('I configure and start {name:w} with a tag {tag_name:w} {tag_value:w}') +@step('I configure and start {name:name} with a tag {tag_name:w} {tag_value}') def start_patroni_with_a_name_value_tag(context, name, tag_name, tag_value): return context.pctl.start(name, custom_config={'tags': {tag_name: tag_value}}) -@then('There is a {label} with "{content}" in {name:w} data directory') +@then('There is a {label} with "{content}" in {name:name} data directory') def check_label(context, label, content, name): value = (context.pctl.read_label(name, label) or '').replace('\n', '\\n') assert content in value, "\"{0}\" in {1} doesn't contain {2}".format(value, label, content) -@step('I create label with "{content:w}" in {name:w} data directory') +@step('I create label with "{content}" in {name:name} data directory') def write_label(context, content, name): context.pctl.write_label(name, content) diff --git a/features/steps/citus.py b/features/steps/citus.py index 274a8b51..0f7b7cfa 100644 --- a/features/steps/citus.py +++ b/features/steps/citus.py @@ -11,8 +11,8 @@ from dateutil import tz tzutc = tz.tzutc() -@step('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds') -@then('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds') +@step('{name:name} is a leader in a group {group:d} after {time_limit:d} seconds') +@then('{name:name} is a leader in a group {group:d} after {time_limit:d} seconds') def is_a_group_leader(context, name, group, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -40,12 +40,12 @@ def check_group_member(context, name, group, key, value, time_limit): " after {5} seconds").format(name, group, key, value, response, time_limit) -@step('I start {name:w} in citus group {group:d}') +@step('I start {name:name} in citus group {group:d}') def start_citus(context, name, group): return context.pctl.start(name, custom_config={"citus": {"database": "postgres", "group": int(group)}}) -@step('{name1:w} is registered in the {name2:w} as the {role:w} in group {group:d} after {time_limit:d} seconds') +@step('{name1:name} is registered in the {name2:name} as the {role:w} in group {group:d} after {time_limit:d} seconds') def check_registration(context, name1, name2, role, group, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -65,13 +65,13 @@ def check_registration(context, name1, name2, role, group, time_limit): assert False, "Node {0} is not registered in pg_dist_node on the node {1}".format(name1, name2) -@step('I create a distributed table on {name:w}') +@step('I create a distributed table on {name:name}') def create_distributed_table(context, name): context.pctl.query(name, 'CREATE TABLE public.d(id int not null)') context.pctl.query(name, "SELECT create_distributed_table('public.d', 'id')") -@step('I cleanup a distributed table on {name:w}') +@step('I cleanup a distributed table on {name:name}') def cleanup_distributed_table(context, name): context.pctl.query(name, 'TRUNCATE public.d') @@ -87,7 +87,7 @@ def insert_thread(query_func, context): context.thread_stop_event.wait(0.01) -@step('I start a thread inserting data on {name:w}') +@step('I start a thread inserting data on {name:name}') def start_insert_thread(context, name): context.thread_stop_event = Event() context.insert_counter = 0 @@ -110,13 +110,13 @@ def stop_insert_thread(context): assert not context.thread.is_alive(), "Thread is still alive" -@step("a distributed table on {name:w} has expected rows") +@step("a distributed table on {name:name} has expected rows") def count_rows(context, name): rows = context.pctl.query(name, "SELECT COUNT(*) FROM public.d").fetchone()[0] assert rows == context.insert_counter, "Distributed table doesn't have expected amount of rows" -@step("there is a transaction in progress on {name:w} changing pg_dist_node after {time_limit:d} seconds") +@step("there is a transaction in progress on {name:name} changing pg_dist_node after {time_limit:d} seconds") def check_transaction(context, name, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) diff --git a/features/steps/custom_bootstrap.py b/features/steps/custom_bootstrap.py index 2ab56ea7..ea3e91a1 100644 --- a/features/steps/custom_bootstrap.py +++ b/features/steps/custom_bootstrap.py @@ -3,17 +3,17 @@ import time from behave import step, then -@step('I start {name:w} in a cluster {cluster_name:w} as a clone of {name2:w}') +@step('I start {name:name} in a cluster {cluster_name:w} as a clone of {name2:name}') def start_cluster_clone(context, name, cluster_name, name2): context.pctl.clone(name2, cluster_name, name) -@step('I start {name:w} in a cluster {cluster_name:w} from backup') +@step('I start {name:name} in a cluster {cluster_name:w} from backup') def start_cluster_from_backup(context, name, cluster_name): context.pctl.bootstrap_from_backup(name, cluster_name) -@then('{name:w} is a leader of {cluster_name:w} after {time_limit:d} seconds') +@then('{name:name} is a leader of {cluster_name:w} after {time_limit:d} seconds') def is_a_leader(context, name, cluster_name, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -22,6 +22,6 @@ def is_a_leader(context, name, cluster_name, time_limit): assert time.time() < max_time, "{0} is not a leader in dcs after {1} seconds".format(name, time_limit) -@step('I do a backup of {name:w}') +@step('I do a backup of {name:name}') def do_backup(context, name): context.pctl.backup(name) diff --git a/features/steps/dcs_failsafe_mode.py b/features/steps/dcs_failsafe_mode.py index 8273e996..2e488257 100644 --- a/features/steps/dcs_failsafe_mode.py +++ b/features/steps/dcs_failsafe_mode.py @@ -11,6 +11,6 @@ def stop_dcs_outage(context): context.dcs_ctl.stop_outage() -@step('I start {name:w} in a cluster {cluster_name:w} from backup with no_leader') +@step('I start {name:name} in a cluster {cluster_name:w} from backup with no_leader') def start_cluster_from_backup_no_leader(context, name, cluster_name): context.pctl.bootstrap_from_backup_no_leader(name, cluster_name) diff --git a/features/steps/patroni_api.py b/features/steps/patroni_api.py index b8935914..3606b09b 100644 --- a/features/steps/patroni_api.py +++ b/features/steps/patroni_api.py @@ -28,8 +28,8 @@ register_type(url=parse_url) # just rely on the database availability, since there is # a short gap between the time PostgreSQL becomes available # and Patroni assuming the leader role. -@step('{name:w} is a leader after {time_limit:d} seconds') -@then('{name:w} is a leader after {time_limit:d} seconds') +@step('{name:name} is a leader after {time_limit:d} seconds') +@then('{name:name} is a leader after {time_limit:d} seconds') def is_a_leader(context, name, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -97,7 +97,7 @@ def do_run(context, cmd): context.response = response.decode('utf-8').strip() -@then('I receive a response {component:w} {data}') +@then('I receive a response {component:name} {data}') def check_response(context, component, data): if component == 'code': assert context.status_code == int(data), \ @@ -116,7 +116,7 @@ def check_response(context, component, data): assert str(context.response[component]) == str(data), "{0} does not contain {1}".format(component, data) -@step('I issue a scheduled switchover from {from_host:w} to {to_host:w} in {in_seconds:d} seconds') +@step('I issue a scheduled switchover from {from_host:name} to {to_host:name} in {in_seconds:d} seconds') def scheduled_switchover(context, from_host, to_host, in_seconds): context.execute_steps(u""" Given I run patronictl.py switchover batman --primary {0} --candidate {1} --scheduled "{2}" --force @@ -130,13 +130,13 @@ def scheduled_restart(context, url, in_seconds, data): context.execute_steps(u"""Given I issue a POST request to {0}/restart with {1}""".format(url, json.dumps(data))) -@step('I {action:w} {tag:w} tag in {pg_name:w} config') +@step('I {action:w} {tag:w} tag in {pg_name:name} config') def add_bool_tag_to_config(context, action, tag, pg_name): value = action == 'set' context.pctl.add_tag_to_config(pg_name, tag, value) -@step('I add tag {tag:w} {value:w} to {pg_name:w} config') +@step('I add tag {tag:w} {value:w} to {pg_name:name} config') def add_tag_to_config(context, tag, value, pg_name): context.pctl.add_tag_to_config(pg_name, tag, value) diff --git a/features/steps/quorum_commit.py b/features/steps/quorum_commit.py index 48fc4b62..9753ae26 100644 --- a/features/steps/quorum_commit.py +++ b/features/steps/quorum_commit.py @@ -36,7 +36,7 @@ def _parse_synchronous_standby_names(value): return expected_num, expected_value -@then('synchronous_standby_names on {name:2} is set to "{value}" after {time_limit:d} seconds') +@then("synchronous_standby_names on {name:2} is set to '{value}' after {time_limit:d} seconds") def check_synchronous_standby_names(context, name, value, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) diff --git a/features/steps/slots.py b/features/steps/slots.py index d7d7d0cb..aca00866 100644 --- a/features/steps/slots.py +++ b/features/steps/slots.py @@ -6,7 +6,7 @@ from behave import step, then import patroni.psycopg as pg -@step('I create a logical replication slot {slot_name} on {pg_name:w} with the {plugin:w} plugin') +@step('I create a logical replication slot {slot_name} on {pg_name:name} with the {plugin:w} plugin') def create_logical_replication_slot(context, slot_name, pg_name, plugin): try: output = context.pctl.query(pg_name, ("SELECT pg_create_logical_replication_slot('{0}', '{1}')," @@ -17,9 +17,9 @@ def create_logical_replication_slot(context, slot_name, pg_name, plugin): assert False, "Error creating slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin) -@step('{pg_name:w} has a logical replication slot named {slot_name}' +@step('{pg_name:name} has a logical replication slot named {slot_name}' ' with the {plugin:w} plugin after {time_limit:d} seconds') -@then('{pg_name:w} has a logical replication slot named {slot_name}' +@then('{pg_name:name} has a logical replication slot named {slot_name}' ' with the {plugin:w} plugin after {time_limit:d} seconds') def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit): time_limit *= context.timeout_multiplier @@ -38,8 +38,8 @@ def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit assert False, f"Error looking for slot {slot_name} on {pg_name} with plugin {plugin}" -@step('{pg_name:w} does not have a replication slot named {slot_name:w}') -@then('{pg_name:w} does not have a replication slot named {slot_name:w}') +@step('{pg_name:name} does not have a replication slot named {slot_name:w}') +@then('{pg_name:name} does not have a replication slot named {slot_name:w}') def does_not_have_replication_slot(context, pg_name, slot_name): try: row = context.pctl.query(pg_name, ("SELECT 1 FROM pg_replication_slots" @@ -49,7 +49,8 @@ def does_not_have_replication_slot(context, pg_name, slot_name): assert False, "Error looking for slot {0} on {1}".format(slot_name, pg_name) -@step('{slot_type:w} slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds') +@step('{slot_type:w} slot {slot_name:w} is in sync between ' + '{pg_name1:name} and {pg_name2:name} after {time_limit:d} seconds') def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -68,17 +69,17 @@ def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit) f"{slot_type} slot {slot_name} is not in sync between {pg_name1} and {pg_name2} after {time_limit} seconds" -@step('I get all changes from logical slot {slot_name:w} on {pg_name:w}') +@step('I get all changes from logical slot {slot_name:w} on {pg_name:name}') def logical_slot_get_changes(context, slot_name, pg_name): context.pctl.query(pg_name, "SELECT * FROM pg_logical_slot_get_changes('{0}', NULL, NULL)".format(slot_name)) -@step('I get all changes from physical slot {slot_name:w} on {pg_name:w}') +@step('I get all changes from physical slot {slot_name:w} on {pg_name:name}') def physical_slot_get_changes(context, slot_name, pg_name): context.pctl.query(pg_name, f"SELECT * FROM pg_replication_slot_advance('{slot_name}', pg_current_wal_lsn())") -@step('{pg_name:w} has a physical replication slot named {slot_name} after {time_limit:d} seconds') +@step('{pg_name:name} has a physical replication slot named {slot_name} after {time_limit:d} seconds') def has_physical_replication_slot(context, pg_name, slot_name, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -94,7 +95,7 @@ def has_physical_replication_slot(context, pg_name, slot_name, time_limit): assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds" -@step('physical replication slot named {slot_name} on {pg_name:w} has no xmin value after {time_limit:d} seconds') +@step('physical replication slot named {slot_name} on {pg_name:name} has no xmin value after {time_limit:d} seconds') def physical_slot_no_xmin(context, pg_name, slot_name, time_limit): time_limit *= context.timeout_multiplier max_time = time.time() + int(time_limit) @@ -114,13 +115,13 @@ def physical_slot_no_xmin(context, pg_name, slot_name, time_limit): else f"Physical slot {slot_name} has xmin value after {time_limit} seconds" -@step('"{name}" key in DCS has {subkey:w} in {key:w}') +@step('"{name}" key in DCS has {subkey} in {key:w}') def dcs_key_contains(context, name, subkey, key): response = json.loads(context.dcs_ctl.query(name)) assert key in response and subkey in response[key], f"{name} key in DCS doesn't have {subkey} in {key}" -@step('"{name}" key in DCS does not have {subkey:w} in {key:w}') +@step('"{name}" key in DCS does not have {subkey} in {key:w}') def dcs_key_does_not_contain(context, name, subkey, key): response = json.loads(context.dcs_ctl.query(name)) assert key not in response or subkey not in response[key], f"{name} key in DCS has {subkey} in {key}" diff --git a/features/steps/standby_cluster.py b/features/steps/standby_cluster.py index 1bf3b3fb..092b95f3 100644 --- a/features/steps/standby_cluster.py +++ b/features/steps/standby_cluster.py @@ -9,7 +9,7 @@ def callbacks(context, name): for c in ('on_start', 'on_stop', 'on_restart', 'on_role_change')} -@step('I start {name:w} in a cluster {cluster_name:w}') +@step('I start {name:name} in a cluster {cluster_name:w}') def start_patroni(context, name, cluster_name): return context.pctl.start(name, custom_config={ "scope": cluster_name, @@ -20,7 +20,7 @@ def start_patroni(context, name, cluster_name): }) -@step('I start {name:w} in a standby cluster {cluster_name:w} as a clone of {name2:w}') +@step('I start {name:name} in a standby cluster {cluster_name:w} as a clone of {name2:name}') def start_patroni_standby_cluster(context, name, cluster_name, name2): # we need to remove patroni.dynamic.json in order to "bootstrap" standby cluster with existing PGDATA os.unlink(os.path.join(context.pctl._processes[name]._data_dir, 'patroni.dynamic.json')) @@ -49,7 +49,7 @@ def start_patroni_standby_cluster(context, name, cluster_name, name2): return context.pctl.start(name) -@step('{pg_name1:w} is replicating from {pg_name2:w} after {timeout:d} seconds') +@step('{pg_name1:name} is replicating from {pg_name2:name} after {timeout:d} seconds') def check_replication_status(context, pg_name1, pg_name2, timeout): bound_time = time.time() + timeout * context.timeout_multiplier diff --git a/features/steps/watchdog.py b/features/steps/watchdog.py index ebec646e..bd48da14 100644 --- a/features/steps/watchdog.py +++ b/features/steps/watchdog.py @@ -14,12 +14,12 @@ def polling_loop(timeout, interval=1): time.sleep(interval) -@step('I start {name:w} with watchdog') +@step('I start {name:name} with watchdog') def start_patroni_with_watchdog(context, name): return context.pctl.start(name, custom_config={'watchdog': True, 'bootstrap': {'dcs': {'ttl': 20}}}) -@step('{name:w} watchdog has been pinged after {timeout:d} seconds') +@step('{name:name} watchdog has been pinged after {timeout:d} seconds') def watchdog_was_pinged(context, name, timeout): for _ in polling_loop(timeout): if context.pctl.get_watchdog(name).was_pinged: @@ -27,22 +27,22 @@ def watchdog_was_pinged(context, name, timeout): return False -@then('{name:w} watchdog has been closed') +@then('{name:name} watchdog has been closed') def watchdog_was_closed(context, name): assert context.pctl.get_watchdog(name).was_closed -@step('{name:w} watchdog has a {timeout:d} second timeout') +@step('{name:name} watchdog has a {timeout:d} second timeout') def watchdog_has_timeout(context, name, timeout): assert context.pctl.get_watchdog(name).timeout == timeout -@step('I reset {name:w} watchdog state') +@step('I reset {name:name} watchdog state') def watchdog_reset_pinged(context, name): context.pctl.get_watchdog(name).reset() -@then('{name:w} watchdog is triggered after {timeout:d} seconds') +@then('{name:name} watchdog is triggered after {timeout:d} seconds') def watchdog_was_triggered(context, name, timeout): for _ in polling_loop(timeout): if context.pctl.get_watchdog(name).was_triggered: @@ -50,6 +50,6 @@ def watchdog_was_triggered(context, name, timeout): assert False -@step('{name:w} hangs for {timeout:d} seconds') +@step('{name:name} hangs for {timeout:d} seconds') def patroni_hang(context, name, timeout): return context.pctl.patroni_hang(name, timeout) diff --git a/features/watchdog.feature b/features/watchdog.feature index cea38f02..e5290b35 100644 --- a/features/watchdog.feature +++ b/features/watchdog.feature @@ -2,38 +2,38 @@ Feature: watchdog Verify that watchdog gets pinged and triggered under appropriate circumstances. Scenario: watchdog is opened and pinged - Given I start postgres0 with watchdog - Then postgres0 is a leader after 10 seconds - And postgres0 role is the primary after 10 seconds - And postgres0 watchdog has been pinged after 10 seconds - And postgres0 watchdog has a 15 second timeout + Given I start postgres-0 with watchdog + Then postgres-0 is a leader after 10 seconds + And postgres-0 role is the primary after 10 seconds + And postgres-0 watchdog has been pinged after 10 seconds + And postgres-0 watchdog has a 15 second timeout Scenario: watchdog is reconfigured after global ttl changed Given I run patronictl.py edit-config batman -s ttl=30 --force Then I receive a response returncode 0 And I receive a response output "+ttl: 30" When I sleep for 4 seconds - Then postgres0 watchdog has a 25 second timeout + Then postgres-0 watchdog has a 25 second timeout Scenario: watchdog is disabled during pause Given I run patronictl.py pause batman Then I receive a response returncode 0 When I sleep for 2 seconds - Then postgres0 watchdog has been closed + Then postgres-0 watchdog has been closed Scenario: watchdog is opened and pinged after resume - Given I reset postgres0 watchdog state + Given I reset postgres-0 watchdog state And I run patronictl.py resume batman Then I receive a response returncode 0 - And postgres0 watchdog has been pinged after 10 seconds + And postgres-0 watchdog has been pinged after 10 seconds Scenario: watchdog is disabled when shutting down - Given I shut down postgres0 - Then postgres0 watchdog has been closed + Given I shut down postgres-0 + Then postgres-0 watchdog has been closed Scenario: watchdog is triggered if patroni stops responding - Given I reset postgres0 watchdog state - And I start postgres0 with watchdog - Then postgres0 role is the primary after 10 seconds - When postgres0 hangs for 30 seconds - Then postgres0 watchdog is triggered after 30 seconds + Given I reset postgres-0 watchdog state + And I start postgres-0 with watchdog + Then postgres-0 role is the primary after 10 seconds + When postgres-0 hangs for 30 seconds + Then postgres-0 watchdog is triggered after 30 seconds