Reduce flakiness of citus behave tests, take 2 (#2742)

Reorder some checks and verify that the old primary is already in the `running` state before checking replication. This check elliminates the race condition when replication started to work but node name is removed from the `synchronous_standby_names` because state isn't `running`.
This commit is contained in:
Alexander Kukushkin
2023-07-11 15:04:10 +02:00
committed by GitHub
parent 412c51ddf1
commit b8cff3515a
+7 -5
View File
@@ -16,14 +16,15 @@ Feature: citus
Scenario: coordinator failover updates pg_dist_node Scenario: coordinator failover updates pg_dist_node
Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force
Then postgres1 role is the primary after 10 seconds Then postgres1 role is the primary after 10 seconds
And "members/postgres0" key in a group 0 in DCS has state=running after 15 seconds
And replication works from postgres1 to postgres0 after 15 seconds And replication works from postgres1 to postgres0 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds
And postgres1 is registered in the postgres2 as the primary in group 0 after 5 seconds And postgres1 is registered in the postgres2 as the primary in group 0 after 5 seconds
When I run patronictl.py failover batman --group 0 --candidate postgres0 --force And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds
When I run patronictl.py switchover batman --group 0 --candidate postgres0 --force
Then postgres0 role is the primary after 10 seconds Then postgres0 role is the primary after 10 seconds
And replication works from postgres0 to postgres1 after 15 seconds And replication works from postgres0 to postgres1 after 15 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds
And postgres0 is registered in the postgres2 as the primary in group 0 after 5 seconds And postgres0 is registered in the postgres2 as the primary in group 0 after 5 seconds
And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds
Scenario: worker switchover doesn't break client queries on the coordinator Scenario: worker switchover doesn't break client queries on the coordinator
Given I create a distributed table on postgres0 Given I create a distributed table on postgres0
@@ -31,16 +32,17 @@ Feature: citus
When I run patronictl.py switchover batman --group 1 --force When I run patronictl.py switchover batman --group 1 --force
Then I receive a response returncode 0 Then I receive a response returncode 0
And postgres3 role is the primary after 10 seconds And postgres3 role is the primary after 10 seconds
And "members/postgres2" key in a group 1 in DCS has state=running after 15 seconds
And replication works from postgres3 to postgres2 after 15 seconds And replication works from postgres3 to postgres2 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds
And postgres3 is registered in the postgres0 as the primary in group 1 after 5 seconds And postgres3 is registered in the postgres0 as the primary in group 1 after 5 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds
And a thread is still alive And a thread is still alive
When I run patronictl.py switchover batman --group 1 --force When I run patronictl.py switchover batman --group 1 --force
Then I receive a response returncode 0 Then I receive a response returncode 0
And postgres2 role is the primary after 10 seconds And postgres2 role is the primary after 10 seconds
And replication works from postgres2 to postgres3 after 15 seconds And replication works from postgres2 to postgres3 after 15 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds
And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds And postgres2 is registered in the postgres0 as the primary in group 1 after 5 seconds
And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds
And a thread is still alive And a thread is still alive
When I stop a thread When I stop a thread
Then a distributed table on postgres0 has expected rows Then a distributed table on postgres0 has expected rows