Feature: failover priority (#2780)

The priority is configured with `failover_priority` tag. Possible values are from `0` till infinity, where `0` means that the node will never become the leader, which is the same as `nofailover` tag set to `true`. As a result, in the configuration file one should set only one of `failover_priority` or `nofailover` tags.

The failover priority kicks in only when there are more than one node have the same receive/replay LSN and are ahead of other nodes in the cluster. In this case the node with higher value of `failover_priority` is preferred. If there is a node with higher values of receive/replay LSN, it will become the new leader even if it has lower value of `failover_priority` (except when priority is set to 0).

Close https://github.com/zalando/patroni/issues/2759

features/basic_replication.feature

@@ -82,4 +82,4 @@ Feature: basic replication
   @reject-duplicate-name
   Scenario: check graceful rejection when two nodes have the same name
     Given I start duplicate postgres0 on port 8011
-    Then there is a "Can't start; there is already a node named 'postgres0' running" CRITICAL in the dup-postgres0 patroni log
+    Then there is one of ["Can't start; there is already a node named 'postgres0' running"] CRITICAL in the dup-postgres0 patroni log after 5 seconds

features/priority_failover.feature

@@ -0,0 +1,23 @@
+Feature: priority replication
+  We should check that we can give nodes priority during failover
+
+  Scenario: check failover priority 0 prevents leaderships
+    Given I configure and start postgres0 with a tag failover_priority 1
+    And I configure and start postgres1 with a tag failover_priority 0
+    Then replication works from postgres0 to postgres1 after 20 seconds
+    When I shut down postgres0
+    And I sleep for 5 seconds
+    Then postgres1 role is the secondary after 10 seconds
+    And there is one of ["following a different leader because I am not allowed to promote"] INFO in the postgres1 patroni log after 5 seconds
+    Given I start postgres0
+    Then postgres0 role is the primary after 10 seconds
+
+  Scenario: check higher failover priority is respected
+    Given I configure and start postgres2 with a tag failover_priority 1
+    And I configure and start postgres3 with a tag failover_priority 2
+    Then replication works from postgres0 to postgres2 after 20 seconds
+    And replication works from postgres0 to postgres3 after 20 seconds
+    When I shut down postgres0
+    And I sleep for 5 seconds
+    Then postgres3 role is the primary after 10 seconds
+    And there is one of ["postgres3 has equally tolerable WAL position and priority 2, while this node has priority 1","Wal position of postgres3 is ahead of my wal position"] INFO in the postgres2 patroni log after 5 seconds

features/steps/basic_replication.py

@@ -1,3 +1,4 @@
+import json
 import patroni.psycopg as pg
 
 from behave import step, then
@@ -113,8 +114,15 @@ def replication_works(context, primary, replica, time_limit):
     """.format(str(time()).replace('.', '_').replace(',', '_'), primary, replica, time_limit))
 
 
-@then('there is a "{message}" {level:w} in the {node} patroni log')
-def check_patroni_log(context, message, level, node):
-    messsages_of_level = context.pctl.read_patroni_log(node, level)
-    assert any(message in line for line in messsages_of_level), \
-        "There was no {0} {1} in the {2} patroni log".format(message, level, node)
+@then('there is one of {message_list} {level:w} in the {node} patroni log after {timeout:d} seconds')
+def check_patroni_log(context, message_list, level, node, timeout):
+    timeout *= context.timeout_multiplier
+    message_list = json.loads(message_list)
+
+    for _ in range(int(timeout)):
+        messsages_of_level = context.pctl.read_patroni_log(node, level)
+        if any(any(message in line for line in messsages_of_level) for message in message_list):
+            break
+        time.sleep(1)
+    else:
+        assert False, f"There were none of {message_list} {level} in the {node} patroni log after {timeout} seconds"