Add watchdog support on Linux (#343)

Ensures that system gets rebooted before TTL runs out.

Initial version. Open questions:

    Do we want to disable watchdog while we are not master?
This commit is contained in:
Ants Aasma
2017-06-01 16:53:46 +02:00
committed by Alexander Kukushkin
parent e3a01727a9
commit a70b46ef13
20 changed files with 1369 additions and 108 deletions
+43
View File
@@ -0,0 +1,43 @@
Feature: watchdog
Verify that watchdog gets pinged and triggered under appropriate circumstances.
Scenario: watchdog is opened, pinged and closed
Given I start postgres0 with watchdog
Then postgres0 is a leader after 10 seconds
And postgres0 role is the primary after 10 seconds
And postgres0 watchdog has been pinged after 10 seconds
When I shut down postgres0
Then postgres0 watchdog has been closed
Scenario: watchdog is updated during pause
Given I start postgres0 with watchdog
Then postgres0 role is the primary after 10 seconds
When I run patronictl.py pause batman
And I wait for next postgres0 watchdog ping
Then I receive a response returncode 0
And postgres0 watchdog has been pinged after 10 seconds
When I shut down postgres0
Then postgres0 watchdog has been closed
And postgres0 database is running
Scenario: watchdog is updated during shutdown checkpoint
Given I start postgres0 with watchdog
Then postgres0 role is the primary after 10 seconds
And Sleep for 10 seconds
Given I run patronictl.py resume batman
Then I receive a response returncode 0
When I start postgres1
Then postgres1 role is the secondary after 10 seconds
When postgres0 checkpoint takes 30 seconds
And I shut down postgres0
Then postgres0 watchdog was not triggered
And postgres1 role is the primary after 10 seconds
Scenario: watchdog is triggered if postgres stops responding
Given I start postgres0 with watchdog
Then postgres0 role is the secondary after 10 seconds
When I shut down postgres1
Then postgres0 role is the primary after 10 seconds
When postgres0 hangs for 30 seconds
And I terminate postgres0 user processes
Then postgres0 watchdog is triggered after 30 seconds