mirror of
https://github.com/outbackdingo/patroni.git
synced 2026-08-25 14:53:37 +00:00
Ants AasmaandAlexander Kukushkin
70d718a058
* Only activate watchdog while master and not paused We don't really need the protections while we are not master. This way we only need to tickle the watchdog when we are updating leader key or while demotion is happening. As implemented we might fail to notice to shut down the watchdog if someone demotes postgres and removes leader key behind Patroni's back. There are probably other similar cases. Basically if the administrator if being actively stupid they might get unexpected restarts. That seems fine. * Add configuration change support. Change MODE_REQUIRED to disable leader eligibility instead of closing Patroni. Changes watchdog timeout during the next keepalive when ttl is changed. Watchdog driver and requirement can also be switched online. When watchdog mode is `required` and watchdog setup does not work then the effect is similar to nofailover. Add watchdog_failed to status API to signify this. This is True only when watchdog does not work **AND** it is required. * Reset implementation when config changed while active. * Add watchdog safety margin configuration Defaults to 5 seconds. Basically this is the maximum amount of time that can pass between the calls to odcs.update_leader()` and `watchdog.keepalive()`, which are called right after each other. Should be safe for pretty much any sane scenario and allows the default settings to not trigger watchdog when DCS is not responding. * Cancel bootstrap if watchdog activation fails The system would have demoted itself anyway the next HA loop. Doing it in bootstrap gives at least some other node chance to try bootstrapping in the hope that it is configured correctly. If all nodes are unable to activate they will continue to try until the disk is filled with moved datadirs. Perhaps not ideal behavior, but as the situation is unlikely to resolve itself without administrator intervention it doesn't seem too bad.
20 lines
802 B
Gherkin
20 lines
802 B
Gherkin
Feature: watchdog
|
|
Verify that watchdog gets pinged and triggered under appropriate circumstances.
|
|
|
|
Scenario: watchdog is opened, pinged and closed
|
|
Given I start postgres0 with watchdog
|
|
Then postgres0 is a leader after 10 seconds
|
|
And postgres0 role is the primary after 10 seconds
|
|
And postgres0 watchdog has been pinged after 10 seconds
|
|
When I shut down postgres0
|
|
Then postgres0 watchdog has been closed
|
|
|
|
#TODO: test watchdog is disabled during pause
|
|
#TODO: test watchdog is disabled properly when shutting down
|
|
|
|
Scenario: watchdog is triggered if patroni stops responding
|
|
Given I start postgres0 with watchdog
|
|
Then postgres0 role is the primary after 10 seconds
|
|
When postgres0 hangs for 30 seconds
|
|
Then postgres0 watchdog is triggered after 30 seconds
|