Commit Graph
100 Commits
Author SHA1 Message Date
Alexander Kukushkin ff41818a84 Make unit-tests output less verbose
It will capture output to stdout and stderr and print it when test is
failed. Please set LOGLEVEL env variable to INFO or DEBUG if you want
to see everything (as it was before).
2016-04-13 13:32:39 +02:00
Alexander Kukushkin 15d30a2d35 Try to stabilize acceptance tests 2016-04-13 13:32:39 +02:00
Alexander Kukushkin f8bf1bb0ab Disable sudo, reshuffle travis tasks and introduce caching
Without sudo travis is executing build tasks using docker and waiting
time in this case is really small, usually not longer then 10 seconds.

postgresql-9.5 is installed via addons.apt.packages (without sudo)
But ports 5432 and 5433 are busy. So I had to ajust environment.py to
assign port from higher diapason.

And a few words about build tasks:
First task is used for executing unit tests for all different python versions
The second one is used for executing acceptance tests against etcd
The third one is used for executing acceptance tests against zookeeper
acceptance tests are executed with python2.7 and python3.5

In addition that I've introduced caching of python virtual environment.
It really helps to reduce time needed to install python modules.
2016-04-13 13:32:39 +02:00
Alexander Kukushkin d7d69af12f In some cases output_members function was showing cluster=None 2016-04-08 16:50:22 +02:00
Alexander Kukushkin ada50e418c Update scenario description 2016-03-31 17:13:29 +02:00
Alexander Kukushkin 7c683336fb Don't expose empty tags dict in API and DCS 2016-03-30 15:16:57 +02:00
Alexander Kukushkin 24cc93ac70 Don't expose known boolean tags with value not True 2016-03-30 15:00:25 +02:00
Alexander Kukushkin db5999a639 Correct implementation of 'clonefrom' feature
According to https://github.com/zalando/patroni/issues/48 'clonefrom'
tag should be boolean and it should be used to mark node as a suitable
for creation of a new replica from. If there are more then one such node
in the cluster (with tag clonefrom=true), one of them will be chosed
randomly.
2016-03-30 11:30:05 +02:00
Alexander Kukushkin 0469d4374d Set metaclass of AbstractDCS with six to be python 2&3 compatible 2016-03-24 17:07:56 +01:00
Alexander Kukushkin d4cb15179f Return "demote reason" more precise depending on cluster state 2016-03-24 14:52:27 +01:00
Alexander Kukushkin 9c41ce9f1c We should run checkpoint before shutdown all the time except one case
(when patroni is being shutdown)
2016-03-24 14:50:34 +01:00
Alexander Kukushkin e6af18f0bb Former leader was not able to reattach to cluster without pg_rewind
It was shutdown correctly and I expected such 'join' working, but it was
not, because new leader didn't had enough time to catch up with the
master before promote.
2016-03-24 14:45:21 +01:00
Alexander Kukushkin 9dc588231f Speed up reattach of former leader to the cluster
Instead of starting it up in "read-only", it wil wait 2 seconds, to give
a time to somebody to prompte and after it will execute normal `recover`
procedure.
2016-03-24 12:29:30 +01:00
Alexander Kukushkin 0d3dca56ff In some cases Ha.cluster can be None after calling get_cluster
Such situation is causing patroni crash. Usually it was happening during
manual failover, after former master has demoted and `reset_cluster`
method has been called. In this case `fetch_cluster` was `False` and
`_load_cluster` method was returning value from `self._cluster`, which
was `None`.
2016-03-24 12:06:39 +01:00
Alexander Kukushkin 3a7d2c3874 Remove unused code from unit tests 2016-03-21 20:48:17 +01:00
Alexander Kukushkin 54055c1ff8 Rename ambiguous Failover.member to candidate
But! 'member' is still accepted by REST API and also name 'member' is
used to strore/read this value to/from DCS (for backward comatibility)
2016-03-18 15:59:47 +01:00
Alexander Kukushkin ca3414bda0 Ha loop was writing into scheduled_at during "manual_failover" 2016-03-18 13:24:30 +01:00
Alexander Kukushkin 8a7d2b53e0 Add forgotten mock to the test_ha.py 2016-03-15 16:40:55 +01:00
Alexander Kukushkin 0e0c8ed8d7 Implement delete_cluster interface in for all available dcs
In addition to that rename confusing `Etcd.client` and
`ZooKeeper.client` into `_client`. This attribute is available from
AbstractDCS and people had wrong impression that it provides the same
interface for different DCS implementations, which is obviously not the
case. For Etcd it has type etcd.Client and for ZooKeeper - KazooClient.
2016-03-15 16:25:48 +01:00
Alexander Kukushkin ffebfd84fd Merge branch 'master' of github.com:zalando/patroni into feature/acceptance_tests_behave 2016-03-14 11:54:44 +01:00
Alexander Kukushkin 7d4ec19bca Enable etcd cluster id check 2016-03-14 10:51:50 +01:00
Alexander Kukushkin 3319c3eeea replace unused variables with _ 2016-03-11 15:36:24 +01:00
Alexander Kukushkin ba444adb67 make codacy and quantifiedcode happier 2016-03-11 15:32:16 +01:00
Alexander Kukushkin d3c2b8b2aa replace unused variables with _ 2016-03-11 15:18:58 +01:00
Alexander Kukushkin ea59692b82 Merge branch 'master' of github.com:zalando/patroni into feature/remove-unused-code 2016-03-11 15:15:13 +01:00
Alexander Kukushkin 5f6beae22f Enforce data-type checks for step matcher
and increase default timeout for patroni start
2016-03-11 14:46:14 +01:00
Alexander Kukushkin 8b81d270bc BUGFIX: Assertion Failed: Steps must be unicode 2016-03-11 13:47:57 +01:00
Alexander Kukushkin 30d3982d25 Acceptance tests with behave 2016-03-11 12:56:29 +01:00
Alexander Kukushkin ad7910c025 Merge branch 'feature/acceptance_tests' of github.com:zalando/patroni into feature/acceptance_tests_travis 2016-03-11 09:04:08 +01:00
Alexander Kukushkin 33a1de7828 Fix .travis.yml 2016-03-10 17:23:44 +01:00
Alexander Kukushkin c2d1eea7d0 disable clonefrom test 2016-03-10 17:19:43 +01:00
Alexander Kukushkin 42d798a3de acceptance tests on travis 2016-03-10 17:19:10 +01:00
Alexander Kukushkin 77e6795432 Merge branch 'master' of github.com:zalando/patroni into feature/acceptance_tests_travis 2016-03-10 16:19:06 +01:00
Alexander Kukushkin 5decc61fc8 Merge branch 'master' of github.com:zalando/patroni into bugfix/zookeeper-support-for-ctl 2016-03-07 12:02:57 +01:00
Alexander Kukushkin b77daffc98 Merge branch 'master' of github.com:zalando/patroni into feature/remove-unused-code 2016-02-26 11:39:28 +01:00
Alexander Kukushkin cb38e50ac1 Remove unused code 2016-02-26 08:50:53 +01:00
Alexander Kukushkin 9c12eb671d Fix unit tests 2016-02-24 12:10:58 +01:00
Alexander Kukushkin a38af0949b Merge branch 'master' of github.com:zalando/patroni into feature/xlog_lag_interval 2016-02-23 14:48:49 +01:00
Alexander Kukushkin ec85e2eb49 make quantifiedcode happier 2016-02-23 12:05:02 +01:00
Alexander Kukushkin 756158a735 make codacy and quantifiedcode happier 2016-02-23 11:59:02 +01:00
Alexander Kukushkin 6b3c4697fc Remove unused code 2016-02-23 11:49:22 +01:00
Alexander Kukushkin ce33090c0d Mock dcs.watch directly instead of using wraper 2016-02-23 11:48:52 +01:00
Alexander Kukushkin dd20fc7e71 Refactor follow method 2016-02-23 11:47:47 +01:00
Alexander Kukushkin 641cc4013e Mock a few of methods in Postgresql class instead of the whole class 2016-02-23 11:46:49 +01:00
Alexander Kukushkin 4038d94c5a Fix more codacy issues 2016-02-17 14:59:17 +01:00
Alexander Kukushkin a210cfd1ab Fix more codacy issues 2016-02-17 14:51:59 +01:00
Alexander Kukushkin f7d60c61b6 remove unused code 2016-02-17 14:09:00 +01:00
Alexander Kukushkin de129b733d Fix unit tests 2016-02-17 12:46:32 +01:00
Alexander Kukushkin 7928a3a0d6 Merge branch 'master' of github.com:zalando/patroni into feature/xlog_lag_interval 2016-02-17 12:41:09 +01:00
Alexander Kukushkin f079a9f308 remove unused code 2016-02-17 12:34:22 +01:00
Alexander Kukushkin 1b9e77fe83 pep8 formatting 2016-02-17 12:34:04 +01:00
Alexander Kukushkin 1b14229da4 Catch TypeError within ha loop not in the unit test
In addition to that use sleep function from patroni.utils instead of
time.sleep which is interruptable
2016-02-17 12:18:50 +01:00
Alexander Kukushkin a875e93f2e Merge branch 'master' of github.com:zalando/patroni into feature/scheduled_failover_squashed 2016-02-17 12:14:10 +01:00
Alexander Kukushkin 0710bdfead directory could disappear after successfull call of isdir 2016-02-15 16:01:11 +01:00
Alexander Kukushkin 1bc22727d5 patroni/postgresql.py
directory could disappear after successfull call of isdir
2016-02-15 14:50:59 +01:00
Alexander Kukushkin 3d4fdea8d5 remove unused import 2016-02-15 13:46:53 +01:00
Alexander Kukushkin 31bad6df49 revert some changes which changed functionality of original code 2016-02-15 13:41:50 +01:00
Alexander Kukushkin 58508c3404 remove uneeded code 2016-02-12 16:57:56 +01:00
Alexander Kukushkin b973ed7e4f improve test coverage 2016-02-12 16:52:26 +01:00
Alexander Kukushkin 544ecdc1be make quantifiedcode and codacy happier 2016-02-12 15:53:24 +01:00
Alexander Kukushkin ab0ef91f24 Improve quality of code by resolving issues found by quantifiedcode and codacy 2016-02-12 14:52:23 +01:00
Alexander Kukushkin 36aef07738 Merge branch 'master' of github.com:zalando/patroni into codequality 2016-02-12 12:33:02 +01:00
Alexander Kukushkin df9b8fed2e Improve quality of code by resolving issues found by quantifiedcode and codacy 2016-02-12 12:23:49 +01:00
Alexander Kukushkin 7b524bc557 Create new session and set the process group ID for the postmaster
Otherwice it was receiving signal when you pressed Ctrl+C in the
terminal where patroni is running.
2016-02-10 14:51:20 +01:00
Alexander Kukushkin b4af126bc3 Apply superuser name and password when doing checkpoint 2016-02-10 13:50:05 +01:00
Alexander Kukushkin ad17b2070a Create superuser with the name specified in config file
And later use this name to connect to the cluster
2016-02-10 12:37:34 +01:00
Alexander Kukushkin b77908ce58 Execute tests against python 3.5 2015-12-24 09:59:01 +01:00
Alexander Kukushkin 1b032ab618 stick to the python-etcd=0.4.2 until issues=152 is not fixed 2015-12-24 09:56:24 +01:00
Alexander Kukushkin 41bf010cc1 stick to the python-etcd=0.4.2 until issues=152 is not fixed 2015-12-24 09:54:13 +01:00
Alexander Kukushkin 91a10bf122 Execute tests against python 3.5 2015-12-24 09:51:42 +01:00
Alexander Kukushkin 42a5cf70e6 mock socket.getaddrinfo for patroni-ctl unit tests 2015-12-24 09:49:58 +01:00
Alexander Kukushkin 092bf8defa Merge branch 'master' of github.com:zalando/patroni into feature/initdb 2015-11-12 12:22:14 +01:00
Alexander Kukushkin 13cb8e7402 set synchronous_commit to local when creating roles 2015-11-12 12:20:05 +01:00
Alexander Kukushkin d39d297d92 blacklist some options for initdb instead of having big whitelist 2015-11-12 12:07:36 +01:00
Alexander Kukushkin f0a6c86caa Make it possible to specify custom options for initdb
In the initial implementation we were using the only option
--encoding=UTF8. In order to have pg_rewind working with postgresql-9.3
we have to enable data-checksums. The naive approach was to enable it
globaly but taking into account some performance degradation it's better
not to do it but make it possible to configure it.

In addition to that fix all problems with setting up password of default
postgres user: execute CREATE ROLE | ALTER ROLE depending on content of
pg_authid
2015-11-11 15:59:34 +01:00
Alexander Kukushkin 9c16830fc0 Do not fail when there is no postgresql/parameters section in a config file 2015-11-11 10:46:37 +01:00
Alexander Kukushkin 8a70cc4d7b Call touch_member right after demote and promote.
This is necessary to propagate actual information about node into DCS.
2015-11-09 16:22:51 +01:00
Alexander Kukushkin 2d709a48e6 fix test_create_replica unit test 2015-10-26 11:30:04 +01:00
Alexander Kukushkin ab64ae7fe7 execute delete_trigger_file() after successfull call of create_replica() 2015-10-26 11:29:32 +01:00
Alexander Kukushkin 553129a981 Revert "Fix unit-test for Postgresql.controldata()"
This reverts commit deaaf8ad1a.
2015-10-23 15:59:20 +02:00
Alexander Kukushkin ad719297c2 Merge branch 'master' of github.com:zalando/patroni into feature/custom-namespace 2015-10-23 15:58:42 +02:00
Alexander Kukushkin 8e41d14283 Merge branch 'master' of github.com:zalando/patroni into feature/custom-namespace 2015-10-23 09:13:52 +02:00
Alexander Kukushkin deaaf8ad1a Fix unit-test for Postgresql.controldata() 2015-10-21 15:38:51 +02:00
Alexander Kukushkin 2c7e3f60cc Make possible to override default namespace (/service/) from a config file
If the namespace is not specified in a config file /service/ would be
used.
Also it's possible to use just '/' as a namespace. It means we would
have following structure:
  /scope1
  /scope2
  ...
2015-10-21 15:34:55 +02:00
Alexander Kukushkin c4a6dd48d3 remove debug print statement 2015-10-21 11:09:37 +02:00
Alexander Kukushkin 8bd28507a9 format tests according to the latest pep8 standards 2015-10-21 11:08:06 +02:00
Alexander Kukushkin 0096b6b06f Schedule update of machines cache when api_execute call has failed
Such situation could happen if we replaced all etcd nodes except one
which was used by patroni. After replacing the last node patroni will
try to execute request on all other nodes from machines_cache but non of
them are available. Michines cache would became empty and patroni will
stick to the latest node which was available in the machines_cache and
will never try to refresh machines_cache from dns for example.

Currently machines cache is refreshed only when one request to the etcd
cluster has failed, but probably it should be done periodically, for
example every minute...
2015-10-21 10:56:43 +02:00
Alexander Kukushkin f53c968d8b Improve tests 2015-10-20 14:36:49 +02:00
Alexander Kukushkin 18eebdadaa Watch for change of failover key.
If the value is empty and leader didn't changed, this probably means
that failover failed.

After 15 seconds timeout we will consider failover status = unknown
2015-10-19 15:00:06 +02:00
Alexander Kukushkin 2ebbebac7f Merge branch 'master' of github.com:zalando/patroni into feature/api 2015-10-19 09:37:11 +02:00
Alexander Kukushkin 921e4fc323 psycopg2 should be not older than 2.6.1 2015-10-16 10:28:15 +02:00
Alexander Kukushkin 3ed82ae22c Manual failover via rest api
curl -XPOST --data '{"leader": "leader_name", "member": "member_name"}' http://127.0.0.1:8008/failover

It will execute some preliminary checks and write failover key into DCS.
Afterward it will wait until new leader key will appear in a DCS.

It's better to execute this request on the master node. It will send a
signal to the main HA loop which makes possible to release leader key
immidiately even if you are working with etcd.
2015-10-15 16:18:28 +02:00
Alexander Kukushkin f35d109810 Bugfix: do not try to double encode data 2015-10-15 16:17:11 +02:00
Alexander Kukushkin 16a0a3481d fix pep8 formatting 2015-10-15 09:08:33 +02:00
Alexander Kukushkin 2f0cf1db06 Mock etcd client delete method 2015-10-15 09:08:16 +02:00
Alexander Kukushkin cf6be5f58e add missing tests for async_executor 2015-10-09 16:02:34 +02:00
Alexander Kukushkin a588ddc23c Merge branch 'master' of github.com:zalando/patroni into feature/manual-failover 2015-10-09 15:58:38 +02:00
Alexander Kukushkin a6603e8b48 bugfix in zookeeper module:
when master node was being attached to patroni/zookeeper (no cluster in
zookeeper yet) patroni has never tried to "refetch" cluster from DCS.
It was leeding to demote...
2015-10-08 13:07:38 +02:00
Alexander Kukushkin 8a844285ff Set fetch_cluster flag to False when _inner_load_cluster called
Set the same flag to True if the cluster does not yet exists in
ZooKeeper
2015-10-07 16:48:39 +02:00
Alexander Kukushkin d48f8384ed leader variable should be None if the leader.name == my name.
This check has to be performed even check_recovery_conf call
2015-10-06 10:07:40 +02:00