Release 3.0.1 (#2561 )

* Bump version * Update release notes * Return 3.6 to supported versions in setup.py
Pass 'master' role to a callback script instead of 'promoted' (#2554 )
2026-08-26 07:30:14 +00:00 · 2023-02-16 08:51:47 +01:00 · 2023-02-08 14:09:51 +01:00 · 2023-02-03 15:29:25 +01:00 · 2023-02-02 11:50:19 +01:00 · 2023-01-30 10:29:08 +01:00
96 changed files with 5410 additions and 1521 deletions
@@ -20,9 +20,9 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.

 **Environment**
- Patroni version:
- PostgreSQL version:
- DCS (and its version):
+-   Patroni version:
+-   PostgreSQL version:
+-   DCS (and its version):

 **Patroni configuration file**
 ```
@@ -19,7 +19,7 @@ def install_requirements(what):
    requirements = ['mock>=2.0.0', 'flake8', 'pytest', 'pytest-cov'] if what == 'all' else ['behave']
    requirements += ['coverage']
    # try to split tests between psycopg2 and psycopg3
-    requirements += ['psycopg[binary]'] if sys.version_info >= (3, 6, 0) and\
+    requirements += ['psycopg[binary]'] if sys.version_info > (3, 7, 0) and\
        (sys.platform != 'darwin' or what == 'etcd3') else ['psycopg2-binary']
    for r in read('requirements.txt').split('\n'):
        r = r.strip()
@@ -45,6 +45,8 @@ def install_packages(what):
    packages['exhibitor'] = packages['zookeeper']
    packages = packages.get(what, [])
    ver = versions.get(what)
+    if float(ver) >= 15:
+        packages += ['postgresql-{0}-citus-11.2'.format(ver)]
    subprocess.call(['sudo', 'apt-get', 'update', '-y'])
    return subprocess.call(['sudo', 'apt-get', 'install', '-y', 'postgresql-' + ver, 'expect-dev'] + packages)

@@ -96,7 +98,7 @@ def unpack(archive, name):


 def install_etcd():
-    version = os.environ.get('ETCDVERSION', '3.3.13')
+    version = os.environ.get('ETCDVERSION', '3.4.23')
    platform = {'linux2': 'linux', 'win32': 'windows', 'cygwin': 'windows'}.get(sys.platform, sys.platform)
    dirname = 'etcd-v{0}-{1}-amd64'.format(version, platform)
    ext = 'tar.gz' if platform == 'linux' else 'zip'
@@ -108,16 +110,17 @@ def install_etcd():


 def install_postgres():
-    version = os.environ.get('PGVERSION', '14.1-1')
+    version = os.environ.get('PGVERSION', '15.1-1')
    platform = {'darwin': 'osx', 'win32': 'windows-x64', 'cygwin': 'windows-x64'}[sys.platform]
+    if platform == 'osx':
+        return subprocess.call(['brew', 'install', 'expect', 'postgresql@{0}'.format(version.split('.')[0])])
    name = 'postgresql-{0}-{1}-binaries.zip'.format(version, platform)
    get_file('http://get.enterprisedb.com/postgresql/' + name, name)
    unzip_all(name)
    bin_dir = os.path.join('pgsql', 'bin')
    for f in os.listdir(bin_dir):
        chmod_755(os.path.join(bin_dir, f))
-    subprocess.call(['pgsql/bin/postgres', '-V'])
-    return 0
+    return subprocess.call(['pgsql/bin/postgres', '-V'])


 def main():
@@ -29,22 +29,19 @@ def main():
        path = '/usr/lib/postgresql/{0}/bin:.'.format(version)
        unbuffer = ['timeout', '900', 'unbuffer']
    else:
-        path = os.path.abspath(os.path.join('pgsql', 'bin'))
        if sys.platform == 'darwin':
-            path += ':.'
-        unbuffer = []
+            version = os.environ.get('PGVERSION', '15.1-1')
+            path = '/usr/local/opt/postgresql@{0}/bin:.'.format(version.split('.')[0])
+            unbuffer = ['unbuffer']
+        else:
+            path = os.path.abspath(os.path.join('pgsql', 'bin'))
+            unbuffer = []
    env['PATH'] = path + os.pathsep + env['PATH']
    env['DCS'] = what
    if what == 'kubernetes':
        env['PATRONI_KUBERNETES_CONTEXT'] = 'k3d-k3s-default'

-    ret = subprocess.call(unbuffer + [sys.executable, '-m', 'behave'], env=env)
-
-    if ret != 0:
-        if subprocess.call('grep . features/output/*_failed/*postgres?.*', shell=True) != 0:
-            subprocess.call('grep . features/output/*/*postgres?.*', shell=True)
-        return 1
-    return 0
+    return subprocess.call(unbuffer + [sys.executable, '-m', 'behave'], env=env)


 if __name__ == '__main__':
@@ -5,12 +5,14 @@ on:
  push:
    branches:
    - master
-    tags:
-    - v.*
+
+env:
+  CODACY_PROJECT_TOKEN: ${{ secrets.CODACY_PROJECT_TOKEN }}
+  SECRETS_AVAILABLE: ${{ secrets.CODACY_PROJECT_TOKEN != '' }}

 jobs:
  unit:
-    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-20.04","windows":"windows-latest","macos":"macos-latest"}')[matrix.os] }}
+    runs-on: ${{ matrix.os }}-latest
    strategy:
      fail-fast: false
      matrix:
@@ -18,26 +20,6 @@ jobs:

    steps:
    - uses: actions/checkout@v3
-    - name: Set up Python 2.7
-      uses: actions/setup-python@v4
-      with:
-        python-version: 2.7
-      if: matrix.os != 'windows'
-    - name: Install dependencies
-      run: python .github/workflows/install_deps.py
-      if: matrix.os != 'windows'
-    - name: Run tests and flake8
-      run: python .github/workflows/run_tests.py
-      if: matrix.os != 'windows'
-
-    - name: Set up Python 3.6
-      uses: actions/setup-python@v4
-      with:
-        python-version: 3.6
-    - name: Install dependencies
-      run: python .github/workflows/install_deps.py
-    - name: Run tests and flake8
-      run: python .github/workflows/run_tests.py

    - name: Set up Python 3.7
      uses: actions/setup-python@v4
@@ -75,6 +57,15 @@ jobs:
    - name: Run tests and flake8
      run: python .github/workflows/run_tests.py

+    - name: Set up Python 3.11
+      uses: actions/setup-python@v4
+      with:
+        python-version: 3.11
+    - name: Install dependencies
+      run: python .github/workflows/install_deps.py
+    - name: Run tests and flake8
+      run: python .github/workflows/run_tests.py
+
    - name: Combine coverage
      run: python .github/workflows/run_tests.py combine

@@ -89,29 +80,26 @@ jobs:
      run: python -m coveralls --service=github

  behave:
-    runs-on: ${{ fromJson('{"ubuntu":"ubuntu-20.04","windows":"windows-latest","macos":"macos-latest"}')[matrix.os] }}
+    runs-on: ${{ matrix.os }}-latest
    env:
      DCS: ${{ matrix.dcs }}
-      ETCDVERSION: 3.3.13
-      PGVERSION: 12.1-1  # for windows and macos
+      ETCDVERSION: 3.4.23
+      PGVERSION: 15.1-1  # for windows and macos
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu]
-        python-version: [2.7, 3.6, 3.9]
+        python-version: [3.7, '3.10']
        dcs: [etcd, etcd3, consul, exhibitor, kubernetes, raft]
-        exclude:
-        - dcs: kubernetes
-          python-version: 2.7
        include:
        - os: macos
-          python-version: 3.7
+          python-version: 3.8
          dcs: raft
        - os: macos
-          python-version: 3.8
+          python-version: 3.9
          dcs: etcd
        - os: macos
-          python-version: '3.10'
+          python-version: 3.11
          dcs: etcd3

    steps:
@@ -122,32 +110,38 @@ jobs:
        python-version: ${{ matrix.python-version }}
    - uses: nolar/setup-k3d-k3s@v1
      if: matrix.dcs == 'kubernetes'
-    - name: Add postgresql apt repo
+    - name: Add postgresql and citus apt repo
      run: |
        sudo apt-get update -y
-        sudo apt-get install -y wget ca-certificates gnupg
+        sudo apt-get install -y wget ca-certificates gnupg debian-archive-keyring apt-transport-https
        sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
        sudo sh -c 'wget -qO - https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor > /etc/apt/trusted.gpg.d/apt.postgresql.org.gpg'
+        sudo sh -c 'echo "deb [signed-by=/etc/apt/trusted.gpg.d/citusdata_community.gpg] https://repos.citusdata.com/community/ubuntu/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/citusdata_community.list'
+        sudo sh -c 'wget -qO - https://repos.citusdata.com/community/gpgkey | gpg --dearmor > /etc/apt/trusted.gpg.d/citusdata_community.gpg'
      if: matrix.os == 'ubuntu'
    - name: Install dependencies
      run: python .github/workflows/install_deps.py
    - name: Run behave tests
      run: python .github/workflows/run_tests.py
-    - uses: actions/setup-python@v4
+    - name: Upload logs if behave failed
+      uses: actions/upload-artifact@v3
+      if: failure()
      with:
-        python-version: '3.10'
-    - name: Install coveralls
-      run: python -m pip install coveralls
-    - name: Upload Coverage
-      env:
-        COVERALLS_FLAG_NAME: behave-${{ matrix.os }}-${{ matrix.dcs }}-${{ matrix.python-version }}
-        COVERALLS_PARALLEL: 'true'
-        GITHUB_TOKEN: ${{ secrets.github_token }}
-      run: python -m coveralls --service=github
+        name: behave-${{ matrix.os }}-${{ matrix.dcs }}-${{ matrix.python-version }}-logs
+        path: |
+          features/output/*_failed/*postgres?.*
+          features/output/*.log
+        if-no-files-found: error
+        retention-days: 5
+    - name: Generate coverage xml report
+      run: python -m coverage xml -o cobertura.xml
+    - name: Upload coverage to Codacy
+      run: bash <(curl -Ls https://coverage.codacy.com/get.sh) report -r cobertura.xml -l Python --partial
+      if: ${{ env.SECRETS_AVAILABLE == 'true' }}

  coveralls-finish:
    name: Finalize coveralls.io
-    needs: [unit, behave]
+    needs: unit
    runs-on: ubuntu-latest
    steps:
    - uses: actions/setup-python@v4
@@ -155,3 +149,11 @@ jobs:
    - run: python -m coveralls --service=github --finish
      env:
        GITHUB_TOKEN: ${{ secrets.github_token }}
+
+  codacy-final:
+    name: Finalize Codacy
+    needs: behave
+    runs-on: ubuntu-latest
+    steps:
+    - run: bash <(curl -Ls https://coverage.codacy.com/get.sh) final
+      if: ${{ env.SECRETS_AVAILABLE == 'true' }}
@@ -43,18 +43,18 @@ RUN set -ex \
    && echo 'syntax on\nfiletype plugin indent on\nset mouse-=a\nautocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab' > /etc/vim/vimrc.local \
 \
    # Prepare postgres/patroni/haproxy environment
-    && mkdir -p $PGHOME/.config/patroni /patroni /run/haproxy \
-    && ln -s ../../postgres0.yml $PGHOME/.config/patroni/patronictl.yaml \
+    && mkdir -p "$PGHOME/.config/patroni" /patroni /run/haproxy \
+    && ln -s ../../postgres0.yml "$PGHOME/.config/patroni/patronictl.yaml" \
    && ln -s /patronictl.py /usr/local/bin/patronictl \
    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
    && chown -R postgres:postgres /var/log \
 \
    # Download etcd
-    && curl -sL https://github.com/coreos/etcd/releases/download/v${ETCDVERSION}/etcd-v${ETCDVERSION}-linux-$(dpkg --print-architecture).tar.gz \
+    && curl -sL "https://github.com/coreos/etcd/releases/download/v$ETCDVERSION/etcd-v$ETCDVERSION-linux-$(dpkg --print-architecture).tar.gz" \
            | tar xz -C /usr/local/bin --strip=1 --wildcards --no-anchored etcd etcdctl \
 \
    # Download confd
-    && curl -sL https://github.com/kelseyhightower/confd/releases/download/v${CONFDVERSION}/confd-${CONFDVERSION}-linux-$(dpkg --print-architecture) \
+    && curl -sL "https://github.com/kelseyhightower/confd/releases/download/v$CONFDVERSION/confd-$CONFDVERSION-linux-$(dpkg --print-architecture)" \
            > /usr/local/bin/confd && chmod +x /usr/local/bin/confd \
 \
    # Clean up all useless packages and some files
@@ -153,7 +153,7 @@ RUN sed -i 's/env python/&3/' /patroni*.py \
    && sed -i 's/^      parameters:/      pg_hba:\n      - local all all trust\n      - host replication all all md5\n      - host all all all md5\n&\n        max_connections: 100/'  postgres?.yml \
    && if [ "$COMPRESS" = "true" ]; then chmod u+s /usr/bin/sudo; fi \
    && chmod +s /bin/ping \
-    && chown -R postgres:postgres $PGHOME /run /etc/haproxy
+    && chown -R postgres:postgres "$PGHOME" /run /etc/haproxy

 USER postgres

@@ -0,0 +1,173 @@
+## This Dockerfile is meant to aid in the building and debugging patroni whilst developing on your local machine
+## It has all the necessary components to play/debug with a single node appliance, running etcd
+ARG PG_MAJOR=15
+ARG COMPRESS=false
+ARG PGHOME=/home/postgres
+ARG PGDATA=$PGHOME/data
+ARG LC_ALL=C.UTF-8
+ARG LANG=C.UTF-8
+
+FROM postgres:$PG_MAJOR as builder
+
+ARG PGHOME
+ARG PGDATA
+ARG LC_ALL
+ARG LANG
+
+ENV ETCDVERSION=3.3.13 CONFDVERSION=0.16.0
+
+RUN set -ex \
+    && export DEBIAN_FRONTEND=noninteractive \
+    && echo 'APT::Install-Recommends "0";\nAPT::Install-Suggests "0";' > /etc/apt/apt.conf.d/01norecommend \
+    && apt-get update -y \
+    # postgres:10 is based on debian, which has the patroni package. We will install all required dependencies
+    && apt-cache depends patroni | sed -n -e 's/.*Depends: \(python3-.\+\)$/\1/p' \
+            | grep -Ev '^python3-(sphinx|etcd|consul|kazoo|kubernetes)' \
+            | xargs apt-get install -y vim curl less jq locales haproxy sudo \
+                            python3-etcd python3-kazoo python3-pip busybox \
+                            net-tools iputils-ping --fix-missing \
+    && curl https://install.citusdata.com/community/deb.sh | bash \
+    && apt-get -y install postgresql-$PG_MAJOR-citus-11.2 \
+    && pip3 install dumb-init \
+\
+    # Cleanup all locales but en_US.UTF-8
+    && find /usr/share/i18n/charmaps/ -type f ! -name UTF-8.gz -delete \
+    && find /usr/share/i18n/locales/ -type f ! -name en_US ! -name en_GB ! -name i18n* ! -name iso14651_t1 ! -name iso14651_t1_common ! -name 'translit_*' -delete \
+    && echo 'en_US.UTF-8 UTF-8' > /usr/share/i18n/SUPPORTED \
+\
+    # Make sure we have a en_US.UTF-8 locale available
+    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
+\
+    # haproxy dummy config
+    && echo 'global\n        stats socket /run/haproxy/admin.sock mode 660 level admin' > /etc/haproxy/haproxy.cfg \
+\
+    # vim config
+    && echo 'syntax on\nfiletype plugin indent on\nset mouse-=a\nautocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab' > /etc/vim/vimrc.local \
+\
+    # Prepare postgres/patroni/haproxy environment
+    && mkdir -p $PGHOME/.config/patroni /patroni /run/haproxy \
+    && ln -s ../../postgres0.yml $PGHOME/.config/patroni/patronictl.yaml \
+    && ln -s /patronictl.py /usr/local/bin/patronictl \
+    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
+    && chown -R postgres:postgres /var/log \
+\
+    # Download etcd
+    && curl -sL https://github.com/coreos/etcd/releases/download/v${ETCDVERSION}/etcd-v${ETCDVERSION}-linux-$(dpkg --print-architecture).tar.gz \
+            | tar xz -C /usr/local/bin --strip=1 --wildcards --no-anchored etcd etcdctl \
+\
+    # Download confd
+    && curl -sL https://github.com/kelseyhightower/confd/releases/download/v${CONFDVERSION}/confd-${CONFDVERSION}-linux-$(dpkg --print-architecture) \
+            > /usr/local/bin/confd && chmod +x /usr/local/bin/confd \
+    # Prepare client cert for HAProxy
+    && cat /etc/ssl/private/ssl-cert-snakeoil.key  /etc/ssl/certs/ssl-cert-snakeoil.pem > /etc/ssl/private/ssl-cert-snakeoil.crt \
+\
+    # Clean up all useless packages and some files
+    && apt-get purge -y --allow-remove-essential python3-pip gzip bzip2 util-linux e2fsprogs \
+                libmagic1 bsdmainutils login ncurses-bin libmagic-mgc e2fslibs bsdutils \
+                exim4-config gnupg-agent dirmngr libpython2.7-stdlib libpython2.7-minimal \
+    && apt-get autoremove -y \
+    && apt-get clean -y \
+    && rm -rf /var/lib/apt/lists/* \
+        /root/.cache \
+        /var/cache/debconf/* \
+        /etc/rc?.d \
+        /etc/systemd \
+        /docker-entrypoint* \
+        /sbin/pam* \
+        /sbin/swap* \
+        /sbin/unix* \
+        /usr/local/bin/gosu \
+        /usr/sbin/[acgipr]* \
+        /usr/sbin/*user* \
+        /usr/share/doc* \
+        /usr/share/man \
+        /usr/share/info \
+        /usr/share/i18n/locales/translit_hangul \
+        /usr/share/locale/?? \
+        /usr/share/locale/??_?? \
+        /usr/share/postgresql/*/man \
+        /usr/share/postgresql-common/pg_wrapper \
+        /usr/share/vim/vim80/doc \
+        /usr/share/vim/vim80/lang \
+        /usr/share/vim/vim80/tutor \
+#        /var/lib/dpkg/info/* \
+    && find /usr/bin -xtype l -delete \
+    && find /var/log -type f -exec truncate --size 0 {} \; \
+    && find /usr/lib/python3/dist-packages -name '*test*' | xargs rm -fr \
+    && find /lib/$(uname -m)-linux-gnu/security -type f ! -name pam_env.so ! -name pam_permit.so ! -name pam_unix.so -delete
+
+# perform compression if it is necessary
+ARG COMPRESS
+RUN if [ "$COMPRESS" = "true" ]; then \
+        set -ex \
+        # Allow certain sudo commands from postgres
+        && echo 'postgres ALL=(ALL) NOPASSWD: /bin/tar xpJf /a.tar.xz -C /, /bin/rm /a.tar.xz, /bin/ln -snf dash /bin/sh' >> /etc/sudoers \
+        && ln -snf busybox /bin/sh \
+        && arch=$(uname -m) \
+        && darch=$(uname -m | sed 's/_/-/') \
+        && files="/bin/sh /usr/bin/sudo /usr/lib/sudo/sudoers.so /lib/$arch-linux-gnu/security/pam_*.so" \
+        && libs="$(ldd $files | awk '{print $3;}' | grep '^/' | sort -u) /lib/ld-linux-$darch.so.* /lib/$arch-linux-gnu/ld-linux-$darch.so.* /lib/$arch-linux-gnu/libnsl.so.* /lib/$arch-linux-gnu/libnss_compat.so.* /lib/$arch-linux-gnu/libnss_files.so.*" \
+        && (echo /var/run $files $libs | tr ' ' '\n' && realpath $files $libs) | sort -u | sed 's/^\///' > /exclude \
+        && find /etc/alternatives -xtype l -delete \
+        && save_dirs="usr lib var bin sbin etc/ssl etc/init.d etc/alternatives etc/apt" \
+        && XZ_OPT=-e9v tar -X /exclude -cpJf a.tar.xz $save_dirs \
+        # we call "cat /exclude" to avoid including files from the $save_dirs that are also among
+        # the exceptions listed in the /exclude, as "uniq -u" eliminates all non-unique lines.
+        # By calling "cat /exclude" a second time we guarantee that there will be at least two lines
+        # for each exception and therefore they will be excluded from the output passed to 'rm'.
+        && /bin/busybox sh -c "(find $save_dirs -not -type d && cat /exclude /exclude && echo exclude) | sort | uniq -u | xargs /bin/busybox rm" \
+        && /bin/busybox --install -s \
+        && /bin/busybox sh -c "find $save_dirs -type d -depth -exec rmdir -p {} \; 2> /dev/null"; \
+    else \
+        /bin/busybox --install -s; \
+    fi
+
+FROM scratch
+COPY --from=builder / /
+
+LABEL maintainer="Alexander Kukushkin <[email protected]>"
+
+ARG PG_MAJOR
+ARG COMPRESS
+ARG PGHOME
+ARG PGDATA
+ARG LC_ALL
+ARG LANG
+
+ARG PGBIN=/usr/lib/postgresql/$PG_MAJOR/bin
+
+ENV LC_ALL=$LC_ALL LANG=$LANG EDITOR=/usr/bin/editor
+ENV PGDATA=$PGDATA PATH=$PATH:$PGBIN
+
+COPY patroni /patroni/
+COPY extras/confd/conf.d/haproxy.toml /etc/confd/conf.d/
+COPY extras/confd/templates/haproxy-citus.tmpl /etc/confd/templates/haproxy.tmpl
+COPY patroni*.py docker/entrypoint.sh /
+COPY postgres?.yml $PGHOME/
+
+WORKDIR $PGHOME
+
+RUN sed -i 's/env python/&3/' /patroni*.py \
+    # "fix" patroni configs
+    && sed -i 's/^\(  connect_address:\|  - host\)/#&/' postgres?.yml \
+    && sed -i 's/^  listen: 127.0.0.1/  listen: 0.0.0.0/' postgres?.yml \
+    && sed -i "s|^\(  data_dir: \).*|\1$PGDATA|" postgres?.yml \
+    && sed -i "s|^#\(  bin_dir: \).*|\1$PGBIN|" postgres?.yml \
+    && sed -i 's/^  - encoding: UTF8/  - locale: en_US.UTF-8\n&/' postgres?.yml \
+    && sed -i 's/^scope:/log:\n  loggers:\n    patroni.postgresql.citus: DEBUG\n#&/' postgres?.yml \
+    && sed -i 's/^\(name\|etcd\|  host\|  authentication\|  pg_hba\|  parameters\):/#&/' postgres?.yml \
+    && sed -i 's/^    \(replication\|superuser\|rewind\|unix_socket_directories\|\(\(  \)\{0,1\}\(username\|password\)\)\):/#&/' postgres?.yml \
+    && sed -i 's/^postgresql:/&\n  basebackup:\n    checkpoint: fast/' postgres?.yml \
+    && sed -i 's|^      parameters:|      pg_hba:\n      - local all all trust\n      - hostssl replication all all md5 clientcert=verify-ca\n      - hostssl all all all md5 clientcert=verify-ca\n&\n        max_connections: 100\n        shared_buffers: 16MB\n        ssl: "on"\n        ssl_ca_file: /etc/ssl/certs/ssl-cert-snakeoil.pem\n        ssl_cert_file: /etc/ssl/certs/ssl-cert-snakeoil.pem\n        ssl_key_file: /etc/ssl/private/ssl-cert-snakeoil.key\n        citus.node_conninfo: "sslrootcert=/etc/ssl/certs/ssl-cert-snakeoil.pem sslkey=/etc/ssl/private/ssl-cert-snakeoil.key sslcert=/etc/ssl/certs/ssl-cert-snakeoil.pem sslmode=verify-ca"|'  postgres?.yml \
+    && sed -i 's/^#\(ctl\|  certfile\|  keyfile\)/\1/' postgres?.yml \
+    && sed -i 's|^#  cafile: .*$|  verify_client: required\n  cafile: /etc/ssl/certs/ssl-cert-snakeoil.pem|' postgres?.yml \
+    && sed -i 's|^#  cacert: .*$|  cacert: /etc/ssl/certs/ssl-cert-snakeoil.pem|' postgres?.yml \
+    && sed -i 's/^#  insecure: .*/  insecure: on/' postgres?.yml \
+    # client cert for HAProxy to access Patroni REST API
+    && if [ "$COMPRESS" = "true" ]; then chmod u+s /usr/bin/sudo; fi \
+    && chmod +s /bin/ping \
+    && chown -R postgres:postgres $PGHOME /run /etc/haproxy
+
+USER postgres
+
+ENTRYPOINT ["/bin/sh", "/entrypoint.sh"]
@@ -14,6 +14,8 @@ We call Patroni a "template" because it is far from being a one-size-fits-all or

 Currently supported PostgreSQL versions: 9.3 to 15.

+**Note to Citus users**: Starting from 3.0 Patroni nicely integrates with the `Citus <https://github.com/citusdata/citus>`__ database extension to Postgres. Please check the `Citus support page <https://github.com/zalando/patroni/blob/master/docs/citus.rst>`__ in the Patroni documentation for more info about how to use Patroni high availability together with a Citus distributed cluster.
+
 **Note to Kubernetes users**: Patroni can run natively on top of Kubernetes. Take a look at the `Kubernetes <https://github.com/zalando/patroni/blob/master/docs/kubernetes.rst>`__ chapter of the Patroni documentation.

 .. contents::
@@ -47,7 +49,7 @@ We report new releases information `here <https://github.com/zalando/patroni/rel
 Community
 =========

-There are two places to connect with the Patroni community: `on github <https://github.com/zalando/patroni>`__, via Issues and PRs, and on channel #patroni in the `PostgreSQL Slack <https://postgres-slack.herokuapp.com/>`__.  If you're using Patroni, or just interested, please join us.
+There are two places to connect with the Patroni community: `on github <https://github.com/zalando/patroni>`__, via Issues and PRs, and on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://postgresteam.slack.com/>`__.  If you're using Patroni, or just interested, please join us.

 ===================================
 Technical Requirements/Installation
@@ -1,12 +0,0 @@
-Failover
-========
- When determining who should become master, include the minor version of PostgreSQL in the decision.
-
-Configuration
-==============
- Provide a way to change pg_hba.conf of a running cluster on the Patroni level, without changing individual nodes.
- Provide hooks to store and retrieve cluster-wide passwords without exposing them in a plain-text form to unauthorized users.
-
-Documentation
-==============
- Document how to run cascading replication and possibly initialize the cluster without an access to the master node.
@@ -0,0 +1,139 @@
+# docker compose file for running a Citus cluster
+# with 3-node etcd v3 cluster as the DCS and one haproxy node.
+# The Citus cluster has a coordinator (3 nodes)
+# and two worker clusters (2 nodes).
+#
+# Before starting it up you need to build the docker image:
+# $ docker build -f Dockerfile.citus -t patroni-citus .
+# The cluster could be started as:
+# $ docker-compose -f docker-compose-citus.yml up -d
+# You can read more about it in the:
+# https://github.com/zalando/patroni/blob/master/docker/README.md#citus-cluster
+version: "2"
+
+networks:
+    demo:
+
+services:
+    etcd1: &etcd
+        image: patroni-citus
+        networks: [ demo ]
+        environment:
+            ETCDCTL_API: 3
+            ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
+            ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
+            ETCD_INITIAL_CLUSTER: etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
+            ETCD_INITIAL_CLUSTER_STATE: new
+            ETCD_INITIAL_CLUSTER_TOKEN: tutorial
+        container_name: demo-etcd1
+        hostname: etcd1
+        command: etcd -name etcd1 -initial-advertise-peer-urls http://etcd1:2380
+
+    etcd2:
+        <<: *etcd
+        container_name: demo-etcd2
+        hostname: etcd2
+        command: etcd -name etcd2 -initial-advertise-peer-urls http://etcd2:2380
+
+    etcd3:
+        <<: *etcd
+        container_name: demo-etcd3
+        hostname: etcd3
+        command: etcd -name etcd3 -initial-advertise-peer-urls http://etcd3:2380
+
+    haproxy:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: haproxy
+        container_name: demo-haproxy
+        ports:
+            - "5000:5000"  # Access to the coorinator primary
+            - "5001:5001"  # Load-balancing across workers primaries
+        command: haproxy
+        environment: &haproxy_env
+            ETCDCTL_API: 3
+            ETCDCTL_ENDPOINTS: http://etcd1:2379,http://etcd2:2379,http://etcd3:2379
+            PATRONI_ETCD3_HOSTS: "'etcd1:2379','etcd2:2379','etcd3:2379'"
+            PATRONI_SCOPE: demo
+            PATRONI_CITUS_GROUP: 0
+            PATRONI_CITUS_DATABASE: citus
+            PGSSLMODE: verify-ca
+            PGSSLKEY: /etc/ssl/private/ssl-cert-snakeoil.key
+            PGSSLCERT: /etc/ssl/certs/ssl-cert-snakeoil.pem
+            PGSSLROOTCERT: /etc/ssl/certs/ssl-cert-snakeoil.pem
+
+    coord1:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: coord1
+        container_name: demo-coord1
+        environment: &coord_env
+            <<: *haproxy_env
+            PATRONI_NAME: coord1
+            PATRONI_CITUS_GROUP: 0
+
+    coord2:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: coord2
+        container_name: demo-coord2
+        environment:
+            <<: *coord_env
+            PATRONI_NAME: coord2
+
+    coord3:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: coord3
+        container_name: demo-coord3
+        environment:
+            <<: *coord_env
+            PATRONI_NAME: coord3
+
+
+    work1-1:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: work1-1
+        container_name: demo-work1-1
+        environment: &work1_env
+            <<: *haproxy_env
+            PATRONI_NAME: work1-1
+            PATRONI_CITUS_GROUP: 1
+
+    work1-2:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: work1-2
+        container_name: demo-work1-2
+        environment:
+            <<: *work1_env
+            PATRONI_NAME: work1-2
+
+
+    work2-1:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: work2-1
+        container_name: demo-work2-1
+        environment: &work2_env
+            <<: *haproxy_env
+            PATRONI_NAME: work2-1
+            PATRONI_CITUS_GROUP: 2
+
+    work2-2:
+        image: patroni-citus
+        networks: [ demo ]
+        env_file: docker/patroni.env
+        hostname: work2-2
+        container_name: demo-work2-2
+        environment:
+            <<: *work2_env
+            PATRONI_NAME: work2-2
@@ -1,5 +1,12 @@
 # docker compose file for running a 3-node PostgreSQL cluster
 # with 3-node etcd cluster as the DCS and one haproxy node
+#
+# requires a patroni image build from the Dockerfile:
+# $ docker build -t patroni .
+# The cluster could be started as:
+# $ docker-compose up -d
+# You can read more about it in the:
+# https://github.com/zalando/patroni/blob/master/docker/README.md
 version: "2"

 networks:
@@ -1,10 +1,10 @@
-# Patroni Dockerfile
-You can run Patroni in a docker container using this Dockerfile
+# Dockerfile and Dockerfile.citus
+You can run Patroni in a docker container using these Dockerfiles

-This Dockerfile is meant in aiding development of Patroni and quick testing of features. It is not a production-worthy
-Dockerfile
+They are meant in aiding development of Patroni and quick testing of features and not a production-worthy!

    docker build -t patroni .
+    docker build -f Dockerfile.citus -t patroni-citus .

 # Examples

@@ -12,7 +12,10 @@ Dockerfile

    docker run -d patroni

-## Three-node Patroni cluster with three-node etcd cluster and one haproxy container using docker-compose
+## Three-node Patroni cluster
+
+In addition to three Patroni containers the stack starts three containers with etcd (forming a three-node cluster), and one container with haproxy.
+The haproxy listens on ports 5000 (connects to the primary) and 5001 (does load-balancing between healthy standbys).

 Example session:

@@ -92,7 +95,8 @@ Example session:
    b2e169fcb8a34028: name=etcd1 peerURLs=http://etcd1:2380 clientURLs=http://etcd1:2379 isLeader=false
    postgres@patroni1:~$ exit

-    $ psql -h localhost -p 5000 -U postgres -W
+    $ docker exec -ti demo-haproxy bash
+    postgres@haproxy:~$ psql -h localhost -p 5000 -U postgres -W
    Password: postgres
    psql (11.2 (Ubuntu 11.2-1.pgdg18.04+1), server 10.7 (Debian 10.7-1.pgdg90+1))
    Type "help" for help.
@@ -105,7 +109,7 @@ Example session:

    localhost/postgres=# \q

-    $ psql -h localhost -p 5001 -U postgres -W
+    $postgres@haproxy:~ psql -h localhost -p 5001 -U postgres -W
    Password: postgres
    psql (11.2 (Ubuntu 11.2-1.pgdg18.04+1), server 10.7 (Debian 10.7-1.pgdg90+1))
    Type "help" for help.
@@ -115,3 +119,188 @@ Example session:
    ───────────────────
     t
    (1 row)
+
+## Citus cluster
+
+The stack starts three containers with etcd (forming a three-node etcd cluster), seven containers with Patroni+PostgreSQL+Citus (three coordinator nodes, and two worker clusters with two nodes each), and one container with haproxy.
+The haproxy listens on ports 5000 (connects to the coordinator primary) and 5001 (does load-balancing between worker primary nodes).
+
+Example session:
+
+    $ docker-compose -f docker-compose-citus.yml up -d
+    Creating demo-work2-1 ... done
+    Creating demo-work1-1 ... done
+    Creating demo-etcd2   ... done
+    Creating demo-etcd1   ... done
+    Creating demo-coord3  ... done
+    Creating demo-etcd3   ... done
+    Creating demo-coord1  ... done
+    Creating demo-haproxy ... done
+    Creating demo-work2-2 ... done
+    Creating demo-coord2  ... done
+    Creating demo-work1-2 ... done
+
+    $ docker ps
+    CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS         PORTS                              NAMES
+    852d8885a612   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 3 seconds                                      demo-coord3
+    cdd692f947ab   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 3 seconds                                      demo-work1-2
+    9f4e340b36da   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 3 seconds                                      demo-etcd3
+    d69c129a960a   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 4 seconds                                      demo-etcd1
+    c5849689b8cd   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 4 seconds                                      demo-coord1
+    c9d72bd6217d   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 3 seconds                                      demo-work2-1
+    24b1b43efa05   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 4 seconds                                      demo-coord2
+    cb0cc2b4ca0a   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 3 seconds                                      demo-work2-2
+    9796c6b8aad5   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 5 seconds                                      demo-work1-1
+    8baccd74dcae   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 4 seconds                                      demo-etcd2
+    353ec62a0187   patroni-citus          "/bin/sh /entrypoint…"   6 seconds ago   Up 4 seconds   0.0.0.0:5000-5001->5000-5001/tcp   demo-haproxy
+
+    $ docker logs demo-coord1
+    2023-01-05 15:09:31,295 INFO: Selected new etcd server http://172.27.0.4:2379
+    2023-01-05 15:09:31,388 INFO: Lock owner: None; I am coord1
+    2023-01-05 15:09:31,501 INFO: trying to bootstrap a new cluster
+    ...
+    2023-01-05 15:09:45,096 INFO: postmaster pid=39
+    localhost:5432 - no response
+    2023-01-05 15:09:45.137 UTC [39] LOG:  starting PostgreSQL 15.1 (Debian 15.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
+    2023-01-05 15:09:45.137 UTC [39] LOG:  listening on IPv4 address "0.0.0.0", port 5432
+    2023-01-05 15:09:45.152 UTC [39] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
+    2023-01-05 15:09:45.177 UTC [43] LOG:  database system was shut down at 2023-01-05 15:09:32 UTC
+    2023-01-05 15:09:45.193 UTC [39] LOG:  database system is ready to accept connections
+    localhost:5432 - accepting connections
+    localhost:5432 - accepting connections
+    2023-01-05 15:09:46,139 INFO: establishing a new patroni connection to the postgres cluster
+    2023-01-05 15:09:46,208 INFO: running post_bootstrap
+    2023-01-05 15:09:47.209 UTC [55] LOG:  starting maintenance daemon on database 16386 user 10
+    2023-01-05 15:09:47.209 UTC [55] CONTEXT:  Citus maintenance daemon for database 16386 user 10
+    2023-01-05 15:09:47,215 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
+    2023-01-05 15:09:47.446 UTC [41] LOG:  checkpoint starting: immediate force wait
+    2023-01-05 15:09:47,466 INFO: initialized a new cluster
+    2023-01-05 15:09:47,594 DEBUG: query(SELECT nodeid, groupid, nodename, nodeport, noderole FROM pg_catalog.pg_dist_node WHERE noderole = 'primary', ())
+    2023-01-05 15:09:47,594 INFO: establishing a new patroni connection to the postgres cluster
+    2023-01-05 15:09:47,467 INFO: Lock owner: coord1; I am coord1
+    2023-01-05 15:09:47,613 DEBUG: query(SELECT pg_catalog.citus_set_coordinator_host(%s, %s, 'primary', 'default'), ('172.27.0.6', 5432))
+    2023-01-05 15:09:47,924 INFO: no action. I am (coord1), the leader with the lock
+    2023-01-05 15:09:51.282 UTC [41] LOG:  checkpoint complete: wrote 1086 buffers (53.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.029 s, sync=3.746 s, total=3.837 s; sync files=280, longest=0.028 s, average=0.014 s; distance=8965 kB, estimate=8965 kB
+    2023-01-05 15:09:51.283 UTC [41] LOG:  checkpoint starting: immediate force wait
+    2023-01-05 15:09:51.495 UTC [41] LOG:  checkpoint complete: wrote 18 buffers (0.9%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.044 s, sync=0.091 s, total=0.212 s; sync files=15, longest=0.015 s, average=0.007 s; distance=67 kB, estimate=8076 kB
+    2023-01-05 15:09:57,467 INFO: Lock owner: coord1; I am coord1
+    2023-01-05 15:09:57,569 INFO: Assigning synchronous standby status to ['coord3']
+    server signaled
+    2023-01-05 15:09:57.574 UTC [39] LOG:  received SIGHUP, reloading configuration files
+    2023-01-05 15:09:57.580 UTC [39] LOG:  parameter "synchronous_standby_names" changed to "coord3"
+    2023-01-05 15:09:59,637 INFO: Synchronous standby status assigned to ['coord3']
+    2023-01-05 15:09:59,638 DEBUG: query(SELECT pg_catalog.citus_add_node(%s, %s, %s, 'primary', 'default'), ('172.27.0.2', 5432, 1))
+    2023-01-05 15:09:59.690 UTC [67] LOG:  standby "coord3" is now a synchronous standby with priority 1
+    2023-01-05 15:09:59.690 UTC [67] STATEMENT:  START_REPLICATION SLOT "coord3" 0/3000000 TIMELINE 1
+    2023-01-05 15:09:59,694 INFO: no action. I am (coord1), the leader with the lock
+    2023-01-05 15:09:59,704 DEBUG: query(SELECT pg_catalog.citus_add_node(%s, %s, %s, 'primary', 'default'), ('172.27.0.8', 5432, 2))
+    2023-01-05 15:10:07,625 INFO: no action. I am (coord1), the leader with the lock
+    2023-01-05 15:10:17,579 INFO: no action. I am (coord1), the leader with the lock
+
+    $ docker exec -ti demo-haproxy bash
+    postgres@haproxy:~$ etcdctl member list
+    1bab629f01fa9065, started, etcd3, http://etcd3:2380, http://172.27.0.10:2379
+    8ecb6af518d241cc, started, etcd2, http://etcd2:2380, http://172.27.0.4:2379
+    b2e169fcb8a34028, started, etcd1, http://etcd1:2380, http://172.27.0.7:2379
+
+    postgres@haproxy:~$ etcdctl get --keys-only --prefix /service/demo
+    /service/demo/0/config
+    /service/demo/0/initialize
+    /service/demo/0/leader
+    /service/demo/0/members/coord1
+    /service/demo/0/members/coord2
+    /service/demo/0/members/coord3
+    /service/demo/0/status
+    /service/demo/0/sync
+    /service/demo/1/config
+    /service/demo/1/initialize
+    /service/demo/1/leader
+    /service/demo/1/members/work1-1
+    /service/demo/1/members/work1-2
+    /service/demo/1/status
+    /service/demo/1/sync
+    /service/demo/2/config
+    /service/demo/2/initialize
+    /service/demo/2/leader
+    /service/demo/2/members/work2-1
+    /service/demo/2/members/work2-2
+    /service/demo/2/status
+    /service/demo/2/sync
+
+    postgres@haproxy:~$ psql -h localhost -p 5000 -U postgres -d citus
+    Password for user postgres: postgres
+    psql (15.1 (Debian 15.1-1.pgdg110+1))
+    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
+    Type "help" for help.
+
+    citus=# select pg_is_in_recovery();
+     pg_is_in_recovery
+    -------------------
+     f
+    (1 row)
+
+    citus=# table pg_dist_node;
+     nodeid | groupid |  nodename  | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
+    --------+---------+------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
+          1 |       0 | 172.27.0.6 |     5432 | default  | t           | t        | primary  | default     | t              | f
+          2 |       1 | 172.27.0.2 |     5432 | default  | t           | t        | primary  | default     | t              | t
+          3 |       2 | 172.27.0.8 |     5432 | default  | t           | t        | primary  | default     | t              | t
+    (3 rows)
+
+    citus=# \q
+
+    postgres@haproxy:~$ patronictl list
+    + Citus cluster: demo ----------+--------------+---------+----+-----------+
+    | Group | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    |     0 | coord1  | 172.27.0.6  | Leader       | running |  1 |           |
+    |     0 | coord2  | 172.27.0.5  | Replica      | running |  1 |         0 |
+    |     0 | coord3  | 172.27.0.9  | Sync Standby | running |  1 |         0 |
+    |     1 | work1-1 | 172.27.0.2  | Leader       | running |  1 |           |
+    |     1 | work1-2 | 172.27.0.12 | Sync Standby | running |  1 |         0 |
+    |     2 | work2-1 | 172.27.0.11 | Sync Standby | running |  1 |         0 |
+    |     2 | work2-2 | 172.27.0.8  | Leader       | running |  1 |           |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+
+    postgres@haproxy:~$ patronictl switchover --group 2 --force
+    Current cluster topology
+    + Citus cluster: demo (group: 2, 7185185529556963355) +-----------+
+    | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +---------+-------------+--------------+---------+----+-----------+
+    | work2-1 | 172.27.0.11 | Sync Standby | running |  1 |         0 |
+    | work2-2 | 172.27.0.8  | Leader       | running |  1 |           |
+    +---------+-------------+--------------+---------+----+-----------+
+    2023-01-05 15:29:29.54204 Successfully switched over to "work2-1"
+    + Citus cluster: demo (group: 2, 7185185529556963355) -------+
+    | Member  | Host        | Role    | State   | TL | Lag in MB |
+    +---------+-------------+---------+---------+----+-----------+
+    | work2-1 | 172.27.0.11 | Leader  | running |  1 |           |
+    | work2-2 | 172.27.0.8  | Replica | stopped |    |   unknown |
+    +---------+-------------+---------+---------+----+-----------+
+
+    postgres@haproxy:~$ patronictl list
+    + Citus cluster: demo ----------+--------------+---------+----+-----------+
+    | Group | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    |     0 | coord1  | 172.27.0.6  | Leader       | running |  1 |           |
+    |     0 | coord2  | 172.27.0.5  | Replica      | running |  1 |         0 |
+    |     0 | coord3  | 172.27.0.9  | Sync Standby | running |  1 |         0 |
+    |     1 | work1-1 | 172.27.0.2  | Leader       | running |  1 |           |
+    |     1 | work1-2 | 172.27.0.12 | Sync Standby | running |  1 |         0 |
+    |     2 | work2-1 | 172.27.0.11 | Leader       | running |  2 |           |
+    |     2 | work2-2 | 172.27.0.8  | Sync Standby | running |  2 |         0 |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+
+    postgres@haproxy:~$ psql -h localhost -p 5000 -U postgres -d citus
+    Password for user postgres: postgres
+    psql (15.1 (Debian 15.1-1.pgdg110+1))
+    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
+    Type "help" for help.
+
+    citus=# table pg_dist_node;
+     nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
+    --------+---------+-------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
+          1 |       0 | 172.27.0.6  |     5432 | default  | t           | t        | primary  | default     | t              | f
+          3 |       2 | 172.27.0.11 |     5432 | default  | t           | t        | primary  | default     | t              | t
+          2 |       1 | 172.27.0.2  |     5432 | default  | t           | t        | primary  | default     | t              | t
+    (3 rows)
@@ -7,29 +7,36 @@ if [ -f /a.tar.xz ]; then
    sudo ln -snf dash /bin/sh
 fi

-readonly PATRONI_SCOPE=${PATRONI_SCOPE:-batman}
-PATRONI_NAMESPACE=${PATRONI_NAMESPACE:-/service}
-readonly PATRONI_NAMESPACE=${PATRONI_NAMESPACE%/}
-readonly DOCKER_IP=$(hostname --ip-address)
+readonly PATRONI_SCOPE="${PATRONI_SCOPE:-batman}"
+PATRONI_NAMESPACE="${PATRONI_NAMESPACE:-/service}"
+readonly PATRONI_NAMESPACE="${PATRONI_NAMESPACE%/}"
+DOCKER_IP=$(hostname --ip-address)
+readonly DOCKER_IP

 case "$1" in
    haproxy)
        haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D
-        CONFD="confd -prefix=$PATRONI_NAMESPACE/$PATRONI_SCOPE -interval=10 -backend"
-        if [ ! -z "$PATRONI_ZOOKEEPER_HOSTS" ]; then
-            while ! /usr/share/zookeeper/bin/zkCli.sh -server $PATRONI_ZOOKEEPER_HOSTS ls /; do
+        set -- confd "-prefix=$PATRONI_NAMESPACE/$PATRONI_SCOPE" -interval=10 -backend
+        if [ -n "$PATRONI_ZOOKEEPER_HOSTS" ]; then
+            while ! /usr/share/zookeeper/bin/zkCli.sh -server "$PATRONI_ZOOKEEPER_HOSTS" ls /; do
                sleep 1
            done
-            exec dumb-init $CONFD zookeeper -node $PATRONI_ZOOKEEPER_HOSTS
+            set -- "$@" zookeeper -node "$PATRONI_ZOOKEEPER_HOSTS"
        else
-            while ! etcdctl cluster-health 2> /dev/null; do
+            while ! etcdctl member list 2> /dev/null; do
                sleep 1
            done
-            exec dumb-init $CONFD etcdv3 -node $(echo $ETCDCTL_ENDPOINTS | sed 's/,/ -node /g')
+            set -- "$@" etcdv3
+            while IFS='' read -r line; do
+                set -- "$@" -node "$line"
+            done <<-EOT
+$(echo "$ETCDCTL_ENDPOINTS" | sed 's/,/\n/g')
+EOT
        fi
+        exec dumb-init "$@"
        ;;
    etcd)
-        exec "$@" -advertise-client-urls http://$DOCKER_IP:2379
+        exec "$@" -advertise-client-urls "http://$DOCKER_IP:2379"
        ;;
    zookeeper)
        exec /usr/share/zookeeper/bin/zkServer.sh start-foreground
@@ -56,5 +63,13 @@ export PATRONI_REPLICATION_USERNAME="${PATRONI_REPLICATION_USERNAME:-replicator}
 export PATRONI_REPLICATION_PASSWORD="${PATRONI_REPLICATION_PASSWORD:-replicate}"
 export PATRONI_SUPERUSER_USERNAME="${PATRONI_SUPERUSER_USERNAME:-postgres}"
 export PATRONI_SUPERUSER_PASSWORD="${PATRONI_SUPERUSER_PASSWORD:-postgres}"
+export PATRONI_REPLICATION_SSLMODE="${PATRONI_REPLICATION_SSLMODE:-$PGSSLMODE}"
+export PATRONI_REPLICATION_SSLKEY="${PATRONI_REPLICATION_SSLKEY:-$PGSSLKEY}"
+export PATRONI_REPLICATION_SSLCERT="${PATRONI_REPLICATION_SSLCERT:-$PGSSLCERT}"
+export PATRONI_REPLICATION_SSLROOTCERT="${PATRONI_REPLICATION_SSLROOTCERT:-$PGSSLROOTCERT}"
+export PATRONI_SUPERUSER_SSLMODE="${PATRONI_SUPERUSER_SSLMODE:-$PGSSLMODE}"
+export PATRONI_SUPERUSER_SSLKEY="${PATRONI_SUPERUSER_SSLKEY:-$PGSSLKEY}"
+export PATRONI_SUPERUSER_SSLCERT="${PATRONI_SUPERUSER_SSLCERT:-$PGSSLCERT}"
+export PATRONI_SUPERUSER_SSLROOTCERT="${PATRONI_SUPERUSER_SSLROOTCERT:-$PGSSLROOTCERT}"

 exec python3 /patroni.py postgres0.yml
@@ -8,7 +8,7 @@ Wanna contribute to Patroni? Yay - here is how!
 Chatting
 --------

-Just want to chat with other Patroni users?  Looking for interactive troubleshooting help? Join us on channel #patroni in the `PostgreSQL Slack <https://postgres-slack.herokuapp.com/>`__.
+Just want to chat with other Patroni users?  Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://postgresteam.slack.com/>`__.

 Running tests
 -------------
@@ -33,6 +33,13 @@ It is possible to create new database users right after the successful initializ

 Example: defining ``PATRONI_admin_PASSWORD=strongpasswd`` and ``PATRONI_admin_OPTIONS='createrole,createdb'`` will cause creation of the user **admin** with the password **strongpasswd** that is allowed to create other users and databases.

+Citus
+-----
+Enables integration Patroni with `Citus <https://docs.citusdata.com>`__. If configured, Patroni will take care of registering Citus worker nodes on the coordinator. You can find more information about Citus support :ref:`here <citus>`.
+
+-  **PATRONI\_CITUS\_GROUP**: the Citus group id, integer. Use ``0`` for coordinator and ``1``, ``2``, etc... for workers
+-  **PATRONI\_CITUS\_DATABASE**: the database where ``citus`` extension should be created. Must be the same on the coordinator and all workers. Currently only one database is supported.
+
 Consul
 ------
 -  **PATRONI\_CONSUL\_HOST**: the host:port for the Consul local agent.
@@ -47,7 +54,8 @@ Consul
 -  **PATRONI\_CONSUL\_DC**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **PATRONI\_CONSUL\_CONSISTENCY**: (optional) Select consul consistency mode. Possible values are ``default``, ``consistent``, or ``stale`` (more details in `consul API reference <https://www.consul.io/api/features/consistency.html/>`__)
 -  **PATRONI\_CONSUL\_CHECKS**: (optional) list of Consul health checks used for the session. By default an empty list is used.
-  **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**
+-  **PATRONI\_CONSUL\_REGISTER\_SERVICE**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, primary, replica, or standby-leader depending on the node's role. Defaults to **false**
+-  **PATRONI\_CONSUL\_SERVICE\_TAGS**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``primary``/``replica``/``standby-leader``). By default an empty list is used.
 -  **PATRONI\_CONSUL\_SERVICE\_CHECK\_INTERVAL**: (optional) how often to perform health check against registered url
 -  **PATRONI\_CONSUL\_SERVICE\_CHECK\_TLS\_SERVER\_NAME**: (optional) overide SNI host when connecting via TLS, see also `consul agent check API reference <https://www.consul.io/api-docs/agent/check#tlsservername>`__.

@@ -110,8 +118,8 @@ Kubernetes
 -  **PATRONI\_KUBERNETES\_PORTS**: (optional) if the Service object has the name for the port, the same name must appear in the Endpoint object, otherwise service won't work. For example, if your service is defined as ``{Kind: Service, spec: {ports: [{name: postgresql, port: 5432, targetPort: 5432}]}}``, then you have to set ``PATRONI_KUBERNETES_PORTS='[{"name": "postgresql", "port": 5432}]'`` and Patroni will use it for updating subsets of the leader Endpoint. This parameter is used only if `PATRONI_KUBERNETES_USE_ENDPOINTS` is set.
 -  **PATRONI\_KUBERNETES\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file with certificates of trusted CAs to use while verifying Kubernetes API SSL certs. If not provided, patroni will use the value provided by the ServiceAccount secret.

-Raft
----
+Raft (deprecated)
+-----------------

 -  **PATRONI\_RAFT\_SELF\_ADDR**: ``ip:port`` to listen on for Raft connections. The ``self_addr`` must be accessible from other nodes of the cluster. If not set, the node will not participate in consensus.
 -  **PATRONI\_RAFT\_BIND\_ADDR**: (optional) ``ip:port`` to listen on for Raft connections. If not specified the ``self_addr`` will be used.
@@ -109,7 +109,7 @@ Planning the Number of PostgreSQL Nodes
 ---------------------------------------

 Patroni/PostgreSQL nodes are decoupled from DCS nodes (except when Patroni implements RAFT on its own) and therefore
-there is no requirement on the minimal number of nodes. Running a cluster consisting of one primary and one standby is 
+there is no requirement on the minimal number of nodes. Running a cluster consisting of one primary and one standby is
 perfectly fine. You can add more standby nodes later.

 Running and Configuring
@@ -177,7 +177,7 @@ Testing an HA solution is a time consuming process, with many variables. This is
 That said, here are some pieces of your infrastructure you should be sure to test:

 * Network (the network in front of your system as well as the NICs [physical or virtual] themselves)
-* Disk IO 
+* Disk IO
 * file limits (nofile in Linux)
 * RAM. Even if you have oomkiller turned off as suggested, the unavailability of RAM could cause issues.
 * CPU
@@ -17,10 +17,11 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
 -  **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
 -  **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
 -  **max\_timelines\_history**: maximum number of timeline history items kept in DCS.  Default value: 0. When set to 0, it keeps the full history in DCS.
-  **master\_start\_timeout**: the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for primary failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
- **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
+-  **primary\_start\_timeout**: the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for primary failure is: loop\_wait + primary\_start\_timeout + loop\_wait, unless primary\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
+- **primary\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by primary\_stop\_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, primary\_stop\_timeout does not apply.
 -  **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that successfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See :ref:`replication modes documentation <replication_modes>` for details.
 -  **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the primary. See :ref:`replication modes documentation <replication_modes>` for details.
+-  **failsafe\_mode**: Enables :ref:`DCS Failsafe Mode <dcs_failsafe_mode>`. Defaults to `false`.
 -  **postgresql**:
    -  **use\_pg\_rewind**: whether or not to use pg_rewind. Defaults to `false`.
    -  **use\_slots**: whether or not to use replication slots. Defaults to `true` on PostgreSQL 9.4+.
@@ -111,6 +112,15 @@ Bootstrap configuration
                -  **- createdb**
    -  **post\_bootstrap** or **post\_init**: An additional script that will be executed after initializing the cluster. The script receives a connection string URL (with the cluster superuser as a user name). The PGPASSFILE variable is set to the location of pgpass file.

+.. _citus_settings:
+
+Citus
+-----
+Enables integration Patroni with `Citus <https://docs.citusdata.com>`__. If configured, Patroni will take care of registering Citus worker nodes on the coordinator. You can find more information about Citus support :ref:`here <citus>`.
+
+-  **group**: the Citus group id, integer. Use ``0`` for coordinator and ``1``, ``2``, etc... for workers
+-  **database**: the database where ``citus`` extension should be created. Must be the same on the coordinator and all workers. Currently only one database is supported.
+
 .. _consul_settings:

 Consul
@@ -129,8 +139,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **dc**: (optional) Datacenter to communicate with. By default the datacenter of the host is used.
 -  **consistency**: (optional) Select consul consistency mode. Possible values are ``default``, ``consistent``, or ``stale`` (more details in `consul API reference <https://www.consul.io/api/features/consistency.html/>`__)
 -  **checks**: (optional) list of Consul health checks used for the session. By default an empty list is used.
-  **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, replica or standby-leader depending on the node's role. Defaults to **false**.
-  **service\_tags**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``replica``/``standby-leader``).  By default an empty list is used.
+-  **register\_service**: (optional) whether or not to register a service with the name defined by the scope parameter and the tag master, primary, replica, or standby-leader depending on the node's role. Defaults to **false**.
+-  **service\_tags**: (optional) additional static tags to add to the Consul service apart from the role (``master``/``primary``/``replica``/``standby-leader``). By default an empty list is used.
 -  **service\_check\_interval**: (optional) how often to perform health check against registered url. Defaults to '5s'.
 -  **service\_check\_tls\_server\_name**: (optional) overide SNI host when connecting via TLS, see also `consul agent check API reference <https://www.consul.io/api-docs/agent/check#tlsservername>`__.

@@ -183,7 +193,7 @@ ZooKeeper
 -  **key**: (optional) File with the client key.
 -  **key_password**: (optional) The client key password.
 -  **verify**: (optional) Whether to verify certificate or not. Defaults to ``true``.
-  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``. 
+-  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``.

 .. note::
    It is required to install ``kazoo>=2.6.0`` to support SSL.
@@ -212,8 +222,8 @@ Kubernetes

 .. _raft_settings:

-Raft
----
+Raft (deprecated)
+-----------------
 -  **self\_addr**: ``ip:port`` to listen on for Raft connections. The ``self_addr`` must be accessible from other nodes of the cluster. If not set, the node will not participate in consensus.
 -  **bind\_addr**: (optional) ``ip:port`` to listen on for Raft connections. If not specified the ``self_addr`` will be used.
 -  **partner\_addrs**: list of other Patroni nodes in the cluster in format: ['ip1:port', 'ip2:port', 'etc...']
@@ -319,6 +329,8 @@ PostgreSQL
    -  **replica\_method**: for each create_replica_methods other than basebackup, you would add a configuration section of the same name. At a minimum, this should include "command" with a full path to the actual script to be executed. Other configuration parameters will be passed along to the script in the form "parameter=value".
    -  **pre\_promote**: a fencing script that executes during a failover after acquiring the leader lock but before promoting the replica. If the script exits with a non-zero code, Patroni does not promote the replica and removes the leader key from DCS.

+.. _restapi_settings:
+
 REST API
 --------
 -  **restapi**:
@@ -0,0 +1,355 @@
+.. _citus:
+
+Citus support
+=============
+
+Patroni makes it extremely simple to deploy `Multi-Node Citus`__ clusters.
+
+__ https://docs.citusdata.com/en/stable/installation/multi_node.html
+
+TL;DR
+-----
+
+There are only a few simple rules you need to follow:
+
+1. `Citus <https://github.com/citusdata/citus>`__ database extension to
+   PostgreSQL must be available on all nodes.  Absolute minimum supported Citus
+   version is 10.0, but, to take all benefits from transparent switchovers and
+   restarts of workers we recommend using at least Citus 11.2.
+2. Cluster name (``scope``) must be the same for all Citus nodes!
+3. Superuser credentials must be the same on coordinator and all worker
+   nodes, and ``pg_hba.conf`` should allow superuser access between all nodes.
+4. :ref:`REST API <restapi_settings>` access should be allowed from worker
+   nodes to the coordinator. E.g., credentials should be the same and if
+   configured, client certificates from worker nodes must be accepted by the
+   coordinator.
+5. Add the following section to the ``patroni.yaml``:
+
+.. code:: YAML
+
+        citus:
+          group: X  # 0 for coordinator and 1, 2, 3, etc for workers
+          database: citus  # must be the same on all nodes
+
+
+After that you just need to start Patroni and it will handle the rest:
+
+1. ``citus`` extension will be automatically added to ``shared_preload_libraries``.
+2. If ``max_prepared_transactions`` isn't explicitly set in the global
+   :ref:`dynamic configuration <dynamic_configuration>` Patroni will
+   automatically set it to ``2*max_connections``.
+3. The ``citus.database`` will be automatically created followed by ``CREATE EXTENSION citus``.
+4. Current superuser :ref:`credentials <postgresql_settings>` will be added to the ``pg_dist_authinfo``
+   table to allow cross-node communication. Don't forget to update them if
+   later you decide to change superuser username/password/sslcert/sslkey!
+5. The coordinator primary node will automatically discover worker primary
+   nodes and add them to the ``pg_dist_node`` table using the
+   ``citus_add_node()`` function.
+6. Patroni will also maintain ``pg_dist_node`` in case failover/switchover
+   on the coordinator or worker clusters occurs.
+
+patronictl
+----------
+
+Coordinator and worker clusters are physically different PostgreSQL/Patroni
+clusters that are just logically groupped together using the
+`Citus <https://github.com/citusdata/citus>`__ database extension to
+PostgreSQL. Therefore in most cases it is not possible to manage them as a
+single entity.
+
+It results in two major differences in ``patronictl`` behaviour when
+``patroni.yaml`` has the ``citus`` section comparing with the usual:
+
+1. The ``list`` and the ``topology`` by default output all members of the Citus
+   formation (coordinators and workers). The new column ``Group`` indicates
+   which Citus group they belong to.
+2. For all ``patronictl`` commands the new option is introduced, named
+   ``--group``. For some commands the default value for the group might be
+   taken from the ``patroni.yaml``. For example, ``patronictl pause`` will
+   enable the maintenance mode by default for the ``group`` that is set in the
+   ``citus`` section, but for example for ``patronictl  switchover`` or
+   ``patronictl remove`` the group must be explicitly specified.
+
+An example of ``patronictl list`` output for the Citus cluster::
+
+    postgres@coord1:~$ patronictl list demo
+    + Citus cluster: demo ----------+--------------+---------+----+-----------+
+    | Group | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    |     0 | coord1  | 172.27.0.10 | Replica      | running |  1 |         0 |
+    |     0 | coord2  | 172.27.0.6  | Sync Standby | running |  1 |         0 |
+    |     0 | coord3  | 172.27.0.4  | Leader       | running |  1 |           |
+    |     1 | work1-1 | 172.27.0.8  | Sync Standby | running |  1 |         0 |
+    |     1 | work1-2 | 172.27.0.2  | Leader       | running |  1 |           |
+    |     2 | work2-1 | 172.27.0.5  | Sync Standby | running |  1 |         0 |
+    |     2 | work2-2 | 172.27.0.7  | Leader       | running |  1 |           |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+
+If we add the ``--group`` option, the output will change to::
+
+    postgres@coord1:~$ patronictl list demo --group 0
+    + Citus cluster: demo (group: 0, 7179854923829112860) -----------+
+    | Member | Host        | Role         | State   | TL | Lag in MB |
+    +--------+-------------+--------------+---------+----+-----------+
+    | coord1 | 172.27.0.10 | Replica      | running |  1 |         0 |
+    | coord2 | 172.27.0.6  | Sync Standby | running |  1 |         0 |
+    | coord3 | 172.27.0.4  | Leader       | running |  1 |           |
+    +--------+-------------+--------------+---------+----+-----------+
+
+    postgres@coord1:~$ patronictl list demo --group 1
+    + Citus cluster: demo (group: 1, 7179854923881963547) -----------+
+    | Member  | Host       | Role         | State   | TL | Lag in MB |
+    +---------+------------+--------------+---------+----+-----------+
+    | work1-1 | 172.27.0.8 | Sync Standby | running |  1 |         0 |
+    | work1-2 | 172.27.0.2 | Leader       | running |  1 |           |
+    +---------+------------+--------------+---------+----+-----------+
+
+Citus worker switchover
+-----------------------
+
+When a switchover is orchestrated for a Citus worker node, Citus offers the
+opportunity to make the switchover close to transparent for an application.
+Because the application connects to the coordinator, which in turn connects to
+the worker nodes, then it is possible with Citus to `pause` the SQL traffic on
+the coordinator for the shards hosted on a worker node. The switchover then
+happens while the traffic is kept on the coordinator, and resumes as soon as a
+new primary worker node is ready to accept read-write queries.
+
+An example of ``patronictl switchover`` on the worker cluster::
+
+    postgres@coord1:~$ patronictl switchover demo
+    + Citus cluster: demo ----------+--------------+---------+----+-----------+
+    | Group | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    |     0 | coord1  | 172.27.0.10 | Replica      | running |  1 |         0 |
+    |     0 | coord2  | 172.27.0.6  | Sync Standby | running |  1 |         0 |
+    |     0 | coord3  | 172.27.0.4  | Leader       | running |  1 |           |
+    |     1 | work1-1 | 172.27.0.8  | Leader       | running |  1 |           |
+    |     1 | work1-2 | 172.27.0.2  | Sync Standby | running |  1 |         0 |
+    |     2 | work2-1 | 172.27.0.5  | Sync Standby | running |  1 |         0 |
+    |     2 | work2-2 | 172.27.0.7  | Leader       | running |  1 |           |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    Citus group: 2
+    Primary [work2-2]:
+    Candidate ['work2-1'] []:
+    When should the switchover take place (e.g. 2022-12-22T08:02 )  [now]:
+    Current cluster topology
+    + Citus cluster: demo (group: 2, 7179854924063375386) -----------+
+    | Member  | Host       | Role         | State   | TL | Lag in MB |
+    +---------+------------+--------------+---------+----+-----------+
+    | work2-1 | 172.27.0.5 | Sync Standby | running |  1 |         0 |
+    | work2-2 | 172.27.0.7 | Leader       | running |  1 |           |
+    +---------+------------+--------------+---------+----+-----------+
+    Are you sure you want to switchover cluster demo, demoting current primary work2-2? [y/N]: y
+    2022-12-22 07:02:40.33003 Successfully switched over to "work2-1"
+    + Citus cluster: demo (group: 2, 7179854924063375386) ------+
+    | Member  | Host       | Role    | State   | TL | Lag in MB |
+    +---------+------------+---------+---------+----+-----------+
+    | work2-1 | 172.27.0.5 | Leader  | running |  1 |           |
+    | work2-2 | 172.27.0.7 | Replica | stopped |    |   unknown |
+    +---------+------------+---------+---------+----+-----------+
+
+    postgres@coord1:~$ patronictl list demo
+    + Citus cluster: demo ----------+--------------+---------+----+-----------+
+    | Group | Member  | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+    |     0 | coord1  | 172.27.0.10 | Replica      | running |  1 |         0 |
+    |     0 | coord2  | 172.27.0.6  | Sync Standby | running |  1 |         0 |
+    |     0 | coord3  | 172.27.0.4  | Leader       | running |  1 |           |
+    |     1 | work1-1 | 172.27.0.8  | Leader       | running |  1 |           |
+    |     1 | work1-2 | 172.27.0.2  | Sync Standby | running |  1 |         0 |
+    |     2 | work2-1 | 172.27.0.5  | Leader       | running |  2 |           |
+    |     2 | work2-2 | 172.27.0.7  | Sync Standby | running |  2 |         0 |
+    +-------+---------+-------------+--------------+---------+----+-----------+
+
+And this is how it looks on the coordinator side::
+
+    # The worker primary notifies the coordinator that it is going to execute "pg_ctl stop".
+    2022-12-22 07:02:38,636 DEBUG: query("BEGIN")
+    2022-12-22 07:02:38,636 DEBUG: query("SELECT pg_catalog.citus_update_node(3, '172.27.0.7-demoted', 5432, true, 10000)")
+    # From this moment all application traffic on the coordinator to the worker group 2 is paused.
+
+    # The future worker primary notifies the coordinator that it acquired the leader lock in DCS and about to run "pg_ctl promote".
+    2022-12-22 07:02:40,085 DEBUG: query("SELECT pg_catalog.citus_update_node(3, '172.27.0.5', 5432)")
+
+    # The new worker primary just finished promote and notifies coordinator that it is ready to accept read-write traffic.
+    2022-12-22 07:02:41,485 DEBUG: query("COMMIT")
+    # From this moment the application traffic on the coordinator to the worker group 2 is unblocked.
+
+Peek into DCS
+-------------
+
+The Citus cluster (coordinator and workers) are stored in DCS as a fleet of
+Patroni clusters logically grouped together::
+
+    /service/batman/              # scope=batman
+    /service/batman/0/            # citus.group=0, coordinator
+    /service/batman/0/initialize
+    /service/batman/0/leader
+    /service/batman/0/members/
+    /service/batman/0/members/m1
+    /service/batman/0/members/m2
+    /service/batman/1/            # citus.group=1, worker
+    /service/batman/1/initialize
+    /service/batman/1/leader
+    /service/batman/1/members/
+    /service/batman/1/members/m3
+    /service/batman/1/members/m4
+    ...
+
+Such an approach was chosen because for most DCS it becomes possible to fetch
+the entire Citus cluster with a single recursive read request. Only Citus
+coordinator nodes are reading the whole tree, because they have to discover
+worker nodes. Worker nodes are reading only the subtree for their own group and
+in some cases they could read the subtree of the coordinator group.
+
+Citus on Kubernetes
+-------------------
+
+Since Kubernetes doesn't support hierarchical structures we had to include the
+citus group to all K8s objects Patroni creates::
+
+    batman-0-leader  # the leader config map for the coordinator
+    batman-0-config  # the config map holding initialize, config, and history "keys"
+    ...
+    batman-1-leader  # the leader config map for worker group 1
+    batman-1-config
+    ...
+
+I.e., the naming pattern is: ``${scope}-${citus.group}-${type}``.
+
+All Kubernetes objects are discovered by Patroni using the `label selector`__,
+therefore all Pods with Patroni&Citus and Endpoints/ConfigMaps must have
+similar labels, and Patroni must be configured to use them using Kubernetes
+:ref:`settings <kubernetes_settings>` or :ref:`environment variables
+<kubernetes_environment>`.
+
+__ https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
+
+A couple of examples of Patroni configuration using Pods environment variables:
+
+1. for the coordinator cluster
+
+.. code:: YAML
+
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          labels:
+            application: patroni
+            citus-group: "0"
+            citus-type: coordinator
+            cluster-name: citusdemo
+          name: citusdemo-0-0
+          namespace: default
+        spec:
+          containers:
+          - env:
+            - name: PATRONI_SCOPE
+              value: citusdemo
+            - name: PATRONI_NAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.name
+            - name: PATRONI_KUBERNETES_POD_IP
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: status.podIP
+            - name: PATRONI_KUBERNETES_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.namespace
+            - name: PATRONI_KUBERNETES_LABELS
+              value: '{application: patroni}'
+            - name: PATRONI_CITUS_DATABASE
+              value: citus
+            - name: PATRONI_CITUS_GROUP
+              value: "0"
+
+2. for the worker cluster from the group 2
+
+.. code:: YAML
+
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          labels:
+            application: patroni
+            citus-group: "2"
+            citus-type: worker
+            cluster-name: citusdemo
+          name: citusdemo-2-0
+          namespace: default
+        spec:
+          containers:
+          - env:
+            - name: PATRONI_SCOPE
+              value: citusdemo
+            - name: PATRONI_NAME
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.name
+            - name: PATRONI_KUBERNETES_POD_IP
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: status.podIP
+            - name: PATRONI_KUBERNETES_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  apiVersion: v1
+                  fieldPath: metadata.namespace
+            - name: PATRONI_KUBERNETES_LABELS
+              value: '{application: patroni}'
+            - name: PATRONI_CITUS_DATABASE
+              value: citus
+            - name: PATRONI_CITUS_GROUP
+              value: "2"
+
+As you may noticed, both examples have ``citus-group`` label set. This label
+allows Patroni to identify object as belonging to a certain Citus group. In
+addition to that, there is also ``PATRONI_CITUS_GROUP`` environment variable,
+which has the same value as the ``citus-group`` label. When Patroni creates
+new Kubernetes objects ConfigMaps or Endpoints, it automatically puts the
+``citus-group: ${env.PATRONI_CITUS_GROUP}`` label on them:
+
+.. code:: YAML
+
+        apiVersion: v1
+        kind: ConfigMap
+        metadata:
+          name: citusdemo-0-leader  # Is generated as ${env.PATRONI_SCOPE}-${env.PATRONI_CITUS_GROUP}-leader
+          labels:
+            application: patroni    # Is set from the ${env.PATRONI_KUBERNETES_LABELS}
+            cluster-name: citusdemo # Is automatically set from the ${env.PATRONI_SCOPE}
+            citus-group: '0'        # Is automatically set from the ${env.PATRONI_CITUS_GROUP}
+
+You can find a complete example of Patroni deployment on Kubernetes with Citus
+support in the `kubernetes`__ folder of the Patroni repository.
+
+__ https://github.com/zalando/patroni/tree/master/kubernetes
+
+There are two important files for you:
+
+1. Dockerfile.citus
+2. citus_k8s.yaml
+
+Citus upgrades and PostgreSQL major upgrades
+--------------------------------------------
+
+First, please read about upgrading Citus version in the `documentation`__.
+There is one minor change in the process. When executing upgrade, you have to
+use ``patronictl restart`` instead of ``systemctl restart`` to restart
+PostgreSQL.
+
+__ https://docs.citusdata.com/en/latest/admin_guide/upgrading_citus.html
+
+The PostgreSQL major upgrade with Citus is a bit more complex. You will have to
+combine techniques used in the Citus documentation about major upgrades and
+Patroni documentation about :ref:`PostgreSQL major upgrade<major_upgrade>`.
+Please keep in mind that Citus cluster consists of many Patroni clusters
+(coordinator and workers) and they all have to be upgraded independently.
@@ -0,0 +1,63 @@
+.. _dcs_failsafe_mode:
+
+DCS Failsafe Mode
+=================
+
+The problem
+-----------
+
+Patroni is heavily relying on Distributed Configuration Store (DCS) to solve the task of leader elections and detect network partitioning. That is, the node is allowed to run Postgres as the primary only if it can update the leader lock in DCS. In case the update of the leader lock fails, Postgres is immediately demoted and started as read-only. Depending on which DCS is used, the chances of hitting the "problem" differ. For example, with Etcd which is only used for Patroni, chances are close to zero, while with K8s API (backed by Etcd) it could be observed more frequently.
+
+
+Reasons for the current implementation
+---------------------------------------
+
+The leader lock update failure could be caused by two main reasons:
+
+1. Network partitioning
+2. DCS being down
+
+In general, it is impossible to distinguish between these two from a single node, and therefore Patroni assumes the worst case - network partitioning. In the case of a partitioned network, other nodes of the Patroni cluster may successfully grab the leader lock and promote Postgres to primary. In order to avoid a split-brain, the old primary is demoted before the leader lock expires.
+
+
+DCS Failsafe Mode
+-----------------
+
+We introduce a new special option, the ``failsafe_mode``. It could be enabled only via global configuration stored in the DCS ``/config`` key. If the failsafe mode is enabled and the leader lock update in DCS failed due to reasons different from the version/value/index mismatch, Postgres may continue to run as a primary if it can access all known members of the cluster via Patroni REST API.
+
+
+Low-level implementation details
+--------------------------------
+
+- We introduce a new, permanent key in DCS, named ``/failsafe``.
+- The ``/failsafe`` key contains all known members of the given Patroni cluster at a given time.
+- The current leader maintains the ``/failsafe`` key.
+- The member is allowed to participate in the leader race and become the new leader only if it is present in the ``/failsafe`` key.
+- If the cluster consists of a single node the ``/failsafe`` key will contain a single member.
+- In the case of DCS "outage" the existing primary connects to all members presented in the ``/failsafe`` key via the ``POST /failsafe`` REST API and may continue to run as the primary if all replicas acknowledge it.
+- If one of the members doesn't respond, the primary is demoted.
+- Replicas are using incoming ``POST /failsafe`` REST API requests as an indicator that the primary is still alive. This information is cached for ``ttl`` seconds.
+
+
+F.A.Q.
+------
+
+- Why MUST the current primary see ALL other members? Can’t we rely on quorum here?
+
+  This is a great question! The problem is that the view on the quorum might be different from the perspective of DCS and Patroni. While DCS nodes must be evenly distributed across availability zones, there is no such rule for Patroni, and more importantly, there is no mechanism for introducing and enforcing such a rule. If the majority of Patroni nodes ends up in the losing part of the partitioned network (including primary) while minority nodes are in the winning part, the primary must be demoted. Only checking ALL other members allows detecting such a situation.
+
+- What if node/pod gets terminated while DCS is down?
+
+  If DCS isn’t accessible, the check “are ALL other cluster members accessible?” is executed every cycle of the heartbeat loop (every ``loop_wait`` seconds). If pod/node is terminated, the check will fail and Postgres will be demoted to a read-only and will not recover until DCS is restored.
+
+- What if all members of the Patroni cluster are lost while DCS is down?
+
+  Patroni could be configured to create the new replica from the backup even when the cluster doesn't have a leader. But, if the new member isn't present in the ``/failsafe`` key, it will not be able to grab the leader lock and promote.
+  
+- What will happen if the primary lost access to DCS while replicas didn't?
+
+  The primary will execute the failsafe code and contact all known replicas. These replicas will use this information as an indicator that the primary is alive and will not start the leader race even if the leader lock in DCS has expired.
+
+- How to enable the Failsafe Mode?
+
+  Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or ``patronictl edit-config -s failsafe_mode=true``
@@ -23,6 +23,8 @@ A Patroni cluster can be started with a data directory from a single-node Postgr
 3. Start Patroni (e.g. ``patroni /etc/patroni/patroni.yml``). It automatically detects that PostgreSQL daemon is already running but its configuration might be out-of-date.
 4. Ask Patroni to restart the node with ``patronictl restart cluster-name node-name``. This step is only required if PostgreSQL configuration is out-of-date.

+.. _major_upgrade:
+
 Major Upgrade of PostgreSQL Version
 ===================================

@@ -12,6 +12,8 @@ We call Patroni a "template" because it is far from being a one-size-fits-all or

 Currently supported PostgreSQL versions: 9.3 to 15.

+**Note to Citus users**: Starting from 3.0 Patroni nicely integrates with the `Citus <https://github.com/citusdata/citus>`__ database extension to Postgres. Please check the :ref:`Citus support page <citus>` in the Patroni documentation for more info about how to use Patroni high availability together with a Citus distributed cluster.
+
 **Note to Kubernetes users**: Patroni can run natively on top of Kubernetes. Take a look at the :ref:`Kubernetes <kubernetes>` chapter of the Patroni documentation.


@@ -20,7 +22,9 @@ Currently supported PostgreSQL versions: 9.3 to 15.
   :caption: Contents:

   README
+   citus
   dynamic_configuration
+   dcs_failsafe_mode
   rest_api
   existing_data
   ENVIRONMENT
@@ -23,10 +23,7 @@ Use ConfigMaps
 In this mode, Patroni will create ConfigMaps instead of Endpoints and store keys inside meta-data of those ConfigMaps.
 Changing the leader takes at least two updates, one to the leader ConfigMap and another to the respective Endpoint.

-There are two ways to direct the traffic to the Postgres leader:
-
- use the `callback script <https://github.com/zalando/patroni/blob/master/kubernetes/callback.py>`_ provided by Patroni
- configure the Kubernetes Postgres service to use the label selector with the `role_label` (configured in patroni configuration).
+To direct the traffic to the Postgres leader you need to configure the Kubernetes Postgres service to use the label selector with the `role_label` (configured in patroni configuration).

 Note that in some cases, for instance, when running on OpenShift, there is no alternative to using ConfigMaps.

@@ -39,7 +36,7 @@ Examples
 --------

 - The `kubernetes <https://github.com/zalando/patroni/tree/master/kubernetes>`__ folder of the Patroni repository contains
-  examples of the Docker image, the Kubernetes manifest and the callback script in order to test Patroni Kubernetes setup.
+  examples of the Docker image, and the Kubernetes manifest to test Patroni Kubernetes setup.
  Note that in the current state it will not be able to use PersistentVolumes because of permission issues.

 - You can find the full-featured Docker image that can use Persistent Volumes in the
@@ -3,6 +3,76 @@
 Release notes
 =============

+Version 3.0.1
+-------------
+
+**Bugfixes**
+
+- Pass proper role name to an ``on_role_change`` callback script'. (Alexander Kukushkin, Polina Bungina)
+
+  Patroni used to erroneously pass ``promoted`` role to an ``on_role_change`` callback script on promotion. The passed role name changed back to ``master``. This regression was introduced in 3.0.0.
+
+
+Version 3.0.0
+-------------
+
+This version adds integration with `Citus <https://www.citusdata.com>`__ and makes it possible to survive temporary DCS outages without demoting primary.
+
+.. warning::
+   - Version 3.0.0 is the last release supporting Python 2.7. Upcoming release will drop support of Python versions older than 3.7.
+
+   - The RAFT support is deprecated. We will do our best to maintain it, but take neither guarantee nor responsibility for possible issues.
+
+   - This version is the first step in getting rid of the "master", in favor of "primary". Upgrading to the next major release will work reliably only if you run at least 3.0.0.
+
+
+**New features**
+
+- DCS failsafe mode (Alexander Kukushkin, Polina Bungina)
+
+  If the feature is enabled it will allow Patroni cluster to survive temporary DCS outages. You can find more details in the :ref:`documentation <dcs_failsafe_mode>`.
+
+- Citus support (Alexander, Polina, Jelte Fennema)
+
+  Patroni enables easy deployment and management of `Citus <https://www.citusdata.com>`__ clusters with HA. Please check :ref:`here <citus>` page for more information.
+
+
+**Improvements**
+
+- Suppress recurring errors when dropping unknown but active replication slots (Michael Banck)
+
+  Patroni will still write these logs, but only in DEBUG.
+
+- Run only one monitoring query per HA loop (Alexander)
+
+  It wasn't the case if synchronous replication is enabled.
+
+- Keep only latest failed data directory (William Albertus Dembo)
+
+  If bootstrap failed Patroni used to rename $PGDATA folder with timestamp suffix. From now on the suffix will be ``.failed`` and if such folder exists it is removed before renaming.
+
+- Improved check of synchronous replication connections (Alexander)
+
+  When the new host is added to the ``synchronous_standby_names`` it will be set as synchronous in DCS only when it managed to catch up with the primary in addition to ``pg_stat_replication.sync_state = 'sync'``.
+
+
+**Removed functionality**
+
+- Remove ``patronictl scaffold`` (Alexander)
+
+  The only reason for having it was a hacky way of running standby clusters.
+
+
+Version 2.1.7
+-------------
+
+**Bugfixes**
+
+- Fixed little incompatibilities with legacy python modules (Alexander Kukushkin)
+
+  They prevented from building/running Patroni on Debian buster/Ubuntu bionic.
+
+
 Version 2.1.6
 -------------

@@ -1261,7 +1331,7 @@ Version 1.6.1

 - Kill all children along with the callback process before starting the new one (Alexander Kukushkin)

-  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time. 
+  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time.

 - Fix 'start failed' issue (Alexander Kukushkin)

@@ -77,7 +77,7 @@ scripts to clone a new replica. Those are configured in the ``postgresql`` confi
            command: <command name>
            keep_data: True
            no_params: True
-            no_master: 1
+            no_leader: 1

 example: wal_e

@@ -89,7 +89,7 @@ example: wal_e
            - basebackup
        wal_e:
            command: patroni_wale_restore
-            no_master: 1
+            no_leader: 1
            envdir: {{WALE_ENV_DIR}}
            use_iam: 1
        basebackup:
@@ -126,7 +126,7 @@ to execute and any custom parameters that should be passed to that command. All
    Connection string to connect to the cluster member to clone from (primary or other replica). The user in the
    connection string can execute SQL and replication protocol commands.

-A special ``no_master`` parameter, if defined, allows Patroni to call the replica creation method even if there is no
+A special ``no_leader`` parameter, if defined, allows Patroni to call the replica creation method even if there is no
 running leader or replicas. In that case, an empty string will be passed in a connection string. This is useful for
 restoring the formerly running cluster from the binary backup.

@@ -12,7 +12,6 @@ For all health check ``GET`` requests Patroni returns a JSON document with the s
 - The following requests to Patroni REST API will return HTTP status code **200** only when the Patroni node is running as the primary with leader lock:

  - ``GET /``
-  - ``GET /master``
  - ``GET /primary``
  - ``GET /read-write``

@@ -33,7 +32,6 @@ For all health check ``GET`` requests Patroni returns a JSON document with the s

 In the following requests, since we are checking for the leader or standby-leader status, Patroni doesn't apply any of the user defined tags and they will be ignored.
  - ``GET /?tag_key1=value1&tag_key2=value2``
-  - ``GET /master?tag_key1=value1&tag_key2=value2``
  - ``GET /leader?tag_key1=value1&tag_key2=value2``
  - ``GET /primary?tag_key1=value1&tag_key2=value2``
  - ``GET /read-write?tag_key1=value1&tag_key2=value2``
@@ -368,7 +366,7 @@ Restart endpoint
  - **restart_pending**: boolean, if set to ``true`` Patroni will restart PostgreSQL only when restart is pending in order to apply some changes in the PostgreSQL config.
  - **role**: perform restart only if the current role of the node matches with the role from the POST request.
  - **postgres_version**: perform restart only if the current version of postgres is smaller than specified in the POST request.
-  - **timeout**: how long we should wait before PostgreSQL starts accepting connections. Overrides ``master_start_timeout``.
+  - **timeout**: how long we should wait before PostgreSQL starts accepting connections. Overrides ``primary_start_timeout``.
  - **schedule**: timestamp with time zone, schedule the restart somewhere in the future.

 - ``DELETE /restart``: delete the scheduled restart
@@ -9,11 +9,11 @@ A Patroni cluster has two interfaces to be protected from unauthorized access: t
 Protecting DCS
 ==============

-Patroni and patronictl both store and retrieve data to/from the DCS. 
+Patroni and patronictl both store and retrieve data to/from the DCS.

 Despite DCS doesn't contain any sensitive information, it allows changing some of Patroni/Postgres configuration. Therefore the very first thing that should be protected is DCS itself.

-The details of protection depend on the type of DCS used. The authentication and encryption parameters (tokens/basic-auth/client certificates) for the supported types of DCS are covered in :ref:`SETTINGS <bootstrap_settings>` 
+The details of protection depend on the type of DCS used. The authentication and encryption parameters (tokens/basic-auth/client certificates) for the supported types of DCS are covered in :ref:`SETTINGS <bootstrap_settings>`

 The general recommendation is to enable TLS for all DCS communication.

@@ -22,7 +22,7 @@ Protecting the REST API

 Protecting the REST API is a more complicated task.

-The Patroni REST API is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. 
+The Patroni REST API is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.

 From the point of view of security, REST API contains safe (``GET`` requests, only retrieve information) and unsafe (``PUT``, ``POST``, ``PATCH`` and ``DELETE`` requests, change the state of nodes) endpoints.

@@ -9,5 +9,5 @@ check_cmd = "/usr/sbin/haproxy -c -f {{ .src }}"
 reload_cmd = "haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf $(cat /var/run/haproxy.pid)"

 keys = [
-    "/members/",
+    "/",
 ]
@@ -0,0 +1,32 @@
+global
+	maxconn 100
+
+defaults
+	log	global
+	mode	tcp
+	retries 2
+	timeout client 30m
+	timeout connect 4s
+	timeout server 30m
+	timeout check 5s
+
+listen stats
+	mode http
+	bind *:7000
+	stats enable
+	stats uri /
+
+listen coordinator
+	bind *:5000
+	option httpchk HEAD /primary
+	http-check expect status 200
+	default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
+{{range gets "/0/members/*"}}	server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check check-ssl port {{index (split (index (split $data.api_url "/") 2) ":") 1}} verify required ca-file /etc/ssl/certs/ssl-cert-snakeoil.pem crt /etc/ssl/private/ssl-cert-snakeoil.crt
+{{end}}
+listen workers
+	bind *:5001
+	option httpchk HEAD /primary
+	http-check expect status 200
+	default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
+{{range gets "/*/members/*"}}{{$group := index (split .Key "/") 1}}{{if ne $group "0"}}	server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check check-ssl port {{index (split (index (split $data.api_url "/") 2) ":") 1}} verify required ca-file /etc/ssl/certs/ssl-cert-snakeoil.pem crt /etc/ssl/private/ssl-cert-snakeoil.crt
+{{end}}{{end}}
@@ -16,9 +16,9 @@ listen stats
 	stats enable
 	stats uri /

-listen master
+listen primary
 	bind *:5000
-	option httpchk HEAD /master
+	option httpchk HEAD /primary
 	http-check expect status 200
 	default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
 {{range gets "/members/*"}}	server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
@@ -1,6 +1,6 @@
 # startup scripts for Patroni

-This directory contains sample startup scripts for various OSes 
+This directory contains sample startup scripts for various OSes
 and management tools for Patroni.

 Scripts supplied:
@@ -10,7 +10,7 @@ Scripts supplied:
 Upstart job for Ubuntu 12.04 or 14.04.  Requires Upstart > 1.4. Intended for systems where Patroni has been installed on a base system, rather than in Docker.

 ### patroni.service
-Systemd service file, to be copied to /etc/systemd/system/patroni.service, tested on Centos 7.1 with Patroni installed from pip. 
+Systemd service file, to be copied to /etc/systemd/system/patroni.service, tested on Centos 7.1 with Patroni installed from pip.

 ### patroni
 Init.d service file for Debian-like distributions. Copy it to /etc/init.d/, make executable:
@@ -21,12 +21,9 @@ Feature: basic replication
    And I shut down postgres1
    Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds
    When I start postgres1
-    And "members/postgres1" key in DCS has state=running after 10 seconds
-    And I sleep for 2 seconds
-    When I issue a GET request to http://127.0.0.1:8010/sync
-    Then I receive a response code 200
-    When I issue a GET request to http://127.0.0.1:8009/async
-    Then I receive a response code 200
+    Then "members/postgres1" key in DCS has state=running after 10 seconds
+    And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
+    And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds

  Scenario: check stuck sync replica
    Given I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": true, "maximum_lag_on_syncnode": 15000000, "postgresql": {"parameters": {"synchronous_commit": "remote_apply"}}}
@@ -38,11 +35,8 @@ Feature: basic replication
    And I load data on postgres0
    Then "sync" key in DCS has sync_standby=postgres1 after 15 seconds
    And I resume wal replay on postgres2
-    And I sleep for 2 seconds
-    And I issue a GET request to http://127.0.0.1:8009/sync
-    Then I receive a response code 200
-    When I issue a GET request to http://127.0.0.1:8010/async
-    Then I receive a response code 200
+    And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds
+    And Status code on GET http://127.0.0.1:8010/async is 200 after 3 seconds
    When I issue a PATCH request to http://127.0.0.1:8008/config with {"pause": null, "maximum_lag_on_syncnode": -1, "postgresql": {"parameters": {"synchronous_commit": "on"}}}
    Then I receive a response code 200
    And I drop table on postgres0
@@ -50,23 +44,17 @@ Feature: basic replication
  Scenario: check multi sync replication
    Given I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 2}
    Then I receive a response code 200
-    And I sleep for 10 seconds
-    Then "sync" key in DCS has sync_standby=postgres1,postgres2 after 5 seconds
-    When I issue a GET request to http://127.0.0.1:8010/sync
-    Then I receive a response code 200
-    When I issue a GET request to http://127.0.0.1:8009/sync
-    Then I receive a response code 200
+    Then "sync" key in DCS has sync_standby=postgres1,postgres2 after 10 seconds
+    And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
+    And Status code on GET http://127.0.0.1:8009/sync is 200 after 3 seconds
    When I issue a PATCH request to http://127.0.0.1:8008/config with {"synchronous_node_count": 1}
    Then I receive a response code 200
    And I shut down postgres1
-    And I sleep for 10 seconds
    Then "sync" key in DCS has sync_standby=postgres2 after 10 seconds
    When I start postgres1
-    And "members/postgres1" key in DCS has state=running after 10 seconds
-    When I issue a GET request to http://127.0.0.1:8010/sync
-    Then I receive a response code 200
-    When I issue a GET request to http://127.0.0.1:8009/async
-    Then I receive a response code 200
+    Then "members/postgres1" key in DCS has state=running after 10 seconds
+    And Status code on GET http://127.0.0.1:8010/sync is 200 after 3 seconds
+    And Status code on GET http://127.0.0.1:8009/async is 200 after 3 seconds

  Scenario: check the basic failover in synchronous mode
    Given I run patronictl.py pause batman
@@ -77,6 +65,7 @@ Feature: basic replication
    Then I receive a response returncode 0
    And postgres2 role is the primary after 24 seconds
    And Response on GET http://127.0.0.1:8010/history contains recovery after 10 seconds
+    And there is a postgres2_cb.log with "on_role_change master batman" in postgres2 data directory
    When I issue a PATCH request to http://127.0.0.1:8010/config with {"synchronous_mode": null, "master_start_timeout": 0}
    Then I receive a response code 200
    When I add the table bar to postgres2
@@ -88,7 +77,7 @@ Feature: basic replication
    Then postgres1 is a leader after 10 seconds
    And postgres1 role is the primary after 10 seconds

-  Scenario: check rejoin of the former master with pg_rewind
+  Scenario: check rejoin of the former primary with pg_rewind
    Given I add the table splitbrain to postgres0
    And I start postgres0
    Then postgres0 role is the secondary after 20 seconds
@@ -0,0 +1,72 @@
+Feature: citus
+  We should check that coordinator discovers and registers workers and clients don't have errors when worker cluster switches over
+
+  Scenario: check that worker cluster is registered in the coordinator
+    Given I start postgres0 in citus group 0
+    And I start postgres2 in citus group 1
+    Then postgres0 is a leader in a group 0 after 10 seconds
+    And postgres2 is a leader in a group 1 after 10 seconds
+    When I start postgres1 in citus group 0
+    And I start postgres3 in citus group 1
+    Then replication works from postgres0 to postgres1 after 15 seconds
+    Then replication works from postgres2 to postgres3 after 15 seconds
+    And postgres0 is registered in the coordinator postgres0 as the worker in group 0
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+
+  Scenario: coordinator failover updates pg_dist_node
+    Given I run patronictl.py failover batman --group 0 --candidate postgres1 --force
+    Then postgres1 role is the primary after 10 seconds
+    And replication works from postgres1 to postgres0 after 15 seconds
+    And "sync" key in a group 0 in DCS has sync_standby=postgres0 after 15 seconds
+    And postgres1 is registered in the coordinator postgres1 as the worker in group 0
+    When I run patronictl.py failover batman --group 0 --candidate postgres0 --force
+    Then postgres0 role is the primary after 10 seconds
+    And replication works from postgres0 to postgres1 after 15 seconds
+    And "sync" key in a group 0 in DCS has sync_standby=postgres1 after 15 seconds
+    And postgres0 is registered in the coordinator postgres0 as the worker in group 0
+
+  Scenario: worker switchover doesn't break client queries on the coordinator
+    Given I create a distributed table on postgres0
+    And I start a thread inserting data on postgres0
+    When I run patronictl.py switchover batman --group 1 --force
+    Then I receive a response returncode 0
+    And postgres3 role is the primary after 10 seconds
+    And replication works from postgres3 to postgres2 after 15 seconds
+    And "sync" key in a group 1 in DCS has sync_standby=postgres2 after 15 seconds
+    And postgres3 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I run patronictl.py switchover batman --group 1 --force
+    Then I receive a response returncode 0
+    And postgres2 role is the primary after 10 seconds
+    And replication works from postgres2 to postgres3 after 15 seconds
+    And "sync" key in a group 1 in DCS has sync_standby=postgres3 after 15 seconds
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I stop a thread
+    Then a distributed table on postgres0 has expected rows
+
+  Scenario: worker primary restart doesn't break client queries on the coordinator
+    Given I cleanup a distributed table on postgres0
+    And I start a thread inserting data on postgres0
+    When I run patronictl.py restart batman postgres2 --group 1 --force
+    Then I receive a response returncode 0
+    And postgres2 role is the primary after 10 seconds
+    And replication works from postgres2 to postgres3 after 15 seconds
+    And postgres2 is registered in the coordinator postgres0 as the worker in group 1
+    And a thread is still alive
+    When I stop a thread
+    Then a distributed table on postgres0 has expected rows
+
+  Scenario: check that in-flight transaction is rolled back after timeout when other workers need to change pg_dist_node
+    Given I start postgres4 in citus group 2
+    Then postgres4 is a leader in a group 2 after 10 seconds
+    And "members/postgres4" key in a group 2 in DCS has role=master after 3 seconds
+    When I run patronictl.py edit-config batman --group 2 -s ttl=20 --force
+    Then I receive a response returncode 0
+    And I receive a response output "+ttl: 20"
+    When I sleep for 2 seconds
+    Then postgres4 is registered in the coordinator postgres0 as the worker in group 2
+    When I shut down postgres4
+    Then There is a transaction in progress on postgres0 changing pg_dist_node
+    When I run patronictl.py restart batman postgres2 --group 1 --force
+    Then a transaction finishes in 20 seconds
@@ -0,0 +1,85 @@
+Feature: dcs failsafe mode
+  We should check the basic dcs failsafe mode functioning
+
+  Scenario: check failsafe mode can be successfully enabled
+    Given I start postgres0
+    And postgres0 is a leader after 10 seconds
+    And I sleep for 3 seconds
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 5, "failsafe_mode": true}
+    Then I receive a response code 200
+    And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds
+    When I issue a GET request to http://127.0.0.1:8008/failsafe
+    Then I receive a response code 200
+    And I receive a response postgres0 http://127.0.0.1:8008/patroni
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"postgresql": {"parameters": {"wal_level": "logical"}}}
+    Then I receive a response code 200
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"dcs_slot_0": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
+    Then I receive a response code 200
+
+  @dcs-failsafe
+  Scenario: check one-node cluster is functioning while DCS is down
+    Given DCS is down
+    Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
+    And postgres0 role is the primary after 10 seconds
+
+  @dcs-failsafe
+  Scenario: check new replica isn't promoted when leader is down and DCS is up
+    Given DCS is up
+    When I do a backup of postgres0
+    And I shut down postgres0
+    When I start postgres1 in a cluster batman from backup with no_leader
+    And I sleep for 2 seconds
+    Then postgres1 role is the replica after 12 seconds
+
+  Scenario: check leader and replica are both in /failsafe key after leader is back
+    Given I start postgres0
+    And I start postgres1
+    Then "members/postgres0" key in DCS has state=running after 10 seconds
+    And "members/postgres1" key in DCS has state=running after 2 seconds
+    And Response on GET http://127.0.0.1:8009/failsafe contains postgres1 after 10 seconds
+    When I issue a GET request to http://127.0.0.1:8009/failsafe
+    Then I receive a response code 200
+    And I receive a response postgres0 http://127.0.0.1:8008/patroni
+    And I receive a response postgres1 http://127.0.0.1:8009/patroni
+
+  @dcs-failsafe
+  @slot-advance
+  Scenario: check leader and replica are functioning while DCS is down
+    Given logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 10 seconds
+    And DCS is down
+    Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
+    Then postgres0 role is the primary after 10 seconds
+    And postgres1 role is the replica after 2 seconds
+    And replication works from postgres0 to postgres1 after 10 seconds
+    And I get all changes from logical slot dcs_slot_0 on postgres0
+    And logical slot dcs_slot_0 is in sync between postgres0 and postgres1 after 20 seconds
+
+  @dcs-failsafe
+  Scenario: check primary is demoted when one replica is shut down and DCS is down
+    Given DCS is down
+    And I kill postgres1
+    And I kill postmaster on postgres1
+    And I sleep for 2 seconds
+    Then postgres0 role is the replica after 12 seconds
+
+  @dcs-failsafe
+  Scenario: check known replica is promoted when leader is down and DCS is up
+    Given I shut down postgres0
+    And DCS is up
+    When I start postgres1
+    Then "members/postgres1" key in DCS has state=running after 10 seconds
+    And postgres1 role is the primary after 25 seconds
+
+  @dcs-failsafe
+  Scenario: check three-node cluster is functioning while DCS is down
+    Given I start postgres0
+    And I start postgres2
+    Then "members/postgres2" key in DCS has state=running after 10 seconds
+    And "members/postgres0" key in DCS has state=running after 20 seconds
+    And Response on GET http://127.0.0.1:8008/failsafe contains postgres2 after 10 seconds
+    And replication works from postgres1 to postgres0 after 10 seconds
+    Given DCS is down
+    Then Response on GET http://127.0.0.1:8008/primary contains failsafe_mode_is_active after 12 seconds
+    Then postgres1 role is the primary after 10 seconds
+    And postgres0 role is the replica after 2 seconds
+    And postgres2 role is the replica after 2 seconds
@@ -1,7 +1,9 @@
 import abc
 import datetime
+import glob
 import os
 import json
+import psutil
 import re
 import shutil
 import signal
@@ -101,6 +103,7 @@ class PatroniController(AbstractController):
            self.watchdog = None

        self._scope = (custom_config or {}).get('scope', 'batman')
+        self._citus_group = (custom_config or {}).get('citus', {}).get('group')
        self._config = self._make_patroni_test_config(name, custom_config)
        self._closables = []

@@ -142,7 +145,7 @@ class PatroniController(AbstractController):
            self.watchdog.start()
        env = os.environ.copy()
        if isinstance(self._context.dcs_ctl, KubernetesController):
-            self._context.dcs_ctl.create_pod(self._name[8:], self._scope)
+            self._context.dcs_ctl.create_pod(self._name[8:], self._scope, self._citus_group)
            env['PATRONI_KUBERNETES_POD_IP'] = '10.0.0.' + self._name[-1]
        if os.name == 'nt':
            env['BEHAVE_DEBUG'] = 'true'
@@ -183,7 +186,9 @@ class PatroniController(AbstractController):
            config.pop('etcd', None)

        raft_port = os.environ.get('RAFT_PORT')
-        if raft_port:
+        # If patroni_raft_controller is suspended two Patroni members is enough to get a quorum,
+        # therefore we don't want Patroni to join as a voting member when testing dcs_failsafe_mode.
+        if raft_port and not self._output_dir.endswith('dcs_failsafe_mode'):
            os.environ['RAFT_PORT'] = str(int(raft_port) + 1)
            config['raft'] = {'data_dir': self._output_dir, 'self_addr': 'localhost:' + os.environ['RAFT_PORT']}

@@ -196,6 +201,8 @@ class PatroniController(AbstractController):
        config['name'] = name
        config['postgresql']['data_dir'] = self._data_dir.replace('\\', '/')
        config['postgresql']['basebackup'] = [{'checkpoint': 'fast'}]
+        config['postgresql']['callbacks'] = {
+            'on_role_change': '{0} features/callback2.py {1}'.format(self._context.pctl.PYTHON, name)}
        config['postgresql']['use_unix_socket'] = os.name != 'nt'  # windows doesn't yet support unix-domain sockets
        config['postgresql']['use_unix_socket_repl'] = os.name != 'nt'
        config['postgresql']['pgpass'] = os.path.join(tempfile.gettempdir(), 'pgpass_' + name).replace('\\', '/')
@@ -239,7 +246,22 @@ class PatroniController(AbstractController):
            self.recursive_update(config, custom_config)

        self.recursive_update(config, {
-            'bootstrap': {'dcs': {'loop_wait': 2, 'postgresql': {'parameters': {'wal_keep_segments': 100}}}}})
+            'bootstrap': {
+                'dcs': {
+                    'loop_wait': 2,
+                    'postgresql': {
+                        'parameters': {
+                            'wal_keep_segments': 100,
+                            'archive_mode': 'on',
+                            'archive_command': (PatroniPoolController.ARCHIVE_RESTORE_SCRIPT +
+                                                ' --mode archive ' +
+                                                '--dirname {} --filename %f --pathname %p').format(
+                                                    os.path.join(self._work_directory, 'data', 'wal_archive'))
+                        }
+                    }
+                }
+            }
+        })
        if config['postgresql'].get('callbacks', {}).get('on_role_change'):
            config['postgresql']['callbacks']['on_role_change'] += ' ' + str(self.__PORT)

@@ -355,6 +377,7 @@ class AbstractDcsController(AbstractController):

    def __init__(self, context, mktemp=True):
        work_directory = mktemp and tempfile.mkdtemp() or None
+        self._paused = False
        super(AbstractDcsController, self).__init__(context, self.name(), work_directory, context.pctl.output_dir)

    def _is_accessible(self):
@@ -366,11 +389,22 @@ class AbstractDcsController(AbstractController):
        if self._work_directory:
            shutil.rmtree(self._work_directory)

-    def path(self, key=None, scope='batman'):
-        return self._CLUSTER_NODE.format(scope) + (key and '/' + key or '')
+    def path(self, key=None, scope='batman', group=None):
+        citus_group = '/{0}'.format(group) if group is not None else ''
+        return self._CLUSTER_NODE.format(scope) + citus_group + (key and '/' + key or '')
+
+    def start_outage(self):
+        if not self._paused and self._handle:
+            self._handle.suspend()
+            self._paused = True
+
+    def stop_outage(self):
+        if self._paused and self._handle:
+            self._handle.resume()
+            self._paused = False

    @abc.abstractmethod
-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        """ query for a value of a given key """

    @abc.abstractmethod
@@ -404,8 +438,8 @@ class ConsulController(AbstractDcsController):
        self._config_file = self._work_directory + '.json'
        with open(self._config_file, 'wb') as f:
            f.write(b'{"session_ttl_min":"5s","server":true,"bootstrap":true,"advertise_addr":"127.0.0.1"}')
-        return subprocess.Popen(['consul', 'agent', '-config-file', self._config_file, '-data-dir',
-                                 self._work_directory], stdout=self._log, stderr=subprocess.STDOUT)
+        return psutil.Popen(['consul', 'agent', '-config-file', self._config_file, '-data-dir',
+                             self._work_directory], stdout=self._log, stderr=subprocess.STDOUT)

    def stop(self, kill=False, timeout=15):
        super(ConsulController, self).stop(kill=kill, timeout=timeout)
@@ -418,11 +452,11 @@ class ConsulController(AbstractDcsController):
        except Exception:
            return False

-    def path(self, key=None, scope='batman'):
-        return super(ConsulController, self).path(key, scope)[1:]
+    def path(self, key=None, scope='batman', group=None):
+        return super(ConsulController, self).path(key, scope, group)[1:]

-    def query(self, key, scope='batman'):
-        _, value = self._client.kv.get(self.path(key, scope))
+    def query(self, key, scope='batman', group=None):
+        _, value = self._client.kv.get(self.path(key, scope, group))
        return value and value['Value'].decode('utf-8')

    def cleanup_service_tree(self):
@@ -441,8 +475,8 @@ class AbstractEtcdController(AbstractDcsController):
        self._client_cls = client_cls

    def _start(self):
-        return subprocess.Popen(["etcd", "--data-dir", self._work_directory],
-                                stdout=self._log, stderr=subprocess.STDOUT)
+        return psutil.Popen(["etcd", "--enable-v2=true", "--data-dir", self._work_directory],
+                            stdout=self._log, stderr=subprocess.STDOUT)

    def _is_running(self):
        from patroni.dcs.etcd import DnsCachingResolver
@@ -462,10 +496,10 @@ class EtcdController(AbstractEtcdController):
        super(EtcdController, self).__init__(context, EtcdClient)
        os.environ['PATRONI_ETCD_HOST'] = 'localhost:2379'

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import etcd
        try:
-            return self._client.get(self.path(key, scope)).value
+            return self._client.get(self.path(key, scope, group)).value
        except etcd.EtcdKeyNotFound:
            return None

@@ -486,9 +520,9 @@ class Etcd3Controller(AbstractEtcdController):
        super(Etcd3Controller, self).__init__(context, Etcd3Client)
        os.environ['PATRONI_ETCD3_HOST'] = 'localhost:2379'

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import base64
-        response = self._client.range(self.path(key, scope))
+        response = self._client.range(self.path(key, scope, group))
        for k in response.get('kvs', []):
            return base64.b64decode(k['value']).decode('utf-8') if 'value' in k else None

@@ -499,7 +533,43 @@ class Etcd3Controller(AbstractEtcdController):
            assert False, "exception when cleaning up etcd contents: {0}".format(e)


-class KubernetesController(AbstractDcsController):
+class AbstractExternalDcsController(AbstractDcsController):
+
+    def __init__(self, context, mktemp=True):
+        super(AbstractExternalDcsController, self).__init__(context, mktemp)
+        self._wrapper = ['sudo']
+
+    def _start(self):
+        return self._external_pid
+
+    def start_outage(self):
+        if not self._paused:
+            subprocess.call(self._wrapper + ['kill', '-SIGSTOP', self._external_pid])
+            self._paused = True
+
+    def stop_outage(self):
+        if self._paused:
+            subprocess.call(self._wrapper + ['kill', '-SIGCONT', self._external_pid])
+            self._paused = False
+
+    def _has_started(self):
+        return True
+
+    @abc.abstractmethod
+    def process_name():
+        """process name to search with pgrep"""
+
+    def _is_running(self):
+        if not self._handle:
+            self._external_pid = subprocess.check_output(['pgrep', '-nf', self.process_name()]).decode('utf-8').strip()
+            return False
+        return True
+
+    def stop(self):
+        pass
+
+
+class KubernetesController(AbstractExternalDcsController):

    def __init__(self, context):
        super(KubernetesController, self).__init__(context)
@@ -515,12 +585,41 @@ class KubernetesController(AbstractDcsController):
        self._client = k8s_client
        self._api = self._client.CoreV1Api()

-    def _start(self):
-        pass
+    def process_name(self):
+        return "localkube"

-    def create_pod(self, name, scope):
+    def _is_running(self):
+        if not self._handle:
+            context = os.environ.get('PATRONI_KUBERNETES_CONTEXT')
+            if context.startswith('kind-'):
+                container = '{0}-control-plane'.format(context[5:])
+                api_process = 'kube-apiserver'
+            elif context.startswith('k3d-'):
+                container = '{0}-server-0'.format(context)
+                api_process = 'k3s'
+            else:
+                return super(KubernetesController, self)._is_running()
+            try:
+                docker = 'docker'
+                with open(os.devnull, 'w') as null:
+                    if subprocess.call([docker, 'info'], stdout=null, stderr=null) != 0:
+                        raise Exception
+            except Exception:
+                docker = 'podman'
+                with open(os.devnull, 'w') as null:
+                    if subprocess.call([docker, 'info'], stdout=null, stderr=null) != 0:
+                        raise Exception
+            self._wrapper = [docker, 'exec', container]
+            self._external_pid = subprocess.check_output(self._wrapper + ['pidof', api_process]).decode('utf-8').strip()
+            return False
+        return True
+
+    def create_pod(self, name, scope, group=None):
+        self.delete_pod(name)
        labels = self._labels.copy()
        labels['cluster-name'] = scope
+        if group is not None:
+            labels['citus-group'] = str(group)
        metadata = self._client.V1ObjectMeta(namespace=self._namespace, name=name, labels=labels)
        spec = self._client.V1PodSpec(containers=[self._client.V1Container(name=name, image='empty')])
        body = self._client.V1Pod(metadata=metadata, spec=spec)
@@ -537,12 +636,14 @@ class KubernetesController(AbstractDcsController):
            except Exception:
                break

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        if key.startswith('members/'):
            pod = self._api.read_namespaced_pod(key[8:], self._namespace)
            return (pod.metadata.annotations or {}).get('status', '')
        else:
            try:
+                if group is not None:
+                    scope = '{0}-{1}'.format(scope, group)
                ep = scope + {'leader': '', 'history': '-config', 'initialize': '-config'}.get(key, '-' + key)
                e = self._api.read_namespaced_endpoints(ep, self._namespace)
                if key != 'sync':
@@ -567,11 +668,8 @@ class KubernetesController(AbstractDcsController):
            if len(result.items) < 1:
                break

-    def _is_running(self):
-        return True

-
-class ZooKeeperController(AbstractDcsController):
+class ZooKeeperController(AbstractExternalDcsController):

    """ handles all zookeeper related tasks, used for the tests setup and cleanup """

@@ -583,13 +681,13 @@ class ZooKeeperController(AbstractDcsController):
        import kazoo.client
        self._client = kazoo.client.KazooClient()

-    def _start(self):
-        pass  # TODO: implement later
+    def process_name(self):
+        return "zookeeper"

-    def query(self, key, scope='batman'):
+    def query(self, key, scope='batman', group=None):
        import kazoo.exceptions
        try:
-            return self._client.get(self.path(key, scope))[0].decode('utf-8')
+            return self._client.get(self.path(key, scope, group))[0].decode('utf-8')
        except kazoo.exceptions.NoNodeError:
            return None

@@ -603,6 +701,9 @@ class ZooKeeperController(AbstractDcsController):
            assert False, "exception when cleaning up zookeeper contents: {0}".format(e)

    def _is_running(self):
+        if not super(ZooKeeperController, self)._is_running():
+            return False
+
        # if zookeeper is running, but we didn't start it
        if self._client.connected:
            return True
@@ -652,12 +753,12 @@ class RaftController(AbstractDcsController):
        del env['PATRONI_RAFT_PARTNER_ADDRS']
        env['PATRONI_RAFT_SELF_ADDR'] = self.CONTROLLER_ADDR
        env['PATRONI_RAFT_DATA_DIR'] = self._work_directory
-        return subprocess.Popen([sys.executable, '-m', 'coverage', 'run',
-                                '--source=patroni', '-p', 'patroni_raft_controller.py'],
-                                stdout=self._log, stderr=subprocess.STDOUT, env=env)
+        return psutil.Popen([sys.executable, '-m', 'coverage', 'run',
+                             '--source=patroni', '-p', 'patroni_raft_controller.py'],
+                            stdout=self._log, stderr=subprocess.STDOUT, env=env)

-    def query(self, key, scope='batman'):
-        ret = self._raft.get(self.path(key, scope))
+    def query(self, key, scope='batman', group=None):
+        ret = self._raft.get(self.path(key, scope, group))
        return ret and ret['value']

    def set(self, key, value):
@@ -683,6 +784,7 @@ class PatroniPoolController(object):

    PYTHON = sys.executable.replace('\\', '/')
    BACKUP_SCRIPT = [PYTHON, 'features/backup_create.py']
+    BACKUP_RESTORE_SCRIPT = ' '.join((PYTHON, os.path.abspath('features/backup_restore.py'))).replace('\\', '/')
    ARCHIVE_RESTORE_SCRIPT = ' '.join((PYTHON, os.path.abspath('features/archive-restore.py')))

    def __init__(self, context):
@@ -769,7 +871,7 @@ class PatroniPoolController(object):
                    'archive_mode': 'on',
                    'archive_command': (self.ARCHIVE_RESTORE_SCRIPT + ' --mode archive ' +
                                        '--dirname {} --filename %f --pathname %p').format(
-                                        os.path.join(self.patroni_path, 'data', 'wal_archive').replace('\\', '/'))
+                                        os.path.join(self.patroni_path, 'data', 'wal_archive_clone').replace('\\', '/'))
                },
                'authentication': {
                    'superuser': {'password': 'zalando1'},
@@ -785,14 +887,14 @@ class PatroniPoolController(object):
            'bootstrap': {
                'method': 'backup_restore',
                'backup_restore': {
-                    'command': (self.PYTHON + ' features/backup_restore.py --sourcedir=' +
+                    'command': (self.BACKUP_RESTORE_SCRIPT + ' --sourcedir=' +
                                os.path.join(self.patroni_path, 'data', 'basebackup').replace('\\', '/')),
                    'recovery_conf': {
                        'recovery_target_action': 'promote',
                        'recovery_target_timeline': 'latest',
                        'restore_command': (self.ARCHIVE_RESTORE_SCRIPT + ' --mode restore ' +
                                            '--dirname {} --filename %f --pathname %p').format(
-                                            os.path.join(self.patroni_path, 'data', 'wal_archive').replace('\\', '/'))
+                            os.path.join(self.patroni_path, 'data', 'wal_archive_clone').replace('\\', '/'))
                    }
                }
            },
@@ -805,6 +907,25 @@ class PatroniPoolController(object):
        }
        self.start(name, custom_config=custom_config)

+    def bootstrap_from_backup_no_leader(self, name, cluster_name):
+        custom_config = {
+            'scope': cluster_name,
+            'postgresql': {
+                'recovery_conf': {
+                    'restore_command': (self.ARCHIVE_RESTORE_SCRIPT + ' --mode restore ' +
+                                        '--dirname {} --filename %f --pathname %p').format(
+                                        os.path.join(self.patroni_path, 'data', 'wal_archive').replace('\\', '/'))
+                },
+                'create_replica_methods': ['no_leader_bootstrap'],
+                'no_leader_bootstrap': {
+                    'command': (self.BACKUP_RESTORE_SCRIPT + ' --sourcedir=' +
+                                os.path.join(self.patroni_path, 'data', 'basebackup').replace('\\', '/')),
+                    'no_leader': '1'
+                }
+            }
+        }
+        self.start(name, custom_config=custom_config)
+
    @property
    def dcs(self):
        if self._dcs is None:
@@ -975,20 +1096,37 @@ def after_all(context):
 def before_feature(context, feature):
    """ create per-feature output directory to collect Patroni and PostgreSQL logs """
    if feature.name == 'watchdog' and os.name == 'nt':
-        feature.skip("Watchdog isn't supported on Windows")
-    else:
-        context.pctl.create_and_set_output_directory(feature.name)
+        return feature.skip("Watchdog isn't supported on Windows")
+    elif feature.name == 'citus':
+        lib = subprocess.check_output(['pg_config', '--pkglibdir']).decode('utf-8').strip()
+        if not os.path.exists(os.path.join(lib, 'citus.so')):
+            return feature.skip("Citus extenstion isn't available")
+    context.pctl.create_and_set_output_directory(feature.name)


 def after_feature(context, feature):
-    """ stop all Patronis, remove their data directory and cleanup the keys in etcd """
+    """ send SIGCONT to a dcs if neccessary,
+    stop all Patronis remove their data directory and cleanup the keys in etcd """
+    context.dcs_ctl.stop_outage()
    context.pctl.stop_all()
    data = os.path.join(context.pctl.patroni_path, 'data')
    if os.path.exists(data):
        shutil.rmtree(data)
    context.dcs_ctl.cleanup_service_tree()
-    if feature.status == 'failed':
+
+    found = False
+    logs = glob.glob(context.pctl.output_dir + '/patroni_*.log')
+    for log in logs:
+        with open(log) as f:
+            for line in f:
+                if 'please report it as a BUG' in line:
+                    print(':'.join([log, line.rstrip()]))
+                    found = True
+
+    if feature.status == 'failed' or found:
        shutil.copytree(context.pctl.output_dir, context.pctl.output_dir + '_failed')
+    if found:
+        raise Exception('Unexpected errors in Patroni log files')


 def before_scenario(context, scenario):
@@ -997,3 +1135,5 @@ def before_scenario(context, scenario):
            if p._conn and p._conn.server_version < 110000:
                scenario.skip('pg_replication_slot_advance() is not supported on {0}'.format(p._conn.server_version))
                break
+    if 'dcs-failsafe' in scenario.effective_tags and not context.dcs_ctl._handle:
+        scenario.skip('it is not possible to control state of {0} from tests'.format(context.dcs_ctl.name()))
@@ -52,7 +52,7 @@ Feature: ignored slots
    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
    And postgres1 does not have a logical replication slot named dummy_slot

-    # 3. After a failover the server (now a master) still has the slot.
+    # 3. After a failover the server (now a primary) still has the slot.
    When I shut down postgres0
    Then "members/postgres1" key in DCS has role=master after 3 seconds
    And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
@@ -14,9 +14,9 @@ Scenario: check API requests on a stand-alone server
 	Then I receive a response code 200
 	When I issue a GET request to http://127.0.0.1:8008/replica
 	Then I receive a response code 503
-	When I run patronictl.py reinit batman postgres0 --force
-	Then I receive a response returncode 0
-	And I receive a response output "Failed: reinitialize for member postgres0, status code=503, (I am the leader, can not reinitialize)"
+	When I issue a POST request to http://127.0.0.1:8008/reinitialize with {"force": true}
+	Then I receive a response code 503
+	And I receive a response text I am the leader, can not reinitialize
 	When I run patronictl.py switchover batman --master postgres0 --force
 	Then I receive a response returncode 1
 	And I receive a response output "Error: No candidates found to switchover to"
@@ -94,11 +94,11 @@ Scenario: check the switchover via the API in the pause mode
 	And postgres0 role is the secondary after 10 seconds
 	And replication works from postgres1 to postgres0 after 20 seconds
 	And "members/postgres0" key in DCS has state=running after 10 seconds
-	When I issue a GET request to http://127.0.0.1:8008/master
+	When I issue a GET request to http://127.0.0.1:8008/primary
 	Then I receive a response code 503
 	When I issue a GET request to http://127.0.0.1:8008/replica
 	Then I receive a response code 200
-	When I issue a GET request to http://127.0.0.1:8009/master
+	When I issue a GET request to http://127.0.0.1:8009/primary
 	Then I receive a response code 200
 	When I issue a GET request to http://127.0.0.1:8009/replica
 	Then I receive a response code 503
@@ -116,11 +116,11 @@ Scenario: check the scheduled switchover
 	And postgres1 role is the secondary after 10 seconds
 	And replication works from postgres0 to postgres1 after 25 seconds
 	And "members/postgres1" key in DCS has state=running after 10 seconds
-	When I issue a GET request to http://127.0.0.1:8008/master
+	When I issue a GET request to http://127.0.0.1:8008/primary
 	Then I receive a response code 200
 	When I issue a GET request to http://127.0.0.1:8008/replica
 	Then I receive a response code 503
-	When I issue a GET request to http://127.0.0.1:8009/master
+	When I issue a GET request to http://127.0.0.1:8009/primary
 	Then I receive a response code 503
 	When I issue a GET request to http://127.0.0.1:8009/replica
 	Then I receive a response code 200
@@ -35,7 +35,7 @@ Feature: standby cluster
    When I add the table foo to postgres0
    Then table foo is present on postgres1 after 20 seconds
    And I sleep for 3 seconds
-    When I issue a GET request to http://127.0.0.1:8009/master
+    When I issue a GET request to http://127.0.0.1:8009/primary
    Then I receive a response code 503
    When I issue a GET request to http://127.0.0.1:8009/standby_leader
    Then I receive a response code 200
@@ -50,7 +50,7 @@ Feature: standby cluster
    When I kill postgres1
    And I kill postmaster on postgres1
    Then postgres2 is replicating from postgres0 after 32 seconds
-    When I issue a GET request to http://127.0.0.1:8010/master
+    When I issue a GET request to http://127.0.0.1:8010/primary
    Then I receive a response code 503
    And I sleep for 3 seconds
    When I issue a GET request to http://127.0.0.1:8010/standby_leader
@@ -83,10 +83,10 @@ def check_role(context, pg_name, pg_role, max_promotion_timeout):
        "{0} role didn't change to {1} after {2} seconds".format(pg_name, pg_role, max_promotion_timeout)


-@step('replication works from {master:w} to {replica:w} after {time_limit:d} seconds')
-@then('replication works from {master:w} to {replica:w} after {time_limit:d} seconds')
-def replication_works(context, master, replica, time_limit):
+@step('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
+@then('replication works from {primary:w} to {replica:w} after {time_limit:d} seconds')
+def replication_works(context, primary, replica, time_limit):
    context.execute_steps(u"""
        When I add the table test_{0} to {1}
        Then table test_{0} is present on {2} after {3} seconds
-    """.format(int(time()), master, replica, time_limit))
+    """.format(int(time()), primary, replica, time_limit))
@@ -11,11 +11,8 @@ def start_patroni_with_a_name_value_tag(context, name, tag_name, tag_value):

@then('There is a {label} with "{content}" in {name:w} data directory')
 def check_label(context, label, content, name):
-    label = context.pctl.read_label(name, label)
-    if label is None:
-        label = ""
-    label = label.replace('\n', '\\n')
-    assert content in label, "\"{0}\" doesn't contain {1}".format(label, content)
+    value = (context.pctl.read_label(name, label) or '').replace('\n', '\\n')
+    assert content in value, "\"{0}\" in {1} doesn't contain {2}".format(value, label, content)


@step('I create label with "{content:w}" in {name:w} data directory')
@@ -0,0 +1,117 @@
+import json
+import time
+
+from behave import step, then
+from dateutil import tz
+from datetime import datetime
+from functools import partial
+from threading import Thread, Event
+
+tzutc = tz.tzutc()
+
+
+@step('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
+@then('{name:w} is a leader in a group {group:d} after {time_limit:d} seconds')
+def is_a_group_leader(context, name, group, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    while (context.dcs_ctl.query("leader", group=group) != name):
+        time.sleep(1)
+        assert time.time() < max_time, "{0} is not a leader in dcs after {1} seconds".format(name, time_limit)
+
+
+@step('"{name}" key in a group {group:d} in DCS has {key:w}={value} after {time_limit:d} seconds')
+def check_group_member(context, name, group, key, value, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    dcs_value = None
+    response = None
+    while time.time() < max_time:
+        try:
+            response = json.loads(context.dcs_ctl.query(name, group=group))
+            dcs_value = response.get(key)
+            if dcs_value == value:
+                return
+        except Exception:
+            pass
+        time.sleep(1)
+    assert False, ("{0} in a group {1} does not have {2}={3} (found {4}) in dcs" +
+                   " after {5} seconds").format(name, group, key, value, response, time_limit)
+
+
+@step('I start {name:w} in citus group {group:d}')
+def start_citus(context, name, group):
+    return context.pctl.start(name, custom_config={"citus": {"database": "postgres", "group": int(group)}})
+
+
+@step('{name1:w} is registered in the coordinator {name2:w} as the worker in group {group:d}')
+def check_registration(context, name1, name2, group):
+    worker_port = int(context.pctl.query(name1, "SHOW port").fetchone()[0])
+    r = context.pctl.query(name2, "SELECT nodeport FROM pg_catalog.pg_dist_node WHERE groupid = {0}".format(group))
+    assert worker_port == r.fetchone()[0],\
+        "Worker {0} is not registered in pg_dist_node on the coordinator {1}".format(name1, name2)
+
+
+@step('I create a distributed table on {name:w}')
+def create_distributed_table(context, name):
+    context.pctl.query(name, 'CREATE TABLE public.d(id int not null)')
+    context.pctl.query(name, "SELECT create_distributed_table('public.d', 'id')")
+
+
+@step('I cleanup a distributed table on {name:w}')
+def cleanup_distributed_table(context, name):
+    context.pctl.query(name, 'TRUNCATE public.d')
+
+
+def insert_thread(query_func, context):
+    while True:
+        if context.thread_stop_event.is_set():
+            break
+
+        context.insert_counter += 1
+        query_func('INSERT INTO public.d VALUES({0})'.format(context.insert_counter))
+
+        context.thread_stop_event.wait(0.01)
+
+
+@step('I start a thread inserting data on {name:w}')
+def start_insert_thread(context, name):
+    context.thread_stop_event = Event()
+    context.insert_counter = 0
+    query_func = partial(context.pctl.query, name)
+    thread_func = partial(insert_thread, query_func, context)
+    context.thread = Thread(target=thread_func)
+    context.thread.daemon = True
+    context.thread.start()
+
+
+@then('a thread is still alive')
+def thread_is_alive(context):
+    assert context.thread.is_alive(), "Thread is not alive"
+
+
+@step("I stop a thread")
+def stop_insert_thread(context):
+    context.thread_stop_event.set()
+    context.thread.join(1*context.timeout_multiplier)
+    assert not context.thread.is_alive(), "Thread is still alive"
+
+
+@step("a distributed table on {name:w} has expected rows")
+def count_rows(context, name):
+    rows = context.pctl.query(name, "SELECT COUNT(*) FROM public.d").fetchone()[0]
+    assert rows == context.insert_counter, "Distributed table doesn't have expected amount of rows"
+
+
+@step("There is a transaction in progress on {name:w} changing pg_dist_node")
+def check_transaction(context, name):
+    cur = context.pctl.query(name, "SELECT xact_start FROM pg_stat_activity WHERE pid <> pg_backend_pid()"
+                                   " AND state = 'idle in transaction' AND query ~ 'citus_update_node'")
+    assert cur.rowcount == 1, "There is no idle in transaction updating pg_dist_node"
+    context.xact_start = cur.fetchone()[0]
+
+
+@step("a transaction finishes in {timeout:d} seconds")
+def check_transaction_timeout(context, timeout):
+    assert (datetime.now(tzutc) - context.xact_start).seconds > timeout,\
+            "a transaction finished earlier than in {0} seconds".format(timeout)
@@ -0,0 +1,16 @@
+from behave import step
+
+
+@step('DCS is down')
+def start_dcs_outage(context):
+    context.dcs_ctl.start_outage()
+
+
+@step('DCS is up')
+def stop_dcs_outage(context):
+    context.dcs_ctl.stop_outage()
+
+
+@step('I start {name:w} in a cluster {cluster_name:w} from backup with no_leader')
+def start_cluster_from_backup_no_leader(context, name, cluster_name):
+    context.pctl.bootstrap_from_backup_no_leader(name, cluster_name)
@@ -109,6 +109,8 @@ def check_response(context, component, data):
        assert data.strip('"') in context.response, "response {0} does not contain {1}".format(context.response, data)
    else:
        assert component in context.response, "{0} is not part of the response".format(component)
+        if context.certfile:
+            data = data.replace('http://', 'https://')
        assert str(context.response[component]) == str(data), "{0} does not contain {1}".format(component, data)


@@ -131,7 +133,21 @@ def add_tag_to_config(context, tag, value, pg_name):
    context.pctl.add_tag_to_config(pg_name, tag, value)


-@then('Response on GET {url} contains {value} after {timeout:d} seconds')
+@then('Status code on GET {url:url} is {code:d} after {timeout:d} seconds')
+def check_http_code(context, url, code, timeout):
+    if context.certfile:
+        url = url.replace('http://', 'https://')
+    timeout *= context.timeout_multiplier
+    for _ in range(int(timeout)):
+        r = context.request_executor.request('GET', url)
+        if int(code) == int(r.status):
+            break
+        time.sleep(1)
+    else:
+        assert False, "HTTP Status Code is not {0} after {1} seconds".format(code, timeout)
+
+
+@then('Response on GET {url:url} contains {value} after {timeout:d} seconds')
 def check_http_response(context, url, value, timeout, negate=False):
    if context.certfile:
        url = url.replace('http://', 'https://')
@@ -4,7 +4,6 @@ LABEL maintainer="Alexander Kukushkin <[email protected]>"
 RUN export DEBIAN_FRONTEND=noninteractive \
    && echo 'APT::Install-Recommends "0";\nAPT::Install-Suggests "0";' > /etc/apt/apt.conf.d/01norecommend \
    && apt-get update -y \
-    && apt-get upgrade -y \
    && apt-cache depends patroni | sed -n -e 's/.* Depends: \(python3-.\+\)$/\1/p' \
            | grep -Ev '^python3-(sphinx|etcd|consul|kazoo|kubernetes)' \
            | xargs apt-get install -y vim-tiny curl jq locales git python3-pip python3-wheel \
@@ -25,7 +24,7 @@ RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* /root/.cache

-ADD entrypoint.sh /
+COPY entrypoint.sh /

 EXPOSE 5432 8008
 ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 EDITOR=/usr/bin/editor
@@ -0,0 +1,42 @@
+FROM postgres:15
+LABEL maintainer="Alexander Kukushkin <[email protected]>"
+
+RUN export DEBIAN_FRONTEND=noninteractive \
+    && echo 'APT::Install-Recommends "0";\nAPT::Install-Suggests "0";' > /etc/apt/apt.conf.d/01norecommend \
+    && apt-get update -y \
+    && apt-get upgrade -y \
+    && apt-cache depends patroni | sed -n -e 's/.* Depends: \(python3-.\+\)$/\1/p' \
+            | grep -Ev '^python3-(sphinx|etcd|consul|kazoo|kubernetes)' \
+            | xargs apt-get install -y busybox vim-tiny curl jq less locales git python3-pip python3-wheel \
+    ## Make sure we have a en_US.UTF-8 locale available
+    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
+    && curl https://install.citusdata.com/community/deb.sh | bash \
+    && apt-get -y install postgresql-15-citus-11.2 \
+    && pip3 install setuptools \
+    && pip3 install 'git+https://github.com/zalando/patroni.git#egg=patroni[kubernetes]' \
+    && PGHOME=/home/postgres \
+    && mkdir -p $PGHOME \
+    && chown postgres $PGHOME \
+    && sed -i "s|/var/lib/postgresql.*|$PGHOME:/bin/bash|" /etc/passwd \
+    && /bin/busybox --install -s \
+    # Set permissions for OpenShift
+    && chmod 775 $PGHOME \
+    && chmod 664 /etc/passwd \
+    # Clean up
+    && apt-get remove -y git python3-pip python3-wheel \
+    && apt-get autoremove -y \
+    && apt-get clean -y \
+    && rm -rf /var/lib/apt/lists/* /root/.cache
+
+ADD entrypoint.sh /
+ENV PGSSLMODE=verify-ca PGSSLKEY=/etc/ssl/private/ssl-cert-snakeoil.key PGSSLCERT=/etc/ssl/certs/ssl-cert-snakeoil.pem PGSSLROOTCERT=/etc/ssl/certs/ssl-cert-snakeoil.pem
+
+RUN sed -i 's/^postgresql:/&\n  basebackup:\n    checkpoint: fast/' /entrypoint.sh \
+    && sed -i "s|^    postgresql:|&\n      pg_hba:\n      - local all all trust\n      - hostssl replication all all md5 clientcert=$PGSSLMODE\n      - hostssl all all all md5 clientcert=$PGSSLMODE\n      parameters:\n        max_connections: 100\n        shared_buffers: 16MB\n        ssl: 'on'\n        ssl_ca_file: $PGSSLROOTCERT\n        ssl_cert_file: $PGSSLCERT\n        ssl_key_file: $PGSSLKEY\n        citus.node_conninfo: 'sslrootcert=$PGSSLROOTCERT sslkey=$PGSSLKEY sslcert=$PGSSLCERT sslmode=$PGSSLMODE'|"  /entrypoint.sh \
+    && sed -i "s#^    \(superuser\|replication\):#&\n      sslmode: $PGSSLMODE\n      sslkey: $PGSSLKEY\n      sslcert: $PGSSLCERT\n      sslrootcert: $PGSSLROOTCERT#" /entrypoint.sh
+
+EXPOSE 5432 8008
+ENV LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 EDITOR=/usr/bin/editor
+USER postgres
+WORKDIR /home/postgres
+CMD ["/bin/bash", "/entrypoint.sh"]
@@ -0,0 +1,154 @@
+# Kubernetes deployment examples
+Below you will find examples of Patroni deployments using [kind](https://kind.sigs.k8s.io/).
+
+# Patroni on K8s
+The Patroni cluster deployment with a StatefulSet consisting of three Pods.
+
+Example session:
+
+    $ kind create cluster
+    Creating cluster "kind" ...
+     ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
+     ✓ Preparing nodes 📦
+     ✓ Writing configuration 📜
+     ✓ Starting control-plane 🕹️
+     ✓ Installing CNI 🔌
+     ✓ Installing StorageClass 💾
+    Set kubectl context to "kind-kind"
+    You can now use your cluster with:
+
+    kubectl cluster-info --context kind-kind
+
+    Thanks for using kind! 😊
+
+    $ docker build -t patroni .
+    Sending build context to Docker daemon  138.8kB
+    Step 1/9 : FROM postgres:15
+    ...
+    Successfully built e9bfe69c5d2b
+    Successfully tagged patroni:latest
+
+    $ kind load docker-image patroni
+    Image: "" with ID "sha256:e9bfe69c5d2b319dec0cf564fb895484537664775e18f37f9b707914cc5537e6" not yet present on node "kind-control-plane", loading...
+
+    $ kubectl apply -f patroni_k8s.yaml
+    service/patronidemo-config created
+    statefulset.apps/patronidemo created
+    endpoints/patronidemo created
+    service/patronidemo created
+    service/patronidemo-repl created
+    secret/patronidemo created
+    serviceaccount/patronidemo created
+    role.rbac.authorization.k8s.io/patronidemo created
+    rolebinding.rbac.authorization.k8s.io/patronidemo created
+    clusterrole.rbac.authorization.k8s.io/patroni-k8s-ep-access created
+    clusterrolebinding.rbac.authorization.k8s.io/patroni-k8s-ep-access created
+
+    $ kubectl get pods -L role
+    NAME            READY   STATUS    RESTARTS   AGE   ROLE
+    patronidemo-0   1/1     Running   0          34s   master
+    patronidemo-1   1/1     Running   0          30s   replica
+    patronidemo-2   1/1     Running   0          26s   replica
+
+    $ kubectl exec -ti patronidemo-0 -- bash
+    postgres@patronidemo-0:~$ patronictl list
+    + Cluster: patronidemo (7186662553319358497) ----+----+-----------+
+    | Member        | Host       | Role    | State   | TL | Lag in MB |
+    +---------------+------------+---------+---------+----+-----------+
+    | patronidemo-0 | 10.244.0.5 | Leader  | running |  1 |           |
+    | patronidemo-1 | 10.244.0.6 | Replica | running |  1 |         0 |
+    | patronidemo-2 | 10.244.0.7 | Replica | running |  1 |         0 |
+    +---------------+------------+---------+---------+----+-----------+
+
+# Citus on K8s
+The Citus cluster with the StatefulSets, one coordinator with three Pods and two workers with two pods each.
+
+Example session:
+
+    $ kind create cluster
+    Creating cluster "kind" ...
+     ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
+     ✓ Preparing nodes 📦
+     ✓ Writing configuration 📜
+     ✓ Starting control-plane 🕹️
+     ✓ Installing CNI 🔌
+     ✓ Installing StorageClass 💾
+    Set kubectl context to "kind-kind"
+    You can now use your cluster with:
+
+    kubectl cluster-info --context kind-kind
+
+    Thanks for using kind! 😊
+
+    demo@localhost:~/git/patroni/kubernetes$ docker build -f Dockerfile.citus -t patroni-citus-k8s .
+    Sending build context to Docker daemon  138.8kB
+    Step 1/11 : FROM postgres:15
+    ...
+    Successfully built 8cd73e325028
+    Successfully tagged patroni-citus-k8s:latest
+
+    $ kind load docker-image patroni-citus-k8s
+    Image: "" with ID "sha256:8cd73e325028d7147672494965e53453f5540400928caac0305015eb2c7027c7" not yet present on node "kind-control-plane", loading...
+
+    $ kubectl apply -f citus_k8s.yaml
+    service/citusdemo-0-config created
+    service/citusdemo-1-config created
+    service/citusdemo-2-config created
+    statefulset.apps/citusdemo-0 created
+    statefulset.apps/citusdemo-1 created
+    statefulset.apps/citusdemo-2 created
+    endpoints/citusdemo-0 created
+    service/citusdemo-0 created
+    endpoints/citusdemo-1 created
+    service/citusdemo-1 created
+    endpoints/citusdemo-2 created
+    service/citusdemo-2 created
+    service/citusdemo-workers created
+    secret/citusdemo created
+    serviceaccount/citusdemo created
+    role.rbac.authorization.k8s.io/citusdemo created
+    rolebinding.rbac.authorization.k8s.io/citusdemo created
+    clusterrole.rbac.authorization.k8s.io/patroni-k8s-ep-access created
+    clusterrolebinding.rbac.authorization.k8s.io/patroni-k8s-ep-access created
+
+    $ kubectl get sts
+    NAME          READY   AGE
+    citusdemo-0   1/3     6s  # coodinator (group=0)
+    citusdemo-1   1/2     6s  # worker (group=1)
+    citusdemo-2   1/2     6s  # worker (group=2)
+
+    $ kubectl get pods -l cluster-name=citusdemo -L role
+    NAME            READY   STATUS    RESTARTS   AGE    ROLE
+    citusdemo-0-0   1/1     Running   0          105s   master
+    citusdemo-0-1   1/1     Running   0          101s   replica
+    citusdemo-0-2   1/1     Running   0          96s    replica
+    citusdemo-1-0   1/1     Running   0          105s   master
+    citusdemo-1-1   1/1     Running   0          101s   replica
+    citusdemo-2-0   1/1     Running   0          105s   master
+    citusdemo-2-1   1/1     Running   0          101s   replica
+
+    $ kubectl exec -ti citusdemo-0-0 -- bash
+    postgres@citusdemo-0-0:~$ patronictl list
+    + Citus cluster: citusdemo -----------+--------------+---------+----+-----------+
+    | Group | Member        | Host        | Role         | State   | TL | Lag in MB |
+    +-------+---------------+-------------+--------------+---------+----+-----------+
+    |     0 | citusdemo-0-0 | 10.244.0.10 | Leader       | running |  1 |           |
+    |     0 | citusdemo-0-1 | 10.244.0.12 | Replica      | running |  1 |         0 |
+    |     0 | citusdemo-0-2 | 10.244.0.14 | Sync Standby | running |  1 |         0 |
+    |     1 | citusdemo-1-0 | 10.244.0.8  | Leader       | running |  1 |           |
+    |     1 | citusdemo-1-1 | 10.244.0.11 | Sync Standby | running |  1 |         0 |
+    |     2 | citusdemo-2-0 | 10.244.0.9  | Leader       | running |  1 |           |
+    |     2 | citusdemo-2-1 | 10.244.0.13 | Sync Standby | running |  1 |         0 |
+    +-------+---------------+-------------+--------------+---------+----+-----------+
+
+    postgres@citusdemo-0-0:~$ psql citus
+    psql (15.1 (Debian 15.1-1.pgdg110+1))
+    Type "help" for help.
+
+    citus=# table pg_dist_node;
+     nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
+    --------+---------+-------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
+          1 |       0 | 10.244.0.10 |     5432 | default  | t           | t        | primary  | default     | t              | f
+          2 |       1 | 10.244.0.8  |     5432 | default  | t           | t        | primary  | default     | t              | t
+          3 |       2 | 10.244.0.9  |     5432 | default  | t           | t        | primary  | default     | t              | t
+    (3 rows)
@@ -0,0 +1,590 @@
+# headless services to avoid deletion of citusdemo-*-config endpoints
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-0-config
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '0'
+spec:
+  clusterIP: None
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-1-config
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '1'
+spec:
+  clusterIP: None
+
+---
+
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-2-config
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '2'
+spec:
+  clusterIP: None
+
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: &cluster_name citusdemo-0
+  labels: &labels
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '0'
+    citus-type: coordinator
+spec:
+  replicas: 3
+  serviceName: *cluster_name
+  selector:
+    matchLabels:
+      <<: *labels
+  template:
+    metadata:
+      labels:
+        <<: *labels
+    spec:
+      serviceAccountName: citusdemo
+      containers:
+      - name: *cluster_name
+        image: patroni-citus-k8s  # docker build -f Dockerfile.citus -t patroni-citus-k8s .
+        imagePullPolicy: IfNotPresent
+        readinessProbe:
+          httpGet:
+            scheme: HTTP
+            path: /readiness
+            port: 8008
+          initialDelaySeconds: 3
+          periodSeconds: 10
+          timeoutSeconds: 5
+          successThreshold: 1
+          failureThreshold: 3
+        ports:
+        - containerPort: 8008
+          protocol: TCP
+        - containerPort: 5432
+          protocol: TCP
+        volumeMounts:
+        - mountPath: /home/postgres/pgdata
+          name: pgdata
+        env:
+        - name: PATRONI_KUBERNETES_POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        - name: PATRONI_KUBERNETES_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: PATRONI_KUBERNETES_BYPASS_API_SERVICE
+          value: 'true'
+        - name: PATRONI_KUBERNETES_USE_ENDPOINTS
+          value: 'true'
+        - name: PATRONI_KUBERNETES_LABELS
+          value: '{application: patroni, cluster-name: citusdemo}'
+        - name: PATRONI_CITUS_DATABASE
+          value: citus
+        - name: PATRONI_CITUS_GROUP
+          value: '0'
+        - name: PATRONI_SUPERUSER_USERNAME
+          value: postgres
+        - name: PATRONI_SUPERUSER_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: superuser-password
+        - name: PATRONI_REPLICATION_USERNAME
+          value: standby
+        - name: PATRONI_REPLICATION_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: replication-password
+        - name: PATRONI_SCOPE
+          value: citusdemo
+        - name: PATRONI_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: PATRONI_POSTGRESQL_DATA_DIR
+          value: /home/postgres/pgdata/pgroot/data
+        - name: PATRONI_POSTGRESQL_PGPASS
+          value: /tmp/pgpass
+        - name: PATRONI_POSTGRESQL_LISTEN
+          value: '0.0.0.0:5432'
+        - name: PATRONI_RESTAPI_LISTEN
+          value: '0.0.0.0:8008'
+      terminationGracePeriodSeconds: 0
+      volumes:
+      - name: pgdata
+        emptyDir: {}
+#  volumeClaimTemplates:
+#  - metadata:
+#      labels:
+#        application: spilo
+#        spilo-cluster: *cluster_name
+#      annotations:
+#        volume.alpha.kubernetes.io/storage-class: anything
+#      name: pgdata
+#    spec:
+#      accessModes:
+#      - ReadWriteOnce
+#      resources:
+#        requests:
+#          storage: 5Gi
+
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: &cluster_name citusdemo-1
+  labels: &labels
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '1'
+    citus-type: worker
+spec:
+  replicas: 2
+  serviceName: *cluster_name
+  selector:
+    matchLabels:
+      <<: *labels
+  template:
+    metadata:
+      labels:
+        <<: *labels
+    spec:
+      serviceAccountName: citusdemo
+      containers:
+      - name: *cluster_name
+        image: patroni-citus-k8s  # docker build -f Dockerfile.citus -t patroni-citus-k8s .
+        imagePullPolicy: IfNotPresent
+        readinessProbe:
+          httpGet:
+            scheme: HTTP
+            path: /readiness
+            port: 8008
+          initialDelaySeconds: 3
+          periodSeconds: 10
+          timeoutSeconds: 5
+          successThreshold: 1
+          failureThreshold: 3
+        ports:
+        - containerPort: 8008
+          protocol: TCP
+        - containerPort: 5432
+          protocol: TCP
+        volumeMounts:
+        - mountPath: /home/postgres/pgdata
+          name: pgdata
+        env:
+        - name: PATRONI_KUBERNETES_POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        - name: PATRONI_KUBERNETES_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: PATRONI_KUBERNETES_BYPASS_API_SERVICE
+          value: 'true'
+        - name: PATRONI_KUBERNETES_USE_ENDPOINTS
+          value: 'true'
+        - name: PATRONI_KUBERNETES_LABELS
+          value: '{application: patroni, cluster-name: citusdemo}'
+        - name: PATRONI_CITUS_DATABASE
+          value: citus
+        - name: PATRONI_CITUS_GROUP
+          value: '1'
+        - name: PATRONI_SUPERUSER_USERNAME
+          value: postgres
+        - name: PATRONI_SUPERUSER_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: superuser-password
+        - name: PATRONI_REPLICATION_USERNAME
+          value: standby
+        - name: PATRONI_REPLICATION_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: replication-password
+        - name: PATRONI_SCOPE
+          value: citusdemo
+        - name: PATRONI_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: PATRONI_POSTGRESQL_DATA_DIR
+          value: /home/postgres/pgdata/pgroot/data
+        - name: PATRONI_POSTGRESQL_PGPASS
+          value: /tmp/pgpass
+        - name: PATRONI_POSTGRESQL_LISTEN
+          value: '0.0.0.0:5432'
+        - name: PATRONI_RESTAPI_LISTEN
+          value: '0.0.0.0:8008'
+      terminationGracePeriodSeconds: 0
+      volumes:
+      - name: pgdata
+        emptyDir: {}
+#  volumeClaimTemplates:
+#  - metadata:
+#      labels:
+#        application: spilo
+#        spilo-cluster: *cluster_name
+#      annotations:
+#        volume.alpha.kubernetes.io/storage-class: anything
+#      name: pgdata
+#    spec:
+#      accessModes:
+#      - ReadWriteOnce
+#      resources:
+#        requests:
+#          storage: 5Gi
+
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: &cluster_name citusdemo-2
+  labels: &labels
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '2'
+    citus-type: worker
+spec:
+  replicas: 2
+  serviceName: *cluster_name
+  selector:
+    matchLabels:
+      <<: *labels
+  template:
+    metadata:
+      labels:
+        <<: *labels
+    spec:
+      serviceAccountName: citusdemo
+      containers:
+      - name: *cluster_name
+        image: patroni-citus-k8s  # docker build -f Dockerfile.citus -t patroni-citus-k8s .
+        imagePullPolicy: IfNotPresent
+        readinessProbe:
+          httpGet:
+            scheme: HTTP
+            path: /readiness
+            port: 8008
+          initialDelaySeconds: 3
+          periodSeconds: 10
+          timeoutSeconds: 5
+          successThreshold: 1
+          failureThreshold: 3
+        ports:
+        - containerPort: 8008
+          protocol: TCP
+        - containerPort: 5432
+          protocol: TCP
+        volumeMounts:
+        - mountPath: /home/postgres/pgdata
+          name: pgdata
+        env:
+        - name: PATRONI_KUBERNETES_POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        - name: PATRONI_KUBERNETES_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: PATRONI_KUBERNETES_BYPASS_API_SERVICE
+          value: 'true'
+        - name: PATRONI_KUBERNETES_USE_ENDPOINTS
+          value: 'true'
+        - name: PATRONI_KUBERNETES_LABELS
+          value: '{application: patroni, cluster-name: citusdemo}'
+        - name: PATRONI_CITUS_DATABASE
+          value: citus
+        - name: PATRONI_CITUS_GROUP
+          value: '2'
+        - name: PATRONI_SUPERUSER_USERNAME
+          value: postgres
+        - name: PATRONI_SUPERUSER_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: superuser-password
+        - name: PATRONI_REPLICATION_USERNAME
+          value: standby
+        - name: PATRONI_REPLICATION_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: citusdemo
+              key: replication-password
+        - name: PATRONI_SCOPE
+          value: citusdemo
+        - name: PATRONI_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: PATRONI_POSTGRESQL_DATA_DIR
+          value: /home/postgres/pgdata/pgroot/data
+        - name: PATRONI_POSTGRESQL_PGPASS
+          value: /tmp/pgpass
+        - name: PATRONI_POSTGRESQL_LISTEN
+          value: '0.0.0.0:5432'
+        - name: PATRONI_RESTAPI_LISTEN
+          value: '0.0.0.0:8008'
+      terminationGracePeriodSeconds: 0
+      volumes:
+      - name: pgdata
+        emptyDir: {}
+#  volumeClaimTemplates:
+#  - metadata:
+#      labels:
+#        application: spilo
+#        spilo-cluster: *cluster_name
+#      annotations:
+#        volume.alpha.kubernetes.io/storage-class: anything
+#      name: pgdata
+#    spec:
+#      accessModes:
+#      - ReadWriteOnce
+#      resources:
+#        requests:
+#          storage: 5Gi
+          
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: citusdemo-0
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '0'
+    citus-type: coordinator
+subsets: []
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-0
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '0'
+    citus-type: coordinator
+spec:
+  type: ClusterIP
+  ports:
+  - port: 5432
+    targetPort: 5432
+
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: citusdemo-1
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '1'
+    citus-type: worker
+subsets: []
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-1
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '1'
+    citus-type: worker
+spec:
+  type: ClusterIP
+  ports:
+  - port: 5432
+    targetPort: 5432
+
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: citusdemo-2
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '2'
+    citus-type: worker
+subsets: []
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-2
+  labels:
+    application: patroni
+    cluster-name: citusdemo
+    citus-group: '2'
+    citus-type: worker
+spec:
+  type: ClusterIP
+  ports:
+  - port: 5432
+    targetPort: 5432
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: citusdemo-workers
+  labels: &labels
+    application: patroni
+    cluster-name: citusdemo
+    citus-type: worker
+    role: master
+spec:
+  type: ClusterIP
+  selector:
+    <<: *labels
+  ports:
+  - port: 5432
+    targetPort: 5432
+
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: &cluster_name citusdemo
+  labels:
+    application: patroni
+    cluster-name: *cluster_name
+type: Opaque
+data:
+  superuser-password: emFsYW5kbw==
+  replication-password: cmVwLXBhc3M=
+
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: citusdemo
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: citusdemo
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - configmaps
+  verbs:
+  - create
+  - get
+  - list
+  - patch
+  - update
+  - watch
+  # delete and deletecollection are required only for 'patronictl remove'
+  - delete
+  - deletecollection
+- apiGroups:
+  - ""
+  resources:
+  - endpoints
+  verbs:
+  - get
+  - patch
+  - update
+  # the following three privileges are necessary only when using endpoints
+  - create
+  - list
+  - watch
+  # delete and deletecollection are required only for for 'patronictl remove'
+  - delete
+  - deletecollection
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  verbs:
+  - get
+  - list
+  - patch
+  - update
+  - watch
+# The following privilege is only necessary for creation of headless service
+# for citusdemo-config endpoint, in order to prevent cleaning it up by the
+# k8s master. You can avoid giving this privilege by explicitly creating the
+# service like it is done in this manifest (lines 2..10)
+- apiGroups:
+  - ""
+  resources:
+  - services
+  verbs:
+  - create
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: citusdemo
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: citusdemo
+subjects:
+- kind: ServiceAccount
+  name: citusdemo
+
+# Following privileges are only required if deployed not in the "default"
+# namespace and you want Patroni to bypass kubernetes service
+# (PATRONI_KUBERNETES_BYPASS_API_SERVICE=true)
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: patroni-k8s-ep-access
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - endpoints
+  resourceNames:
+  - kubernetes
+  verbs:
+  - get
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: patroni-k8s-ep-access
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: patroni-k8s-ep-access
+subjects:
+- kind: ServiceAccount
+  name: citusdemo
+# The namespace must be specified explicitly.
+# If deploying to the different namespace you have to change it.
+  namespace: default
@@ -1,5 +1,5 @@
 # Patroni OpenShift Configuration
-Patroni can be run in OpenShift. Based on the kubernetes configuration, the Dockerfile and Entrypoint has been modified to support the dynamic UID/GID configuration that is applied in OpenShift. This can be run under the standard `restricted` SCC. 
+Patroni can be run in OpenShift. Based on the kubernetes configuration, the Dockerfile and Entrypoint has been modified to support the dynamic UID/GID configuration that is applied in OpenShift. This can be run under the standard `restricted` SCC.

 # Examples

@@ -11,39 +11,39 @@ oc new-project patroni-test

 ## Build the image

-Note: Update the references when merged upstream. 
-Note: If deploying as a template for multiple users, the following commands should be performed in a shared namespace like `openshift`. 
+Note: Update the references when merged upstream.
+Note: If deploying as a template for multiple users, the following commands should be performed in a shared namespace like `openshift`.

 ```
 oc import-image postgres:10 --confirm -n openshift
 oc new-build https://github.com/zalando/patroni --context-dir=kubernetes -n openshift
 ```

-## Deploy the Image 
-Two configuration templates exist in [templates](templates) directory: 
- Patroni Ephemeral
- Patroni Persistent
+## Deploy the Image
+Two configuration templates exist in [templates](templates) directory:
+-   Patroni Ephemeral
+-   Patroni Persistent

-The only difference is whether or not the statefulset requests persistent storage. 
+The only difference is whether or not the statefulset requests persistent storage.

 ## Create the Template
-Install the template into the `openshift` namespace if this should be shared across projects: 
+Install the template into the `openshift` namespace if this should be shared across projects:

 ```
 oc create -f templates/template_patroni_ephemeral.yml -n openshift
 ```

-Then, from your own project: 
+Then, from your own project:

 ```
 oc new-app patroni-pgsql-ephemeral
 ```

-Once the pods are running, two configmaps should be available: 
+Once the pods are running, two configmaps should be available:

 ```
 $ oc get configmap
 NAME                DATA      AGE
 patroniocp-config   0         1m
 patroniocp-leader   0         1m
-```
+```
@@ -1,2 +1,2 @@
 # Jenkins Test
-This pipeline test will create a separate deployment config for a pgbench pod and execute a test against the patroni cluster. This is a sample and should be customized. 
+This pipeline test will create a separate deployment config for a pgbench pod and execute a test against the patroni cluster. This is a sample and should be customized.
@@ -1,5 +1,5 @@
 #!/bin/sh
 set -e

-pip install --ignore-installed setuptools==19.2 pyinstaller
-pyinstaller --clean --onefile patroni.spec
+pip install --ignore-installed pyinstaller
+pyinstaller --clean patroni.spec
@@ -8,7 +8,7 @@ def hiddenimports():
    sys.path.insert(0, '.')
    try:
        import patroni.dcs
-        return patroni.dcs.dcs_modules()
+        return patroni.dcs.dcs_modules() + ['http.server']
    finally:
        sys.path.pop(0)

@@ -3,7 +3,7 @@ import os
 import signal
 import time

-from .daemon import AbstractPatroniDaemon, abstract_main
+from patroni.daemon import AbstractPatroniDaemon, abstract_main

 logger = logging.getLogger(__name__)

@@ -11,13 +11,13 @@ logger = logging.getLogger(__name__)
 class Patroni(AbstractPatroniDaemon):

    def __init__(self, config):
-        from .api import RestApiServer
-        from .dcs import get_dcs
-        from .ha import Ha
-        from .postgresql import Postgresql
-        from .request import PatroniRequest
-        from .version import __version__
-        from .watchdog import Watchdog
+        from patroni.api import RestApiServer
+        from patroni.dcs import get_dcs
+        from patroni.ha import Ha
+        from patroni.postgresql import Postgresql
+        from patroni.request import PatroniRequest
+        from patroni.version import __version__
+        from patroni.watchdog import Watchdog

        super(Patroni, self).__init__(config)

@@ -138,7 +138,7 @@ def patroni_main():

 def main():
    if os.getpid() != 1:
-        from . import check_psycopg
+        from patroni import check_psycopg

        check_psycopg()
        return patroni_main()
@@ -97,7 +97,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
    def do_GET(self, write_status_code_only=False):
        """Default method for processing all GET requests which can not be routed to other methods"""

-        path = '/master' if self.path == '/' else self.path
+        path = '/primary' if self.path == '/' else self.path
        response = self.get_postgresql_status()

        patroni = self.server.patroni
@@ -114,8 +114,8 @@ class RestApiHandler(BaseHTTPRequestHandler):
            response.get('role') == 'replica' and response.get('state') == 'running' else 503

        if not cluster and patroni.ha.is_paused():
-            leader_status_code = 200 if response.get('role') in ('master', 'standby_leader') else 503
-            primary_status_code = 200 if response.get('role') == 'master' else 503
+            leader_status_code = 200 if response.get('role') in ('master', 'primary', 'standby_leader') else 503
+            primary_status_code = 200 if response.get('role') in ('master', 'primary') else 503
            standby_leader_status_code = 200 if response.get('role') == 'standby_leader' else 503
        elif patroni.ha.is_leader():
            leader_status_code = 200
@@ -191,7 +191,7 @@ class RestApiHandler(BaseHTTPRequestHandler):

    def do_GET_liveness(self):
        patroni = self.server.patroni
-        is_primary = patroni.postgresql.role == 'master' and patroni.postgresql.is_running()
+        is_primary = patroni.postgresql.role in ('master', 'primary') and patroni.postgresql.is_running()
        # We can tolerate Patroni problems longer on the replica.
        # On the primary the liveness probe most likely will start failing only after the leader key expired.
        # It should not be a big problem because replicas will see that the primary is still alive via REST API call.
@@ -255,7 +255,11 @@ class RestApiHandler(BaseHTTPRequestHandler):

        metrics.append("# HELP patroni_master Value is 1 if this node is the leader, 0 otherwise.")
        metrics.append("# TYPE patroni_master gauge")
-        metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] == 'master')))
+        metrics.append("patroni_master{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))
+
+        metrics.append("# HELP patroni_primary Value is 1 if this node is the leader, 0 otherwise.")
+        metrics.append("# TYPE patroni_primary gauge")
+        metrics.append("patroni_primary{0} {1}".format(scope_label, int(postgres['role'] in ('master', 'primary'))))

        metrics.append("# HELP patroni_xlog_location Current location of the Postgres"
                       " transaction log, 0 if this node is not the leader.")
@@ -302,6 +306,11 @@ class RestApiHandler(BaseHTTPRequestHandler):
        metrics.append("# TYPE patroni_cluster_unlocked gauge")
        metrics.append("patroni_cluster_unlocked{0} {1}".format(scope_label, int(postgres.get('cluster_unlocked', 0))))

+        metrics.append("# HELP patroni_failsafe_mode_is_active Value is 1 if the cluster is unlocked, 0 if locked.")
+        metrics.append("# TYPE patroni_failsafe_mode_is_active gauge")
+        metrics.append("patroni_failsafe_mode_is_active{0} {1}"
+                       .format(scope_label, int(postgres.get('failsafe_mode_is_active', 0))))
+
        metrics.append("# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.")
        metrics.append("# TYPE patroni_postgres_timeline counter")
        metrics.append("patroni_postgres_timeline{0} {1}".format(scope_label, postgres.get('timeline', 0)))
@@ -368,6 +377,24 @@ class RestApiHandler(BaseHTTPRequestHandler):
        self.server.patroni.sighup_handler()
        self._write_response(202, 'reload scheduled')

+    def do_GET_failsafe(self):
+        failsafe = self.server.patroni.dcs.failsafe
+        if isinstance(failsafe, dict):
+            self._write_json_response(200, failsafe)
+        else:
+            self.send_error(502)
+
+    @check_access
+    def do_POST_failsafe(self):
+        if self.server.patroni.ha.is_failsafe_mode():
+            request = self._read_json_content()
+            if request:
+                message = self.server.patroni.ha.update_failsafe(request) or 'Accepted'
+                code = 200 if message == 'Accepted' else 500
+                self._write_response(code, message)
+        else:
+            self.send_error(502)
+
    @check_access
    def do_POST_sigterm(self):
        """Only for behave testing on windows"""
@@ -420,9 +447,9 @@ class RestApiHandler(BaseHTTPRequestHandler):
                    status_code = _
                    break
            elif k == 'role':
-                if request[k] not in ('master', 'replica'):
+                if request[k] not in ('master', 'primary', 'replica'):
                    status_code = 400
-                    data = "PostgreSQL role should be either master or replica"
+                    data = "PostgreSQL role should be either primary or replica"
                    break
            elif k == 'postgres_version':
                try:
@@ -592,6 +619,18 @@ class RestApiHandler(BaseHTTPRequestHandler):
    def do_POST_switchover(self):
        self.do_POST_failover(action='switchover')

+    @check_access
+    def do_POST_citus(self):
+        request = self._read_json_content()
+        if not request:
+            return
+
+        patroni = self.server.patroni
+        if patroni.postgresql.citus_handler.is_coordinator() and patroni.ha.is_leader():
+            cluster = patroni.dcs.get_cluster(True)
+            patroni.postgresql.citus_handler.handle_event(cluster, request)
+        self._write_response(200, 'OK')
+
    def parse_request(self):
        """Override parse_request method to enrich basic functionality of `BaseHTTPRequestHandler` class

@@ -670,6 +709,8 @@ class RestApiHandler(BaseHTTPRequestHandler):

        if not cluster or cluster.is_unlocked():
            result['cluster_unlocked'] = True
+        if self.server.patroni.ha.failsafe_is_active():
+            result['failsafe_mode_is_active'] = True
        result['dcs_last_seen'] = self.server.patroni.dcs.last_seen
        return result

@@ -734,16 +775,17 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
    def __members_ips(self):
        cluster = self.patroni.dcs.cluster
        if self.__allowlist_include_members and cluster:
-            for member in cluster.members:
-                if member.api_url:
-                    try:
-                        r = urlparse(member.api_url)
-                        host = r.hostname
-                        port = r.port or (443 if r.scheme == 'https' else 80)
-                        for ip in self.__resolve_ips(host, port):
-                            yield ip
-                    except Exception as e:
-                        logger.debug('Failed to parse url %s: %r', member.api_url, e)
+            for cluster in [cluster] + list(cluster.workers.values()):
+                for member in cluster.members:
+                    if member.api_url:
+                        try:
+                            r = urlparse(member.api_url)
+                            host = r.hostname
+                            port = r.port or (443 if r.scheme == 'https' else 80)
+                            for ip in self.__resolve_ips(host, port):
+                                yield ip
+                        except Exception as e:
+                            logger.debug('Failed to parse url %s: %r', member.api_url, e)

    def check_access(self, rh):
        if self.__allowlist or self.__allowlist_include_members:
@@ -2,6 +2,7 @@ import json
 import logging
 import os
 import shutil
+import six
 import tempfile
 import yaml

@@ -58,16 +59,21 @@ class Config(object):
    PATRONI_CONFIG_VARIABLE = PATRONI_ENV_PREFIX + 'CONFIGURATION'

    __CACHE_FILENAME = 'patroni.dynamic.json'
+    __REMAP_KEYS = {
+        'master_start_timeout': 'primary_start_timeout',
+        'master_stop_timeout': 'primary_stop_timeout'
+    }
    __DEFAULT_CONFIG = {
        'ttl': 30, 'loop_wait': 10, 'retry_timeout': 10,
        'maximum_lag_on_failover': 1048576,
        'maximum_lag_on_syncnode': -1,
        'check_timeline': False,
-        'master_start_timeout': 300,
-        'master_stop_timeout': 0,
+        'primary_start_timeout': 300,
+        'primary_stop_timeout': 0,
        'synchronous_mode': False,
        'synchronous_mode_strict': False,
        'synchronous_node_count': 1,
+        'failsafe_mode': False,
        'standby_cluster': {
            'create_replica_methods': '',
            'host': '',
@@ -223,6 +229,9 @@ class Config(object):
        config = deepcopy(self.__DEFAULT_CONFIG)

        for name, value in dynamic_configuration.items():
+            # allow copying master_start_timeout->primary_start_timeout when the latter isn't in dynamic_configuration
+            if name in self.__REMAP_KEYS and self.__REMAP_KEYS[name] not in dynamic_configuration:
+                name = self.__REMAP_KEYS[name]
            if name == 'postgresql':
                for name, value in (value or {}).items():
                    if name == 'parameters':
@@ -235,7 +244,7 @@ class Config(object):
                    if name in self.__DEFAULT_CONFIG['standby_cluster']:
                        config['standby_cluster'][name] = deepcopy(value)
            elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overridden from DCS
-                if name in ('synchronous_mode', 'synchronous_mode_strict'):
+                if name in ('synchronous_mode', 'synchronous_mode_strict', 'failsafe_mode'):
                    config[name] = value
                else:
                    config[name] = int(value)
@@ -353,18 +362,24 @@ class Config(object):
                if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'SRV_SUFFIX', 'URL', 'PROXY',
                              'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY',
                              'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'SERVICE_CHECK_TLS_SERVER_NAME',
-                              'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP',
-                              'PORTS', 'LABELS', 'BYPASS_API_SERVICE', 'KEY_PASSWORD', 'USE_SSL', 'SET_ACLS') and name:
+                              'SERVICE_TAGS', 'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL',
+                              'POD_IP', 'PORTS', 'LABELS', 'BYPASS_API_SERVICE', 'KEY_PASSWORD', 'USE_SSL', 'SET_ACLS',
+                              'GROUP', 'DATABASE') and name:
                    value = os.environ.pop(param)
-                    if suffix == 'PORT':
+                    if name == 'CITUS':
+                        if suffix == 'GROUP':
+                            value = parse_int(value)
+                        elif suffix != 'DATABASE':
+                            continue
+                    elif suffix == 'PORT':
                        value = value and parse_int(value)
-                    elif suffix in ('HOSTS', 'PORTS', 'CHECKS'):
+                    elif suffix in ('HOSTS', 'PORTS', 'CHECKS', 'SERVICE_TAGS'):
                        value = value and _parse_list(value)
                    elif suffix in ('LABELS', 'SET_ACLS'):
                        value = _parse_dict(value)
                    elif suffix in ('USE_PROXIES', 'REGISTER_SERVICE', 'USE_ENDPOINTS', 'BYPASS_API_SERVICE', 'VERIFY'):
                        value = parse_bool(value)
-                    if value:
+                    if value is not None:
                        ret[name.lower()][suffix.lower()] = value
        for dcs in ('etcd', 'etcd3'):
            if dcs in ret:
@@ -392,7 +407,11 @@ class Config(object):
    def _build_effective_configuration(self, dynamic_configuration, local_configuration):
        config = self._safe_copy_dynamic_configuration(dynamic_configuration)
        for name, value in local_configuration.items():
-            if name == 'postgresql':
+            if name == 'citus':  # remove invalid citus configuration
+                if isinstance(value, dict) and isinstance(value.get('group'), six.integer_types)\
+                        and isinstance(value.get('database'), six.string_types):
+                    config[name] = value
+            elif name == 'postgresql':
                for name, value in (value or {}).items():
                    if name == 'parameters':
                        config['postgresql'][name].update(self._process_postgresql_parameters(value, True))
@@ -430,6 +449,12 @@ class Config(object):
        if 'name' not in config and 'name' in pg_config:
            config['name'] = pg_config['name']

+        # when bootstrapping the new Citus cluster (coordinator/worker) enable sync replication in global configuration
+        if 'citus' in config:
+            bootstrap = config.setdefault('bootstrap', {})
+            dcs = bootstrap.setdefault('dcs', {})
+            dcs.setdefault('synchronous_mode', True)
+
        updated_fields = (
            'name',
            'scope',
@@ -437,7 +462,8 @@ class Config(object):
            'synchronous_mode',
            'synchronous_mode_strict',
            'synchronous_node_count',
-            'maximum_lag_on_syncnode'
+            'maximum_lag_on_syncnode',
+            'citus'
        )

        pg_config.update({p: config[p] for p in updated_fields if p in config})
@@ -34,9 +34,8 @@ except ImportError:  # pragma: no cover

 from .dcs import get_dcs as _get_dcs
 from .exceptions import PatroniException
-from .postgresql import Postgresql
 from .postgresql.misc import postgres_version_to_int
-from .utils import cluster_as_json, find_executable, patch_config, polling_loop
+from .utils import cluster_as_json, find_executable, patch_config, polling_loop, is_standby_cluster
 from .request import PatroniRequest
 from .version import __version__

@@ -137,13 +136,17 @@ option_watch = click.option('-W', is_flag=True, help='Auto update the screen eve
 option_force = click.option('--force', is_flag=True, help='Do not ask for confirmation at any point')
 arg_cluster_name = click.argument('cluster_name', required=False,
                                  default=lambda: click.get_current_context().obj.get('scope'))
+option_default_citus_group = click.option('--group', required=False, type=int, help='Citus group',
+                                          default=lambda: click.get_current_context().obj.get('citus', {}).get('group'))
+option_citus_group = click.option('--group', required=False, type=int, help='Citus group')
 option_insecure = click.option('-k', '--insecure', is_flag=True, help='Allow connections to SSL sites without certs')
+role_choice = click.Choice(['leader', 'primary', 'standby-leader', 'replica', 'standby', 'any', 'master'])


@click.group()
@click.option('--config-file', '-c', help='Configuration file',
              envvar='PATRONICTL_CONFIG_FILE', default=CONFIG_FILE_PATH)
-@click.option('--dcs-url', '--dcs', '-d', help='The DCS connect url', envvar='DCS_URL')
+@click.option('--dcs-url', '--dcs', '-d', 'dcs_url', help='The DCS connect url', envvar='DCS_URL')
@option_insecure
@click.pass_context
 def ctl(ctx, config_file, dcs_url, insecure):
@@ -157,11 +160,16 @@ def ctl(ctx, config_file, dcs_url, insecure):
    ctx.obj.setdefault('ctl', {})['insecure'] = ctx.obj.get('ctl', {}).get('insecure') or insecure


-def get_dcs(config, scope):
+def get_dcs(config, scope, group):
    config.update({'scope': scope, 'patronictl': True})
+    if group is not None:
+        config['citus'] = {'group': group}
    config.setdefault('name', scope)
    try:
-        return _get_dcs(config)
+        dcs = _get_dcs(config)
+        if config.get('citus') and group is None:
+            dcs.get_cluster = dcs._get_citus_cluster
+        return dcs
    except PatroniException as e:
        raise PatroniCtlException(str(e))

@@ -186,9 +194,10 @@ def print_output(columns, rows, alignment=None, fmt='pretty', header=None, delim
            for row in rows:
                if row[i]:
                    row[i] = format_config_for_editing(row[i], fmt != 'pretty').strip()
-        if list_cluster and fmt != 'tsv':  # skip cluster name if pretty-printing
-            columns = columns[1:] if columns else []
-            rows = [row[1:] for row in rows]
+        if list_cluster and fmt != 'tsv':  # skip cluster name and maybe Citus group if pretty-printing
+            skip_cols = 2 if ' (group: ' in header else 1
+            columns = columns[skip_cols:] if columns else []
+            rows = [row[skip_cols:] for row in rows]

        if fmt == 'tsv':
            for r in ([columns] if columns else []) + rows:
@@ -232,21 +241,29 @@ def watching(w, watch, max_count=None, clear=True):
        yield 0


-def get_all_members(cluster, role='master'):
-    if role == 'master':
-        if cluster.leader is not None and cluster.leader.name:
-            yield cluster.leader
+def get_all_members(obj, cluster, group, role='leader'):
+    clusters = {0: cluster}
+    if obj.get('citus') and group is None:
+        clusters.update(cluster.workers)
+    if role in ('leader', 'master', 'primary', 'standby-leader'):
+        role = {'primary': 'master', 'standby-leader': 'standby_leader'}.get(role, role)
+        for cluster in clusters.values():
+            if cluster.leader is not None and cluster.leader.name and\
+                    (role == 'leader' or
+                     cluster.leader.data.get('role') != 'master' and role == 'standby_leader' or
+                     cluster.leader.data.get('role') != 'standby_leader' and role == 'master'):
+                yield cluster.leader.member
        return

-    leader_name = (cluster.leader.member.name if cluster.leader else None)
-    for m in cluster.members:
-        if role == 'any' or role == 'replica' and m.name != leader_name:
-            yield m
+    for cluster in clusters.values():
+        leader_name = (cluster.leader.member.name if cluster.leader else None)
+        for m in cluster.members:
+            if role == 'any' or role in ('replica', 'standby') and m.name != leader_name:
+                yield m


-def get_any_member(cluster, role='master', member=None):
-    members = get_all_members(cluster, role)
-    for m in members:
+def get_any_member(obj, cluster, group, role='leader', member=None):
+    for m in get_all_members(obj, cluster, group, role):
        if member is None or m.name == member:
            return m

@@ -260,8 +277,8 @@ def get_all_members_leader_first(cluster):
            yield member


-def get_cursor(cluster, connect_parameters, role='master', member=None):
-    member = get_any_member(cluster, role=role, member=member)
+def get_cursor(obj, cluster, group, connect_parameters, role='leader', member=None):
+    member = get_any_member(obj, cluster, group, role=role, member=member)
    if member is None:
        return None

@@ -275,13 +292,14 @@ def get_cursor(cluster, connect_parameters, role='master', member=None):
    from . import psycopg
    conn = psycopg.connect(**params)
    cursor = conn.cursor()
-    if role == 'any':
+    if role in ('any', 'leader'):
        return cursor

    cursor.execute('SELECT pg_catalog.pg_is_in_recovery()')
    in_recovery = cursor.fetchone()[0]

-    if in_recovery and role == 'replica' or not in_recovery and role == 'master':
+    if in_recovery and role in ('replica', 'standby', 'standby-leader')\
+            or not in_recovery and role in ('master', 'primary'):
        return cursor

    conn.close()
@@ -289,32 +307,31 @@ def get_cursor(cluster, connect_parameters, role='master', member=None):
    return None


-def get_members(cluster, cluster_name, member_names, role, force, action, ask_confirmation=True):
-    candidates = {m.name: m for m in cluster.members}
+def get_members(obj, cluster, cluster_name, member_names, role, force, action, ask_confirmation=True, group=None):
+    members = list(get_all_members(obj, cluster, group, role))

+    candidates = {m.name for m in members}
    if not force or role:
        if not member_names and not candidates:
            raise PatroniCtlException('{0} cluster doesn\'t have any members'.format(cluster_name))
-        output_members(cluster, cluster_name)
+        output_members(obj, cluster, cluster_name, group=group)

-    if role:
-        role_names = [m.name for m in get_all_members(cluster, role)]
-        if member_names:
-            member_names = list(set(member_names) & set(role_names))
-            if not member_names:
-                raise PatroniCtlException('No {0} among provided members'.format(role))
-        else:
-            member_names = role_names
+    if member_names:
+        member_names = list(set(member_names) & candidates)
+        if not member_names:
+            raise PatroniCtlException('No {0} among provided members'.format(role))
+    elif action != 'reinitialize':
+        member_names = list(candidates)

    if not member_names and not force:
        member_names = [click.prompt('Which member do you want to {0} [{1}]?'.format(action,
-                        ', '.join(candidates.keys())), type=str, default='')]
+                        ', '.join(candidates)), type=str, default='')]

    for member_name in member_names:
        if member_name not in candidates:
            raise PatroniCtlException('{0} is not a member of cluster'.format(member_name))

-    members = [candidates[n] for n in member_names]
+    members = [m for m in members if m.name in member_names]
    if ask_confirmation:
        confirm_members_action(members, force, action)
    return members
@@ -335,20 +352,22 @@ def confirm_members_action(members, force, action, scheduled_at=None):
                raise PatroniCtlException('Aborted {0}'.format(action))


-@ctl.command('dsn', help='Generate a dsn for the provided member, defaults to a dsn of the master')
-@click.option('--role', '-r', help='Give a dsn of any member with this role', type=click.Choice(['master', 'replica',
-              'any']), default=None)
+@ctl.command('dsn', help='Generate a dsn for the provided member, defaults to a dsn of the leader')
+@click.option('--role', '-r', help='Give a dsn of any member with this role', type=role_choice, default=None)
@click.option('--member', '-m', help='Generate a dsn for this member', type=str)
@arg_cluster_name
+@option_citus_group
@click.pass_obj
-def dsn(obj, cluster_name, role, member):
-    if role is not None and member is not None:
-        raise PatroniCtlException('--role and --member are mutually exclusive options')
+def dsn(obj, cluster_name, group, role, member):
+    if member is not None:
+        if role is not None:
+            raise PatroniCtlException('--role and --member are mutually exclusive options')
+        role = 'any'
    if member is None and role is None:
-        role = 'master'
+        role = 'leader'

-    cluster = get_dcs(obj, cluster_name).get_cluster()
-    m = get_any_member(cluster, role=role, member=member)
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()
+    m = get_any_member(obj, cluster, group, role=role, member=member)
    if m is None:
        raise PatroniCtlException('Can not find a suitable member')

@@ -358,14 +377,14 @@ def dsn(obj, cluster_name, role, member):

@ctl.command('query', help='Query a Patroni PostgreSQL member')
@arg_cluster_name
+@option_citus_group
@click.option('--format', 'fmt', help='Output format (pretty, tsv, json, yaml)', default='tsv')
@click.option('--file', '-f', 'p_file', help='Execute the SQL commands from this file', type=click.File('rb'))
@click.option('--password', help='force password prompt', is_flag=True)
@click.option('-U', '--username', help='database user name', type=str)
@option_watch
@option_watchrefresh
-@click.option('--role', '-r', help='The role of the query', type=click.Choice(['master', 'replica', 'any']),
-              default=None)
+@click.option('--role', '-r', help='The role of the query', type=role_choice, default=None)
@click.option('--member', '-m', help='Query a specific member', type=str)
@click.option('--delimiter', help='The column delimiter', default='\t')
@click.option('--command', '-c', help='The SQL commands to execute')
@@ -374,6 +393,7 @@ def dsn(obj, cluster_name, role, member):
 def query(
    obj,
    cluster_name,
+    group,
    role,
    member,
    w,
@@ -386,10 +406,12 @@ def query(
    dbname,
    fmt='tsv',
 ):
-    if role is not None and member is not None:
-        raise PatroniCtlException('--role and --member are mutually exclusive options')
+    if member is not None:
+        if role is not None:
+            raise PatroniCtlException('--role and --member are mutually exclusive options')
+        role = 'any'
    if member is None and role is None:
-        role = 'master'
+        role = 'leader'

    if p_file is not None and command is not None:
        raise PatroniCtlException('--file and --command are mutually exclusive options')
@@ -408,25 +430,25 @@ def query(
    if p_file is not None:
        command = p_file.read()

-    dcs = get_dcs(obj, cluster_name)
+    dcs = get_dcs(obj, cluster_name, group)

    cursor = None
    for _ in watching(w, watch, clear=False):
        if cursor is None:
            cluster = dcs.get_cluster()

-        output, header = query_member(cluster, cursor, member, role, command, connect_parameters)
+        output, header = query_member(obj, cluster, group, cursor, member, role, command, connect_parameters)
        print_output(header, output, fmt=fmt, delimiter=delimiter)


-def query_member(cluster, cursor, member, role, command, connect_parameters):
+def query_member(obj, cluster, group, cursor, member, role, command, connect_parameters):
    from . import psycopg
    try:
        if cursor is None:
-            cursor = get_cursor(cluster, connect_parameters, role=role, member=member)
+            cursor = get_cursor(obj, cluster, group, connect_parameters, role=role, member=member)

        if cursor is None:
-            if role is None:
+            if member is not None:
                message = 'No connection to member {0} is available'.format(member)
            else:
                message = 'No connection to role={0} is available'.format(role)
@@ -446,13 +468,16 @@ def query_member(cluster, cursor, member, role, command, connect_parameters):

@ctl.command('remove', help='Remove cluster from DCS')
@click.argument('cluster_name')
+@option_citus_group
@option_format
@click.pass_obj
-def remove(obj, cluster_name, fmt):
-    dcs = get_dcs(obj, cluster_name)
+def remove(obj, cluster_name, group, fmt):
+    dcs = get_dcs(obj, cluster_name, group)
    cluster = dcs.get_cluster()

-    output_members(cluster, cluster_name, fmt=fmt)
+    if obj.get('citus') and group is None:
+        raise PatroniCtlException('For Citus clusters the --group must me specified')
+    output_members(obj, cluster, cluster_name, fmt=fmt)

    confirm = click.prompt('Please confirm the cluster name to remove', type=str)
    if confirm != cluster_name:
@@ -466,9 +491,9 @@ def remove(obj, cluster_name, fmt):
        raise PatroniCtlException('You did not exactly type "{0}"'.format(message))

    if cluster.leader and cluster.leader.name:
-        confirm = click.prompt('This cluster currently is healthy. Please specify the master name to continue')
+        confirm = click.prompt('This cluster currently is healthy. Please specify the leader name to continue')
        if confirm != cluster.leader.name:
-            raise PatroniCtlException('You did not specify the current master of the cluster')
+            raise PatroniCtlException('You did not specify the current leader of the cluster')

    dcs.delete_cluster()

@@ -501,14 +526,14 @@ def parse_scheduled(scheduled):
@ctl.command('reload', help='Reload cluster member configuration')
@click.argument('cluster_name')
@click.argument('member_names', nargs=-1)
-@click.option('--role', '-r', help='Reload only members with this role', default='any',
-              type=click.Choice(['master', 'replica', 'any']))
+@option_citus_group
+@click.option('--role', '-r', help='Reload only members with this role', type=role_choice, default='any')
@option_force
@click.pass_obj
-def reload(obj, cluster_name, member_names, force, role):
-    cluster = get_dcs(obj, cluster_name).get_cluster()
+def reload(obj, cluster_name, member_names, group, force, role):
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()

-    members = get_members(cluster, cluster_name, member_names, role, force, 'reload')
+    members = get_members(obj, cluster, cluster_name, member_names, role, force, 'reload', group=group)

    for member in members:
        r = request_patroni(member, 'post', 'reload')
@@ -527,8 +552,8 @@ def reload(obj, cluster_name, member_names, force, role):
@ctl.command('restart', help='Restart cluster member')
@click.argument('cluster_name')
@click.argument('member_names', nargs=-1)
-@click.option('--role', '-r', help='Restart only members with this role', default='any',
-              type=click.Choice(['master', 'replica', 'any']))
+@option_citus_group
+@click.option('--role', '-r', help='Restart only members with this role', type=role_choice, default='any')
@click.option('--any', 'p_any', help='Restart a single member only', is_flag=True)
@click.option('--scheduled', help='Timestamp of a scheduled restart in unambiguous format (e.g. ISO 8601)',
              default=None)
@@ -539,10 +564,10 @@ def reload(obj, cluster_name, member_names, force, role):
              help='Return error and fail over if necessary when restarting takes longer than this.')
@option_force
@click.pass_obj
-def restart(obj, cluster_name, member_names, force, role, p_any, scheduled, version, pending, timeout):
-    cluster = get_dcs(obj, cluster_name).get_cluster()
+def restart(obj, cluster_name, group, member_names, force, role, p_any, scheduled, version, pending, timeout):
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()

-    members = get_members(cluster, cluster_name, member_names, role, force, 'restart', False)
+    members = get_members(obj, cluster, cluster_name, member_names, role, force, 'restart', False, group=group)
    if scheduled is None and not force:
        next_hour = (datetime.datetime.now() + datetime.timedelta(hours=1)).strftime('%Y-%m-%dT%H:%M')
        scheduled = click.prompt('When should the restart take place (e.g. ' + next_hour + ') ',
@@ -600,13 +625,14 @@ def restart(obj, cluster_name, member_names, force, role, p_any, scheduled, vers

@ctl.command('reinit', help='Reinitialize cluster member')
@click.argument('cluster_name')
+@option_citus_group
@click.argument('member_names', nargs=-1)
@option_force
@click.option('--wait', help='Wait until reinitialization completes', is_flag=True)
@click.pass_obj
-def reinit(obj, cluster_name, member_names, force, wait):
-    cluster = get_dcs(obj, cluster_name).get_cluster()
-    members = get_members(cluster, cluster_name, member_names, None, force, 'reinitialize')
+def reinit(obj, cluster_name, group, member_names, force, wait):
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()
+    members = get_members(obj, cluster, cluster_name, member_names, 'replica', force, 'reinitialize', group=group)

    wait_on_members = []
    for member in members:
@@ -637,31 +663,42 @@ def reinit(obj, cluster_name, member_names, force, wait):
                wait_on_members.remove(member)


-def _do_failover_or_switchover(obj, action, cluster_name, master, candidate, force, scheduled=None):
+def _do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force, scheduled=None):
    """
        We want to trigger a failover or switchover for the specified cluster name.

-        We verify that the cluster name, master name and candidate name are correct.
+        We verify that the cluster name, leader name and candidate name are correct.
        If so, we trigger an action and keep the client up to date.
    """

-    dcs = get_dcs(obj, cluster_name)
+    dcs = get_dcs(obj, cluster_name, group)
    cluster = dcs.get_cluster()
+    click.echo('Current cluster topology')
+    output_members(obj, cluster, cluster_name, group=group)
+
+    if obj.get('citus') and group is None:
+        if force:
+            raise PatroniCtlException('For Citus clusters the --group must me specified')
+        else:
+            group = click.prompt('Citus group', type=int)
+            dcs = get_dcs(obj, cluster_name, group)
+            cluster = dcs.get_cluster()

    if action == 'switchover' and (cluster.leader is None or not cluster.leader.name):
-        raise PatroniCtlException('This cluster has no master')
+        raise PatroniCtlException('This cluster has no leader')

-    if master is None:
+    if leader is None:
        if force or action == 'failover':
-            master = cluster.leader and cluster.leader.name
+            leader = cluster.leader and cluster.leader.name
        else:
-            master = click.prompt('Master', type=str, default=cluster.leader.member.name)
+            prompt = 'Standby Leader' if is_standby_cluster(cluster.config) else 'Primary'
+            leader = click.prompt(prompt, type=str, default=cluster.leader.member.name)

-    if master is not None and cluster.leader and cluster.leader.member.name != master:
-        raise PatroniCtlException('Member {0} is not the leader of cluster {1}'.format(master, cluster_name))
+    if leader is not None and cluster.leader and cluster.leader.member.name != leader:
+        raise PatroniCtlException('Member {0} is not the leader of cluster {1}'.format(leader, cluster_name))

    # excluding members with nofailover tag
-    candidate_names = [str(m.name) for m in cluster.members if m.name != master and not m.nofailover]
+    candidate_names = [str(m.name) for m in cluster.members if m.name != leader and not m.nofailover]
    # We sort the names for consistent output to the client
    candidate_names.sort()

@@ -674,7 +711,7 @@ def _do_failover_or_switchover(obj, action, cluster_name, master, candidate, for
    if action == 'failover' and not candidate:
        raise PatroniCtlException('Failover could be performed only to a specific candidate')

-    if candidate == master:
+    if candidate == leader:
        raise PatroniCtlException(action.title() + ' target and source are the same.')

    if candidate and candidate not in candidate_names:
@@ -695,16 +732,13 @@ def _do_failover_or_switchover(obj, action, cluster_name, master, candidate, for
                raise PatroniCtlException("Can't schedule switchover in the paused state")
            scheduled_at_str = scheduled_at.isoformat()

-    failover_value = {'leader': master, 'candidate': candidate, 'scheduled_at': scheduled_at_str}
+    failover_value = {'leader': leader, 'candidate': candidate, 'scheduled_at': scheduled_at_str}

    logging.debug(failover_value)

    # By now we have established that the leader exists and the candidate exists
-    click.echo('Current cluster topology')
-    output_members(dcs.get_cluster(), cluster_name)
-
    if not force:
-        demote_msg = ', demoting current master ' + master if master else ''
+        demote_msg = ', demoting current leader ' + leader if leader else ''
        if scheduled_at_str:
            if not click.confirm('Are you sure you want to schedule {0} of cluster {1} at {2}{3}?'
                                 .format(action, cluster_name, scheduled_at_str, demote_msg)):
@@ -736,32 +770,34 @@ def _do_failover_or_switchover(obj, action, cluster_name, master, candidate, for
        logging.exception(r)
        logging.warning('Failing over to DCS')
        click.echo('{0} Could not {1} using Patroni api, falling back to DCS'.format(timestamp(), action))
-        dcs.manual_failover(master, candidate, scheduled_at=scheduled_at)
+        dcs.manual_failover(leader, candidate, scheduled_at=scheduled_at)

-    output_members(cluster, cluster_name)
+    output_members(obj, cluster, cluster_name, group=group)


@ctl.command('failover', help='Failover to a replica')
@arg_cluster_name
-@click.option('--master', help='The name of the current master', default=None)
+@option_citus_group
+@click.option('--leader', '--primary', '--master', 'leader', help='The name of the current leader', default=None)
@click.option('--candidate', help='The name of the candidate', default=None)
@option_force
@click.pass_obj
-def failover(obj, cluster_name, master, candidate, force):
-    action = 'switchover' if master else 'failover'
-    _do_failover_or_switchover(obj, action, cluster_name, master, candidate, force)
+def failover(obj, cluster_name, group, leader, candidate, force):
+    action = 'switchover' if leader else 'failover'
+    _do_failover_or_switchover(obj, action, cluster_name, group, leader, candidate, force)


@ctl.command('switchover', help='Switchover to a replica')
@arg_cluster_name
-@click.option('--master', help='The name of the current master', default=None)
+@option_citus_group
+@click.option('--leader', '--primary', '--master', 'leader', help='The name of the current leader', default=None)
@click.option('--candidate', help='The name of the candidate', default=None)
@click.option('--scheduled', help='Timestamp of a scheduled switchover in unambiguous format (e.g. ISO 8601)',
              default=None)
@option_force
@click.pass_obj
-def switchover(obj, cluster_name, master, candidate, force, scheduled):
-    _do_failover_or_switchover(obj, 'switchover', cluster_name, master, candidate, force, scheduled)
+def switchover(obj, cluster_name, group, leader, candidate, force, scheduled):
+    _do_failover_or_switchover(obj, 'switchover', cluster_name, group, leader, candidate, force, scheduled)


 def generate_topology(level, member, topology):
@@ -791,48 +827,7 @@ def topology_sort(members):
        yield member


-def output_members(cluster, name, extended=False, fmt='pretty'):
-    rows = []
-    logging.debug(cluster)
-    initialize = {None: 'uninitialized', '': 'initializing'}.get(cluster.initialize, cluster.initialize)
-    cluster = cluster_as_json(cluster)
-
-    columns = ['Cluster', 'Member', 'Host', 'Role', 'State', 'TL', 'Lag in MB']
-    for c in ('Pending restart', 'Scheduled restart', 'Tags'):
-        if extended or any(m.get(c.lower().replace(' ', '_')) for m in cluster['members']):
-            columns.append(c)
-
-    # Show Host as 'host:port' if somebody is running on non-standard port or two nodes are running on the same host
-    members = [m for m in cluster['members'] if 'host' in m]
-    append_port = any('port' in m and m['port'] != 5432 for m in members) or\
-        len(set(m['host'] for m in members)) < len(members)
-
-    sort = topology_sort if fmt == 'topology' else iter
-    for m in sort(cluster['members']):
-        logging.debug(m)
-
-        lag = m.get('lag', '')
-        m.update(cluster=name, member=m['name'], host=m.get('host', ''), tl=m.get('timeline', ''),
-                 role=m['role'].replace('_', ' ').title(),
-                 lag_in_mb=round(lag/1024/1024) if isinstance(lag, six.integer_types) else lag,
-                 pending_restart='*' if m.get('pending_restart') else '')
-
-        if append_port and m['host'] and m.get('port'):
-            m['host'] = ':'.join([m['host'], str(m['port'])])
-
-        if 'scheduled_restart' in m:
-            value = m['scheduled_restart']['schedule']
-            if 'postgres_version' in m['scheduled_restart']:
-                value += ' if version < {0}'.format(m['scheduled_restart']['postgres_version'])
-            m['scheduled_restart'] = value
-
-        rows.append([m.get(n.lower().replace(' ', '_'), '') for n in columns])
-
-    print_output(columns, rows, {'Lag in MB': 'r', 'TL': 'r'}, fmt, ' Cluster: {0} ({1}) '.format(name, initialize))
-
-    if fmt not in ('pretty', 'topology'):  # Omit service info when using machine-readable formats
-        return
-
+def get_cluster_service_info(cluster):
    service_info = []
    if cluster.get('pause'):
        service_info.append('Maintenance mode: on')
@@ -843,44 +838,109 @@ def output_members(cluster, name, extended=False, fmt='pretty'):
            if name in cluster['scheduled_switchover']:
                info += '\n{0:>24}: {1}'.format(name, cluster['scheduled_switchover'][name])
        service_info.append(info)
+    return service_info

-    if service_info:
-        click.echo(' ' + '\n '.join(service_info))
+
+def output_members(obj, cluster, name, extended=False, fmt='pretty', group=None):
+    rows = []
+    logging.debug(cluster)
+
+    initialize = {None: 'uninitialized', '': 'initializing'}.get(cluster.initialize, cluster.initialize)
+    columns = ['Cluster', 'Member', 'Host', 'Role', 'State', 'TL', 'Lag in MB']
+
+    clusters = {group or 0: cluster_as_json(cluster)}
+
+    is_citus_cluster = obj.get('citus')
+    if is_citus_cluster:
+        columns.insert(1, 'Group')
+        if group is None:
+            clusters.update({g: cluster_as_json(c) for g, c in cluster.workers.items()})
+
+    all_members = [m for c in clusters.values() for m in c['members'] if 'host' in m]
+
+    for c in ('Pending restart', 'Scheduled restart', 'Tags'):
+        if extended or any(m.get(c.lower().replace(' ', '_')) for m in all_members):
+            columns.append(c)
+
+    # Show Host as 'host:port' if somebody is running on non-standard port or two nodes are running on the same host
+    append_port = any('port' in m and m['port'] != 5432 for m in all_members) or\
+        len(set(m['host'] for m in all_members)) < len(all_members)
+
+    sort = topology_sort if fmt == 'topology' else iter
+    for g, cluster in sorted(clusters.items()):
+        for member in sort(cluster['members']):
+            logging.debug(member)
+
+            lag = member.get('lag', '')
+            member.update(cluster=name, member=member['name'], group=g,
+                          host=member.get('host', ''), tl=member.get('timeline', ''),
+                          role=member['role'].replace('_', ' ').title(),
+                          lag_in_mb=round(lag/1024/1024) if isinstance(lag, six.integer_types) else lag,
+                          pending_restart='*' if member.get('pending_restart') else '')
+
+            if append_port and member['host'] and member.get('port'):
+                member['host'] = ':'.join([member['host'], str(member['port'])])
+
+            if 'scheduled_restart' in member:
+                value = member['scheduled_restart']['schedule']
+                if 'postgres_version' in member['scheduled_restart']:
+                    value += ' if version < {0}'.format(member['scheduled_restart']['postgres_version'])
+                member['scheduled_restart'] = value
+
+            rows.append([member.get(n.lower().replace(' ', '_'), '') for n in columns])
+
+    title = 'Citus cluster' if is_citus_cluster else 'Cluster'
+    group_title = '' if group is None else 'group: {0}, '.format(group)
+    title_details = group_title and ' ({0}{1})'.format(group_title, initialize)
+    title = ' {0}: {1}{2} '.format(title, name, title_details)
+    print_output(columns, rows, {'Group': 'r', 'Lag in MB': 'r', 'TL': 'r'}, fmt, title)
+
+    if fmt not in ('pretty', 'topology'):  # Omit service info when using machine-readable formats
+        return
+
+    for g, cluster in sorted(clusters.items()):
+        service_info = get_cluster_service_info(cluster)
+        if service_info:
+            if is_citus_cluster and group is None:
+                click.echo('Citus group: {0}'.format(g))
+            click.echo(' ' + '\n '.join(service_info))


@ctl.command('list', help='List the Patroni members for a given Patroni')
@click.argument('cluster_names', nargs=-1)
+@option_citus_group
@click.option('--extended', '-e', help='Show some extra information', is_flag=True)
@click.option('--timestamp', '-t', 'ts', help='Print timestamp', is_flag=True)
@option_format
@option_watch
@option_watchrefresh
@click.pass_obj
-def members(obj, cluster_names, fmt, watch, w, extended, ts):
+def members(obj, cluster_names, group, fmt, watch, w, extended, ts):
    if not cluster_names:
        if 'scope' in obj:
            cluster_names = [obj['scope']]
        if not cluster_names:
            return logging.warning('Listing members: No cluster names were provided')

-    for cluster_name in cluster_names:
-        dcs = get_dcs(obj, cluster_name)
+    for _ in watching(w, watch):
+        if ts:
+            click.echo(timestamp(0))

-        for _ in watching(w, watch):
-            if ts:
-                click.echo(timestamp(0))
+        for cluster_name in cluster_names:
+            dcs = get_dcs(obj, cluster_name, group)

            cluster = dcs.get_cluster()
-            output_members(cluster, cluster_name, extended, fmt)
+            output_members(obj, cluster, cluster_name, extended, fmt, group)


@ctl.command('topology', help='Prints ASCII topology for given cluster')
@click.argument('cluster_names', nargs=-1)
+@option_citus_group
@option_watch
@option_watchrefresh
@click.pass_obj
@click.pass_context
-def topology(ctx, obj, cluster_names, watch, w):
+def topology(ctx, obj, cluster_names, group, watch, w):
    ctx.forward(members, fmt='topology')


@@ -888,75 +948,20 @@ def timestamp(precision=6):
    return datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')[:precision - 7]


-def touch_member(config, dcs):
-    ''' Rip-off of the ha.touch_member without inter-class dependencies '''
-    p = Postgresql(config['postgresql'])
-    p.set_state('running')
-    p.set_role('master')
-
-    def restapi_connection_string(config):
-        protocol = 'https' if config.get('certfile') else 'http'
-        connect_address = config.get('connect_address')
-        listen = config['listen']
-        return '{0}://{1}/patroni'.format(protocol, connect_address or listen)
-
-    data = {
-        'conn_url': p.connection_string,
-        'api_url': restapi_connection_string(config['restapi']),
-        'state': p.state,
-        'role': p.role
-    }
-
-    return dcs.touch_member(data, permanent=True)
-
-
-def set_defaults(config, cluster_name):
-    """fill-in some basic configuration parameters if config file is not set """
-    config['postgresql'].setdefault('name', cluster_name)
-    config['postgresql'].setdefault('scope', cluster_name)
-    config['postgresql'].setdefault('listen', '127.0.0.1')
-    config['postgresql']['authentication'] = {'replication': None}
-    config['restapi']['listen'] = ':' in config['restapi']['listen'] and config['restapi']['listen'] or '127.0.0.1:8008'
-
-
-@ctl.command('scaffold', help='Create a structure for the cluster in DCS')
-@click.argument('cluster_name')
-@click.option('--sysid', '-s', help='System ID of the cluster to put into the initialize key', default="")
-@click.pass_obj
-def scaffold(obj, cluster_name, sysid):
-    dcs = get_dcs(obj, cluster_name)
-    cluster = dcs.get_cluster()
-    if cluster and cluster.initialize is not None:
-        raise PatroniCtlException("This cluster is already initialized")
-
-    if not dcs.initialize(create_new=True, sysid=sysid):
-        # initialize key already exists, don't touch this cluster
-        raise PatroniCtlException("Initialize key for cluster {0} already exists".format(cluster_name))
-
-    set_defaults(obj, cluster_name)
-
-    # make sure the leader keys will never expire
-    if not (touch_member(obj, dcs) and dcs.attempt_to_acquire_leader(permanent=True)):
-        # we did initialize this cluster, but failed to write the leader or member keys, wipe it down completely.
-        dcs.delete_cluster()
-        raise PatroniCtlException("Unable to install permanent leader for cluster {0}".format(cluster_name))
-    click.echo("Cluster {0} has been created successfully".format(cluster_name))
-
-
@ctl.command('flush', help='Discard scheduled events')
@click.argument('cluster_name')
+@option_citus_group
@click.argument('member_names', nargs=-1)
@click.argument('target', type=click.Choice(['restart', 'switchover']))
-@click.option('--role', '-r', help='Flush only members with this role', default='any',
-              type=click.Choice(['master', 'replica', 'any']))
+@click.option('--role', '-r', help='Flush only members with this role', type=role_choice, default='any')
@option_force
@click.pass_obj
-def flush(obj, cluster_name, member_names, force, role, target):
-    dcs = get_dcs(obj, cluster_name)
+def flush(obj, cluster_name, group, member_names, force, role, target):
+    dcs = get_dcs(obj, cluster_name, group)
    cluster = dcs.get_cluster()

    if target == 'restart':
-        for member in get_members(cluster, cluster_name, member_names, role, force, 'flush'):
+        for member in get_members(obj, cluster, cluster_name, member_names, role, force, 'flush', group=group):
            if member.data.get('scheduled_restart'):
                r = request_patroni(member, 'delete', 'restart')
                check_response(r, member.name, 'flush scheduled restart')
@@ -1002,8 +1007,8 @@ def wait_until_pause_is_applied(dcs, paused, old_cluster):
    return click.echo('Success: cluster management is {0}'.format(paused and 'paused' or 'resumed'))


-def toggle_pause(config, cluster_name, paused, wait):
-    dcs = get_dcs(config, cluster_name)
+def toggle_pause(config, cluster_name, group, paused, wait):
+    dcs = get_dcs(config, cluster_name, group)
    cluster = dcs.get_cluster()
    if cluster.is_paused() == paused:
        raise PatroniCtlException('Cluster is {0} paused'.format(paused and 'already' or 'not'))
@@ -1031,18 +1036,20 @@ def toggle_pause(config, cluster_name, paused, wait):

@ctl.command('pause', help='Disable auto failover')
@arg_cluster_name
+@option_default_citus_group
@click.pass_obj
@click.option('--wait', help='Wait until pause is applied on all nodes', is_flag=True)
-def pause(obj, cluster_name, wait):
-    return toggle_pause(obj, cluster_name, True, wait)
+def pause(obj, cluster_name, group, wait):
+    return toggle_pause(obj, cluster_name, group, True, wait)


@ctl.command('resume', help='Resume auto failover')
@arg_cluster_name
+@option_default_citus_group
@click.option('--wait', help='Wait until pause is cleared on all nodes', is_flag=True)
@click.pass_obj
-def resume(obj, cluster_name, wait):
-    return toggle_pause(obj, cluster_name, False, wait)
+def resume(obj, cluster_name, group, wait):
+    return toggle_pause(obj, cluster_name, group, False, wait)


@contextmanager
@@ -1199,6 +1206,7 @@ def invoke_editor(before_editing, cluster_name):

@ctl.command('edit-config', help="Edit cluster configuration")
@arg_cluster_name
+@option_default_citus_group
@click.option('--quiet', '-q', is_flag=True, help='Do not show changes')
@click.option('--set', '-s', 'kvpairs', multiple=True,
              help='Set specific configuration value. Can be specified multiple times')
@@ -1210,8 +1218,8 @@ def invoke_editor(before_editing, cluster_name):
              ' Use - for stdin.')
@option_force
@click.pass_obj
-def edit_config(obj, cluster_name, force, quiet, kvpairs, pgkvpairs, apply_filename, replace_filename):
-    dcs = get_dcs(obj, cluster_name)
+def edit_config(obj, cluster_name, group, force, quiet, kvpairs, pgkvpairs, apply_filename, replace_filename):
+    dcs = get_dcs(obj, cluster_name, group)
    cluster = dcs.get_cluster()

    before_editing = format_config_for_editing(cluster.config.data)
@@ -1253,9 +1261,10 @@ def edit_config(obj, cluster_name, force, quiet, kvpairs, pgkvpairs, apply_filen

@ctl.command('show-config', help="Show cluster configuration")
@arg_cluster_name
+@option_default_citus_group
@click.pass_obj
-def show_config(obj, cluster_name):
-    cluster = get_dcs(obj, cluster_name).get_cluster()
+def show_config(obj, cluster_name, group):
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()

    click.echo(format_config_for_editing(cluster.config.data))

@@ -1263,16 +1272,17 @@ def show_config(obj, cluster_name):
@ctl.command('version', help='Output version of patronictl command or a running Patroni instance')
@click.argument('cluster_name', required=False)
@click.argument('member_names', nargs=-1)
+@option_citus_group
@click.pass_obj
-def version(obj, cluster_name, member_names):
+def version(obj, cluster_name, group, member_names):
    click.echo("patronictl version {0}".format(__version__))

    if not cluster_name:
        return

    click.echo("")
-    cluster = get_dcs(obj, cluster_name).get_cluster()
-    for m in cluster.members:
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()
+    for m in get_all_members(obj, cluster, group, 'any'):
        if m.api_url:
            if not member_names or m.name in member_names:
                try:
@@ -1288,10 +1298,11 @@ def version(obj, cluster_name, member_names):

@ctl.command('history', help="Show the history of failovers/switchovers")
@arg_cluster_name
+@option_default_citus_group
@option_format
@click.pass_obj
-def history(obj, cluster_name, fmt):
-    cluster = get_dcs(obj, cluster_name).get_cluster()
+def history(obj, cluster_name, group, fmt):
+    cluster = get_dcs(obj, cluster_name, group).get_cluster()
    history = cluster.history and cluster.history.lines or []
    table_header_row = ['TL', 'LSN', 'Reason', 'Timestamp', 'New Leader']
    for line in history:
@@ -20,6 +20,8 @@ from threading import Event, Lock
 from ..exceptions import PatroniFatalException
 from ..utils import deep_compare, parse_bool, uri

+CITUS_COORDINATOR_GROUP_ID = 0
+citus_group_re = re.compile('^(0|[1-9][0-9]*)$')
 slot_name_re = re.compile('^[a-z0-9_]{1,63}$')
 logger = logging.getLogger(__name__)

@@ -94,6 +96,9 @@ def get_dcs(config):
                        # propagate some parameters
                        config[name].update({p: config[p] for p in ('namespace', 'name', 'scope', 'loop_wait',
                                             'patronictl', 'ttl', 'retry_timeout') if p in config})
+                        # From citus section we only need "group" parameter, but will propagate everything just in case.
+                        if isinstance(config.get('citus'), dict):
+                            config[name].update(config['citus'])
                        return item(config[name])
            except ImportError:
                logger.debug('Failed to import %s', module_name)
@@ -224,8 +229,7 @@ class Member(namedtuple('Member', 'index,name,session,data')):


 class RemoteMember(Member):
-    """ Represents a remote master for a standby cluster
-    """
+    """Represents a remote member (typically a primary) for a standby cluster"""
    def __new__(cls, name, data):
        return super(RemoteMember, cls).__new__(cls, None, name, None, data)

@@ -278,7 +282,7 @@ class Leader(namedtuple('Leader', 'index,session,member')):
        version = self.member.version
        # 1.5.6 is the last version which doesn't expose checkpoint_after_promote: false
        if version and version > (1, 5, 6):
-            return self.data.get('role') == 'master' and 'checkpoint_after_promote' not in self.data
+            return self.data.get('role') in ('master', 'primary') and 'checkpoint_after_promote' not in self.data


 class Failover(namedtuple('Failover', 'index,leader,candidate,scheduled_at')):
@@ -444,7 +448,8 @@ class TimelineHistory(namedtuple('TimelineHistory', 'index,value,lines')):
        return TimelineHistory(index, value, lines)


-class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_lsn,members,failover,sync,history,slots,failsafe')):
+class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_lsn,members,'
+                                    'failover,sync,history,slots,failsafe,workers')):

    """Immutable object (namedtuple) which represents PostgreSQL cluster.
    Consists of the following fields:
@@ -458,7 +463,14 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_lsn,members,f
    :param sync: reference to `SyncState` object, last observed synchronous replication state.
    :param history: reference to `TimelineHistory` object
    :param slots: state of permanent logical replication slots on the primary in the format: {"slot_name": int}
-    """
+    :param failsafe: failsafe topology. Node is allowed to become the leader only if its name is found in this list.
+    :param workers: workers of the Citus cluster, optional. Format: {int(group): Cluster()}"""
+
+    def __new__(cls, *args):
+        # Make workers argument optional
+        if len(cls._fields) == len(args) + 1:
+            args = args + ({},)
+        return super(Cluster, cls).__new__(cls, *args)

    @property
    def leader_name(self):
@@ -507,16 +519,16 @@ class Cluster(namedtuple('Cluster', 'initialize,config,leader,last_lsn,members,f

    def get_replication_slots(self, my_name, role, nofailover, major_version, show_error=False):
        # if the replicatefrom tag is set on the member - we should not create the replication slot for it on
-        # the current master, because that member would replicate from elsewhere. We still create the slot if
+        # the current primary, because that member would replicate from elsewhere. We still create the slot if
        # the replicatefrom destination member is currently not a member of the cluster (fallback to the
-        # master), or if replicatefrom destination member happens to be the current master
+        # primary), or if replicatefrom destination member happens to be the current primary
        use_slots = self.use_slots
-        if role in ('master', 'standby_leader'):
+        if role in ('master', 'primary', 'standby_leader'):
            slot_members = [m.name for m in self.members if use_slots and m.name != my_name and
                            (m.replicatefrom is None or m.replicatefrom == my_name or
                             not self.has_member(m.replicatefrom))]
            permanent_slots = self.__permanent_slots if use_slots and \
-                role == 'master' else self.__permanent_physical_slots
+                role in ('master', 'primary') else self.__permanent_physical_slots
        else:
            # only manage slots for replicas that replicate from this one, except for the leader among them
            slot_members = [m.name for m in self.members if use_slots and
@@ -664,6 +676,7 @@ class AbstractDCS(object):
        """
        self._name = config['name']
        self._base_path = re.sub('/+', '/', '/'.join(['', config.get('namespace', 'service'), config['scope']]))
+        self._citus_group = str(config['group']) if isinstance(config.get('group'), six.integer_types) else None
        self._set_loop_wait(config.get('loop_wait', 10))

        self._ctl = bool(config.get('patronictl', False))
@@ -677,7 +690,11 @@ class AbstractDCS(object):
        self.event = Event()

    def client_path(self, path):
-        return '/'.join([self._base_path, path.lstrip('/')])
+        components = [self._base_path]
+        if self._citus_group:
+            components.append(self._citus_group)
+        components.append(path.lstrip('/'))
+        return '/'.join(components)

    @property
    def initialize_path(self):
@@ -752,23 +769,62 @@ class AbstractDCS(object):
        return self._last_seen

    @abc.abstractmethod
-    def _load_cluster(self):
-        """Internally this method should build  `Cluster` object which
-           represents current state and topology of the cluster in DCS.
-           this method supposed to be called only by `get_cluster` method.
+    def _cluster_loader(self, path):
+        """Load and build the `Cluster` object from DCS, which
+        represents a single Patroni cluster.

-           raise `~DCSError` in case of communication or other problems with DCS.
-           If the current node was running as a master and exception raised,
-           instance would be demoted."""
+        :param path: the path in DCS where to load Cluster(s) from.
+        :returns: `Cluster`"""
+
+    def _citus_cluster_loader(self, path):
+        """Load and build `Cluster` onjects from DCS that represent all
+        Patroni clusters from a single Citus cluster.
+
+        :param path: the path in DCS where to load Cluster(s) from.
+        :returns: all Citus groups as `dict`, with group ids as keys"""
+
+    @abc.abstractmethod
+    def _load_cluster(self, path, loader):
+        """Internally this method should call the `loader` method that
+        will build `Cluster` object which represents current state and
+        topology of the cluster in DCS. This method supposed to be
+        called only by `get_cluster` method.
+
+        :param path: the path in DCS where to load Cluster(s) from.
+        :param loader: one of `_cluster_loader` or `_citus_cluster_loader`
+        :raise: `~DCSError` in case of communication problems with DCS.
+        If the current node was running as a primary and exception
+        raised, instance would be demoted."""

    def _bypass_caches(self):
        """Used only in zookeeper"""

+    def is_citus_coordinator(self):
+        return self._citus_group == str(CITUS_COORDINATOR_GROUP_ID)
+
+    def get_citus_coordinator(self):
+        try:
+            path = '{0}/{1}/'.format(self._base_path, CITUS_COORDINATOR_GROUP_ID)
+            return self._load_cluster(path, self._cluster_loader)
+        except Exception as e:
+            logger.error('Failed to load Citus coordinator cluster from %s: %r', self.__class__.__name__, e)
+
+    def _get_citus_cluster(self):
+        groups = self._load_cluster(self._base_path + '/', self._citus_cluster_loader)
+        if isinstance(groups, Cluster):  # Zookeeper could return a cached version
+            cluster = groups
+        else:
+            cluster = groups.pop(CITUS_COORDINATOR_GROUP_ID,
+                                 Cluster(None, None, None, None, [], None, None, None, None, None))
+            cluster.workers.update(groups)
+        return cluster
+
    def get_cluster(self, force=False):
        if force:
            self._bypass_caches()
        try:
-            cluster = self._load_cluster()
+            cluster = self._get_citus_cluster() if self.is_citus_coordinator()\
+                else self._load_cluster(self.client_path(''), self._cluster_loader)
        except Exception:
            self.reset_cluster()
            raise
@@ -830,6 +886,10 @@ class AbstractDCS(object):
                and self._write_failsafe(json.dumps(value, separators=(',', ':'))):
            self._last_failsafe = value

+    @property
+    def failsafe(self):
+        return self._last_failsafe
+
    @abc.abstractmethod
    def _update_leader(self):
        """Update leader key (or session) ttl
@@ -861,11 +921,9 @@ class AbstractDCS(object):
        return ret

    @abc.abstractmethod
-    def attempt_to_acquire_leader(self, permanent=False):
+    def attempt_to_acquire_leader(self):
        """Attempt to acquire leader lock
        This method should create `/leader` key with value=`~self._name`
-        :param permanent: if set to `!True`, the leader key will never expire.
-         Used in patronictl for the external master
        :returns: `!True` if key has been created successfully.

        Key must be created atomically. In case if key already exists it should not be
@@ -895,15 +953,13 @@ class AbstractDCS(object):
        """Create or update `/config` key"""

    @abc.abstractmethod
-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        """Update member key in DCS.
        This method should create or update key with the name = '/members/' + `~self._name`
        and value = data in a given DCS.

        :param data: information about instance (including connection strings)
        :param ttl: ttl for member key, optional parameter. If it is None `~self.member_ttl will be used`
-        :param permanent: if set to `!True`, the member key will never expire.
-         Used in patronictl for the external master.
        :returns: `!True` on success otherwise `!False`
        """

@@ -970,7 +1026,7 @@ class AbstractDCS(object):
        """"""

    def watch(self, leader_index, timeout):
-        """If the current node is a master it should just sleep.
+        """If the current node is a leader it should just sleep.
        Any other node should watch for changes of leader key with a given timeout

        :param leader_index: index of a leader key
@@ -8,14 +8,14 @@ import ssl
 import time
 import urllib3

-from collections import namedtuple
+from collections import defaultdict, namedtuple
 from consul import ConsulException, NotFound, base
 from urllib3.exceptions import HTTPError
 from six.moves.urllib.parse import urlencode, urlparse, quote
 from six.moves.http_client import HTTPException

-from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member,\
-        SyncState, TimelineHistory, ReturnFalseException, catch_return_false_exception
+from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState,\
+        TimelineHistory, ReturnFalseException, catch_return_false_exception, citus_group_re
 from ..exceptions import DCSError
 from ..utils import deep_compare, parse_bool, Retry, RetryFailedError, split_host_port, uri, USER_AGENT

@@ -190,6 +190,7 @@ class Consul(AbstractDCS):

    def __init__(self, config):
        super(Consul, self).__init__(config)
+        self._base_path = self._base_path[1:]
        self._scope = config['scope']
        self._session = None
        self.__do_not_watch = False
@@ -318,85 +319,95 @@ class Consul(AbstractDCS):
            logger.exception('refresh_session')
        raise ConsulError('Failed to renew/create session')

-    def client_path(self, path):
-        return super(Consul, self).client_path(path)[1:]
-
    @staticmethod
    def member(node):
        return Member.from_node(node['ModifyIndex'], os.path.basename(node['Key']), node.get('Session'), node['Value'])

-    def _load_cluster(self):
+    def _cluster_from_nodes(self, nodes):
+        # get initialize flag
+        initialize = nodes.get(self._INITIALIZE)
+        initialize = initialize and initialize['Value']
+
+        # get global dynamic configuration
+        config = nodes.get(self._CONFIG)
+        config = config and ClusterConfig.from_node(config['ModifyIndex'], config['Value'])
+
+        # get timeline history
+        history = nodes.get(self._HISTORY)
+        history = history and TimelineHistory.from_node(history['ModifyIndex'], history['Value'])
+
+        # get last known leader lsn and slots
+        status = nodes.get(self._STATUS)
+        if status:
+            try:
+                status = json.loads(status['Value'])
+                last_lsn = status.get(self._OPTIME)
+                slots = status.get('slots')
+            except Exception:
+                slots = last_lsn = None
+        else:
+            last_lsn = nodes.get(self._LEADER_OPTIME)
+            last_lsn = last_lsn and last_lsn['Value']
+            slots = None
+
        try:
-            path = self.client_path('/')
-            _, results = self.retry(self._client.kv.get, path, recurse=True)
+            last_lsn = int(last_lsn)
+        except Exception:
+            last_lsn = 0

-            if results is None:
-                raise NotFound
+        # get list of members
+        members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]

-            nodes = {}
-            for node in results:
+        # get leader
+        leader = nodes.get(self._LEADER)
+
+        if leader:
+            member = Member(-1, leader['Value'], None, {})
+            member = ([m for m in members if m.name == leader['Value']] or [member])[0]
+            leader = Leader(leader['ModifyIndex'], leader.get('Session'), member)
+
+        # failover key
+        failover = nodes.get(self._FAILOVER)
+        if failover:
+            failover = Failover.from_node(failover['ModifyIndex'], failover['Value'])
+
+        # get synchronization state
+        sync = nodes.get(self._SYNC)
+        sync = SyncState.from_node(sync and sync['ModifyIndex'], sync and sync['Value'])
+
+        # get failsafe topology
+        failsafe = nodes.get(self._FAILSAFE)
+        try:
+            failsafe = json.loads(failsafe['Value']) if failsafe else None
+        except Exception:
+            failsafe = None
+
+        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+
+    def _cluster_loader(self, path):
+        _, results = self.retry(self._client.kv.get, path, recurse=True)
+        if results is None:
+            raise NotFound
+        nodes = {}
+        for node in results:
+            node['Value'] = (node['Value'] or b'').decode('utf-8')
+            nodes[node['Key'][len(path):]] = node
+
+        return self._cluster_from_nodes(nodes)
+
+    def _citus_cluster_loader(self, path):
+        _, results = self.retry(self._client.kv.get, path, recurse=True)
+        clusters = defaultdict(dict)
+        for node in results or []:
+            key = node['Key'][len(path):].split('/', 1)
+            if len(key) == 2 and citus_group_re.match(key[0]):
                node['Value'] = (node['Value'] or b'').decode('utf-8')
-                nodes[node['Key'][len(path):].lstrip('/')] = node
+                clusters[int(key[0])][key[1]] = node
+        return {group: self._cluster_from_nodes(nodes) for group, nodes in clusters.items()}

-            # get initialize flag
-            initialize = nodes.get(self._INITIALIZE)
-            initialize = initialize and initialize['Value']
-
-            # get global dynamic configuration
-            config = nodes.get(self._CONFIG)
-            config = config and ClusterConfig.from_node(config['ModifyIndex'], config['Value'])
-
-            # get timeline history
-            history = nodes.get(self._HISTORY)
-            history = history and TimelineHistory.from_node(history['ModifyIndex'], history['Value'])
-
-            # get last known leader lsn and slots
-            status = nodes.get(self._STATUS)
-            if status:
-                try:
-                    status = json.loads(status['Value'])
-                    last_lsn = status.get(self._OPTIME)
-                    slots = status.get('slots')
-                except Exception:
-                    slots = last_lsn = None
-            else:
-                last_lsn = nodes.get(self._LEADER_OPTIME)
-                last_lsn = last_lsn and last_lsn['Value']
-                slots = None
-
-            try:
-                last_lsn = int(last_lsn)
-            except Exception:
-                last_lsn = 0
-
-            # get list of members
-            members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
-
-            # get leader
-            leader = nodes.get(self._LEADER)
-
-            if leader:
-                member = Member(-1, leader['Value'], None, {})
-                member = ([m for m in members if m.name == leader['Value']] or [member])[0]
-                leader = Leader(leader['ModifyIndex'], leader.get('Session'), member)
-
-            # failover key
-            failover = nodes.get(self._FAILOVER)
-            if failover:
-                failover = Failover.from_node(failover['ModifyIndex'], failover['Value'])
-
-            # get synchronization state
-            sync = nodes.get(self._SYNC)
-            sync = SyncState.from_node(sync and sync['ModifyIndex'], sync and sync['Value'])
-
-            # get failsafe topology
-            failsafe = nodes.get(self._FAILSAFE)
-            try:
-                failsafe = json.loads(failsafe['Value']) if failsafe else None
-            except Exception:
-                failsafe = None
-
-            return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+    def _load_cluster(self, path, loader):
+        try:
+            return loader(path)
        except NotFound:
            return Cluster(None, None, None, None, [], None, None, None, None, None)
        except Exception:
@@ -404,12 +415,12 @@ class Consul(AbstractDCS):
            raise ConsulError('Consul is not responding properly')

    @catch_consul_errors
-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        cluster = self.cluster
        member = cluster and cluster.get_member(self._name, fallback_to_leader=False)

        try:
-            create_member = not permanent and self.refresh_session()
+            create_member = self.refresh_session()
        except DCSError:
            return False

@@ -427,8 +438,7 @@ class Consul(AbstractDCS):
            return True

        try:
-            args = {} if permanent else {'acquire': self._session}
-            self._client.kv.put(self.member_path, json.dumps(data, separators=(',', ':')), **args)
+            self._client.kv.put(self.member_path, json.dumps(data, separators=(',', ':')), acquire=self._session)
            return True
        except InvalidSession:
            self._session = None
@@ -467,6 +477,10 @@ class Consul(AbstractDCS):
            check['TLSServerName'] = self._service_check_tls_server_name
        tags = self._service_tags[:]
        tags.append(role)
+        if role == 'master':
+            tags.append('primary')
+        elif role == 'primary':
+            tags.append('master')
        self._previous_loop_service_tags = self._service_tags
        self._previous_loop_token = self._client.token

@@ -484,7 +498,7 @@ class Consul(AbstractDCS):
            return self.deregister_service(params['service_id'])

        self._previous_loop_register_service = self._register_service
-        if role in ['master', 'replica', 'standby-leader']:
+        if role in ['master', 'primary', 'replica', 'standby-leader']:
            if state != 'running':
                return
            return self.register_service(service_name, **params)
@@ -509,10 +523,9 @@ class Consul(AbstractDCS):
        ):
            return self._update_service(new_data)

-    def _do_attempt_to_acquire_leader(self, permanent, retry):
+    def _do_attempt_to_acquire_leader(self, retry):
        try:
-            kwargs = {} if permanent else {'acquire': self._session}
-            return retry(self._client.kv.put, self.leader_path, self._name, **kwargs)
+            return retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)
        except InvalidSession:
            logger.error('Our session disappeared from Consul. Will try to get a new one and retry attempt')
            self._session = None
@@ -527,16 +540,15 @@ class Consul(AbstractDCS):
            return retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)

    @catch_return_false_exception
-    def attempt_to_acquire_leader(self, permanent=False):
+    def attempt_to_acquire_leader(self):
        retry = self._retry.copy()
-        if not permanent:
-            self._run_and_handle_exceptions(self._do_refresh_session, retry=retry)
+        self._run_and_handle_exceptions(self._do_refresh_session, retry=retry)

-            retry.deadline = retry.stoptime - time.time()
-            if retry.deadline < 1:
-                raise ConsulError('attempt_to_acquire_leader timeout')
+        retry.deadline = retry.stoptime - time.time()
+        if retry.deadline < 1:
+            raise ConsulError('attempt_to_acquire_leader timeout')

-        ret = self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, permanent, retry, retry=None)
+        ret = self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, retry, retry=None)
        if not ret:
            logger.info('Could not take out TTL lock')

@@ -10,6 +10,8 @@ import six
 import socket
 import time

+from collections import defaultdict
+from copy import deepcopy
 from dns.exception import DNSException
 from dns import resolver
 from urllib3 import Timeout
@@ -19,8 +21,8 @@ from six.moves.http_client import HTTPException
 from six.moves.urllib_parse import urlparse
 from threading import Thread

-from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member,\
-        SyncState, TimelineHistory, ReturnFalseException, catch_return_false_exception
+from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState,\
+        TimelineHistory, ReturnFalseException, catch_return_false_exception, citus_group_re
 from ..exceptions import DCSError
 from ..request import get as requests_get
 from ..utils import Retry, RetryFailedError, split_host_port, uri, USER_AGENT
@@ -480,6 +482,7 @@ class AbstractEtcd(AbstractDCS):
                sock.setsockopt(*opt)

    def get_etcd_client(self, config, client_cls):
+        config = deepcopy(config)
        if 'proxy' in config:
            config['use_proxies'] = True
            config['url'] = config['proxy']
@@ -604,71 +607,85 @@ class Etcd(AbstractEtcd):
    def member(node):
        return Member.from_node(node.modifiedIndex, os.path.basename(node.key), node.ttl, node.value)

-    def _load_cluster(self):
+    def _cluster_from_nodes(self, etcd_index, nodes):
+        # get initialize flag
+        initialize = nodes.get(self._INITIALIZE)
+        initialize = initialize and initialize.value
+
+        # get global dynamic configuration
+        config = nodes.get(self._CONFIG)
+        config = config and ClusterConfig.from_node(config.modifiedIndex, config.value)
+
+        # get timeline history
+        history = nodes.get(self._HISTORY)
+        history = history and TimelineHistory.from_node(history.modifiedIndex, history.value)
+
+        # get last know leader lsn and slots
+        status = nodes.get(self._STATUS)
+        if status:
+            try:
+                status = json.loads(status.value)
+                last_lsn = status.get(self._OPTIME)
+                slots = status.get('slots')
+            except Exception:
+                slots = last_lsn = None
+        else:
+            last_lsn = nodes.get(self._LEADER_OPTIME)
+            last_lsn = last_lsn and last_lsn.value
+            slots = None
+
+        try:
+            last_lsn = int(last_lsn)
+        except Exception:
+            last_lsn = 0
+
+        # get list of members
+        members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
+
+        # get leader
+        leader = nodes.get(self._LEADER)
+        if leader:
+            member = Member(-1, leader.value, None, {})
+            member = ([m for m in members if m.name == leader.value] or [member])[0]
+            index = etcd_index if etcd_index > leader.modifiedIndex else leader.modifiedIndex + 1
+            leader = Leader(index, leader.ttl, member)
+
+        # failover key
+        failover = nodes.get(self._FAILOVER)
+        if failover:
+            failover = Failover.from_node(failover.modifiedIndex, failover.value)
+
+        # get synchronization state
+        sync = nodes.get(self._SYNC)
+        sync = SyncState.from_node(sync and sync.modifiedIndex, sync and sync.value)
+
+        # get failsafe topology
+        failsafe = nodes.get(self._FAILSAFE)
+        try:
+            failsafe = json.loads(failsafe.value) if failsafe else None
+        except Exception:
+            failsafe = None
+
+        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+
+    def _cluster_loader(self, path):
+        result = self.retry(self._client.read, path, recursive=True)
+        nodes = {node.key[len(result.key):].lstrip('/'): node for node in result.leaves}
+        return self._cluster_from_nodes(result.etcd_index, nodes)
+
+    def _citus_cluster_loader(self, path):
+        clusters = defaultdict(dict)
+        result = self.retry(self._client.read, path, recursive=True)
+        for node in result.leaves:
+            key = node.key[len(result.key):].lstrip('/').split('/', 1)
+            if len(key) == 2 and citus_group_re.match(key[0]):
+                clusters[int(key[0])][key[1]] = node
+        return {group: self._cluster_from_nodes(result.etcd_index, nodes) for group, nodes in clusters.items()}
+
+    def _load_cluster(self, path, loader):
        cluster = None
        try:
-            result = self.retry(self._client.read, self.client_path(''), recursive=True)
-            nodes = {node.key[len(result.key):].lstrip('/'): node for node in result.leaves}
-
-            # get initialize flag
-            initialize = nodes.get(self._INITIALIZE)
-            initialize = initialize and initialize.value
-
-            # get global dynamic configuration
-            config = nodes.get(self._CONFIG)
-            config = config and ClusterConfig.from_node(config.modifiedIndex, config.value)
-
-            # get timeline history
-            history = nodes.get(self._HISTORY)
-            history = history and TimelineHistory.from_node(history.modifiedIndex, history.value)
-
-            # get last know leader lsn and slots
-            status = nodes.get(self._STATUS)
-            if status:
-                try:
-                    status = json.loads(status.value)
-                    last_lsn = status.get(self._OPTIME)
-                    slots = status.get('slots')
-                except Exception:
-                    slots = last_lsn = None
-            else:
-                last_lsn = nodes.get(self._LEADER_OPTIME)
-                last_lsn = last_lsn and last_lsn.value
-                slots = None
-
-            try:
-                last_lsn = int(last_lsn)
-            except Exception:
-                last_lsn = 0
-
-            # get list of members
-            members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
-
-            # get leader
-            leader = nodes.get(self._LEADER)
-            if leader:
-                member = Member(-1, leader.value, None, {})
-                member = ([m for m in members if m.name == leader.value] or [member])[0]
-                index = result.etcd_index if result.etcd_index > leader.modifiedIndex else leader.modifiedIndex + 1
-                leader = Leader(index, leader.ttl, member)
-
-            # failover key
-            failover = nodes.get(self._FAILOVER)
-            if failover:
-                failover = Failover.from_node(failover.modifiedIndex, failover.value)
-
-            # get synchronization state
-            sync = nodes.get(self._SYNC)
-            sync = SyncState.from_node(sync and sync.modifiedIndex, sync and sync.value)
-
-            # get failsafe topology
-            failsafe = nodes.get(self._FAILSAFE)
-            try:
-                failsafe = json.loads(failsafe.value) if failsafe else None
-            except Exception:
-                failsafe = None
-
-            cluster = Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+            cluster = loader(path)
        except etcd.EtcdKeyNotFound:
            cluster = Cluster(None, None, None, None, [], None, None, None, None, None)
        except Exception as e:
@@ -677,28 +694,24 @@ class Etcd(AbstractEtcd):
        return cluster

    @catch_etcd_errors
-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        data = json.dumps(data, separators=(',', ':'))
-        return self._client.set(self.member_path, data, None if permanent else self._ttl)
+        return self._client.set(self.member_path, data, self._ttl)

    @catch_etcd_errors
    def take_leader(self):
        return self.retry(self._client.write, self.leader_path, self._name, ttl=self._ttl)

-    def _do_attempt_to_acquire_leader(self, permanent=False):
+    def _do_attempt_to_acquire_leader(self):
        try:
-            return bool(self.retry(self._client.write,
-                                   self.leader_path,
-                                   self._name,
-                                   ttl=None if permanent else self._ttl,
-                                   prevExist=False))
+            return bool(self.retry(self._client.write, self.leader_path, self._name, ttl=self._ttl, prevExist=False))
        except etcd.EtcdAlreadyExist:
            logger.info('Could not take out TTL lock')
            return False

    @catch_return_false_exception
-    def attempt_to_acquire_leader(self, permanent=False):
-        return self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, permanent=permanent, retry=None)
+    def attempt_to_acquire_leader(self):
+        return self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, retry=None)

    @catch_etcd_errors
    def set_failover_value(self, value, index=None):
@@ -10,11 +10,12 @@ import sys
 import time
 import urllib3

+from collections import defaultdict
 from threading import Condition, Lock, Thread
 from urllib3.exceptions import ReadTimeoutError, ProtocolError

-from . import ClusterConfig, Cluster, Failover, Leader, Member,\
-        SyncState, TimelineHistory, ReturnFalseException, catch_return_false_exception
+from . import ClusterConfig, Cluster, Failover, Leader, Member, SyncState,\
+        TimelineHistory, ReturnFalseException, catch_return_false_exception, citus_group_re
 from .etcd import AbstractEtcdClientWithFailover, AbstractEtcd, catch_etcd_errors
 from ..exceptions import DCSError, PatroniException
 from ..utils import deep_compare, enable_keepalive, iter_response_objects, RetryFailedError, USER_AGENT
@@ -91,7 +92,7 @@ class Unavailable(Etcd3ClientError):
    code = GRPCCode.Unavailable


-# https://github.com/etcd-io/etcd/blob/master/etcdserver/api/v3rpc/rpctypes/error.go
+# https://github.com/etcd-io/etcd/commits/main/api/v3rpc/rpctypes/error.go
 class LeaseNotFound(NotFound):
    error = "etcdserver: requested lease not found"

@@ -558,13 +559,18 @@ class PatroniEtcd3Client(Etcd3Client):
                raise RetryFailedError('Exceeded retry deadline')
            self._kv_cache.condition.wait(timeout)

-    def get_cluster(self):
-        if self._kv_cache:
+    def get_cluster(self, path):
+        if self._kv_cache and path.startswith(self._etcd3.cluster_prefix):
            with self._kv_cache.condition:
                self._wait_cache(self._etcd3._retry.deadline)
-                return self._kv_cache.copy()
+                ret = self._kv_cache.copy()
        else:
-            return self._etcd3.retry(self.prefix, self._etcd3.cluster_prefix).get('kvs', [])
+            ret = self._etcd3.retry(self.prefix, path).get('kvs', [])
+        for node in ret:
+            node.update({'key': base64_decode(node['key']),
+                         'value': base64_decode(node.get('value', '')),
+                         'lease': node.get('lease')})
+        return ret

    def call_rpc(self, method, fields, retry=None):
        ret = super(PatroniEtcd3Client, self).call_rpc(method, fields, retry)
@@ -641,85 +647,94 @@ class Etcd3(AbstractEtcd):

    @property
    def cluster_prefix(self):
-        return self.client_path('')
+        return self._base_path + '/' if self.is_citus_coordinator() else self.client_path('')

    @staticmethod
    def member(node):
        return Member.from_node(node['mod_revision'], os.path.basename(node['key']), node['lease'], node['value'])

-    def _load_cluster(self):
+    def _cluster_from_nodes(self, nodes):
+        # get initialize flag
+        initialize = nodes.get(self._INITIALIZE)
+        initialize = initialize and initialize['value']
+
+        # get global dynamic configuration
+        config = nodes.get(self._CONFIG)
+        config = config and ClusterConfig.from_node(config['mod_revision'], config['value'])
+
+        # get timeline history
+        history = nodes.get(self._HISTORY)
+        history = history and TimelineHistory.from_node(history['mod_revision'], history['value'])
+
+        # get last know leader lsn and slots
+        status = nodes.get(self._STATUS)
+        if status:
+            try:
+                status = json.loads(status['value'])
+                last_lsn = status.get(self._OPTIME)
+                slots = status.get('slots')
+            except Exception:
+                slots = last_lsn = None
+        else:
+            last_lsn = nodes.get(self._LEADER_OPTIME)
+            last_lsn = last_lsn and last_lsn['value']
+            slots = None
+
+        try:
+            last_lsn = int(last_lsn)
+        except Exception:
+            last_lsn = 0
+
+        # get list of members
+        members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
+
+        # get leader
+        leader = nodes.get(self._LEADER)
+        if not self._ctl and leader and leader['value'] == self._name and self._lease != leader.get('lease'):
+            logger.warning('I am the leader but not owner of the lease')
+
+        if leader:
+            member = Member(-1, leader['value'], None, {})
+            member = ([m for m in members if m.name == leader['value']] or [member])[0]
+            leader = Leader(leader['mod_revision'], leader['lease'], member)
+
+        # failover key
+        failover = nodes.get(self._FAILOVER)
+        if failover:
+            failover = Failover.from_node(failover['mod_revision'], failover['value'])
+
+        # get synchronization state
+        sync = nodes.get(self._SYNC)
+        sync = SyncState.from_node(sync and sync['mod_revision'], sync and sync['value'])
+
+        # get failsafe topology
+        failsafe = nodes.get(self._FAILSAFE)
+        try:
+            failsafe = json.loads(failsafe['value']) if failsafe else None
+        except Exception:
+            failsafe = None
+
+        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+
+    def _cluster_loader(self, path):
+        nodes = {node['key'][len(path):]: node
+                 for node in self._client.get_cluster(path)
+                 if node['key'].startswith(path)}
+        return self._cluster_from_nodes(nodes)
+
+    def _citus_cluster_loader(self, path):
+        clusters = defaultdict(dict)
+        path = self._base_path + '/'
+        for node in self._client.get_cluster(path):
+            key = node['key'][len(path):].split('/', 1)
+            if len(key) == 2 and citus_group_re.match(key[0]):
+                clusters[int(key[0])][key[1]] = node
+        return {group: self._cluster_from_nodes(nodes) for group, nodes in clusters.items()}
+
+    def _load_cluster(self, path, loader):
        cluster = None
        try:
-            path_len = len(self.cluster_prefix)
-
-            nodes = {}
-            for node in self._client.get_cluster():
-                node['key'] = base64_decode(node['key'])
-                node['value'] = base64_decode(node.get('value', ''))
-                node['lease'] = node.get('lease')
-                nodes[node['key'][path_len:].lstrip('/')] = node
-
-            # get initialize flag
-            initialize = nodes.get(self._INITIALIZE)
-            initialize = initialize and initialize['value']
-
-            # get global dynamic configuration
-            config = nodes.get(self._CONFIG)
-            config = config and ClusterConfig.from_node(config['mod_revision'], config['value'])
-
-            # get timeline history
-            history = nodes.get(self._HISTORY)
-            history = history and TimelineHistory.from_node(history['mod_revision'], history['value'])
-
-            # get last know leader lsn and slots
-            status = nodes.get(self._STATUS)
-            if status:
-                try:
-                    status = json.loads(status['value'])
-                    last_lsn = status.get(self._OPTIME)
-                    slots = status.get('slots')
-                except Exception:
-                    slots = last_lsn = None
-            else:
-                last_lsn = nodes.get(self._LEADER_OPTIME)
-                last_lsn = last_lsn and last_lsn['value']
-                slots = None
-
-            try:
-                last_lsn = int(last_lsn)
-            except Exception:
-                last_lsn = 0
-
-            # get list of members
-            members = [self.member(n) for k, n in nodes.items() if k.startswith(self._MEMBERS) and k.count('/') == 1]
-
-            # get leader
-            leader = nodes.get(self._LEADER)
-            if not self._ctl and leader and leader['value'] == self._name and self._lease != leader.get('lease'):
-                logger.warning('I am the leader but not owner of the lease')
-
-            if leader:
-                member = Member(-1, leader['value'], None, {})
-                member = ([m for m in members if m.name == leader['value']] or [member])[0]
-                leader = Leader(leader['mod_revision'], leader['lease'], member)
-
-            # failover key
-            failover = nodes.get(self._FAILOVER)
-            if failover:
-                failover = Failover.from_node(failover['mod_revision'], failover['value'])
-
-            # get synchronization state
-            sync = nodes.get(self._SYNC)
-            sync = SyncState.from_node(sync and sync['mod_revision'], sync and sync['value'])
-
-            # get failsafe topology
-            failsafe = nodes.get(self._FAILSAFE)
-            try:
-                failsafe = json.loads(failsafe['value']) if failsafe else None
-            except Exception:
-                failsafe = None
-
-            cluster = Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+            cluster = loader(path)
        except UnsupportedEtcdVersion:
            raise
        except Exception as e:
@@ -728,12 +743,11 @@ class Etcd3(AbstractEtcd):
        return cluster

    @catch_etcd_errors
-    def touch_member(self, data, permanent=False):
-        if not permanent:
-            try:
-                self.refresh_lease()
-            except Etcd3Error:
-                return False
+    def touch_member(self, data):
+        try:
+            self.refresh_lease()
+        except Etcd3Error:
+            return False

        cluster = self.cluster
        member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
@@ -743,7 +757,7 @@ class Etcd3(AbstractEtcd):

        data = json.dumps(data, separators=(',', ':'))
        try:
-            return self._client.put(self.member_path, data, None if permanent else self._lease)
+            return self._client.put(self.member_path, data, self._lease)
        except LeaseNotFound:
            self._lease = None
            logger.error('Our lease disappeared from Etcd, can not "touch_member"')
@@ -752,13 +766,13 @@ class Etcd3(AbstractEtcd):
    def take_leader(self):
        return self.retry(self._client.put, self.leader_path, self._name, self._lease)

-    def _do_attempt_to_acquire_leader(self, permanent, retry):
+    def _do_attempt_to_acquire_leader(self, retry):
        def _retry(*args, **kwargs):
            kwargs['retry'] = retry
            return retry(*args, **kwargs)

        try:
-            return _retry(self._client.put, self.leader_path, self._name, None if permanent else self._lease, 0)
+            return _retry(self._client.put, self.leader_path, self._name, self._lease, 0)
        except LeaseNotFound:
            logger.error('Our lease disappeared from Etcd. Will try to get a new one and retry attempt')
            self._lease = None
@@ -770,24 +784,23 @@ class Etcd3(AbstractEtcd):
            if retry.deadline < 1:
                raise Etcd3Error('_do_attempt_to_acquire_leader timeout')

-            return _retry(self._client.put, self.leader_path, self._name, None if permanent else self._lease, 0)
+            return _retry(self._client.put, self.leader_path, self._name, self._lease, 0)

    @catch_return_false_exception
-    def attempt_to_acquire_leader(self, permanent=False):
+    def attempt_to_acquire_leader(self):
        retry = self._retry.copy()

        def _retry(*args, **kwargs):
            kwargs['retry'] = retry
            return retry(*args, **kwargs)

-        if not permanent:
-            self._run_and_handle_exceptions(self._do_refresh_lease, retry=_retry)
+        self._run_and_handle_exceptions(self._do_refresh_lease, retry=_retry)

-            retry.deadline = retry.stoptime - time.time()
-            if retry.deadline < 1:
-                raise Etcd3Error('attempt_to_acquire_leader timeout')
+        retry.deadline = retry.stoptime - time.time()
+        if retry.deadline < 1:
+            raise Etcd3Error('attempt_to_acquire_leader timeout')

-        ret = self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, permanent, retry, retry=None)
+        ret = self._run_and_handle_exceptions(self._do_attempt_to_acquire_leader, retry, retry=None)
        if not ret:
            logger.info('Could not take out TTL lock')
        return ret
@@ -853,7 +866,7 @@ class Etcd3(AbstractEtcd):

    @catch_etcd_errors
    def delete_cluster(self):
-        return self.retry(self._client.deleteprefix, self.cluster_prefix)
+        return self.retry(self._client.deleteprefix, self.client_path(''))

    @catch_etcd_errors
    def set_history_value(self, value):
@@ -19,7 +19,7 @@ class ExhibitorEnsembleProvider(object):
        self._uri_path = uri_path
        self._poll_interval = poll_interval
        self._exhibitors = hosts
-        self._master_exhibitors = hosts
+        self._boot_exhibitors = hosts
        self._zookeeper_hosts = ''
        self._next_poll = None
        while not self.poll():
@@ -32,7 +32,7 @@ class ExhibitorEnsembleProvider(object):

        json = self._query_exhibitors(self._exhibitors)
        if not json:
-            json = self._query_exhibitors(self._master_exhibitors)
+            json = self._query_exhibitors(self._boot_exhibitors)

        if isinstance(json, dict) and 'servers' in json and 'port' in json:
            self._next_poll = time.time() + self._poll_interval
@@ -68,7 +68,7 @@ class Exhibitor(ZooKeeper):
        config['hosts'] = self._ensemble_provider.zookeeper_hosts
        super(Exhibitor, self).__init__(config)

-    def _load_cluster(self):
+    def _load_cluster(self, path, loader):
        if self._ensemble_provider.poll():
            self._client.set_hosts(self._ensemble_provider.zookeeper_hosts)
-        return super(Exhibitor, self)._load_cluster()
+        return super(Exhibitor, self)._load_cluster(path, loader)
@@ -8,18 +8,20 @@ import os
 import random
 import socket
 import six
-import sys
 import tempfile
 import time
 import urllib3
 import yaml

+from collections import defaultdict
+from copy import deepcopy
 from urllib3 import Timeout
 from urllib3.exceptions import HTTPError
 from six.moves.http_client import HTTPException
 from threading import Condition, Lock, Thread

-from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState, TimelineHistory
+from . import AbstractDCS, Cluster, ClusterConfig, Failover, Leader, Member, SyncState,\
+        TimelineHistory, CITUS_COORDINATOR_GROUP_ID, citus_group_re
 from ..exceptions import DCSError
 from ..utils import deep_compare, iter_response_objects, keepalive_socket_options,\
        Retry, RetryFailedError, tzutc, uri, USER_AGENT
@@ -239,7 +241,7 @@ class K8sClient(object):

        def set_base_uri(self, value):
            logger.info('Selected new K8s API server endpoint %s', value)
-            # We will connect by IP of the master node which is not listed as alternative name
+            # We will connect by IP of the K8s master node which is not listed as alternative name
            self.pool_manager.connection_pool_kw['assert_hostname'] = False
            self._base_uri = value

@@ -687,8 +689,10 @@ class ObjectCache(Thread):

 class Kubernetes(AbstractDCS):

+    _CITUS_LABEL = 'citus-group'
+
    def __init__(self, config):
-        self._labels = config['labels']
+        self._labels = deepcopy(config['labels'])
        self._labels[config.get('scope_label', 'cluster-name')] = config['scope']
        self._label_selector = ','.join('{0}={1}'.format(k, v) for k, v in self._labels.items())
        self._namespace = config.get('namespace') or 'default'
@@ -696,6 +700,9 @@ class Kubernetes(AbstractDCS):
        self._ca_certs = os.environ.get('PATRONI_KUBERNETES_CACERT', config.get('cacert')) or SERVICE_CERT_FILENAME
        config['namespace'] = ''
        super(Kubernetes, self).__init__(config)
+        if self._citus_group:
+            self._labels[self._CITUS_LABEL] = self._citus_group
+
        self._retry = Retry(deadline=config['retry_timeout'], max_delay=1, max_tries=-1,
                            retry_exceptions=KubernetesRetriableException)
        self._ttl = None
@@ -755,7 +762,7 @@ class Kubernetes(AbstractDCS):

    @property
    def leader_path(self):
-        return self._base_path[1:] if self._api.use_endpoints else super(Kubernetes, self).leader_path
+        return super(Kubernetes, self).leader_path[:-7 if self._api.use_endpoints else None]

    def set_ttl(self, ttl):
        ttl = int(ttl)
@@ -787,95 +794,137 @@ class Kubernetes(AbstractDCS):
                raise RetryFailedError('Exceeded retry deadline')
            self._condition.wait(timeout)

-    def _load_cluster(self):
+    def _cluster_from_nodes(self, group, nodes, pods):
+        members = [self.member(pod) for pod in pods]
+        path = self._base_path[1:] + '-'
+        if group:
+            path += group + '-'
+
+        config = nodes.get(path + self._CONFIG)
+        metadata = config and config.metadata
+        annotations = metadata and metadata.annotations or {}
+
+        # get initialize flag
+        initialize = annotations.get(self._INITIALIZE)
+
+        # get global dynamic configuration
+        config = ClusterConfig.from_node(metadata and metadata.resource_version,
+                                         annotations.get(self._CONFIG) or '{}',
+                                         metadata.resource_version if self._CONFIG in annotations else 0)
+
+        # get timeline history
+        history = TimelineHistory.from_node(metadata and metadata.resource_version,
+                                            annotations.get(self._HISTORY) or '[]')
+
+        leader_path = path[:-1] if self._api.use_endpoints else path + self._LEADER
+        leader = nodes.get(leader_path)
+        metadata = leader and leader.metadata
+        if leader_path == self.leader_path:  # We want to memorize leader_resource_version only for our cluster
+            self._leader_resource_version = metadata.resource_version if metadata else None
+        annotations = metadata and metadata.annotations or {}
+
+        # get last known leader lsn
+        last_lsn = annotations.get(self._OPTIME)
+        try:
+            last_lsn = 0 if last_lsn is None else int(last_lsn)
+        except Exception:
+            last_lsn = 0
+
+        # get permanent slots state (confirmed_flush_lsn)
+        slots = annotations.get('slots')
+        try:
+            slots = slots and json.loads(slots)
+        except Exception:
+            slots = None
+
+        # get failsafe topology
+        failsafe = annotations.get(self._FAILSAFE)
+        try:
+            failsafe = json.loads(failsafe) if failsafe else None
+        except Exception:
+            failsafe = None
+
+        # get leader
+        leader_record = {n: annotations.get(n) for n in (self._LEADER, 'acquireTime',
+                         'ttl', 'renewTime', 'transitions') if n in annotations}
+        # We want to memorize leader_observed_record and update leader_observed_time only for our cluster
+        if leader_path == self.leader_path and (leader_record or self._leader_observed_record)\
+                and leader_record != self._leader_observed_record:
+            self._leader_observed_record = leader_record
+            self._leader_observed_time = time.time()
+
+        leader = leader_record.get(self._LEADER)
+        try:
+            ttl = int(leader_record.get('ttl')) or self._ttl
+        except (TypeError, ValueError):
+            ttl = self._ttl
+
+        # We want to check validity of the leader record only for our own cluster
+        if leader_path == self.leader_path and\
+                not (metadata and self._leader_observed_time and self._leader_observed_time + ttl >= time.time()):
+            leader = None
+
+        if metadata:
+            member = Member(-1, leader, None, {})
+            member = ([m for m in members if m.name == leader] or [member])[0]
+            leader = Leader(metadata.resource_version, None, member)
+
+        # failover key
+        failover = nodes.get(path + self._FAILOVER)
+        metadata = failover and failover.metadata
+        failover = Failover.from_node(metadata and metadata.resource_version,
+                                      metadata and (metadata.annotations or {}).copy())
+
+        # get synchronization state
+        sync = nodes.get(path + self._SYNC)
+        metadata = sync and sync.metadata
+        sync = SyncState.from_node(metadata and metadata.resource_version,  metadata and metadata.annotations)
+
+        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+
+    def _cluster_loader(self, path):
+        return self._cluster_from_nodes(path['group'], path['nodes'], path['pods'])
+
+    def _citus_cluster_loader(self, path):
+        clusters = defaultdict(lambda: {'pods': [], 'nodes': {}})
+
+        for pod in path['pods']:
+            group = pod.metadata.labels.get(self._CITUS_LABEL)
+            if group and citus_group_re.match(group):
+                clusters[group]['pods'].append(pod)
+
+        for name, kind in path['nodes'].items():
+            group = kind.metadata.labels.get(self._CITUS_LABEL)
+            if group and citus_group_re.match(group):
+                clusters[group]['nodes'][name] = kind
+        return {int(group): self._cluster_from_nodes(group, value['nodes'], value['pods'])
+                for group, value in clusters.items()}
+
+    def __load_cluster(self, group, loader):
        stop_time = time.time() + self._retry.deadline
        self._api.refresh_api_servers_cache()
        try:
            with self._condition:
                self._wait_caches(stop_time)
-
-                members = [self.member(pod) for pod in self._pods.copy().values()]
-                nodes = self._kinds.copy()
-
-            config = nodes.get(self.config_path)
-            metadata = config and config.metadata
-            annotations = metadata and metadata.annotations or {}
-
-            # get initialize flag
-            initialize = annotations.get(self._INITIALIZE)
-
-            # get global dynamic configuration
-            config = ClusterConfig.from_node(metadata and metadata.resource_version,
-                                             annotations.get(self._CONFIG) or '{}',
-                                             metadata.resource_version if self._CONFIG in annotations else 0)
-
-            # get timeline history
-            history = TimelineHistory.from_node(metadata and metadata.resource_version,
-                                                annotations.get(self._HISTORY) or '[]')
-
-            leader = nodes.get(self.leader_path)
-            metadata = leader and leader.metadata
-            self._leader_resource_version = metadata.resource_version if metadata else None
-            annotations = metadata and metadata.annotations or {}
-
-            # get last known leader lsn
-            last_lsn = annotations.get(self._OPTIME)
-            try:
-                last_lsn = 0 if last_lsn is None else int(last_lsn)
-            except Exception:
-                last_lsn = 0
-
-            # get permanent slots state (confirmed_flush_lsn)
-            slots = annotations.get('slots')
-            try:
-                slots = slots and json.loads(slots)
-            except Exception:
-                slots = None
-
-            # get failsafe topology
-            failsafe = annotations.get(self._FAILSAFE)
-            try:
-                failsafe = json.loads(failsafe) if failsafe else None
-            except Exception:
-                failsafe = None
-
-            # get leader
-            leader_record = {n: annotations.get(n) for n in (self._LEADER, 'acquireTime',
-                             'ttl', 'renewTime', 'transitions') if n in annotations}
-            if (leader_record or self._leader_observed_record) and leader_record != self._leader_observed_record:
-                self._leader_observed_record = leader_record
-                self._leader_observed_time = time.time()
-
-            leader = leader_record.get(self._LEADER)
-            try:
-                ttl = int(leader_record.get('ttl')) or self._ttl
-            except (TypeError, ValueError):
-                ttl = self._ttl
-
-            if not metadata or not self._leader_observed_time or self._leader_observed_time + ttl < time.time():
-                leader = None
-
-            if metadata:
-                member = Member(-1, leader, None, {})
-                member = ([m for m in members if m.name == leader] or [member])[0]
-                leader = Leader(metadata.resource_version, None, member)
-
-            # failover key
-            failover = nodes.get(self.failover_path)
-            metadata = failover and failover.metadata
-            failover = Failover.from_node(metadata and metadata.resource_version,
-                                          metadata and (metadata.annotations or {}).copy())
-
-            # get synchronization state
-            sync = nodes.get(self.sync_path)
-            metadata = sync and sync.metadata
-            sync = SyncState.from_node(metadata and metadata.resource_version,  metadata and metadata.annotations)
-
-            return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)
+                pods = [pod for pod in self._pods.copy().values()
+                        if not group or pod.metadata.labels.get(self._CITUS_LABEL) == group]
+                nodes = {name: kind for name, kind in self._kinds.copy().items()
+                         if not group or kind.metadata.labels.get(self._CITUS_LABEL) == group}
+            return loader({'group': group, 'pods': pods, 'nodes': nodes})
        except Exception:
            logger.exception('get_cluster')
            raise KubernetesError('Kubernetes API is not responding properly')

+    def _load_cluster(self, path, loader):
+        group = self._citus_group if path == self.client_path('') else None
+        return self.__load_cluster(group, loader)
+
+    def get_citus_coordinator(self):
+        try:
+            return self.__load_cluster(str(CITUS_COORDINATOR_GROUP_ID), self._cluster_loader)
+        except Exception as e:
+            logger.error('Failed to load Citus coordinator cluster from Kubernetes: %r', e)
+
    @staticmethod
    def compare_ports(p1, p2):
        return p1.name == p2.name and p1.port == p2.port and (p1.protocol or 'TCP') == (p2.protocol or 'TCP')
@@ -1082,9 +1131,9 @@ class Kubernetes(AbstractDCS):
        resource_version = kind and kind.metadata.resource_version
        return self._update_leader_with_retry(annotations, resource_version, self.__ips)

-    def attempt_to_acquire_leader(self, permanent=False):
+    def attempt_to_acquire_leader(self):
        now = datetime.datetime.now(tzutc).isoformat()
-        annotations = {self._LEADER: self._name, 'ttl': str(sys.maxsize if permanent else self._ttl),
+        annotations = {self._LEADER: self._name, 'ttl': str(self._ttl),
                       'renewTime': now, 'acquireTime': now, 'transitions': '0'}
        if self._leader_observed_record:
            try:
@@ -1136,11 +1185,11 @@ class Kubernetes(AbstractDCS):
        return self.patch_or_create_config({self._CONFIG: value}, index, bool(self._config_resource_version), False)

    @catch_kubernetes_errors
-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        cluster = self.cluster
        if cluster and cluster.leader and cluster.leader.name == self._name:
            role = 'master'
-        elif data['state'] == 'running' and data['role'] != 'master':
+        elif data['state'] == 'running' and data['role'] not in ('master', 'primary'):
            role = data['role']
        else:
            role = None
@@ -4,13 +4,14 @@ import os
 import threading
 import time

+from collections import defaultdict
 from pysyncobj import SyncObj, SyncObjConf, replicated, FAIL_REASON
 from pysyncobj.dns_resolver import globalDnsResolver
 from pysyncobj.node import TCPNode
 from pysyncobj.transport import TCPTransport, CONNECTION_STATE
 from pysyncobj.utility import TcpUtility

-from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory
+from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory, citus_group_re
 from ..exceptions import DCSError
 from ..utils import validate_directory

@@ -319,13 +320,7 @@ class Raft(AbstractDCS):
    def member(key, value):
        return Member.from_node(value['index'], os.path.basename(key), None, value['value'])

-    def _load_cluster(self):
-        prefix = self.client_path('')
-        response = self._sync_obj.get(prefix, recursive=True)
-        if not response:
-            return Cluster(None, None, None, None, [], None, None, None, None, None)
-        nodes = {os.path.relpath(key, prefix).replace('\\', '/'): value for key, value in response.items()}
-
+    def _cluster_from_nodes(self, nodes):
        # get initialize flag
        initialize = nodes.get(self._INITIALIZE)
        initialize = initialize and initialize['value']
@@ -385,6 +380,25 @@ class Raft(AbstractDCS):

        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)

+    def _cluster_loader(self, path):
+        response = self._sync_obj.get(path, recursive=True)
+        if not response:
+            return Cluster(None, None, None, None, [], None, None, None, None, None)
+        nodes = {key[len(path):]: value for key, value in response.items()}
+        return self._cluster_from_nodes(nodes)
+
+    def _citus_cluster_loader(self, path):
+        clusters = defaultdict(dict)
+        response = self._sync_obj.get(path, recursive=True)
+        for key, value in response.items():
+            key = key[len(path):].split('/', 1)
+            if len(key) == 2 and citus_group_re.match(key[0]):
+                clusters[int(key[0])][key[1]] = value
+        return {group: self._cluster_from_nodes(nodes) for group, nodes in clusters.items()}
+
+    def _load_cluster(self, path, loader):
+        return loader(path)
+
    def _write_leader_optime(self, last_lsn):
        return self._sync_obj.set(self.leader_optime_path, last_lsn, timeout=1)

@@ -401,9 +415,8 @@ class Raft(AbstractDCS):
            ret = self.attempt_to_acquire_leader()
        return ret

-    def attempt_to_acquire_leader(self, permanent=False):
-        return self._sync_obj.set(self.leader_path, self._name, ttl=None if permanent else self._ttl,
-                                  handle_raft_error=False, prevExist=False)
+    def attempt_to_acquire_leader(self):
+        return self._sync_obj.set(self.leader_path, self._name, ttl=self._ttl, handle_raft_error=False, prevExist=False)

    def set_failover_value(self, value, index=None):
        return self._sync_obj.set(self.failover_path, value, prevIndex=index)
@@ -411,9 +424,9 @@ class Raft(AbstractDCS):
    def set_config_value(self, value, index=None):
        return self._sync_obj.set(self.config_path, value, prevIndex=index)

-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        data = json.dumps(data, separators=(',', ':'))
-        return self._sync_obj.set(self.member_path, data, None if permanent else self._ttl, timeout=2)
+        return self._sync_obj.set(self.member_path, data, self._ttl, timeout=2)

    def take_leader(self):
        return self._sync_obj.set(self.leader_path, self._name, ttl=self._ttl)
@@ -11,7 +11,7 @@ from kazoo.protocol.states import KeeperState
 from kazoo.retry import RetryFailedError
 from kazoo.security import make_acl

-from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory
+from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory, citus_group_re
 from ..exceptions import DCSError
 from ..utils import deep_compare

@@ -149,7 +149,11 @@ class ZooKeeper(AbstractDCS):

    def cluster_watcher(self, event):
        self._fetch_cluster = True
-        self.status_watcher(event)
+        if not event or event.state != KazooState.CONNECTED or event.path.startswith(self.client_path('')):
+            self.status_watcher(event)
+
+    def members_watcher(self, event):
+        self._fetch_cluster = True

    def reload_config(self, config):
        self.set_retry_timeout(config['retry_timeout'])
@@ -194,10 +198,10 @@ class ZooKeeper(AbstractDCS):
        except NoNodeError:
            return None

-    def get_status(self, leader):
+    def get_status(self, path, leader):
        watch = self.status_watcher if not leader or leader.name != self._name else None

-        status = self.get_node(self.status_path, watch)
+        status = self.get_node(path + self._STATUS, watch)
        if status:
            try:
                status = json.loads(status[0])
@@ -206,7 +210,7 @@ class ZooKeeper(AbstractDCS):
            except Exception:
                slots = last_lsn = None
        else:
-            last_lsn = self.get_node(self.leader_optime_path, watch)
+            last_lsn = self.get_node(path + self._LEADER_OPTIME, watch)
            last_lsn = last_lsn and last_lsn[0]
            slots = None

@@ -228,41 +232,41 @@ class ZooKeeper(AbstractDCS):
        except NoNodeError:
            return []

-    def load_members(self):
+    def load_members(self, path):
        members = []
-        for member in self.get_children(self.members_path, self.cluster_watcher):
-            data = self.get_node(self.members_path + member)
+        for member in self.get_children(path + self._MEMBERS, self.cluster_watcher):
+            data = self.get_node(path + self._MEMBERS + member)
            if data is not None:
                members.append(self.member(member, *data))
        return members

-    def _inner_load_cluster(self):
+    def _cluster_loader(self, path):
        self._fetch_cluster = False
        self.event.clear()
-        nodes = set(self.get_children(self.client_path(''), self.cluster_watcher))
+        nodes = set(self.get_children(path, self.cluster_watcher))
        if not nodes:
            self._fetch_cluster = True

        # get initialize flag
-        initialize = (self.get_node(self.initialize_path) or [None])[0] if self._INITIALIZE in nodes else None
+        initialize = (self.get_node(path + self._INITIALIZE) or [None])[0] if self._INITIALIZE in nodes else None

        # get global dynamic configuration
-        config = self.get_node(self.config_path, watch=self.cluster_watcher) if self._CONFIG in nodes else None
+        config = self.get_node(path + self._CONFIG, watch=self.cluster_watcher) if self._CONFIG in nodes else None
        config = config and ClusterConfig.from_node(config[1].version, config[0], config[1].mzxid)

        # get timeline history
-        history = self.get_node(self.history_path, watch=self.cluster_watcher) if self._HISTORY in nodes else None
+        history = self.get_node(path + self._HISTORY, watch=self.cluster_watcher) if self._HISTORY in nodes else None
        history = history and TimelineHistory.from_node(history[1].mzxid, history[0])

        # get synchronization state
-        sync = self.get_node(self.sync_path, watch=self.cluster_watcher) if self._SYNC in nodes else None
+        sync = self.get_node(path + self._SYNC, watch=self.cluster_watcher) if self._SYNC in nodes else None
        sync = SyncState.from_node(sync and sync[1].version, sync and sync[0])

        # get list of members
-        members = self.load_members() if self._MEMBERS[:-1] in nodes else []
+        members = self.load_members(path) if self._MEMBERS[:-1] in nodes else []

        # get leader
-        leader = self.get_node(self.leader_path) if self._LEADER in nodes else None
+        leader = self.get_node(path + self._LEADER) if self._LEADER in nodes else None
        if leader:
            member = Member(-1, leader[0], None, {})
            member = ([m for m in members if m.name == leader[0]] or [member])[0]
@@ -270,14 +274,14 @@ class ZooKeeper(AbstractDCS):
            self._fetch_cluster = member.index == -1

        # get last known leader lsn and slots
-        last_lsn, slots = self.get_status(leader)
+        last_lsn, slots = self.get_status(path, leader)

        # failover key
-        failover = self.get_node(self.failover_path, watch=self.cluster_watcher) if self._FAILOVER in nodes else None
+        failover = self.get_node(path + self._FAILOVER, watch=self.cluster_watcher) if self._FAILOVER in nodes else None
        failover = failover and Failover.from_node(failover[1].version, failover[0])

        # get failsafe topology
-        failsafe = self.get_node(self.failsafe_path, watch=self.cluster_watcher) if self._FAILSAFE in nodes else None
+        failsafe = self.get_node(path + self._FAILSAFE, watch=self.cluster_watcher) if self._FAILSAFE in nodes else None
        try:
            failsafe = json.loads(failsafe[0]) if failsafe else None
        except Exception:
@@ -285,11 +289,21 @@ class ZooKeeper(AbstractDCS):

        return Cluster(initialize, config, leader, last_lsn, members, failover, sync, history, slots, failsafe)

-    def _load_cluster(self):
-        cluster = self.cluster
+    def _citus_cluster_loader(self, path):
+        fetch_cluster = False
+        ret = {}
+        for node in self.get_children(path, self.cluster_watcher):
+            if citus_group_re.match(node):
+                ret[int(node)] = self._cluster_loader(path + node + '/')
+                fetch_cluster = fetch_cluster or self._fetch_cluster
+        self._fetch_cluster = fetch_cluster
+        return ret
+
+    def _load_cluster(self, path, loader):
+        cluster = self.cluster if path == self._base_path + '/' else None
        if self._fetch_cluster or cluster is None:
            try:
-                cluster = self._client.retry(self._inner_load_cluster)
+                cluster = self._client.retry(loader, path)
            except Exception:
                logger.exception('get_cluster')
                self.cluster_watcher(None)
@@ -302,10 +316,12 @@ class ZooKeeper(AbstractDCS):
                self.event.clear()
            else:
                try:
-                    last_lsn, slots = self.get_status(cluster.leader)
+                    last_lsn, slots = self.get_status(self.client_path(''), cluster.leader)
                    self.event.clear()
-                    cluster = Cluster(cluster.initialize, cluster.config, cluster.leader, last_lsn, cluster.members,
-                                      cluster.failover, cluster.sync, cluster.history, slots, cluster.failsafe)
+                    cluster = list(cluster)
+                    cluster[3] = last_lsn
+                    cluster[8] = slots
+                    cluster = Cluster(*cluster)
                except Exception:
                    pass
        return cluster
@@ -324,10 +340,10 @@ class ZooKeeper(AbstractDCS):
            logger.exception('Failed to create %s', path)
        return False

-    def attempt_to_acquire_leader(self, permanent=False):
+    def attempt_to_acquire_leader(self):
        try:
            self._client.retry(self._client.create, self.leader_path, self._name.encode('utf-8'),
-                               makepath=True, ephemeral=not permanent)
+                               makepath=True, ephemeral=True)
            return True
        except (ConnectionClosedError, RetryFailedError) as e:
            raise ZooKeeperError(e)
@@ -367,12 +383,15 @@ class ZooKeeper(AbstractDCS):
        return self._create(self.initialize_path, sysid, retry=True) if create_new \
            else self._client.retry(self._client.set, self.initialize_path, sysid)

-    def touch_member(self, data, permanent=False):
+    def touch_member(self, data):
        cluster = self.cluster
        member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
        member_data = self.__last_member_data or member and member.data
+        #  We want to notify leader if some important fields in the member key changed by removing ZNode
        if member and (self._client.client_id is not None and member.session != self._client.client_id[0] or
                       not (deep_compare(member_data.get('tags', {}), data.get('tags', {})) and
+                            (member_data.get('state') == data.get('state') or
+                                'running' not in (member_data.get('state'), data.get('state'))) and
                            member_data.get('version') == data.get('version') and
                            member_data.get('checkpoint_after_promote') == data.get('checkpoint_after_promote'))):
            try:
@@ -389,8 +408,7 @@ class ZooKeeper(AbstractDCS):
                return True
        else:
            try:
-                self._client.create_async(self.member_path, encoded_data, makepath=True,
-                                          ephemeral=not permanent).get(timeout=1)
+                self._client.create_async(self.member_path, encoded_data, makepath=True, ephemeral=True).get(timeout=1)
                self.__last_member_data = data
                return True
            except Exception as e:
@@ -18,7 +18,7 @@ from .postgresql import ACTION_ON_START, ACTION_ON_ROLE_CHANGE
 from .postgresql.misc import postgres_version_to_int
 from .postgresql.rewind import Rewind
 from .utils import polling_loop, tzutc, is_standby_cluster as _is_standby_cluster, parse_int
-from .dcs import RemoteMember
+from .dcs import Cluster, Leader, RemoteMember

 logger = logging.getLogger(__name__)

@@ -70,6 +70,64 @@ class _MemberStatus(namedtuple('_MemberStatus', ['member', 'reachable', 'in_reco
        return None


+class Failsafe(object):
+
+    def __init__(self, dcs):
+        self._lock = RLock()
+        self._dcs = dcs
+        self._last_update = 0
+        self._name = None
+        self._conn_url = None
+        self._api_url = None
+        self._slots = None
+
+    def update(self, data):
+        with self._lock:
+            self._last_update = time.time()
+            self._name = data['name']
+            self._conn_url = data['conn_url']
+            self._api_url = data['api_url']
+            self._slots = data.get('slots')
+
+    @property
+    def leader(self):
+        with self._lock:
+            if self._last_update + self._dcs.ttl > time.time():
+                return Leader(None, None,
+                              RemoteMember(self._name, {'api_url': self._api_url,
+                                                        'conn_url': self._conn_url,
+                                                        'slots': self._slots}))
+
+    def update_cluster(self, cluster):
+        # Enreach cluster with the real leader if there was a ping from it
+        leader = self.leader
+        if leader:
+            cluster = list(cluster)
+            # We rely on the strict order of fields in the namedtuple
+            cluster[2] = leader
+            cluster[8] = leader.member.data['slots']
+            cluster = Cluster(*cluster)
+        return cluster
+
+    def is_active(self):
+        """Is used to report in REST API whether the failsafe mode was activated.
+
+           On primary the self._last_update is set from the
+           set_is_active() method and always returns the correct value.
+
+           On replicas the self._last_update is set at the moment when
+           the primary performs POST /failsafe REST API calls.
+           The side-effect - it is possible that replicas will show
+           failsafe_is_active values different from the primary."""
+
+        with self._lock:
+            return self._last_update + self._dcs.ttl > time.time()
+
+    def set_is_active(self, value):
+        with self._lock:
+            self._last_update = value
+
+
 class Ha(object):

    def __init__(self, patroni):
@@ -81,6 +139,7 @@ class Ha(object):
        self.old_cluster = None
        self._is_leader = False
        self._is_leader_lock = RLock()
+        self._failsafe = Failsafe(patroni.dcs)
        self._was_paused = False
        self._leader_timeline = None
        self.recovering = False
@@ -97,6 +156,8 @@ class Ha(object):
        # Count of concurrent sync disabling requests. Value above zero means that we don't want to be synchronous
        # standby. Changes protected by _member_state_lock.
        self._disable_sync = 0
+        # Remember the last known member role and state written to the DCS in order to notify Citus coordinator
+        self._last_state = None

        # We need following property to avoid shutdown of postgres when join of Patroni to the postgres
        # already running as replica was aborted due to cluster not being initialized in DCS.
@@ -112,9 +173,9 @@ class Ha(object):
        else:
            return self.patroni.config.check_mode(mode)

-    def master_stop_timeout(self):
-        """ Master stop timeout """
-        ret = parse_int(self.patroni.config['master_stop_timeout'])
+    def primary_stop_timeout(self):
+        """ Primary stop timeout """
+        ret = parse_int(self.patroni.config['primary_stop_timeout'])
        return ret if ret and ret > 0 and self.is_synchronous_mode() else None

    def is_paused(self):
@@ -149,6 +210,10 @@ class Ha(object):
            self.old_cluster = cluster
        self.cluster = cluster

+        if self.cluster.is_unlocked() and self.is_failsafe_mode():
+            # If failsafe mode is enabled we want to inject the "real" leader to the cluster
+            self.cluster = cluster = self._failsafe.update_cluster(cluster)
+
        if not self.has_lock(False):
            self.set_is_leader(False)

@@ -165,6 +230,13 @@ class Ha(object):
        self.set_is_leader(ret)
        return ret

+    def _failsafe_config(self):
+        if self.is_failsafe_mode():
+            ret = {m.name: m.api_url for m in self.cluster.members}
+            if self.state_handler.name not in ret:
+                ret[self.state_handler.name] = self.patroni.api.connection_string
+            return ret
+
    def update_lock(self, write_leader_optime=False):
        last_lsn = slots = None
        if write_leader_optime:
@@ -174,7 +246,7 @@ class Ha(object):
            except Exception:
                logger.exception('Exception when called state_handler.last_operation()')
        try:
-            ret = self.dcs.update_leader(last_lsn, slots)
+            ret = self.dcs.update_leader(last_lsn, slots, self._failsafe_config())
        except DCSError:
            raise
        except Exception:
@@ -199,6 +271,22 @@ class Ha(object):
            tags['nosync'] = True
        return tags

+    def notify_citus_coordinator(self, event):
+        if self.state_handler.citus_handler.is_worker():
+            coordinator = self.dcs.get_citus_coordinator()
+            if coordinator and coordinator.leader and coordinator.leader.conn_kwargs:
+                try:
+                    data = {'type': event,
+                            'group': self.state_handler.citus_handler.group(),
+                            'leader': self.state_handler.name,
+                            'timeout': self.dcs.ttl,
+                            'cooldown': self.patroni.config['retry_timeout']}
+                    timeout = self.dcs.ttl if event == 'before_demote' else 2
+                    self.patroni.request(coordinator.leader.member, 'post', 'citus', data, timeout=timeout, retries=0)
+                except Exception as e:
+                    logger.warning('Request to Citus coordinator leader %s %s failed: %r',
+                                   coordinator.leader.name, coordinator.leader.member.api_url, e)
+
    def touch_member(self):
        with self._member_state_lock:
            data = {
@@ -251,7 +339,13 @@ class Ha(object):
            if self.is_paused():
                data['pause'] = True

-            return self.dcs.touch_member(data)
+            ret = self.dcs.touch_member(data)
+            if ret:
+                new_state = (data['state'], {'master': 'primary'}.get(data['role'], data['role']))
+                if self._last_state != new_state and new_state == ('running', 'primary'):
+                    self.notify_citus_coordinator('after_promote')
+                self._last_state = new_state
+            return ret

    def clone(self, clone_member=None, msg='(without leader)'):
        if self.is_standby_cluster() and not isinstance(clone_member, RemoteMember):
@@ -275,7 +369,7 @@ class Ha(object):
            ret = self._async_executor.try_run_async('bootstrap {0}'.format(msg), self.clone, args=(clone_member, msg))
            return ret or 'trying to bootstrap {0}'.format(msg)

-        # no initialize key and node is allowed to be master and has 'bootstrap' section in a configuration file
+        # no initialize key and node is allowed to be primary and has 'bootstrap' section in a configuration file
        elif self.cluster.initialize is None and not self.patroni.nofailover and 'bootstrap' in self.patroni.config:
            if self.dcs.initialize(create_new=True):  # race for initialization
                self.state_handler.bootstrapping = True
@@ -303,11 +397,11 @@ class Ha(object):

    def bootstrap_standby_leader(self):
        """ If we found 'standby' key in the configuration, we need to bootstrap
-            not a real master, but a 'standby leader', that will take base backup
-            from a remote master and start follow it.
+            not a real primary, but a 'standby leader', that will take base backup
+            from a remote member and start follow it.
        """
-        clone_source = self.get_remote_master()
-        msg = 'clone from remote master {0}'.format(clone_source.conn_url)
+        clone_source = self.get_remote_member()
+        msg = 'clone from remote member {0}'.format(clone_source.conn_url)
        result = self.clone(clone_source, msg)
        with self._async_response:  # pretend that post_bootstrap was already executed
            self._async_response.complete(result)
@@ -324,7 +418,7 @@ class Ha(object):
            return self._async_executor.try_run_async(msg, self._rewind.ensure_clean_shutdown) or msg

    def _handle_rewind_or_reinitialize(self):
-        leader = self.get_remote_master() if self.is_standby_cluster() else self.cluster.leader
+        leader = self.get_remote_member() if self.is_standby_cluster() else self.cluster.leader
        if not self._rewind.rewind_or_reinitialize_needed_and_possible(leader):
            return None

@@ -348,12 +442,12 @@ class Ha(object):
        self.watchdog.disable()

        if self.has_lock() and self.update_lock():
-            timeout = self.patroni.config['master_start_timeout']
+            timeout = self.patroni.config['primary_start_timeout']
            if timeout == 0:
-                # We are requested to prefer failing over to restarting master. But see first if there
+                # We are requested to prefer failing over to restarting primary. But see first if there
                # is anyone to fail over to.
                if self.is_failover_possible(self.cluster.members):
-                    logger.info("Master crashed. Failing over.")
+                    logger.info("Primary crashed. Failing over.")
                    self.demote('immediate')
                    return 'stopped PostgreSQL to fail over after a crash'
        else:
@@ -381,11 +475,14 @@ class Ha(object):
                role = 'standby_leader'
                node_to_follow = self._get_node_to_follow(self.cluster)
            elif self.is_standby_cluster() and self.cluster.is_unlocked():
-                msg = "trying to follow a remote master because standby cluster is unhealthy"
-                node_to_follow = self.get_remote_master()
+                msg = "trying to follow a remote member because standby cluster is unhealthy"
+                node_to_follow = self.get_remote_member()
            else:
                msg = "starting as a secondary"
                node_to_follow = self._get_node_to_follow(self.cluster)
+
+            if self.is_synchronous_mode():
+                self.state_handler.sync_handler.set_synchronous_standby_names([])
        elif self.has_lock():
            msg = "starting as readonly because i had the session lock"
            node_to_follow = None
@@ -401,7 +498,7 @@ class Ha(object):
        standby_config = self.get_standby_cluster_config()
        is_standby_cluster = _is_standby_cluster(standby_config)
        if is_standby_cluster and (self.cluster.is_unlocked() or self.has_lock(False)):
-            node_to_follow = self.get_remote_master()
+            node_to_follow = self.get_remote_member()
        elif self.patroni.replicatefrom and self.patroni.replicatefrom != self.state_handler.name:
            node_to_follow = cluster.get_member(self.patroni.replicatefrom)
        else:
@@ -432,7 +529,7 @@ class Ha(object):
                    or self.cluster.is_unlocked():
                if is_leader:
                    self.state_handler.set_role('master')
-                    return 'continue to run as master without lock'
+                    return 'continue to run as primary without lock'
                elif self.state_handler.role != 'standby_leader':
                    self.state_handler.set_role('replica')

@@ -480,11 +577,14 @@ class Ha(object):
    def is_synchronous_mode_strict(self):
        return self.check_mode('synchronous_mode_strict')

+    def is_failsafe_mode(self):
+        return self.check_mode('failsafe_mode')
+
    def process_sync_replication(self):
        """Process synchronous standby beahvior.

        Synchronous standbys are registered in two places postgresql.conf and DCS. The order of updating them must
-        be right. The invariant that should be kept is that if a node is master and sync_standby is set in DCS,
+        be right. The invariant that should be kept is that if a node is primary and sync_standby is set in DCS,
        then that node must have synchronous_standby set to that value. Or more simple, first set in postgresql.conf
        and then in DCS. When removing, first remove in DCS, then in postgresql.conf. This is so we only consider
        promoting standbys that were guaranteed to be replicating synchronously.
@@ -492,9 +592,9 @@ class Ha(object):
        if self.is_synchronous_mode():
            sync_node_count = self.patroni.config['synchronous_node_count']
            current = self.cluster.sync.leader and self.cluster.sync.members or []
-            picked, allow_promote = self.state_handler.pick_synchronous_standby(self.cluster, sync_node_count,
-                                                                                self.patroni.config[
-                                                                                    'maximum_lag_on_syncnode'])
+            picked, allow_promote = self.state_handler.sync_handler.current_state(self.cluster, sync_node_count,
+                                                                                  self.patroni.config[
+                                                                                      'maximum_lag_on_syncnode'])
            if set(picked) != set(current):
                # update synchronous standby list in dcs temporarily to point to common nodes in current and picked
                sync_common = list(set(current).intersection(set(allow_promote)))
@@ -512,15 +612,15 @@ class Ha(object):
                    logger.warning("No standbys available!")

                logger.info("Assigning synchronous standby status to %s", picked)
-                self.state_handler.config.set_synchronous_standby(picked)
+                self.state_handler.sync_handler.set_synchronous_standby_names(picked)

                if picked and picked[0] != '*' and set(allow_promote) != set(picked) and not allow_promote:
                    # Wait for PostgreSQL to enable synchronous mode and see if we can immediately set sync_standby
                    time.sleep(2)
-                    _, allow_promote = self.state_handler.pick_synchronous_standby(self.cluster,
-                                                                                   sync_node_count,
-                                                                                   self.patroni.config[
-                                                                                       'maximum_lag_on_syncnode'])
+                    _, allow_promote = self.state_handler.sync_handler.current_state(self.cluster,
+                                                                                     sync_node_count,
+                                                                                     self.patroni.config[
+                                                                                         'maximum_lag_on_syncnode'])
                if allow_promote and set(allow_promote) != set(sync_common):
                    try:
                        cluster = self.dcs.get_cluster()
@@ -536,7 +636,7 @@ class Ha(object):
        else:
            if self.cluster.sync.leader and self.dcs.delete_sync_state(index=self.cluster.sync.index):
                logger.info("Disabled synchronous replication")
-            self.state_handler.config.set_synchronous_standby([])
+            self.state_handler.sync_handler.set_synchronous_standby_names([])

    def is_sync_standby(self, cluster):
        return cluster.leader and cluster.sync.leader == cluster.leader.name \
@@ -550,7 +650,7 @@ class Ha(object):

        If the connection to DCS fails we run the action anyway, as this is only a hint.

-        There is a small race window where this function runs between a master picking us the sync standby and
+        There is a small race window where this function runs between a primary picking us the sync standby and
        publishing it to the DCS. As the window is rather tiny consequences are holding up commits for one cycle
        period we don't worry about it here."""

@@ -561,7 +661,7 @@ class Ha(object):
            self._disable_sync += 1
        try:
            if self.touch_member():
-                # Master should notice the updated value during the next cycle. We will wait double that, if master
+                # Primary should notice the updated value during the next cycle. We will wait double that, if primary
                # hasn't noticed the value by then not disabling sync replication is not likely to matter.
                for _ in polling_loop(timeout=self.dcs.loop_wait*2, interval=2):
                    try:
@@ -570,7 +670,7 @@ class Ha(object):
                    except DCSError:
                        logger.warning("Could not get cluster state, skipping synchronous standby disable")
                        break
-                    logger.info("Waiting for master to release us from synchronous standby")
+                    logger.info("Waiting for primary to release us from synchronous standby")
            else:
                logger.warning("Updating member state failed, skipping synchronous standby disable")

@@ -580,14 +680,14 @@ class Ha(object):
                self._disable_sync -= 1

    def update_cluster_history(self):
-        master_timeline = self.state_handler.get_master_timeline()
+        primary_timeline = self.state_handler.get_primary_timeline()
        cluster_history = self.cluster.history and self.cluster.history.lines
-        if master_timeline == 1:
+        if primary_timeline == 1:
            if cluster_history:
                self.dcs.set_history_value('[]')
-        elif not cluster_history or cluster_history[-1][0] != master_timeline - 1 or len(cluster_history[-1]) != 5:
+        elif not cluster_history or cluster_history[-1][0] != primary_timeline - 1 or len(cluster_history[-1]) != 5:
            cluster_history = {line[0]: line for line in cluster_history or []}
-            history = self.state_handler.get_history(master_timeline)
+            history = self.state_handler.get_history(primary_timeline)
            if history and self.cluster.config:
                history = history[-self.cluster.config.max_timelines_history:]
                for line in history:
@@ -600,14 +700,14 @@ class Ha(object):
                            line.append(cluster_history[line[0]][4])
                self.dcs.set_history_value(json.dumps(history, separators=(',', ':')))

-    def enforce_follow_remote_master(self, message):
-        demote_reason = 'cannot be a real master in standby cluster'
+    def enforce_follow_remote_member(self, message):
+        demote_reason = 'cannot be a real primary in standby cluster'
        return self.follow(demote_reason, message)

-    def enforce_master_role(self, message, promote_message):
+    def enforce_primary_role(self, message, promote_message):
        """
        Ensure the node that has won the race for the leader key meets criteria
-        for promoting its PG server to the 'master' role.
+        for promoting its PG server to the 'primary' role.
        """
        if not self.is_paused():
            if not self.watchdog.is_running and not self.watchdog.activate():
@@ -628,13 +728,14 @@ class Ha(object):
                    return 'Promotion cancelled because the pre-promote script failed'

        if self.state_handler.is_leader():
-            # Inform the state handler about its master role.
+            # Inform the state handler about its primary role.
            # It may be unaware of it if postgres is promoted manually.
            self.state_handler.set_role('master')
            self.process_sync_replication()
            self.update_cluster_history()
+            self.state_handler.citus_handler.sync_pg_dist_node(self.cluster)
            return message
-        elif self.state_handler.role == 'master':
+        elif self.state_handler.role in ('master', 'promoted', 'primary'):
            self.process_sync_replication()
            return message
        else:
@@ -645,16 +746,21 @@ class Ha(object):
                    # Somebody else updated sync state, it may be due to us losing the lock. To be safe, postpone
                    # promotion until next cycle. TODO: trigger immediate retry of run_cycle
                    return 'Postponing promotion because synchronous replication state was updated by somebody else'
-                self.state_handler.config.set_synchronous_standby(['*'] if self.is_synchronous_mode_strict() else [])
-            if self.state_handler.role != 'master':
+                self.state_handler.sync_handler.set_synchronous_standby_names(
+                    ['*'] if self.is_synchronous_mode_strict() else [])
+            if self.state_handler.role not in ('master', 'promoted', 'primary'):
                def on_success():
                    self._rewind.reset_state()
                    logger.info("cleared rewind state after becoming the leader")

+                def before_promote():
+                    self.notify_citus_coordinator('before_promote')
+
                with self._async_response:
                    self._async_response.reset()
                self._async_executor.try_run_async('promote', self.state_handler.promote,
-                                                   args=(self.dcs.loop_wait, self._async_response, on_success))
+                                                   args=(self.dcs.loop_wait, self._async_response,
+                                                         before_promote, on_success))
            return promote_message

    def fetch_node_status(self, member):
@@ -678,6 +784,49 @@ class Ha(object):
        pool.join()
        return results

+    def update_failsafe(self, data):
+        if self.state_handler.state == 'running' and self.state_handler.role in ('master', 'primary'):
+            return 'Running as a leader'
+        self._failsafe.update(data)
+
+    def failsafe_is_active(self):
+        return self._failsafe.is_active()
+
+    def call_failsafe_member(self, data, member):
+        try:
+            response = self.patroni.request(member, 'post', 'failsafe', data, timeout=2, retries=1)
+            data = response.data.decode('utf-8')
+            logger.info('Got response from %s %s: %s', member.name, member.api_url, data)
+            return response.status == 200 and data == 'Accepted'
+        except Exception as e:
+            logger.warning("Request failed to %s: POST %s (%s)", member.name, member.api_url, e)
+        return False
+
+    def check_failsafe_topology(self):
+        failsafe = self.dcs.failsafe
+        if not isinstance(failsafe, dict) or self.state_handler.name not in failsafe:
+            return False
+        data = {
+            'name': self.state_handler.name,
+            'conn_url': self.state_handler.connection_string,
+            'api_url': self.patroni.api.connection_string,
+        }
+        try:
+            data['slots'] = self.state_handler.slots()
+        except Exception:
+            logger.exception('Exception when called state_handler.slots()')
+        members = [RemoteMember(name, {'api_url': url})
+                   for name, url in failsafe.items()
+                   if name != self.state_handler.name]
+        if not members:  # A sinlge node cluster
+            return True
+        pool = ThreadPool(len(members))
+        call_failsafe_member = functools.partial(self.call_failsafe_member, data)
+        results = pool.map(call_failsafe_member, members)
+        pool.close()
+        pool.join()
+        return all(results)
+
    def is_lagging(self, wal_position):
        """Returns if instance with an wal should consider itself unhealthy to be promoted due to replication lag.

@@ -693,7 +842,7 @@ class Ha(object):
        my_wal_position = self.state_handler.last_operation()
        if check_replication_lag and self.is_lagging(my_wal_position):
            logger.info('My wal position exceeds maximum replication lag')
-            return False  # Too far behind last reported wal position on master
+            return False  # Too far behind last reported wal position on primary

        if not self.is_standby_cluster() and self.check_timeline():
            cluster_timeline = self.cluster.timeline
@@ -709,7 +858,7 @@ class Ha(object):
            for st in self.fetch_nodes_statuses(members):
                if st.failover_limitation() is None:
                    if not st.in_recovery:
-                        logger.warning('Master (%s) is still alive', st.member.name)
+                        logger.warning('Primary (%s) is still alive', st.member.name)
                        return False
                    if my_wal_position < st.wal_position:
                        logger.info('Wal position of %s is ahead of my wal position', st.member.name)
@@ -752,7 +901,7 @@ class Ha(object):
                return True
            elif self.is_paused():
                # Remove failover key if the node to failover has terminated to avoid waiting for it indefinitely
-                # In order to avoid attempts to delete this key from all nodes only the master is allowed to do it.
+                # In order to avoid attempts to delete this key from all nodes only the primary is allowed to do it.
                if (not self.cluster.get_member(failover.candidate, fallback_to_leader=False) and
                   self.state_handler.is_leader()):
                    logger.warning("manual failover: removing failover key because failover candidate is not running")
@@ -805,7 +954,7 @@ class Ha(object):
        if self.is_paused() and not self.patroni.nofailover and \
                self.cluster.failover and not self.cluster.failover.scheduled_at:
            ret = self.manual_failover_process_no_leader()
-            if ret is not None:  # continue if we just deleted the stale failover key as a master
+            if ret is not None:  # continue if we just deleted the stale failover key as a leader
                return ret

        if self.state_handler.is_starting():  # postgresql still starting up is unhealthy
@@ -833,8 +982,19 @@ class Ha(object):
            logger.warning('Watchdog device is not usable')
            return False

-        # When in sync mode, only last known master and sync standby are allowed to promote automatically.
-        all_known_members = self.cluster.members + self.old_cluster.members
+        all_known_members = self.old_cluster.members
+        if self.is_failsafe_mode():
+            failsafe_members = self.dcs.failsafe
+            # We want to discard failsafe_mode if the /failsafe key contains garbage or empty.
+            if isinstance(failsafe_members, dict):
+                # If current node is missing in the /failsafe key we immediately disqualify it from the race.
+                if failsafe_members and self.state_handler.name not in failsafe_members:
+                    return False
+                # Race among not only existing cluster members, but also all known members from the failsafe config
+                all_known_members += [RemoteMember(name, {'api_url': url}) for name, url in failsafe_members.items()]
+        all_known_members += self.cluster.members
+
+        # When in sync mode, only last known primary and sync standby are allowed to promote automatically.
        if self.is_synchronous_mode() and self.cluster.sync and self.cluster.sync.leader:
            if not self.cluster.sync.matches(self.state_handler.name):
                return False
@@ -857,14 +1017,14 @@ class Ha(object):
        logger.info("Leader key released")

    def demote(self, mode):
-        """Demote PostgreSQL running as master.
+        """Demote PostgreSQL running as primary.

        :param mode: One of offline, graceful or immediate.
            offline is used when connection to DCS is not available.
            graceful is used when failing over to another node due to user request. May only be called running async.
-            immediate is used when we determine that we are not suitable for master and want to failover quickly
+            immediate is used when we determine that we are not suitable for primary and want to failover quickly
                without regard for data durability. May only be called synchronously.
-            immediate-nolock is used when find out that we have lost the lock to be master. Need to bring down
+            immediate-nolock is used when find out that we have lost the lock to be primary. Need to bring down
                PostgreSQL as quickly as possible without regard for data durability. May only be called synchronously.
        """
        mode_control = {
@@ -891,10 +1051,17 @@ class Ha(object):
                    self.release_leader_key_voluntarily(checkpoint_location)
                    status['released'] = True

+        def before_shutdown():
+            if self.state_handler.citus_handler.is_coordinator():
+                self.state_handler.citus_handler.on_demote()
+            else:
+                self.notify_citus_coordinator('before_demote')
+
        self.state_handler.stop(mode_control['stop'], checkpoint=mode_control['checkpoint'],
                                on_safepoint=self.watchdog.disable if self.watchdog.is_running else None,
                                on_shutdown=on_shutdown if mode_control['release'] else None,
-                                stop_timeout=self.master_stop_timeout())
+                                before_shutdown=before_shutdown if mode == 'graceful' else None,
+                                stop_timeout=self.primary_stop_timeout())
        self.state_handler.set_role('demoted')
        self.set_is_leader(False)

@@ -913,14 +1080,15 @@ class Ha(object):
            except Exception:
                node_to_follow, leader = None, None

+        if self.is_synchronous_mode():
+            self.state_handler.sync_handler.set_synchronous_standby_names([])
+
        # FIXME: with mode offline called from DCS exception handler and handle_long_action_in_progress
        # there could be an async action already running, calling follow from here will lead
        # to racy state handler state updates.
        if mode_control['async_req']:
            self._async_executor.try_run_async('starting after demotion', self.state_handler.follow, (node_to_follow,))
        else:
-            if self.is_synchronous_mode():
-                self.state_handler.config.set_synchronous_standby([])
            if self._rewind.rewind_or_reinitialize_needed_and_possible(leader):
                return False  # do not start postgres, but run pg_rewind on the next iteration
            self.state_handler.follow(node_to_follow)
@@ -1021,11 +1189,11 @@ class Ha(object):
                if self.is_standby_cluster():
                    # standby leader disappeared, and this is the healthiest
                    # replica, so it should become a new standby leader.
-                    # This implies we need to start following a remote master
+                    # This implies we need to start following a remote member
                    msg = 'promoted self to a standby leader by acquiring session lock'
-                    return self.enforce_follow_remote_master(msg)
+                    return self.enforce_follow_remote_member(msg)
                else:
-                    return self.enforce_master_role(
+                    return self.enforce_primary_role(
                        'acquired session lock as a leader',
                        'promoted self to leader by acquiring session lock'
                    )
@@ -1040,7 +1208,7 @@ class Ha(object):
                time.sleep(2)  # Give a time to somebody to take the leader lock

            if self.patroni.nofailover:
-                return self.follow('demoting self because I am not allowed to become master',
+                return self.follow('demoting self because I am not allowed to become primary',
                                   'following a different leader because I am not allowed to promote')
            return self.follow('demoting self because i am not the healthiest node',
                               'following a different leader because i am not the healthiest node')
@@ -1049,11 +1217,11 @@ class Ha(object):
        if self.has_lock():
            if self.is_paused() and not self.state_handler.is_leader():
                if self.cluster.failover and self.cluster.failover.candidate == self.state_handler.name:
-                    return 'waiting to become master after promote...'
+                    return 'waiting to become primary after promote...'

                if not self.is_standby_cluster():
                    self._delete_leader()
-                    return 'removed leader lock because postgres is not running as master'
+                    return 'removed leader lock because postgres is not running as primary'

            if self.update_lock(True):
                msg = self.process_manual_failover_from_leader()
@@ -1065,14 +1233,14 @@ class Ha(object):

                if self.is_standby_cluster():
                    # in case of standby cluster we don't really need to
-                    # enforce anything, since the leader is not a master.
+                    # enforce anything, since the leader is not a primary
                    # So just remind the role.
                    msg = 'no action. I am ({0}), the standby leader with the lock'.format(self.state_handler.name) \
                          if self.state_handler.role == 'standby_leader' else \
                          'promoted self to a standby leader because i had the session lock'
-                    return self.enforce_follow_remote_master(msg)
+                    return self.enforce_follow_remote_member(msg)
                else:
-                    return self.enforce_master_role(
+                    return self.enforce_primary_role(
                        'no action. I am ({0}), the leader with the lock'.format(self.state_handler.name),
                        'promoted self to leader because I had the session lock'
                    )
@@ -1081,7 +1249,7 @@ class Ha(object):
                logger.error('failed to update leader lock')
                if self.state_handler.is_leader():
                    if self.is_paused():
-                        return 'continue to run as master after failing to update leader lock in DCS'
+                        return 'continue to run as primary after failing to update leader lock in DCS'
                    self.demote('immediate-nolock')
                    return 'demoted self because failed to update leader lock in DCS'
                else:
@@ -1188,11 +1356,19 @@ class Ha(object):

        # Now that restart is scheduled we can set timeout for startup, it will get reset
        # once async executor runs and main loop notices PostgreSQL as up.
-        timeout = restart_data.get('timeout', self.patroni.config['master_start_timeout'])
+        timeout = restart_data.get('timeout', self.patroni.config['primary_start_timeout'])
        self.set_start_timeout(timeout)

+        def before_shutdown():
+            self.notify_citus_coordinator('before_demote')
+
+        def after_start():
+            self.notify_citus_coordinator('after_promote')
+
        # For non async cases we want to wait for restart to complete or timeout before returning.
-        do_restart = functools.partial(self.state_handler.restart, timeout, self._async_executor.critical_task)
+        do_restart = functools.partial(self.state_handler.restart, timeout, self._async_executor.critical_task,
+                                       before_shutdown=before_shutdown if self.has_lock() else None,
+                                       after_start=after_start if self.has_lock() else None)
        if self.is_synchronous_mode() and not self.has_lock():
            do_restart = functools.partial(self.while_not_sync_standby, do_restart)

@@ -1243,7 +1419,7 @@ class Ha(object):
        """
        if self.has_lock() and self.update_lock():
            if self._async_executor.scheduled_action == 'doing crash recovery in a single user mode':
-                time_left = self.patroni.config['master_start_timeout'] - (time.time() - self._crash_recovery_started)
+                time_left = self.patroni.config['primary_start_timeout'] - (time.time() - self._crash_recovery_started)
                if time_left <= 0 and self.is_failover_possible(self.cluster.members):
                    logger.info("Demoting self because crash recovery is taking too long")
                    self.state_handler.cancellable.cancel(True)
@@ -1252,7 +1428,7 @@ class Ha(object):

            return 'updated leader lock during ' + self._async_executor.scheduled_action
        elif not self.state_handler.bootstrapping and not self.is_paused():
-            # Don't have lock, make sure we are not promoting or starting up a master in the background
+            # Don't have lock, make sure we are not promoting or starting up a primary in the background
            if self._async_executor.scheduled_action == 'promote':
                with self._async_response:
                    cancel = self._async_response.cancel()
@@ -1260,8 +1436,8 @@ class Ha(object):
                    self.state_handler.cancellable.cancel()
                    return 'lost leader before promote'

-            if self.state_handler.role == 'master':
-                logger.info("Demoting master during " + self._async_executor.scheduled_action)
+            if self.state_handler.role in ('master', 'primary'):
+                logger.info("Demoting primary during " + self._async_executor.scheduled_action)
                if self._async_executor.scheduled_action == 'restart':
                    # Restart needs a special interlocking cancel because postmaster may be just started in a
                    # background thread and has not even written a pid file yet.
@@ -1286,7 +1462,7 @@ class Ha(object):
        if not self.state_handler.is_running():
            self.watchdog.disable()
            if self.has_lock():
-                if self.state_handler.role in ('master', 'standby_leader'):
+                if self.state_handler.role in ('master', 'primary', 'standby_leader'):
                    self.state_handler.set_role('demoted')
                self._delete_leader()
                return 'removed leader key after trying and failing to start postgres'
@@ -1320,6 +1496,7 @@ class Ha(object):
        if not self.watchdog.activate():
            logger.error('Cancelling bootstrap because watchdog activation failed')
            self.cancel_initialization()
+
        self._rewind.ensure_checkpoint_after_promote(self.wakeup)
        self.dcs.initialize(create_new=(self.cluster.initialize is None), sysid=self.state_handler.sysid)
        self.dcs.set_config_value(json.dumps(self.patroni.config.dynamic_configuration, separators=(',', ':')))
@@ -1348,16 +1525,16 @@ class Ha(object):
                self.demote('immediate-nolock')
                return 'stopped PostgreSQL while starting up because leader key was lost'

-            timeout = self._start_timeout or self.patroni.config['master_start_timeout']
+            timeout = self._start_timeout or self.patroni.config['primary_start_timeout']
            time_left = timeout - self.state_handler.time_in_state()

            if time_left <= 0:
                if self.is_failover_possible(self.cluster.members):
-                    logger.info("Demoting self because master startup is taking too long")
+                    logger.info("Demoting self because primary startup is taking too long")
                    self.demote('immediate')
                    return 'stopped PostgreSQL because of startup timeout'
                else:
-                    return 'master start has timed out, but continuing to wait because failover is not possible'
+                    return 'primary start has timed out, but continuing to wait because failover is not possible'
            else:
                msg = self.process_manual_failover_from_leader()
                if msg is not None:
@@ -1370,7 +1547,7 @@ class Ha(object):
            return None

    def set_start_timeout(self, value):
-        """Sets timeout for starting as master before eligible for failover.
+        """Sets timeout for starting as primary before eligible for failover.

        Must be called when async_executor is busy or in the main thread."""
        self._start_timeout = value
@@ -1445,7 +1622,7 @@ class Ha(object):
            if not data_directory_is_accessible or data_directory_is_empty:
                self.state_handler.set_role('uninitialized')
                self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
-                # In case datadir went away while we were master.
+                # In case datadir went away while we were primary
                self.watchdog.disable()

                # is this instance the leader?
@@ -1485,7 +1662,7 @@ class Ha(object):
                            and not self.state_handler.is_leader():
                        self._join_aborted = True
                        logger.error('No initialize key in DCS and PostgreSQL is running as replica, aborting start')
-                        logger.error('Please first start Patroni on the node running as master')
+                        logger.error('Please first start Patroni on the node running as primary')
                        sys.exit(1)
                    self.dcs.initialize(create_new=(self.cluster.initialize is None), sysid=data_sysid)

@@ -1509,48 +1686,84 @@ class Ha(object):
                # try to start dead postgres
                return self.recover()

-            try:
-                if self.cluster.is_unlocked():
-                    ret = self.process_unhealthy_cluster()
-                else:
-                    msg = self.process_healthy_cluster()
-                    ret = self.evaluate_scheduled_restart() or msg
-            finally:
-                # we might not have a valid PostgreSQL connection here if another thread
-                # stops PostgreSQL, therefore, we only reload replication slots if no
-                # asynchronous processes are running (should be always the case for the master)
-                if not self._async_executor.busy and not self.state_handler.is_starting():
-                    create_slots = self.state_handler.slots_handler.sync_replication_slots(self.cluster,
-                                                                                           self.patroni.nofailover,
-                                                                                           self.patroni.replicatefrom,
-                                                                                           self.is_paused())
-                    if not self.state_handler.cb_called:
-                        if not self.state_handler.is_leader():
-                            self._rewind.trigger_check_diverged_lsn()
-                        self.state_handler.call_nowait(ACTION_ON_START)
-                    if create_slots and self.cluster.leader:
-                        err = self._async_executor.try_run_async('copy_logical_slots',
-                                                                 self.state_handler.slots_handler.copy_logical_slots,
-                                                                 args=(self.cluster, create_slots))
-                        if not err:
-                            ret = 'Copying logical slots {0} from the primary'.format(create_slots)
+            if self.cluster.is_unlocked():
+                ret = self.process_unhealthy_cluster()
+            else:
+                msg = self.process_healthy_cluster()
+                ret = self.evaluate_scheduled_restart() or msg
+
+            # we might not have a valid PostgreSQL connection here if another thread
+            # stops PostgreSQL, therefore, we only reload replication slots if no
+            # asynchronous processes are running (should be always the case for the primary)
+            if not self._async_executor.busy and not self.state_handler.is_starting():
+                create_slots = self._sync_replication_slots(False)
+                if not self.state_handler.cb_called:
+                    if not self.state_handler.is_leader():
+                        self._rewind.trigger_check_diverged_lsn()
+                    self.state_handler.call_nowait(ACTION_ON_START)
+                if create_slots and self.cluster.leader:
+                    err = self._async_executor.try_run_async('copy_logical_slots',
+                                                             self.state_handler.slots_handler.copy_logical_slots,
+                                                             args=(self.cluster, create_slots))
+                    if not err:
+                        ret = 'Copying logical slots {0} from the primary'.format(create_slots)
            return ret
        except DCSError:
            dcs_failed = True
            logger.error('Error communicating with DCS')
-            if not self.is_paused() and self.state_handler.is_running() and self.state_handler.is_leader():
+            return self._handle_dcs_error()
+        except (psycopg.Error, PostgresConnectionException):
+            return 'Error communicating with PostgreSQL. Will try again later'
+        finally:
+            if not dcs_failed:
+                if self.is_leader():
+                    self._failsafe.set_is_active(0)
+                self.touch_member()
+
+    def _handle_dcs_error(self):
+        if not self.is_paused() and self.state_handler.is_running():
+            if self.state_handler.is_leader():
+                if self.is_failsafe_mode() and self.check_failsafe_topology():
+                    self.set_is_leader(True)
+                    self._failsafe.set_is_active(time.time())
+                    self.watchdog.keepalive()
+                    return 'continue to run as a leader because failsafe mode is enabled and all members are accessible'
+                self._failsafe.set_is_active(0)
                msg = 'demoting self because DCS is not accessible and I was a leader'
                if not self._async_executor.try_run_async(msg, self.demote, ('offline',)):
                    return msg
                logger.warning('AsyncExecutor is busy, demoting from the main thread')
                self.demote('offline')
                return 'demoted self because DCS is not accessible and I was a leader'
-            return 'DCS is not accessible'
-        except (psycopg.Error, PostgresConnectionException):
-            return 'Error communicating with PostgreSQL. Will try again later'
-        finally:
-            if not dcs_failed:
-                self.touch_member()
+            else:
+                self._sync_replication_slots(True)
+        return 'DCS is not accessible'
+
+    def _sync_replication_slots(self, dcs_failed):
+        """Handles replication slots.
+
+        :param dcs_failed: bool, indicates that communication with DCS failed (get_cluster() or update_leader())
+        :returns: list[str], replication slots names that should be copied from the primary"""
+
+        slots = []
+
+        # If dcs_failed we don't want to touch replication slots on a leader or replicas if failsafe_mode isn't enabled.
+        if not self.cluster or dcs_failed and (self.is_leader() or not self.is_failsafe_mode()):
+            return slots
+
+        # It could be that DCS is read-only, or only the leader can't access it.
+        # Only the second one could be handled by `load_cluster_from_dcs()`.
+        # The first one affects advancing logical replication slots on replicas, therefore we rely on
+        # Failsafe.update_cluster(), that will return "modified" Cluster if failsafe mode is active.
+        cluster = self._failsafe.update_cluster(self.cluster)\
+            if self.is_failsafe_mode() and not self.is_leader() else self.cluster
+        if cluster:
+            slots = self.state_handler.slots_handler.sync_replication_slots(cluster,
+                                                                            self.patroni.nofailover,
+                                                                            self.patroni.replicatefrom,
+                                                                            self.is_paused())
+        # Don't copy replication slots if failsafe_mode is active
+        return [] if self.failsafe_is_active() else slots

    def run_cycle(self):
        with self._async_executor:
@@ -1588,10 +1801,15 @@ class Ha(object):
                    else:
                        self.dcs.write_leader_optime(checkpoint_location)

+            def _before_shutdown():
+                self.notify_citus_coordinator('before_demote')
+
            on_shutdown = _on_shutdown if self.is_leader() else None
+            before_shutdown = _before_shutdown if self.is_leader() else None
            self.while_not_sync_standby(lambda: self.state_handler.stop(checkpoint=False, on_safepoint=disable_wd,
                                                                        on_shutdown=on_shutdown,
-                                                                        stop_timeout=self.master_stop_timeout()))
+                                                                        before_shutdown=before_shutdown,
+                                                                        stop_timeout=self.primary_stop_timeout()))
            if not self.state_handler.is_running():
                if self.is_leader() and not status['deleted']:
                    checkpoint_location = self.state_handler.latest_checkpoint_location()
@@ -1616,18 +1834,18 @@ class Ha(object):
    def wakeup(self):
        """Call of this method will trigger the next run of HA loop if there is
        no "active" leader watch request in progress.
-        This usually happens on the master or if the node is running async action"""
+        This usually happens on the leader or if the node is running async action"""
        self.dcs.event.set()

    def get_remote_member(self, member=None):
        """ In case of standby cluster this will tel us from which remote
-            master to stream. Config can be both patroni config or
+            member to stream. Config can be both patroni config or
            cluster.config.data
        """
        cluster_params = self.get_standby_cluster_config()

        if cluster_params:
-            name = member.name if member else 'remote_master:{}'.format(uuid.uuid1())
+            name = member.name if member else 'remote_member:{}'.format(uuid.uuid1())

            data = {k: v for k, v in cluster_params.items() if k in RemoteMember.allowed_keys()}
            data['no_replication_slot'] = 'primary_slot_name' not in cluster_params
@@ -1637,6 +1855,3 @@ class Ha(object):
                data['conn_kwargs'] = conn_kwargs

            return RemoteMember(name, data)
-
-    def get_remote_master(self):
-        return self.get_remote_member()
@@ -19,9 +19,11 @@ from .callback_executor import CallbackExecutor
 from .cancellable import CancellableSubprocess
 from .config import ConfigHandler, mtime
 from .connection import Connection, get_connection_cursor
+from .citus import CitusHandler
 from .misc import parse_history, parse_lsn, postgres_major_version_to_int
 from .postmaster import PostmasterProcess
 from .slots import SlotsHandler
+from .sync import SyncHandler
 from .. import psycopg
 from ..exceptions import PostgresConnectionException
 from ..utils import Retry, RetryFailedError, polling_loop, data_directory_is_empty, parse_int
@@ -54,7 +56,7 @@ class Postgresql(object):
    POSTMASTER_START_TIME = "pg_catalog.pg_postmaster_start_time()"
    TL_LSN = ("CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
              "ELSE ('x' || pg_catalog.substr(pg_catalog.pg_{0}file_name("
-              "pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, "  # master timeline
+              "pg_catalog.pg_current_{0}_{1}()), 1, 8))::bit(32)::int END, "  # primary timeline
              "CASE WHEN pg_catalog.pg_is_in_recovery() THEN 0 "
              "ELSE pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_current_{0}_{1}(), '0/0')::bigint END, "  # write_lsn
              "pg_catalog.pg_{0}_{1}_diff(pg_catalog.pg_last_{0}_replay_{1}(), '0/0')::bigint, "
@@ -75,6 +77,7 @@ class Postgresql(object):

        self._pending_restart = False
        self._connection = Connection()
+        self.citus_handler = CitusHandler(self, config.get('citus'))
        self.config = ConfigHandler(self, config)
        self.config.check_directories()

@@ -84,6 +87,7 @@ class Postgresql(object):
        self.__thread_ident = current_thread().ident

        self.slots_handler = SlotsHandler(self)
+        self.sync_handler = SyncHandler(self)

        self._callback_executor = CallbackExecutor()
        self.__cb_called = False
@@ -106,6 +110,7 @@ class Postgresql(object):
        self._cluster_info_state = {}
        self._has_permanent_logical_slots = True
        self._enforce_hot_standby_feedback = False
+        self._is_synchronous_mode = True
        self._cached_replica_timeline = None

        # Last known running process
@@ -121,7 +126,7 @@ class Postgresql(object):
            ident_saved = self.config.replace_pg_ident()
            if hba_saved or ident_saved:
                self.reload()
-        elif self.role == 'master':
+        elif self.role in ('master', 'primary'):
            self.set_role('demoted')

    @property
@@ -158,11 +163,36 @@ class Postgresql(object):

    @property
    def cluster_info_query(self):
+        """Returns the monitoring query with a fixed number of fields.
+
+        The query text is constructed based on current state in DCS and PostgreSQL version:
+        1. function names depend on version. wal/lsn for v10+ and xlog/location for pre v10.
+        2. for primary we query timeline_id (extracted from pg_walfile_name()) and pg_current_wal_lsn()
+        3. for replicas we query pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), and  pg_is_wal_replay_paused()
+        4. for v9.6+ we query primary_slot_name and primary_conninfo from pg_stat_get_wal_receiver()
+        5. for v11+ with permanent logical slots we query from pg_replication_slots and aggregate the result
+        6. for standby_leader node running v9.6+ we also query pg_control_checkpoint to fetch timeline_id
+        7. if sync replication is enabled we query pg_stat_replication and aggregate the result.
+           In addition to that we get current values of synchronous_commit and synchronous_standby_names GUCs.
+
+        If some conditions are not satisfied we simply put static values instead. E.g., NULL, 0, '', and so on."""
+
+        extra = ", " + (("pg_catalog.current_setting('synchronous_commit'), " +
+                         "pg_catalog.current_setting('synchronous_standby_names'), "
+                         "(SELECT pg_catalog.json_agg(r.*) FROM (SELECT w.pid as pid, application_name, sync_state," +
+                         " pg_catalog.pg_{0}_{1}_diff(write_{1}, '0/0')::bigint AS write_lsn," +
+                         " pg_catalog.pg_{0}_{1}_diff(flush_{1}, '0/0')::bigint AS flush_lsn," +
+                         " pg_catalog.pg_{0}_{1}_diff(replay_{1}, '0/0')::bigint AS replay_lsn " +
+                         "FROM pg_catalog.pg_stat_get_wal_senders() w," +
+                         " pg_catalog.pg_stat_get_activity(w.pid)" +
+                         " WHERE w.state = 'streaming') r)").format(self.wal_name, self.lsn_name)
+                        if self._is_synchronous_mode and self.role in ('master', 'primary') else "'on', '', NULL")
+
        if self._major_version >= 90600:
-            extra = "(SELECT pg_catalog.json_agg(s.*) FROM (SELECT slot_name, slot_type as type, datoid::bigint, " +\
-                    "plugin, catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint" + \
-                    " AS confirmed_flush_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"\
-                            if self._has_permanent_logical_slots and self._major_version >= 110000 else "NULL"
+            extra = ("(SELECT pg_catalog.json_agg(s.*) FROM (SELECT slot_name, slot_type as type, datoid::bigint, " +
+                     "plugin, catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint" +
+                     " AS confirmed_flush_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"
+                     if self._has_permanent_logical_slots and self._major_version >= 110000 else "NULL") + extra
            extra = (", CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END,"
                     " slot_name, conninfo, {0} FROM pg_catalog.pg_stat_get_wal_receiver()").format(extra)
            if self.role == 'standby_leader':
@@ -170,7 +200,7 @@ class Postgresql(object):
            else:
                extra = "0" + extra
        else:
-            extra = "0, NULL, NULL, NULL, NULL"
+            extra = "0, NULL, NULL, NULL, NULL" + extra

        return ("SELECT " + self.TL_LSN + ", {2}").format(self.wal_name, self.lsn_name, extra)

@@ -255,7 +285,8 @@ class Postgresql(object):
        return self._connection.get()

    def set_connection_kwargs(self, kwargs):
-        self._connection.set_conn_kwargs(kwargs)
+        self._connection.set_conn_kwargs(kwargs.copy())
+        self.citus_handler.set_conn_kwargs(kwargs.copy())

    def _query(self, sql, *params):
        """We are always using the same cursor, therefore this method is not thread-safe!!!
@@ -300,7 +331,8 @@ class Postgresql(object):
        return deepcopy(self.config.get(method, {}))

    def replica_method_can_work_without_replication_connection(self, method):
-        return method != 'basebackup' and self.replica_method_options(method).get('no_master')
+        return method != 'basebackup' and (self.replica_method_options(method).get('no_master') or
+                                           self.replica_method_options(method).get('no_leader'))

    def can_create_replica_without_replication_connection(self, replica_methods=None):
        """ go through the replication methods to see if there are ones
@@ -334,13 +366,16 @@ class Postgresql(object):
                self._has_permanent_logical_slots or
                cluster.should_enforce_hot_standby_feedback(self.name, nofailover, self.major_version))

+            self._is_synchronous_mode = cluster.is_synchronous_mode()
+
    def _cluster_info_state_get(self, name):
        if not self._cluster_info_state:
            try:
                result = self._is_leader_retry(self._query, self.cluster_info_query).fetchone()
                cluster_info_state = dict(zip(['timeline', 'wal_position', 'replayed_location',
                                               'received_location', 'replay_paused', 'pg_control_timeline',
-                                               'received_tli', 'slot_name', 'conninfo', 'slots'], result))
+                                               'received_tli', 'slot_name', 'conninfo', 'slots', 'synchronous_commit',
+                                               'synchronous_standby_names', 'pg_stat_replication'], result))
                if self._has_permanent_logical_slots:
                    cluster_info_state['slots'] =\
                        self.slots_handler.process_permanent_slots(cluster_info_state['slots'])
@@ -373,12 +408,21 @@ class Postgresql(object):
    def received_timeline(self):
        return self._cluster_info_state_get('received_tli')

+    def synchronous_commit(self):
+        return self._cluster_info_state_get('synchronous_commit')
+
+    def synchronous_standby_names(self):
+        return self._cluster_info_state_get('synchronous_standby_names')
+
+    def pg_stat_replication(self):
+        return self._cluster_info_state_get('pg_stat_replication') or []
+
    def is_leader(self):
        try:
            return bool(self._cluster_info_state_get('timeline'))
        except PostgresConnectionException:
            logger.warning('Failed to determine PostgreSQL state from the connection, falling back to cached role')
-            return bool(self.is_running() and self.role == 'master')
+            return bool(self.is_running() and self.role in ('master', 'primary'))

    def replay_paused(self):
        return self._cluster_info_state_get('replay_paused')
@@ -461,11 +505,12 @@ class Postgresql(object):

        if self.callback and cb_name in self.callback:
            cmd = self.callback[cb_name]
+            role = 'master' if self.role == 'promoted' else self.role
            try:
-                cmd = shlex.split(self.callback[cb_name]) + [cb_name, self.role, self.scope]
+                cmd = shlex.split(self.callback[cb_name]) + [cb_name, role, self.scope]
                self._callback_executor.call(cmd)
            except Exception:
-                logger.exception('callback %s %s %s %s failed', cmd, cb_name, self.role, self.scope)
+                logger.exception('callback %s %s %s %s failed', cmd, cb_name, role, self.scope)

    @property
    def role(self):
@@ -512,7 +557,7 @@ class Postgresql(object):
        logger.warning("Timed out waiting for PostgreSQL to start")
        return False

-    def start(self, timeout=None, task=None, block_callbacks=False, role=None):
+    def start(self, timeout=None, task=None, block_callbacks=False, role=None, after_start=None):
        """Start PostgreSQL

        Waits for postmaster to open ports or terminate so pg_isready can be used to check startup completion
@@ -583,6 +628,8 @@ class Postgresql(object):

        ret = self.wait_for_startup(start_timeout)
        if ret is not None:
+            if ret and after_start:
+                after_start()
            return ret
        elif timeout is not None:
            return False
@@ -609,7 +656,7 @@ class Postgresql(object):
            return 'not accessible or not healty'

    def stop(self, mode='fast', block_callbacks=False, checkpoint=None,
-             on_safepoint=None, on_shutdown=None, stop_timeout=None):
+             on_safepoint=None, on_shutdown=None, before_shutdown=None, stop_timeout=None):
        """Stop PostgreSQL

        Supports a callback when a safepoint is reached. A safepoint is when no user backend can return a successful
@@ -618,11 +665,13 @@ class Postgresql(object):

        :param on_safepoint: This callback is called when no user backends are running.
        :param on_shutdown: is called when pg_controldata starts reporting `Database cluster state: shut down`
+        :param before_shutdown: is called after running optional CHECKPOINT and before running pg_ctl stop
        """
        if checkpoint is None:
            checkpoint = False if mode == 'immediate' else True

-        success, pg_signaled = self._do_stop(mode, block_callbacks, checkpoint, on_safepoint, on_shutdown, stop_timeout)
+        success, pg_signaled = self._do_stop(mode, block_callbacks, checkpoint, on_safepoint,
+                                             on_shutdown, before_shutdown, stop_timeout)
        if success:
            # block_callbacks is used during restart to avoid
            # running start/stop callbacks in addition to restart ones
@@ -635,7 +684,7 @@ class Postgresql(object):
            self.set_state('stop failed')
        return success

-    def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint, on_shutdown, stop_timeout):
+    def _do_stop(self, mode, block_callbacks, checkpoint, on_safepoint, on_shutdown, before_shutdown, stop_timeout):
        postmaster = self.is_running()
        if not postmaster:
            if on_safepoint:
@@ -648,6 +697,9 @@ class Postgresql(object):
        if not block_callbacks:
            self.set_state('stopping')

+        if before_shutdown:
+            before_shutdown()
+
        # Send signal to postmaster to stop
        success = postmaster.signal_stop(mode, self.pgcommand('pg_ctl'))
        if success is not None:
@@ -775,7 +827,8 @@ class Postgresql(object):

        return self.state == 'running'

-    def restart(self, timeout=None, task=None, block_callbacks=False, role=None):
+    def restart(self, timeout=None, task=None, block_callbacks=False,
+                role=None, before_shutdown=None, after_start=None):
        """Restarts PostgreSQL.

        When timeout parameter is set the call will block either until PostgreSQL has started, failed to start or
@@ -786,7 +839,8 @@ class Postgresql(object):
        self.set_state('restarting')
        if not block_callbacks:
            self.__cb_pending = ACTION_ON_RESTART
-        ret = self.stop(block_callbacks=True) and self.start(timeout, task, True, role)
+        ret = self.stop(block_callbacks=True, before_shutdown=before_shutdown)\
+            and self.start(timeout, task, True, role, after_start)
        if not ret and not self.is_starting():
            self.set_state('restart failed ({0})'.format(self.state))
        return ret
@@ -853,12 +907,13 @@ class Postgresql(object):
        except Exception:
            logger.exception('Can not fetch local timeline and lsn from replication connection')

-    def replica_cached_timeline(self, master_timeline):
-        if not self._cached_replica_timeline or not master_timeline or self._cached_replica_timeline != master_timeline:
+    def replica_cached_timeline(self, primary_timeline):
+        if not self._cached_replica_timeline or not primary_timeline\
+                or self._cached_replica_timeline != primary_timeline:
            self._cached_replica_timeline = self.get_replica_timeline()
        return self._cached_replica_timeline

-    def get_master_timeline(self):
+    def get_primary_timeline(self):
        return self._cluster_info_state_get('timeline')

    def get_history(self, timeline):
@@ -881,11 +936,11 @@ class Postgresql(object):
        recovery_params = self.config.build_recovery_params(member)
        self.config.write_recovery_conf(recovery_params)

-        # When we demoting the master or standby_leader to replica or promoting replica to a standby_leader
+        # When we demoting the primary or standby_leader to replica or promoting replica to a standby_leader
        # and we know for sure that postgres was already running before, we will only execute on_role_change
        # callback and prevent execution of on_restart/on_start callback.
        # If the role remains the same (replica or standby_leader), we will execute on_start or on_restart
-        change_role = self.cb_called and (self.role in ('master', 'demoted') or
+        change_role = self.cb_called and (self.role in ('master', 'primary', 'demoted') or
                                          not {'standby_leader', 'replica'} - {self.role, role})
        if change_role:
            self.__cb_pending = ACTION_NOOP
@@ -911,6 +966,7 @@ class Postgresql(object):
        for _ in polling_loop(wait_seconds):
            data = self.controldata()
            if data.get('Database cluster state') == 'in production':
+                self.set_role('master')
                return True

    def _pre_promote(self):
@@ -928,8 +984,8 @@ class Postgresql(object):
            logger.info('pre_promote script `%s` exited with %s', cmd, ret)
        return ret == 0

-    def promote(self, wait_seconds, task, on_success=None):
-        if self.role == 'master':
+    def promote(self, wait_seconds, task, before_promote=None, on_success=None):
+        if self.role in ('promoted', 'master', 'primary'):
            return True

        ret = self._pre_promote()
@@ -945,11 +1001,15 @@ class Postgresql(object):
            logger.info("PostgreSQL promote cancelled.")
            return False

+        if before_promote is not None:
+            before_promote()
+
        self.slots_handler.on_promote()
+        self.citus_handler.schedule_cache_rebuild()

        ret = self.pg_ctl('promote', '-W')
        if ret:
-            self.set_role('master')
+            self.set_role('promoted')
            if on_success is not None:
                on_success()
            self.call_nowait(ACTION_ON_ROLE_CHANGE)
@@ -1023,13 +1083,15 @@ class Postgresql(object):
    def move_data_directory(self):
        if os.path.isdir(self._data_dir) and not self.is_running():
            try:
-                postfix = time.strftime('%Y-%m-%d-%H-%M-%S')
+                postfix = 'failed'

                # let's see if the wal directory is a symlink, in this case we
                # should move the target
                for (source, pg_wal_realpath) in self.pg_wal_realpath().items():
                    logger.info('renaming WAL directory and updating symlink: %s', pg_wal_realpath)
-                    new_name = '{0}_{1}'.format(pg_wal_realpath, postfix)
+                    new_name = '{0}.{1}'.format(pg_wal_realpath, postfix)
+                    if os.path.exists(new_name):
+                        shutil.rmtree(new_name)
                    os.rename(pg_wal_realpath, new_name)
                    os.unlink(source)
                    os.symlink(new_name, source)
@@ -1037,13 +1099,17 @@ class Postgresql(object):
                # Move user defined tablespace directory
                for (source, pg_tsp_rpath) in self.pg_tblspc_realpaths().items():
                    logger.info('renaming user defined tablespace directory and updating symlink: %s', pg_tsp_rpath)
-                    new_name = '{0}_{1}'.format(pg_tsp_rpath, postfix)
+                    new_name = '{0}.{1}'.format(pg_tsp_rpath, postfix)
+                    if os.path.exists(new_name):
+                        shutil.rmtree(new_name)
                    os.rename(pg_tsp_rpath, new_name)
                    os.unlink(source)
                    os.symlink(new_name, source)

-                new_name = '{0}_{1}'.format(self._data_dir, postfix)
+                new_name = '{0}.{1}'.format(self._data_dir, postfix)
                logger.info('renaming data directory to %s', new_name)
+                if os.path.exists(new_name):
+                    shutil.rmtree(new_name)
                os.rename(self._data_dir, new_name)
            except OSError:
                logger.exception("Could not rename data directory %s", self._data_dir)
@@ -1076,58 +1142,6 @@ class Postgresql(object):
            logger.exception('Could not remove data directory %s', self._data_dir)
            self.move_data_directory()

-    def _get_synchronous_commit_param(self):
-        return self.query("SHOW synchronous_commit").fetchone()[0]
-
-    def pick_synchronous_standby(self, cluster, sync_node_count=1, sync_node_maxlag=-1):
-        """Finds the best candidate to be the synchronous standby.
-
-        Current synchronous standby is always preferred, unless it has disconnected or does not want to be a
-        synchronous standby any longer.
-        Parameter sync_node_maxlag(maximum_lag_on_syncnode) would help swapping unhealthy sync replica in case
-        if it stops responding (or hung). Please set the value high enough so it won't unncessarily swap sync
-        standbys during high loads. Any less or equal of 0 value keep the behavior backward compatible and
-        will not swap. Please note that it will not also swap sync standbys in case where all replicas are hung.
-
-        :returns tuple of candidates list and synchronous standby list.
-        """
-        if self._major_version < 90600:
-            sync_node_count = 1
-        members = {m.name.lower(): m for m in cluster.members}
-        candidates = []
-        sync_nodes = []
-        replica_list = []
-        # Pick candidates based on who has higher replay/remote_write/flush lsn.
-        sync_commit_par = self._get_synchronous_commit_param()
-        sort_col = {'remote_apply': 'replay', 'remote_write': 'write'}.get(sync_commit_par, 'flush')
-        # pg_stat_replication.sync_state has 4 possible states - async, potential, quorum, sync.
-        # Sort clause "ORDER BY sync_state DESC" is to get the result in required order and to keep
-        # the result consistent in case if a synchronous standby member is slowed down OR async node
-        # receiving changes faster than the sync member (very rare but possible). Such cases would
-        # trigger sync standby member swapping frequently and the sort on sync_state desc should
-        # help in keeping the query result consistent.
-        for app_name, sync_state, replica_lsn in self.query(
-                "SELECT pg_catalog.lower(application_name), sync_state, pg_{2}_{1}_diff({0}_{1}, '0/0')::bigint"
-                " FROM pg_catalog.pg_stat_replication"
-                " WHERE state = 'streaming' AND {0}_{1} IS NOT NULL"
-                " ORDER BY sync_state DESC, {0}_{1} DESC".format(sort_col, self.lsn_name, self.wal_name)):
-            member = members.get(app_name)
-            if member and not member.tags.get('nosync', False):
-                replica_list.append((member.name, sync_state, replica_lsn, bool(member.nofailover)))
-
-        max_lsn = max(replica_list, key=lambda x: x[2])[2] if len(replica_list) > 1 else int(str(self.last_operation()))
-
-        # Prefer members without nofailover tag. We are relying on the fact that sorts are guaranteed to be stable.
-        for app_name, sync_state, replica_lsn, _ in sorted(replica_list, key=lambda x: x[3]):
-            if sync_node_maxlag <= 0 or max_lsn - replica_lsn <= sync_node_maxlag:
-                candidates.append(app_name)
-                if sync_state == 'sync':
-                    sync_nodes.append(app_name)
-            if len(candidates) >= sync_node_count:
-                break
-
-        return candidates, sync_nodes
-
    def schedule_sanity_checks_after_pause(self):
        """
            After coming out of pause we have to:
@@ -1138,4 +1152,5 @@ class Postgresql(object):
        if not self._major_version:
            self.configure_server_parameters()
        self.slots_handler.schedule()
+        self.citus_handler.schedule_cache_rebuild()
        self._sysid = None
@@ -155,12 +155,12 @@ class Bootstrap(object):
        self._postgresql.set_state('creating replica')
        self._postgresql.schedule_sanity_checks_after_pause()

-        is_remote_master = isinstance(clone_member, RemoteMember)
+        is_remote_member = isinstance(clone_member, RemoteMember)

        # get list of replica methods either from clone member or from
        # the config. If there is no configuration key, or no value is
        # specified, use basebackup
-        replica_methods = (clone_member.create_replica_methods if is_remote_master
+        replica_methods = (clone_member.create_replica_methods if is_remote_member
                           else self._postgresql.create_replica_methods) or ['basebackup']

        if clone_member and clone_member.conn_url:
@@ -212,7 +212,7 @@ class Bootstrap(object):
                                          "datadir": self._postgresql.data_dir,
                                          "connstring": connstring})
                else:
-                    for param in ('no_params', 'no_master', 'keep_data'):
+                    for param in ('no_params', 'no_master', 'no_leader', 'keep_data'):
                        method_config.pop(param, None)
                params = ["--{0}={1}".format(arg, val) for arg, val in method_config.items()]
                try:
@@ -269,7 +269,7 @@ class Bootstrap(object):

    def clone(self, clone_member):
        """
-             - initialize the replica from an existing member (master or replica)
+             - initialize the replica from an existing member (primary or replica)
             - initialize the replica using the replica creation method that
               works without the replication connection (i.e. restore from on-disk
               base backup)
@@ -345,7 +345,7 @@ END;$$""".format(quote_literal(name), quote_ident(name, self._postgresql.connect
 BEGIN
    SET local synchronous_commit = 'local';
    GRANT EXECUTE ON function pg_catalog.{0} TO {1};
-END;$$""".format(f, quote_ident(rewind['username'], self._postgresql.connection()))
+END;$$""".format(f, quote_ident(rewind['username'], postgresql.connection()))
                        postgresql.query(sql)

                for name, value in (config.get('users') or {}).items():
@@ -377,6 +377,9 @@ END;$$""".format(f, quote_ident(rewind['username'], self._postgresql.connection(
                            postgresql.reload()
                            time.sleep(1)  # give a time to postgres to "reload" configuration files
                            postgresql.connection().close()  # close connection to reconnect with a new password
+                else:  # initdb
+                    # We may want create database and extension for citus
+                    self._postgresql.citus_handler.bootstrap()
        except Exception:
            logger.exception('post_bootstrap')
            task.complete(False)
@@ -0,0 +1,389 @@
+import logging
+import re
+import time
+
+from six.moves.urllib_parse import urlparse
+from threading import Condition, Event, Thread
+
+from .connection import Connection
+from ..dcs import CITUS_COORDINATOR_GROUP_ID
+from ..psycopg import connect, quote_ident
+
+CITUS_SLOT_NAME_RE = re.compile(r'^citus_shard_(move|split)_slot(_[1-9][0-9]*){2,3}$')
+logger = logging.getLogger(__name__)
+
+
+class PgDistNode(object):
+    """Represents a single row in the `pg_dist_node` table"""
+
+    def __init__(self, group, host, port, event, nodeid=None, timeout=None, cooldown=None):
+        self.group = group
+        # A weird way of pausing client connections by adding the `-demoted` suffix to the hostname
+        self.host = host + ('-demoted' if event == 'before_demote' else '')
+        self.port = port
+        # Event that is trying to change or changed the given row.
+        # Possible values: before_demote, before_promote, after_promote.
+        self.event = event
+        self.nodeid = nodeid
+
+        # If transaction was started, we need to COMMIT/ROLLBACK before the deadline
+        self.timeout = timeout
+        self.cooldown = cooldown or 10000  # 10s by default
+        self.deadline = 0
+
+        # All changes in the pg_dist_node are serialized on the Patroni
+        # side by performing them from a thread. The thread, that is
+        # requested a change, sometimes needs to wait for a result.
+        # For example, we want to pause client connections before demoting
+        # the worker, and once it is done notify the calling thread.
+        self._event = Event()
+
+    def wait(self):
+        self._event.wait()
+
+    def wakeup(self):
+        self._event.set()
+
+    def __eq__(self, other):
+        return isinstance(other, PgDistNode) and self.event == other.event\
+            and self.host == other.host and self.port == other.port
+
+    def __ne__(self, other):
+        return not self == other
+
+    def __str__(self):
+        return ('PgDistNode(nodeid={0},group={1},host={2},port={3},event={4})'
+                .format(self.nodeid, self.group, self.host, self.port, self.event))
+
+    def __repr__(self):
+        return str(self)
+
+
+class CitusHandler(Thread):
+
+    def __init__(self, postgresql, config):
+        super(CitusHandler, self).__init__()
+        self.daemon = True
+        self._postgresql = postgresql
+        self._config = config
+        self._connection = Connection()
+        self._pg_dist_node = {}  # Cache of pg_dist_node: {groupid: PgDistNode()}
+        self._tasks = []  # Requests to change pg_dist_node, every task is a `PgDistNode`
+        self._condition = Condition()  # protects _pg_dist_node, _tasks, and _schedule_load_pg_dist_node
+        self._in_flight = None  # Reference to the `PgDistNode` if there is a transaction in progress changing it
+        self.schedule_cache_rebuild()
+
+    def is_enabled(self):
+        return isinstance(self._config, dict)
+
+    def group(self):
+        return self._config['group']
+
+    def is_coordinator(self):
+        return self.is_enabled() and self.group() == CITUS_COORDINATOR_GROUP_ID
+
+    def is_worker(self):
+        return self.is_enabled() and not self.is_coordinator()
+
+    def set_conn_kwargs(self, kwargs):
+        if self.is_enabled():
+            kwargs.update({'dbname': self._config['database'],
+                           'options': '-c statement_timeout=0 -c idle_in_transaction_session_timeout=0'})
+            self._connection.set_conn_kwargs(kwargs)
+
+    def schedule_cache_rebuild(self):
+        with self._condition:
+            self._schedule_load_pg_dist_node = True
+
+    def on_demote(self):
+        with self._condition:
+            self._pg_dist_node.clear()
+            self._tasks[:] = []
+            self._in_flight = None
+
+    def query(self, sql, *params):
+        try:
+            logger.debug('query(%s, %s)', sql, params)
+            cursor = self._connection.cursor()
+            cursor.execute(sql, params or None)
+            return cursor
+        except Exception as e:
+            logger.error('Exception when executing query "%s", (%s): %r', sql, params, e)
+            self._connection.close()
+            self._in_flight = None
+            self.schedule_cache_rebuild()
+            raise e
+
+    def load_pg_dist_node(self):
+        """Read from the `pg_dist_node` table and put it into the local cache"""
+
+        with self._condition:
+            if not self._schedule_load_pg_dist_node:
+                return True
+            self._schedule_load_pg_dist_node = False
+
+        try:
+            cursor = self.query("SELECT nodeid, groupid, nodename, nodeport, noderole"
+                                " FROM pg_catalog.pg_dist_node WHERE noderole = 'primary'")
+        except Exception:
+            return False
+
+        with self._condition:
+            self._pg_dist_node = {r[1]: PgDistNode(r[1], r[2], r[3], 'after_promote', r[0]) for r in cursor}
+        return True
+
+    def sync_pg_dist_node(self, cluster):
+        """Maintain the `pg_dist_node` from the coordinator leader every heartbeat loop.
+
+        We can't always rely on REST API calls from worker nodes in order
+        to maintain `pg_dist_node`, therefore at least once per heartbeat
+        loop we make sure that workes registered in `self._pg_dist_node`
+        cache are matching the cluster view from DCS by creating tasks
+        the same way as it is done from the REST API."""
+
+        if not self.is_coordinator():
+            return
+
+        with self._condition:
+            if not self.is_alive():
+                self.start()
+
+        self.add_task('after_promote', CITUS_COORDINATOR_GROUP_ID, self._postgresql.connection_string)
+
+        for group, worker in cluster.workers.items():
+            leader = worker.leader
+            if leader and leader.conn_url\
+                    and leader.data.get('role') in ('master', 'primary') and leader.data.get('state') == 'running':
+                self.add_task('after_promote', group, leader.conn_url)
+
+    def find_task_by_group(self, group):
+        for i, task in enumerate(self._tasks):
+            if task.group == group:
+                return i
+
+    def pick_task(self):
+        """Returns the tuple(i, task), where `i` - is the task index in the self._tasks list
+
+        Tasks are picked by following priorities:
+        1. If there is already a transaction in progress, pick a task
+           that that will change already affected worker primary.
+        2. If the coordinator address should be changed - pick a task
+           with group=0 (coordinators are always in group 0).
+        3. Pick a task that is the oldest (first from the self._tasks)"""
+
+        with self._condition:
+            if self._in_flight:
+                i = self.find_task_by_group(self._in_flight.group)
+            else:
+                while True:
+                    i = self.find_task_by_group(CITUS_COORDINATOR_GROUP_ID)  # set_coordinator
+                    if i is None and self._tasks:
+                        i = 0
+                    if i is None:
+                        break
+                    task = self._tasks[i]
+                    if task == self._pg_dist_node.get(task.group):
+                        self._tasks.pop(i)  # nothing to do because cached version of pg_dist_node already matches
+                    else:
+                        break
+            task = self._tasks[i] if i is not None else None
+
+            # When tasks are added it could happen that self._pg_dist_node
+            # wasn't ready (self._schedule_load_pg_dist_node is False)
+            # and hence the nodeid wasn't filled.
+            if task and task.group in self._pg_dist_node:
+                task.nodeid = self._pg_dist_node[task.group].nodeid
+            return i, task
+
+    def update_node(self, task):
+        if task.group == CITUS_COORDINATOR_GROUP_ID:
+            return self.query("SELECT pg_catalog.citus_set_coordinator_host(%s, %s, 'primary', 'default')",
+                              task.host, task.port)
+
+        if task.nodeid is None and task.event != 'before_demote':
+            task.nodeid = self.query("SELECT pg_catalog.citus_add_node(%s, %s, %s, 'primary', 'default')",
+                                     task.host, task.port, task.group).fetchone()[0]
+        elif task.nodeid is not None:
+            # XXX: statement_timeout?
+            self.query('SELECT pg_catalog.citus_update_node(%s, %s, %s, true, %s)',
+                       task.nodeid, task.host, task.port, task.cooldown)
+
+    def process_task(self, task):
+        """Updates a single row in `pg_dist_node` table, optionally in a transaction.
+
+        The transaction is started if we do a demote of the worker node
+        or before promoting the other worker if there is not transaction
+        in progress. And, the transaction it is committed when the
+        switchover/failover completed.
+
+        This method returns `True` if node was updated (optionally,
+        transaction was committed) as an indicator that
+        the `self._pg_dist_node` cache should be updated.
+
+        The maximum lifetime of the transaction in progress
+        is controlled outside of this method."""
+
+        if task.event == 'after_promote':
+            # The after_promote may happen without previous before_demote and/or
+            # before_promore.  In this case we just call self.update_node() method.
+            # If there is a transaction in progress, it could be that it already did
+            # required changes and we can simply COMMIT.
+            if not self._in_flight or self._in_flight.host != task.host or self._in_flight.port != task.port:
+                self.update_node(task)
+            if self._in_flight:
+                self.query('COMMIT')
+                self._in_flight = None
+            return True
+        else:  # before_demote, before_promote
+            if task.timeout:
+                task.deadline = time.time() + task.timeout
+            if not self._in_flight:
+                self.query('BEGIN')
+            self.update_node(task)
+            self._in_flight = task
+        return False
+
+    def process_tasks(self):
+        while True:
+            if not self._in_flight and not self.load_pg_dist_node():
+                break
+
+            i, task = self.pick_task()
+            if not task:
+                break
+            try:
+                update_cache = self.process_task(task)
+            except Exception as e:
+                logger.error('Exception when working with pg_dist_node: %r', e)
+                update_cache = False
+            with self._condition:
+                if self._tasks:
+                    if update_cache:
+                        self._pg_dist_node[task.group] = task
+                    if id(self._tasks[i]) == id(task):
+                        self._tasks.pop(i)
+            task.wakeup()
+
+    def run(self):
+        while True:
+            try:
+                with self._condition:
+                    if self._schedule_load_pg_dist_node:
+                        timeout = -1
+                    elif self._in_flight:
+                        timeout = self._in_flight.deadline - time.time() if self._tasks else None
+                    else:
+                        timeout = -1 if self._tasks else None
+
+                    if timeout is None or timeout > 0:
+                        self._condition.wait(timeout)
+                    elif self._in_flight:
+                        logger.warning('Rolling back transaction. Last known status: %s', self._in_flight)
+                        self.query('ROLLBACK')
+                        self._in_flight = None
+                self.process_tasks()
+            except Exception:
+                logger.exception('run')
+
+    def _add_task(self, task):
+        with self._condition:
+            i = self.find_task_by_group(task.group)
+
+            # task.timeout is None is an indicator that it was scheduled
+            # from the sync_pg_dist_node() and we don't want to override
+            # already existing task created from REST API.
+            if task.timeout is None and (i is not None or self._in_flight and self._in_flight.group == task.group):
+                return False
+
+            # Override already existing task for the same worker group
+            if i is not None:
+                if task != self._tasks[i]:
+                    logger.debug('Overriding existing task: %s != %s', self._tasks[i], task)
+                    self._tasks[i] = task
+                    self._condition.notify()
+                    return True
+            # Add the task to the list if Worker node state is different from the cached `pg_dist_node`
+            elif self._schedule_load_pg_dist_node or task != self._pg_dist_node.get(task.group)\
+                    or self._in_flight and task.group == self._in_flight.group:
+                logger.debug('Adding the new task: %s', task)
+                self._tasks.append(task)
+                self._condition.notify()
+                return True
+        return False
+
+    def add_task(self, event, group, conn_url, timeout=None, cooldown=None):
+        try:
+            r = urlparse(conn_url)
+        except Exception as e:
+            return logger.error('Failed to parse connection url %s: %r', conn_url, e)
+        host = r.hostname
+        port = r.port or 5432
+        task = PgDistNode(group, host, port, event, timeout=timeout, cooldown=cooldown)
+        return task if self._add_task(task) else None
+
+    def handle_event(self, cluster, event):
+        if not self.is_alive():
+            return
+
+        cluster = cluster.workers.get(event['group'])
+        if not (cluster and cluster.leader and cluster.leader.name == event['leader'] and cluster.leader.conn_url):
+            return
+
+        task = self.add_task(event['type'], event['group'],
+                             cluster.leader.conn_url,
+                             event['timeout'], event['cooldown']*1000)
+        if task and event['type'] == 'before_demote':
+            task.wait()
+
+    def bootstrap(self):
+        if not self.is_enabled():
+            return
+
+        conn_kwargs = self._postgresql.config.local_connect_kwargs
+        conn_kwargs['options'] = '-c synchronous_commit=local -c statement_timeout=0'
+        if self._config['database'] != self._postgresql.database:
+            conn = connect(**conn_kwargs)
+            try:
+                with conn.cursor() as cur:
+                    cur.execute('CREATE DATABASE {0}'.format(quote_ident(self._config['database'], conn)))
+            finally:
+                conn.close()
+
+        conn_kwargs['dbname'] = self._config['database']
+        conn = connect(**conn_kwargs)
+        try:
+            with conn.cursor() as cur:
+                cur.execute('CREATE EXTENSION citus')
+
+                superuser = self._postgresql.config.superuser
+                params = {k: superuser[k] for k in ('password', 'sslcert', 'sslkey') if k in superuser}
+                if params:
+                    cur.execute("INSERT INTO pg_catalog.pg_dist_authinfo VALUES"
+                                "(0, pg_catalog.current_user(), %s)",
+                                (self._postgresql.config.format_dsn(params),))
+        finally:
+            conn.close()
+
+    def adjust_postgres_gucs(self, parameters):
+        if not self.is_enabled():
+            return
+
+        # citus extension must be on the first place in shared_preload_libraries
+        shared_preload_libraries = list(filter(
+            lambda el: el and el != 'citus',
+            [p.strip() for p in parameters.get('shared_preload_libraries', '').split(',')]))
+        parameters['shared_preload_libraries'] = ','.join(['citus'] + shared_preload_libraries)
+
+        # if not explicitly set Citus overrides max_prepared_transactions to max_connections*2
+        if parameters.get('max_prepared_transactions') == 0:
+            parameters['max_prepared_transactions'] = parameters['max_connections'] * 2
+
+        # Resharding in Citus implemented using logical replication
+        parameters['wal_level'] = 'logical'
+
+    def ignore_replication_slot(self, slot):
+        if self.is_enabled() and self._postgresql.is_leader() and\
+                slot['type'] == 'logical' and slot['database'] == self._config['database']:
+            m = CITUS_SLOT_NAME_RE.match(slot['name'])
+            return m and {'move': 'pgoutput', 'split': 'citus'}.get(m.group(1)) == slot['plugin']
+        return False
@@ -12,21 +12,14 @@ from .validator import CaseInsensitiveDict, recovery_parameters,\
        transform_postgresql_parameter_value, transform_recovery_parameter_value
 from ..dcs import slot_name_from_member_name, RemoteMember
 from ..exceptions import PatroniFatalException
-from ..psycopg import quote_ident as _quote_ident
 from ..utils import compare_values, parse_bool, parse_int, split_host_port, uri, \
        validate_directory, is_subpath

 logger = logging.getLogger(__name__)

-SYNC_STANDBY_NAME_RE = re.compile(r'^[A-Za-z_][A-Za-z_0-9\$]*$')
 PARAMETER_RE = re.compile(r'([a-z_]+)\s*=\s*')


-def quote_ident(value):
-    """Very simplified version of quote_ident"""
-    return value if SYNC_STANDBY_NAME_RE.match(value) else _quote_ident(value)
-
-
 def conninfo_uri_parse(dsn):
    ret = {}
    r = urlparse(dsn)
@@ -536,24 +529,24 @@ class ConfigHandler(object):
            recovery_params.update({'recovery_target': '', 'recovery_target_name': '', 'recovery_target_time': '',
                                    'recovery_target_xid': '', 'recovery_target_lsn': ''})

-        is_remote_master = isinstance(member, RemoteMember)
+        is_remote_member = isinstance(member, RemoteMember)
        primary_conninfo = self.primary_conninfo_params(member)
        if primary_conninfo:
            use_slots = self.get('use_slots', True) and self._postgresql.major_version >= 90400
-            if use_slots and not (is_remote_master and member.no_replication_slot):
-                primary_slot_name = member.primary_slot_name if is_remote_master else self._postgresql.name
+            if use_slots and not (is_remote_member and member.no_replication_slot):
+                primary_slot_name = member.primary_slot_name if is_remote_member else self._postgresql.name
                recovery_params['primary_slot_name'] = slot_name_from_member_name(primary_slot_name)
                # We are a standby leader and are using a replication slot. Make sure we connect to
                # the leader of the main cluster (in case more than one host is specified in the
                # connstr) by adding 'target_session_attrs=read-write' to primary_conninfo.
-                if is_remote_master and 'target_sesions_attrs' not in primary_conninfo and\
+                if is_remote_member and 'target_sesions_attrs' not in primary_conninfo and\
                        self._postgresql.major_version >= 100000:
                    primary_conninfo['target_session_attrs'] = 'read-write'
            recovery_params['primary_conninfo'] = primary_conninfo

        # standby_cluster config might have different parameters, we want to override them
        standby_cluster_params = ['restore_command', 'archive_cleanup_command']\
-            + (['recovery_min_apply_delay'] if is_remote_master else [])
+            + (['recovery_min_apply_delay'] if is_remote_member else [])
        recovery_params.update({p: member.data.get(p) for p in standby_cluster_params if member and member.data.get(p)})
        return recovery_params

@@ -869,6 +862,9 @@ class ConfigHandler(object):
        elif self._postgresql.major_version:
            wal_keep_size = parse_int(parameters.pop('wal_keep_size', self.CMDLINE_OPTIONS['wal_keep_size'][0]), 'MB')
            parameters.setdefault('wal_keep_segments', int((wal_keep_size + 8) / 16))
+
+        self._postgresql.citus_handler.adjust_postgres_gucs(parameters)
+
        ret = CaseInsensitiveDict({k: v for k, v in parameters.items() if not self._postgresql.major_version or
                                   self._postgresql.major_version >= self.CMDLINE_OPTIONS.get(k, (0, 1, 90100))[2]})
        ret.update({k: os.path.join(self._config_dir, ret[k]) for k in ('hba_file', 'ident_file') if k in ret})
@@ -1044,23 +1040,19 @@ class ConfigHandler(object):
        else:
            logger.info('No PostgreSQL configuration items changed, nothing to reload.')

-    def set_synchronous_standby(self, sync_members):
-        """Sets a node to be synchronous standby and if changed does a reload for PostgreSQL."""
-        if sync_members and sync_members != ['*']:
-            sync_members = [quote_ident(x) for x in sync_members]
-        if self._postgresql.major_version >= 90600 and len(sync_members) > 1:
-            sync_param = '{0} ({1})'.format(len(sync_members), ','.join(sync_members))
-        else:
-            sync_param = next(iter(sync_members), None)
-        if sync_param != self._synchronous_standby_names:
-            if sync_param is None:
+    def set_synchronous_standby_names(self, value):
+        """Updates synchronous_standby_names and reloads if necessary.
+        :returns: True if value was updated."""
+        if value != self._synchronous_standby_names:
+            if value is None:
                self._server_parameters.pop('synchronous_standby_names', None)
            else:
-                self._server_parameters['synchronous_standby_names'] = sync_param
-            self._synchronous_standby_names = sync_param
+                self._server_parameters['synchronous_standby_names'] = value
+            self._synchronous_standby_names = value
            if self._postgresql.state == 'running':
                self.write_postgresql_conf()
                self._postgresql.reload()
+            return True

    @property
    def effective_configuration(self):
@@ -1073,7 +1065,7 @@ class ConfigHandler(object):
        As a workaround we will start it with the values from controldata and set `pending_restart`
        to true as an indicator that current values of parameters are not matching expectations."""

-        if self._postgresql.role == 'master':
+        if self._postgresql.role in ('master', 'primary'):
            return self._server_parameters

        options_mapping = {
@@ -1,4 +1,6 @@
+import errno
 import logging
+import os

 from patroni.exceptions import PostgresException

@@ -73,3 +75,16 @@ def parse_history(data):
 def format_lsn(lsn, full=False):
    template = '{0:X}/{1:08X}' if full else '{0:X}/{1:X}'
    return template.format(lsn >> 32, lsn & 0xFFFFFFFF)
+
+
+def fsync_dir(path):
+    if os.name != 'nt':
+        fd = os.open(path, os.O_DIRECTORY)
+        try:
+            os.fsync(fd)
+        except OSError as e:
+            # Some filesystems don't like fsyncing directories and raise EINVAL. Ignoring it is usually safe.
+            if e.errno != errno.EINVAL:
+                raise
+        finally:
+            os.close(fd)
@@ -9,7 +9,7 @@ import subprocess
 from threading import Lock, Thread

 from .connection import get_connection_cursor
-from .misc import format_lsn, parse_history, parse_lsn
+from .misc import format_lsn, fsync_dir, parse_history, parse_lsn
 from ..async_executor import CriticalTask
 from ..dcs import Leader

@@ -126,7 +126,7 @@ class Rewind(object):
                in_recovery = True
                lsn = data.get('Minimum recovery ending location')
                timeline = int(data.get("Min recovery ending loc's timeline"))
-                if lsn == '0/0' or timeline == 0:  # it was a master when it crashed
+                if lsn == '0/0' or timeline == 0:  # it was a primary when it crashed
                    data['Database cluster state'] = 'shut down'
            if data.get('Database cluster state') == 'shut down':
                in_recovery = False
@@ -157,7 +157,7 @@ class Rewind(object):
        return in_recovery, timeline, lsn

    @staticmethod
-    def _log_master_history(history, i):
+    def _log_primary_history(history, i):
        start = max(0, i - 3)
        end = None if i + 4 >= len(history) else i + 2
        history_show = []
@@ -172,7 +172,7 @@ class Rewind(object):
            history_show.append('...')
            history_show.append(format_history_line(history[-1]))

-        logger.info('master: history=%s', '\n'.join(history_show))
+        logger.info('primary: history=%s', '\n'.join(history_show))

    def _conn_kwargs(self, member, auth):
        ret = member.conn_kwargs(auth)
@@ -189,7 +189,7 @@ class Rewind(object):
        if local_timeline is None or local_lsn is None:
            return

-        if isinstance(leader, Leader) and leader.member.data.get('role') != 'master':
+        if isinstance(leader, Leader) and leader.member.data.get('role') not in ('master', 'primary'):
            return

        # We want to use replication credentials when connecting to the "postgres" database in case if
@@ -206,20 +206,20 @@ class Rewind(object):
        try:
            with self._postgresql.get_replication_connection_cursor(**leader.conn_kwargs()) as cur:
                cur.execute('IDENTIFY_SYSTEM')
-                master_timeline = cur.fetchone()[1]
-                logger.info('master_timeline=%s', master_timeline)
-                if local_timeline > master_timeline:  # Not always supported by pg_rewind
+                primary_timeline = cur.fetchone()[1]
+                logger.info('primary_timeline=%s', primary_timeline)
+                if local_timeline > primary_timeline:  # Not always supported by pg_rewind
                    need_rewind = True
-                elif local_timeline == master_timeline:
+                elif local_timeline == primary_timeline:
                    need_rewind = False
-                elif master_timeline > 1:
-                    cur.execute('TIMELINE_HISTORY {0}'.format(master_timeline))
+                elif primary_timeline > 1:
+                    cur.execute('TIMELINE_HISTORY {0}'.format(primary_timeline))
                    history = cur.fetchone()[1]
                    if not isinstance(history, six.string_types):
                        history = bytes(history).decode('utf-8')
-                    logger.debug('master: history=%s', history)
+                    logger.debug('primary: history=%s', history)
        except Exception:
-            return logger.exception('Exception when working with master via replication connection')
+            return logger.exception('Exception when working with primary via replication connection')

        if history is not None:
            history = list(parse_history(history))
@@ -240,7 +240,7 @@ class Rewind(object):
                    break
            else:
                need_rewind = True
-            self._log_master_history(history, i)
+            self._log_primary_history(history, i)

        self._state = need_rewind and REWIND_STATUS.NEED or REWIND_STATUS.NOT_NEED

@@ -270,7 +270,7 @@ class Rewind(object):
                        if self._checkpoint_task.result is not None:
                            self._state = REWIND_STATUS.CHECKPOINT
                            self._checkpoint_task = None
-                elif self._postgresql.get_master_timeline() == self._postgresql.pg_control_timeline():
+                elif self._postgresql.get_primary_timeline() == self._postgresql.pg_control_timeline():
                    self._state = REWIND_STATUS.CHECKPOINT
                else:
                    self._checkpoint_task = CriticalTask()
@@ -362,6 +362,18 @@ class Rewind(object):
                else:
                    logger.info('Failed to archive WAL segment %s', wal)

+    def _maybe_clean_pg_replslot(self):
+        """Clean pg_replslot directory if pg version is less then 11
+        (pg_rewind deletes $PGDATA/pg_replslot content only since pg11)."""
+        if self._postgresql.major_version < 110000:
+            replslot_dir = self._postgresql.slots_handler.pg_replslot_dir
+            try:
+                for f in os.listdir(replslot_dir):
+                    shutil.rmtree(os.path.join(replslot_dir, f))
+                fsync_dir(replslot_dir)
+            except Exception as e:
+                logger.warning('Unable to clean %s: %r', replslot_dir, e)
+
    def pg_rewind(self, r):
        # prepare pg_rewind connection
        env = self._postgresql.config.write_pgpass(r)
@@ -419,12 +431,12 @@ class Rewind(object):
        # prepare pg_rewind connection
        r = self._conn_kwargs(leader, self._postgresql.config.rewind_credentials)

-        # 1. make sure that we are really trying to rewind from the master
+        # 1. make sure that we are really trying to rewind from the primary
        # 2. make sure that pg_control contains the new timeline by:
        #   running a checkpoint or
-        #   waiting until Patroni on the master will expose checkpoint_after_promote=True
+        #   waiting until Patroni on the primary will expose checkpoint_after_promote=True
        checkpoint_status = leader.checkpoint_after_promote if isinstance(leader, Leader) else None
-        if checkpoint_status is None:  # we are the standby-cluster leader or master still runs the old Patroni
+        if checkpoint_status is None:  # we are the standby-cluster leader or primary still runs the old Patroni
            # superuser credentials match rewind_credentials if the latter are not provided or we run 10 or older
            if self._postgresql.config.superuser == self._postgresql.config.rewind_credentials:
                leader_status = self._postgresql.checkpoint(
@@ -439,14 +451,15 @@ class Rewind(object):
            return

        if self.pg_rewind(r):
+            self._maybe_clean_pg_replslot()
            self._state = REWIND_STATUS.SUCCESS
        else:
            if not self.check_leader_is_not_in_recovery(r):
-                logger.warning('Failed to rewind because master %s become unreachable', leader.name)
+                logger.warning('Failed to rewind because primary %s become unreachable', leader.name)
                if not self.can_rewind:  # It is possible that the previous attempt damaged pg_control file!
                    self._state = REWIND_STATUS.FAILED
            else:
-                logger.error('Failed to rewind from healty master: %s', leader.name)
+                logger.error('Failed to rewind from healty primary: %s', leader.name)
                self._state = REWIND_STATUS.FAILED

            if self.failed:
@@ -1,4 +1,3 @@
-import errno
 import logging
 import os
 import shutil
@@ -8,7 +7,7 @@ from contextlib import contextmanager
 from threading import Condition, Thread

 from .connection import get_connection_cursor
-from .misc import format_lsn
+from .misc import format_lsn, fsync_dir
 from ..psycopg import OperationalError

 logger = logging.getLogger(__name__)
@@ -19,19 +18,6 @@ def compare_slots(s1, s2, dbid='database'):
                                         s1.get(dbid) == s2.get(dbid) and s1['plugin'] == s2['plugin'])


-def fsync_dir(path):
-    if os.name != 'nt':
-        fd = os.open(path, os.O_DIRECTORY)
-        try:
-            os.fsync(fd)
-        except OSError as e:
-            # Some filesystems don't like fsyncing directories and raise EINVAL. Ignoring it is usually safe.
-            if e.errno != errno.EINVAL:
-                raise
-        finally:
-            os.close(fd)
-
-
 class SlotsAdvanceThread(Thread):

    def __init__(self, slots_handler):
@@ -122,6 +108,7 @@ class SlotsHandler(object):
        self._advance = None
        self._replication_slots = {}  # already existing replication slots
        self._unready_logical_slots = {}
+        self.pg_replslot_dir = os.path.join(self._postgresql.data_dir, 'pg_replslot')
        self.schedule()

    def _query(self, sql, *params):
@@ -189,26 +176,36 @@ class SlotsHandler(object):
            if ((matcher.get("name") is None or matcher["name"] == name)
               and all(not matcher.get(a) or matcher[a] == slot.get(a) for a in ('database', 'plugin', 'type'))):
                return True
-        return False
+        return self._postgresql.citus_handler.ignore_replication_slot(slot)

    def drop_replication_slot(self, name):
-        cursor = self._query(('SELECT pg_catalog.pg_drop_replication_slot(%s) WHERE EXISTS (SELECT 1 ' +
-                              'FROM pg_catalog.pg_replication_slots WHERE slot_name = %s AND NOT active)'), name, name)
-        # In normal situation rowcount should be 1, otherwise either slot doesn't exists or it is still active
-        return cursor.rowcount == 1
+        """Returns a tuple(active, dropped)"""
+        cursor = self._query(('WITH slots AS (SELECT slot_name, active' +
+                              ' FROM pg_catalog.pg_replication_slots WHERE slot_name = %s),' +
+                              ' dropped AS (SELECT pg_catalog.pg_drop_replication_slot(slot_name),' +
+                              ' true AS dropped FROM slots WHERE not active) ' +
+                              'SELECT active, COALESCE(dropped, false) FROM slots' +
+                              ' FULL OUTER JOIN dropped ON true'), name)
+        return cursor.fetchone() if cursor.rowcount == 1 else (False, False)

    def _drop_incorrect_slots(self, cluster, slots, paused):
        # drop old replication slots which are not presented in desired slots
        for name in set(self._replication_slots) - set(slots):
-            if not paused and not self.ignore_replication_slot(cluster, name) and not self.drop_replication_slot(name):
-                logger.error("Failed to drop replication slot '%s'", name)
-                self._schedule_load_slots = True
-
+            if not paused and not self.ignore_replication_slot(cluster, name):
+                active, dropped = self.drop_replication_slot(name)
+                if dropped:
+                    logger.info("Dropped unknown replication slot '%s'", name)
+                else:
+                    self._schedule_load_slots = True
+                    if active:
+                        logger.debug("Unable to drop unknown replication slot '%s', slot is still active", name)
+                    else:
+                        logger.error("Failed to drop replication slot '%s'", name)
        for name, value in slots.items():
            if name in self._replication_slots and not compare_slots(value, self._replication_slots[name]):
                logger.info("Trying to drop replication slot '%s' because value is changing from %s to %s",
                            name, self._replication_slots[name], value)
-                if self.drop_replication_slot(name):
+                if self.drop_replication_slot(name) == (False, True):
                    self._replication_slots.pop(name)
                else:
                    logger.error("Failed to drop replication slot '%s'", name)
@@ -387,9 +384,8 @@ class SlotsHandler(object):
                logger.error("Failed to copy logical slots from the %s via postgresql connection: %r", leader.name, e)

        if isinstance(create_slots, dict) and create_slots and self._postgresql.stop():
-            pg_replslot_dir = os.path.join(self._postgresql.data_dir, 'pg_replslot')
            for name, value in create_slots.items():
-                slot_dir = os.path.join(pg_replslot_dir, name)
+                slot_dir = os.path.join(self._postgresql.slots_handler.pg_replslot_dir, name)
                slot_tmp_dir = slot_dir + '.tmp'
                if os.path.exists(slot_tmp_dir):
                    shutil.rmtree(slot_tmp_dir)
@@ -404,7 +400,7 @@ class SlotsHandler(object):
                os.rename(slot_tmp_dir, slot_dir)
                fsync_dir(slot_dir)
                self._unready_logical_slots[name] = None
-            fsync_dir(pg_replslot_dir)
+            fsync_dir(self._postgresql.slots_handler.pg_replslot_dir)
            self._postgresql.start()

    def schedule(self, value=None):
@@ -0,0 +1,258 @@
+import logging
+import re
+import time
+
+from copy import deepcopy
+
+from .validator import CaseInsensitiveDict
+from ..psycopg import quote_ident as _quote_ident
+
+logger = logging.getLogger(__name__)
+
+SYNC_STANDBY_NAME_RE = re.compile(r'^[A-Za-z_][A-Za-z_0-9\$]*$')
+SYNC_REP_PARSER_RE = re.compile(r"""
+           (?P<first> [fF][iI][rR][sS][tT] )
+         | (?P<any> [aA][nN][yY] )
+         | (?P<space> \s+ )
+         | (?P<ident> [A-Za-z_][A-Za-z_0-9\$]* )
+         | (?P<dquot> " (?: [^"]+ | "" )* " )
+         | (?P<star> [*] )
+         | (?P<num> \d+ )
+         | (?P<comma> , )
+         | (?P<parenstart> \( )
+         | (?P<parenend> \) )
+         | (?P<JUNK> . )
+        """, re.X)
+_EMPTY_SSN = {'type': 'off', 'num': 0, 'members': CaseInsensitiveDict({})}
+
+
+def quote_ident(value):
+    """Very simplified version of quote_ident"""
+    return value if SYNC_STANDBY_NAME_RE.match(value) else _quote_ident(value)
+
+
+def parse_sync_standby_names(value):
+    """Parse postgresql synchronous_standby_names to constituent parts.
+    Returns dict with the following keys:
+    * type: 'quorum'|'priority'
+    * num: int
+    * members: CaseInsensitiveDict, with names as keys
+    * has_star: bool - Present if true
+    If the configuration value can not be parsed, raises a ValueError.
+
+    >>> parse_sync_standby_names('')['type']
+    'off'
+
+    >>> parse_sync_standby_names('FiRsT')['type']
+    'priority'
+
+    >>> parse_sync_standby_names('FiRsT')['members']
+    {'FiRsT': True}
+
+    >>> parse_sync_standby_names('"1"')['members']
+    {'1': True}
+
+    >>> parse_sync_standby_names(' a , b ')['members']
+    {'a': True, 'b': True}
+
+    >>> parse_sync_standby_names(' a , b ')['num']
+    1
+
+    >>> parse_sync_standby_names('ANY 4("a",*,b)')['has_star']
+    True
+
+    >>> parse_sync_standby_names('ANY 4("a",*,b)')['num']
+    4
+
+    >>> parse_sync_standby_names('1')  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    ValueError: Unparseable synchronous_standby_names value
+
+    >>> parse_sync_standby_names('a,')  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    ValueError: Unparseable synchronous_standby_names value
+
+    >>> parse_sync_standby_names('ANY 4("a" b,"c c")')  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    ValueError: Unparseable synchronous_standby_names value
+
+    >>> parse_sync_standby_names('FIRST 4("a",)')  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    ValueError: Unparseable synchronous_standby_names value
+
+    >>> parse_sync_standby_names('2 (,)')  # doctest: +IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    ValueError: Unparseable synchronous_standby_names value
+    """
+    tokens = [(m.lastgroup, m.group(0), m.start())
+              for m in SYNC_REP_PARSER_RE.finditer(value)
+              if m.lastgroup != 'space']
+    if not tokens:
+        return deepcopy(_EMPTY_SSN)
+
+    if [t[0] for t in tokens[0:3]] == ['any', 'num', 'parenstart'] and tokens[-1][0] == 'parenend':
+        result = {'type': 'quorum', 'num': int(tokens[1][1])}
+        synclist = tokens[3:-1]
+    elif [t[0] for t in tokens[0:3]] == ['first', 'num', 'parenstart'] and tokens[-1][0] == 'parenend':
+        result = {'type': 'priority', 'num': int(tokens[1][1])}
+        synclist = tokens[3:-1]
+    elif [t[0] for t in tokens[0:2]] == ['num', 'parenstart'] and tokens[-1][0] == 'parenend':
+        result = {'type': 'priority', 'num': int(tokens[0][1])}
+        synclist = tokens[2:-1]
+    else:
+        result = {'type': 'priority', 'num': 1}
+        synclist = tokens
+    result['members'] = CaseInsensitiveDict({})
+    for i, (a_type, a_value, a_pos) in enumerate(synclist):
+        if i % 2 == 1:  # odd elements are supposed to be commas
+            if len(synclist) == i + 1:  # except the last token
+                raise ValueError("Unparseable synchronous_standby_names value %r: Unexpected token %s %r at %d" %
+                                 (value, a_type, a_value, a_pos))
+            elif a_type != 'comma':
+                raise ValueError("Unparseable synchronous_standby_names value %r: ""Got token %s %r while"
+                                 " expecting comma at %d" % (value, a_type, a_value, a_pos))
+        elif a_type in {'ident', 'first', 'any'}:
+            result['members'][a_value] = True
+        elif a_type == 'star':
+            result['members'][a_value] = True
+            result['has_star'] = True
+        elif a_type == 'dquot':
+            result['members'][a_value[1:-1].replace('""', '"')] = True
+        else:
+            raise ValueError("Unparseable synchronous_standby_names value %r: Unexpected token %s %r at %d" %
+                             (value, a_type, a_value, a_pos))
+    return result
+
+
+class SyncHandler(object):
+    """Class responsible for working with the `synchronous_standby_names`.
+
+    Sync standbys are chosen based on their state in `pg_stat_replication`.
+    When `synchronous_standby_names` is changed we memorize the `_primary_flush_lsn`
+    and the `current_state()` method will count newly added names as "sync" only when
+    they reached memorized LSN and also reported as "sync" by `pg_stat_replication`"""
+
+    def __init__(self, postgresql):
+        self._postgresql = postgresql
+        self._synchronous_standby_names = ''  # last known value of synchronous_standby_names
+        self._ssn_data = deepcopy(_EMPTY_SSN)
+        self._primary_flush_lsn = 0
+        # "sync" replication connections, that were verified to reach self._primary_flush_lsn at some point
+        self._ready_replicas = CaseInsensitiveDict({})  # keys: member names, values: connection pids
+
+    def _handle_synchronous_standby_names_change(self):
+        """If synchronous_standby_names has changed we need to check that newly added replicas
+        have reached self._primary_flush_lsn. Only after that they could be counted as sync."""
+        synchronous_standby_names = self._postgresql.synchronous_standby_names()
+        if synchronous_standby_names == self._synchronous_standby_names:
+            return False
+
+        self._synchronous_standby_names = synchronous_standby_names
+        try:
+            self._ssn_data = parse_sync_standby_names(synchronous_standby_names)
+        except ValueError as e:
+            logger.warning('%s', e)
+            self._ssn_data = deepcopy(_EMPTY_SSN)
+
+        # Invalidate cache of "sync" connections
+        for app_name in list(self._ready_replicas.keys()):
+            if app_name not in self._ssn_data['members']:
+                del self._ready_replicas[app_name]
+
+        # Newly connected replicas will be counted as sync only when reached self._primary_flush_lsn
+        self._primary_flush_lsn = self._postgresql.last_operation()
+        self._postgresql.query('SELECT pg_catalog.txid_current()')  # Ensure some WAL traffic to move replication
+        self._postgresql.reset_cluster_info_state(None)  # Reset internal cache to query fresh values
+
+    def current_state(self, cluster, sync_node_count=1, sync_node_maxlag=-1):
+        """Finds best candidates to be the synchronous standbys.
+
+        Current synchronous standby is always preferred, unless it has disconnected or does not want to be a
+        synchronous standby any longer.
+        Parameter sync_node_maxlag(maximum_lag_on_syncnode) would help swapping unhealthy sync replica in case
+        if it stops responding (or hung). Please set the value high enough so it won't unncessarily swap sync
+        standbys during high loads. Any less or equal of 0 value keep the behavior backward compatible and
+        will not swap. Please note that it will not also swap sync standbys in case where all replicas are hung.
+
+        :returns: tuple of candidates list and synchronous standby list."""
+
+        self._handle_synchronous_standby_names_change()
+
+        # Pick candidates based on who has higher replay/remote_write/flush lsn.
+        sort_col = {
+            'remote_apply': 'replay',
+            'remote_write': 'write'
+        }.get(self._postgresql.synchronous_commit(), 'flush') + '_lsn'
+
+        pg_stat_replication = [(r['pid'], r['application_name'], r['sync_state'], r[sort_col])
+                               for r in self._postgresql.pg_stat_replication()
+                               if r[sort_col] is not None]
+
+        members = CaseInsensitiveDict({m.name: m for m in cluster.members})
+        replica_list = []
+        # pg_stat_replication.sync_state has 4 possible states - async, potential, quorum, sync.
+        # That is, alphabetically they are in the reversed order of priority.
+        # Since we are doing reversed sort on (sync_state, lsn) tuples, it helps to keep the result
+        # consistent in case if a synchronous standby member is slowed down OR async node receiving
+        # changes faster than the sync member (very rare but possible).
+        # Such cases would trigger sync standby member swapping, but only if lag on a sync node exceeding a threshold.
+        for pid, app_name, sync_state, replica_lsn in sorted(pg_stat_replication, key=lambda r: r[2:4], reverse=True):
+            member = members.get(app_name)
+            if member and member.is_running and not member.tags.get('nosync', False):
+                replica_list.append((pid, member.name, sync_state, replica_lsn, bool(member.nofailover)))
+
+        max_lsn = max(replica_list, key=lambda x: x[3])[3]\
+            if len(replica_list) > 1 else self._postgresql.last_operation()
+
+        if self._postgresql.major_version < 90600:
+            sync_node_count = 1
+
+        candidates = []
+        sync_nodes = []
+        # Prefer members without nofailover tag. We are relying on the fact that sorts are guaranteed to be stable.
+        for pid, app_name, sync_state, replica_lsn, _ in sorted(replica_list, key=lambda x: x[4]):
+            # if standby name is listed in the /sync key we can count it as synchronous, otherwice
+            # ig becomes really synchronous when sync_state = 'sync' and it is known that it managed to catch up
+            if app_name not in self._ready_replicas and app_name in self._ssn_data['members'] and\
+                    (cluster.sync and app_name in cluster.sync.members or
+                     sync_state == 'sync' and replica_lsn >= self._primary_flush_lsn):
+                self._ready_replicas[app_name] = pid
+
+            if sync_node_maxlag <= 0 or max_lsn - replica_lsn <= sync_node_maxlag:
+                candidates.append(app_name)
+                if sync_state == 'sync' and app_name in self._ready_replicas:
+                    sync_nodes.append(app_name)
+            if len(candidates) >= sync_node_count:
+                break
+
+        return candidates, sync_nodes
+
+    def set_synchronous_standby_names(self, value):
+        """Constructs and sets `synchronous_standby_names` value.
+
+        :param value: list[str] - the list of wanted sync members"""
+        if value and value != ['*']:
+            value = [quote_ident(x) for x in value]
+
+        if self._postgresql.major_version >= 90600 and len(value) > 1:
+            sync_param = '{0} ({1})'.format(len(value), ','.join(value))
+        else:
+            sync_param = next(iter(value), None)
+
+        if not (self._postgresql.config.set_synchronous_standby_names(sync_param) and
+                self._postgresql.state == 'running' and self._postgresql.is_leader()) or value == ['*']:
+            return
+
+        time.sleep(0.1)  # Usualy it takes 1ms to reload postgresql.conf, but we will give it 100ms
+
+        # Reset internal cache to query fresh values
+        self._postgresql.reset_cluster_info_state(None)
+
+        # timeline == 0 -- indicates that this is the replica, shoudn't ever happen
+        if self._postgresql.get_primary_timeline() > 0:
+            self._handle_synchronous_standby_names_change()
@@ -11,7 +11,7 @@
 # arguments are:
 #   - cluster scope
 #   - cluster role
-#   - master connection string
+#   - leader connection string
 #   - number of retries
 #   - envdir for the WALE env
 # - WALE_BACKUP_THRESHOLD_MEGABYTES if WAL amount is above that - use pg_basebackup
@@ -104,11 +104,11 @@ WALEConfig = namedtuple(

 class WALERestore(object):
    def __init__(self, scope, datadir, connstring, env_dir, threshold_mb,
-                 threshold_pct, use_iam, no_master, retries):
+                 threshold_pct, use_iam, no_leader, retries):
        self.scope = scope
-        self.master_connection = connstring
+        self.leader_connection = connstring
        self.data_dir = datadir
-        self.no_master = no_master
+        self.no_leader = no_leader

        wale_cmd = [
            'envdir',
@@ -213,11 +213,11 @@ class WALERestore(object):
        diff_in_bytes = backup_size
        attempts_no = 0
        while True:
-            if self.master_connection:
+            if self.leader_connection:
                con = None
                try:
                    # get the difference in bytes between the current WAL location and the backup start offset
-                    con = psycopg.connect(self.master_connection)
+                    con = psycopg.connect(self.leader_connection)
                    if con.server_version >= 100000:
                        wal_name = 'wal'
                        lsn_name = 'lsn'
@@ -235,22 +235,22 @@ class WALERestore(object):

                        diff_in_bytes = int(cur.fetchone()[0])
                except psycopg.Error:
-                    logger.exception('could not determine difference with the master location')
+                    logger.exception('could not determine difference with the leader location')
                    if attempts_no < self.retries:  # retry in case of a temporarily connection issue
                        attempts_no = attempts_no + 1
                        time.sleep(RETRY_SLEEP_INTERVAL)
                        continue
                    else:
-                        if not self.no_master:
+                        if not self.no_leader:
                            return False  # do no more retries on the outer level
-                        logger.info("continue with base backup from S3 since master is not available")
+                        logger.info("continue with base backup from S3 since leader is not available")
                        diff_in_bytes = 0
                        break
                finally:
                    if con:
                        con.close()
            else:
-                # always try to use WAL-E if master connection string is not available
+                # always try to use WAL-E if leader connection string is not available
                diff_in_bytes = 0
            break

@@ -346,22 +346,22 @@ def main():
    parser.add_argument('--threshold_megabytes', type=int, default=10240)
    parser.add_argument('--threshold_backup_size_percentage', type=int, default=30)
    parser.add_argument('--use_iam', type=int, default=0)
-    parser.add_argument('--no_master', type=int, default=0)
+    parser.add_argument('--no_leader', '--no_master', type=int, default=0)
    args = parser.parse_args()

    exit_code = None
    assert args.retries >= 0

-    # Retry cloning in a loop. We do separate retries for the master
+    # Retry cloning in a loop. We do separate retries for the leader
    # connection attempt inside should_use_s3_to_create_replica,
    # because we need to differentiate between the last attempt and
    # the rest and make a decision when the last attempt fails on
-    # whether to use WAL-E or not depending on the no_master flag.
+    # whether to use WAL-E or not depending on the no_leader flag.
    for _ in range(0, args.retries + 1):
        restore = WALERestore(scope=args.scope, datadir=args.datadir, connstring=args.connstring,
                              env_dir=args.envdir, threshold_mb=args.threshold_megabytes,
                              threshold_pct=args.threshold_backup_size_percentage, use_iam=args.use_iam,
-                              no_master=args.no_master, retries=args.retries)
+                              no_leader=args.no_leader, retries=args.retries)
        exit_code = restore.run()
        if not exit_code == ExitCode.RETRY_LATER:  # only WAL-E failures lead to the retry
            logger.debug('exit_code is %r, not retrying', exit_code)
@@ -367,6 +367,10 @@ schema = Schema({
          Optional("ports"): [{"name": str, "port": int}],
          },
      }),
+  Optional("citus"): {
+    "database": str,
+    "group": int
+  },
  "postgresql": {
    "listen": validate_host_port_listen_multiple_hosts,
    "connect_address": validate_connect_address,
@@ -1 +1 @@
-__version__ = '2.1.6'
+__version__ = '3.0.1'
@@ -18,6 +18,10 @@ restapi:
 #  keyfile: /etc/ssl/private/ssl-cert-snakeoil.key
 #  cacert: /etc/ssl/certs/ssl-cacert-snakeoil.pem

+#citus:
+#  database: citus
+#  group: 0  # coordinator
+
 etcd:
  #Provide host to do the initial discovery of the cluster topology:
  host: 127.0.0.1:2379
@@ -47,7 +51,7 @@ bootstrap:
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
-#    master_start_timeout: 300
+#    primary_start_timeout: 300
 #    synchronous_mode: false
    #standby_cluster:
      #host: 127.0.0.1
@@ -18,6 +18,10 @@ restapi:
 #  keyfile: /etc/ssl/private/ssl-cert-snakeoil.key
 #  cacert: /etc/ssl/certs/ssl-cacert-snakeoil.pem

+#citus:
+#  database: citus
+#  group: 1  # worker
+
 etcd:
  #Provide host to do the initial discovery of the cluster topology:
  host: 127.0.0.1:2379
@@ -18,6 +18,10 @@ restapi:
 #  keyfile: /etc/ssl/private/ssl-cert-snakeoil.key
 #  cacert: /etc/ssl/certs/ssl-cacert-snakeoil.pem

+#citus:
+#  database: citus
+#  group: 1  # worker
+
 etcd:
  #Provide host to do the initial discovery of the cluster topology:
  host: 127.0.0.1:2379
@@ -41,15 +41,13 @@ CLASSIFIERS = [
    'Operating System :: POSIX :: BSD :: FreeBSD',
    'Operating System :: Microsoft :: Windows',
    'Programming Language :: Python',
-    'Programming Language :: Python :: 2.7',
    'Programming Language :: Python :: 3',
-    'Programming Language :: Python :: 3.4',
-    'Programming Language :: Python :: 3.5',
    'Programming Language :: Python :: 3.6',
    'Programming Language :: Python :: 3.7',
    'Programming Language :: Python :: 3.8',
    'Programming Language :: Python :: 3.9',
    'Programming Language :: Python :: 3.10',
+    'Programming Language :: Python :: 3.11',
    'Programming Language :: Python :: Implementation :: CPython',
 ]

@@ -160,7 +158,6 @@ def setup_package(version):
        classifiers=CLASSIFIERS,
        packages=find_packages(exclude=['tests', 'tests.*']),
        package_data={MAIN_PACKAGE: ["*.json"]},
-        python_requires='>=2.7',
        install_requires=install_requires,
        extras_require=EXTRAS_REQUIRE,
        cmdclass=cmdclass,
@@ -171,14 +168,12 @@ def setup_package(version):
 if __name__ == '__main__':
    old_modules = sys.modules.copy()
    try:
-        from patroni import check_psycopg, fatal
+        from patroni import check_psycopg
        from patroni.version import __version__
    finally:
        sys.modules.clear()
        sys.modules.update(old_modules)

-    if sys.version_info < (2, 7, 0):
-        fatal('Patroni needs to be run with Python 2.7+')
    check_psycopg()

    setup_package(__version__)
@@ -43,11 +43,13 @@ class MockResponse(object):
        return {'content-type': 'json'}


-def requests_get(url, **kwargs):
+def requests_get(url, method='GET', endpoint=None, data='', **kwargs):
    members = '[{"id":14855829450254237642,"peerURLs":["http://localhost:2380","http://localhost:7001"],' +\
              '"name":"default","clientURLs":["http://localhost:2379","http://localhost:4001"]}]'
    response = MockResponse()
-    if url.startswith('http://local'):
+    if endpoint == 'failsafe':
+        response.content = 'Accepted'
+    elif url.startswith('http://local'):
        raise urllib3.exceptions.HTTPError()
    elif ':8011/patroni' in url:
        response.content = '{"role": "replica", "wal": {"received_location": 0}, "tags": {}}'
@@ -56,7 +58,6 @@ def requests_get(url, **kwargs):
    elif url.startswith('http://exhibitor'):
        response.content = '{"servers":["127.0.0.1","127.0.0.2","127.0.0.3"],"port":2181}'
    elif url.endswith(':8011/reinitialize'):
-        data = kwargs.get('data', '')
        if ' false}' in data:
            response.status_code = 503
            response.content = 'restarting after failure already in progress'
@@ -66,9 +67,8 @@ def requests_get(url, **kwargs):


 class MockPostmaster(object):
-    def __init__(self, is_running=True, is_single_master=False):
-        self.is_running = Mock(return_value=is_running)
-        self.is_single_master = Mock(return_value=is_single_master)
+    def __init__(self, pid=1):
+        self.is_running = Mock(return_value=self)
        self.wait_for_user_backends_to_close = Mock()
        self.signal_stop = Mock(return_value=None)
        self.wait = Mock()
@@ -97,8 +97,12 @@ class MockCursor(object):
            self.results = [('ls', 'logical', 'a', 'b', 100, 500, b'123456')]
        elif sql.startswith('SELECT slot_name'):
            self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'a', 'b', 5, 100, 500)]
+        elif sql.startswith('WITH slots AS (SELECT slot_name, active'):
+            self.results = [(False, True)]
        elif sql.startswith('SELECT CASE WHEN pg_catalog.pg_is_in_recovery()'):
-            self.results = [(1, 2, 1, 0, False, 1, 1, None, None, [{"slot_name": "ls", "confirmed_flush_lsn": 12345}])]
+            self.results = [(1, 2, 1, 0, False, 1, 1, None, None,
+                             [{"slot_name": "ls", "confirmed_flush_lsn": 12345}],
+                             'on', 'n1', None)]
        elif sql.startswith('SELECT pg_catalog.pg_is_in_recovery()'):
            self.results = [(False, 2)]
        elif sql.startswith('SELECT pg_catalog.pg_postmaster_start_time'):
@@ -123,6 +127,10 @@ class MockCursor(object):
                                 b'1\t0/40159C0\tno recovery target specified\n\n'
                                 b'2\t0/402DD98\tno recovery target specified\n\n'
                                 b'3\t0/403DD98\tno recovery target specified\n')]
+        elif sql.startswith('SELECT pg_catalog.citus_add_node'):
+            self.results = [(2,)]
+        elif sql.startswith('SELECT nodeid, groupid'):
+            self.results = [(1, 0, 'host1', 5432, 'primary'), (2, 1, 'host2', 5432, 'primary')]
        else:
            self.results = [(None, None, None, None, None, None, None, None, None, None)]

@@ -182,7 +190,7 @@ class PostgresInit(unittest.TestCase):
    @patch.object(ConfigHandler, 'write_postgresql_conf', Mock())
    @patch.object(ConfigHandler, 'replace_pg_hba', Mock())
    @patch.object(ConfigHandler, 'replace_pg_ident', Mock())
-    @patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='master'))
+    @patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='primary'))
    def setUp(self):
        data_dir = os.path.join('data', 'test0')
        self.p = Postgresql({'name': 'postgresql0', 'scope': 'batman', 'data_dir': data_dir,
@@ -200,7 +208,8 @@ class PostgresInit(unittest.TestCase):
                             'pg_hba': ['host all all 0.0.0.0/0 md5'],
                             'pg_ident': ['krb realm postgres'],
                             'callbacks': {'on_start': 'true', 'on_stop': 'true', 'on_reload': 'true',
-                                           'on_restart': 'true', 'on_role_change': 'true'}})
+                                           'on_restart': 'true', 'on_role_change': 'true'},
+                             'citus': {'group': 0, 'database': 'citus'}})


 class BaseTestPostgresql(PostgresInit):
@@ -211,11 +220,13 @@ class BaseTestPostgresql(PostgresInit):
        if not os.path.exists(self.p.data_dir):
            os.makedirs(self.p.data_dir)

-        self.leadermem = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres'})
+        self.leadermem = Member(0, 'leader', 28, {
+            'state': 'running', 'conn_url': 'postgres://replicator:[email protected]:5435/postgres'})
        self.leader = Leader(-1, 28, self.leadermem)
        self.other = Member(0, 'test-1', 28, {'conn_url': 'postgres://replicator:[email protected]:5433/postgres',
-                                              'tags': {'replicatefrom': 'leader'}})
-        self.me = Member(0, 'test0', 28, {'conn_url': 'postgres://replicator:[email protected]:5434/postgres'})
+                                              'state': 'running', 'tags': {'replicatefrom': 'leader'}})
+        self.me = Member(0, 'test0', 28, {
+            'state': 'running', 'conn_url': 'postgres://replicator:[email protected]:5434/postgres'})

    def tearDown(self):
        if os.path.exists(self.p.data_dir):
@@ -25,7 +25,7 @@ class MockPostgresql(object):

    name = 'test'
    state = 'running'
-    role = 'master'
+    role = 'primary'
    server_version = '999999'
    sysid = 'dummysysid'
    scope = 'dummy'
@@ -34,6 +34,7 @@ class MockPostgresql(object):
    lsn_name = 'lsn'
    POSTMASTER_START_TIME = 'pg_catalog.pg_postmaster_start_time()'
    TL_LSN = 'CASE WHEN pg_catalog.pg_is_in_recovery()'
+    citus_handler = Mock()

    @staticmethod
    def connection():
@@ -61,6 +62,14 @@ class MockHa(object):
    state_handler = MockPostgresql()
    watchdog = MockWatchdog()

+    @staticmethod
+    def update_failsafe(*args):
+        return 'foo'
+
+    @staticmethod
+    def failsafe_is_active(*args):
+        return True
+
    @staticmethod
    def is_leader():
        return False
@@ -180,7 +189,7 @@ class TestRestApiHandler(unittest.TestCase):
        MockRestApiServer(RestApiHandler, 'GET /read-only')
        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={})):
            MockRestApiServer(RestApiHandler, 'GET /replica')
-        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
            MockRestApiServer(RestApiHandler, 'GET /replica')
        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'state': 'running'})):
            MockRestApiServer(RestApiHandler, 'GET /health')
@@ -199,11 +208,11 @@ class TestRestApiHandler(unittest.TestCase):
            with patch.object(MockHa, 'is_standby_cluster', Mock(return_value=True)):
                MockRestApiServer(RestApiHandler, 'GET /standby_leader')
        MockPatroni.dcs.cluster = None
-        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
-            MockRestApiServer(RestApiHandler, 'GET /master')
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
+            MockRestApiServer(RestApiHandler, 'GET /primary')
        with patch.object(MockHa, 'restart_scheduled', Mock(return_value=True)):
-            MockRestApiServer(RestApiHandler, 'GET /master')
-        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /master'))
+            MockRestApiServer(RestApiHandler, 'GET /primary')
+        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /primary'))
        with patch.object(RestApiServer, 'query', Mock(return_value=[('', 1, '', '', '', '', False, '')])):
            self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /patroni'))
        with patch.object(MockHa, 'is_standby_cluster', Mock(return_value=True)):
@@ -211,30 +220,30 @@ class TestRestApiHandler(unittest.TestCase):

        # test tags
        #
-        MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+        MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                          'tag_key1=true&tag_key2=false&'
                                          'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
-        MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+        MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                          'tag_key1=true&tag_key2=False&'
                                          'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
-        MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+        MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                          'tag_key1=true&tag_key2=false&'
                                          'tag_key3=1.0&tag_key4=1.4&tag_key5=RandomTag')
-        MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+        MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                          'tag_key1=true&tag_key2=false&'
                                          'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
        #
-        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
-            MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
+            MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                              'tag_key1=true&tag_key2=false&'
                                              'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
-            MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+            MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                              'tag_key1=true&tag_key2=False&'
                                              'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
-            MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+            MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                              'tag_key1=true&tag_key2=false&'
                                              'tag_key3=1.0&tag_key4=1.4&tag_key5=RandomTag')
-            MockRestApiServer(RestApiHandler, 'GET /master?lag=1M&'
+            MockRestApiServer(RestApiHandler, 'GET /primary?lag=1M&'
                                              'tag_key1=true&tag_key2=false&'
                                              'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
        #
@@ -265,7 +274,7 @@ class TestRestApiHandler(unittest.TestCase):
                                          'tag_key1=true&tag_key2=false&'
                                          'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag&tag_key6=RandomTag2')
        #
-        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'master'})):
+        with patch.object(RestApiHandler, 'get_postgresql_status', Mock(return_value={'role': 'primary'})):
            MockRestApiServer(RestApiHandler, 'GET /replica?lag=1M&'
                                              'tag_key1=true&tag_key2=false&'
                                              'tag_key3=1&tag_key4=1.4&tag_key5=RandomTag')
@@ -371,6 +380,20 @@ class TestRestApiHandler(unittest.TestCase):
        mock_dcs.get_cluster.return_value.config = ClusterConfig.from_node(1, config)
        MockRestApiServer(RestApiHandler, request)

+    @patch.object(MockPatroni, 'dcs')
+    def test_do_GET_failsafe(self, mock_dcs):
+        type(mock_dcs).failsafe = PropertyMock(return_value={'node1': 'http://foo:8080/patroni'})
+        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /failsafe'))
+        type(mock_dcs).failsafe = PropertyMock(return_value=None)
+        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /failsafe'))
+
+    def test_do_POST_failsafe(self):
+        with patch.object(MockHa, 'is_failsafe_mode', Mock(return_value=False), create=True):
+            self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'POST /failsafe HTTP/1.0' + self._authorization))
+        with patch.object(MockHa, 'is_failsafe_mode', Mock(return_value=True), create=True):
+            self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'POST /failsafe HTTP/1.0' + self._authorization +
+                                                                   '\nContent-Length: 9\n\n{"a":"b"}'))
+
    @patch.object(MockPatroni, 'sighup_handler', Mock())
    def test_do_POST_reload(self):
        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'POST /reload HTTP/1.0' + self._authorization))
@@ -405,27 +428,27 @@ class TestRestApiHandler(unittest.TestCase):
        request = make_request(schedule=future_restart_time.isoformat(), role='unknown', postgres_version='9.5.3')
        MockRestApiServer(RestApiHandler, request)
        # wrong version
-        request = make_request(schedule=future_restart_time.isoformat(), role='master', postgres_version='9.5.3.1')
+        request = make_request(schedule=future_restart_time.isoformat(), role='primary', postgres_version='9.5.3.1')
        MockRestApiServer(RestApiHandler, request)
        # unknown filter
        request = make_request(schedule=future_restart_time.isoformat(), batman='lives')
        MockRestApiServer(RestApiHandler, request)
        # incorrect schedule
-        request = make_request(schedule='2016-08-42 12:45TZ+1', role='master')
+        request = make_request(schedule='2016-08-42 12:45TZ+1', role='primary')
        MockRestApiServer(RestApiHandler, request)
        # everything fine, but the schedule is missing
-        request = make_request(role='master', postgres_version='9.5.2')
+        request = make_request(role='primary', postgres_version='9.5.2')
        MockRestApiServer(RestApiHandler, request)
        for retval in (True, False):
            with patch.object(MockHa, 'schedule_future_restart', Mock(return_value=retval)):
                request = make_request(schedule=future_restart_time.isoformat())
                MockRestApiServer(RestApiHandler, request)
            with patch.object(MockHa, 'restart', Mock(return_value=(retval, "foo"))):
-                request = make_request(role='master', postgres_version='9.5.2')
+                request = make_request(role='primary', postgres_version='9.5.2')
                MockRestApiServer(RestApiHandler, request)

        mock_dcs.get_cluster.return_value.is_paused.return_value = True
-        MockRestApiServer(RestApiHandler, make_request(schedule='2016-08-42 12:45TZ+1', role='master'))
+        MockRestApiServer(RestApiHandler, make_request(schedule='2016-08-42 12:45TZ+1', role='primary'))
        # Valid timeout
        MockRestApiServer(RestApiHandler, make_request(timeout='60s'))
        # Invalid timeout
@@ -551,6 +574,13 @@ class TestRestApiHandler(unittest.TestCase):
        MockRestApiServer(RestApiHandler, post + '14\n\n{"leader":"1"}')
        MockRestApiServer(RestApiHandler, post + '37\n\n{"candidate":"2","scheduled_at": "1"}')

+    @patch.object(MockPatroni, 'dcs', Mock())
+    @patch.object(MockHa, 'is_leader', Mock(return_value=True))
+    def test_do_POST_citus(self):
+        post = 'POST /citus HTTP/1.0' + self._authorization + '\nContent-Length: '
+        MockRestApiServer(RestApiHandler, post + '0\n\n')
+        MockRestApiServer(RestApiHandler, post + '14\n\n{"leader":"1"}')
+

 class TestRestApiServer(unittest.TestCase):

@@ -37,15 +37,15 @@ class TestAWSConnection(unittest.TestCase):
        self.conn = AWSConnection('test')

    def test_on_role_change(self):
-        self.assertTrue(self.conn.on_role_change('master'))
+        self.assertTrue(self.conn.on_role_change('primary'))
        with patch.object(MockVolumes, 'filter', Mock(return_value=[])):
            self.conn._retry.max_tries = 1
-            self.assertFalse(self.conn.on_role_change('master'))
+            self.assertFalse(self.conn.on_role_change('primary'))

    @patch('patroni.scripts.aws.requests_get', Mock(side_effect=Exception('foo')))
    def test_non_aws(self):
        conn = AWSConnection('test')
-        self.assertFalse(conn.on_role_change("master"))
+        self.assertFalse(conn.on_role_change("primary"))

    @patch('patroni.scripts.aws.requests_get', Mock(return_value=urllib3.HTTPResponse(status=200, body=b'foo')))
    def test_aws_bizare_response(self):
@@ -0,0 +1,159 @@
+from mock import Mock, patch
+from patroni.postgresql.citus import CitusHandler
+
+from . import BaseTestPostgresql, MockCursor, psycopg_connect, SleepException
+from .test_ha import get_cluster_initialized_with_leader
+
+
+@patch('patroni.postgresql.citus.Thread', Mock())
+@patch('patroni.psycopg.connect', psycopg_connect)
+class TestCitus(BaseTestPostgresql):
+
+    def setUp(self):
+        super(TestCitus, self).setUp()
+        self.c = self.p.citus_handler
+        self.c.set_conn_kwargs({'host': 'localhost', 'dbname': 'postgres'})
+        self.cluster = get_cluster_initialized_with_leader()
+        self.cluster.workers[1] = self.cluster
+
+    @patch('time.time', Mock(side_effect=[100, 130, 160, 190, 220, 250, 280]))
+    @patch('patroni.postgresql.citus.logger.exception', Mock(side_effect=SleepException))
+    @patch('patroni.postgresql.citus.logger.warning')
+    @patch('patroni.postgresql.citus.PgDistNode.wait', Mock())
+    @patch.object(CitusHandler, 'is_alive', Mock(return_value=True))
+    def test_run(self, mock_logger_warning):
+        # `before_demote` or `before_promote` REST API calls starting a
+        # transaction. We want to make sure that it finishes during
+        # certain timeout. In case if it is not, we want to roll it back
+        # in order to not block other workers that want to update
+        # `pg_dist_node`.
+        self.c._condition.wait = Mock(side_effect=[Mock(), Mock(), Mock(), SleepException])
+
+        self.c.handle_event(self.cluster, {'type': 'before_demote', 'group': 1,
+                                           'leader': 'leader', 'timeout': 30, 'cooldown': 10})
+        self.c.add_task('after_promote', 2, 'postgres://host3:5432/postgres')
+        self.assertRaises(SleepException, self.c.run)
+        mock_logger_warning.assert_called_once()
+        self.assertTrue(mock_logger_warning.call_args[0][0].startswith('Rolling back transaction'))
+        self.assertTrue(repr(mock_logger_warning.call_args[0][1]).startswith('PgDistNode'))
+
+    @patch.object(CitusHandler, 'is_alive', Mock(return_value=False))
+    @patch.object(CitusHandler, 'start', Mock())
+    def test_sync_pg_dist_node(self):
+        with patch.object(CitusHandler, 'is_enabled', Mock(return_value=False)):
+            self.c.sync_pg_dist_node(self.cluster)
+        self.c.sync_pg_dist_node(self.cluster)
+
+    def test_handle_event(self):
+        self.c.handle_event(self.cluster, {})
+        with patch.object(CitusHandler, 'is_alive', Mock(return_value=True)):
+            self.c.handle_event(self.cluster, {'type': 'after_promote', 'group': 2,
+                                               'leader': 'leader', 'timeout': 30, 'cooldown': 10})
+
+    def test_add_task(self):
+        with patch('patroni.postgresql.citus.logger.error') as mock_logger,\
+                patch('patroni.postgresql.citus.urlparse', Mock(side_effect=Exception)):
+            self.c.add_task('', 1, None)
+            mock_logger.assert_called_once()
+
+        with patch('patroni.postgresql.citus.logger.debug') as mock_logger:
+            self.c.add_task('before_demote', 1, 'postgres://host:5432/postgres', 30)
+            mock_logger.assert_called_once()
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Adding the new task:'))
+
+        with patch('patroni.postgresql.citus.logger.debug') as mock_logger:
+            self.c.add_task('before_promote', 1, 'postgres://host:5432/postgres', 30)
+            mock_logger.assert_called_once()
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Overriding existing task:'))
+
+        # add_task called from sync_pg_dist_node should not override already scheduled or in flight task
+        self.assertIsNotNone(self.c.add_task('after_promote', 1, 'postgres://host:5432/postgres', 30))
+        self.assertIsNone(self.c.add_task('after_promote', 1, 'postgres://host:5432/postgres'))
+        self.c._in_flight = self.c._tasks.pop()
+        self.assertIsNone(self.c.add_task('after_promote', 1, 'postgres://host:5432/postgres'))
+
+        # If there is no transaction in progress and cached pg_dist_node matching desired state task should not be added
+        self.c._schedule_load_pg_dist_node = False
+        self.c._pg_dist_node[self.c._in_flight.group] = self.c._in_flight
+        self.c._in_flight = None
+        self.assertIsNone(self.c.add_task('after_promote', 1, 'postgres://host:5432/postgres'))
+
+    def test_pick_task(self):
+        self.c.add_task('after_promote', 1, 'postgres://host2:5432/postgres')
+        with patch.object(CitusHandler, 'process_task') as mock_process_task:
+            self.c.process_tasks()
+            # process_task() shouln't be called because pick_task double checks with _pg_dist_node
+            mock_process_task.assert_not_called()
+
+    def test_process_task(self):
+        self.c.add_task('after_promote', 0, 'postgres://host2:5432/postgres')
+        task = self.c.add_task('before_promote', 1, 'postgres://host4:5432/postgres', 30)
+        self.c.process_tasks()
+        self.assertTrue(task._event.is_set())
+
+        # the after_promote should result only in COMMIT
+        task = self.c.add_task('after_promote', 1, 'postgres://host4:5432/postgres', 30)
+        with patch.object(CitusHandler, 'query') as mock_query:
+            self.c.process_tasks()
+            mock_query.assert_called_once()
+            self.assertEqual(mock_query.call_args[0][0], 'COMMIT')
+
+    def test_process_tasks(self):
+        self.c.add_task('after_promote', 0, 'postgres://host2:5432/postgres')
+        self.c.process_tasks()
+
+        self.c.add_task('after_promote', 0, 'postgres://host3:5432/postgres')
+        with patch('patroni.postgresql.citus.logger.error') as mock_logger,\
+                patch.object(CitusHandler, 'query', Mock(side_effect=Exception)):
+            self.c.process_tasks()
+            mock_logger.assert_called_once()
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Exception when working with pg_dist_node: '))
+
+    def test_on_demote(self):
+        self.c.on_demote()
+
+    @patch('patroni.postgresql.citus.logger.error')
+    @patch.object(MockCursor, 'execute', Mock(side_effect=Exception))
+    def test_load_pg_dist_node(self, mock_logger):
+        # load_pg_dist_node() triggers, query fails and exception is property handled
+        self.c.process_tasks()
+        self.assertTrue(self.c._schedule_load_pg_dist_node)
+        mock_logger.assert_called_once()
+        self.assertTrue(mock_logger.call_args[0][0].startswith('Exception when executing query'))
+        self.assertTrue(mock_logger.call_args[0][1].startswith('SELECT nodeid, groupid, '))
+
+    def test_wait(self):
+        task = self.c.add_task('before_demote', 1, 'postgres://host:5432/postgres', 30)
+        task._event.wait = Mock()
+        task.wait()
+
+    def test_adjust_postgres_gucs(self):
+        parameters = {'max_connections': 101,
+                      'max_prepared_transactions': 0,
+                      'shared_preload_libraries': 'foo , citus, bar '}
+        self.c.adjust_postgres_gucs(parameters)
+        self.assertEqual(parameters['max_prepared_transactions'], 202)
+        self.assertEqual(parameters['shared_preload_libraries'], 'citus,foo,bar')
+        self.assertEqual(parameters['wal_level'], 'logical')
+
+    @patch.object(CitusHandler, 'is_enabled', Mock(return_value=False))
+    def test_bootstrap(self):
+        self.c.bootstrap()
+
+    def test_ignore_replication_slot(self):
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'foo', 'type': 'physical',
+                                                         'database': 'bar', 'plugin': 'wal2json'}))
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'foo', 'type': 'logical',
+                                                         'database': 'bar', 'plugin': 'wal2json'}))
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'foo', 'type': 'logical',
+                                                         'database': 'bar', 'plugin': 'pgoutput'}))
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'foo', 'type': 'logical',
+                                                         'database': 'citus', 'plugin': 'pgoutput'}))
+        self.assertTrue(self.c.ignore_replication_slot({'name': 'citus_shard_move_slot_1_2_3',
+                                                        'type': 'logical', 'database': 'citus', 'plugin': 'pgoutput'}))
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'citus_shard_move_slot_1_2_3',
+                                                         'type': 'logical', 'database': 'citus', 'plugin': 'citus'}))
+        self.assertFalse(self.c.ignore_replication_slot({'name': 'citus_shard_split_slot_1_2_3',
+                                                         'type': 'logical', 'database': 'citus', 'plugin': 'pgoutput'}))
+        self.assertTrue(self.c.ignore_replication_slot({'name': 'citus_shard_split_slot_1_2_3',
+                                                        'type': 'logical', 'database': 'citus', 'plugin': 'citus'}))
@@ -21,7 +21,9 @@ class TestConfig(unittest.TestCase):
    def test_set_dynamic_configuration(self):
        with patch.object(Config, '_build_effective_configuration', Mock(side_effect=Exception)):
            self.assertIsNone(self.config.set_dynamic_configuration({'foo': 'bar'}))
-        self.assertTrue(self.config.set_dynamic_configuration({'synchronous_mode': True, 'standby_cluster': {}}))
+        self.assertTrue(self.config.set_dynamic_configuration({'synchronous_mode': True,
+                                                               'standby_cluster': {}, 'master_start_timeout': 1}))
+        self.assertEqual(self.config.get('primary_start_timeout'), 1)

    def test_reload_local_configuration(self):
        os.environ.update({
@@ -31,6 +33,9 @@ class TestConfig(unittest.TestCase):
            'PATRONI_LOGLEVEL': 'ERROR',
            'PATRONI_LOG_LOGGERS': 'patroni.postmaster: WARNING, urllib3: DEBUG',
            'PATRONI_LOG_FILE_NUM': '5',
+            'PATRONI_CITUS_DATABASE': 'citus',
+            'PATRONI_CITUS_GROUP': '0',
+            'PATRONI_CITUS_HOST': '0',
            'PATRONI_RESTAPI_USERNAME': 'username',
            'PATRONI_RESTAPI_PASSWORD': 'password',
            'PATRONI_RESTAPI_LISTEN': '0.0.0.0:8008',
@@ -18,6 +18,8 @@ def kv_get(self, key, **kwargs):
    good_cls = ('6429',
                [{'CreateIndex': 1334, 'Flags': 0, 'Key': key + 'failover', 'LockIndex': 0,
                  'ModifyIndex': 1334, 'Value': b''},
+                 {'CreateIndex': 1334, 'Flags': 0, 'Key': key + '1/initialize', 'LockIndex': 0,
+                  'ModifyIndex': 1334, 'Value': b'postgresql0'},
                 {'CreateIndex': 1334, 'Flags': 0, 'Key': key + 'initialize', 'LockIndex': 0,
                  'ModifyIndex': 1334, 'Value': b'postgresql0'},
                 {'CreateIndex': 2621, 'Flags': 0, 'Key': key + 'leader', 'LockIndex': 1,
@@ -94,7 +96,7 @@ class TestConsul(unittest.TestCase):
                'verify': 'on', 'cert': 'bar', 'cacert': 'buz', 'register_service': True})
        self.c = Consul({'ttl': 30, 'scope': 'test', 'name': 'postgresql1', 'host': 'localhost:1', 'retry_timeout': 10,
                         'register_service': True, 'service_check_tls_server_name': True})
-        self.c._base_path = '/service/good'
+        self.c._base_path = 'service/good'
        self.c.get_cluster()

    @patch('time.sleep', Mock(side_effect=SleepException))
@@ -115,16 +117,22 @@ class TestConsul(unittest.TestCase):

    @patch.object(consul.Consul.KV, 'delete', Mock())
    def test_get_cluster(self):
-        self.c._base_path = '/service/test'
+        self.c._base_path = 'service/test'
        self.assertIsInstance(self.c.get_cluster(), Cluster)
        self.assertIsInstance(self.c.get_cluster(), Cluster)
-        self.c._base_path = '/service/fail'
+        self.c._base_path = 'service/fail'
        self.assertRaises(ConsulError, self.c.get_cluster)
-        self.c._base_path = '/service/broken'
+        self.c._base_path = 'service/broken'
        self.assertIsInstance(self.c.get_cluster(), Cluster)
-        self.c._base_path = '/service/legacy'
+        self.c._base_path = 'service/legacy'
        self.assertIsInstance(self.c.get_cluster(), Cluster)

+    def test__get_citus_cluster(self):
+        self.c._citus_group = '0'
+        cluster = self.c.get_cluster()
+        self.assertIsInstance(cluster, Cluster)
+        self.assertIsInstance(cluster.workers[1], Cluster)
+
    @patch.object(consul.Consul.KV, 'delete', Mock(side_effect=[ConsulException, True, True, True]))
    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=[True, ConsulException, InvalidSession]))
    def test_touch_member(self):
@@ -221,7 +229,7 @@ class TestConsul(unittest.TestCase):
    def test_set_history_value(self):
        self.assertTrue(self.c.set_history_value('{}'))

-    @patch.object(consul.Consul.Agent.Service, 'register', Mock(side_effect=(False, True)))
+    @patch.object(consul.Consul.Agent.Service, 'register', Mock(side_effect=(False, True, True, True)))
    @patch.object(consul.Consul.Agent.Service, 'deregister', Mock(return_value=True))
    def test_update_service(self):
        d = {'role': 'replica', 'api_url': 'http://a/t', 'conn_url': 'pg://c:1', 'state': 'running'}
@@ -236,6 +244,9 @@ class TestConsul(unittest.TestCase):
        d['state'] = 'running'
        d['role'] = 'bla'
        self.assertIsNone(self.c.update_service({}, d))
+        for role in ('master', 'primary'):
+            d['role'] = role
+            self.assertTrue(self.c.update_service({}, d))

    @patch.object(consul.Consul.KV, 'put', Mock(side_effect=ConsulException))
    def test_reload_config(self):
@@ -21,16 +21,19 @@ from .test_ha import get_cluster_initialized_without_leader, get_cluster_initial


@patch('patroni.ctl.load_config', Mock(return_value={
-    'scope': 'alpha', 'restapi': {'listen': '::', 'certfile': 'a'}, 'etcd': {'host': 'localhost:2379'},
+    'scope': 'alpha', 'restapi': {'listen': '::', 'certfile': 'a'},
+    'etcd': {'host': 'localhost:2379'}, 'citus': {'database': 'citus', 'group': 0},
    'postgresql': {'data_dir': '.', 'pgpass': './pgpass', 'parameters': {}, 'retry_timeout': 5}}))
 class TestCtl(unittest.TestCase):
+    TEST_ROLES = ('master', 'primary', 'leader')

    @patch('socket.getaddrinfo', socket_getaddrinfo)
    def setUp(self):
        with patch.object(AbstractEtcdClientWithFailover, 'machines') as mock_machines:
            mock_machines.__get__ = Mock(return_value=['http://remotehost:2379'])
            self.runner = CliRunner()
-            self.e = get_dcs({'etcd': {'ttl': 30, 'host': 'ok:2379', 'retry_timeout': 10}}, 'foo')
+            self.e = get_dcs({'etcd': {'ttl': 30, 'host': 'ok:2379', 'retry_timeout': 10},
+                              'citus': {'group': 0}}, 'foo', None)

    @patch('patroni.ctl.logging.debug')
    def test_load_config(self, mock_logger_debug):
@@ -39,7 +42,7 @@ class TestCtl(unittest.TestCase):
            self.assertRaises(PatroniCtlException, load_config, './non-existing-config-file', None)

        with patch('os.path.exists', Mock(return_value=True)), \
-             patch('patroni.config.Config._load_config_path', Mock(return_value={})):
+                patch('patroni.config.Config._load_config_path', Mock(return_value={})):
            load_config(CONFIG_FILE_PATH, None)
            mock_logger_debug.assert_called_once()
            self.assertEqual(('Ignoring configuration file "%s". It does not exists or is not readable.',
@@ -56,14 +59,14 @@ class TestCtl(unittest.TestCase):

    @patch('patroni.psycopg.connect', psycopg_connect)
    def test_get_cursor(self):
-        self.assertIsNone(get_cursor(get_cluster_initialized_without_leader(), {}, role='master'))
-
-        self.assertIsNotNone(get_cursor(get_cluster_initialized_with_leader(), {}, role='master'))
+        for role in self.TEST_ROLES:
+            self.assertIsNone(get_cursor({}, get_cluster_initialized_without_leader(), None, {}, role=role))
+            self.assertIsNotNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {}, role=role))

        # MockCursor returns pg_is_in_recovery as false
-        self.assertIsNone(get_cursor(get_cluster_initialized_with_leader(), {}, role='replica'))
+        self.assertIsNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {}, role='replica'))

-        self.assertIsNotNone(get_cursor(get_cluster_initialized_with_leader(), {'dbname': 'foo'}, role='any'))
+        self.assertIsNotNone(get_cursor({}, get_cluster_initialized_with_leader(), None, {'dbname': 'foo'}, role='any'))

    def test_parse_dcs(self):
        assert parse_dcs(None) is None
@@ -80,7 +83,7 @@ class TestCtl(unittest.TestCase):
        cluster = get_cluster_initialized_with_leader(Failover(1, 'foo', 'bar', scheduled_at))
        del cluster.members[1].data['conn_url']
        for fmt in ('pretty', 'json', 'yaml', 'tsv', 'topology'):
-            self.assertIsNone(output_members(cluster, name='abc', fmt=fmt))
+            self.assertIsNone(output_members({}, cluster, name='abc', fmt=fmt))

    @patch('patroni.ctl.get_dcs')
    @patch.object(PoolManager, 'request', Mock(return_value=MockResponse()))
@@ -88,74 +91,79 @@ class TestCtl(unittest.TestCase):
        mock_get_dcs.return_value = self.e
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
        mock_get_dcs.return_value.set_failover_value = Mock()
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
        assert 'leader' in result.output

-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n2300-01-01T12:23:00\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'],
+                                    input='leader\nother\n2300-01-01T12:23:00\ny')
        assert result.exit_code == 0

        with patch('patroni.dcs.Cluster.is_paused', Mock(return_value=True)):
-            result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force', '--scheduled', '2015-01-01T12:00:00'])
+            result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
+                                              '--force', '--scheduled', '2015-01-01T12:00:00'])
            assert result.exit_code == 1

        # Aborting switchover, as we answer NO to the confirmation
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\nN')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\nN')
        assert result.exit_code == 1

        # Aborting scheduled switchover, as we answer NO to the confirmation
-        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--scheduled', '2015-01-01T12:00:00+01:00'],
-                                    input='leader\nother\n\nN')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
+                                    '--scheduled', '2015-01-01T12:00:00+01:00'], input='leader\nother\n\nN')
        assert result.exit_code == 1

        # Target and source are equal
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nleader\n\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nleader\n\ny')
        assert result.exit_code == 1

        # Reality is not part of this cluster
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nReality\n\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nReality\n\ny')
        assert result.exit_code == 1

-        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force'])
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0', '--force'])
        assert 'Member' in result.output

-        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force', '--scheduled', '2015-01-01T12:00:00+01:00'])
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
+                                          '--force', '--scheduled', '2015-01-01T12:00:00+01:00'])
        assert result.exit_code == 0

        # Invalid timestamp
-        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force', '--scheduled', 'invalid'])
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0', '--force', '--scheduled', 'invalid'])
        assert result.exit_code != 0

        # Invalid timestamp
-        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--force', '--scheduled', '2115-02-30T12:00:00+01:00'])
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0',
+                                          '--force', '--scheduled', '2115-02-30T12:00:00+01:00'])
        assert result.exit_code != 0

        # Specifying wrong leader
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='dummy')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='dummy')
        assert result.exit_code == 1

        with patch.object(PoolManager, 'request', Mock(side_effect=Exception)):
            # Non-responding patroni
-            result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n2300-01-01T12:23:00\ny')
+            result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'],
+                                        input='leader\nother\n2300-01-01T12:23:00\ny')
            assert 'falling back to DCS' in result.output

        with patch.object(PoolManager, 'request') as mocked:
            mocked.return_value.status = 500
-            result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\ny')
+            result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
            assert 'Switchover failed' in result.output

            mocked.return_value.status = 501
            mocked.return_value.data = b'Server does not support this operation'
-            result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\ny')
+            result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
            assert 'Switchover failed' in result.output

        # No members available
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_only_leader
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
        assert result.exit_code == 1

-        # No master available
+        # No leader available
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_without_leader
-        result = self.runner.invoke(ctl, ['switchover', 'dummy'], input='leader\nother\n\ny')
+        result = self.runner.invoke(ctl, ['switchover', 'dummy', '--group', '0'], input='leader\nother\n\ny')
        assert result.exit_code == 1

    @patch('patroni.ctl.get_dcs')
@@ -164,12 +172,14 @@ class TestCtl(unittest.TestCase):
        mock_get_dcs.return_value = self.e
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
        mock_get_dcs.return_value.set_failover_value = Mock()
-        result = self.runner.invoke(ctl, ['failover', 'dummy'], input='\n')
+        result = self.runner.invoke(ctl, ['failover', 'dummy', '--force'], input='\n')
+        assert 'For Citus clusters the --group must me specified' in result.output
+        result = self.runner.invoke(ctl, ['failover', 'dummy'], input='0\n')
        assert 'Failover could be performed only to a specific candidate' in result.output

    @patch('patroni.dcs.dcs_modules', Mock(return_value=['patroni.dcs.dummy', 'patroni.dcs.etcd']))
    def test_get_dcs(self):
-        self.assertRaises(PatroniCtlException, get_dcs, {'dummy': {}}, 'dummy')
+        self.assertRaises(PatroniCtlException, get_dcs, {'dummy': {}}, 'dummy', 0)

    @patch('patroni.psycopg.connect', psycopg_connect)
    @patch('patroni.ctl.query_member', Mock(return_value=([['mock column']], None)))
@@ -178,8 +188,9 @@ class TestCtl(unittest.TestCase):
    def test_query(self, mock_get_dcs):
        mock_get_dcs.return_value = self.e
        # Mutually exclusive
-        result = self.runner.invoke(ctl, ['query', 'alpha', '--member', 'abc', '--role', 'master'])
-        assert result.exit_code == 1
+        for role in self.TEST_ROLES:
+            result = self.runner.invoke(ctl, ['query', 'alpha', '--member', 'abc', '--role', role])
+            assert result.exit_code == 1

        with self.runner.isolated_filesystem():
            with open('dummy', 'w') as dummy_file:
@@ -189,7 +200,7 @@ class TestCtl(unittest.TestCase):
            result = self.runner.invoke(ctl, ['query', 'alpha', '--file', 'dummy', '--command', 'dummy'])
            assert result.exit_code == 1

-            result = self.runner.invoke(ctl, ['query', 'alpha', '--file', 'dummy'])
+            result = self.runner.invoke(ctl, ['query', 'alpha', '--member', 'abc', '--file', 'dummy'])
            assert result.exit_code == 0

            os.remove('dummy')
@@ -207,21 +218,22 @@ class TestCtl(unittest.TestCase):

    def test_query_member(self):
        with patch('patroni.ctl.get_cursor', Mock(return_value=MockConnect().cursor())):
-            rows = query_member(None, None, None, 'master', 'SELECT pg_catalog.pg_is_in_recovery()', {})
-            self.assertTrue('False' in str(rows))
+            for role in self.TEST_ROLES:
+                rows = query_member({}, None, None, None, None, role, 'SELECT pg_catalog.pg_is_in_recovery()', {})
+                self.assertTrue('False' in str(rows))

            with patch.object(MockCursor, 'execute', Mock(side_effect=OperationalError('bla'))):
-                rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
+                rows = query_member({}, None, None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})

        with patch('patroni.ctl.get_cursor', Mock(return_value=None)):
-            rows = query_member(None, None, None, None, 'SELECT pg_catalog.pg_is_in_recovery()', {})
+            rows = query_member({}, None, None, None, None, None, 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertTrue('No connection to' in str(rows))

-            rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
+            rows = query_member({}, None, None, None, 'foo', 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
            self.assertTrue('No connection to' in str(rows))

        with patch('patroni.ctl.get_cursor', Mock(side_effect=OperationalError('bla'))):
-            rows = query_member(None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})
+            rows = query_member({}, None, None, None, None, 'replica', 'SELECT pg_catalog.pg_is_in_recovery()', {})

    @patch('patroni.ctl.get_dcs')
    def test_dsn(self, mock_get_dcs):
@@ -230,8 +242,9 @@ class TestCtl(unittest.TestCase):
        assert 'host=127.0.0.1 port=5435' in result.output

        # Mutually exclusive options
-        result = self.runner.invoke(ctl, ['dsn', 'alpha', '--role', 'master', '--member', 'dummy'])
-        assert result.exit_code == 1
+        for role in self.TEST_ROLES:
+            result = self.runner.invoke(ctl, ['dsn', 'alpha', '--role', role, '--member', 'dummy'])
+            assert result.exit_code == 1

        # Non-existing member
        result = self.runner.invoke(ctl, ['dsn', 'alpha', '--member', 'dummy'])
@@ -334,21 +347,23 @@ class TestCtl(unittest.TestCase):
    @patch('patroni.ctl.get_dcs')
    def test_remove(self, mock_get_dcs):
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
-        result = self.runner.invoke(ctl, ['-k', 'remove', 'alpha'], input='alpha\nslave')
+        result = self.runner.invoke(ctl, ['remove', 'dummy'], input='\n')
+        assert 'For Citus clusters the --group must me specified' in result.output
+        result = self.runner.invoke(ctl, ['-k', 'remove', 'alpha', '--group', '0'], input='alpha\nstandby')
        assert 'Please confirm' in result.output
        assert 'You are about to remove all' in result.output
        # Not typing an exact confirmation
        assert result.exit_code == 1

-        # master specified does not match master of cluster
-        result = self.runner.invoke(ctl, ['remove', 'alpha'], input='alpha\nYes I am aware\nslave')
+        # leader specified does not match leader of cluster
+        result = self.runner.invoke(ctl, ['remove', 'alpha', '--group', '0'], input='alpha\nYes I am aware\nstandby')
        assert result.exit_code == 1

        # cluster specified on cmdline does not match verification prompt
-        result = self.runner.invoke(ctl, ['remove', 'alpha'], input='beta\nleader')
+        result = self.runner.invoke(ctl, ['remove', 'alpha', '--group', '0'], input='beta\nleader')
        assert result.exit_code == 1

-        result = self.runner.invoke(ctl, ['remove', 'alpha'], input='alpha\nYes I am aware\nleader')
+        result = self.runner.invoke(ctl, ['remove', 'alpha', '--group', '0'], input='alpha\nYes I am aware\nleader')
        assert result.exit_code == 0

    def test_ctl(self):
@@ -358,23 +373,26 @@ class TestCtl(unittest.TestCase):
        assert 'Usage:' in result.output

    def test_get_any_member(self):
-        self.assertIsNone(get_any_member(get_cluster_initialized_without_leader(), role='master'))
+        for role in self.TEST_ROLES:
+            self.assertIsNone(get_any_member({}, get_cluster_initialized_without_leader(), None, role=role))

-        m = get_any_member(get_cluster_initialized_with_leader(), role='master')
-        self.assertEqual(m.name, 'leader')
+            m = get_any_member({}, get_cluster_initialized_with_leader(), None, role=role)
+            self.assertEqual(m.name, 'leader')

    def test_get_all_members(self):
-        self.assertEqual(list(get_all_members(get_cluster_initialized_without_leader(), role='master')), [])
+        for role in self.TEST_ROLES:
+            self.assertEqual(list(get_all_members({}, get_cluster_initialized_without_leader(), None, role=role)), [])

-        r = list(get_all_members(get_cluster_initialized_with_leader(), role='master'))
-        self.assertEqual(len(r), 1)
-        self.assertEqual(r[0].name, 'leader')
+            r = list(get_all_members({}, get_cluster_initialized_with_leader(), None, role=role))
+            self.assertEqual(len(r), 1)
+            self.assertEqual(r[0].name, 'leader')

-        r = list(get_all_members(get_cluster_initialized_with_leader(), role='replica'))
+        r = list(get_all_members({}, get_cluster_initialized_with_leader(), None, role='replica'))
        self.assertEqual(len(r), 1)
        self.assertEqual(r[0].name, 'other')

-        self.assertEqual(len(list(get_all_members(get_cluster_initialized_without_leader(), role='replica'))), 2)
+        self.assertEqual(len(list(get_all_members({}, get_cluster_initialized_without_leader(),
+                                                  None, role='replica'))), 2)

    @patch('patroni.ctl.get_dcs')
    def test_members(self, mock_get_dcs):
@@ -385,30 +403,6 @@ class TestCtl(unittest.TestCase):
        with patch('patroni.ctl.load_config', Mock(return_value={})):
            self.runner.invoke(ctl, ['list'])

-    @patch('patroni.ctl.get_dcs')
-    def test_scaffold(self, mock_get_dcs):
-        mock_get_dcs.return_value = self.e
-        mock_get_dcs.return_value.get_cluster = get_cluster_not_initialized_without_leader
-        mock_get_dcs.return_value.initialize = Mock(return_value=True)
-        mock_get_dcs.return_value.touch_member = Mock(return_value=True)
-        mock_get_dcs.return_value.attempt_to_acquire_leader = Mock(return_value=True)
-        mock_get_dcs.return_value.delete_cluster = Mock()
-
-        with patch.object(self.e, 'initialize', return_value=False):
-            result = self.runner.invoke(ctl, ['scaffold', 'alpha'])
-            assert result.exception
-
-        with patch.object(mock_get_dcs.return_value, 'touch_member', Mock(return_value=False)):
-            result = self.runner.invoke(ctl, ['scaffold', 'alpha'])
-            assert result.exception
-
-        result = self.runner.invoke(ctl, ['scaffold', 'alpha'])
-        assert result.exit_code == 0
-
-        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader
-        result = self.runner.invoke(ctl, ['scaffold', 'alpha'])
-        assert result.exception
-
    @patch('patroni.ctl.get_dcs')
    def test_list_extended(self, mock_get_dcs):
        mock_get_dcs.return_value = self.e
@@ -438,16 +432,16 @@ class TestCtl(unittest.TestCase):
        cluster.members.append(cascade_member_wrong_tags)
        mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
        result = self.runner.invoke(ctl, ['topology', 'dummy'])
-        assert '+\n| leader          | 127.0.0.1:5435 | Leader  |' in result.output
-        assert '|\n| + other         | 127.0.0.1:5436 | Replica |' in result.output
-        assert '|\n|   + cascade     | 127.0.0.1:5437 | Replica |' in result.output
-        assert '|\n| + wrong_cascade | 127.0.0.1:5438 | Replica |' in result.output
+        assert '+\n|     0 | leader          | 127.0.0.1:5435 | Leader  |' in result.output
+        assert '|\n|     0 | + other         | 127.0.0.1:5436 | Replica |' in result.output
+        assert '|\n|     0 |   + cascade     | 127.0.0.1:5437 | Replica |' in result.output
+        assert '|\n|     0 | + wrong_cascade | 127.0.0.1:5438 | Replica |' in result.output

        cluster = get_cluster_initialized_without_leader()
        mock_get_dcs.return_value.get_cluster = Mock(return_value=cluster)
        result = self.runner.invoke(ctl, ['topology', 'dummy'])
-        assert '+\n| + leader | 127.0.0.1:5435 | Replica |' in result.output
-        assert '|\n| + other  | 127.0.0.1:5436 | Replica |' in result.output
+        assert '+\n|     0 | + leader | 127.0.0.1:5435 | Replica |' in result.output
+        assert '|\n|     0 | + other  | 127.0.0.1:5436 | Replica |' in result.output

    @patch('patroni.ctl.get_dcs')
    @patch.object(PoolManager, 'request', Mock(return_value=MockResponse()))
@@ -455,8 +449,9 @@ class TestCtl(unittest.TestCase):
        mock_get_dcs.return_value = self.e
        mock_get_dcs.return_value.get_cluster = get_cluster_initialized_with_leader

-        result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '-r', 'master'], input='y')
-        assert 'No scheduled restart' in result.output
+        for role in self.TEST_ROLES:
+            result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '-r', role], input='y')
+            assert 'No scheduled restart' in result.output

        result = self.runner.invoke(ctl, ['flush', 'dummy', 'restart', '--force'])
        assert 'Success: flush scheduled restart' in result.output
@@ -40,7 +40,11 @@ def etcd_read(self, key, **kwargs):
        raise etcd.EtcdKeyNotFound

    response = {"action": "get", "node": {"key": "/service/batman5", "dir": True, "nodes": [
-                {"key": "/service/batman5/config", "value": '{"synchronous_mode": 0}',
+                {"key": "/service/batman5/1", "dir": True, "nodes": [
+                    {"key": "/service/batman5/1/initialize", "value": "2164261704",
+                     "modifiedIndex": 20729, "createdIndex": 20729}],
+                 "modifiedIndex": 20437, "createdIndex": 20437},
+                {"key": "/service/batman5/config", "value": '{"synchronous_mode": 0, "failsafe_mode": true}',
                 "modifiedIndex": 1582, "createdIndex": 1582},
                {"key": "/service/batman5/failover", "value": "",
                 "modifiedIndex": 1582, "createdIndex": 1582},
@@ -266,8 +270,14 @@ class TestEtcd(unittest.TestCase):
        self.etcd._base_path = '/service/noleader'
        self.assertRaises(EtcdError, self.etcd.get_cluster)

+    def test__get_citus_cluster(self):
+        self.etcd._citus_group = '0'
+        cluster = self.etcd.get_cluster()
+        self.assertIsInstance(cluster, Cluster)
+        self.assertIsInstance(cluster.workers[1], Cluster)
+
    def test_touch_member(self):
-        self.assertFalse(self.etcd.touch_member('', ''))
+        self.assertFalse(self.etcd.touch_member(''))

    def test_take_leader(self):
        self.assertFalse(self.etcd.take_leader())
@@ -30,6 +30,8 @@ def mock_urlopen(self, method, url, **kwargs):
        ret.content = json.dumps({
            "header": {"revision": "1"},
            "kvs": [
+                {"key": base64_encode('/patroni/test/1/initialize'),
+                 "value": base64_encode('12345'), "mod_revision": '1'},
                {"key": base64_encode('/patroni/test/leader'),
                 "value": base64_encode('foo'), "lease": "bla", "mod_revision": '1'},
                {"key": base64_encode('/patroni/test/members/foo'),
@@ -207,6 +209,12 @@ class TestEtcd3(BaseTestEtcd3):
            mock_urlopen.side_effect = SleepException()
            self.assertRaises(Etcd3Error, self.etcd3.get_cluster)

+    def test__get_citus_cluster(self):
+        self.etcd3._citus_group = '0'
+        cluster = self.etcd3.get_cluster()
+        self.assertIsInstance(cluster, Cluster)
+        self.assertIsInstance(cluster.workers[1], Cluster)
+
    def test_touch_member(self):
        self.etcd3.touch_member({})
        self.etcd3._lease = 'bla'
@@ -13,6 +13,7 @@ from patroni.postgresql import Postgresql
 from patroni.postgresql.bootstrap import Bootstrap
 from patroni.postgresql.cancellable import CancellableSubprocess
 from patroni.postgresql.config import ConfigHandler
+from patroni.postgresql.postmaster import PostmasterProcess
 from patroni.postgresql.rewind import Rewind
 from patroni.postgresql.slots import SlotsHandler
 from patroni.utils import tzutc
@@ -33,12 +34,12 @@ def false(*args, **kwargs):
    return False


-def get_cluster(initialize, leader, members, failover, sync, cluster_config=None):
+def get_cluster(initialize, leader, members, failover, sync, cluster_config=None, failsafe=None):
    t = datetime.datetime.now().isoformat()
    history = TimelineHistory(1, '[[1,67197376,"no recovery target specified","' + t + '","foo"]]',
                              [(1, 67197376, 'no recovery target specified', t, 'foo')])
    cluster_config = cluster_config or ClusterConfig(1, {'check_timeline': True}, 1)
-    return Cluster(initialize, cluster_config, leader, 10, members, failover, sync, history, None, None)
+    return Cluster(initialize, cluster_config, leader, 10, members, failover, sync, history, None, failsafe)


 def get_cluster_not_initialized_without_leader(cluster_config=None):
@@ -49,9 +50,10 @@ def get_cluster_bootstrapping_without_leader(cluster_config=None):
    return get_cluster("", None, [], None, SyncState(None, None, None), cluster_config)


-def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None, cluster_config=None):
+def get_cluster_initialized_without_leader(leader=False, failover=None, sync=None, cluster_config=None, failsafe=False):
    m1 = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:[email protected]:5435/postgres',
-                                  'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4})
+                                  'api_url': 'http://127.0.0.1:8008/patroni', 'xlog_location': 4,
+                                  'role': 'primary', 'state': 'running'})
    leader = Leader(0, 0, m1 if leader else Member(0, '', 28, {}))
    m2 = Member(0, 'other', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres',
                                 'api_url': 'http://127.0.0.1:8011/patroni',
@@ -61,7 +63,8 @@ def get_cluster_initialized_without_leader(leader=False, failover=None, sync=Non
                                 'scheduled_restart': {'schedule': "2100-01-01 10:53:07.560445+00:00",
                                                       'postgres_version': '99.0.0'}})
    syncstate = SyncState(0 if sync else None, sync and sync[0], sync and sync[1])
-    return get_cluster(SYSID, leader, [m1, m2], failover, syncstate, cluster_config)
+    failsafe = {m.name: m.api_url for m in (m1, m2)} if failsafe else None
+    return get_cluster(SYSID, leader, [m1, m2], failover, syncstate, cluster_config, failsafe)


 def get_cluster_initialized_with_leader(failover=None, sync=None):
@@ -84,6 +87,11 @@ def get_standby_cluster_initialized_with_only_leader(failover=None, sync=None):
    )


+def get_cluster_initialized_with_leader_and_failsafe():
+    return get_cluster_initialized_without_leader(leader=True, failsafe=True,
+                                                  cluster_config=ClusterConfig(1, {'failsafe_mode': True}, 1))
+
+
 def get_node_status(reachable=True, in_recovery=True, dcs_last_seen=0,
                    timeline=2, wal_position=10, nofailover=False,
                    watchdog_failed=False):
@@ -143,7 +151,7 @@ zookeeper:
        self.scheduled_restart = {'schedule': future_restart_time,
                                  'postmaster_start_time': str(postmaster_start_time)}
        self.watchdog = Watchdog(self.config)
-        self.request = lambda member, **kwargs: requests_get(member.api_url, **kwargs)
+        self.request = lambda *args, **kwargs: requests_get(args[0].api_url, *args[1:], **kwargs)


 def run_async(self, func, args=()):
@@ -173,7 +181,7 @@ def run_async(self, func, args=()):
@patch.object(Postgresql, 'checkpoint', Mock())
@patch.object(CancellableSubprocess, 'call', Mock(return_value=0))
@patch.object(Postgresql, 'get_replica_timeline', Mock(return_value=2))
-@patch.object(Postgresql, 'get_master_timeline', Mock(return_value=2))
+@patch.object(Postgresql, 'get_primary_timeline', Mock(return_value=2))
@patch.object(ConfigHandler, 'restore_configuration_files', Mock())
@patch.object(etcd.Client, 'write', etcd_write)
@patch.object(etcd.Client, 'read', etcd_read)
@@ -182,6 +190,7 @@ def run_async(self, func, args=()):
@patch('patroni.async_executor.AsyncExecutor.busy', PropertyMock(return_value=False))
@patch('patroni.async_executor.AsyncExecutor.run_async', run_async)
@patch('patroni.postgresql.rewind.Thread', Mock())
+@patch('patroni.postgresql.citus.CitusHandler.start', Mock())
@patch('subprocess.call', Mock(return_value=0))
@patch('time.sleep', Mock())
 class TestHa(PostgresInit):
@@ -198,13 +207,15 @@ class TestHa(PostgresInit):
            self.p.postmaster_start_time = MagicMock(return_value=str(postmaster_start_time))
            self.p.can_create_replica_without_replication_connection = MagicMock(return_value=False)
            self.e = get_dcs({'etcd': {'ttl': 30, 'host': 'ok:2379', 'scope': 'test',
-                                       'name': 'foo', 'retry_timeout': 10}})
+                                       'name': 'foo', 'retry_timeout': 10},
+                              'citus': {'database': 'citus', 'group': None}})
            self.ha = Ha(MockPatroni(self.p, self.e))
            self.ha.old_cluster = self.e.get_cluster()
            self.ha.cluster = get_cluster_initialized_without_leader()
            self.ha.load_cluster_from_dcs = Mock()

    def test_update_lock(self):
+        self.ha.is_failsafe_mode = true
        self.p.last_operation = Mock(side_effect=PostgresConnectionException(''))
        self.ha.dcs.update_leader = Mock(side_effect=[DCSError(''), Exception])
        self.assertRaises(DCSError, self.ha.update_lock)
@@ -218,6 +229,9 @@ class TestHa(PostgresInit):
        self.p.timeline_wal_position = Mock(return_value=(0, 1, 1))
        self.p.set_role('standby_leader')
        self.ha.touch_member()
+        self.p.set_role('primary')
+        self.ha.dcs.touch_member = true
+        self.ha.touch_member()

    def test_is_leader(self):
        self.assertFalse(self.ha.is_leader())
@@ -266,11 +280,11 @@ class TestHa(PostgresInit):
        self.ha.dcs.__class__.__name__ = 'Raft'
        self.assertEqual(self.ha.run_cycle(), 'started as a secondary')

-    def test_recover_former_master(self):
+    def test_recover_former_primary(self):
        self.p.follow = false
        self.p.is_running = false
        self.p.name = 'leader'
-        self.p.set_role('master')
+        self.p.set_role('primary')
        self.p.controldata = lambda: {'Database cluster state': 'shut down', 'Database system identifier': SYSID}
        self.ha.cluster = get_cluster_initialized_with_leader()
        self.assertEqual(self.ha.run_cycle(), 'starting as readonly because i had the session lock')
@@ -303,15 +317,18 @@ class TestHa(PostgresInit):

    @patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
    @patch.object(Rewind, 'can_rewind', PropertyMock(return_value=True))
+    @patch('os.listdir', Mock(return_value=[]))
+    @patch('patroni.postgresql.rewind.fsync_dir', Mock())
    def test_recover_with_rewind(self):
        self.p.is_running = false
        self.ha.cluster = get_cluster_initialized_with_leader()
-        self.ha.cluster.leader.member.data.update(version='2.0.2', role='master')
+        self.ha.cluster.leader.member.data.update(version='2.0.2', role='primary')
        self.ha._rewind.pg_rewind = true
        self.ha._rewind.check_leader_is_not_in_recovery = true
        with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True)):
            self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')
-        with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=False)):
+        with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=False)),\
+                patch.object(Ha, 'is_synchronous_mode', Mock(return_value=True)):
            self.p.follow = true
            self.assertEqual(self.ha.run_cycle(), 'starting as a secondary')
            self.p.is_running = true
@@ -342,7 +359,7 @@ class TestHa(PostgresInit):
        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader because I had the session lock')

    @patch('patroni.psycopg.connect', psycopg_connect)
-    def test_acquire_lock_as_master(self):
+    def test_acquire_lock_as_primary(self):
        self.assertEqual(self.ha.run_cycle(), 'acquired session lock as a leader')

    def test_promoted_by_acquiring_lock(self):
@@ -367,7 +384,7 @@ class TestHa(PostgresInit):
        self.ha.cluster.is_unlocked = false
        self.ha.has_lock = true
        self.p.is_leader = false
-        self.p.set_role('master')
+        self.p.set_role('primary')
        self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')

    def test_demote_after_failing_to_obtain_lock(self):
@@ -410,6 +427,12 @@ class TestHa(PostgresInit):
        self.ha.has_lock = true
        self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')

+    def test_coordinator_leader_with_lock(self):
+        self.ha.cluster = get_cluster_initialized_with_leader()
+        self.ha.cluster.is_unlocked = false
+        self.ha.has_lock = true
+        self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')
+
    @patch.object(Postgresql, '_wait_for_connection_close', Mock())
    def test_demote_because_not_having_lock(self):
        self.ha.cluster.is_unlocked = false
@@ -445,7 +468,7 @@ class TestHa(PostgresInit):
    def test_follow_in_pause(self):
        self.ha.cluster.is_unlocked = false
        self.ha.is_paused = true
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
        self.p.is_leader = false
        self.assertEqual(self.ha.run_cycle(), 'PAUSE: no action. I am (postgresql0)')

@@ -457,12 +480,68 @@ class TestHa(PostgresInit):
        self.ha.cluster = get_cluster_initialized_with_leader()
        self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')

-    def test_no_etcd_connection_master_demote(self):
+    def test_no_dcs_connection_primary_demote(self):
        self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
        self.assertEqual(self.ha.run_cycle(), 'demoting self because DCS is not accessible and I was a leader')
        self.ha._async_executor.schedule('dummy')
        self.assertEqual(self.ha.run_cycle(), 'demoted self because DCS is not accessible and I was a leader')

+    def test_check_failsafe_topology(self):
+        self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
+        self.ha.cluster = get_cluster_initialized_with_leader_and_failsafe()
+        self.ha.dcs._last_failsafe = self.ha.cluster.failsafe
+        self.assertEqual(self.ha.run_cycle(), 'demoting self because DCS is not accessible and I was a leader')
+        self.ha.state_handler.name = self.ha.cluster.leader.name
+        self.assertFalse(self.ha.failsafe_is_active())
+        self.assertEqual(self.ha.run_cycle(),
+                         'continue to run as a leader because failsafe mode is enabled and all members are accessible')
+        self.assertTrue(self.ha.failsafe_is_active())
+        with patch.object(Postgresql, 'slots', Mock(side_effect=Exception)):
+            self.ha.patroni.request = Mock(side_effect=Exception)
+            self.assertEqual(self.ha.run_cycle(), 'demoting self because DCS is not accessible and I was a leader')
+            self.assertFalse(self.ha.failsafe_is_active())
+        self.ha.dcs._last_failsafe.clear()
+        self.ha.dcs._last_failsafe[self.ha.cluster.leader.name] = self.ha.cluster.leader.member.api_url
+        self.assertEqual(self.ha.run_cycle(),
+                         'continue to run as a leader because failsafe mode is enabled and all members are accessible')
+
+    def test_no_dcs_connection_primary_failsafe(self):
+        self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
+        self.ha.cluster = get_cluster_initialized_with_leader_and_failsafe()
+        self.ha.dcs._last_failsafe = self.ha.cluster.failsafe
+        self.ha.state_handler.name = self.ha.cluster.leader.name
+        self.assertEqual(self.ha.run_cycle(),
+                         'continue to run as a leader because failsafe mode is enabled and all members are accessible')
+
+    def test_readonly_dcs_primary_failsafe(self):
+        self.ha.cluster = get_cluster_initialized_with_leader_and_failsafe()
+        self.ha.dcs.update_leader = Mock(side_effect=DCSError('Etcd is not responding properly'))
+        self.ha.dcs._last_failsafe = self.ha.cluster.failsafe
+        self.ha.state_handler.name = self.ha.cluster.leader.name
+        self.assertEqual(self.ha.run_cycle(),
+                         'continue to run as a leader because failsafe mode is enabled and all members are accessible')
+
+    def test_no_dcs_connection_replica_failsafe(self):
+        self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
+        self.ha.cluster = get_cluster_initialized_with_leader_and_failsafe()
+        self.ha.update_failsafe({'name': 'leader', 'api_url': 'http://127.0.0.1:8008/patroni',
+                                 'conn_url': 'postgres://127.0.0.1:5432/postgres', 'slots': {'foo': 1000}})
+        self.p.is_leader = false
+        self.assertEqual(self.ha.run_cycle(), 'DCS is not accessible')
+
+    def test_no_dcs_connection_replica_failsafe_not_enabled_but_active(self):
+        self.ha.load_cluster_from_dcs = Mock(side_effect=DCSError('Etcd is not responding properly'))
+        self.ha.cluster = get_cluster_initialized_with_leader()
+        self.ha.update_failsafe({'name': 'leader', 'api_url': 'http://127.0.0.1:8008/patroni',
+                                 'conn_url': 'postgres://127.0.0.1:5432/postgres', 'slots': {'foo': 1000}})
+        self.p.is_leader = false
+        self.assertEqual(self.ha.run_cycle(), 'DCS is not accessible')
+
+    def test_update_failsafe(self):
+        self.assertRaises(Exception, self.ha.update_failsafe, {})
+        self.p.set_role('primary')
+        self.assertEqual(self.ha.update_failsafe({}), 'Running as a leader')
+
    @patch('time.sleep', Mock())
    def test_bootstrap_from_another_member(self):
        self.ha.cluster = get_cluster_initialized_with_leader()
@@ -487,6 +566,8 @@ class TestHa(PostgresInit):
        self.assertEqual(self.ha.bootstrap(), 'failed to acquire initialize lock')

    @patch('patroni.psycopg.connect', psycopg_connect)
+    @patch('patroni.postgresql.citus.connect', psycopg_connect)
+    @patch('patroni.postgresql.citus.quote_ident', Mock())
    @patch.object(Postgresql, 'connection', Mock(return_value=None))
    def test_bootstrap_initialized_new_cluster(self):
        self.ha.cluster = get_cluster_not_initialized_without_leader()
@@ -506,16 +587,20 @@ class TestHa(PostgresInit):
        self.assertRaises(PatroniFatalException, self.ha.post_bootstrap)

    @patch('patroni.psycopg.connect', psycopg_connect)
+    @patch('patroni.postgresql.citus.connect', psycopg_connect)
+    @patch('patroni.postgresql.citus.quote_ident', Mock())
    @patch.object(Postgresql, 'connection', Mock(return_value=None))
    def test_bootstrap_release_initialize_key_on_watchdog_failure(self):
        self.ha.cluster = get_cluster_not_initialized_without_leader()
        self.e.initialize = true
        self.ha.bootstrap()
-        self.p.is_running.return_value = MockPostmaster()
        self.p.is_leader = true
-        with patch.object(Watchdog, 'activate', Mock(return_value=False)):
+        with patch.object(Watchdog, 'activate', Mock(return_value=False)),\
+                patch('patroni.ha.logger.error') as mock_logger:
            self.assertEqual(self.ha.post_bootstrap(), 'running post_bootstrap')
            self.assertRaises(PatroniFatalException, self.ha.post_bootstrap)
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Cancelling bootstrap because'
+                                                                   ' watchdog activation failed'))

    @patch('patroni.psycopg.connect', psycopg_connect)
    def test_reinitialize(self):
@@ -542,6 +627,20 @@ class TestHa(PostgresInit):
        with patch.object(self.ha, "restart_matches", return_value=False):
            self.assertEqual(self.ha.restart({'foo': 'bar'}), (False, "restart conditions are not satisfied"))

+    @patch('time.sleep', Mock())
+    @patch.object(ConfigHandler, 'replace_pg_hba', Mock())
+    @patch.object(ConfigHandler, 'replace_pg_ident', Mock())
+    @patch.object(PostmasterProcess, 'start', Mock(return_value=MockPostmaster()))
+    @patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
+    def test_worker_restart(self):
+        self.ha.has_lock = true
+        self.ha.patroni.request = Mock()
+        self.p.is_running = Mock(side_effect=[Mock(), False])
+        self.assertEqual(self.ha.restart({}), (True, 'restarted successfully'))
+        self.ha.patroni.request.assert_called()
+        self.assertEqual(self.ha.patroni.request.call_args_list[0][0][3]['type'], 'before_demote')
+        self.assertEqual(self.ha.patroni.request.call_args_list[1][0][3]['type'], 'after_promote')
+
    @patch('os.kill', Mock())
    def test_restart_in_progress(self):
        with patch('patroni.async_executor.AsyncExecutor.busy', PropertyMock(return_value=True)):
@@ -556,7 +655,7 @@ class TestHa(PostgresInit):
            self.assertEqual(self.ha.run_cycle(), 'updated leader lock during restart')

            self.ha.update_lock = false
-            self.p.set_role('master')
+            self.p.set_role('primary')
            with patch('patroni.async_executor.CriticalTask.cancel', Mock(return_value=False)):
                with patch('patroni.postgresql.Postgresql.terminate_starting_postmaster') as mock_terminate:
                    self.assertEqual(self.ha.run_cycle(), 'lost leader lock during restart')
@@ -565,6 +664,7 @@ class TestHa(PostgresInit):
            self.ha.is_paused = true
            self.assertEqual(self.ha.run_cycle(), 'PAUSE: restart in progress')

+    @patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
    def test_manual_failover_from_leader(self):
        self.ha.fetch_node_status = get_node_status()
        self.ha.has_lock = true
@@ -691,7 +791,7 @@ class TestHa(PostgresInit):
        # manual failover when the `other` node isn't available but our name is in the /sync key
        self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None),
                                                                 sync=('leader1', 'postgresql0'))
-        self.p.pick_synchronous_standby = Mock(return_value=([], []))
+        self.p.sync_handler.current_state = Mock(return_value=([], []))
        self.ha.dcs.write_sync_state = true
        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')

@@ -707,15 +807,15 @@ class TestHa(PostgresInit):
        self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'postgresql0', None),
                                                                 sync=('leader1', 'other'))
        self.p.set_role('replica')
-        self.p.pick_synchronous_standby = Mock(return_value=(['leader1'], ['leader1']))
+        self.p.sync_handler.current_state = Mock(return_value=(['leader1'], ['leader1']))
        self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')

    def test_manual_failover_process_no_leader_in_pause(self):
        self.ha.is_paused = true
        self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, '', 'other', None))
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
        self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', '', None))
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')
        self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', 'blabla', None))
        self.assertEqual('PAUSE: acquired session lock as a leader', self.ha.run_cycle())
        self.p.is_leader = false
@@ -724,10 +824,15 @@ class TestHa(PostgresInit):
        self.assertEqual(self.ha.run_cycle(), 'PAUSE: promoted self to leader by acquiring session lock')

    def test_is_healthiest_node(self):
+        self.ha.is_failsafe_mode = true
        self.ha.state_handler.is_leader = false
        self.ha.patroni.nofailover = False
        self.ha.fetch_node_status = get_node_status()
+        self.ha.dcs._last_failsafe = {'foo': ''}
+        self.assertFalse(self.ha.is_healthiest_node())
+        self.ha.dcs._last_failsafe = {'postgresql0': ''}
        self.assertTrue(self.ha.is_healthiest_node())
+        self.ha.dcs._last_failsafe = None
        with patch.object(Watchdog, 'is_healthy', PropertyMock(return_value=False)):
            self.assertFalse(self.ha.is_healthiest_node())
        with patch('patroni.postgresql.Postgresql.is_starting', return_value=True):
@@ -764,14 +869,16 @@ class TestHa(PostgresInit):

    @patch.object(Rewind, 'pg_rewind', true)
    @patch.object(Rewind, 'check_leader_is_not_in_recovery', true)
+    @patch('os.listdir', Mock(return_value=[]))
+    @patch('patroni.postgresql.rewind.fsync_dir', Mock())
    def test_post_recover(self):
        self.p.is_running = false
        self.ha.has_lock = true
-        self.p.set_role('master')
+        self.p.set_role('primary')
        self.assertEqual(self.ha.post_recover(), 'removed leader key after trying and failing to start postgres')
        self.ha.has_lock = false
        self.assertEqual(self.ha.post_recover(), 'failed to start postgres')
-        leader = Leader(0, 0, Member(0, 'l', 2, {"version": "1.6", "conn_url": "postgres://a", "role": "master"}))
+        leader = Leader(0, 0, Member(0, 'l', 2, {"version": "1.6", "conn_url": "postgres://a", "role": "primary"}))
        self.ha._rewind.execute(leader)
        self.p.is_running = true
        self.assertIsNone(self.ha.post_recover())
@@ -818,7 +925,7 @@ class TestHa(PostgresInit):
        self.p._role = 'replica'
        self.p._connection.server_version = 90500
        self.p._pending_restart = True
-        self.assertFalse(self.ha.restart_matches("master", "9.5.0", True))
+        self.assertFalse(self.ha.restart_matches("primary", "9.5.0", True))
        self.assertFalse(self.ha.restart_matches("replica", "9.4.3", True))
        self.p._pending_restart = False
        self.assertFalse(self.ha.restart_matches("replica", "9.5.2", True))
@@ -829,9 +936,9 @@ class TestHa(PostgresInit):
        self.ha.is_paused = true
        self.p.name = 'leader'
        self.ha.cluster = get_cluster_initialized_with_leader()
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: removed leader lock because postgres is not running as master')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: removed leader lock because postgres is not running as primary')
        self.ha.cluster = get_cluster_initialized_with_leader(Failover(0, '', self.p.name, None))
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: waiting to become master after promote...')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: waiting to become primary after promote...')

    @patch('patroni.postgresql.mtime', Mock(return_value=1588316884))
    @patch.object(builtins, 'open', mock_open(read_data='1\t0/40159C0\tno recovery target specified\n'))
@@ -872,7 +979,7 @@ class TestHa(PostgresInit):
        self.p.name = 'replica'
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        self.ha.is_unlocked = true
-        self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_master:'))
+        self.assertTrue(self.ha.run_cycle().startswith('running pg_rewind from remote_member:'))

    def test_recover_unhealthy_leader_in_standby_cluster(self):
        self.p.is_leader = false
@@ -890,7 +997,7 @@ class TestHa(PostgresInit):
        self.ha.cluster = get_standby_cluster_initialized_with_only_leader()
        self.ha.cluster.is_unlocked = true
        self.ha.has_lock = false
-        self.assertEqual(self.ha.run_cycle(), 'trying to follow a remote master because standby cluster is unhealthy')
+        self.assertEqual(self.ha.run_cycle(), 'trying to follow a remote member because standby cluster is unhealthy')

    def test_failed_to_update_lock_in_pause(self):
        self.ha.update_lock = false
@@ -898,7 +1005,7 @@ class TestHa(PostgresInit):
        self.p.name = 'leader'
        self.ha.cluster = get_cluster_initialized_with_leader()
        self.assertEqual(self.ha.run_cycle(),
-                         'PAUSE: continue to run as master after failing to update leader lock in DCS')
+                         'PAUSE: continue to run as primary after failing to update leader lock in DCS')

    def test_postgres_unhealthy_in_pause(self):
        self.ha.is_paused = true
@@ -932,7 +1039,7 @@ class TestHa(PostgresInit):
        self.p.time_in_state = lambda: 350
        self.ha.fetch_node_status = get_node_status(reachable=False)  # inaccessible, in_recovery
        self.assertEqual(self.ha.run_cycle(),
-                         'master start has timed out, but continuing to wait because failover is not possible')
+                         'primary start has timed out, but continuing to wait because failover is not possible')
        check_calls([(update_lock, True), (demote, False)])

        self.ha.fetch_node_status = get_node_status()  # accessible, in_recovery
@@ -958,27 +1065,27 @@ class TestHa(PostgresInit):
        self.assertEqual(self.ha.run_cycle(), 'manual failover: demoting myself')

    @patch('patroni.ha.Ha.demote')
-    def test_failover_immediately_on_zero_master_start_timeout(self, demote):
+    def test_failover_immediately_on_zero_primary_start_timeout(self, demote):
        self.p.is_running = false
        self.ha.cluster = get_cluster_initialized_with_leader(sync=(self.p.name, 'other'))
        self.ha.cluster.config.data['synchronous_mode'] = True
-        self.ha.patroni.config.set_dynamic_configuration({'master_start_timeout': 0})
+        self.ha.patroni.config.set_dynamic_configuration({'primary_start_timeout': 0})
        self.ha.has_lock = true
        self.ha.update_lock = true
        self.ha.fetch_node_status = get_node_status()  # accessible, in_recovery
        self.assertEqual(self.ha.run_cycle(), 'stopped PostgreSQL to fail over after a crash')
        demote.assert_called_once()

-    def test_master_stop_timeout(self):
-        self.assertEqual(self.ha.master_stop_timeout(), None)
-        self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
+    def test_primary_stop_timeout(self):
+        self.assertEqual(self.ha.primary_stop_timeout(), None)
+        self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': 30})
        with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=True)):
-            self.assertEqual(self.ha.master_stop_timeout(), 30)
-        self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': 30})
+            self.assertEqual(self.ha.primary_stop_timeout(), 30)
+        self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': 30})
        with patch.object(Ha, 'is_synchronous_mode', Mock(return_value=False)):
-            self.assertEqual(self.ha.master_stop_timeout(), None)
-            self.ha.patroni.config.set_dynamic_configuration({'master_stop_timeout': None})
-            self.assertEqual(self.ha.master_stop_timeout(), None)
+            self.assertEqual(self.ha.primary_stop_timeout(), None)
+            self.ha.patroni.config.set_dynamic_configuration({'primary_stop_timeout': None})
+            self.assertEqual(self.ha.primary_stop_timeout(), None)

    @patch('patroni.postgresql.Postgresql.follow')
    def test_demote_immediate(self, follow):
@@ -989,7 +1096,7 @@ class TestHa(PostgresInit):

    def test_process_sync_replication(self):
        self.ha.has_lock = true
-        mock_set_sync = self.p.config.set_synchronous_standby = Mock()
+        mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
        self.p.name = 'leader'

        # Test sync key removed when sync mode disabled
@@ -1012,7 +1119,7 @@ class TestHa(PostgresInit):
        self.ha.is_synchronous_mode = true

        # Test sync standby not touched when picking the same node
-        self.p.pick_synchronous_standby = Mock(return_value=(['other'], ['other']))
+        self.p.sync_handler.current_state = Mock(return_value=(['other'], ['other']))
        self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
        self.ha.run_cycle()
        mock_set_sync.assert_not_called()
@@ -1020,13 +1127,13 @@ class TestHa(PostgresInit):
        mock_set_sync.reset_mock()

        # Test sync standby is replaced when switching standbys
-        self.p.pick_synchronous_standby = Mock(return_value=(['other2'], []))
+        self.p.sync_handler.current_state = Mock(return_value=(['other2'], []))
        self.ha.dcs.write_sync_state = Mock(return_value=True)
        self.ha.run_cycle()
        mock_set_sync.assert_called_once_with(['other2'])

        # Test sync standby is replaced when new standby is joined
-        self.p.pick_synchronous_standby = Mock(return_value=(['other2', 'other3'], ['other2']))
+        self.p.sync_handler.current_state = Mock(return_value=(['other2', 'other3'], ['other2']))
        self.ha.dcs.write_sync_state = Mock(return_value=True)
        self.ha.run_cycle()
        self.assertEqual(mock_set_sync.call_args_list[0][0], (['other2'],))
@@ -1043,7 +1150,7 @@ class TestHa(PostgresInit):
        self.ha.dcs.write_sync_state = Mock(return_value=True)
        self.ha.dcs.get_cluster = Mock(return_value=get_cluster_initialized_with_leader(sync=('leader', 'other')))
        # self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'other'))
-        self.p.pick_synchronous_standby = Mock(return_value=(['other2'], ['other2']))
+        self.p.sync_handler.current_state = Mock(return_value=(['other2'], ['other2']))
        self.ha.run_cycle()
        self.ha.dcs.get_cluster.assert_called_once()
        self.assertEqual(self.ha.dcs.write_sync_state.call_count, 2)
@@ -1066,14 +1173,14 @@ class TestHa(PostgresInit):
        # Test sync set to '*' when synchronous_mode_strict is enabled
        mock_set_sync.reset_mock()
        self.ha.is_synchronous_mode_strict = true
-        self.p.pick_synchronous_standby = Mock(return_value=([], []))
+        self.p.sync_handler.current_state = Mock(return_value=([], []))
        self.ha.run_cycle()
        mock_set_sync.assert_called_once_with(['*'])

-    def test_sync_replication_become_master(self):
+    def test_sync_replication_become_primary(self):
        self.ha.is_synchronous_mode = true

-        mock_set_sync = self.p.config.set_synchronous_standby = Mock()
+        mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
        self.p.is_leader = false
        self.p.set_role('replica')
        self.ha.has_lock = true
@@ -1081,17 +1188,17 @@ class TestHa(PostgresInit):
        self.p.name = 'leader'
        self.ha.cluster = get_cluster_initialized_with_leader(sync=('other', None))

-        # When we just became master nobody is sync
-        self.assertEqual(self.ha.enforce_master_role('msg', 'promote msg'), 'promote msg')
+        # When we just became primary nobody is sync
+        self.assertEqual(self.ha.enforce_primary_role('msg', 'promote msg'), 'promote msg')
        mock_set_sync.assert_called_once_with([])
        mock_write_sync.assert_called_once_with('leader', None, index=0)

        mock_set_sync.reset_mock()

-        # When we just became master nobody is sync
+        # When we just became primary nobody is sync
        self.p.set_role('replica')
        mock_write_sync.return_value = False
-        self.assertTrue(self.ha.enforce_master_role('msg', 'promote msg') != 'promote msg')
+        self.assertTrue(self.ha.enforce_primary_role('msg', 'promote msg') != 'promote msg')
        mock_set_sync.assert_not_called()

    def test_unhealthy_sync_mode(self):
@@ -1193,6 +1300,16 @@ class TestHa(PostgresInit):
        self.ha.is_failover_possible = true
        self.ha.shutdown()

+    @patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
+    def test_shutdown_citus_worker(self):
+        self.ha.is_leader = true
+        self.p.is_running = Mock(side_effect=[Mock(), False])
+        self.ha.patroni.request = Mock()
+        self.ha.shutdown()
+        self.ha.patroni.request.assert_called()
+        self.assertEqual(self.ha.patroni.request.call_args[0][2], 'citus')
+        self.assertEqual(self.ha.patroni.request.call_args[0][3]['type'], 'before_demote')
+
    @patch('time.sleep', Mock())
    def test_leader_with_not_accessible_data_directory(self):
        self.ha.cluster = get_cluster_initialized_with_leader()
@@ -1215,7 +1332,7 @@ class TestHa(PostgresInit):
        self.ha.has_lock = true
        self.ha.cluster.is_unlocked = false
        for tl in (1, 3):
-            self.p.get_master_timeline = Mock(return_value=tl)
+            self.p.get_primary_timeline = Mock(return_value=tl)
            self.assertEqual(self.ha.run_cycle(), 'no action. I am (postgresql0), the leader with the lock')

    @patch('sys.exit', return_value=1)
@@ -1260,7 +1377,7 @@ class TestHa(PostgresInit):
    def test_sysid_no_match_in_pause(self):
        self.ha.is_paused = true
        self.p.controldata = lambda: {'Database cluster state': 'in recovery', 'Database system identifier': '123'}
-        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as master without lock')
+        self.assertEqual(self.ha.run_cycle(), 'PAUSE: continue to run as primary without lock')

        self.ha.has_lock = true
        self.assertEqual(self.ha.run_cycle(), 'PAUSE: released leader key voluntarily due to the system ID mismatch')
@@ -1293,3 +1410,16 @@ class TestHa(PostgresInit):
        self.ha.dcs.attempt_to_acquire_leader = Mock(side_effect=[DCSError('foo'), Exception])
        self.assertRaises(DCSError, self.ha.acquire_lock)
        self.assertFalse(self.ha.acquire_lock())
+
+    @patch('patroni.postgresql.citus.CitusHandler.is_coordinator', Mock(return_value=False))
+    def test_notify_citus_coordinator(self):
+        self.ha.patroni.request = Mock()
+        self.ha.notify_citus_coordinator('before_demote')
+        self.ha.patroni.request.assert_called_once()
+        self.assertEqual(self.ha.patroni.request.call_args[1]['timeout'], 30)
+        self.ha.patroni.request = Mock(side_effect=Exception)
+        with patch('patroni.ha.logger.warning') as mock_logger:
+            self.ha.notify_citus_coordinator('before_promote')
+            self.assertEqual(self.ha.patroni.request.call_args[1]['timeout'], 2)
+            mock_logger.assert_called()
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Request to Citus coordinator'))
@@ -1,12 +1,13 @@
 import base64
 import datetime
 import json
+import mock
 import socket
 import time
 import unittest

-from mock import call, Mock, PropertyMock, mock_open, patch
-from patroni.dcs.kubernetes import k8s_client, k8s_config, K8sConfig, K8sConnectionFailed,\
+from mock import Mock, PropertyMock, mock_open, patch
+from patroni.dcs.kubernetes import Cluster, k8s_client, k8s_config, K8sConfig, K8sConnectionFailed,\
        K8sException, K8sObject, Kubernetes, KubernetesError, KubernetesRetriableException,\
        Retry, RetryFailedError, SERVICE_HOST_ENV_NAME, SERVICE_PORT_ENV_NAME
 from six.moves import builtins
@@ -25,6 +26,15 @@ def mock_list_namespaced_config_map(*args, **kwargs):
    items.append(k8s_client.V1ConfigMap(metadata=k8s_client.V1ObjectMeta(**metadata)))
    metadata.update({'name': 'test-sync', 'annotations': {'leader': 'p-0'}})
    items.append(k8s_client.V1ConfigMap(metadata=k8s_client.V1ObjectMeta(**metadata)))
+    metadata.update({'name': 'test-0-leader', 'labels': {Kubernetes._CITUS_LABEL: '0'},
+                     'annotations': {'optime': '1234x', 'leader': 'p-0', 'ttl': '30s', 'slots': '{', 'failsafe': '{'}})
+    items.append(k8s_client.V1ConfigMap(metadata=k8s_client.V1ObjectMeta(**metadata)))
+    metadata.update({'name': 'test-0-config', 'labels': {Kubernetes._CITUS_LABEL: '0'},
+                     'annotations': {'initialize': '123', 'config': '{}'}})
+    items.append(k8s_client.V1ConfigMap(metadata=k8s_client.V1ObjectMeta(**metadata)))
+    metadata.update({'name': 'test-1-leader', 'labels': {Kubernetes._CITUS_LABEL: '1'},
+                     'annotations': {'leader': 'p-3', 'ttl': '30s'}})
+    items.append(k8s_client.V1ConfigMap(metadata=k8s_client.V1ObjectMeta(**metadata)))
    metadata = k8s_client.V1ObjectMeta(resource_version='1')
    return k8s_client.V1ConfigMapList(metadata=metadata, items=items, kind='ConfigMapList')

@@ -47,7 +57,8 @@ def mock_list_namespaced_endpoints(*args, **kwargs):


 def mock_list_namespaced_pod(*args, **kwargs):
-    metadata = k8s_client.V1ObjectMeta(resource_version='1', name='p-0', annotations={'status': '{}'},
+    metadata = k8s_client.V1ObjectMeta(resource_version='1', labels={'f': 'b', Kubernetes._CITUS_LABEL: '1'},
+                                       name='p-0', annotations={'status': '{}'},
                                       uid='964dfeae-e79b-4476-8a5a-1920b5c2a69d')
    status = k8s_client.V1PodStatus(pod_ip='10.0.0.0')
    spec = k8s_client.V1PodSpec(hostname='p-0', node_name='kind-control-plane', containers=[])
@@ -134,7 +145,7 @@ class TestK8sConfig(unittest.TestCase):
            mock_atexit.assert_called_once()
            mock_remove.side_effect = OSError
            mock_atexit.call_args[0][0]()  # call _cleanup_temp_files
-            mock_remove.assert_has_calls([call('1.tmp'), call('2.tmp')])
+            mock_remove.assert_has_calls([mock.call('1.tmp'), mock.call('2.tmp')])


@patch('urllib3.PoolManager.request')
@@ -212,9 +223,10 @@ class BaseTestKubernetes(unittest.TestCase):
    @patch.object(k8s_client.CoreV1Api, 'list_namespaced_config_map', mock_list_namespaced_config_map, create=True)
    def setUp(self, config=None):
        config = config or {}
-        config.update(ttl=30, scope='test', name='p-0', loop_wait=10,
+        config.update(ttl=30, scope='test', name='p-0', loop_wait=10, group=0,
                      retry_timeout=10, labels={'f': 'b'}, bypass_api_service=True)
        self.k = Kubernetes(config)
+        self.k._citus_group = None
        self.assertRaises(AttributeError, self.k._pods._build_cache)
        self.k._pods._is_ready = True
        self.assertRaises(TypeError, self.k._kinds._build_cache)
@@ -238,6 +250,20 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
        with patch.object(Kubernetes, '_wait_caches', Mock(side_effect=Exception)):
            self.assertRaises(KubernetesError, self.k.get_cluster)

+    def test__get_citus_cluster(self):
+        self.k._citus_group = '0'
+        cluster = self.k.get_cluster()
+        self.assertIsInstance(cluster, Cluster)
+        self.assertIsInstance(cluster.workers[1], Cluster)
+
+    @patch('patroni.dcs.kubernetes.logger.error')
+    def test_get_citus_coordinator(self, mock_logger):
+        self.assertIsInstance(self.k.get_citus_coordinator(), Cluster)
+        with patch.object(Kubernetes, '_cluster_loader', Mock(side_effect=Exception)):
+            self.assertIsNone(self.k.get_citus_coordinator())
+            mock_logger.assert_called()
+            self.assertTrue(mock_logger.call_args[0][0].startswith('Failed to load Citus coordinator'))
+
    def test_attempt_to_acquire_leader(self):
        with patch.object(k8s_client.CoreV1Api, 'patch_namespaced_config_map', create=True) as mock_patch:
            mock_patch.side_effect = K8sException
@@ -267,7 +293,7 @@ class TestKubernetesConfigMaps(BaseTestKubernetes):
        self.k.touch_member({'role': 'replica'})
        self.k._name = 'p-1'
        self.k.touch_member({'state': 'running', 'role': 'replica'})
-        self.k.touch_member({'state': 'stopped', 'role': 'master'})
+        self.k.touch_member({'state': 'stopped', 'role': 'primary'})

    def test_initialize(self):
        self.k.initialize()
@@ -372,6 +398,7 @@ class TestCacheBuilder(BaseTestKubernetes):
    @patch.object(k8s_client.CoreV1Api, 'list_namespaced_config_map', mock_list_namespaced_config_map, create=True)
    @patch('patroni.dcs.kubernetes.ObjectCache._watch')
    def test__build_cache(self, mock_response):
+        self.k._citus_group = '0'
        mock_response.return_value.read_chunked.return_value = [json.dumps(
            {'type': 'MODIFIED', 'object': {'metadata': {
                'name': self.k.config_path, 'resourceVersion': '2', 'annotations': {self.k._CONFIG: 'foo'}}}}
@@ -143,7 +143,8 @@ class TestPatroni(unittest.TestCase):
        self.p.api.start = Mock()
        self.p.logger.start = Mock()
        self.p.config._dynamic_configuration = {}
-        self.assertRaises(SleepException, self.p.run)
+        with patch('patroni.dcs.Cluster.is_unlocked', Mock(return_value=True)):
+            self.assertRaises(SleepException, self.p.run)
        with patch('patroni.config.Config.reload_local_configuration', Mock(return_value=False)):
            self.p.sighup_handler()
            self.assertRaises(SleepException, self.p.run)
@@ -10,7 +10,7 @@ from mock import Mock, MagicMock, PropertyMock, patch, mock_open
 import patroni.psycopg as psycopg

 from patroni.async_executor import CriticalTask
-from patroni.dcs import Cluster, RemoteMember, SyncState
+from patroni.dcs import RemoteMember
 from patroni.exceptions import PostgresConnectionException, PatroniException
 from patroni.postgresql import Postgresql, STATE_REJECT, STATE_NO_RESPONSE
 from patroni.postgresql.bootstrap import Bootstrap
@@ -144,7 +144,8 @@ class TestPostgresql(BaseTestPostgresql):
    @patch('patroni.postgresql.polling_loop', Mock(return_value=range(1)))
    def test_wait_for_port_open(self, mock_pg_isready):
        mock_pg_isready.return_value = STATE_NO_RESPONSE
-        mock_postmaster = MockPostmaster(is_running=False)
+        mock_postmaster = MockPostmaster()
+        mock_postmaster.is_running.return_value = None

        # No pid file and postmaster death
        self.assertFalse(self.p.wait_for_port_open(mock_postmaster, 1))
@@ -438,6 +439,8 @@ class TestPostgresql(BaseTestPostgresql):
        self.assertTrue(self.p.stop())

    @patch('os.rename', Mock())
+    @patch('os.path.exists', Mock(return_value=True))
+    @patch('shutil.rmtree', Mock())
    @patch('os.path.isdir', Mock(return_value=True))
    @patch('os.unlink', Mock())
    @patch('os.symlink', Mock())
@@ -500,13 +503,13 @@ class TestPostgresql(BaseTestPostgresql):
        self.p.config._config['create_replica_method'] = []
        self.assertFalse(self.p.can_create_replica_without_replication_connection())
        self.p.config._config['create_replica_method'] = ['wale', 'basebackup']
-        self.p.config._config['wale'] = {'command': 'foo', 'no_master': 1}
+        self.p.config._config['wale'] = {'command': 'foo', 'no_leader': 1}
        self.assertTrue(self.p.can_create_replica_without_replication_connection())

    def test_replica_method_can_work_without_replication_connection(self):
        self.assertFalse(self.p.replica_method_can_work_without_replication_connection('basebackup'))
        self.assertFalse(self.p.replica_method_can_work_without_replication_connection('foobar'))
-        self.p.config._config['foo'] = {'command': 'bar', 'no_master': 1}
+        self.p.config._config['foo'] = {'command': 'bar', 'no_leader': 1}
        self.assertTrue(self.p.replica_method_can_work_without_replication_connection('foo'))
        self.p.config._config['foo'] = {'command': 'bar'}
        self.assertFalse(self.p.replica_method_can_work_without_replication_connection('foo'))
@@ -638,79 +641,12 @@ class TestPostgresql(BaseTestPostgresql):
            self.p._state = 'starting'
            self.assertIsNone(self.p.wait_for_startup())

-    def test_pick_sync_standby(self):
-        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None,
-                          SyncState(0, self.me.name, self.leadermem.name), None, None, None)
-        mock_cursor = Mock()
-        mock_cursor.fetchone.return_value = ('remote_apply',)
-
-        with patch.object(Postgresql, "query", side_effect=[
-                    mock_cursor,
-                    [(self.leadermem.name, 'sync', 1),
-                     (self.me.name, 'async', 2),
-                     (self.other.name, 'async', 2)]
-                ]):
-            self.assertEqual(self.p.pick_synchronous_standby(cluster), ([self.leadermem.name], [self.leadermem.name]))
-
-        with patch.object(Postgresql, "query", side_effect=[
-                    mock_cursor,
-                    [(self.leadermem.name, 'potential', 1),
-                     (self.me.name, 'async', 2),
-                     (self.other.name, 'async', 2)]
-                ]):
-            self.assertEqual(self.p.pick_synchronous_standby(cluster), ([self.leadermem.name], []))
-
-        with patch.object(Postgresql, "query", side_effect=[
-                    mock_cursor,
-                    [(self.me.name, 'async', 1),
-                     (self.other.name, 'async', 2)]
-                ]):
-            self.assertEqual(self.p.pick_synchronous_standby(cluster), ([self.me.name], []))
-
-        with patch.object(Postgresql, "query", side_effect=[
-                    mock_cursor,
-                    [('missing', 'sync', 1),
-                     (self.me.name, 'async', 2),
-                     (self.other.name, 'async', 3)]
-                ]):
-            self.assertEqual(self.p.pick_synchronous_standby(cluster), ([self.me.name], []))
-
-        with patch.object(Postgresql, "query", side_effect=[mock_cursor, []]):
-            self.p._major_version = 90400
-            self.assertEqual(self.p.pick_synchronous_standby(cluster), ([], []))
-
-    def test_set_sync_standby(self):
-        def value_in_conf():
-            with open(os.path.join(self.p.data_dir, 'postgresql.conf')) as f:
-                for line in f:
-                    if line.startswith('synchronous_standby_names'):
-                        return line.strip()
-
-        mock_reload = self.p.reload = Mock()
-        self.p.config.set_synchronous_standby(['n1'])
-        self.assertEqual(value_in_conf(), "synchronous_standby_names = 'n1'")
-        mock_reload.assert_called()
-
-        mock_reload.reset_mock()
-        self.p.config.set_synchronous_standby(['n1'])
-        mock_reload.assert_not_called()
-        self.assertEqual(value_in_conf(), "synchronous_standby_names = 'n1'")
-
-        self.p.config.set_synchronous_standby(['n1', 'n2'])
-        mock_reload.assert_called()
-        self.assertEqual(value_in_conf(), "synchronous_standby_names = '2 (n1,n2)'")
-
-        mock_reload.reset_mock()
-        self.p.config.set_synchronous_standby([])
-        mock_reload.assert_called()
-        self.assertEqual(value_in_conf(), None)
-
    def test_get_server_parameters(self):
        config = {'synchronous_mode': True, 'parameters': {'wal_level': 'hot_standby'}, 'listen': '0'}
        self.p.config.get_server_parameters(config)
        config['synchronous_mode_strict'] = True
        self.p.config.get_server_parameters(config)
-        self.p.config.set_synchronous_standby('foo')
+        self.p.config.set_synchronous_standby_names('foo')
        self.assertTrue(str(self.p.config.get_server_parameters(config)).startswith('{'))

    @patch('time.sleep', Mock())
@@ -734,8 +670,8 @@ class TestPostgresql(BaseTestPostgresql):
    def test_replica_cached_timeline(self):
        self.assertEqual(self.p.replica_cached_timeline(2), 3)

-    def test_get_master_timeline(self):
-        self.assertEqual(self.p.get_master_timeline(), 1)
+    def test_get_primary_timeline(self):
+        self.assertEqual(self.p.get_primary_timeline(), 1)

    @patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='replica'))
    @patch.object(Bootstrap, 'running_custom_bootstrap', PropertyMock(return_value=True))
@@ -4,7 +4,8 @@ import tempfile
 import time

 from mock import Mock, PropertyMock, patch
-from patroni.dcs.raft import DynMemberSyncObj, KVStoreTTL, Raft, RaftError, SyncObjUtility, TCPTransport, _TCPTransport
+from patroni.dcs.raft import Cluster, DynMemberSyncObj, KVStoreTTL,\
+        Raft, RaftError, SyncObjUtility, TCPTransport, _TCPTransport
 from pysyncobj import SyncObjConf, FAIL_REASON


@@ -128,7 +129,8 @@ class TestRaft(unittest.TestCase):

    def test_raft(self):
        raft = Raft({'ttl': 30, 'scope': 'test', 'name': 'pg', 'self_addr': '127.0.0.1:1234',
-                     'retry_timeout': 10, 'data_dir': self._TMP})
+                     'retry_timeout': 10, 'data_dir': self._TMP,
+                     'database': 'citus', 'group': 0})
        raft.reload_config({'retry_timeout': 20, 'ttl': 60, 'loop_wait': 10})
        self.assertTrue(raft._sync_obj.set(raft.members_path + 'legacy', '{"version":"2.0.0"}'))
        self.assertTrue(raft.touch_member(''))
@@ -136,20 +138,28 @@ class TestRaft(unittest.TestCase):
        self.assertTrue(raft.cancel_initialization())
        self.assertTrue(raft.set_config_value('{}'))
        self.assertTrue(raft.write_sync_state('foo', 'bar'))
+        raft._citus_group = '1'
        self.assertTrue(raft.manual_failover('foo', 'bar'))
-        raft.get_cluster()
+        raft._citus_group = '0'
+        cluster = raft.get_cluster()
+        self.assertIsInstance(cluster, Cluster)
+        self.assertIsInstance(cluster.workers[1], Cluster)
        self.assertTrue(raft._sync_obj.set(raft.status_path, '{"optime":1234567,"slots":{"ls":12345}}'))
        raft.get_cluster()
        self.assertTrue(raft.update_leader('1', failsafe={'foo': 'bat'}))
        self.assertTrue(raft._sync_obj.set(raft.failsafe_path, '{"foo"}'))
        self.assertTrue(raft._sync_obj.set(raft.status_path, '{'))
-        raft.get_cluster()
+        raft.get_citus_coordinator()
        self.assertTrue(raft.delete_sync_state())
        self.assertTrue(raft.delete_leader())
        self.assertTrue(raft.set_history_value(''))
        self.assertTrue(raft.delete_cluster())
+        raft._citus_group = '1'
+        self.assertTrue(raft.delete_cluster())
+        raft._citus_group = None
        raft.get_cluster()
        self.assertTrue(raft.take_leader())
+        raft.get_cluster()
        raft.watch(None, 0.001)
        raft._sync_obj.destroy()

@@ -117,7 +117,7 @@ class TestRewind(BaseTestPostgresql):
            self.r.trigger_check_diverged_lsn()
            self.r.execute(self.leader)

-        self.leader.member.data.update(version='1.5.7', checkpoint_after_promote=False, role='master')
+        self.leader.member.data.update(version='1.5.7', checkpoint_after_promote=False, role='primary')
        self.assertIsNone(self.r.execute(self.leader))

        del self.leader.member.data['checkpoint_after_promote']
@@ -128,9 +128,9 @@ class TestRewind(BaseTestPostgresql):
            self.r.execute(self.leader)

    @patch('patroni.postgresql.rewind.logger.info')
-    def test__log_master_history(self, mock_logger):
+    def test__log_primary_history(self, mock_logger):
        history = [[n, n, ''] for n in range(1, 10)]
-        self.r._log_master_history(history, 1)
+        self.r._log_primary_history(history, 1)
        expected = '\n'.join(['{0}\t0/{0}\t'.format(n) for n in range(1, 4)] + ['...', '9\t0/9\t'])
        self.assertEqual(mock_logger.call_args[0][1], expected)

@@ -274,6 +274,19 @@ class TestRewind(BaseTestPostgresql):
            self.r._archive_ready_wals()
            mock_logger_info.assert_not_called()

+    @patch.object(Postgresql, 'major_version', PropertyMock(return_value=100000))
+    @patch('os.listdir', Mock(side_effect=[OSError, ['something', 'something_else']]))
+    @patch('shutil.rmtree', Mock())
+    @patch('patroni.postgresql.rewind.fsync_dir', Mock())
+    @patch('patroni.postgresql.rewind.logger.warning')
+    def test_maybe_clean_pg_replslot(self, mock_logger):
+        # failed to list pg_replslot/
+        self.assertIsNone(self.r._maybe_clean_pg_replslot())
+        mock_logger.assert_called_once()
+        mock_logger.reset_mock()
+
+        self.assertIsNone(self.r._maybe_clean_pg_replslot())
+
    @patch('os.unlink', Mock())
    @patch('os.listdir', Mock(return_value=[]))
    @patch('os.path.isfile', Mock(return_value=True))
@@ -285,14 +298,14 @@ class TestRewind(BaseTestPostgresql):
    @patch('patroni.postgresql.rewind.Thread', MockThread)
    @patch.object(Postgresql, 'controldata')
    @patch.object(Postgresql, 'checkpoint')
-    @patch.object(Postgresql, 'get_master_timeline')
-    def test_ensure_checkpoint_after_promote(self, mock_get_master_timeline, mock_checkpoint, mock_controldata):
+    @patch.object(Postgresql, 'get_primary_timeline')
+    def test_ensure_checkpoint_after_promote(self, mock_get_primary_timeline, mock_checkpoint, mock_controldata):
        mock_controldata.return_value = {"Latest checkpoint's TimeLineID": 1}
-        mock_get_master_timeline.return_value = 1
+        mock_get_primary_timeline.return_value = 1
        self.r.ensure_checkpoint_after_promote(Mock())

        self.r.reset_state()
-        mock_get_master_timeline.return_value = 2
+        mock_get_primary_timeline.return_value = 2
        mock_checkpoint.return_value = 0
        self.r.ensure_checkpoint_after_promote(Mock())
        self.r.ensure_checkpoint_after_promote(Mock())
@@ -9,7 +9,8 @@ from threading import Thread
 from patroni import psycopg
 from patroni.dcs import Cluster, ClusterConfig, Member
 from patroni.postgresql import Postgresql
-from patroni.postgresql.slots import SlotsAdvanceThread, SlotsHandler, fsync_dir
+from patroni.postgresql.misc import fsync_dir
+from patroni.postgresql.slots import SlotsAdvanceThread, SlotsHandler

 from . import BaseTestPostgresql, psycopg_connect, MockCursor

@@ -42,17 +43,19 @@ class TestSlotsHandler(BaseTestPostgresql):
        with mock.patch('patroni.postgresql.Postgresql._query', Mock(side_effect=psycopg.OperationalError)):
            self.s.sync_replication_slots(cluster, False)
        self.p.set_role('standby_leader')
-        self.s.sync_replication_slots(cluster, False)
+        with patch.object(SlotsHandler, 'drop_replication_slot', Mock(return_value=(True, False))),\
+                patch('patroni.postgresql.slots.logger.debug') as mock_debug:
+            self.s.sync_replication_slots(cluster, False)
+            mock_debug.assert_called_once()
        self.p.set_role('replica')
        with patch.object(Postgresql, 'is_leader', Mock(return_value=False)),\
                patch.object(SlotsHandler, 'drop_replication_slot') as mock_drop:
            self.s.sync_replication_slots(cluster, False, paused=True)
            mock_drop.assert_not_called()
-        self.p.set_role('master')
+        self.p.set_role('primary')
        with mock.patch('patroni.postgresql.Postgresql.role', new_callable=PropertyMock(return_value='replica')):
            self.s.sync_replication_slots(cluster, False)
-        with patch.object(SlotsHandler, 'drop_replication_slot', Mock(return_value=True)),\
-                patch('patroni.dcs.logger.error', new_callable=Mock()) as errorlog_mock:
+        with patch('patroni.dcs.logger.error', new_callable=Mock()) as errorlog_mock:
            alias1 = Member(0, 'test-3', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres'})
            alias2 = Member(0, 'test.3', 28, {'conn_url': 'postgres://replicator:[email protected]:5436/postgres'})
            cluster.members.extend([alias1, alias2])
@@ -0,0 +1,89 @@
+import os
+
+from mock import Mock, patch
+
+from patroni.dcs import Cluster, SyncState
+from patroni.postgresql import Postgresql
+
+from . import BaseTestPostgresql, psycopg_connect
+
+
+@patch('subprocess.call', Mock(return_value=0))
+@patch('patroni.psycopg.connect', psycopg_connect)
+class TestSync(BaseTestPostgresql):
+
+    @patch('subprocess.call', Mock(return_value=0))
+    @patch('os.rename', Mock())
+    @patch('patroni.postgresql.CallbackExecutor', Mock())
+    @patch.object(Postgresql, 'get_major_version', Mock(return_value=140000))
+    @patch.object(Postgresql, 'is_running', Mock(return_value=True))
+    def setUp(self):
+        super(TestSync, self).setUp()
+        self.p.config.write_postgresql_conf()
+        self.s = self.p.sync_handler
+
+    @patch.object(Postgresql, 'last_operation', Mock(return_value=1))
+    def test_pick_sync_standby(self):
+        cluster = Cluster(True, None, self.leader, 0, [self.me, self.other, self.leadermem], None,
+                          SyncState(0, self.me.name, self.leadermem.name), None, None, None)
+
+        pg_stat_replication = [
+            {'pid': 100, 'application_name': self.leadermem.name, 'sync_state': 'sync', 'flush_lsn': 1},
+            {'pid': 101, 'application_name': self.me.name, 'sync_state': 'async', 'flush_lsn': 2},
+            {'pid': 102, 'application_name': self.other.name, 'sync_state': 'async', 'flush_lsn': 2}]
+
+        # sync node is a bit behind of async, but we prefer it anyway
+        with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name,
+                                                                              'on', pg_stat_replication]):
+            self.assertEqual(self.s.current_state(cluster), ([self.leadermem.name], [self.leadermem.name]))
+
+        # prefer node with sync_state='potential', even if it is slightly behind of async
+        pg_stat_replication[0]['sync_state'] = 'potential'
+        for r in pg_stat_replication:
+            r['write_lsn'] = r.pop('flush_lsn')
+        with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_write', pg_stat_replication]):
+            self.assertEqual(self.s.current_state(cluster), ([self.leadermem.name], []))
+
+        # when there are no sync or potential candidates we pick async with the minimal replication lag
+        for i, r in enumerate(pg_stat_replication):
+            r.update(replay_lsn=3 - i, application_name=r['application_name'].upper())
+        missing = pg_stat_replication.pop(0)
+        with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_apply', pg_stat_replication]):
+            self.assertEqual(self.s.current_state(cluster), ([self.me.name], []))
+
+        # unknown sync node is ignored
+        missing.update(application_name='missing', sync_state='sync')
+        pg_stat_replication.insert(0, missing)
+        with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_apply', pg_stat_replication]):
+            self.assertEqual(self.s.current_state(cluster), ([self.me.name], []))
+
+        # invalid synchronous_standby_names and empty pg_stat_replication
+        with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['a b', 'remote_apply', None]):
+            self.p._major_version = 90400
+            self.assertEqual(self.s.current_state(cluster), ([], []))
+
+    def test_set_sync_standby(self):
+        def value_in_conf():
+            with open(os.path.join(self.p.data_dir, 'postgresql.conf')) as f:
+                for line in f:
+                    if line.startswith('synchronous_standby_names'):
+                        return line.strip()
+
+        mock_reload = self.p.reload = Mock()
+        self.s.set_synchronous_standby_names(['n1'])
+        self.assertEqual(value_in_conf(), "synchronous_standby_names = 'n1'")
+        mock_reload.assert_called()
+
+        mock_reload.reset_mock()
+        self.s.set_synchronous_standby_names(['n1'])
+        mock_reload.assert_not_called()
+        self.assertEqual(value_in_conf(), "synchronous_standby_names = 'n1'")
+
+        self.s.set_synchronous_standby_names(['n1', 'n2'])
+        mock_reload.assert_called()
+        self.assertEqual(value_in_conf(), "synchronous_standby_names = '2 (n1,n2)'")
+
+        mock_reload.reset_mock()
+        self.s.set_synchronous_standby_names([])
+        mock_reload.assert_called()
+        self.assertEqual(value_in_conf(), None)
@@ -59,22 +59,22 @@ class TestWALERestore(unittest.TestCase):
            self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())

        with patch('patroni.psycopg.connect', Mock(side_effect=psycopg.Error("foo"))):
-            save_no_master = self.wale_restore.no_master
-            save_master_connection = self.wale_restore.master_connection
+            save_no_leader = self.wale_restore.no_leader
+            save_leader_connection = self.wale_restore.leader_connection

            self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())

            with patch('time.sleep', mock_sleep):
-                self.wale_restore.no_master = 1
+                self.wale_restore.no_leader = 1
                self.assertTrue(self.wale_restore.should_use_s3_to_create_replica())
                # verify retries
                self.assertEqual(sleeps[0], WALE_TEST_RETRIES)

-            self.wale_restore.master_connection = ''
+            self.wale_restore.leader_connection = ''
            self.assertTrue(self.wale_restore.should_use_s3_to_create_replica())

-            self.wale_restore.no_master = save_no_master
-            self.wale_restore.master_connection = save_master_connection
+            self.wale_restore.no_leader = save_no_leader
+            self.wale_restore.leader_connection = save_leader_connection

        with patch('subprocess.check_output', Mock(side_effect=subprocess.CalledProcessError(1, "cmd", "foo"))):
            self.assertFalse(self.wale_restore.should_use_s3_to_create_replica())
@@ -62,7 +62,7 @@ class MockKazooClient(Mock):
        if path.startswith('/no_node'):
            raise NoNodeError
        elif path in ['/service/bla/', '/service/test/']:
-            return ['initialize', 'leader', 'members', 'optime', 'failover', 'sync', 'failsafe']
+            return ['initialize', 'leader', 'members', 'optime', 'failover', 'sync', 'failsafe', '0', '1']
        return ['foo', 'bar', 'buzz']

    def create(self, path, value=b"", acl=None, ephemeral=False, sequence=False, makepath=False):
@@ -155,6 +155,11 @@ class TestZooKeeper(unittest.TestCase):
    def test_session_listener(self):
        self.zk.session_listener(KazooState.SUSPENDED)

+    def test_members_watcher(self):
+        self.zk._fetch_cluster = False
+        self.zk.members_watcher(None)
+        self.assertTrue(self.zk._fetch_cluster)
+
    def test_reload_config(self):
        self.zk.reload_config({'ttl': 20, 'retry_timeout': 10, 'loop_wait': 10})
        self.zk.reload_config({'ttl': 20, 'retry_timeout': 10, 'loop_wait': 5})
@@ -165,15 +170,15 @@ class TestZooKeeper(unittest.TestCase):
    def test_get_children(self):
        self.assertListEqual(self.zk.get_children('/no_node'), [])

-    def test__inner_load_cluster(self):
+    def test__cluster_loader(self):
        self.zk._base_path = self.zk._base_path.replace('test', 'bla')
-        self.zk._inner_load_cluster()
+        self.zk._cluster_loader(self.zk.client_path(''))
        self.zk._base_path = self.zk._base_path = '/broken'
-        self.zk._inner_load_cluster()
+        self.zk._cluster_loader(self.zk.client_path(''))
        self.zk._base_path = self.zk._base_path = '/legacy'
-        self.zk._inner_load_cluster()
+        self.zk._cluster_loader(self.zk.client_path(''))
        self.zk._base_path = self.zk._base_path = '/no_node'
-        self.zk._inner_load_cluster()
+        self.zk._cluster_loader(self.zk.client_path(''))

    def test_get_cluster(self):
        cluster = self.zk.get_cluster(True)
@@ -188,6 +193,19 @@ class TestZooKeeper(unittest.TestCase):
        cluster = self.zk.get_cluster()
        self.assertEqual(cluster.last_lsn, 500)

+    def test__get_citus_cluster(self):
+        self.zk._citus_group = '0'
+        for _ in range(0, 2):
+            cluster = self.zk.get_cluster()
+            self.assertIsInstance(cluster, Cluster)
+            self.assertIsInstance(cluster.workers[1], Cluster)
+
+    @patch('patroni.dcs.zookeeper.logger.error')
+    @patch.object(ZooKeeper, '_cluster_loader', Mock(side_effect=Exception))
+    def test_get_citus_coordinator(self, mock_logger):
+        self.assertIsNone(self.zk.get_citus_coordinator())
+        mock_logger.assert_called_once()
+
    def test_delete_leader(self):
        self.assertTrue(self.zk.delete_leader())
Author	SHA1	Message	Date
Polina BunginaandGitHub	422047f105	Release 3.0.1 (#2561 ) * Bump version * Update release notes * Return 3.6 to supported versions in setup.py	2023-02-16 08:51:47 +01:00
Polina Bungina GitHub Alexander Kukushkin	b85f155dbe	Pass 'master' role to a callback script instead of 'promoted' (#2554 ) Co-authored-by: Alexander Kukushkin <[email protected]>	2023-02-08 14:09:51 +01:00
Alexander KukushkinandGitHub	1669a49b2d	Switch to Citus 11.2 (#2548 ) - Update Dockerfile.citus files - Enable behave tests with Citus	2023-02-03 15:29:25 +01:00
Alexander KukushkinandGitHub	8ac8ed6584	Update Citus link to the github.com repo (#2546 ) Per suggestion from @clairegiordano	2023-02-02 11:50:19 +01:00
Alexander KukushkinandGitHub	7869f5e211	Release 3.0.0 (#2545 ) * bump version * update release notes * removed 2.7, 3.4, 3.5, and 3.6 from supported versions in setup.py * switched GH actions back to ubuntu-latest, removed tests with 2.7 and 3.6, and added 3.11 * some little fixes in Citus documentation and behave tests	2023-01-30 10:29:08 +01:00
Alexander KukushkinandGitHub	45e5ac2baf	Remove patronictl scaffold (#2544 ) The only reason for having it was a hacky way of running standby clusters.	2023-01-27 08:52:59 +01:00
Alexander KukushkinandGitHub	4c3af2d1a0	Change master->primary/leader/member (#2541 ) keep as much backward compatibility as possible. Following changes were made: 1. All internal checks are performed as `role in ('master', 'primary')` 2. All internal variables/functions/methods are renamed 3. `GET /metrics` endpoint returns `patroni_primary` in addition to `patroni_master`. 4. Logs are changed to use leader/primary/member/remote depending on the context 5. Unit-tests are using only role = 'primary' instead of 'master' to verify that 1 works. 6. patronictl still supports old syntax, but also accepts `--leader` and `--primary`. 7. `master_(start\|stop)_timeout` is automatically translated to `primary_(start\|stop)_timeout` if the last one is not set. 8. updated the documentation and some examples Future plan: in the next major release switch role name from `master` to `primary` and maybe drop `master` altogether. The Kubernetes implementation will require more work and keep two labels in parallel. Label values should probably be configurable as described in https://github.com/zalando/patroni/issues/2495.	2023-01-27 07:40:24 +01:00
Alexander KukushkinandGitHub	0273eac15e	Compatibility with pyinstaller (#2537 ) it doesn't like relative imports and not recognise `http.server` imported with `six`. The last one is explicitly added to the list of `hiddenimports()` and will break compatibility with python 2.7, which support will be dropped in the next Patroni release anyway. Close https://github.com/zalando/patroni/issues/2535	2023-01-26 16:35:30 +01:00
Alexander KukushkinandGitHub	79458688d1	Check unexpected exceptions in Patroni logs after behave (#2538 ) and make behave fail if there are anything unexpected found. In addition to that fix globing rule when uploading artifacts with logs.	2023-01-25 11:02:52 +01:00
Alexander KukushkinandGitHub	4872ac51e0	Citus integration (#2504 ) Citus cluster (coordinator and workers) will be stored in DCS as a fleet of Patroni logically grouped together: ``` /service/batman/ /service/batman/0/ /service/batman/0/initialize /service/batman/0/leader /service/batman/0/members/ /service/batman/0/members/m1 /service/batman/0/members/m2 /service/batman/ /service/batman/1/ /service/batman/1/initialize /service/batman/1/leader /service/batman/1/members/ /service/batman/1/members/m1 /service/batman/1/members/m2 ... ``` Where 0 is a Citus group for coordinator and 1, 2, etc are worker groups. Such hierarchy allows reading the entire Citus cluster with a single call to DCS (except Zookeeper). The get_cluster() method will be reading the entire Citus cluster on the coordinator because it needs to discover workers. For the worker cluster it will be reading the subtree of its own group. Besides that we introduce a new method get_citus_coordinator(). It will be used only by worker clusters. Since there is no hierarchical structures on K8s we will use the citus group suffix on all objects that Patroni creates. E.g. ``` batman-0-leader # the leader config map for the coordinator batman-0-config # the config map holding initialize, config, and history "keys" ... batman-1-leader # the leader config map for worker group 1 batman-1-config ... ``` Citus integration is enabled from patroni.yaml: ```yaml citus: database: citus group: 0 # 0 is for coordinator, 1, 2, etc are for workers ``` If enabled, Patroni will create the database, citus extension in it, and INSERTs INTO `pg_dist_authinfo` information required for Citus nodes to communicate between each other, i.e. 'password', 'sslcert', 'sslkey' for superuser if they are defined in the Patroni configuration file. When the new Citus coordinator/worker is bootstrapped, Patroni adds `synchronous_mode: on` to the `bootstrap.dcs` section. Besides that, Patroni takes over management of some Postgres GUCs: - `shared_preload_libraries` - Patroni ensures that the "citus" is added to the first place - `max_prepared_transactions` - if not set or set to 0, Patroni changes the value to `max_connections*2` - wal_level - automatically set to logical. It is used by Citus to move/split shards. Under the hood Citus is creating/removing replication slots and they are automatically added by Patroni to the `ignore_slots` configuration to avoid accidental removal. The coordinator primary actively discovers worker primary nodes and registers/updates them in the `pg_dist_node` table using citus_add_node() and citus_update_node() functions. Patroni running on the coordinator provides the new REST API endpoint: `POST /citus`. It is used by workers to facilitate controlled switchovers and restarts of worker primaries. When the worker primary needs to shut down Postgres because of restart or switchover, it calls the `POST /citus` endpoint on the coordinator and the Patroni on the coordinator starts a transaction and calls `citus_update_node(nodeid, 'host-demoted', port)` in order to pause client connections that work with the given worker. Once the new leader is elected or postgres started back, they perform another call to the `POST/citus` endpoint, that does another `citus_update_node()` call with actual hostname and port and commits a transaction. After transaction is committed, coordinator reestablishes connections to the worker node and client connections are unblocked. If clients don't run long transaction the operation finishes without client visible errors, but only a short latency spike. All operations on the `pg_dist_node` are serialized by Patroni on the coordinator. It allows to have more control and ROLLBACK transaction in progress if its lifetime exceeding a certain threshold and there are other worker nodes should be updated.	2023-01-24 16:14:58 +01:00
Alexander KukushkinandGitHub	3161f31088	Enhanced sync connections check (#2524 ) When `synchronous_standby_names` GUC is changed PostgreSQL nearly immediately starts reporting corresponding walsenders as synchronous, while in fact they maybe didn't reach this state yet. To mitigate this problem we memorize current flush lsn on the primary right after change of `synchronous_standby_names` got visible and use it as an additional check for walsenders. The walsender will be counted as truly "sync" only when write/flush/replay_lsn on it reached memorized LSN and the `application_name` is known to be a part of `synchronous_standby_names`. The size of PR mostly related to refactoring and moving the code responsible for working with `synchronous_standby_names` and `pg_stat_replication` to the dedicated file. And `parse_sync_standby_names()` function was mostly copied from #672.	2023-01-24 15:05:54 +01:00
Alexander KukushkinandGitHub	40d16443f9	Fixes and improvements in failsafe (#2532 ) 1. Fix problem with logical slots not advancing when only the primary lost access to DCS 2. Don't let Patroni to join as a raft voting member when running failsafe behave tests. It allows to test exactly the same conditions as for other DCS 3. Speed up dcs_failsafe_mode behave tests by getting rid from long sleeps, slight reshuffling of places when we start/stop outage, and by killing Patroni/Postgres to avoid long shutdown due to the leader key removal attempts.	2023-01-24 14:07:31 +01:00
Alexander KukushkinandGitHub	1e208736f8	Refactor drop_replication_slot() and _drop_incorrect_slots() (#2534 ) Use CTE to avoid running the second query if pg_drop_replication_slot() failed	2023-01-23 16:46:07 +01:00
William Albertus DemboandGitHub	f06d432dab	Keep only latest failed data directory (#2471 ) Use constant postfix when moving data directory due to failure so it only keeps data from the latest failure.	2023-01-19 21:47:41 +01:00
Polina Bungina GitHub Alexander Kukushkin	838653325a	Clean pg_replslot/ after pg_rewind (#2531 ) As pg_rewind cleans this directory on target only since pg11 Co-authored-by: Alexander Kukushkin <[email protected]>	2023-01-19 15:50:30 +01:00
Michael BanckandGitHub	06bbe2eadc	Suppress recurring errors when dropping unknown but active replication slots (#2502 ) When a replication slot is not registered with Patroni but is active, Patroni would log an error during each HA cycle in certain conditions (after a restart or role change). To avoid this, first check if the replication slot we are about to drop is still active and if so, only log a warning. Otherwise, log the slot we are dropping for informational purposes. Close: #2499	2023-01-19 09:53:17 +01:00
Alexander Kukushkin GitHub Polina Bungina	b75cd5a7d9	Submit coverage to codacy only if secret is available (#2528 ) If PR is open from the external GH repo secrets are not set due to security reasons. It makes codacy coverage report to fail. Co-authored-by: Polina Bungina <[email protected]>	2023-01-17 15:28:39 +01:00
Polina Bungina GitHub Alexander Kukushkin	acecbe0d8f	Fix a couple of linter problems, delete TODO.md (#2526 ) Fix a couple of linter problems, remove trailing whitespaces Co-authored-by: Alexander Kukushkin <[email protected]>	2023-01-17 10:52:03 +01:00
Alexander Kukushkin GitHub Polina Bungina	2ea0357854	DCS failsafe mode (#2379 ) If enabled it will allow Patroni to cope with DCS outages. In case of a DCS outage the leader tries to call all remaining members in the cluster via API and if all of them respond with success the leader will not be demoted. The failsafe_mode could be enabled by running ```sh patronictl edit-config -s failsafe_mode=true ``` or by calling the `/config` REST API endpoint. Co-authored-by: Polina Bungina <[email protected]>	2023-01-13 13:35:05 +01:00
Polina BunginaandGitHub	b13354b6a3	Make launch.sh pass shellcheck (#2522 )	2023-01-12 09:14:47 +01:00
Alexander KukushkinandGitHub	5bbb5dceeb	Improve /(a)sync checks in behave tests (#2521 ) They are frequently failing because sometimes replicas are a bit slow realizing that they are synchronous. Instead of instroducing more sleeps we will poll for required http status code with some timeout.	2023-01-12 08:23:59 +01:00
Polina BunginaandGitHub	650344fca8	Update Slack link in README.rst and CONTRIBUTING.rst (#2520 ) * Update Slack link in README.rst and CONTRIBUTING.rst	2023-01-11 16:06:25 +01:00
Polina BunginaandGitHub	9de22e667b	Report coverage to Codacy for behave tests (#2518 )	2023-01-11 11:47:08 +01:00
Alexander KukushkinandGitHub	c12fe4146d	Run only one query per HA loop (#2516 ) If the cluster is stable (no nodes are joining/leaving/lagging) we want to run at most one monitor query per every HA loop. So far it worker perfectly except when synchronous_mode is enabled, where we run two additional queries: 1. SHOW synchronous_mode 2. SELECT ... FROM pg_stat_replication In order to solve it, we will include these "queries" to the common monitoring query is synchronous_mode is enabled. In addition to that make sure that `synchronous_standby_names` is reset on replicas that used to be a primary and avoid using replicas which are not in the 'running' state. P.S.: in the monitoring query we also extract the current value of synchronous_standby_names, because it will be useful for the quorum commit feature. Close https://github.com/zalando/patroni/issues/2469	2023-01-10 10:44:17 +01:00
Alexander KukushkinandGitHub	baaf187c81	Fix behave tests on GH actions MacOS (#2515 ) - the new MacOS doesn't play well with old go binaries (bump etcd) - use brew to install Postgres and expect (unbuffer, to make behave output colorful) and use the latest version - upload failed logs instead of grepping them to stdout	2023-01-05 12:32:39 +01:00
Alexander KukushkinandGitHub	442bd3f434	Compatibility with some old modules (#2514 ) - old click differently handles argument names - old pytest doesn't like `from mock import call` Bump version and update release notes. Close: https://github.com/zalando/patroni/issues/2508 Close: https://github.com/zalando/patroni/issues/2512	2023-01-04 07:24:52 +01:00
Michael BanckandGitHub	e3e4ad0ada	Start etcd with V2 API enabled for V2 etcd acceptance tests (#2509 ) Otherwise, the etcd (not etcd3) behave tests fail to connect: ``` Jan 02 09:56:18 HOOK-ERROR in before_all: AssertionError: etcd instance is not available for queries after 5 seconds ```	2023-01-03 15:39:30 +01:00