diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 0540cca0..bc6a1be3 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -174,3 +174,27 @@ jobs: - uses: jakebailey/pyright-action@v1 with: version: 1.1.320 + + docs: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Set up Python 3.11 + uses: actions/setup-python@v4 + with: + python-version: 3.11 + cache: pip + + - name: Install dependencies + run: pip install tox + + - name: Install package dependencies + run: | + sudo apt update \ + && sudo apt install -y \ + latexmk texlive-latex-extra tex-gyre \ + --no-install-recommends + + - name: Generate documentation + run: tox -m docs diff --git a/.gitignore b/.gitignore index c3af6eb0..a5191907 100644 --- a/.gitignore +++ b/.gitignore @@ -51,6 +51,7 @@ scm-source.json docs/build/ docs/source/_static/ docs/source/_templates/ +docs/modules/ # Pycharm IDE .idea/ diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 724e2418..9c08ad79 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -19,3 +19,8 @@ formats: - epub - pdf - htmlzip + +python: + install: + - requirements: requirements.docs.txt + - requirements: requirements.txt diff --git a/docs/CONTRIBUTING.rst b/docs/CONTRIBUTING.rst index 44733a55..b5e0be0e 100644 --- a/docs/CONTRIBUTING.rst +++ b/docs/CONTRIBUTING.rst @@ -1,182 +1,12 @@ .. _contributing: -Contributing guidelines -======================= +Contributing +============ -Wanna contribute to Patroni? Yay - here is how! +Resources and information for developers can be found in the pages below. -Chatting --------- +.. toctree:: + :maxdepth: 2 -Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni `__ in the `PostgreSQL Slack `__. - -Running tests -------------- - -Requirements for running behave tests: - -1. PostgreSQL packages need to be installed. -2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`. -3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`. - -Install dependencies: - -.. code-block:: bash - - # You may want to use Virtualenv or specify pip3. - pip install -r requirements.txt - pip install -r requirements.dev.txt - -After you have all dependencies installed, you can run the various test suites: - -.. code-block:: bash - - # You may want to use Virtualenv or specify python3. - - # Run flake8 to check syntax and formatting: - python setup.py flake8 - - # Run the pytest suite in tests/: - python setup.py test - - # Run the behave (https://behave.readthedocs.io/en/latest/) test suite in features/; - # modify DCS as desired (raft has no dependencies so is the easiest to start with): - DCS=raft python -m behave - -Testing with tox ----------------- - -To run tox tests you only need to install one dependency (other than Python) - -.. code-block:: bash - - pip install tox>=4 - -If you wish to run `behave` tests then you also need docker installed. - -Tox configuration in `tox.ini` has "environments" to run the following tasks: - -* lint: Python code lint with `flake8` -* test: unit tests for all available python interpreters with `pytest`, - generates XML reports or HTML reports if a TTY is detected -* dep: detect package dependency conflicts using `pipdeptree` -* type: static type checking with `pyright` -* black: code formatting with `black` -* docker-build: build docker image used for the `behave` env -* docker-cmd: run arbitrary command with the above image -* docker-behave-etcd: run tox for behave tests with above image -* py*behave: run behave with available python interpreters (without docker, although - this is what is called inside docker containers) -* docs: build docs with `sphinx` - -Running tox -^^^^^^^^^^^ - -To run the default env list; dep, lint, test, and docs, just run: - -.. code-block:: bash - - tox - -The `test` envs can be run with the label `test`: - -.. code-block:: bash - - tox -m test - -The `behave` docker tests can be run with the label `behave`: - -.. code-block:: bash - - tox -m behave - -Similarly, docs has the label `docs`. - -All other envs can be run with their respective env names: - -.. code-block:: bash - - tox -e lint - tox -e py39-test-lin - -It is also possible to select partial env lists using `factors`. For example, if you want to run -all envs for python 3.10: - -.. code-block:: bash - - tox -f py310 - -This is equivalent to running all the envs listed below: - -.. code-block:: bash - - $ tox -l -f py310 - py310-test-lin - py310-test-mac - py310-test-win - py310-type-lin - py310-type-mac - py310-type-win - py310-behave-etcd-lin - py310-behave-etcd-win - py310-behave-etcd-mac - - -You can list all configured combinations of environments with tox (>=v4) like so - -.. code-block:: bash - - tox l - -The envs `test` and `docs` will attempt to open the HTML output files -when the job completes, if tox is run with an active terminal. This -is intended to be for benefit of the developer running this env locally. -It will attempt to run `open` on a mac and `xdg-open` on Linux. -To use a different command set the env var `OPEN_CMD` to the name or path of -the command. If this step fails it will not fail the run overall. -If you want to disable this facility set the env var `OPEN_CMD` to the `:` no-op command. - -.. code-block:: bash - - OPEN_CMD=: tox -m docs - -Behave tests -^^^^^^^^^^^^ - -Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 15 and then run all -behave tests. This can take quite a long time to run so you might want to limit the scope to a select version of -Postgres or to a specific feature set or steps. - -To specify the version of postgres include the full name of the dependent image build env that you want and then the -behave env name. For instance if you want Postgres 15 use: - -.. code-block:: bash - - tox -e pg14-docker-build,pg14-docker-behave-etcd-lin - -If on the other hand you want to test a specific feature you can pass positional arguments to behave. This will run -the watchdog behave feature test scenario with all versions of Postgres. - -.. code-block:: bash - - tox -m behave -- features/watchdog.feature - -Of course you can combine the two. - -Reporting issues ----------------- - -If you have a question about patroni or have a problem using it, please read the :ref:`README ` before filing an issue. -Also double check with the current issues on our `Issues Tracker `__. - -Contributing a pull request ---------------------------- - -1) Submit a comment to the relevant issue or create a new issue describing your proposed change. -2) Do a fork, develop and test your code changes. -3) Include documentation -4) Submit a pull request. - -You'll get feedback about your pull request as soon as possible. - -Happy Patroni hacking ;-) + contributing_guidelines + Patroni API docs diff --git a/docs/conf.py b/docs/conf.py index 6f136d00..2228c72e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -20,10 +20,15 @@ import os import sys + sys.path.insert(0, os.path.abspath('..')) from patroni.version import __version__ +project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) +module_dir = os.path.abspath(os.path.join(project_root, 'patroni')) +excludes = ['tests', 'setup.py', 'conf'] + # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. @@ -33,11 +38,21 @@ from patroni.version import __version__ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['sphinx.ext.intersphinx', +extensions = [ + 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.mathjax', 'sphinx.ext.ifconfig', - 'sphinx.ext.viewcode'] + # 'sphinx.ext.viewcode', + 'sphinx_github_style', # Generate "View on GitHub" for source code + 'sphinxcontrib.apidoc', # For generating module docs from code + 'sphinx.ext.autodoc', # For generating module docs from docstrings + 'sphinx.ext.napoleon', # For Google and Numpy formatted docstrings +] +apidoc_module_dir = module_dir +apidoc_output_dir = 'modules' +apidoc_excluded_paths = excludes +apidoc_separate_modules = True # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] @@ -107,6 +122,34 @@ if not on_rtd: # only import and set the theme if we're building docs locally # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] +# Replace "source" links with "edit on GitHub" when using rtd theme +html_context = { + 'display_github': True, + 'github_user': 'zalando', + 'github_repo': 'patroni', + 'github_version': 'master', + 'conf_py_path': '/docs/', +} + +# sphinx-github-style options, https://sphinx-github-style.readthedocs.io/en/latest/index.html + +# The name of the top-level package. +top_level = "patroni" + +# The blob to link to on GitHub - any of "head", "last_tag", or "{blob}" +# linkcode_blob = 'head' + +# The link to your GitHub repository formatted as https://github.com/user/repo +# If not provided, will attempt to create the link from the html_context dict +# linkcode_url = f"https://github.com/{html_context['github_user']}/" \ +# f"{html_context['github_repo']}/{html_context['github_version']}" + +# The text to use for the linkcode link +# linkcode_link_text: str = "View on GitHub" + +# A linkcode_resolve() function to use for resolving the link target +# linkcode_resolve: types.FunctionType + # -- Options for HTMLHelp output ------------------------------------------ @@ -165,7 +208,6 @@ texinfo_documents = [ ] - # -- Options for Epub output ---------------------------------------------- # Bibliographic Dublin Core info. @@ -187,10 +229,57 @@ epub_copyright = copyright epub_exclude_files = ['search.html'] - # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = {'python': ('https://docs.python.org/', None)} + +# Remove these pages from index, references, toc trees, etc. +# If the builder is not 'html' then add the API docs modules index to pages to be removed. +exclude_from_builder = { + 'latex': ['modules/modules'], + 'epub': ['modules/modules'], +} +# Internal holding list, anything added here will always be excluded +_docs_to_remove = [] + + +def builder_inited(app): + """Run during Sphinx `builder-inited` phase. + + Set a config value to builder name and add module docs to `docs_to_remove`. + """ + print(f'The builder is: {app.builder.name}') + app.add_config_value('builder', app.builder.name, 'env') + + # Remove pages when builder matches any referenced in exclude_from_builder + if exclude_from_builder.get(app.builder.name): + _docs_to_remove.extend(exclude_from_builder[app.builder.name]) + + +def env_get_outdated(app, env, added, changed, removed): + """Run during Sphinx `env-get-outdated` phase. + + Remove the items listed in `docs_to_remove` from known pages. + """ + added.difference_update(_docs_to_remove) + changed.difference_update(_docs_to_remove) + removed.update(_docs_to_remove) + return [] + + +def doctree_read(app, doctree): + """Run during Sphinx `doctree-read` phase. + + Remove the items listed in `docs_to_remove` from the table of contents. + """ + from sphinx import addnodes + for toc_tree_node in doctree.traverse(addnodes.toctree): + for e in toc_tree_node['entries']: + ref = str(e[1]) + if ref in _docs_to_remove: + toc_tree_node['entries'].remove(e) + + # A possibility to have an own stylesheet, to add new rules or override existing ones # For the latter case, the CSS specificity of the rules should be higher than the default ones def setup(app): @@ -198,3 +287,8 @@ def setup(app): app.add_css_file('custom.css') else: app.add_stylesheet('custom.css') + + # Run extra steps to remove module docs when running with a non-html builder + app.connect('builder-inited', builder_inited) + app.connect('env-get-outdated', env_get_outdated) + app.connect('doctree-read', doctree_read) diff --git a/docs/contributing_guidelines.rst b/docs/contributing_guidelines.rst new file mode 100644 index 00000000..48710868 --- /dev/null +++ b/docs/contributing_guidelines.rst @@ -0,0 +1,182 @@ +.. _contributing_guidelines: + +Contributing guidelines +======================= + +Wanna contribute to Patroni? Yay - here is how! + +Chatting +-------- + +Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni `__ in the `PostgreSQL Slack `__. + +Running tests +------------- + +Requirements for running behave tests: + +1. PostgreSQL packages need to be installed. +2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`. +3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`. + +Install dependencies: + +.. code-block:: bash + + # You may want to use Virtualenv or specify pip3. + pip install -r requirements.txt + pip install -r requirements.dev.txt + +After you have all dependencies installed, you can run the various test suites: + +.. code-block:: bash + + # You may want to use Virtualenv or specify python3. + + # Run flake8 to check syntax and formatting: + python setup.py flake8 + + # Run the pytest suite in tests/: + python setup.py test + + # Run the behave (https://behave.readthedocs.io/en/latest/) test suite in features/; + # modify DCS as desired (raft has no dependencies so is the easiest to start with): + DCS=raft python -m behave + +Testing with tox +---------------- + +To run tox tests you only need to install one dependency (other than Python) + +.. code-block:: bash + + pip install tox>=4 + +If you wish to run `behave` tests then you also need docker installed. + +Tox configuration in `tox.ini` has "environments" to run the following tasks: + +* lint: Python code lint with `flake8` +* test: unit tests for all available python interpreters with `pytest`, + generates XML reports or HTML reports if a TTY is detected +* dep: detect package dependency conflicts using `pipdeptree` +* type: static type checking with `pyright` +* black: code formatting with `black` +* docker-build: build docker image used for the `behave` env +* docker-cmd: run arbitrary command with the above image +* docker-behave-etcd: run tox for behave tests with above image +* py*behave: run behave with available python interpreters (without docker, although + this is what is called inside docker containers) +* docs: build docs with `sphinx` + +Running tox +^^^^^^^^^^^ + +To run the default env list; dep, lint, test, and docs, just run: + +.. code-block:: bash + + tox + +The `test` envs can be run with the label `test`: + +.. code-block:: bash + + tox -m test + +The `behave` docker tests can be run with the label `behave`: + +.. code-block:: bash + + tox -m behave + +Similarly, docs has the label `docs`. + +All other envs can be run with their respective env names: + +.. code-block:: bash + + tox -e lint + tox -e py39-test-lin + +It is also possible to select partial env lists using `factors`. For example, if you want to run +all envs for python 3.10: + +.. code-block:: bash + + tox -f py310 + +This is equivalent to running all the envs listed below: + +.. code-block:: bash + + $ tox -l -f py310 + py310-test-lin + py310-test-mac + py310-test-win + py310-type-lin + py310-type-mac + py310-type-win + py310-behave-etcd-lin + py310-behave-etcd-win + py310-behave-etcd-mac + + +You can list all configured combinations of environments with tox (>=v4) like so + +.. code-block:: bash + + tox l + +The envs `test` and `docs` will attempt to open the HTML output files +when the job completes, if tox is run with an active terminal. This +is intended to be for benefit of the developer running this env locally. +It will attempt to run `open` on a mac and `xdg-open` on Linux. +To use a different command set the env var `OPEN_CMD` to the name or path of +the command. If this step fails it will not fail the run overall. +If you want to disable this facility set the env var `OPEN_CMD` to the `:` no-op command. + +.. code-block:: bash + + OPEN_CMD=: tox -m docs + +Behave tests +^^^^^^^^^^^^ + +Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 15 and then run all +behave tests. This can take quite a long time to run so you might want to limit the scope to a select version of +Postgres or to a specific feature set or steps. + +To specify the version of postgres include the full name of the dependent image build env that you want and then the +behave env name. For instance if you want Postgres 15 use: + +.. code-block:: bash + + tox -e pg14-docker-build,pg14-docker-behave-etcd-lin + +If on the other hand you want to test a specific feature you can pass positional arguments to behave. This will run +the watchdog behave feature test scenario with all versions of Postgres. + +.. code-block:: bash + + tox -m behave -- features/watchdog.feature + +Of course you can combine the two. + +Reporting issues +---------------- + +If you have a question about patroni or have a problem using it, please read the :ref:`README ` before filing an issue. +Also double check with the current issues on our `Issues Tracker `__. + +Contributing a pull request +--------------------------- + +1) Submit a comment to the relevant issue or create a new issue describing your proposed change. +2) Do a fork, develop and test your code changes. +3) Include documentation +4) Submit a pull request. + +You'll get feedback about your pull request as soon as possible. + +Happy Patroni hacking ;-) diff --git a/docs/ha_multi_dc.rst b/docs/ha_multi_dc.rst index 75a43a5f..0a7f610e 100644 --- a/docs/ha_multi_dc.rst +++ b/docs/ha_multi_dc.rst @@ -12,7 +12,7 @@ In both cases, it is important to be clear about the following concepts: - You should run the odd number of etcd, ZooKeeper or Consul nodes: 3 or 5! Synchronous Replication ----------------------------- +----------------------- To have a multi DC cluster that can automatically tolerate a zone drop, a minimum of 3 is required. @@ -27,7 +27,7 @@ Regarding postgres, we must deploy at least 2 nodes, in different DC. Then you h This enables sync replication and the primary node will choose one of the nodes as synchronous. Asynchronous Replication ----------------------------------- +------------------------ With only two data centers it would be better to have two independent etcd clusters and run Patroni :ref:`standby cluster ` in the second data center. If the first site is down, you can MANUALLY promote the ``standby_cluster``. diff --git a/docs/index.rst b/docs/index.rst index e76ec6d5..c7428a95 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -40,6 +40,13 @@ Currently supported PostgreSQL versions: 9.3 to 15. Indices and tables ================== -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` +.. ifconfig:: builder == 'html' + + * :ref:`genindex` + * :ref:`modindex` + * :ref:`search` + +.. ifconfig:: builder != 'html' + + * :ref:`genindex` + * :ref:`search` diff --git a/docs/patroni_configuration.rst b/docs/patroni_configuration.rst index 559236a2..fafbbbdb 100644 --- a/docs/patroni_configuration.rst +++ b/docs/patroni_configuration.rst @@ -44,7 +44,6 @@ Some of the PostgreSQL parameters **must hold the same values on the primary and - **max_worker_processes**: 8 - **max_prepared_transactions**: 0 - **wal_level**: hot_standby -- **wal_log_hints**: on - **track_commit_timestamp**: off For the parameters below, PostgreSQL does not require equal values among the primary and all the replicas. However, considering the possibility of a replica to become the primary at any time, it doesn't really make sense to set them differently; therefore, **Patroni restricts setting their values to the** :ref:`dynamic configuration `. @@ -62,6 +61,7 @@ There are some other Postgres parameters controlled by Patroni: - **port** - is set either from ``postgresql.listen`` or from ``PATRONI_POSTGRESQL_LISTEN`` environment variable - **cluster_name** - is set either from ``scope`` or from ``PATRONI_SCOPE`` environment variable - **hot_standby: on** +- **wal_log_hints: on** - for Postgres 9.4 and newer. To be on the safe side parameters from the above lists are not written into ``postgresql.conf``, but passed as a list of arguments to the ``pg_ctl start`` which gives them the highest precedence, even above `ALTER SYSTEM `__ diff --git a/docs/rest_api.rst b/docs/rest_api.rst index 5310f73e..98c89013 100644 --- a/docs/rest_api.rst +++ b/docs/rest_api.rst @@ -282,7 +282,6 @@ Config endpoint "use_pg_rewind": true, "parameters": { "hot_standby": "on", - "wal_log_hints": "on", "wal_level": "hot_standby", "max_wal_senders": 5, "max_replication_slots": 5, @@ -309,7 +308,6 @@ Config endpoint "use_pg_rewind": true, "parameters": { "hot_standby": "on", - "wal_log_hints": "on", "wal_level": "hot_standby", "max_wal_senders": 5, "max_replication_slots": 5, @@ -362,7 +360,6 @@ If you want to remove (reset) some setting just patch it with ``null``: "hot_standby": "on", "unix_socket_directories": ".", "wal_level": "hot_standby", - "wal_log_hints": "on", "max_wal_senders": 5, "max_replication_slots": 5 } @@ -376,7 +373,7 @@ The above call removes ``postgresql.parameters.max_connections`` from the dynami .. code-block:: bash $ curl -s -XPUT -d \ - '{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_log_hints":"on","wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \ + '{"maximum_lag_on_failover":1048576,"retry_timeout":10,"postgresql":{"use_slots":true,"use_pg_rewind":true,"parameters":{"hot_standby":"on","wal_level":"hot_standby","unix_socket_directories":".","max_wal_senders":5}},"loop_wait":3,"ttl":20}' \ http://localhost:8008/config | jq . { "ttl": 20, @@ -388,7 +385,6 @@ The above call removes ``postgresql.parameters.max_connections`` from the dynami "hot_standby": "on", "unix_socket_directories": ".", "wal_level": "hot_standby", - "wal_log_hints": "on", "max_wal_senders": 5 }, "use_pg_rewind": true diff --git a/docs/yaml_configuration.rst b/docs/yaml_configuration.rst index 41776f87..f0cb8ef4 100644 --- a/docs/yaml_configuration.rst +++ b/docs/yaml_configuration.rst @@ -49,17 +49,24 @@ Bootstrap configuration - **- data-checksums**: Must be enabled when pg_rewind is needed on 9.3. - **- encoding: UTF8**: default encoding for new databases. - **- locale: UTF8**: default locale for new databases. - - **users**: Some additional users which need to be created after initializing new cluster - - - **admin**: the name of user - - - **password**: (optional) password for the user - - **options**: list of options for CREATE USER statement - - - **- createrole** - - **- createdb** + - **users**: Some additional users which need to be created after initializing new cluster, see :ref:`Bootstrap users configuration ` below. - **post\_bootstrap** or **post\_init**: An additional script that will be executed after initializing the cluster. The script receives a connection string URL (with the cluster superuser as a user name). The PGPASSFILE variable is set to the location of pgpass file. +.. _bootstrap_users_configuration: + +Bootstrap users configuration +============================= + +Users which need to be created after initializing the cluster: + +- **admin**: the name of user + + - **password**: (optional) password for the user + - **options**: list of options for CREATE USER statement + + - **- createrole** + - **- createdb** + .. _citus_settings: Citus diff --git a/patroni/api.py b/patroni/api.py index ce0c1d37..83addb93 100644 --- a/patroni/api.py +++ b/patroni/api.py @@ -49,9 +49,23 @@ def check_access(func: Callable[['RestApiHandler'], None]) -> Callable[..., None :Example: - @check_access - def do_PUT_foo(): - pass + >>> class FooServer: + ... def check_access(self, *args, **kwargs): + ... print(f'In FooServer: {args[0].__class__.__name__}') + ... return True + ... + + >>> class Foo: + ... server = FooServer() + ... @check_access + ... def do_PUT_foo(self): + ... print('In do_PUT_foo') + + >>> f = Foo() + >>> f.do_PUT_foo() + In FooServer: Foo + In do_PUT_foo + """ def wrapper(self: 'RestApiHandler', *args: Any, **kwargs: Any) -> None: @@ -97,6 +111,7 @@ class RestApiHandler(BaseHTTPRequestHandler): """Write a response that is composed only of the HTTP status. The response is written with these values separated by space: + * HTTP protocol version; * *status_code*; * description of *status_code*. @@ -157,19 +172,19 @@ class RestApiHandler(BaseHTTPRequestHandler): Modifies *response* before sending it to the client. Defines the ``patroni`` key, which is a dictionary that contains the mandatory keys: - * ``version``: Patroni version, e.g. ``3.0.2``; - * ``scope``: value of ``scope`` setting from Patroni configuration. + * ``version``: Patroni version, e.g. ``3.0.2``; + * ``scope``: value of ``scope`` setting from Patroni configuration. May also add the following optional keys, depending on the status of this Patroni/PostgreSQL node: - * ``tags``: tags that were set through Patroni configuration merged with dynamically applied tags; - * ``database_system_identifier``: ``Database system identifier`` from ``pg_controldata`` output; - * ``pending_restart``: ``True`` if PostgreSQL is pending to be restarted; - * ``scheduled_restart``: a dictionary with a single key ``schedule``, which is the timestamp for the scheduled - restart; - * ``watchdog_failed``: ``True`` if watchdog device is unhealthy; - * ``logger_queue_size``: log queue length if it is longer than expected; - * ``logger_records_lost``: number of log records that have been lost while the log queue was full. + * ``tags``: tags that were set through Patroni configuration merged with dynamically applied tags; + * ``database_system_identifier``: ``Database system identifier`` from ``pg_controldata`` output; + * ``pending_restart``: ``True`` if PostgreSQL is pending to be restarted; + * ``scheduled_restart``: a dictionary with a single key ``schedule``, which is the timestamp for the + scheduled restart; + * ``watchdog_failed``: ``True`` if watchdog device is unhealthy; + * ``logger_queue_size``: log queue length if it is longer than expected; + * ``logger_records_lost``: number of log records that have been lost while the log queue was full. :param status_code: response HTTP status code. :param response: represents the status of the PostgreSQL node, and is used as a basis for the HTTP response. @@ -204,36 +219,62 @@ class RestApiHandler(BaseHTTPRequestHandler): Is used for handling all health-checks requests. E.g. "GET /(primary|replica|sync|async|etc...)". The (optional) query parameters and the HTTP response status depend on the requested path: + * ``/``, ``primary``, or ``read-write``: + * HTTP status ``200``: if a primary with the leader lock. + * ``/standby-leader``: + * HTTP status ``200``: if holds the leader lock in a standby cluster. + * ``/leader``: + * HTTP status ``200``: if holds the leader lock. + * ``/replica``: + * Query parameters: + * ``lag``: only accept replication lag up to ``lag``. Accepts either an :class:`int`, which represents lag in bytes, or a :class:`str` representing lag in human-readable format (e.g. ``10MB``). * Any custom parameter: will attempt to match them against node tags. + * HTTP status ``200``: if up and running as a standby and without ``noloadbalance`` tag. + * ``/read-only``: + * HTTP status ``200``: if up and running and without ``noloadbalance`` tag. + * ``/quorum``: + * HTTP status ``200``: if up and running as a quorum synchronous standby. + * ``/read-only-quorum``: + * HTTP status ``200``: if up and running as a quorum synchronous standby or primary. + * ``/synchronous`` or ``/sync``: + * HTTP status ``200``: if up and running as a synchronous standby. + * ``/read-only-sync``: + * HTTP status ``200``: if up and running as a synchronous standby or primary. + * ``/asynchronous``: + * Query parameters: + * ``lag``: only accept replication lag up to ``lag``. Accepts either an :class:`int`, which represents lag in bytes, or a :class:`str` representing lag in human-readable format (e.g. ``10MB``). + * HTTP status ``200``: if up and running as an asynchronous standby. + * ``/health``: + * HTTP status ``200``: if up and running. .. note:: @@ -345,16 +386,16 @@ class RestApiHandler(BaseHTTPRequestHandler): def do_OPTIONS(self) -> None: """Handle an ``OPTIONS`` request. - Write a simple HTTP response that represents the current PostgreSQL status. Send only `200 OK` or - `503 Service Unavailable` as a response and nothing more, particularly no headers. + Write a simple HTTP response that represents the current PostgreSQL status. Send only ``200 OK`` or + ``503 Service Unavailable`` as a response and nothing more, particularly no headers. """ self.do_GET(write_status_code_only=True) def do_HEAD(self) -> None: """Handle a ``HEAD`` request. - Write a simple HTTP response that represents the current PostgreSQL status. Send only `200 OK` or - `503 Service Unavailable` as a response and nothing more, particularly no headers. + Write a simple HTTP response that represents the current PostgreSQL status. Send only ``200 OK`` or + ``503 Service Unavailable`` as a response and nothing more, particularly no headers. """ self.do_GET(write_status_code_only=True) @@ -362,11 +403,17 @@ class RestApiHandler(BaseHTTPRequestHandler): """Handle a ``GET`` request to ``/liveness`` path. Write a simple HTTP response with HTTP status: + * ``200``: + * If the cluster is in maintenance mode; or * If Patroni heartbeat loop is properly running; - * ``503`` if Patroni heartbeat loop last run was more than ``ttl`` setting ago on the primary (or twice the - value of ``ttl`` on a replica). + + * ``503``: + + * if Patroni heartbeat loop last run was more than ``ttl`` setting ago on the primary (or twice the + value of ``ttl`` on a replica). + """ patroni: Patroni = self.server.patroni is_primary = patroni.postgresql.role in ('master', 'primary') and patroni.postgresql.is_running() @@ -383,10 +430,14 @@ class RestApiHandler(BaseHTTPRequestHandler): """Handle a ``GET`` request to ``/readiness`` path. Write a simple HTTP response which HTTP status can be: + * ``200``: + * If this Patroni node holds the DCS leader lock; or * If this PostgreSQL instance is up and running; + * ``503``: if none of the previous conditions apply. + """ patroni = self.server.patroni if patroni.ha.is_leader(): @@ -409,8 +460,8 @@ class RestApiHandler(BaseHTTPRequestHandler): def do_GET_cluster(self) -> None: """Handle a ``GET`` request to ``/cluster`` path. - Write an HTTP response with JSON content based on the output of :func:`cluster_as_json`, with HTTP status - ``200`` and the JSON representation of the cluster topology. + Write an HTTP response with JSON content based on the output of :func:`~patroni.utils.cluster_as_json`, with + HTTP status ``200`` and the JSON representation of the cluster topology. """ cluster = self.server.patroni.dcs.get_cluster(True) global_config = self.server.patroni.config.get_global_config(cluster) @@ -424,11 +475,13 @@ class RestApiHandler(BaseHTTPRequestHandler): The response contains a :class:`list` of failover/switchover events. Each item is a :class:`list` with the following items: + * Timeline when the event occurred (class:`int`); * LSN at which the event occurred (class:`int`); * The reason for the event (class:`str`); * Timestamp when the new timeline was created (class:`str`); * Name of the involved Patroni node (class:`str`). + """ cluster = self.server.patroni.dcs.cluster or self.server.patroni.dcs.get_cluster() self._write_json_response(200, cluster.history and cluster.history.lines or []) @@ -455,33 +508,34 @@ class RestApiHandler(BaseHTTPRequestHandler): The response contains the following items: - * ``patroni_version``: Patroni version without periods, e.g. ``030002`` for Patroni ``3.0.2``; - * ``patroni_postgres_running``: ``1`` if PostgreSQL is running, else ``0``; - * ``patroni_postmaster_start_time``: epoch timestamp since Postmaster was started; - * ``patroni_master``: ``1`` if this node holds the leader lock, else ``0``; - * ``patroni_primary``: same as ``patroni_master``; - * ``patroni_xlog_location``: ``pg_wal_lsn_diff(pg_current_wal_flush_lsn(), '0/0')`` if leader, else ``0``; - * ``patroni_standby_leader``: ``1`` if standby leader node, else ``0``; - * ``patroni_replica``: ``1`` if a replica, else ``0``; - * ``patroni_sync_standby``: ``1`` if a sync replica, else ``0``; - * ``patroni_quorum_standby``: ``1`` if a quorum sync replica, else ``0``; - * ``patroni_xlog_received_location``: ``pg_wal_lsn_diff(pg_last_wal_receive_lsn(), '0/0')``; - * ``patroni_xlog_replayed_location``: ``pg_wal_lsn_diff(pg_last_wal_replay_lsn(), '0/0)``; - * ``patroni_xlog_replayed_timestamp``: ``pg_last_xact_replay_timestamp``; - * ``patroni_xlog_paused``: ``pg_is_wal_replay_paused()``; - * ``patroni_postgres_server_version``: Postgres version without periods, e.g. ``150002`` for Postgres ``15.2``; - * ``patroni_cluster_unlocked``: ``1`` if no one holds the leader lock, else ``0``; - * ``patroni_failsafe_mode_is_active``: ``1`` if ``failsafe_mode`` is currently active, else ``0``; - * ``patroni_postgres_timeline``: PostgreSQL timeline based on current WAL file name; - * ``patroni_dcs_last_seen``: epoch timestamp when DCS was last contacted successfully; - * ``patroni_pending_restart``: ``1`` if this PostgreSQL node is pending a restart, else ``0``; - * ``patroni_is_paused``: ``1`` if Patroni is in maintenance node, else ``0``. + * ``patroni_version``: Patroni version without periods, e.g. ``030002`` for Patroni ``3.0.2``; + * ``patroni_postgres_running``: ``1`` if PostgreSQL is running, else ``0``; + * ``patroni_postmaster_start_time``: epoch timestamp since Postmaster was started; + * ``patroni_master``: ``1`` if this node holds the leader lock, else ``0``; + * ``patroni_primary``: same as ``patroni_master``; + * ``patroni_xlog_location``: ``pg_wal_lsn_diff(pg_current_wal_flush_lsn(), '0/0')`` if leader, else ``0``; + * ``patroni_standby_leader``: ``1`` if standby leader node, else ``0``; + * ``patroni_replica``: ``1`` if a replica, else ``0``; + * ``patroni_sync_standby``: ``1`` if a sync replica, else ``0``; + * ``patroni_quorum_standby``: ``1`` if a quorum sync replica, else ``0``; + * ``patroni_xlog_received_location``: ``pg_wal_lsn_diff(pg_last_wal_receive_lsn(), '0/0')``; + * ``patroni_xlog_replayed_location``: ``pg_wal_lsn_diff(pg_last_wal_replay_lsn(), '0/0)``; + * ``patroni_xlog_replayed_timestamp``: ``pg_last_xact_replay_timestamp``; + * ``patroni_xlog_paused``: ``pg_is_wal_replay_paused()``; + * ``patroni_postgres_server_version``: Postgres version without periods, e.g. ``150002`` for Postgres + ``15.2``; + * ``patroni_cluster_unlocked``: ``1`` if no one holds the leader lock, else ``0``; + * ``patroni_failsafe_mode_is_active``: ``1`` if ``failsafe_mode`` is currently active, else ``0``; + * ``patroni_postgres_timeline``: PostgreSQL timeline based on current WAL file name; + * ``patroni_dcs_last_seen``: epoch timestamp when DCS was last contacted successfully; + * ``patroni_pending_restart``: ``1`` if this PostgreSQL node is pending a restart, else ``0``; + * ``patroni_is_paused``: ``1`` if Patroni is in maintenance node, else ``0``. For PostgreSQL v9.6+ the response will also have the following: - * ``patroni_postgres_streaming``: 1 if Postgres is streaming from another node, else ``0``; - * ``patroni_postgres_in_archive_recovery``: ``1`` if Postgres isn't streaming and - there is ``restore_command`` available, else ``0``. + * ``patroni_postgres_streaming``: 1 if Postgres is streaming from another node, else ``0``; + * ``patroni_postgres_in_archive_recovery``: ``1`` if Postgres isn't streaming and + there is ``restore_command`` available, else ``0``. """ postgres = self.get_postgresql_status(True) patroni = self.server.patroni @@ -684,7 +738,7 @@ class RestApiHandler(BaseHTTPRequestHandler): def do_POST_reload(self) -> None: """Handle a ``POST`` request to ``/reload`` path. - Schedules a reload to Patroni and writes a response with HTTP status `202`. + Schedules a reload to Patroni and writes a response with HTTP status ``202``. """ self.server.patroni.sighup_handler() self.write_response(202, 'reload scheduled') @@ -745,13 +799,17 @@ class RestApiHandler(BaseHTTPRequestHandler): :param schedule: a string representing a timestamp, e.g. ``2023-04-14T20:27:00+00:00``. :param action: the action to be scheduled (``restart``, ``switchover``, or ``failover``). - :returns: a tuple composed of 3 items + :returns: a tuple composed of 3 items: + * Suggested HTTP status code for a response: + * ``None``: if no issue was faced while parsing, leaving it up to the caller to decide the status; or * ``400``: if no timezone information could be found in *schedule*; or * ``422``: if *schedule* is invalid -- in the past or not parsable. + * An error message, if any error is faced, otherwise ``None``; * Parsed *schedule*, if able to parse, otherwise ``None``. + """ error = None scheduled_at = None @@ -778,25 +836,31 @@ class RestApiHandler(BaseHTTPRequestHandler): Used to restart postgres (or schedule a restart), mainly by ``patronictl restart``. The request body should be a JSON dictionary, and it can contain the following keys: + * ``schedule``: timestamp at which the restart should occur; * ``role``: restart only nodes which role is ``role``. Can be either: + * ``primary`` (or ``master``); or * ``replica``. + * ``postgres_version``: restart only nodes which PostgreSQL version is less than ``postgres_version``, e.g. ``15.2``; * ``timeout``: if restart takes longer than ``timeout`` return an error and fail over to a replica; * ``restart_pending``: if we should restart only when have ``pending restart`` flag; Response HTTP status codes: + * ``200``: if successfully performed an immediate restart; or * ``202``: if successfully scheduled a restart for later; or * ``500``: if the cluster is in maintenance mode; or * ``400``: if + * ``role`` value is invalid; or * ``postgres_version`` value is invalid; or * ``timeout`` is not a number, or lesser than ``0``; or * request contains an unknown key; or * exception is faced while performing an immediate restart. + * ``409``: if another restart was already previously scheduled; or * ``503``: if any issue was found while performing an immediate restart; or * HTTP status returned by :func:`parse_schedule`, if any error was observed while parsing the schedule. @@ -874,6 +938,7 @@ class RestApiHandler(BaseHTTPRequestHandler): Used to remove a scheduled restart of PostgreSQL. Response HTTP status codes: + * ``200``: if a scheduled restart was removed; or * ``404``: if no scheduled restart could be found. """ @@ -892,6 +957,7 @@ class RestApiHandler(BaseHTTPRequestHandler): Used to remove a scheduled switchover in the cluster. It writes a response, and the HTTP status code can be: + * ``200``: if a scheduled switchover was removed; or * ``404``: if no scheduled switchover could be found; or * ``409``: if not able to update the switchover info in the DCS. @@ -913,11 +979,13 @@ class RestApiHandler(BaseHTTPRequestHandler): """Handle a ``POST`` request to ``/reinitialize`` path. The request body may contain a JSON dictionary with the following key: + * ``force``: ``True`` if we want to cancel an already running task in order to reinit a replica. Response HTTP status codes: + * ``200``: if the reinit operation has started; or - * ``503``: if any error is returned by :func:`Ha.reinitialize`. + * ``503``: if any error is returned by :func:`~patroni.ha.Ha.reinitialize`. """ request = self._read_json_content(body_is_optional=True) @@ -941,11 +1009,15 @@ class RestApiHandler(BaseHTTPRequestHandler): :param candidate: name of the Patroni node to be promoted. :param action: the action that is ongoing (``switchover`` or ``failover``). - :returns: a tuple composed of 2 items + :returns: a tuple composed of 2 items: + * Response HTTP status codes: + * ``200``: if the operation succeeded; or * ``503``: if the operation failed or timed out. + * A status message about the operation. + """ timeout = max(10, self.server.patroni.dcs.loop_wait) for _ in range(0, timeout * 2): @@ -1005,12 +1077,14 @@ class RestApiHandler(BaseHTTPRequestHandler): Handles manual failovers/switchovers, mainly from ``patronictl``. The request body should be a JSON dictionary, and it can contain the following keys: + * ``leader``: name of the current leader in the cluster; * ``candidate``: name of the Patroni node to be promoted; * ``scheduled_at``: a string representing the timestamp when to execute the switchover/failover, e.g. ``2023-04-14T20:27:00+00:00``. Response HTTP status codes: + * ``202``: if operation has been scheduled; * ``412``: if operation is not possible; * ``503``: if unable to register the operation to the DCS; @@ -1087,8 +1161,8 @@ class RestApiHandler(BaseHTTPRequestHandler): def do_POST_citus(self) -> None: """Handle a ``POST`` request to ``/citus`` path. - Call :func:`CitusHandler.handle_event` to handle the request, then write a response with HTTP status code - ``200``. + Call :func:`~patroni.postgresql.CitusHandler.handle_event` to handle the request, then write a response with + HTTP status code ``200``. .. note:: If unable to parse the request body, then the request is silently discarded. @@ -1104,18 +1178,21 @@ class RestApiHandler(BaseHTTPRequestHandler): self.write_response(200, 'OK') def parse_request(self) -> bool: - """Override :func:`parse_request` method to enrich basic functionality of :class:`BaseHTTPRequestHandler`. + """Override :func:`parse_request` to enrich basic functionality of :class:`~http.server.BaseHTTPRequestHandler`. Original class can only invoke :func:`do_GET`, :func:`do_POST`, :func:`do_PUT`, etc method implementations if they are defined. + But we would like to have at least some simple routing mechanism, i.e.: + * ``GET /uri1/part2`` request should invoke :func:`do_GET_uri1()` * ``POST /other`` should invoke :func:`do_POST_other()` If the :func:`do__` method does not exist we'll fall back to original behavior. :returns: ``True`` for success, ``False`` for failure; on failure, any relevant error response has already been - sent back. + sent back. + """ ret = BaseHTTPRequestHandler.parse_request(self) if ret: @@ -1149,36 +1226,46 @@ class RestApiHandler(BaseHTTPRequestHandler): Some of the values are collected by executing a query and other are taken from the state stored in memory. :param retry: whether the query should be retried if failed or give up immediately + :returns: a dict with the status of Postgres/Patroni. The keys are: + * ``state``: Postgres state among ``stopping``, ``stopped``, ``stop failed``, ``crashed``, ``running``, - ``starting``, ``start failed``, ``restarting``, ``restart failed``, ``initializing new cluster``, - ``initdb failed``, ``running custom bootstrap script``, ``custom bootstrap failed``, - ``creating replica``, or ``unknown``; + ``starting``, ``start failed``, ``restarting``, ``restart failed``, ``initializing new cluster``, + ``initdb failed``, ``running custom bootstrap script``, ``custom bootstrap failed``, + ``creating replica``, or ``unknown``; * ``postmaster_start_time``: ``pg_postmaster_start_time()``; * ``role``: ``replica`` or ``master`` based on ``pg_is_in_recovery()`` output; * ``server_version``: Postgres version without periods, e.g. ``150002`` for Postgres ``15.2``; * ``xlog``: dictionary. Its structure depends on ``role``: + * If ``master``: + * ``location``: ``pg_current_wal_flush_lsn()`` + * If ``replica``: + * ``received_location``: ``pg_wal_lsn_diff(pg_last_wal_receive_lsn(), '0/0')``; * ``replayed_location``: ``pg_wal_lsn_diff(pg_last_wal_replay_lsn(), '0/0)``; * ``replayed_timestamp``: ``pg_last_xact_replay_timestamp``; * ``paused``: ``pg_is_wal_replay_paused()``; + * ``sync_standby``: ``True`` if replication mode is synchronous and this is a sync standby; * ``timeline``: PostgreSQL primary node timeline; * ``replication``: :class:`list` of :class:`dict` entries, one for each replication connection. Each entry contains the following keys: + * ``application_name``: ``pg_stat_activity.application_name``; * ``client_addr``: ``pg_stat_activity.client_addr``; * ``state``: ``pg_stat_replication.state``; * ``sync_priority``: ``pg_stat_replication.sync_priority``; * ``sync_state``: ``pg_stat_replication.sync_state``; * ``usename``: ``pg_stat_activity.usename``. + * ``pause``: ``True`` if cluster is in maintenance mode; * ``cluster_unlocked``: ``True`` if cluster has no node holding the leader lock; * ``failsafe_mode_is_active``: ``True`` if DCS failsafe mode is currently active; * ``dcs_last_seen``: epoch timestamp DCS was last reached by Patroni. + """ postgresql = self.server.patroni.postgresql cluster = self.server.patroni.dcs.cluster @@ -1309,8 +1396,10 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): :param params: positional arguments to be used as parameters for *sql*. :returns: a list of rows that were fetched from the database. - :raises psycopg.Error: if had issues while executing *sql*. - :raises PostgresConnectionException: if had issues while connecting to the database. + + :raises: + :class:`psycopg.Error`: if had issues while executing *sql*. + :class:`~patroni.exceptions.PostgresConnectionException`: if had issues while connecting to the database. """ cursor = None try: @@ -1370,7 +1459,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): :param host: hostname to be checked. :param port: port to be checked. - :rtype: Iterator[Union[IPv4Network, IPv6Network]] of *host* + *port* resolved to IP networks. + :yields: *host* + *port* resolved to IP networks. """ try: for _, _, _, _, sa in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM, socket.IPPROTO_TCP): @@ -1384,8 +1473,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): .. note:: Only yields object if ``restapi.allowlist_include_members`` setting is enabled. - :rtype: Iterator[Union[IPv4Network, IPv6Network]] of each node ``restapi.connect_address`` resolved to an IP - network. + :yields: each node ``restapi.connect_address`` resolved to an IP network. """ cluster = self.patroni.dcs.cluster if self.__allowlist_include_members and cluster: @@ -1405,8 +1493,10 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): """Ensure client has enough privileges to perform a given request. Write a response back to the client if any issue is observed, and the HTTP status may be: + * ``401``: if ``Authorization`` header is missing or contain an invalid password; * ``403``: if: + * ``restapi.allowlist`` was configured, but client IP is not in the allowed list; or * ``restapi.allowlist_include_members`` is enabled, but client IP is not in the members list; or * a client certificate is expected by the server, but is missing in the request. @@ -1486,17 +1576,20 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): ``host`` can be a hostname or IP address. It is the value of ``restapi.listen`` setting. :param ssl_options: dictionary that may contain the following keys, depending on what has been configured in ``restapi` section: + * ``certfile``: path to PEM certificate. If given, will start in HTTPS mode; * ``keyfile``: path to key of ``certfile``; * ``keyfile_password``: password for decrypting ``keyfile``; * ``cafile``: path to CA file to validate client certificates; * ``ciphers``: permitted cipher suites; * ``verify_client``: value can be one among: + * ``none``: do not check client certificates; * ``optional``: check client certificate only for unsafe REST API endpoints; * ``required``: check client certificate for all REST API endpoints. - :raises ValueError: if any issue is faced while parsing *listen*. + :raises: + :class:`ValueError`: if any issue is faced while parsing *listen*. """ try: host, port = split_host_port(listen, None) @@ -1544,7 +1637,8 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): client_address: Tuple[str, int]) -> None: """Process a request to the REST API. - Wrapper for :func:`ThreadingMixIn.process_request_thread` that additionally: + Wrapper for :func:`~socketserver.ThreadingMixIn.process_request_thread` that additionally: + * Enable TCP keepalive * Perform SSL handshake (if an SSL socket). @@ -1562,7 +1656,8 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): def shutdown_request(self, request: Union[socket.socket, Tuple[bytes, socket.socket]]) -> None: """Shut down a request to the REST API. - Wrapper for :func:`HTTPServer.shutdown_request` that additionally: + Wrapper for :func:`http.server.HTTPServer.shutdown_request` that additionally: + * Perform SSL shutdown handshake (if a SSL socket). :param request: socket to handle the client request. @@ -1610,7 +1705,7 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): :param value: list of IPs and/or networks contained in ``restapi.allowlist`` setting. Each item can be a host, an IP, or a network in CIDR format. - :rtype: Iterator[Union[IPv4Network, IPv6Network]] of *host* + *port* resolved to IP networks. + :yields: *host* + *port* resolved to IP networks. """ if isinstance(value, list): for v in value: @@ -1627,7 +1722,9 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread): """Reload REST API configuration. :param config: dictionary representing values under the ``restapi`` configuration section. - :raises ValueError: if ``listen`` key is not present in *config*. + + :raises: + :class:`ValueError`: if ``listen`` key is not present in *config*. """ if 'listen' not in config: # changing config in runtime raise ValueError('Can not find "restapi.listen" config') diff --git a/patroni/config.py b/patroni/config.py index 6edfb436..3e0ad39a 100644 --- a/patroni/config.py +++ b/patroni/config.py @@ -1,3 +1,4 @@ +"""Facilities related to Patroni configuration.""" import json import logging import os @@ -35,47 +36,61 @@ _AUTH_ALLOWED_PARAMETERS = ( def default_validator(conf: Dict[str, Any]) -> List[str]: + """Ensure *conf* is not empty. + + Designed to be used as default validator for :class:`Config` objects, if no specific validator is provided. + + :param conf: configuration to be validated. + + :returns: an empty list -- :class:`Config` expects the validator to return a list of 0 or more issues found while + validating the configuration. + + :raises: + :class:`ConfigParseError`: if *conf* is empty. + """ if not conf: raise ConfigParseError("Config is empty.") return [] class GlobalConfig(object): + """A class that wraps global configuration and provides convenient methods to access/check values. - """A class that wrapps global configuration and provides convinient methods to access/check values. - - It is instantiated by calling :func:`Config.global_config` method which picks either a - configuration from provided :class:`Cluster` object (the most up-to-date) or from the - local cache if :class::`ClusterConfig` is not initialized or doesn't have a valid config. + It is instantiated either by calling :func:`get_global_config` or :meth:`Config.get_global_config`, which picks + either a configuration from provided :class:`Cluster` object (the most up-to-date) or from the + local cache if :class:`ClusterConfig` is not initialized or doesn't have a valid config. """ def __init__(self, config: Dict[str, Any]) -> None: - """Initialize :class:`GlobalConfig` object. + """Initialize :class:`GlobalConfig` object with given *config*. :param config: current configuration either from - :class:`ClusterConfig` or from :class:`Config.dynamic_configuration` + :class:`ClusterConfig` or from :func:`Config.dynamic_configuration`. """ self.__config = config def get(self, name: str) -> Any: - """Gets global configuration value by name. + """Gets global configuration value by *name*. - :param name: parameter name - :returns: configuration value or `None` if it is missing + :param name: parameter name. + + :returns: configuration value or ``None`` if it is missing. """ return self.__config.get(name) def check_mode(self, mode: str) -> bool: """Checks whether the certain parameter is enabled. - :param mode: parameter name could be: synchronous_mode, failsafe_mode, pause, check_timeline, and so on - :returns: `True` if *mode* is enabled in the global configuration. + :param mode: parameter name, e.g. ``synchronous_mode``, ``failsafe_mode``, ``pause``, ``check_timeline``, and + so on. + + :returns: ``True`` if parameter *mode* is enabled in the global configuration. """ return bool(parse_bool(self.__config.get(mode))) @property def is_paused(self) -> bool: - """:returns: `True` if cluster is in maintenance mode.""" + """``True`` if cluster is in maintenance mode.""" return self.check_mode('pause') @property @@ -85,76 +100,103 @@ class GlobalConfig(object): @property def is_synchronous_mode(self) -> bool: - """:returns: `True` if synchronous replication is requested.""" + """``True`` if synchronous replication is requested.""" return self.check_mode('synchronous_mode') is True or self.is_quorum_commit_mode @property def is_synchronous_mode_strict(self) -> bool: - """:returns: `True` if at least one synchronous node is required.""" + """``True`` if at least one synchronous node is required.""" return self.check_mode('synchronous_mode_strict') def get_standby_cluster_config(self) -> Union[Dict[str, Any], Any]: - """:returns: "standby_cluster" configuration.""" + """Get ``standby_cluster`` configuration. + + :returns: a copy of ``standby_cluster`` configuration. + """ return deepcopy(self.get('standby_cluster')) @property def is_standby_cluster(self) -> bool: - """:returns: `True` if global configuration has a valid "standby_cluster" section.""" + """``True`` if global configuration has a valid ``standby_cluster`` section.""" config = self.get_standby_cluster_config() return isinstance(config, dict) and\ bool(config.get('host') or config.get('port') or config.get('restore_command')) def get_int(self, name: str, default: int = 0) -> int: - """Gets current value from the global configuration and trying to return it as int. + """Gets current value of *name* from the global configuration and try to return it as :class:`int`. - :param name: name of the parameter - :param default: default value if *name* is not in the configuration or invalid - :returns: currently configured value from the global configuration or *default* if it is not set or invalid. + :param name: name of the parameter. + :param default: default value if *name* is not in the configuration or invalid. + + :returns: currently configured value of *name* from the global configuration or *default* if it is not set or + invalid. """ ret = parse_int(self.get(name)) return default if ret is None else ret @property def min_synchronous_nodes(self) -> int: - """:returns: the minimal number of synchronous nodes based on whether strict mode is requested or not.""" + """The minimal number of synchronous nodes based on whether ``synchronous_mode_strict`` is enabled or not.""" return 1 if self.is_synchronous_mode_strict else 0 @property def synchronous_node_count(self) -> int: - """:returns: currently configured value from the global configuration or 1 if it is not set or invalid.""" + """Currently configured value of ``synchronous_node_count`` from the global configuration. + + Assume ``1`` if it is not set or invalid. + """ return max(self.get_int('synchronous_node_count', 1), self.min_synchronous_nodes) @property def maximum_lag_on_failover(self) -> int: - """:returns: currently configured value from the global configuration or 1048576 if it is not set or invalid.""" + """Currently configured value of ``maximum_lag_on_failover`` from the global configuration. + + Assume ``1048576`` if it is not set or invalid. + """ return self.get_int('maximum_lag_on_failover', 1048576) @property def maximum_lag_on_syncnode(self) -> int: - """:returns: currently configured value from the global configuration or -1 if it is not set or invalid.""" + """Currently configured value of ``maximum_lag_on_syncnode`` from the global configuration. + + Assume ``-1`` if it is not set or invalid. + """ return self.get_int('maximum_lag_on_syncnode', -1) @property def primary_start_timeout(self) -> int: - """:returns: currently configured value from the global configuration or 300 if it is not set or invalid.""" + """Currently configured value of ``primary_start_timeout`` from the global configuration. + + Assume ``300`` if it is not set or invalid. + + .. note:: + ``master_start_timeout`` is still supported to keep backward compatibility. + """ default = 300 return self.get_int('primary_start_timeout', default)\ if 'primary_start_timeout' in self.__config else self.get_int('master_start_timeout', default) @property def primary_stop_timeout(self) -> int: - """:returns: currently configured value from the global configuration or 300 if it is not set or invalid.""" + """Currently configured value of ``primary_stop_timeout`` from the global configuration. + + Assume ``0`` if it is not set or invalid. + + .. note:: + ``master_stop_timeout`` is still supported to keep backward compatibility. + """ default = 0 return self.get_int('primary_stop_timeout', default)\ if 'primary_stop_timeout' in self.__config else self.get_int('master_stop_timeout', default) -def get_global_config(cluster: Union[Cluster, None], default: Optional[Dict[str, Any]] = None) -> GlobalConfig: +def get_global_config(cluster: Optional[Cluster], default: Optional[Dict[str, Any]] = None) -> GlobalConfig: """Instantiates :class:`GlobalConfig` based on the input. - :param cluster: the currently known cluster state from DCS - :param default: default configuration, which will be used if there is no valid *cluster.config* - :returns: :class:`GlobalConfig` object + :param cluster: the currently known cluster state from DCS. + :param default: default configuration, which will be used if there is no valid *cluster.config*. + + :returns: :class:`GlobalConfig` object. """ # Try to protect from the case when DCS was wiped out if cluster and cluster.config and cluster.config.modify_version: @@ -165,23 +207,29 @@ def get_global_config(cluster: Union[Cluster, None], default: Optional[Dict[str, class Config(object): - """ + """Handle Patroni configuration. + This class is responsible for: - 1) Building and giving access to `effective_configuration` from: - * `Config.__DEFAULT_CONFIG` -- some sane default values - * `dynamic_configuration` -- configuration stored in DCS - * `local_configuration` -- configuration from `config.yml` or environment + 1) Building and giving access to ``effective_configuration`` from: - 2) Saving and loading `dynamic_configuration` into 'patroni.dynamic.json' file + * ``Config.__DEFAULT_CONFIG`` -- some sane default values; + * ``dynamic_configuration`` -- configuration stored in DCS; + * ``local_configuration`` -- configuration from `config.yml` or environment. + + 2) Saving and loading ``dynamic_configuration`` into 'patroni.dynamic.json' file located in local_configuration['postgresql']['data_dir'] directory. - This is necessary to be able to restore `dynamic_configuration` - if DCS was accidentally wiped + This is necessary to be able to restore ``dynamic_configuration`` + if DCS was accidentally wiped. - 3) Loading of configuration file in the old format and converting it into new format + 3) Loading of configuration file in the old format and converting it into new format. - 4) Mimicking some of the `dict` interfaces to make it possible - to work with it as with the old `config` object. + 4) Mimicking some ``dict`` interfaces to make it possible + to work with it as with the old ``config`` object. + + :cvar PATRONI_CONFIG_VARIABLE: name of the environment variable that can be used to load Patroni configuration from. + :cvar __CACHE_FILENAME: name of the file used to cache dynamic configuration under Postgres data directory. + :cvar __DEFAULT_CONFIG: default configuration values for some Patroni settings. """ PATRONI_CONFIG_VARIABLE = PATRONI_ENV_PREFIX + 'CONFIGURATION' @@ -207,12 +255,30 @@ class Config(object): def __init__(self, configfile: str, validator: Optional[Callable[[Dict[str, Any]], List[str]]] = default_validator) -> None: + """Create a new instance of :class:`Config` and validate the loaded configuration using *validator*. + + .. note:: + Patroni will read configuration from these locations in this order: + + * file or directory path passed as command-line argument (*configfile*), if it exists and the file or + files found in the directory can be parsed (see :meth:`~Config._load_config_path`), otherwise + * YAML file passed via the environment variable (see :cvar:`PATRONI_CONFIG_VARIABLE`), if the referenced + file exists and can be parsed, otherwise + * from configuration values defined as environment variables, see + :meth:`~Config._build_environment_configuration`. + + :param configfile: path to Patroni configuration file. + :param validator: function used to validate Patroni configuration. It should receive a dictionary which + represents Patroni configuration, and return a list of zero or more error messages based on validation. + + :raises: + :class:`ConfigParseError`: if any issue is reported by *validator*. + """ self._modify_version = -1 self._dynamic_configuration = {} self.__environment_configuration = self._build_environment_configuration() - # Patroni reads the configuration from the command-line argument if it exists, otherwise from the environment self._config_file = configfile if configfile and os.path.exists(configfile) else None if self._config_file: self._local_configuration = self._load_config_file() @@ -233,11 +299,13 @@ class Config(object): self._cache_needs_saving = False @property - def config_file(self) -> Union[str, None]: + def config_file(self) -> Optional[str]: + """Path to Patroni configuration file, if any, else ``None``.""" return self._config_file @property def dynamic_configuration(self) -> Dict[str, Any]: + """Deep copy of cached Patroni dynamic configuration.""" return deepcopy(self._dynamic_configuration) @property @@ -257,9 +325,17 @@ class Config(object): return deepcopy(cls.__DEFAULT_CONFIG) def _load_config_path(self, path: str) -> Dict[str, Any]: - """ - If path is a file, loads the yml file pointed to by path. - If path is a directory, loads all yml files in that directory in alphabetical order + """Load Patroni configuration file(s) from *path*. + + If *path* is a file, load the yml file pointed to by *path*. + If *path* is a directory, load all yml files in that directory in alphabetical order. + + :param path: path to either an YAML configuration file, or to a folder containing YAML configuration files. + + :returns: configuration after reading the configuration file(s) from *path*. + + :raises: + :class:`ConfigParseError`: if *path* is invalid. """ if os.path.isfile(path): files = [path] @@ -278,14 +354,18 @@ class Config(object): return overall_config def _load_config_file(self) -> Dict[str, Any]: - """Loads config.yaml from filesystem and applies some values which were set via ENV""" + """Load configuration file(s) from filesystem and apply values which were set via environment variables. + + :returns: final configuration after merging configuration file(s) and environment variables. + """ if TYPE_CHECKING: # pragma: no cover - assert self._config_file is not None - config = self._load_config_path(self._config_file) + assert self.config_file is not None + config = self._load_config_path(self.config_file) patch_config(config, self.__environment_configuration) return config def _load_cache(self) -> None: + """Load dynamic configuration from ``patroni.dynamic.json``.""" if os.path.isfile(self._cache_file): try: with open(self._cache_file) as f: @@ -294,6 +374,12 @@ class Config(object): logger.exception('Exception when loading file: %s', self._cache_file) def save_cache(self) -> None: + """Save dynamic configuration to ``patroni.dynamic.json`` under Postgres data directory. + + .. note:: + ``patroni.dynamic.jsonXXXXXX`` is created as a temporary file and than renamed to ``patroni.dynamic.json``, + where ``XXXXXX`` is a random suffix. + """ if self._cache_needs_saving: tmpfile = fd = None try: @@ -320,9 +406,16 @@ class Config(object): # configuration could be either ClusterConfig or dict def set_dynamic_configuration(self, configuration: Union[ClusterConfig, Dict[str, Any]]) -> bool: + """Set dynamic configuration values with given *configuration*. + + :param configuration: new dynamic configuration values. Supports :class:`dict` for backward compatibility. + + :returns: ``True`` if changes have been detected between current dynamic configuration and the new dynamic + *configuration*, ``False`` otherwise. + """ if isinstance(configuration, ClusterConfig): if self._modify_version == configuration.modify_version: - return False # If the version didn't changed there is nothing to do + return False # If the version didn't change there is nothing to do self._modify_version = configuration.modify_version configuration = configuration.data @@ -338,6 +431,14 @@ class Config(object): return False def reload_local_configuration(self) -> Optional[bool]: + """Reload configuration values from the configuration file(s). + + .. note:: + Designed to be used when user applies changes to configuration file(s), so Patroni can use the new values + with a reload instead of a restart. + + :returns: ``True`` if changes have been detected between current local configuration + """ if self.config_file: try: configuration = self._load_config_file() @@ -353,6 +454,38 @@ class Config(object): @staticmethod def _process_postgresql_parameters(parameters: Dict[str, Any], is_local: bool = False) -> Dict[str, Any]: + """Process Postgres *parameters*. + + .. note:: + If *is_local* configuration discard any setting from *parameters* that is listed under + :attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` as those are supposed to be set only + through dynamic configuration. + + When setting parameters from :attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` through + dynamic configuration their value will be validated as per the validator defined in that very same + attribute entry. If the given value cannot be validated, a warning will be logged and the default value of + the GUC will be used instead. + + Some parameters from :attr:`~patroni.postgresql.config.ConfigHandler.CMDLINE_OPTIONS` cannot be set even if + not *is_local* configuration: + + * ``listen_addresses``: inferred from ``postgresql.listen`` local configuration or from + ``PATRONI_POSTGRESQL_LISTEN`` environment variable; + * ``port``: inferred from ``postgresql.listen`` local configuration or from + ``PATRONI_POSTGRESQL_LISTEN`` environment variable; + * ``cluster_name``: set through ``scope`` local configuration or through ``PATRONI_SCOPE`` environment + variable; + * ``hot_standby``: always enabled; + * ``wal_log_hints``: always enabled. + + :param parameters: Postgres parameters to be processed. Should be the parsed YAML value of + ``postgresql.parameters`` configuration, either from local or from dynamic configuration. + + :param is_local: should be ``True`` if *parameters* refers to local configuration, or ``False`` if *parameters* + refers to dynamic configuration. + + :returns: new value for ``postgresql.parameters`` after processing and validating *parameters*. + """ pg_params: Dict[str, Any] = {} for name, value in (parameters or {}).items(): @@ -368,6 +501,32 @@ class Config(object): return pg_params def _safe_copy_dynamic_configuration(self, dynamic_configuration: Dict[str, Any]) -> Dict[str, Any]: + """Create a copy of *dynamic_configuration*. + + Merge *dynamic_configuration* with :attr:`__DEFAULT_CONFIG` (*dynamic_configuration* takes precedence), and + process ``postgresql.parameters`` from *dynamic_configuration* through :func:`_process_postgresql_parameters`, + if present. + + .. note:: + The following settings are not allowed in ``postgresql`` section as they are intended to be local + configuration, and are removed if present: + + * ``connect_address``; + * ``proxy_address``; + * ``listen``; + * ``config_dir``; + * ``data_dir``; + * ``pgpass``; + * ``authentication``; + + Besides that any setting present in *dynamic_configuration* but absent from :attr:`__DEFAULT_CONFIG` is + discarded. + + :param dynamic_configuration: Patroni dynamic configuration. + + :returns: copy of *dynamic_configuration*, merged with default dynamic configuration and with some sanity checks + performed over it. + """ config = self.get_default_config() for name, value in dynamic_configuration.items(): @@ -388,9 +547,25 @@ class Config(object): @staticmethod def _build_environment_configuration() -> Dict[str, Any]: + """Get local configuration settings that were specified through environment variables. + + :returns: dictionary containing the found environment variables and their values, respecting the expected + structure of Patroni configuration. + """ ret: Dict[str, Any] = defaultdict(dict) - def _popenv(name: str) -> Union[str, None]: + def _popenv(name: str) -> Optional[str]: + """Get value of environment variable *name*. + + .. note:: + *name* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` when searching in the environment. + + Also, the corresponding environment variable is removed from the environment upon reading its value. + + :param name: name of the environment variable. + + :returns: value of *name*, if present in the environment, otherwise ``None``. + """ return os.environ.pop(PATRONI_ENV_PREFIX + name.upper(), None) for param in ('name', 'namespace', 'scope'): @@ -399,6 +574,23 @@ class Config(object): ret[param] = value def _fix_log_env(name: str, oldname: str) -> None: + """Normalize a log related environment variable. + + .. note:: + Patroni used to support different names for log related environment variables in the past. As the + environment variables were renamed, this function takes care of mapping and normalizing the environment. + + *name* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` and ``LOG`` when searching in the + environment. + + *oldname* is prefixed with :data:`~patroni.PATRONI_ENV_PREFIX` when searching in the environment. + + If both *name* and *oldname* are set in the environment, *name* takes precedence. + + :param name: new name of a log related environment variable. + :param oldname: original name of a log related environment variable. + :type oldname: str + """ value = _popenv(oldname) name = PATRONI_ENV_PREFIX + 'LOG_' + name.upper() if value and name not in os.environ: @@ -408,6 +600,15 @@ class Config(object): _fix_log_env(name, oldname) def _set_section_values(section: str, params: List[str]) -> None: + """Get value of *params* environment variables that are related with *section*. + + .. note:: + The values are retrieved from the environment and updated directly into the returning dictionary of + :func:`_build_environment_configuration`. + + :param section: configuration section the *params* belong to. + :param params: name of the Patroni settings. + """ for param in params: value = _popenv(section + '_' + param) if value: @@ -429,6 +630,7 @@ class Config(object): if value: ret['postgresql'].setdefault('bin_name', {})[binary] = value + # parse all values retrieved from the environment as Python objects, according to the expected type for first, second in (('restapi', 'allowlist_include_members'), ('ctl', 'insecure')): value = ret.get(first, {}).pop(second, None) if value: @@ -445,7 +647,13 @@ class Config(object): if value is not None: ret[first][second] = value - def _parse_list(value: str) -> Union[List[str], None]: + def _parse_list(value: str) -> Optional[List[str]]: + """Parse an YAML list *value* as a :class:`list`. + + :param value: YAML list as a string. + + :returns: *value* as :class:`list`. + """ if not (value.strip().startswith('-') or '[' in value): value = '[{0}]'.format(value) try: @@ -461,7 +669,13 @@ class Config(object): if value: ret[first][second] = value - def _parse_dict(value: str) -> Union[Dict[str, Any], None]: + def _parse_dict(value: str) -> Optional[Dict[str, Any]]: + """Parse an YAML dictionary *value* as a :class:`dict`. + + :param value: YAML dictionary as a string. + + :returns: *value* as :class:`dict`. + """ if not value.strip().startswith('{'): value = '{{{0}}}'.format(value) try: @@ -478,9 +692,16 @@ class Config(object): if value: ret[first][second] = value - def _get_auth(name: str, params: Optional[Collection[str]] = None) -> Dict[str, str]: + def _get_auth(name: str, params: Collection[str] = _AUTH_ALLOWED_PARAMETERS[:2]) -> Dict[str, str]: + """Get authorization related environment variables *params* from section *name*. + + :param name: name of a configuration section that may contain authorization *params*. + :param params: the authorization settings that may be set under section *name*. + + :returns: dictionary containing environment values for authorization *params* of section *name*. + """ ret: Dict[str, str] = {} - for param in params or _AUTH_ALLOWED_PARAMETERS[:2]: + for param in params: value = _popenv(name + '_' + param) if value: ret[param] = value @@ -503,7 +724,7 @@ class Config(object): for param in list(os.environ.keys()): if param.startswith(PATRONI_ENV_PREFIX): # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..) - name, suffix = (param[8:].split('_', 1) + [''])[:2] + name, suffix = (param[len(PATRONI_ENV_PREFIX):].split('_', 1) + [''])[:2] if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'SRV_SUFFIX', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY', 'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'SERVICE_CHECK_TLS_SERVER_NAME', @@ -534,14 +755,14 @@ class Config(object): users = {} for param in list(os.environ.keys()): if param.startswith(PATRONI_ENV_PREFIX): - name, suffix = (param[8:].rsplit('_', 1) + [''])[:2] + name, suffix = (param[len(PATRONI_ENV_PREFIX):].rsplit('_', 1) + [''])[:2] # PATRONI__PASSWORD=, PATRONI__OPTIONS= # CREATE USER "" WITH PASSWORD '' if name and suffix == 'PASSWORD': password = os.environ.pop(param) if password: users[name] = {'password': password} - options = os.environ.pop(param[:-9] + '_OPTIONS', None) + options = os.environ.pop(param[:-9] + '_OPTIONS', None) # replace "_PASSWORD" with "_OPTIONS" options = options and _parse_list(options) if options: users[name]['options'] = options @@ -552,6 +773,16 @@ class Config(object): def _build_effective_configuration(self, dynamic_configuration: Dict[str, Any], local_configuration: Dict[str, Union[Dict[str, Any], Any]]) -> Dict[str, Any]: + """Build effective configuration by merging *dynamic_configuration* and *local_configuration*. + + .. note:: + *local_configuration* takes precedence over *dynamic_configuration* if a setting is defined in both. + + :param dynamic_configuration: Patroni dynamic configuration. + :param local_configuration: Patroni local configuration. + + :returns: _description_ + """ config = self._safe_copy_dynamic_configuration(dynamic_configuration) for name, value in local_configuration.items(): if name == 'citus': # remove invalid citus configuration @@ -615,23 +846,57 @@ class Config(object): return config def get(self, key: str, default: Optional[Any] = None) -> Any: + """Get effective value of ``key`` setting from Patroni configuration root. + + Designed to work the same way as :func:`dict.get`. + + :param key: name of the setting. + :param default: default value if *key* is not present in the effective configuration. + + :returns: value of *key*, if present in the effective configuration, otherwise *default*. + """ return self.__effective_configuration.get(key, default) def __contains__(self, key: str) -> bool: + """Check if setting *key* is present in the effective configuration. + + Designed to work the same way as :func:`dict.__contains__`. + + :param key: name of the setting to be checked. + + :returns: ``True`` if setting *key* exists in effective configuration, else ``False``. + """ return key in self.__effective_configuration def __getitem__(self, key: str) -> Any: + """Get value of setting *key* from effective configuration. + + Designed to work the same way as :func:`dict.__getitem__`. + + :param key: name of the setting. + + :returns: value of setting *key*. + + :raises: + :class:`KeyError`: if *key* is not present in effective configuration. + """ return self.__effective_configuration[key] def copy(self) -> Dict[str, Any]: + """Get a deep copy of effective Patroni configuration. + + :returns: a deep copy of the Patroni configuration. + """ return deepcopy(self.__effective_configuration) - def get_global_config(self, cluster: Union[Cluster, None]) -> GlobalConfig: + def get_global_config(self, cluster: Optional[Cluster]) -> GlobalConfig: """Instantiate :class:`GlobalConfig` based on input. Use the configuration from provided *cluster* (the most up-to-date) or from the local cache if *cluster.config* is not initialized or doesn't have a valid config. - :param cluster: the currently known cluster state from DCS - :returns: :class:`GlobalConfig` object + + :param cluster: the currently known cluster state from DCS. + + :returns: :class:`GlobalConfig` object. """ return get_global_config(cluster, self._dynamic_configuration) diff --git a/patroni/ha.py b/patroni/ha.py index 17528a55..09f23ff7 100644 --- a/patroni/ha.py +++ b/patroni/ha.py @@ -473,16 +473,23 @@ class Ha(object): """Handle the case when postgres isn't running. Depending on the state of Patroni, DCS cluster view, and pg_controldata the following could happen: - - if ``primary_start_timeout`` is 0 and this node owns the leader lock, the lock - will be voluntarily released if there are healthy replicas to take it over. - - if postgres was running as a ``primary`` and this node owns the leader lock, postgres is started as primary. - - crash recover in a single-user mode is executed in the following cases: - - postgres was running as ``primary`` wasn't ``shut down`` cleanly and there is no leader in DCS - - postgres was running as ``replica`` wasn't ``shut down in recovery`` (cleanly) - and we need to run ``pg_rewind`` to join back to the cluster. - - ``pg_rewind`` is executed if it is necessary, or optinally, the data directory could - be removed if it is allowed by configuration. - - after ``crash recovery`` and/or ``pg_rewind`` are executed, postgres is started in recovery. + + - if ``primary_start_timeout`` is 0 and this node owns the leader lock, the lock + will be voluntarily released if there are healthy replicas to take it over. + + - if postgres was running as a ``primary`` and this node owns the leader lock, postgres is started as primary. + + - crash recover in a single-user mode is executed in the following cases: + + - postgres was running as ``primary`` wasn't ``shut down`` cleanly and there is no leader in DCS + + - postgres was running as ``replica`` wasn't ``shut down in recovery`` (cleanly) + and we need to run ``pg_rewind`` to join back to the cluster. + + - ``pg_rewind`` is executed if it is necessary, or optinally, the data directory could + be removed if it is allowed by configuration. + + - after ``crash recovery`` and/or ``pg_rewind`` are executed, postgres is started in recovery. :returns: action message, describing what was performed. """ @@ -1019,6 +1026,7 @@ class Ha(object): """Check if node should consider itself unhealthy to be promoted due to replication lag. :param wal_position: Current wal position. + :returns `True` when node is lagging """ lag = (self.cluster.last_lsn or 0) - wal_position @@ -1132,7 +1140,7 @@ class Ha(object): :returns: - `True` if the current node is the best candidate to become the new leader - `None` if the current node is running as a primary and requested candidate doesn't exist - """ + """ failover = self.cluster.failover if TYPE_CHECKING: # pragma: no cover assert failover is not None @@ -1190,6 +1198,7 @@ class Ha(object): """Performs a series of checks to determine that the current node is the best candidate. In case if manual failover/switchover is requested it calls :func:`manual_failover_process_no_leader` method. + :returns: `True` if the current node is among the best candidates to become the new leader. """ if time.time() - self._released_leader_key_timestamp < self.dcs.ttl: @@ -1276,13 +1285,15 @@ class Ha(object): def demote(self, mode: str) -> Optional[bool]: """Demote PostgreSQL running as primary. - :param mode: One of offline, graceful or immediate. - offline is used when connection to DCS is not available. - graceful is used when failing over to another node due to user request. May only be called running async. - immediate is used when we determine that we are not suitable for primary and want to failover quickly - without regard for data durability. May only be called synchronously. - immediate-nolock is used when find out that we have lost the lock to be primary. Need to bring down - PostgreSQL as quickly as possible without regard for data durability. May only be called synchronously. + :param mode: One of offline, graceful, immediate or immediate-nolock. + ``offline`` is used when connection to DCS is not available. + ``graceful`` is used when failing over to another node due to user request. May only be called + running async. + ``immediate`` is used when we determine that we are not suitable for primary and want to failover + quickly without regard for data durability. May only be called synchronously. + ``immediate-nolock`` is used when find out that we have lost the lock to be primary. Need to bring + down PostgreSQL as quickly as possible without regard for data durability. May only be called + synchronously. """ mode_control = { 'offline': dict(stop='fast', checkpoint=False, release=False, offline=True, async_req=False), # noqa: E241,E501 @@ -1664,9 +1675,7 @@ class Ha(object): self._async_executor.run_async(self._do_reinitialize, args=(cluster, )) def handle_long_action_in_progress(self) -> str: - """ - Figure out what to do with the task AsyncExecutor is performing. - """ + """Figure out what to do with the task AsyncExecutor is performing.""" if self.has_lock() and self.update_lock(): if self._async_executor.scheduled_action == 'doing crash recovery in a single user mode': time_left = self.global_config.primary_start_timeout - (time.time() - self._crash_recovery_started) @@ -1761,8 +1770,7 @@ class Ha(object): return 'initialized a new cluster' def handle_starting_instance(self) -> Optional[str]: - """Starting up PostgreSQL may take a long time. In case we are the leader we may want to - fail over to.""" + """Starting up PostgreSQL may take a long time. In case we are the leader we may want to fail over to.""" # Check if we are in startup, when paused defer to main loop for manual failovers. if not self.state_handler.check_for_startup() or self.is_paused(): @@ -1802,7 +1810,8 @@ class Ha(object): def set_start_timeout(self, value: Optional[int]) -> None: """Sets timeout for starting as primary before eligible for failover. - Must be called when async_executor is busy or in the main thread.""" + Must be called when async_executor is busy or in the main thread. + """ self._start_timeout = value def _run_cycle(self) -> str: @@ -2007,7 +2016,9 @@ class Ha(object): """Handles replication slots. :param dcs_failed: bool, indicates that communication with DCS failed (get_cluster() or update_leader()) - :returns: list[str], replication slots names that should be copied from the primary""" + + :returns: list[str], replication slots names that should be copied from the primary + """ slots: List[str] = [] @@ -2097,15 +2108,16 @@ class Ha(object): return self.dcs.watch(leader_version, timeout) def wakeup(self) -> None: - """Call of this method will trigger the next run of HA loop if there is - no "active" leader watch request in progress. + """Trigger the next run of HA loop if there is no "active" leader watch request in progress. + This usually happens on the leader or if the node is running async action""" self.dcs.event.set() def get_remote_member(self, member: Union[Leader, Member, None] = None) -> RemoteMember: - """ In case of standby cluster this will tel us from which remote - member to stream. Config can be both patroni config or - cluster.config.data + """Get remote member node to stream from. + + In case of standby cluster this will tell us from which remote member to stream. Config can be both patroni + config or cluster.config.data. """ data: Dict[str, Any] = {} cluster_params = self.global_config.get_standby_cluster_config() diff --git a/patroni/log.py b/patroni/log.py index 55b63386..09d73883 100644 --- a/patroni/log.py +++ b/patroni/log.py @@ -21,17 +21,17 @@ _LOGGER = logging.getLogger(__name__) def debug_exception(self: logging.Logger, msg: object, *args: Any, **kwargs: Any) -> None: """Add full stack trace info to debug log messages and partial to others. - Handle :func:`exception` calls for *self*. + Handle :func:`~self.exception` calls for *self*. .. note:: * If *self* log level is set to ``DEBUG``, then issue a ``DEBUG`` message with the complete stack trace; * If *self* log level is ``INFO`` or higher, then issue an ``ERROR`` message with only the last line of the stack trace. - :param self: logger for which :func:`exception` will be processed. + :param self: logger for which :func:`~self.exception` will be processed. :param msg: the message related to the exception to be logged. - :param args: positional arguments to be passed to :func:`self.debug` or :func:`loger_obj.error`. - :param kwargs: keyword arguments to be passed to :func:`self.debug` or :func:`loger_obj.error`. + :param args: positional arguments to be passed to :func:`~self.debug` or :func:`~self.error`. + :param kwargs: keyword arguments to be passed to :func:`~self.debug` or :func:`~self.error`. """ kwargs.pop("exc_info", False) if self.isEnabledFor(logging.DEBUG): @@ -44,16 +44,16 @@ def debug_exception(self: logging.Logger, msg: object, *args: Any, **kwargs: Any def error_exception(self: logging.Logger, msg: object, *args: Any, **kwargs: Any) -> None: """Add full stack trace info to error messages. - Handle :func:`exception` calls for *self*. + Handle :func:`~self.exception` calls for *self*. .. note:: * By default issue an ``ERROR`` message with the complete stack trace. If you do not want to show the complete - stack trace, call with ``exc_info=False``. + stack trace, call with ``exc_info=False``. - :param self: logger for which :func:`exception` will be processed. + :param self: logger for which :func:`~self.exception` will be processed. :param msg: the message related to the exception to be logged. - :param args: positional arguments to be passed to :func:`loger_obj.error`. - :param kwargs: keyword arguments to be passed to :func:`loger_obj.error`. + :param args: positional arguments to be passed to :func:`~self.error`. + :param kwargs: keyword arguments to be passed to :func:`~self.error`. """ exc_info = kwargs.pop("exc_info", True) self.error(msg, *args, exc_info=exc_info, **kwargs) @@ -140,7 +140,7 @@ class ProxyHandler(logging.Handler): def emit(self, record: logging.LogRecord) -> None: """Emit each log record that is handled. - Will push the log record down to :func:`handle` method of the currently configured log handler. + Will push the log record down to :func:`~logging.Handler.handle` method of the currently configured log handler. :param record: the record that was emitted. """ @@ -203,7 +203,7 @@ class PatroniLogger(Thread): self._root_logger.addHandler(self._proxy_handler) def update_loggers(self) -> None: - """Configure loggers' log level as defined in ``log.loggers` section of Patroni configuration. + """Configure loggers' log level as defined in ``log.loggers`` section of Patroni configuration. .. note:: It creates logger objects that are not defined yet in the log manager. @@ -281,7 +281,8 @@ class PatroniLogger(Thread): .. note:: It is used to remove different handlers that were configured previous to a reload in the configuration, - e.g. if we are switching from :class:`RotatingFileHandler` to class:`StreamHandler` and vice-versa. + e.g. if we are switching from :class:`~logging.handlers.RotatingFileHandler` to + class:`~logging.StreamHandler` and vice-versa. """ while True: with self.log_handler_lock: diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py index 376ed071..0d1f9b60 100644 --- a/patroni/postgresql/__init__.py +++ b/patroni/postgresql/__init__.py @@ -183,6 +183,7 @@ class Postgresql(object): """Returns the monitoring query with a fixed number of fields. The query text is constructed based on current state in DCS and PostgreSQL version: + 1. function names depend on version. wal/lsn for v10+ and xlog/location for pre v10. 2. for primary we query timeline_id (extracted from pg_walfile_name()) and pg_current_wal_lsn() 3. for replicas we query pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), and pg_is_wal_replay_paused() @@ -192,7 +193,8 @@ class Postgresql(object): 7. if sync replication is enabled we query pg_stat_replication and aggregate the result. In addition to that we get current values of synchronous_commit and synchronous_standby_names GUCs. - If some conditions are not satisfied we simply put static values instead. E.g., NULL, 0, '', and so on.""" + If some conditions are not satisfied we simply put static values instead. E.g., NULL, 0, '', and so on. + """ extra = ", " + (("pg_catalog.current_setting('synchronous_commit'), " "pg_catalog.current_setting('synchronous_standby_names'), " diff --git a/patroni/postgresql/citus.py b/patroni/postgresql/citus.py index b5c90cf5..a3a8fe45 100644 --- a/patroni/postgresql/citus.py +++ b/patroni/postgresql/citus.py @@ -174,11 +174,13 @@ class CitusHandler(Thread): """Returns the tuple(i, task), where `i` - is the task index in the self._tasks list Tasks are picked by following priorities: + 1. If there is already a transaction in progress, pick a task that that will change already affected worker primary. 2. If the coordinator address should be changed - pick a task with group=0 (coordinators are always in group 0). - 3. Pick a task that is the oldest (first from the self._tasks)""" + 3. Pick a task that is the oldest (first from the self._tasks) + """ with self._condition: if self._in_flight: diff --git a/patroni/postgresql/slots.py b/patroni/postgresql/slots.py index 4df27eaf..681aed00 100644 --- a/patroni/postgresql/slots.py +++ b/patroni/postgresql/slots.py @@ -544,7 +544,7 @@ class SlotsHandler: 1) Retrieve the current ``catalog_xmin`` value for the physical slot from the cluster leader, and 2) using previously stored list of "unready" logical slots, those which have yet to be checked hence have no - stored slot attributes, + stored slot attributes, 3) store logical slot ``catalog_xmin`` when the physical slot ``catalog_xmin`` becomes valid. :param cluster: object containing stateful information for the cluster. diff --git a/patroni/postgresql/sync.py b/patroni/postgresql/sync.py index 8864a166..c55874fc 100644 --- a/patroni/postgresql/sync.py +++ b/patroni/postgresql/sync.py @@ -317,7 +317,7 @@ END;$$""") self._ready_replicas[replica.application_name] = replica.pid def current_state(self, cluster: Cluster) -> _SyncState: - """Finds best candidates to be the synchronous standbys. + """Find the best candidates to be the synchronous standbys. Current synchronous standby is always preferred, unless it has disconnected or does not want to be a synchronous standby any longer. diff --git a/patroni/postgresql/validator.py b/patroni/postgresql/validator.py index 7da0003a..d568fa74 100644 --- a/patroni/postgresql/validator.py +++ b/patroni/postgresql/validator.py @@ -178,10 +178,11 @@ class ValidatorFactory: :returns: the Patroni validator object that corresponds to the specification found in *validator*. - :raises :class:`ValidatorFactoryNoType`: if *validator* contains no ``type`` key. - :raises :class:`ValidatorFactoryInvalidType`: if ``type`` key from *validator* contains an invalid value. - :raises :class:`ValidatorFactoryInvalidSpec`: if *validator* contains an invalid set of attributes for the - given ``type``. + :raises: + :class:`ValidatorFactoryNoType`: if *validator* contains no ``type`` key. + :class:`ValidatorFactoryInvalidType`: if ``type`` key from *validator* contains an invalid value. + :class:`ValidatorFactoryInvalidSpec`: if *validator* contains an invalid set of attributes for the given + ``type``. :Example: @@ -265,7 +266,8 @@ def _read_postgres_gucs_validators_file(file: str) -> Dict[str, Any]: :returns: the YAML content parsed into a Python object. If any issue is faced while reading/parsing the file, then return ``None``. - :raises :class:`InvalidGucValidatorsFile`: if faces an issue while reading or parsing *file*. + :raises: + :class:`InvalidGucValidatorsFile`: if faces an issue while reading or parsing *file*. """ try: with open(file, encoding='UTF-8') as stream: @@ -462,11 +464,13 @@ def transform_postgresql_parameter_value(version: int, name: str, value: Any, :param value: value of the Postgres GUC. :param available_gucs: a set of all GUCs available in Postgres *version*. Each item is the name of a Postgres GUC. Used for a couple purposes: - * Disallow writing GUCs to ``postgresql.conf`` that does not exist in Postgres *version*; - * Avoid ignoring GUC *name* if it does not have a validator in ``parameters``, but is a valid GUC in Postgres - *version*. - :returns: The return value may be one among + * Disallow writing GUCs to ``postgresql.conf`` that does not exist in Postgres *version*; + * Avoid ignoring GUC *name* if it does not have a validator in ``parameters``, but is a valid GUC in + Postgres *version*. + + :returns: The return value may be one among: + * The original *value* if *name* seems to be an extension GUC (contains a period '.'); or * ``None`` if **name** is a recovery GUC; or * *value* transformed to the expected format for GUC *name* in Postgres *version* using validators defined in @@ -490,10 +494,11 @@ def transform_recovery_parameter_value(version: int, name: str, value: Any, :param value: value of the Postgres recovery GUC. :param available_gucs: a set of all GUCs available in Postgres *version*. Each item is the name of a Postgres GUC. Used for a couple purposes: - * Disallow writing GUCs to ``recovery.conf`` (or ``postgresql.conf`` depending on *version*), that does not - exist in Postgres *version*; - * Avoid ignoring recovery GUC *name* if it does not have a validator in ``recovery_parameters``, but is a valid - GUC in Postgres *version*. + + * Disallow writing GUCs to ``recovery.conf`` (or ``postgresql.conf`` depending on *version*), that does not + exist in Postgres *version*; + * Avoid ignoring recovery GUC *name* if it does not have a validator in ``recovery_parameters``, but is a + valid GUC in Postgres *version*. :returns: *value* transformed to the expected format for recovery GUC *name* in Postgres *version* using validators defined in ``recovery_parameters``. It can also return ``None``. See :func:`_transform_parameter_value`. diff --git a/patroni/psycopg.py b/patroni/psycopg.py index d337dd3e..4a92047c 100644 --- a/patroni/psycopg.py +++ b/patroni/psycopg.py @@ -1,7 +1,8 @@ -"""Abstraction layer for ``psycopg`` module. +"""Abstraction layer for :mod:`psycopg` module. -This module is able to handle both ``pyscopg2`` and ``psycopg3``, and it exposes a common interface for both. -``psycopg2`` takes precedence. ``psycopg3`` will only be used if ``psycopg2`` is either absent or older than ``2.5.4``. +This module is able to handle both :mod:`pyscopg2` and :mod:`psycopg`, and it exposes a common interface for both. +:mod:`psycopg2` takes precedence. :mod:`psycopg` will only be used if :mod:`psycopg2` is either absent or older than +``2.5.4``. """ from typing import Any, Optional, TYPE_CHECKING, Union if TYPE_CHECKING: # pragma: no cover @@ -28,7 +29,7 @@ try: """Quote *value* as a SQL literal. .. note:: - *value* is quoted through ``psycopg`` adapters. + *value* is quoted through :mod:`psycopg2` adapters. :param value: value to be quoted. :param conn: if a connection is given then :func:`quote_literal` checks if any special handling based on server @@ -44,14 +45,14 @@ except ImportError: from psycopg import connect as __connect, sql, Error, DatabaseError, OperationalError, ProgrammingError def _connect(dsn: Optional[str] = None, **kwargs: Any) -> 'Connection[Any]': - """Call ``psycopg.connect`` with ``dsn`` and ``**kwargs``. + """Call :func:`psycopg.connect` with *dsn* and ``**kwargs``. .. note:: Will create ``server_version`` attribute in the returning connection, so it keeps compatibility with the - object that would be returned by ``psycopg2.connect``. + object that would be returned by :func:`psycopg2.connect`. - :param dsn: DSN to call ``psycopg.connect`` with. - :param kwargs: keyword arguments to call ``psycopg.connect`` with. + :param dsn: DSN to call :func:`psycopg.connect` with. + :param kwargs: keyword arguments to call :func:`psycopg.connect` with. :returns: a connection to the database. """ @@ -89,11 +90,11 @@ def connect(*args: Any, **kwargs: Any) -> Union['connection', 'Connection[Any]'] It also enforces ``search_path=pg_catalog`` for non-replication connections to mitigate security issues as Patroni relies on superuser connections. - :param args: positional arguments to call ``connect`` function from ``psycopg`` module. - :param kwargs: keyword arguments to call ``connect`` function from ``psycopg`` module. + :param args: positional arguments to call :func:`~psycopg.connect` function from :mod:`psycopg` module. + :param kwargs: keyword arguments to call :func:`~psycopg.connect` function from :mod:`psycopg` module. - :returns: a connection to the database. Can be either a :class:`psycopg.Connection` if using ``psycopg3``, or a - :class:`psycopg2.extensions.connection` if using ``psycopg2``. + :returns: a connection to the database. Can be either a :class:`psycopg.Connection` if using :mod:`psycopg`, or a + :class:`psycopg2.extensions.connection` if using :mod:`psycopg2`. """ if kwargs and 'replication' not in kwargs and kwargs.get('fallback_application_name') != 'Patroni ctl': options = [kwargs['options']] if 'options' in kwargs else [] @@ -109,7 +110,7 @@ def quote_ident(value: Any, conn: Optional[Union['cursor', 'connection', 'Connec :param value: value to be quoted. :param conn: connection to evaluate the returning string into. Can be either a :class:`psycopg.Connection` if - using ``psycopg3``, or a :class:`psycopg2.extensions.connection` if using ``psycopg2``. + using :mod:`psycopg`, or a :class:`psycopg2.extensions.connection` if using :mod:`psycopg2`. :returns: *value* quoted as a SQL identifier. """ diff --git a/patroni/request.py b/patroni/request.py index 0421bf86..16659c96 100644 --- a/patroni/request.py +++ b/patroni/request.py @@ -34,10 +34,11 @@ class PatroniRequest(object): """Create a new :class:`PatroniRequest` instance with given *config*. :param config: Patroni YAML configuration. - :param insecure: how to deal with SSL certs verification + :param insecure: how to deal with SSL certs verification: + * If ``True`` it will perform REST API requests without verifying SSL certs; or * If ``False`` it will perform REST API requests and verify SSL certs; or - * If ``None`` it will behave according to the value of ``ctl -> insecure`` configuration; or + * If ``None`` it will behave according to the value of ``ctl.insecure`` configuration; or * If none of the above applies, then it falls back to ``False``. """ self._insecure = insecure @@ -51,7 +52,7 @@ class PatroniRequest(object): :param config: Patroni YAML configuration. :param name: name of the setting value to be retrieved. - :returns: value of ``ctl -> *name*`` if present, ``None`` otherwise. + :returns: value of ``ctl.*name*`` if present, ``None`` otherwise. """ return config.get('ctl', {}).get(name, default) @@ -83,12 +84,13 @@ class PatroniRequest(object): :param config: Patroni YAML configuration. :param name: prefix of the Patroni SSL related setting name. Currently, supports these: + * ``cert``: gets translated to ``certfile`` * ``key``: gets translated to ``keyfile`` Will attempt to fetch the requested key first from ``ctl`` section. - :returns: value of ``ctl -> *name*file`` if present, ``None`` otherwise. + :returns: value of ``ctl.*name*file`` if present, ``None`` otherwise. """ value = self._get_ctl_value(config, name + 'file') self._apply_pool_param(name + '_file', value) @@ -99,13 +101,13 @@ class PatroniRequest(object): Configure these HTTP headers for requests: - * ``authorization``: based on Patroni' CTL or REST API authentication config; - * ``user-agent``: based on `patroni.utils.USER_AGENT`. + * ``authorization``: based on Patroni' CTL or REST API authentication config; + * ``user-agent``: based on ``patroni.utils.USER_AGENT``. Also configure SSL related settings for requests: - * ``ca_certs`` is configured if ``ctl -> cacert`` or ``restapi -> cafile`` is available; - * ``cert``, ``key`` and ``key_password`` are configured if ``ctl -> certfile`` is available. + * ``ca_certs`` is configured if ``ctl.cacert`` or ``restapi.cafile`` is available; + * ``cert``, ``key`` and ``key_password`` are configured if ``ctl.certfile`` is available. :param config: Patroni YAML configuration. """ diff --git a/patroni/utils.py b/patroni/utils.py index ad04b7fa..8046c6b0 100644 --- a/patroni/utils.py +++ b/patroni/utils.py @@ -130,8 +130,8 @@ def parse_bool(value: Any) -> Union[bool, None]: .. note:: The parsing is case-insensitive, and takes into consideration these values: - * ``on``, ``true``, ``yes``, and ``1`` as ``True``. - * ``off``, ``false``, ``no``, and ``0`` as ``False``. + * ``on``, ``true``, ``yes``, and ``1`` as ``True``. + * ``off``, ``false``, ``no``, and ``0`` as ``False``. :param value: value to be parsed to :class:`bool`. @@ -246,14 +246,16 @@ def convert_to_base_unit(value: Union[int, float], unit: str, base_unit: Optiona """Convert *value* as a *unit* of compute information or time to *base_unit*. :param value: value to be converted to the base unit. - :param unit: unit of *value*. Accepts these units (case sensitive) - * For space: ``B``, ``kB``, ``MB``, ``GB``, or ``TB``; - * For time: ``d``, ``h``, ``min``, ``s``, ``ms``, or ``us``. + :param unit: unit of *value*. Accepts these units (case sensitive): + + * For space: ``B``, ``kB``, ``MB``, ``GB``, or ``TB``; + * For time: ``d``, ``h``, ``min``, ``s``, ``ms``, or ``us``. :param base_unit: target unit in the conversion. May contain the target unit with an associated value, e.g - ``512MB``. Accepts these units (case sensitive) - * For space: ``B``, ``kB``, or ``MB``; - * For time: ``ms``, ``s``, or ``min``. + ``512MB``. Accepts these units (case sensitive): + + * For space: ``B``, ``kB``, or ``MB``; + * For time: ``ms``, ``s``, or ``min``. :returns: *value* in *unit* converted to *base_unit*. Returns ``None`` if *unit* or *base_unit* is invalid. @@ -403,7 +405,8 @@ def compare_values(vartype: str, unit: Optional[str], old_value: Any, new_value: """Check if *old_value* and *new_value* are equivalent after parsing them as *vartype*. :param vartpe: the target type to parse *old_value* and *new_value* before comparing them. Accepts any among of the - following (case sensitive) + following (case sensitive): + * ``bool``: parse values using :func:`parse_bool`; or * ``integer``: parse values using :func:`parse_int`; or * ``real``: parse values using :func:`parse_real`; or @@ -460,7 +463,7 @@ def compare_values(vartype: str, unit: Optional[str], old_value: Any, new_value: def _sleep(interval: Union[int, float]) -> None: - """Wrap :func:`time.sleep`. + """Wrap :func:`~time.sleep`. :param interval: Delay execution for a given number of seconds. The argument may be a floating point number for subsecond precision. @@ -549,6 +552,7 @@ class Retry(object): """Set next cycle delay. It will be the minimum value between: + * current delay with ``backoff``; or * ``max_delay``. """ @@ -562,10 +566,14 @@ class Retry(object): def ensure_deadline(self, timeout: float, raise_ex: Optional[Exception] = None) -> bool: """Calculates, sets, and checks the remaining deadline time. - :param timeout: if the *deadline* is smaller than the provided *timeout* value raise *raise_ex* exception - :param raise_ex: the exception object that will be raised if the *deadline* is smaller than provided *timeout* - :returns: `False` if *deadline* is smaller than a provided *timeout* and *raise_ex* isn't set. Otherwise `True` - :raises Exception: if calculated deadline is smaller than provided *timeout* + :param timeout: if the *deadline* is smaller than the provided *timeout* value raise *raise_ex* exception. + :param raise_ex: the exception object that will be raised if the *deadline* is smaller than provided *timeout*. + + :returns: ``False`` if *deadline* is smaller than a provided *timeout* and *raise_ex* isn't set. Otherwise + ``True``. + + :raises: + :class:`Exception`: *raise_ex* if calculated deadline is smaller than provided *timeout*. """ self.deadline = self.stoptime - time.time() if self.deadline < timeout: @@ -578,9 +586,10 @@ class Retry(object): """Call a function *func* with arguments ``*args`` and ``*kwargs`` in a loop. *func* will be called until one of the following conditions is met: - * It completes without throwing one of the configured ``retry_exceptions``; or - * ``max_retries`` is exceeded.; or - * ``deadline`` is exceeded. + + * It completes without throwing one of the configured ``retry_exceptions``; or + * ``max_retries`` is exceeded.; or + * ``deadline`` is exceeded. .. note:: * It will set loop stop time based on ``deadline`` attribute. @@ -589,9 +598,10 @@ class Retry(object): :param func: function to call. :param args: positional arguments to call *func* with. :params kwargs: keyword arguments to call *func* with. - :raises :class:`RetryFailedError` - * If ``max_tries`` is exceeded; or - * If ``deadline`` is exceeded. + :raises: + :class:`RetryFailedError`: + * If ``max_tries`` is exceeded; or + * If ``deadline`` is exceeded. """ self.reset() @@ -626,7 +636,8 @@ def polling_loop(timeout: Union[int, float], interval: Union[int, float] = 1) -> :param timeout: for how long (in seconds) from now it should keep returning values. :param interval: for how long to sleep before returning a new value. - :rtype: Iterator[:class:`int`] with current iteration counter, starting from ``0``. + + :yields: current iteration counter, starting from ``0``. """ start_time = time.time() iteration = 0 @@ -640,14 +651,16 @@ def polling_loop(timeout: Union[int, float], interval: Union[int, float] = 1) -> def split_host_port(value: str, default_port: Optional[int]) -> Tuple[str, int]: """Extract host(s) and port from *value*. - :param value: string from where host(s) and port will be extracted. Accepts either of these formats - * ``host:port``; or - * ``host1,host2,...,hostn:port``. + :param value: string from where host(s) and port will be extracted. Accepts either of these formats: + + * ``host:port``; or + * ``host1,host2,...,hostn:port``. Each ``host`` portion of *value* can be either: - * A FQDN; or - * An IPv4 address; or - * An IPv6 address, with or without square brackets. + + * A FQDN; or + * An IPv4 address; or + * An IPv6 address, with or without square brackets. :param default_port: if no port can be found in *param*, use *default_port* instead. @@ -682,18 +695,23 @@ def uri(proto: str, netloc: Union[List[str], Tuple[str, Union[int, str]], str], :param proto: the URI protocol. :param netloc: the URI host(s) and port. Can be specified in either way among + * A :class:`list` or :class:`tuple`. The second item should be a port, and the first item should be composed of hosts in either of these formats: + * ``host``; or. * ``host1,host2,...,hostn``. + * A :class:`str` in either of these formats: + * ``host:port``; or * ``host1,host2,...,hostn:port``. In all cases, each ``host`` portion of *netloc* can be either: - * An FQDN; or - * An IPv4 address; or - * An IPv6 address, with or without square brackets. + + * An FQDN; or + * An IPv4 address; or + * An IPv6 address, with or without square brackets. :param path: the URI path. :param user: the authenticating user, if any. @@ -711,10 +729,11 @@ def uri(proto: str, netloc: Union[List[str], Tuple[str, Union[int, str]], str], def iter_response_objects(response: HTTPResponse) -> Iterator[Dict[str, Any]]: - """Iterate over the chunks of a :class:`HTTPResponse` and yield each JSON document that is found along the way. + """Iterate over the chunks of a :class:`~urllib3.response.HTTPResponse` and yield each JSON document that is found. :param response: the HTTP response from which JSON documents will be retrieved. - :rtype: Iterator[:class:`dict`] with current JSON document. + + :yields: current JSON document. """ prev = '' decoder = JSONDecoder() @@ -743,33 +762,36 @@ def iter_response_objects(response: HTTPResponse) -> Iterator[Dict[str, Any]]: def cluster_as_json(cluster: 'Cluster', global_config: Optional['GlobalConfig'] = None) -> Dict[str, Any]: """Get a JSON representation of *cluster*. - :param cluster: the :class:`Cluster` object to be parsed as JSON. - :param global_config: optional :class:`GlobalConfig` object to check the cluster state. + :param cluster: the :class:`~patroni.dcs.Cluster` object to be parsed as JSON. + :param global_config: optional :class:`~patroni.config.GlobalConfig` object to check the cluster state. if not provided will be instantiated from the `Cluster.config`. :returns: JSON representation of *cluster*. These are the possible keys in the returning object depending on the available information in *cluster*: - * ``members``: list of members in the cluster. Each value is a :class:`dict` that may have the following keys: - * ``name``: the name of the host (unique in the cluster). The ``members`` list is sorted by this key; - * ``role``: ``leader``, ``standby_leader``, ``quorum_standby``, ``sync_standby``, or ``replica``; - * ``state``: ``stopping``, ``stopped``, ``stop failed``, ``crashed``, ``running``, ``starting``, - ``start failed``, ``restarting``, ``restart failed``, ``initializing new cluster``, ``initdb failed``, - ``running custom bootstrap script``, ``custom bootstrap failed``, or ``creating replica``; - * ``api_url``: REST API URL based on ``restapi->connect_address`` configuration; - * ``host``: PostgreSQL host based on ``postgresql->connect_address``; - * ``port``: PostgreSQL port based on ``postgresql->connect_address``; - * ``timeline``: PostgreSQL current timeline; - * ``pending_restart``: ``True`` if PostgreSQL is pending to be restarted; - * ``scheduled_restart``: scheduled restart timestamp, if any; - * ``tags``: any tags that were set for this member; - * ``lag``: replication lag, if applicable; - * ``pause``: ``True`` if cluster is in maintenance mode; - * ``scheduled_switchover``: if a switchover has been scheduled, then it contains this entry with these keys: - * ``at``: timestamp when switchover was scheduled to occur; - * ``from``: name of the member to be demoted; - * ``to``: name of the member to be promoted. + * ``members``: list of members in the cluster. Each value is a :class:`dict` that may have the following keys: + + * ``name``: the name of the host (unique in the cluster). The ``members`` list is sorted by this key; + * ``role``: ``leader``, ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``; + * ``state``: ``stopping``, ``stopped``, ``stop failed``, ``crashed``, ``running``, ``starting``, + ``start failed``, ``restarting``, ``restart failed``, ``initializing new cluster``, ``initdb failed``, + ``running custom bootstrap script``, ``custom bootstrap failed``, or ``creating replica``; + * ``api_url``: REST API URL based on ``restapi->connect_address`` configuration; + * ``host``: PostgreSQL host based on ``postgresql->connect_address``; + * ``port``: PostgreSQL port based on ``postgresql->connect_address``; + * ``timeline``: PostgreSQL current timeline; + * ``pending_restart``: ``True`` if PostgreSQL is pending to be restarted; + * ``scheduled_restart``: scheduled restart timestamp, if any; + * ``tags``: any tags that were set for this member; + * ``lag``: replication lag, if applicable; + + * ``pause``: ``True`` if cluster is in maintenance mode; + * ``scheduled_switchover``: if a switchover has been scheduled, then it contains this entry with these keys: + + * ``at``: timestamp when switchover was scheduled to occur; + * ``from``: name of the member to be demoted; + * ``to``: name of the member to be promoted. """ if not global_config: from patroni.config import get_global_config @@ -846,15 +868,18 @@ def validate_directory(d: str, msg: str = "{} {}") -> None: If the directory does not exist, :func:`validate_directory` will attempt to create it. :param d: the directory to be checked. - :param msg: a message to be thrown when raising :class:`PatroniException`, if any issue is faced. It must contain - 2 placeholders to be used by :func:`format`: - * The first placeholder will be replaced with path *d*; - * The second placeholder will be replaced with the error condition. + :param msg: a message to be thrown when raising :class:`~patroni.exceptions.PatroniException`, if any issue is + faced. It must contain 2 placeholders to be used by :func:`format`: - :raises :class:`PatroniException`: if any issue is observed while validating *d*. Can be thrown in these situations - * *d* did not exist, and :func:`validate_directory` was not able to create it; or - * *d* is an existing directory, but Patroni is not able to write to that directory; or - * *d* is an existing file, not a directory. + * The first placeholder will be replaced with path *d*; + * The second placeholder will be replaced with the error condition. + + :raises: + :class:`~patroni.exceptions.PatroniException`: if any issue is observed while validating *d*. Can be thrown if: + + * *d* did not exist, and :func:`validate_directory` was not able to create it; or + * *d* is an existing directory, but Patroni is not able to write to that directory; or + * *d* is an existing file, not a directory. """ if not os.path.exists(d): try: @@ -909,13 +934,22 @@ def keepalive_socket_options(timeout: int, idle: int, cnt: int = 3) -> Iterator[ :param idle: value for ``TCP_KEEPIDLE``. :param cnt: value for ``TCP_KEEPCNT``. - :rtype: Iterator[Tuple[:class:`int`, :class:`int`, :class:`int`]] of all keepalive related socket options to be - set. The first item in the tuple is the protocol, the second item is the option, and the third item is the - value to be used. The return values depend on the platform: - * ``Windows``: yield ``SO_KEEPALIVE``; - * ``Linux``: yield ``SO_KEEPALIVE``, ``TCP_USER_TIMEOUT``, ``TCP_KEEPIDLE`, ``TCP_KEEPINTVL``, and - ``TCP_KEEPCNT``; - * ``MacOS``: yield ``SO_KEEPALIVE``, ``TCP_KEEPIDLE`, ``TCP_KEEPINTVL``, and ``TCP_KEEPCNT`` + :yields: all keepalive related socket options to be set. The first item in the tuple is the protocol, the second + item is the option, and the third item is the value to be used. The return values depend on the platform: + + * ``Windows``: + * ``SO_KEEPALIVE``. + * ``Linux``: + * ``SO_KEEPALIVE``; + * ``TCP_USER_TIMEOUT``; + * ``TCP_KEEPIDLE``; + * ``TCP_KEEPINTVL``; + * ``TCP_KEEPCNT``. + * ``MacOS``: + * ``SO_KEEPALIVE``; + * ``TCP_KEEPIDLE``; + * ``TCP_KEEPINTVL``; + * ``TCP_KEEPCNT``. """ yield (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) @@ -953,7 +987,7 @@ def enable_keepalive(sock: socket.socket, timeout: int, idle: int, cnt: int = 3) :param idle: value for ``TCP_KEEPIDLE``. :param cnt: value for ``TCP_KEEPCNT``. - :returns: output of :func:`socket.ioctl` if we are on Windows, nothing otherwise. + :returns: output of :func:`~socket.ioctl` if we are on Windows, nothing otherwise. """ SIO_KEEPALIVE_VALS = getattr(socket, 'SIO_KEEPALIVE_VALS', None) if SIO_KEEPALIVE_VALS is not None: # Windows @@ -967,23 +1001,27 @@ def enable_keepalive(sock: socket.socket, timeout: int, idle: int, cnt: int = 3) def unquote(string: str) -> str: """Unquote a fully quoted *string*. + :param string: The string to be checked for quoting. + + :returns: The string with quotes removed, if it is a fully quoted single string, or the original string if quoting + is not detected, or unquoting was not possible. + :Examples: A *string* with quotes will have those quotes removed + >>> unquote('"a quoted string"') 'a quoted string' A *string* with multiple quotes will be returned as is + >>> unquote('"a multi" "quoted string"') '"a multi" "quoted string"' So will a *string* with unbalanced quotes + >>> unquote('unbalanced "quoted string') 'unbalanced "quoted string' - - :param string: The string to be checked for quoting. - :returns: The string with quotes removed, if it is a fully quoted single string, - or the original string if quoting is not detected, or unquoting was not possible. """ try: ret = split(string) @@ -999,9 +1037,11 @@ def get_major_version(bin_dir: Optional[str] = None, bin_name: str = 'postgres') It is based on the output of ``postgres --version``. :param bin_dir: path to the PostgreSQL binaries directory. If ``None`` or an empty string, it will use the first - *bin_name* binary that is found by the subprocess in the ``PATH``. + *bin_name* binary that is found by the subprocess in the ``PATH``. :param bin_name: name of the postgres binary to call (``postgres`` by default) + :returns: the PostgreSQL major version. + :raises: :exc:`~patroni.exceptions.PatroniException`: if the postgres binary call failed due to :exc:`OSError`. diff --git a/patroni/validator.py b/patroni/validator.py index 2f70e334..c99b3d32 100644 --- a/patroni/validator.py +++ b/patroni/validator.py @@ -3,7 +3,7 @@ This module contains facilities for validating configuration of Patroni processes. -:var schema: configuration schema of the daemon launched by `patroni` command. +:var schema: configuration schema of the daemon launched by ``patroni`` command. """ import os import shutil @@ -22,6 +22,7 @@ def data_directory_empty(data_dir: str) -> bool: """Check if PostgreSQL data directory is empty. :param data_dir: path to the PostgreSQL data directory to be checked. + :returns: ``True`` if the data directory is empty. """ if os.path.isfile(os.path.join(data_dir, "global", "pg_control")): @@ -32,12 +33,14 @@ def data_directory_empty(data_dir: str) -> bool: def validate_connect_address(address: str) -> bool: """Check if options related to connection address were properly configured. - :param address: address to be validated in the format - ``host:ip``. + :param address: address to be validated in the format ``host:ip``. + :returns: ``True`` if the address is valid. - :raises :class:`patroni.exceptions.ConfigParseError`: - * If the address is not in the expected format; or - * If the host is set to not allowed values (``127.0.0.1``, ``0.0.0.0``, ``*``, ``::1``, or ``localhost``). + + :raises: + :class:`~patroni.exceptions.ConfigParseError`: + * If the address is not in the expected format; or + * If the host is set to not allowed values (``127.0.0.1``, ``0.0.0.0``, ``*``, ``::1``, or ``localhost``). """ try: host, _ = split_host_port(address, 1) @@ -51,20 +54,25 @@ def validate_connect_address(address: str) -> bool: def validate_host_port(host_port: str, listen: bool = False, multiple_hosts: bool = False) -> bool: """Check if host(s) and port are valid and available for usage. - :param host_port: the host(s) and port to be validated. It can be in either of these formats + :param host_port: the host(s) and port to be validated. It can be in either of these formats: + * ``host:ip``, if *multiple_hosts* is ``False``; or * ``host_1,host_2,...,host_n:port``, if *multiple_hosts* is ``True``. :param listen: if the address is expected to be available for binding. ``False`` means it expects to connect to that address, and ``True`` that it expects to bind to that address. :param multiple_hosts: if *host_port* can contain multiple hosts. + :returns: ``True`` if the host(s) and port are valid. - :raises: :class:`patroni.exceptions.ConfigParserError`: - * If the *host_port* is not in the expected format; or - * If ``*`` was specified along with more hosts in *host_port*; or - * If we are expecting to bind to an address that is already in use; or - * If we are not able to connect to an address that we are expecting to do so; or - * If :class:`socket.gaierror` is thrown by socket module when attempting to connect to the given address(es). + + :raises: + :class:`~patroni.exceptions.ConfigParseError`: + * If the *host_port* is not in the expected format; or + * If ``*`` was specified along with more hosts in *host_port*; or + * If we are expecting to bind to an address that is already in use; or + * If we are not able to connect to an address that we are expecting to do so; or + * If :class:`~socket.gaierror` is thrown by socket module when attempting to connect to the given + address(es). """ try: hosts, port = split_host_port(host_port, 1) @@ -104,6 +112,7 @@ def validate_host_port_list(value: List[str]) -> bool: Call :func:`validate_host_port` with each item in *value*. :param value: list of host(s) and port items to be validated. + :returns: ``True`` if all items are valid. """ assert all([validate_host_port(v) for v in value]), "didn't pass the validation" @@ -116,6 +125,7 @@ def comma_separated_host_port(string: str) -> bool: Call :func:`validate_host_port_list` with a list represented by the CSV *string*. :param string: comma-separated list of host and port items. + :returns: ``True`` if all items in the CSV string are valid. """ return validate_host_port_list([s.strip() for s in string.split(",")]) @@ -127,7 +137,7 @@ def validate_host_port_listen(host_port: str) -> bool: Call :func:`validate_host_port` with *listen* set to ``True``. :param host_port: the host and port to be validated. Must be in the format - `host:ip`. + ``host:ip``. :returns: ``True`` if the host and port are valid and available for binding. """ @@ -140,8 +150,9 @@ def validate_host_port_listen_multiple_hosts(host_port: str) -> bool: Call :func:`validate_host_port` with both *listen* and *multiple_hosts* set to ``True``. :param host_port: the host(s) and port to be validated. It can be in either of these formats - * `host:ip`; or - * `host_1,host_2,...,host_n:port` + + * ``host:ip``; or + * ``host_1,host_2,...,host_n:port`` :returns: ``True`` if the host(s) and port are valid and available for binding. """ @@ -152,8 +163,11 @@ def is_ipv4_address(ip: str) -> bool: """Check if *ip* is a valid IPv4 address. :param ip: the IP to be checked. + :returns: ``True`` if the IP is an IPv4 address. - :raises :class:`patroni.exceptions.ConfigParserError`: if *ip* is not a valid IPv4 address. + + :raises: + :class:`~patroni.exceptions.ConfigParseError`: if *ip* is not a valid IPv4 address. """ try: socket.inet_aton(ip) @@ -166,8 +180,11 @@ def is_ipv6_address(ip: str) -> bool: """Check if *ip* is a valid IPv6 address. :param ip: the IP to be checked. + :returns: ``True`` if the IP is an IPv6 address. - :raises :class:`patroni.exceptions.ConfigParserError`: if *ip* is not a valid IPv6 address. + + :raises: + :class:`~patroni.exceptions.ConfigParseError`: if *ip* is not a valid IPv6 address. """ try: socket.inet_pton(socket.AF_INET6, ip) @@ -196,14 +213,17 @@ def validate_data_dir(data_dir: str) -> bool: * Point to a non-empty directory that seems to contain a valid PostgreSQL data directory. :param data_dir: the value of ``postgresql.data_dir`` configuration option. + :returns: ``True`` if the PostgreSQL data directory is valid. - :raises :class:`patroni.exceptions.ConfigParserError`: - * If no *data_dir* was given; or - * If *data_dir* is a file and not a directory; or - * If *data_dir* is a non-empty directory and: - * ``PG_VERSION`` file is not available in the directory - * ``pg_wal``/``pg_xlog`` is not available in the directory - * ``PG_VERSION`` content does not match the major version reported by ``postgres --version`` + + :raises: + :class:`~patroni.exceptions.ConfigParseError`: + * If no *data_dir* was given; or + * If *data_dir* is a file and not a directory; or + * If *data_dir* is a non-empty directory and: + * ``PG_VERSION`` file is not available in the directory + * ``pg_wal``/``pg_xlog`` is not available in the directory + * ``PG_VERSION`` content does not match the major version reported by ``postgres --version`` """ if not data_dir: raise ConfigParseError("is an empty string") @@ -244,11 +264,12 @@ def validate_binary_name(bin_name: str) -> bool: :returns: ``True`` if the conditions are true - :raises :class:`patroni.exceptions.ConfigParserError`: if: - * *bin_name* is not set; or - * the path join of the ``postgresql.bin_dir`` plus *bin_name* does not exist; or - * the path join as above is not executable; or - * the *bin_name* cannot be found in the system PATH + :raises: + :class:`~patroni.exceptions.ConfigParseError` if: + * *bin_name* is not set; or + * the path join of the ``postgresql.bin_dir`` plus *bin_name* does not exist; or + * the path join as above is not executable; or + * the *bin_name* cannot be found in the system PATH """ if not bin_name: @@ -275,7 +296,7 @@ class Result(object): .. note:: - ``error`` attribute is only set if ``status`` is failed. + ``error`` attribute is only set if *status* is failed. :param status: if the validation succeeded. :param error: error message related to the validation that was performed, if the validation failed. @@ -322,8 +343,8 @@ class Case(object): "url": str, }) - That will check that ``host`` configuration, if given, is valid based on ``validate_host_port`` function, and - will also check that ``url`` configuration, if given, is a ``str`` instance. + That will check that ``host`` configuration, if given, is valid based on :func:`validate_host_port`, and will + also check that ``url`` configuration, if given, is a ``str`` instance. """ self._schema = schema @@ -349,7 +370,7 @@ class Or(object): The outer :class:`Or` is used to define that ``host`` and ``hosts`` are possible options in this scope. The inner :class`Or` in the ``hosts`` key value is used to define that ``hosts`` option is valid if either of - the functions ``comma_separated_host_port`` or ``validate_host_port`` succeed to validate it. + :func:`comma_separated_host_port` or :func:`validate_host_port` succeed to validate it. """ self.args = args @@ -391,12 +412,12 @@ class Directory(object): self.contains_executable = contains_executable def _check_executables(self, path: OptionalType[str] = None) -> Iterator[Result]: - """Check that all executables from contains_executable list exist within the given directory or within PATH. + """Check that all executables from contains_executable list exist within the given directory or within ``PATH``. :param path: optional path to the base directory against which executables will be validated. - If not provided, check within PATH. - :rtype: Iterator[:class:`Result`] objects with the error message containing the name of the executable, - if any check fails. + If not provided, check within ``PATH``. + + :yields: objects with the error message containing the name of the executable, if any check fails. """ for program in self.contains_executable or []: if not shutil.which(program, path=path): @@ -406,8 +427,9 @@ class Directory(object): """Check if the expected paths and executables can be found under *name* directory. :param name: path to the base directory against which paths and executables will be validated. - Check against PATH if name is not provided. - :rtype: Iterator[:class:`Result`] objects with the error message related to the failure, if any check fails. + Check against ``PATH`` if name is not provided. + + :yields: objects with the error message related to the failure, if any check fails. """ if not name: yield from self._check_executables() @@ -455,12 +477,13 @@ class Schema(object): be performed against each one of them. The validations will be performed whenever the :class:`Schema` object is called, or its :func:`validate` method is called. - :ivar validator: validator of the configuration schema. Can be any of these + :ivar validator: validator of the configuration schema. Can be any of these: + * :class:`str`: defines that a string value is required; or - * :class:`type`: any subclass of `type`, defines that a value of the given type is required; or - * `callable`: any callable object, defines that validation will follow the code defined in the callable - object. If the callable object contains an ``expected_type`` attribute, then it will check if the - configuration value is of the expected type before calling the code of the callable object; or + * :class:`type`: any subclass of :class:`type`, defines that a value of the given type is required; or + * ``callable``: any callable object, defines that validation will follow the code defined in the callable + object. If the callable object contains an ``expected_type`` attribute, then it will check if the + configuration value is of the expected type before calling the code of the callable object; or * :class:`list`: list representing one or more values in the configuration; or * :class:`dict`: dictionary representing the YAML configuration tree. """ @@ -477,11 +500,12 @@ class Schema(object): nodes, when it performs checks of the actual setting values. :param validator: validator of the configuration schema. Can be any of these: + * :class:`str`: defines that a string value is required; or * :class:`type`: any subclass of :class:`type`, defines that a value of the given type is required; or - * `callable`: Any callable object, defines that validation will follow the code defined in the callable - object. If the callable object contains an ``expected_type`` attribute, then it will check if the - configuration value is of the expected type before calling the code of the callable object; or + * ``callable``: Any callable object, defines that validation will follow the code defined in the callable + object. If the callable object contains an ``expected_type`` attribute, then it will check if the + configuration value is of the expected type before calling the code of the callable object; or * :class:`list`: list representing it expects to contain one or more values in the configuration; or * :class:`dict`: dictionary representing the YAML configuration tree. @@ -489,18 +513,22 @@ class Schema(object): to stop. If *validator* is a :class:`dict`, then you should follow these rules: + * For the keys it can be either: + * A :class:`str` instance. It will be the name of the configuration option; or * An :class:`Optional` instance. The ``name`` attribute of that object will be the name of the - configuration option, and that class makes this configuration option as optional to the - user, allowing it to not be specified in the YAML; or + configuration option, and that class makes this configuration option as optional to the + user, allowing it to not be specified in the YAML; or * An :class:`Or` instance. The ``args`` attribute of that object will contain a tuple of configuration option names. At least one of them should be specified by the user in the YAML; + * For the values it can be either: + * A new :class:`dict` instance. It will represent a new level in the YAML configuration tree; or * A :class:`Case` instance. This is required if the key of this value is an :class:`Or` instance, - and the :class:`Case` instance is used to map each of the ``args`` in :class:`Or` to their - corresponding base validator in :class:`Case`; or + and the :class:`Case` instance is used to map each of the ``args`` in :class:`Or` to their + corresponding base validator in :class:`Case`; or * An :class:`Or` instance with one or more base validators; or * A :class:`list` instance with a single item which is the base validator; or * A base validator. @@ -523,15 +551,16 @@ class Schema(object): }) This sample schema defines that your YAML configuration follows these rules: - * It must contain an ``application_name`` entry which value should be a :class:`str` instance; - * It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``; - * It must contain a ``bind.port`` entry which value should be an :class:`int` instance; - * It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances; - * It may optionally contain a ``data_directory`` entry, with a value which should be a string; - * It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a - :class:`bool` instance; - * It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float` - instance. + + * It must contain an ``application_name`` entry which value should be a :class:`str` instance; + * It must contain a ``bind.host`` entry which value should be valid as per function ``validate_host``; + * It must contain a ``bind.port`` entry which value should be an :class:`int` instance; + * It must contain a ``aliases`` entry which value should be a :class:`list` of :class:`str` instances; + * It may optionally contain a ``data_directory`` entry, with a value which should be a string; + * It must contain at least one of ``log_to_file`` or ``log_to_db``, with a value which should be a + :class:`bool` instance; + * It must contain a ``version`` entry which value should be either an :class:`int` or a :class:`float` + instance. """ self.validator = validator @@ -539,6 +568,7 @@ class Schema(object): """Perform validation of data using the rules defined in this schema. :param data: configuration to be validated against ``validator``. + :returns: list of errors identified while validating the *data*, if any. """ errors: List[str] = [] @@ -553,14 +583,15 @@ class Schema(object): It first checks that *data* argument type is compliant with the type of ``validator`` attribute. Additionally: - * If ``validator`` attribute is a callable object, calls it to validate *data* argument. Before doing so, if - `validator` contains an ``expected_type`` attribute, check if *data* argument is compliant with that - expected type. - * If ``validator`` attribute is an iterable object (:class:`dict`, :class:`list`, :class:`Directory` or - :class:`Or`), then it iterates over it to validate each of the corresponding entries in *data* argument. + * If ``validator`` attribute is a callable object, calls it to validate *data* argument. Before doing so, if + `validator` contains an ``expected_type`` attribute, check if *data* argument is compliant with that + expected type. + * If ``validator`` attribute is an iterable object (:class:`dict`, :class:`list`, :class:`Directory` or + :class:`Or`), then it iterates over it to validate each of the corresponding entries in *data* argument. :param data: configuration to be validated against ``validator``. - :rtype: Iterator[:class:`Result`] objects with the error message related to the failure, if any check fails. + + :yields: objects with the error message related to the failure, if any check fails. """ self.data = data @@ -601,7 +632,7 @@ class Schema(object): Only :class:`dict`, :class:`list`, :class:`Directory` and :class:`Or` objects are considered iterable objects. - :rtype: Iterator[:class:`Result`] objects with the error message related to the failure, if any check fails. + :yields: objects with the error message related to the failure, if any check fails. """ if isinstance(self.validator, dict): if not isinstance(self.data, dict): @@ -629,7 +660,7 @@ class Schema(object): def iter_dict(self) -> Iterator[Result]: """Iterate over a :class:`dict` based ``validator`` to validate the corresponding entries in ``data``. - :rtype: Iterator[:class:`Result`] objects with the error message related to the failure, if any check fails. + :yields: objects with the error message related to the failure, if any check fails. """ # One key in `validator` attribute (`key` variable) can be mapped to one or more keys in `data` attribute (`d` # variable), depending on the `key` type. @@ -652,12 +683,12 @@ class Schema(object): path=(d + ("." + v.path if v.path else "")), level=v.level, data=v.data) def iter_or(self) -> Iterator[Result]: - """Perform all validations defined in an `Or` object for a given configuration option. + """Perform all validations defined in an :class:`Or` object for a given configuration option. This method can be only called against leaf nodes in the configuration tree. :class:`Or` objects defined in the ``validator`` keys will be handled by :func:`iter_dict` method. - :rtype: Iterator[:class:`Result`] objects with the error message related to the failure, if any check fails. + :yields: objects with the error message related to the failure, if any check fails. """ results: List[Result] = [] for a in self.validator.args: @@ -683,7 +714,7 @@ class Schema(object): :param key: key from the ``validator`` attribute. - :rtype: Iterator[str], keys that should be used to access corresponding value in the ``data`` attribute. + :yields: keys that should be used to access corresponding value in the ``data`` attribute. """ # If the key was defined as a `str` object in `validator` attribute, then it is already the final key to access # the `data` dictionary. @@ -710,12 +741,11 @@ class Schema(object): def _get_type_name(python_type: Any) -> str: - """Get a user friendly name for a given Python type. + """Get a user-friendly name for a given Python type. :param python_type: Python type which user friendly name should be taken. - Returns: - User friendly name of the given Python type. + :returns: User friendly name of the given Python type. """ types: Dict[Any, str] = {str: 'a string', int: 'an integer', float: 'a number', bool: 'a boolean', list: 'an array', dict: 'a dictionary'} @@ -736,11 +766,11 @@ def assert_(condition: bool, message: str = "Wrong value") -> None: class IntValidator(object): """Validate an integer setting. - :cvar expected_type: the expect Python type for an integer setting (:class:`int`). + :cvar expected_type: the expected Python type for an integer setting (:class:`int`). :ivar min: minimum allowed value for the setting, if any. :ivar max: maximum allowed value for the setting, if any. - :ivar base_unit: the base unit to convert the value to before checking if it's within `min` and `max` range. - :ivar raise_assert: if an ``assert`` call should be performed regarding expected type and valid range. + :ivar base_unit: the base unit to convert the value to before checking if it's within *min* and *max* range. + :ivar raise_assert: if an ``assert`` test should be performed regarding expected type and valid range. """ expected_type = int @@ -752,7 +782,7 @@ class IntValidator(object): :param min: minimum allowed value for the setting, if any. :param max: maximum allowed value for the setting, if any. :param base_unit: the base unit to convert the value to before checking if it's within *min* and *max* range. - :param raise_assert: if an ``assert`` call should be performed regarding expected type and valid range. + :param raise_assert: if an ``assert`` test should be performed regarding expected type and valid range. """ self.min = min self.max = max @@ -763,8 +793,10 @@ class IntValidator(object): """Check if *value* is a valid integer and within the expected range. .. note:: - If ``raise_assert`` is ``True`` and *value* is not valid, then an ``AssertionError`` will be triggered. + If ``raise_assert`` is ``True`` and *value* is not valid, then an :class:`AssertionError` will be triggered. + :param value: value to be checked against the rules defined for this :class:`IntValidator` instance. + :returns: ``True`` if *value* is valid and within the expected range. """ value = parse_int(value, self.base_unit) or "" diff --git a/requirements.docs.txt b/requirements.docs.txt new file mode 100644 index 00000000..67493140 --- /dev/null +++ b/requirements.docs.txt @@ -0,0 +1,5 @@ +sphinx>=4 +sphinx_rtd_theme +sphinxcontrib-apidoc +sphinx-github-style +pyyaml diff --git a/tox.ini b/tox.ini index 71104357..b145f483 100644 --- a/tox.ini +++ b/tox.ini @@ -77,6 +77,7 @@ platform = {[common]platforms} allowlist_externals = rm + true {env:OPEN_CMD} [testenv:dep] @@ -174,24 +175,52 @@ platform = {[common]platforms} [testenv:docs-{lin,mac,win}] -description = Build Sphinx documentation +description = Build Sphinx documentation in HTML format labels: docs deps = - sphinx>=4 - sphinx_rtd_theme + -r requirements.docs.txt + -r requirements.txt + psycopg[binary] + psycopg2-binary commands = sphinx-build \ -d "{envtmpdir}{/}doctree" docs "{toxworkdir}{/}docs_out" \ --color -b html \ + -T -E -W --keep-going \ {posargs} commands_post = - {tty:{env:OPEN_CMD} "{toxworkdir}{/}docs_out{/}index.html":true:} allowlist_externals = + true {env:OPEN_CMD} platform = {[common]platforms} +[testenv:pdf-{lin,mac,win}] +description = Build Sphinx documentation in PDF format +labels: + docs +deps = + -r requirements.docs.txt + -r requirements.txt + psycopg[binary] + psycopg2-binary +commands = + python -m sphinx -T -E -b latex -d _build/doctrees -D language=en . pdf + - latexmk -r pdf/latexmkrc -cd -C pdf/Patroni.tex + latexmk -r pdf/latexmkrc -cd -pdf -f -dvi- -ps- -jobname=Patroni -interaction=nonstopmode pdf/Patroni.tex + +commands_post = + - {tty:{env:OPEN_CMD} "pdf{/}Patroni.pdf":true:} +allowlist_externals = + true + latexmk + {env:OPEN_CMD} +platform = + {[common]platforms} +change_dir = docs + [flake8] max-line-length = 120 ignore = D401,W503