Generate API docs from code with sphinx autodoc (#2699)

Expanding on the addition of docstrings in code, this adds python module API docs to sphinx documentation.

A developer can preview what this might look like by running this locally:

```
tox -m docs
```

The option `-W` is added to the tox env so that warning messages are considered errors.

Adds doc generation using the above method to the test GitHub workflow to catch documentation problems on PRs.

Some docstrings have been reformatted and fixed to satisfy errors generated with the above setup.
This commit is contained in:
Matt Baker
2023-09-20 12:35:14 +02:00
committed by Alexander Kukushkin
parent dde2331160
commit e796198045
22 changed files with 846 additions and 497 deletions
+7 -177
View File
@@ -1,182 +1,12 @@
.. _contributing:
Contributing guidelines
=======================
Contributing
============
Wanna contribute to Patroni? Yay - here is how!
Resources and information for developers can be found in the pages below.
Chatting
--------
.. toctree::
:maxdepth: 2
Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
Running tests
-------------
Requirements for running behave tests:
1. PostgreSQL packages need to be installed.
2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
Install dependencies:
.. code-block:: bash
# You may want to use Virtualenv or specify pip3.
pip install -r requirements.txt
pip install -r requirements.dev.txt
After you have all dependencies installed, you can run the various test suites:
.. code-block:: bash
# You may want to use Virtualenv or specify python3.
# Run flake8 to check syntax and formatting:
python setup.py flake8
# Run the pytest suite in tests/:
python setup.py test
# Run the behave (https://behave.readthedocs.io/en/latest/) test suite in features/;
# modify DCS as desired (raft has no dependencies so is the easiest to start with):
DCS=raft python -m behave
Testing with tox
----------------
To run tox tests you only need to install one dependency (other than Python)
.. code-block:: bash
pip install tox>=4
If you wish to run `behave` tests then you also need docker installed.
Tox configuration in `tox.ini` has "environments" to run the following tasks:
* lint: Python code lint with `flake8`
* test: unit tests for all available python interpreters with `pytest`,
generates XML reports or HTML reports if a TTY is detected
* dep: detect package dependency conflicts using `pipdeptree`
* type: static type checking with `pyright`
* black: code formatting with `black`
* docker-build: build docker image used for the `behave` env
* docker-cmd: run arbitrary command with the above image
* docker-behave-etcd: run tox for behave tests with above image
* py*behave: run behave with available python interpreters (without docker, although
this is what is called inside docker containers)
* docs: build docs with `sphinx`
Running tox
^^^^^^^^^^^
To run the default env list; dep, lint, test, and docs, just run:
.. code-block:: bash
tox
The `test` envs can be run with the label `test`:
.. code-block:: bash
tox -m test
The `behave` docker tests can be run with the label `behave`:
.. code-block:: bash
tox -m behave
Similarly, docs has the label `docs`.
All other envs can be run with their respective env names:
.. code-block:: bash
tox -e lint
tox -e py39-test-lin
It is also possible to select partial env lists using `factors`. For example, if you want to run
all envs for python 3.10:
.. code-block:: bash
tox -f py310
This is equivalent to running all the envs listed below:
.. code-block:: bash
$ tox -l -f py310
py310-test-lin
py310-test-mac
py310-test-win
py310-type-lin
py310-type-mac
py310-type-win
py310-behave-etcd-lin
py310-behave-etcd-win
py310-behave-etcd-mac
You can list all configured combinations of environments with tox (>=v4) like so
.. code-block:: bash
tox l
The envs `test` and `docs` will attempt to open the HTML output files
when the job completes, if tox is run with an active terminal. This
is intended to be for benefit of the developer running this env locally.
It will attempt to run `open` on a mac and `xdg-open` on Linux.
To use a different command set the env var `OPEN_CMD` to the name or path of
the command. If this step fails it will not fail the run overall.
If you want to disable this facility set the env var `OPEN_CMD` to the `:` no-op command.
.. code-block:: bash
OPEN_CMD=: tox -m docs
Behave tests
^^^^^^^^^^^^
Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 15 and then run all
behave tests. This can take quite a long time to run so you might want to limit the scope to a select version of
Postgres or to a specific feature set or steps.
To specify the version of postgres include the full name of the dependent image build env that you want and then the
behave env name. For instance if you want Postgres 15 use:
.. code-block:: bash
tox -e pg14-docker-build,pg14-docker-behave-etcd-lin
If on the other hand you want to test a specific feature you can pass positional arguments to behave. This will run
the watchdog behave feature test scenario with all versions of Postgres.
.. code-block:: bash
tox -m behave -- features/watchdog.feature
Of course you can combine the two.
Reporting issues
----------------
If you have a question about patroni or have a problem using it, please read the :ref:`README <readme>` before filing an issue.
Also double check with the current issues on our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Contributing a pull request
---------------------------
1) Submit a comment to the relevant issue or create a new issue describing your proposed change.
2) Do a fork, develop and test your code changes.
3) Include documentation
4) Submit a pull request.
You'll get feedback about your pull request as soon as possible.
Happy Patroni hacking ;-)
contributing_guidelines
Patroni API docs<modules/modules>
+98 -4
View File
@@ -20,10 +20,15 @@
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
from patroni.version import __version__
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
module_dir = os.path.abspath(os.path.join(project_root, 'patroni'))
excludes = ['tests', 'setup.py', 'conf']
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
@@ -33,11 +38,21 @@ from patroni.version import __version__
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.intersphinx',
extensions = [
'sphinx.ext.intersphinx',
'sphinx.ext.todo',
'sphinx.ext.mathjax',
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode']
# 'sphinx.ext.viewcode',
'sphinx_github_style', # Generate "View on GitHub" for source code
'sphinxcontrib.apidoc', # For generating module docs from code
'sphinx.ext.autodoc', # For generating module docs from docstrings
'sphinx.ext.napoleon', # For Google and Numpy formatted docstrings
]
apidoc_module_dir = module_dir
apidoc_output_dir = 'modules'
apidoc_excluded_paths = excludes
apidoc_separate_modules = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
@@ -107,6 +122,34 @@ if not on_rtd: # only import and set the theme if we're building docs locally
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Replace "source" links with "edit on GitHub" when using rtd theme
html_context = {
'display_github': True,
'github_user': 'zalando',
'github_repo': 'patroni',
'github_version': 'master',
'conf_py_path': '/docs/',
}
# sphinx-github-style options, https://sphinx-github-style.readthedocs.io/en/latest/index.html
# The name of the top-level package.
top_level = "patroni"
# The blob to link to on GitHub - any of "head", "last_tag", or "{blob}"
# linkcode_blob = 'head'
# The link to your GitHub repository formatted as https://github.com/user/repo
# If not provided, will attempt to create the link from the html_context dict
# linkcode_url = f"https://github.com/{html_context['github_user']}/" \
# f"{html_context['github_repo']}/{html_context['github_version']}"
# The text to use for the linkcode link
# linkcode_link_text: str = "View on GitHub"
# A linkcode_resolve() function to use for resolving the link target
# linkcode_resolve: types.FunctionType
# -- Options for HTMLHelp output ------------------------------------------
@@ -165,7 +208,6 @@ texinfo_documents = [
]
# -- Options for Epub output ----------------------------------------------
# Bibliographic Dublin Core info.
@@ -187,10 +229,57 @@ epub_copyright = copyright
epub_exclude_files = ['search.html']
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'python': ('https://docs.python.org/', None)}
# Remove these pages from index, references, toc trees, etc.
# If the builder is not 'html' then add the API docs modules index to pages to be removed.
exclude_from_builder = {
'latex': ['modules/modules'],
'epub': ['modules/modules'],
}
# Internal holding list, anything added here will always be excluded
_docs_to_remove = []
def builder_inited(app):
"""Run during Sphinx `builder-inited` phase.
Set a config value to builder name and add module docs to `docs_to_remove`.
"""
print(f'The builder is: {app.builder.name}')
app.add_config_value('builder', app.builder.name, 'env')
# Remove pages when builder matches any referenced in exclude_from_builder
if exclude_from_builder.get(app.builder.name):
_docs_to_remove.extend(exclude_from_builder[app.builder.name])
def env_get_outdated(app, env, added, changed, removed):
"""Run during Sphinx `env-get-outdated` phase.
Remove the items listed in `docs_to_remove` from known pages.
"""
added.difference_update(_docs_to_remove)
changed.difference_update(_docs_to_remove)
removed.update(_docs_to_remove)
return []
def doctree_read(app, doctree):
"""Run during Sphinx `doctree-read` phase.
Remove the items listed in `docs_to_remove` from the table of contents.
"""
from sphinx import addnodes
for toc_tree_node in doctree.traverse(addnodes.toctree):
for e in toc_tree_node['entries']:
ref = str(e[1])
if ref in _docs_to_remove:
toc_tree_node['entries'].remove(e)
# A possibility to have an own stylesheet, to add new rules or override existing ones
# For the latter case, the CSS specificity of the rules should be higher than the default ones
def setup(app):
@@ -198,3 +287,8 @@ def setup(app):
app.add_css_file('custom.css')
else:
app.add_stylesheet('custom.css')
# Run extra steps to remove module docs when running with a non-html builder
app.connect('builder-inited', builder_inited)
app.connect('env-get-outdated', env_get_outdated)
app.connect('doctree-read', doctree_read)
+182
View File
@@ -0,0 +1,182 @@
.. _contributing_guidelines:
Contributing guidelines
=======================
Wanna contribute to Patroni? Yay - here is how!
Chatting
--------
Just want to chat with other Patroni users? Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
Running tests
-------------
Requirements for running behave tests:
1. PostgreSQL packages need to be installed.
2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
Install dependencies:
.. code-block:: bash
# You may want to use Virtualenv or specify pip3.
pip install -r requirements.txt
pip install -r requirements.dev.txt
After you have all dependencies installed, you can run the various test suites:
.. code-block:: bash
# You may want to use Virtualenv or specify python3.
# Run flake8 to check syntax and formatting:
python setup.py flake8
# Run the pytest suite in tests/:
python setup.py test
# Run the behave (https://behave.readthedocs.io/en/latest/) test suite in features/;
# modify DCS as desired (raft has no dependencies so is the easiest to start with):
DCS=raft python -m behave
Testing with tox
----------------
To run tox tests you only need to install one dependency (other than Python)
.. code-block:: bash
pip install tox>=4
If you wish to run `behave` tests then you also need docker installed.
Tox configuration in `tox.ini` has "environments" to run the following tasks:
* lint: Python code lint with `flake8`
* test: unit tests for all available python interpreters with `pytest`,
generates XML reports or HTML reports if a TTY is detected
* dep: detect package dependency conflicts using `pipdeptree`
* type: static type checking with `pyright`
* black: code formatting with `black`
* docker-build: build docker image used for the `behave` env
* docker-cmd: run arbitrary command with the above image
* docker-behave-etcd: run tox for behave tests with above image
* py*behave: run behave with available python interpreters (without docker, although
this is what is called inside docker containers)
* docs: build docs with `sphinx`
Running tox
^^^^^^^^^^^
To run the default env list; dep, lint, test, and docs, just run:
.. code-block:: bash
tox
The `test` envs can be run with the label `test`:
.. code-block:: bash
tox -m test
The `behave` docker tests can be run with the label `behave`:
.. code-block:: bash
tox -m behave
Similarly, docs has the label `docs`.
All other envs can be run with their respective env names:
.. code-block:: bash
tox -e lint
tox -e py39-test-lin
It is also possible to select partial env lists using `factors`. For example, if you want to run
all envs for python 3.10:
.. code-block:: bash
tox -f py310
This is equivalent to running all the envs listed below:
.. code-block:: bash
$ tox -l -f py310
py310-test-lin
py310-test-mac
py310-test-win
py310-type-lin
py310-type-mac
py310-type-win
py310-behave-etcd-lin
py310-behave-etcd-win
py310-behave-etcd-mac
You can list all configured combinations of environments with tox (>=v4) like so
.. code-block:: bash
tox l
The envs `test` and `docs` will attempt to open the HTML output files
when the job completes, if tox is run with an active terminal. This
is intended to be for benefit of the developer running this env locally.
It will attempt to run `open` on a mac and `xdg-open` on Linux.
To use a different command set the env var `OPEN_CMD` to the name or path of
the command. If this step fails it will not fail the run overall.
If you want to disable this facility set the env var `OPEN_CMD` to the `:` no-op command.
.. code-block:: bash
OPEN_CMD=: tox -m docs
Behave tests
^^^^^^^^^^^^
Behave tests with `-m behave` will build docker images based on PG_MAJOR version 11 through 15 and then run all
behave tests. This can take quite a long time to run so you might want to limit the scope to a select version of
Postgres or to a specific feature set or steps.
To specify the version of postgres include the full name of the dependent image build env that you want and then the
behave env name. For instance if you want Postgres 15 use:
.. code-block:: bash
tox -e pg14-docker-build,pg14-docker-behave-etcd-lin
If on the other hand you want to test a specific feature you can pass positional arguments to behave. This will run
the watchdog behave feature test scenario with all versions of Postgres.
.. code-block:: bash
tox -m behave -- features/watchdog.feature
Of course you can combine the two.
Reporting issues
----------------
If you have a question about patroni or have a problem using it, please read the :ref:`README <readme>` before filing an issue.
Also double check with the current issues on our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
Contributing a pull request
---------------------------
1) Submit a comment to the relevant issue or create a new issue describing your proposed change.
2) Do a fork, develop and test your code changes.
3) Include documentation
4) Submit a pull request.
You'll get feedback about your pull request as soon as possible.
Happy Patroni hacking ;-)
+2 -2
View File
@@ -12,7 +12,7 @@ In both cases, it is important to be clear about the following concepts:
- You should run the odd number of etcd, ZooKeeper or Consul nodes: 3 or 5!
Synchronous Replication
----------------------------
-----------------------
To have a multi DC cluster that can automatically tolerate a zone drop, a minimum of 3 is required.
@@ -27,7 +27,7 @@ Regarding postgres, we must deploy at least 2 nodes, in different DC. Then you h
This enables sync replication and the primary node will choose one of the nodes as synchronous.
Asynchronous Replication
----------------------------------
------------------------
With only two data centers it would be better to have two independent etcd clusters and run Patroni :ref:`standby cluster <standby_cluster>` in the second data center. If the first site is down, you can MANUALLY promote the ``standby_cluster``.
+10 -3
View File
@@ -40,6 +40,13 @@ Currently supported PostgreSQL versions: 9.3 to 15.
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. ifconfig:: builder == 'html'
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. ifconfig:: builder != 'html'
* :ref:`genindex`
* :ref:`search`
+16 -9
View File
@@ -43,17 +43,24 @@ Bootstrap configuration
- **- data-checksums**: Must be enabled when pg_rewind is needed on 9.3.
- **- encoding: UTF8**: default encoding for new databases.
- **- locale: UTF8**: default locale for new databases.
- **users**: Some additional users which need to be created after initializing new cluster
- **admin**: the name of user
- **password**: (optional) password for the user
- **options**: list of options for CREATE USER statement
- **- createrole**
- **- createdb**
- **users**: Some additional users which need to be created after initializing new cluster, see :ref:`Bootstrap users configuration <bootstrap_users_configuration>` below.
- **post\_bootstrap** or **post\_init**: An additional script that will be executed after initializing the cluster. The script receives a connection string URL (with the cluster superuser as a user name). The PGPASSFILE variable is set to the location of pgpass file.
.. _bootstrap_users_configuration:
Bootstrap users configuration
=============================
Users which need to be created after initializing the cluster:
- **admin**: the name of user
- **password**: (optional) password for the user
- **options**: list of options for CREATE USER statement
- **- createrole**
- **- createdb**
.. _citus_settings:
Citus