Hla 1093 jsonschema docs

.copier-answers.yml

@@ -1,5 +1,5 @@
 # Changes here will be overwritten by Copier
-_commit: 5.0.2
+_commit: 5.1.0
 _src_path: gh:DiamondLightSource/python-copier-template
 author_email: martin.gaughran@diamond.ac.uk
 author_name: Martin Gaughran

.devcontainer/devcontainer.json

@@ -39,7 +39,10 @@
                 "redhat.vscode-yaml",
                 "ryanluker.vscode-coverage-gutters",
                 "charliermarsh.ruff",
-                "ms-azuretools.vscode-docker"
+                "ms-azuretools.vscode-docker",
+                // Embedded HTML preview for built Sphinx docs (right-click
+                // build/html/index.html -> "Show Preview")
+                "ms-vscode.live-server"
             ]
         }
     },

.github/CONTRIBUTING.md

@@ -24,4 +24,4 @@ It is recommended that developers use a [vscode devcontainer](https://code.visua
 
 This project was created using the [Diamond Light Source Copier Template](https://github.com/DiamondLightSource/python-copier-template) for Python projects.
 
-For more information on common tasks like setting up a developer environment, running the tests, and setting a pre-commit hook, see the template's [How-to guides](https://diamondlightsource.github.io/python-copier-template/5.0.2/how-to.html).
+For more information on common tasks like setting up a developer environment, running the tests, and setting a pre-commit hook, see the template's [How-to guides](https://diamondlightsource.github.io/python-copier-template/5.1.0/how-to.html).

.github/workflows/ci.yml

@@ -18,8 +18,8 @@ jobs:
   test:
     strategy:
       matrix:
-        runs-on: [ "ubuntu-latest" ] # can add windows-latest, macos-latest
-        python-version: [ "3.12", "3.13", "3.14" ]
+        runs-on: ["ubuntu-latest"] # can add windows-latest, macos-latest
+        python-version: ["3.12", "3.13", "3.14"]
       fail-fast: false
     uses: ./.github/workflows/_test.yml
     with:

README.md

@@ -12,8 +12,9 @@ access to a shared filesystem. We use the
 [Environment Modules](https://modules.readthedocs.io/en/latest/) package to make these
 applications available to the end user.
 
-Source          | <https://github.com/DiamondLightSource/deploy-tools>
+What            | Where
 :---:           | :---:
+Source          | <https://github.com/DiamondLightSource/deploy-tools>
 PyPI            | `pip install dls-deploy-tools`
 Docker          | `docker run ghcr.io/diamondlightsource/deploy-tools:latest`
 Documentation   | Work in progress <https://diamondlightsource.github.io/deploy-tools>
@@ -23,21 +24,28 @@ The demo_configuration folder in this repository can be passed as the config_fol
 the deploy-tools commands. The deployment_root needs to be a writeable location for all
 files to get deployed under.
 
+The examples below use the `deploy-tools` console script; `python -m deploy_tools` is
+equivalent if you prefer to invoke the module directly.
+
 ```
 deployment_root = /path/to/deployment/root
 config_folder = /path/to/config/folder
 schema_folder = /path/to/schema/folder
 
 # Generate the schema for configuration yaml files
-python -m deploy_tools schema $schema_folder
+deploy-tools schema $schema_folder
 
 # Validate the deployment configuration files, also ensuring that the required updates
 # are compatible with the previous deployments.
-python -m deploy_tools validate $deployment_root $config_folder
+deploy-tools validate $deployment_root $config_folder
 
 # Synchronise the deployment area with the configuration files. This will first run
 # validation as part of determining the required changes
-python -m deploy_tools sync $deployment_root $config_folder
+deploy-tools sync $deployment_root $config_folder
+
+# Compare the previous deployment snapshot against what is actually deployed in the
+# deployment area. CI/CD should run this before a deploy to confirm a healthy state.
+deploy-tools compare $deployment_root
 
 ```
 
@@ -55,15 +63,15 @@ bit different to the CLI commands; see `deploy-tools --help` for more informatio
 
 ## JSON Schema
 
-A set of JSON schema files are provided under `src/deploy_tools/models/schemas`. These are generated from the Pydantic models in the schemas parent directory.
+A set of JSON schema files are provided under `src/deploy_tools/models/schemas`. These are generated from the Pydantic models in `src/deploy_tools/models/` by the `deploy-tools schema` command (see `models/schema.py`).
 
-We strongly recommend that you provide a schema for configuration file validation. The relevant lines at the top of each release file are:
+We strongly recommend that you provide a schema for configuration file validation, by adding a `yaml-language-server` comment to the top of each configuration file. For production configuration, point this at the schema files hosted on GitHub:
 
-```# yaml-language-server: $schema=/workspaces/deploy-tools/src/deploy_tools/models/schemas/release.json```
+```# yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json```
 
-As the demo_configuration is used during development, we set it to use the locally generated schemas. Note that the 'Generate Schema' VSCode task will update the schemas according to any update of the code, but you need to trigger this manually and check the contents in.
+As the demo_configuration is used during development, we instead set it to use the locally generated schemas via an absolute workspace path (e.g. `/workspaces/deploy-tools/src/deploy_tools/models/schemas/release.json`). This dev-container-only path should not be used for production configuration.
 
-For any production configuration, you should set it to use schema files from GitHub.
+Note that the 'Generate Schema' VSCode task will update the schemas according to any update of the code, but you need to trigger this manually and check the contents in.
 
 ## CLI Commands, VSCode Tasks and Debug Configuration
 
@@ -100,7 +108,7 @@ See the Deployment Steps above for an overview of the primary stages of a deploy
 |Deployment         |The sum total of all Releases (deprecated or not) that are to be maintained in the Deployment Area                                                                                                                                                                                                          |
 |Module             |A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration                                                                       |
 |Release (noun)     |A Module, including version, alongside its lifecycle (i.e. deprecation) status                                                                                                                                                                                                                              |
-|Application        |Each Module can be configured with multiple Applications, each one providing one or more executables. As of writing, there are 2 types of Application: `Apptainer` and `Shell` (Bash script)                                                                                                                |
+|Application        |Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash)                                                                                                                |
 |Deployment Area    |The top-level location where all Modules are to be deployed. This is typically a shared filesystem location for general use by multiple people. Note that there are several subdirectories which are used by `deploy-tools` for different purposes                                                          |
 |(Area) Root        |Refers to the filesystem path at the root of the given Area.                                                                                                                                                                                                                                                |
 |Deployment Step    |Refers to one of the primary steps that make up the Deployment process. See the section 'Deployment Steps' above for a breakdown                                                                                                                                                                            |

docs/cli.md

@@ -0,0 +1,9 @@
+# Command-line interface
+
+`deploy-tools` is driven entirely through its command-line interface.
+
+```{typer} deploy_tools.__main__:app
+:prog: deploy-tools
+:make-sections:
+:show-nested:
+```

docs/conf.py

@@ -49,6 +49,12 @@
     "sphinx_design",
     # So we can write markdown files
     "myst_parser",
+    # For the generation of schema docs
+    "sphinx-jsonschema",
+    # Render Pydantic models cleanly (hides internals like model_config / FieldInfo)
+    "sphinxcontrib.autodoc_pydantic",
+    # Render the Typer command-line interface
+    "sphinxcontrib.typer",
 ]
 
 # So we can use the ::: syntax
@@ -71,20 +77,42 @@
     ("py:class", "'object'"),
     ("py:class", "'id'"),
     ("py:class", "typing_extensions.Literal"),
+    # Generic type variable used in save_and_load.load_from_yaml
+    ("py:class", "T"),
+    # PEP 695 type aliases in models.deployment cannot be resolved as classes
+    ("py:class", "deploy_tools.models.deployment.DefaultVersionsByName"),
+    ("py:class", "deploy_tools.models.deployment.ModuleVersionsByName"),
+    ("py:class", "deploy_tools.models.deployment.ReleasesByNameAndVersion"),
+    ("py:class", "deploy_tools.models.deployment.ReleasesByVersion"),
+    ("py:class", "deploy_tools.models.deployment.ModulesByName"),
+    ("py:class", "deploy_tools.models.apptainer_app.MountPoint"),
+    # Pydantic's FieldInfo leaks from the discriminated-union default in the
+    # models.module.Application type alias when rendered in signatures
+    ("py:class", "FieldInfo"),
 ]
 
-# Both the class’ and the __init__ method’s docstring are concatenated and
-# inserted into the main body of the autoclass directive
-autoclass_content = "both"
+# Use only the class docstring in the main body of the autoclass directive. The
+# __init__ docstring is excluded because Pydantic's autogenerated BaseModel.__init__
+# docstring references unresolvable targets (self, ValidationError) under nitpicky mode.
+autoclass_content = "class"
 
 # Order the members by the order they appear in the source code
 autodoc_member_order = "bysource"
 
 # Don't inherit docstrings from baseclasses
 autodoc_inherit_docstrings = False
 
-# Document only what is in __all__
-autosummary_ignore_module_all = False
+# Ignore module __all__ so every public member is documented. Internals are kept out
+# of the docs by marking them private with a leading underscore instead.
+autosummary_ignore_module_all = True
+
+# Set global options for sphinx-jsonschema
+jsonschema_options = {
+    "lift_description": True,
+    "lift_definitions": True,
+    "auto_target": True,
+    "auto_reference": True,
+}
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
@@ -109,7 +137,10 @@
 
 # This means you can link things like `str` and `asyncio` to the relevant
 # docs in the python documentation.
-intersphinx_mapping = {"python": ("https://docs.python.org/3/", None)}
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3/", None),
+    "pydantic": ("https://docs.pydantic.dev/latest/", None),
+}
 
 # A dictionary of graphviz graph attributes for inheritance diagrams.
 inheritance_graph_attrs = {"rankdir": "TB"}

docs/reference.md

@@ -6,7 +6,9 @@ Technical reference material including APIs and release notes.
 :maxdepth: 1
 :glob:
 
+CLI <cli>
 API <_api/deploy_tools>
+Schemas <schemas>
 genindex
 Release Notes <https://github.com/DiamondLightSource/deploy-tools/releases>
 ```

docs/schemas.md

@@ -0,0 +1,27 @@
+# JSON Schema files
+
+These JSON schema files are generated from the Pydantic models in `src/deploy_tools/models`.
+
+They are used by the YAML language server and other tooling to validate deployment configuration files.
+
+This section links to per-schema reference pages which render the schema and list the individual
+properties as tables.
+
+## Schema pages
+
+```{toctree}
+:maxdepth: 1
+
+schemas/deployment-settings
+schemas/release
+```
+
+## How the schemas are generated
+
+The `deploy_tools.models.schema.generate_schema` function writes these files from the corresponding Pydantic models.
+
+To regenerate them manually:
+
+```bash
+deploy-tools schema path/to/output/folder
+```

docs/schemas/deployment-settings.rst

@@ -0,0 +1,4 @@
+.. default-role:: literal
+
+.. jsonschema:: ../../src/deploy_tools/models/schemas/deployment-settings.json
+    :hide_key: /**/additionalProperties

docs/schemas/release.rst

@@ -0,0 +1,4 @@
+.. default-role:: literal
+
+.. jsonschema:: ../../src/deploy_tools/models/schemas/release.json
+    :hide_key: /**/pattern,/**/additionalProperties

pyproject.toml

@@ -9,6 +9,7 @@ classifiers = [
     "License :: OSI Approved :: Apache Software License",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
 ]
 description = "A set of tools used for deploying applications to a shared filesystem."
 dependencies = [
@@ -26,6 +27,7 @@ requires-python = ">=3.12"
 
 [dependency-groups]
 dev = [
+    "autodoc-pydantic",
     "copier",
     "myst-parser",
     "pre-commit",
@@ -36,7 +38,9 @@ dev = [
     "ruff",
     "sphinx-autobuild",
     "sphinx-copybutton",
+    "sphinx-jsonschema",
     "sphinx-design",
+    "sphinxcontrib-typer",
     "tox-uv",
     "types-mock",
 ]
@@ -55,7 +59,7 @@ name = "Martin Gaughran"
 version_file = "src/deploy_tools/_version.py"
 
 [tool.setuptools.package-data]
-deploy_tools = ['templates', 'models/schema/*']
+deploy_tools = ['templates', 'models/schemas/*']
 
 [tool.pyright]
 exclude = ["src/deploy_tools/_version.py"]
@@ -81,6 +85,9 @@ data_file = "/tmp/deploy_tools.coverage"
 # Tests are run from installed location, map back to the src directory
 source = ["src", "**/site-packages/"]
 
+[tool.uv]
+exclude-newer = "3 days"
+
 [tool.tox]
 skipsdist = true
 # envs to runs automatically with tox -p

src/deploy_tools/__main__.py

@@ -13,7 +13,7 @@
 __all__ = ["main"]
 
 
-def verbose_callback(value: int) -> None:
+def _verbose_callback(value: int) -> None:
     match value:
         case 2:
             level = logging.DEBUG
@@ -103,7 +103,7 @@ def verbose_callback(value: int) -> None:
         max=2,
         clamp=True,
         show_default="WARNING",
-        callback=verbose_callback,
+        callback=_verbose_callback,
     ),
 ]
 
@@ -181,7 +181,7 @@ def schema(
     generate_schema(output_path)
 
 
-def version_callback(value: bool) -> None:
+def _version_callback(value: bool) -> None:
     if value:
         typer.echo(__version__)
         raise typer.Exit()
@@ -195,7 +195,7 @@ def common(
             "--version",
             "-V",
             help="Show program's version number and exit.",
-            callback=version_callback,
+            callback=_version_callback,
         ),
     ] = None,
 ) -> None:

src/deploy_tools/app_builder.py

@@ -17,7 +17,7 @@
 
 
 class AppBuilderError(Exception):
-    pass
+    """Raised when building an application's files fails."""
 
 
 class AppBuilder:
@@ -28,6 +28,10 @@ def __init__(self, templater: Templater, build_layout: ModuleBuildLayout) -> Non
         self._build_layout = build_layout
 
     def create_application_files(self, app: Application, module: Module):
+        """Create the entrypoint and supporting files for an application.
+
+        The files produced depend on the application type (Apptainer, shell or binary).
+        """
         match app:
             case ApptainerApp():
                 self._create_apptainer_files(app, module)

src/deploy_tools/apptainer.py

@@ -3,14 +3,15 @@
 
 
 class ApptainerError(Exception):
-    pass
+    """Raised when building an Apptainer SIF file fails."""
 
 
 def create_sif_file(
     output_path: Path,
     container_url: str,
     create_parents: bool = False,
 ) -> None:
+    """Build an Apptainer SIF file at the given path from a container image URL."""
     if create_parents:
         output_path.parent.mkdir(parents=True, exist_ok=True)
 

src/deploy_tools/build.py

@@ -7,6 +7,7 @@
 
 
 def clean_build_area(layout: Layout) -> None:
+    """Remove the build area directory if it exists."""
     build_path = layout.build_layout.build_root
 
     if build_path.exists():

src/deploy_tools/compare.py

@@ -24,7 +24,7 @@
 
 
 class ComparisonError(Exception):
-    pass
+    """Raised when comparing the deployment area to its snapshot fails."""
 
 
 def compare_to_snapshot(
@@ -35,10 +35,10 @@ def compare_to_snapshot(
     This helps us to identify broken environment modules, or a failed deployment step.
     Note that this does not exclude the possibility of all types of issues.
 
-    The `use_ref` argument can be used to compare against a previous Deployment
+    The ``use_ref`` argument can be used to compare against a previous Deployment
     configuration. It is recommended to use a reference relative to HEAD, e.g. 'HEAD~1'.
 
-    The `from_scratch` argument checks that the deployment area is in a suitable state
+    The ``from_scratch`` argument checks that the deployment area is in a suitable state
     for a clean deployment into an empty directory. No snapshot is expected.
 
     Args: