ChangeLog

================================
echemdb-metadata-schema Change Log
================================

.. current developments

v0.8.3
====================

**Added:**

* Added ``pixi run test-generated-artifacts`` (part of ``pixi run test``), which regenerates the JSON Schemas and Pydantic models into a temporary directory and compares them with the committed files, catching stale generated artifacts locally instead of in CI.

**Changed:**

* Changed version bounds of `linkml` and `linkml-runtime` from `>=10,<11` to `>=11,<12`.
* Regenerated JSON Schemas with LinkML 1.11: URL-typed properties now carry ``"format": "uri"`` and abstract base classes (e.g. ``ControlledOperation``) are included in ``$defs``.

**Fixed:**

* Strip MkDocs-Material search front matter and ``data-search-exclude`` divs from ``gen-doc`` output (new in LinkML 1.11), which broke the Sphinx/MyST documentation build.



v0.8.2
====================

**Fixed:**

* Fixed validation of YAML metadata containing unquoted dates (e.g.
  ``date: 2021-07-09``): YAML files are now loaded with
  ``MetadataYamlLoader``, which keeps dates and timestamps as plain strings
  instead of ``datetime.date`` objects that fail validation of string-typed
  schema fields (`#123 <https://github.com/echemdb/metadata-schema/issues/123>`_).



v0.8.1
====================

**Fixed:**

* Fixed migration-registry tests that required the ``UNRELEASED`` placeholder to
  always be present. After a release, ``finalize_migrations`` stamps the
  placeholder to the concrete version, so the checks now accept either the
  placeholder (during development) or a valid released version, and no longer
  fail on the release commit or on ``main`` between breaking releases.

* Fixed a fragile ``validator.validate`` doctest that validated the current
  example against the frozen ``0.5.1`` schema; it now demonstrates (and expects)
  that a newer example does not validate against that older schema version.

* Fixed the release process to pass the previous release tag to
  ``finalize_migrations``, activating the guardrail that refuses a patch-only
  version bump while a breaking migration is still pending (breaking changes
  require at least a minor bump).



v0.8.0
====================

**Added:**

* Added a metadata migration engine (``mdstools.schema.migrate``) that upgrades
  echemdb metadata dicts, files and data packages from older to newer schema
  versions. Breaking-change steps are declared in ``mdstools.schema.migrations``;
  only breaking changes need a step (additive changes are backward-compatible).
* Added the ``mdstools update`` command (and ``pixi run update``) to report or
  apply migrations. Without ``--in-place`` it is a dry run; ``--in-place``
  rewrites the file in its original format, preserving YAML comments and layout.
* Added ``MetadataMigrator.validate`` to check a migrated document against its
  target schema, including the instrument-reference check.
* Added ``experimental.operationParameters`` describing how a measurement was
  operated: ``temperature`` (with active thermal control) and ``massTransport``
  with the forced-convection modes ``rotation`` (RDE/RRDE), ``flow``,
  ``ultrasound`` (sonoelectrochemistry) and ``stirring``. Each mode carries its
  defining quantity(ies) plus a ``control`` block referencing the controlling
  instrument by name.
* Added the ``ControlledQuantity``, ``Control``, ``MassTransport``,
  ``Rotation``, ``Flow``, ``Stirring`` and ``Ultrasound`` classes
  (``linkml/experimental/operation.yaml``).
* Added ``validate_instrument_references`` which checks that every
  ``operationParameters`` control block references an instrument that exists in
  the same ``experimental.instrumentation`` list. This cross-reference cannot
  be expressed in JSON Schema, so it is enforced during example validation.

**Changed:**

* The release process (``rever.xsh``) now stamps unreleased migration steps
  (``to_version="UNRELEASED"``) with the concrete release version via a new
  ``finalize_migrations`` activity.
* Added ``ruamel.yaml`` as a dependency for comment-preserving YAML round-trips.
* ``Instrumentation.name`` is now the class ``identifier``; it is the key that
  ``operationParameters`` control blocks reference.

**Removed:**

* Removed ``temperature`` from ``system.electrolyte``; the temperature and its
  control now live under ``experimental.operationParameters.temperature``.

**Fixed:**

* Made ``update_expected_schemas`` output ASCII so it no longer crashes on
  Windows consoles using the cp1252 code page.



v0.7.1
====================

**Fixed:**

* Fixed missing version bump in 0.7.0 release by correcting the rever workflow to use the ``dev`` pixi environment for schema generation.



v0.7.0
====================

**Added:**

* Added echemdbSchemaVersion to indicate the version of the echemdb metadata schema.

**Changed:**

* Changed ``dataDescription.dialect.delimiters`` to ``dataDescription.dialect.delimiter`` in the source data schema and examples to match the file-loading API.
* Added optional ``dataDescription.dialect.candidateDelimiters`` to provide delimiter candidates for dialect inference.

**Fixed:**

* Fixed CI generation checks to prevent generator drift by regenerating schemas and Pydantic models and failing when tracked generated artifacts differ.
* Fixed OS-dependent absolute paths in generated Pydantic models (``source_file``) by passing relative POSIX paths to ``gen-pydantic``.



v0.6.0
====================

**Added:**

* Added a documentation.
* Added several doctests to docstrings.
* Added ``to_latex()`` export to ``FlattenedMetadata`` and ``EnrichedFlattenedMetadata``.
* Added optional ``filepath`` parameter to ``to_markdown()`` and ``to_latex()`` for saving directly to file.
* Added Markdown (``.md``) and LaTeX (``.tex``) export to the ``flatten`` CLI command.
* Added ``validate()`` function and per-schema convenience wrappers
  (``validate_svgdigitizer()``, ``validate_autotag()``, etc.) that fetch
  JSON schemas directly from the metadata-schema GitHub repository and
  validate metadata dicts or YAML/JSON files against them.  A ``version``
  parameter selects the git tag or branch (default ``main``).

**Changed:**

* Replaced custom ``$ref`` resolution in the schema enricher with the ``jsonref`` library.
  The ``SchemaEnricher`` now resolves all ``$ref`` at load time via ``jsonref.replace_refs()``,
  removing ~95 lines of hand-written ref-walking code (``_resolve_ref``, ``_follow_refs``).
  External Frictionless URLs are mapped to locally cached schema files.
* Changed flattened metadata handling to remove empty nested structures.
* Changed release workflow: replaced local ``ghrelease`` rever activity (broken due to ``github3.py`` authentication issues) with a tag-triggered GitHub Actions workflow that creates GitHub releases from ``rever/LATEST`` changelog.

**Removed:**

* Removed ``twine``, ``build``, and ``setuptools`` from release dependencies (not needed since this project is not published to PyPI).

**Fixed:**

* Fixed schemas, to show ID, description, etc at the top of the file.
* Fixed schemas, allowing SMILES chemical identifier.



v0.5.1
====================

**Changed:**

* Changed backend for the creation and reading of excel files from `openpyxl` to `xlsxwriter`.

**Fixed:**

* Fixed referencing of datapackage resource in datapackage schemas.



v0.5.0
====================



v0.5.0
====================

**Added:**

* Added click-based CLI via ``mdstools.entrypoint`` with ``flatten`` and ``unflatten`` commands, registered as the ``mdstools`` entry point.
* Added ``mdstools/test/cli.py`` with ``invoke()`` helper for click CLI testing, following the unitpackage pattern.
* Added ``mdstools/schema/check_naming.py`` to validate naming conventions across all schema files (camelCase properties, PascalCase definitions, snake_case file names).
* Added ``check-naming`` pixi task, included in ``pixi run validate``.
* Added naming conventions section to ``schemas/README.md``.
* Added ``referencing`` as an explicit dependency.
* Added LinkML YAML schemas under ``linkml/`` as single source of truth for all metadata definitions.
* Added ``mdstools/schema/generate_from_linkml.py`` to generate JSON Schema and Pydantic models from LinkML.
* Added ``ensure_frictionless_schemas()`` to auto-download Frictionless Data Package schemas on demand into ``schemas/frictionless/`` (gitignored).
* Added Frictionless Data Package composition for package schemas (``echemdb_package.json``, ``svgdigitizer_package.json``): resource items use ``allOf`` with local ``frictionless/dataresource.json``.
* Added ``validate_package_schemas()`` in ``mdstools/schema/validate_examples.py`` for Python-based package schema validation with local Frictionless registry.
* Added auto-generated Pydantic models under ``mdstools/models/`` with permissive validation (``extra="allow"``, ``coerce_numbers_to_str=True``).
* Added ``validate_with_pydantic()`` in ``mdstools/schema/validator.py`` for Pydantic-based metadata validation.
* Added snapshot testing for generated schemas via ``mdstools/test/test_resolved_schemas.py``.
* Added ``bibdata`` attribute to ``SvgdigitizerSource`` in LinkML.
* Added pixi tasks: ``generate-schemas``, ``generate-models``, ``generate-all``.
* Added ``linkml`` and ``linkml-runtime`` as dependencies.

**Changed:**

* Consolidated test directory from ``mdstools/tests/`` into ``mdstools/test/``.
* Updated pixi tasks ``flatten`` and ``unflatten`` to use the new click-based CLI.
* Simplified ``SchemaEnricher._register_definitions`` to register under PascalCase name directly instead of lowercase aliasing.
* Simplified ``SchemaEnricher.enrich_row`` definition lookup from a 3-step fallback to a deterministic camelCase-to-PascalCase mapping.
* Migrated ``validate_metadata`` from deprecated ``jsonschema.RefResolver`` to ``referencing.Registry`` / ``referencing.Resource``.
* Changed list item identifiers in flattened metadata from letter-based (``a``, ``b``, ``c``, …) to incremental ``i<n>`` format (``i1``, ``i2``, ``i3``, …), removing the 26-item alphabet limit and improving readability.
* Changed ``Quantity.value`` range from ``string`` to ``float`` in LinkML.
* Changed ``Purity.value`` range from ``string`` to ``float`` in LinkML.
* Updated ``Quantity.unit`` and ``Uncertainty.unit`` descriptions to reference astropy string notation; dimensionless quantities use an empty string.
* ``SchemaEnricher`` updated to handle both ``$defs`` (LinkML) and ``definitions`` (legacy) JSON Schema formats.
* ``check_naming.py`` refactored to validate generated JSON schemas.
* All 6 JSON schemas generated from LinkML definitions.
* ``schemas/README.md`` updated with LinkML-based workflow and Frictionless documentation.
* ``validate-package-schemas`` pixi task switched from ``check-jsonschema`` CLI to Python-based validation.
* Fixed swapped ``value``/``unit`` in ``partialPressure`` quantities across example files (``autotag.yaml``, ``svgdigitizer.yaml``, ``system.yaml``).

**Removed:**

* Removed legacy argparse-based ``mdstools/cli.py``, replaced by ``mdstools/entrypoint.py``.
* Removed ``mdstools/schema/resolver.py`` legacy schema resolver (replaced by ``generate_from_linkml.py``).
* Removed ``schemas/schema_pieces/`` legacy YAML schema definitions (replaced by LinkML-generated JSON schemas).
* Removed ``schemas/resolved/`` empty legacy directory.
* Remove support for Python 3.10 in CI testing (minimum supported version is now Python 3.11).

**Fixed:**

* Fixed ``"title": "scan rate"`` (with space) in ``figure_description.json`` to ``"scanRate"`` (camelCase).
* Fixed ``svgdigitizer_resource`` definition name in ``svgdigitizer_package.json`` to ``SvgdigitizerResource`` (PascalCase).
* Fixed ``DeprecationWarning`` for ``jsonschema.RefResolver`` in ``validator.py``.



v0.4.0
====================

**Added:**

* Added the ``mdstools`` Python package with CLI, metadata flattening/unflattening,
  schema enrichment, schema resolution, and validation utilities.
* Added ``minimum_echemdb`` schema for a minimum set of metadata required by
  the echemdb database.
* Added ``source_data`` schema for source data files including ``data_description``
  schema piece (CSV dialect, field mapping, and field units).
* Added CI workflows for linting, testing, and example validation.
* Added resolved single-file schemas with all ``$ref`` references inlined for
  distribution and snapshot testing against expected baselines.
* Added YAML example files for ``minimum_echemdb`` and ``source_data`` schemas.

**Changed:**

* Updated ``pyproject.toml`` with ``mdstools`` package dependencies and pixi task
  definitions for development, testing, linting, and validation.
* Updated README with usage documentation for the CLI and Python API.



v0.3.3
====================

**Added:**

* Added `originalFilename` to source metadata.



v0.3.2
====================

**Added:**

* Added property `chemicalIdentfiers` to components for inclusion of, i.e., cas, InChi, or InChiKey identifiers.



v0.3.1
====================

**Added:**

* Added pixi configuration file `pixi.toml`, to simplify testing schemas.
* Added `uncertainty` to quantity, which contains keys such as `value`, `unit`, `comment`,.....

**Changed:**

* Changed `pH` into a quantity.



v0.3.0
====================

**Added:**

* Added JSON schema for validation of Data Packages created by [`svgdigitizer`](https://github.com/echemdb/svgdigitizer).

**Changed:**

* Changed metadata keys to use camelCase for consistency with JSON naming conventions.
* Changed draft version to draft-07.
* Changed referencing in schema which allows easier creation of a bundled schema.



v0.2.1
====================

**Added:**

* Added JSON schema for validation of echemdb Data Packages.



v0.2.0
====================

**Added:**

* Added electrochemical system property `atmosphere`, describing the atmosphere above the electrolyte (glove box, UHV, laboratory conditions, etc.).
* Added `electrolyte.type : solute`, denoting anything added to a solvent, which does not match with existing types.
* Added `supplier` and/or `manufacturer` to the source of components, instruments, materials, etc.

**Changed:**

* Changed `supplied purity` to `purity` and use one `purity` schema for all `purity` keys.

**Removed**

* Removed `supplied purity` in favor of `purity`.
*



v0.1.0
====================

First stable version of the metadata-schema.