Skip to content

refactor(config): validated per-canton YAML for cantonal geoservice config - #238

Open
monodo wants to merge 4 commits into
mainfrom
refactor/cantons-config-yaml
Open

monodo wants to merge 4 commits into
mainfrom
refactor/cantons-config-yaml

Conversation

@monodo

@monodo monodo commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replaces the ~1200-line hand-maintained Python dict (cantons_configuration/cantons.py) with one declarative YAML file per canton (cantons_configuration/data/<CODE>.yaml), validated by a hardened Pydantic schema that is enforced in CI so a bad config can never reach production.

Why the dict was a problem

  • No validation — a typo (bad field name, missing URL, wrong harmonized value) failed silently at runtime against a live WMS instead of at load time.
  • No type safety — consumed as raw dict end to end.
  • Config mixed with code — editing cantonal data meant editing Python and redeploying; non-developers couldn't safely touch it.

What changed

  • schema.py — typed CantonConfig / Layer / PropertyValue / HarmonyMapEntry models with extra='forbid' (so typos are rejected), reusing the existing GroundSuitability enum. Lenient exactly where the real data requires it: optional desc, layer-level target_harmonized_value (ZH zones) and ESRI id (GE/FR), empty-string URLs, and 3-to-5-element ground_control_point rows with mixed int/float/str.
  • loader.py — loads and validates all YAML at import time (fail fast), aggregates every problem into one clear ConfigError, caches with lru_cache, and resolves paths package-relative so it works identically under uvicorn, pytest and AWS Lambda (Mangum). Validates on cold start, costs nothing on warm invocations.
  • data/*.yaml — 26 per-canton files. cantons.py is now a thin shim re-exporting CANTONS from the loader, so every existing consumer keeps receiving the exact same legacy-shaped dict.
  • Latent bug fixed — SO and BS carried name: 'NE' (copy-paste); corrected to match their code. The loader enforces name == filename going forward.
  • Adds pyyaml runtime dependency (uv.lock updated).

Hardening / CI (prevents app-breaking config)

  • tests/test_configuration_structure.py rewritten — 88 tests: every file loads + validates, every layer is usable, ESRI layers have an id, names match filenames, and the fail-fast paths (missing dir, empty dir, invalid YAML, unknown key, name/filename mismatch, bad GCP) all raise.
  • New .github/workflows/validate_config.yml — on every push/PR: a fast standalone load_cantons() check plus the config test suite. A malformed YAML fails this check and blocks the merge/release before any deploy.

No API contract change

GET /v1/cantons, /v1/cantons/{code}, /v1/avalaible-cantons and the drill-category canton_config payload are unchanged — verified byte-for-byte JSON-equal between the old dict and the YAML-loaded output (modulo the intended SO/BS name fix).

Testing

  • 131 tests pass (43 pre-existing + 88 new config tests); the full original suite is green.
  • ruff check and ruff format --check clean.
  • App and Lambda handler import cleanly; Dockerfile.lambda already copies src/drillapi so the YAML ships in the image.

Follow-ups (not in this PR)

  • The legacy harmonyMap / loopLayers keys (LU only) are currently unused by any code path; preserved here for lossless migration. Worth a separate decision on whether to remove them.
  • The NE/SO/BS entries remain inactive placeholder stubs with TEST layers; left as-is intentionally.

Replaces the ~1200-line hand-maintained Python dict in cantons_configuration/cantons.py with one declarative YAML file per canton (data/<CODE>.yaml), validated by a hardened Pydantic schema.

Why: the dict was untyped, unvalidated, and mixed config data with code. A typo (bad field name, missing URL, wrong harmonized value) failed silently at runtime against a live WMS. Config changes required editing Python and redeploying.

What:
- schema.py: typed CantonConfig/Layer/PropertyValue/HarmonyMapEntry models with extra='forbid' (catches typos), reusing the existing GroundSuitability enum. Lenient exactly where the real data needs it (optional desc, layer-level target_harmonized_value and ESRI id, empty-string URLs, 3-5 element ground_control_point rows).
- loader.py: loads + validates all YAML at import time (fail fast), aggregates every error into one clear ConfigError, caches results, resolves paths package-relative (works under uvicorn, pytest and Lambda/Mangum).
- data/*.yaml: 26 per-canton files. cantons.py is now a thin shim re-exporting CANTONS from the loader, so every existing consumer keeps receiving the exact same legacy-shaped dict - verified byte-for-byte JSON-equal to the old dict.
- Fixed a latent bug: SO and BS carried name='NE' (copy-paste); now corrected to match their code.
- Hardening: new tests/test_configuration_structure.py (88 tests) validates every file and the fail-fast paths; new .github/workflows/validate_config.yml validates the config on every push/PR so a broken YAML fails CI before deploy.
- Adds pyyaml runtime dependency.

No API contract change: /v1/cantons, /v1/cantons/{code}, /v1/avalaible-cantons and the drill-category canton_config payload are unchanged. All 43 existing tests still pass (131 total with the new config tests).
@monodo
monodo requested a review from a team as a code owner October 8, 2026 14:26
monodo added 3 commits October 8, 2026 17:13
Add a README section describing the per-canton YAML config, startup/CI validation, and the editing rule (filename must match 'name'). Document the uvicorn --reload command that watches the YAML files for local config work, and add watchfiles to the dev extra so that command works out of the box.
Fribourg retired the old map.geo.fr.ch/Theme_environnement MapServer, so every FR drill-category query failed and fell back to harmonized value 4. The data moved to the new open-data endpoint maps.fr.ch/ags/rest/services/opendata.

Updates FR.yaml to the new service (Admissibilite_des_sondes_geothermiques__SGV_, layer id 0) and keys on the stable code field DA_SGV (SGVaut/SGVdem/SGVint) instead of the descriptive DA_SGV_DESC text. Verified end-to-end: the three FR ground control points now return the expected 1/2/3. Pre-existing issue, not caused by the YAML migration (the old dict pointed at the same dead URL).
The FR geoservice fix repointed the config to maps.fr.ch/ags/.../opendata, but two tests still mocked the old map.geo.fr.ch/Theme_environnement URL via respx, so the real request was 'not mocked' and the endpoint returned 500. Update both mocks (test_drill_suitability and test_geoservice_unavailable) to the new URL. The identify_fr.json fixture already carries the DA_SGV code field, so no fixture change needed. Full suite: 131 passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant