feat(connector): STAC 1.0 connector - harvesting, transformation, scheduling, caching, and API layer

[Vishmayraj]

Summary

This PR introduces the istSOS Metadata Connector's STAC 1.0 implementation as a fully integrated package inside api/app/v1/connector/. It replaces the standalone microservice architecture evaluated during the bonding period after benchmarking revealed that sequential HTTP pagination across the SensorThings API was structurally unsuitable for production -- 42.4 seconds across 57 requests on the Fraunhofer air quality dataset (5,610 Things, 22,941 Datastreams).

The connector now runs inside the istSOS4 backend lifecycle, harvesting directly from Postgres via asyncpg, caching to Redis, and serving STAC 1.0 compliant responses through a FastAPI router registered on the existing application.

---

What's in this PR

Harvesting layer (harvester.py)
A single asyncpg JOIN query across Thing, Location, Datastream, ObservedProperty, and Sensor produces all metadata needed for both STAC and DCAT-AP transformation in one database round trip. On the Fraunhofer dataset, this executes in 383 ms. JSON/JSONB columns are decoded locally via _coerce_json() to avoid registering a global codec that would silently affect unrelated STA read paths.

Scheduling layer (scheduler.py)
APScheduler fires the harvest-transform-cache cycle at a configurable interval (HARVEST_INTERVAL_MINUTES in .env). A Postgres advisory lock, acquired before harvest and held for the full cycle, prevents duplicate execution across multiple Uvicorn workers. The advisory lock connection is kept separate from the harvest connection. STAC and DCAT cache writes are independent -- a future DCAT failure does not roll back the STAC write.

STAC transformation layer (stac_transformer.py)
Pure dict-based transformation following the STA-STAC-Mapping-Reference.md spec. pystac was evaluated and dropped -- its per-call catalog validation brought transformation time for the Fraunhofer dataset to nearly five minutes. The dict approach is sub-second and gives explicit control over STAC 1.0 link relations, datetime formatting, bbox derivation, and collection extent computation.

Caching layer (cache.py)
Flattens the transformer output into stac:catalog, stac:collection:{id}, and stac:item:{collection_id}:{id} Redis keys. Uses the existing synchronous Redis client at app.db.redis_db, following the sta2rest pattern. Stale keys are purged via SCAN before each write to prevent orphaned data.

API layer (api.py)
FastAPI router serving /connector, /connector/stac, and the collection and item endpoints. All endpoints return STAC 1.0 compliant responses, validated against Radiant Earth's STAC Browser. GET /connector/stac/collections on the Fraunhofer dataset returns ~11.3 MB of valid STAC data -- pagination and response size optimisation are out of scope for this PR and tracked as a post-midterm item.

---

Notes

• The FROST migration utility commit (feat: add temporary FROST migration utility for STAC development) will be reverted before the end of the development period. It is intentionally included in this branch for the duration of active development.
• DCAT-AP 3.0 implementation (dcat_transformer.py) begins week 7. The scheduler already handles the DCAT path and logs a warning each cycle until it is implemented -- STAC serving is unaffected.
• collections.json is untracked and not included in this PR.

---

Testing

End-to-end validation was performed against the Fraunhofer FROST public instance (5,610 Things, 22,941 Datastreams) using Radiant Earth's STAC Browser. Automated testing infrastructure is the week 6 priority ahead of the midterm evaluation.