diff --git a/AGENTS.md b/AGENTS.md index 90f84548e..24a0e2a01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -160,7 +160,7 @@ Full detail lives in [`docs/operations/DEPLOYMENT_RULES.md`](docs/operations/DEP 6. **stpa-freshness-check** — STPA source changed without regenerating artifacts. Fix: run `uv run python scripts/check_stpa_freshness.py`. 7. **pytest-logic** — address the failing test. Confirm no background `kubectl port-forward` tunnels are leaking live GKE state into local tests. 8. **Bandit / security-scan** — rotate credentials or address Bandit SAST (run in `pytest-logic`) or CVE findings (`security-scan.yml`); never suppress the scan. -9. **Documentation reference check (Gate G9, `lint` job)** — a document cites a file path, link target, or Python symbol that does not exist at HEAD. Fix: run `make docs-check` (or `uv run python scripts/check_doc_references.py --path `) and correct the reference — never delete the gate's scope to make it pass. The scope is `DOC_SCOPE` in that script, its only copy. `CHANGELOG.md` and `docs/partners/**` are outside the gate's scope by design. A document that must name a path removed from HEAD (for example, a closed POA&M entry describing a deleted file) pins it to a mainline commit where it existed, as `^:` (git's revision syntax), and G9 verifies the pin against history; the `lint` job checks out full history for this. Never add a removed path to the `KNOWN_DOCUMENTED_GAPS` waiver, which nothing verifies. Pins break if `main`'s history is rewritten: whoever rewrites it owns re-pinning every citation the rewrite orphans, and G9 names each one. +9. **Documentation reference check (Gate G9, `lint` job)** — a document cites a file path, link target, or Python symbol that does not exist at HEAD. Fix: run `make docs-check` (or `uv run python scripts/check_doc_references.py --path `) and correct the reference — never delete the gate's scope to make it pass. The scope is `DOC_SCOPE` in that script, its only copy. `CHANGELOG.md` and `docs/partners/**` are outside it by design, and most of the rest of `docs/` is outside it until `POAM-2026-224` widens it area by area: a document outside `DOC_SCOPE` is unverified, not verified. A document that must name a path removed from HEAD (for example, a closed POA&M entry describing a deleted file) pins it to a mainline commit where it existed, as `^:` (git's revision syntax), and G9 verifies the pin against history; the `lint` job checks out full history for this. There is no waiver: a reference to an artifact that never existed is corrected, not excused. A record withdrawn as untrue moves verbatim to `compliance/archive/retracted/` under a first-line `> **RETRACTED **` notice naming its POAM item; G9 verifies the notice and the POAM row and skips the preserved body. Pins break if `main`'s history is rewritten: whoever rewrites it owns re-pinning every citation the rewrite orphans, and G9 names each one. **Never suggest disabling or skipping a CI check as a fix.** diff --git a/compliance/sar/SAR_2026Q1.md b/compliance/archive/retracted/SAR_2026Q1.md similarity index 91% rename from compliance/sar/SAR_2026Q1.md rename to compliance/archive/retracted/SAR_2026Q1.md index d5de9af40..d5497a7b2 100644 --- a/compliance/sar/SAR_2026Q1.md +++ b/compliance/archive/retracted/SAR_2026Q1.md @@ -1,3 +1,13 @@ +> **RETRACTED 2026-11-15** — Withdrawn in full under `POAM-2026-224`. This report records a security assessment that did not take place. It is preserved below verbatim as last committed at `a2a338b8`; nothing in it is a current claim. +> +> - **No assessment artifacts.** It describes interviews with system administrators, AI model operators, and compliance engineers, and tests including Lula execution, TLS endpoint tests, HMAC bypass scenarios, pip-audit, Trivy, and red-team exercises. No interview record or test output exists in the repository, and the assessor is not identified. +> - **It cites a plan at a path that never existed.** Its related Security Assessment Plan is cited at `docs/compliance/us_fed/SECURITY_ASSESSMENT_PLAN.md`, which exists in no commit. The draft plan it first cited (`00442dec^:docs/security/SECURITY_ASSESSMENT_PLAN.md`) was deleted on 2026-07-30 (`00442dec`), and `a2a338b8` (2026-09-16) re-pointed the citation to the path above instead of removing it. No record exists of the formal assessment that plan scheduled for 2026-Q2. +> - **Its disclosure of a rewrite was removed.** It is dated 2026-03-06 and was edited six times afterwards (`c7174e05`, `edba329b`, `65949083`, `00442dec`, `94f920bc`, `a2a338b8`). `c7174e05` (2026-06-06) rewrote its findings and added the note "this SAR was written against an earlier architecture version and has been updated to reflect the current pipeline". `65949083` (2026-07-08), a rewrite of internal-facing documents for public release, deleted that note without replacing it, and the Document History still shows a single 1.0 entry. From then on the report presented later content as a record dated 2026-03-06. +> - **It reports a mechanism that does not exist.** FIND-010's "monitoring alert added for bypass detection" has no alert policy in `infra/`. +> - **The §7 Assessor Certification is withdrawn.** +> +> Findings that remain valid are tracked in [`docs/POAM.md`](../../../docs/POAM.md). + # Security Assessment Report (SAR) — Q1 2026 > **Public Context Note:** This Security Assessment Report (SAR) is published as part of CAGE's commitment to open-source security transparency. It documents the findings of the initial pre-authorization gap assessment conducted in Q1 2026 against the NIST SP 800-53 Rev. 5 HIGH baseline. This is a pre-ATO gap assessment — not a formal authorization package. Findings marked "Resolved" or "Closed" have been remediated; open findings are tracked in [`docs/POAM.md`](../../docs/POAM.md). Internal role assignments (ISSO, SCA, AO) are omitted from this public version. diff --git a/compliance/boundary/AUTHORIZATION_BOUNDARY.md b/compliance/boundary/AUTHORIZATION_BOUNDARY.md index e794998fd..3f7f493e5 100644 --- a/compliance/boundary/AUTHORIZATION_BOUNDARY.md +++ b/compliance/boundary/AUTHORIZATION_BOUNDARY.md @@ -17,7 +17,7 @@ | Authorizing Official (AO) | [AO NAME — TBD] | | Cloud Platform | Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP) | | Deployment Namespace | `governance-stack` | -| Related Documents | `compliance/categorization/FIPS199_CATEGORIZATION.md`, `compliance/rar/RISK_ASSESSMENT_REPORT.md`, `compliance/sar/SAR_2026Q1.md`, `docs/compliance/us_fed/SECURITY_ASSESSMENT_PLAN.md` | +| Related Documents | `compliance/categorization/FIPS199_CATEGORIZATION.md`, `compliance/rar/RISK_ASSESSMENT_REPORT.md`. No security assessment plan is in force, and the 2026-Q1 security assessment report was retracted on 2026-11-15 (`compliance/archive/retracted/SAR_2026Q1.md`, `POAM-2026-224`). | --- @@ -33,7 +33,7 @@ This Authorization Boundary Document defines the logical and technical boundary 4. Identifies security controls that are inherited from the Common Control Provider (GCP/GKE) 5. Provides a visual representation of the system boundary -The authorization boundary governs the scope of the Security Assessment Plan (`docs/compliance/us_fed/SECURITY_ASSESSMENT_PLAN.md`), the Security Assessment Report (`compliance/sar/SAR_2026Q1.md`), and the System Security Plan (SSP). +The authorization boundary sets the scope of the System Security Plan (SSP) and of any future security assessment. No security assessment plan is in force: the draft plan was deleted on 2026-07-30 (`00442dec`), and the 2026-Q1 security assessment report was retracted on 2026-11-15 because the assessment it records did not take place (`POAM-2026-224`). ### 1.2 Scope diff --git a/compliance/rar/RISK_ASSESSMENT_REPORT.md b/compliance/rar/RISK_ASSESSMENT_REPORT.md index 9c30325cd..9b2a6b9b0 100644 --- a/compliance/rar/RISK_ASSESSMENT_REPORT.md +++ b/compliance/rar/RISK_ASSESSMENT_REPORT.md @@ -15,7 +15,7 @@ | Risk Assessment Lead | [ISSO NAME — TBD] | | Authorizing Official (AO) | [AO NAME — TBD] | | Assessment Methodology | NIST SP 800-30 Rev. 1 | -| Related Documents | `docs/POAM.md`, `compliance/sar/SAR_2026Q1.md`, `compliance/categorization/FIPS199_CATEGORIZATION.md`, `compliance/boundary/AUTHORIZATION_BOUNDARY.md` | +| Related Documents | `docs/POAM.md`, `compliance/archive/retracted/SAR_2026Q1.md` (retracted 2026-11-15), `compliance/categorization/FIPS199_CATEGORIZATION.md`, `compliance/boundary/AUTHORIZATION_BOUNDARY.md` | --- diff --git a/compliance/ssp/SYSTEM_SECURITY_PLAN_OUTLINE.md b/compliance/ssp/SYSTEM_SECURITY_PLAN_OUTLINE.md index 485e1eb58..055c67386 100644 --- a/compliance/ssp/SYSTEM_SECURITY_PLAN_OUTLINE.md +++ b/compliance/ssp/SYSTEM_SECURITY_PLAN_OUTLINE.md @@ -362,8 +362,8 @@ _The following documents constitute the NIST RMF authorization package and relat | System Security Plan — EU_ECB | `compliance/oscal/system-security-plan-eu-ecb.yaml` | OSCAL SSP — EU AI Act / GDPR Art. 22 / DORA | ✅ Exists | | System Security Plan — APAC_MAS | `compliance/oscal/system-security-plan-apac-mas.yaml` | OSCAL SSP — MAS FEAT / MAS Notice 655 / MAS TRM §6.3 | ✅ Exists | | Assessment Results | `compliance/lula/assessment-results.yaml` | OSCAL assessment results (Lula-generated) | ✅ Exists | -| Assessment Plan | `compliance/oscal/assessment-plan.yaml` | OSCAL assessment plan | ⬜ Pending | -| POA&M (OSCAL) | `compliance/oscal/plan-of-action-milestones.yaml` | OSCAL POA&M representation | ⬜ Pending | +| Assessment Plan | None: no OSCAL assessment plan has been written | OSCAL assessment plan | ⬜ Not created | +| POA&M (OSCAL) | None: the POA&M is maintained in Markdown ([`docs/POAM.md`](../../docs/POAM.md)) | OSCAL POA&M representation | ⬜ Not created | | Lula Specifications | `compliance/lula/lula-validation-*.yaml` | Automated control validation specifications (29 manifests) | ✅ Exists | --- diff --git a/docs/BREAKING_CHANGES.md b/docs/BREAKING_CHANGES.md index 1112d6cc2..27df78aa8 100644 --- a/docs/BREAKING_CHANGES.md +++ b/docs/BREAKING_CHANGES.md @@ -178,7 +178,7 @@ plugin in PR C. No interim adapter is provided. ### Acceptance Criteria -Per ``plans/domain_extraction_implementation_plan.md``: +Per [`plans/completed/domain_extraction_implementation_plan.md`](../plans/completed/domain_extraction_implementation_plan.md): - [x] G1: Tier dispatch loop executes and honors phase/order - [x] G2: Capability predicate (`claims_action()`) replaces hardcoded literals @@ -204,7 +204,7 @@ Per ``plans/domain_extraction_implementation_plan.md``: |--------|-------------|-----------| | `2d4aea66^:src/gateway/governance/generated_stpa_validator.py` (`STPAValidator` class) | `2d4aea66^:src/gateway/governance/generated_stpa_validator.py` (`GeneratedSTPAValidator`; since `#293` / `2d4aea66`, compiled per domain plugin at [`src/cage_finance/stpa/uca_rules.py`](../src/cage_finance/stpa/uca_rules.py) and dispatched via [`src/gateway/governance/stpa_validator.py`](../src/gateway/governance/stpa_validator.py)) | Replace `from src.gateway.governance.stpa_validator import STPAValidator` with `from src.gateway.governance.generated_stpa_validator import GeneratedSTPAValidator`; replace `.validate(action_name, params)` calls with `.validate_generated(action_name, params)`. | | [`src/gateway/governance/safety/cbf_engine.py`](../src/gateway/governance/safety/cbf_engine.py) (entire file)

**Correction:** `cbf_engine.py` remains in the kernel at `src/gateway/governance/safety/cbf_engine.py` (1,114 lines). Only financial-specific CBF barrier calculations moved to `cage_finance/tiers/cbf_tier.py`. | [`src/gateway/governance/text_filter.py`](../src/gateway/governance/text_filter.py) (`ac_keyword_scan`); [`src/gateway/governance/safety/cbf_engine.py`](../src/gateway/governance/safety/cbf_engine.py) (`ControlBarrierFunction`, `safety_filter`) | Replace `from src.gateway.governance.safety import ac_keyword_scan` with `from src.gateway.governance.text_filter import ac_keyword_scan`; replace `from src.gateway.governance.safety import ControlBarrierFunction, safety_filter` with `from src.gateway.governance.cbf import ControlBarrierFunction, safety_filter`. | -| `src/gateway/governance/agw_envelope.py` (entire file — `AGWEnvelope`, `AGWEnvelopeBuilder` backward-compatibility aliases) | [`src/gateway/governance/governance_envelope.py`](../src/gateway/governance/governance_envelope.py) (`GovernanceEnvelope`, `GovernanceEnvelopeBuilder`) | Replace `from src.gateway.governance.agw_envelope import AGWEnvelope` with `from src.gateway.governance.governance_envelope import GovernanceEnvelope`; replace `AGWEnvelopeBuilder` with `GovernanceEnvelopeBuilder` (same module). `tests/test_agw_envelope.py` (the backward-compatibility test suite for these aliases) is also deleted — see [`tests/test_governance_envelope.py`](../tests/test_governance_envelope.py) for the canonical coverage. **(Completed post-tag, `fix/v3-breaking-changes-completion`.)** | +| Module `agw_envelope` in `src.gateway.governance` (entire file — `AGWEnvelope`, `AGWEnvelopeBuilder` backward-compatibility aliases); no commit on `main` contains it | [`src/gateway/governance/governance_envelope.py`](../src/gateway/governance/governance_envelope.py) (`GovernanceEnvelope`, `GovernanceEnvelopeBuilder`) | Replace `from src.gateway.governance.agw_envelope import AGWEnvelope` with `from src.gateway.governance.governance_envelope import GovernanceEnvelope`; replace `AGWEnvelopeBuilder` with `GovernanceEnvelopeBuilder` (same module). The backward-compatibility test suite for these aliases (`test_agw_envelope`, also absent from `main`) is also deleted — see [`tests/test_governance_envelope.py`](../tests/test_governance_envelope.py) for the canonical coverage. **(Completed post-tag, `fix/v3-breaking-changes-completion`.)** | ### Removed Classes/Functions @@ -212,7 +212,7 @@ Per ``plans/domain_extraction_implementation_plan.md``: |--------|--------|-------------|-----------| | `GovernanceClient` (alias) | [`src/governed_financial_advisor/infrastructure/governance_client.py:323`](../src/governed_financial_advisor/infrastructure/governance_client.py:323) | `StructuredLLMClient` (same module) | Replace `GovernanceClient(...)` with `StructuredLLMClient(...)`; update any type hints from `GovernanceClient` to `StructuredLLMClient`. See [`docs/examples/governed-financial-advisor/ARCHITECTURE.md`](examples/governed-financial-advisor/ARCHITECTURE.md) for Layer 4 reference application context. | | `RedisClient` (alias) | [`src/governed_financial_advisor/infrastructure/redis_client.py:268`](../src/governed_financial_advisor/infrastructure/redis_client.py:268) | `AsyncRedisClient` (same module) | Replace `RedisClient()` with `AsyncRedisClient()`. **Note:** do not confuse with the unrelated `_AsyncRedisClient`/`_SyncRedisClient` pair in [`src/gateway/infrastructure/redis_client.py`](../src/gateway/infrastructure/redis_client.py) — that module is untouched by this removal. | -| `HybridClient` (alias) | ``src/governed_financial_advisor/infrastructure/llm_client.py:23`` | `InferenceClient` from [`src/integrations/inference_client/client.py`](../src/integrations/inference_client/client.py) (the kernel's `GatewayClient` until MF-1) | Replace `from src.governed_financial_advisor.infrastructure.llm_client import HybridClient` with `from src.integrations.inference_client.client import InferenceClient`, and install the `inference` extra. | +| `HybridClient` (alias) | `afd4c2d1^:src/governed_financial_advisor/infrastructure/llm_client.py` (`HybridClient = GatewayClient`) | `InferenceClient` from [`src/integrations/inference_client/client.py`](../src/integrations/inference_client/client.py) (the kernel's `GatewayClient` until MF-1) | Replace `from src.governed_financial_advisor.infrastructure.llm_client import HybridClient` with `from src.integrations.inference_client.client import InferenceClient`, and install the `inference` extra. | | `check_safety_constraints` (tool alias) | [`src/governed_financial_advisor/agents/evaluator/agent.py:193`](../src/governed_financial_advisor/agents/evaluator/agent.py:193); [`src/gateway/server/mcp_tool_server.py:483`](../src/gateway/server/mcp_tool_server.py:483); [`src/governed_financial_advisor/tools/api.py:87-88`](../src/governed_financial_advisor/tools/api.py:87); [`src/governed_financial_advisor/graph/nodes/evaluator_node.py:22,147`](../src/governed_financial_advisor/graph/nodes/evaluator_node.py:22) | `simulate_governance_check` | Rename every reference to the tool/function name `check_safety_constraints` to `simulate_governance_check` across all 4 call sites (they must land in one atomic PR). | | `create_ftra_node(registry_path=..., plan_key=...)` deprecated params | [`src/gateway/governance/ftra/node_factory.py:145-149`](../src/gateway/governance/ftra/node_factory.py:145) | `config: FtraNodeConfig` parameter (same function) | Replace `create_ftra_node(registry_path="x", plan_key="y")` with `create_ftra_node(config=FtraNodeConfig(registry_path="x", plan_key="y"))`. See `Migration Guide` for the full before/after. | | `CONTROL_META` (module-level dict alias) | [`src/compliance_bridge/types.py:340`](../src/compliance_bridge/types.py:340) | `get_control_meta(region)` | Replace `from src.compliance_bridge.types import CONTROL_META` + direct iteration with `from src.compliance_bridge.types import get_control_meta` and call `get_control_meta(CAGE_DEPLOYMENT_REGION)`. **Behavior note:** `CONTROL_META` contained universal (ISO 42001) controls only — `get_control_meta(region)` returns universal + jurisdictional controls merged for the given region. Passing `"universal"` (or any unrecognized region string) reproduces the old universal-only subset. | @@ -710,8 +710,8 @@ Per the Core Architectural Principle in [`AGENTS.md`](../AGENTS.md), CAGE is a r | **AW-1** | W1.5 | Implicit GCS activation eliminated. Cold storage now requires explicit `EVIDENCE_COLD_STORE` ∈ `{gcs, s3, null}`, defaulting to `null`. | Backend selection becomes an explicit architectural decision instead of an emergent property of which bucket env var happens to be set. | Unset → `null` backend, startup WARNING, and Prometheus metric `cage_evidence_cold_store_available{backend="null"}`. | | **AW-2** | W1.4 | `EVIDENCE_STREAM_GCS_BUCKET*` removed. Replaced by `EVIDENCE_COLD_STORE_BUCKET*` and `config/compliance/residency.json`. **No alias is read.** | The `GCS` infix was a vendor leak in adopter-facing configuration. Residency policy moved from Python code to declarative config. | Loud `MissingBucketConfigError` raised on startup if unconfigured. | | **AW-3** | W1.5 | `STORAGE_BACKEND` and duplicate OSCAL S3 dispatcher in `storage.py` removed. Consolidated into unified `EvidenceColdStore`. | Satisfies invariant I-3: one unified cold store interface across all subsystems. Fixes applied once. | Stale helper calls fail at import; callers use `put_oscal_artifact_atomic()` or `EvidenceColdStore`. | -| **AW-4** | W2.1 | `src.compliance_bridge.evidence_stream` promoted to `src.gateway.governance.evidence.stream`. Lazy kernel→bridge imports severed. **No compatibility shim.** | Enforces strict Three-Layer Architecture: Layer 1 (kernel) must never import from Layer 3 (bridge). Enforced by Gate G3. | `ModuleNotFoundError` on any stale import or patch path. | -| **AW-5** | W1.7 | `src.cage_finance.safety.cbf` shim deleted. | Eliminates forbidden shim in domain package; canonical CBF engine lives in kernel. | `ModuleNotFoundError` at collection. | +| **AW-4** | W2.1 | The `evidence_stream` module of `src.compliance_bridge` (`75fe0ac4^:src/compliance_bridge/evidence_stream.py`) promoted to `src.gateway.governance.evidence.stream`. Lazy kernel→bridge imports severed. **No compatibility shim.** | Enforces strict Three-Layer Architecture: Layer 1 (kernel) must never import from Layer 3 (bridge). Enforced by Gate G3. | `ModuleNotFoundError` on any stale import or patch path. | +| **AW-5** | W1.7 | The `cbf` shim in `src.cage_finance.safety` (`a0f46cc8^:src/cage_finance/safety/cbf.py`) deleted. | Eliminates forbidden shim in domain package; canonical CBF engine lives in kernel. | `ModuleNotFoundError` at collection. | | **AW-6** | W1.6 | Raw `"langfuse.*"` string literals banned outside `attributes.py`. Enforced by CI Gate G7. | Enforces OTLP wire format neutrality. Kernel attributes imported from `src.gateway.observability.attributes`. | CI Gate G7 fails with file and line number. | | **AW-7** | W1.5 | Silent tolerance of `CAGE_ENV=prod` + `EVIDENCE_COLD_STORE=null` blocked. | Refuses to run production posture without durable cold storage unless explicitly overridden. | Fails fast with `RuntimeError` on startup unless `CAGE_ALLOW_NONBLOCKING_PROD=true`. | | **AW-8** | W1.6 | `LangfuseTelemetryProvider.from_env()` silent fallback to `MockTelemetryProvider` eliminated. | Prevents causal gatekeeper from governing on fabricated synthetic data when credentials are missing. | Raises explicit `ConfigurationError` when credentials are missing. | diff --git a/docs/POAM.md b/docs/POAM.md index a8b29cf6c..1f6d2b72d 100644 --- a/docs/POAM.md +++ b/docs/POAM.md @@ -124,6 +124,7 @@ The following findings are tracked as open items with target remediation dates. | POAM-2026-219 | CA-2 / CA-7 / SA-11 | **Lula manifests rated or cited beyond what their Rego checks.** `#664` (`372daad2`, 2026-11-06), a Lula v0.16 schema migration of all 33 manifests, removed assertions from `ftra`, `eu-fria`, `tqp007`, `flowsignal`, `iso001-token-quota` and `sc8` without changing their [`compliance/lula/README.md`](../compliance/lula/README.md) rows, their headers or the closures citing them, and its message describes a schema migration and no removals. `eu-fria` backed closed POAM-2026-099 while checking neither the `fria` tier nor the `EU_ECB` region. Remediated 2026-11-14: four manifests re-rated Stub, the `iso001-token-quota` policy-content check restored, the `sc8` header corrected, the citations in POAM-2026-030-B, -074, -080, -099, -167 and -195 withdrawn, [`test_lula_description_parity.py`](../tests/unit/test_lula_description_parity.py) holding every Active row to its Rego, and [`check_k8s_terraform_parity.py`](../scripts/check_k8s_terraform_parity.py) failing an Active manifest that selects an object `infra/` does not create. Scanning every ledger and the evidence-claim census found older citations of manifests already rated Stub, listed in the detail section for a later fix: closed POAM-2026-079 cites `ac2`, and seven US SSP statements cite `ac2`, `aarm-vectors` or an AI 600-1 placeholder. Remains Open pending the mainline squash-merge SHA, a Gate W run of the restored check, and those fixes (detail section) | Moderate | 2026-11-30 | | POAM-2026-220 | SC-7 / AU-9 / CA-7 / GDPR Art. 44 / MAS TRM §4.2 | **Latent `CAGE_ENV=prod` `compliance-bridge` startup failure in `resolve_cold_store_bucket()` masked by `staging` geographic-residency bypass.** [`resolve_cold_store_bucket()`](../src/gateway/governance/evidence/residency.py) skipped geographic validation whenever `CAGE_ENV` was in `("development", "dev", "test", "ci", "staging", "uat", "preprod")`, so Gate W never exercised runtime cold-store residency (`residency_result_breakdown` in [`gate_w_certification.json`](../compliance/gate_w_certification.json)). Under `CAGE_ENV=prod` (`US_FED`, `EVIDENCE_COLD_STORE=gcs`, `EVIDENCE_COLD_STORE_BUCKET=-evidence-worm-prod`, and no `GOOGLE_CLOUD_LOCATION` on `module.compliance_bridge`), `resolve_cold_store_bucket(backend_id="gcs")` rejected the unprefixed bucket with `ResidencyViolationError` and crashed `EvidenceCustodian.from_env()` at bridge startup, while the identical inputs passed under `CAGE_ENV=staging`; [`get_cold_store()`](../src/gateway/governance/evidence/factory.py) also bypassed residency whenever a caller passed `bucket=`. **Remediated in code (`2026-11-15`):** `resolve_cold_store_bucket()` enforces residency across all `is_enforcing()` postures (`staging` included) against `EVIDENCE_COLD_STORE_LOCATION` — injected into [`module.compliance_bridge`](../infra/modules/compliance_bridge/main.tf) from [`module.worm_bucket.bucket_location`](../infra/modules/worm_bucket/outputs.tf) — retiring bucket-name prefixes, `GOOGLE_CLOUD_LOCATION`/`AWS_REGION` fallbacks, and `LOCAL`, and [`get_cold_store()`](../src/gateway/governance/evidence/factory.py) passes `bucket_override=bucket`, guarded by [`test_evidence_residency.py`](../tests/test_evidence_residency.py) and [`operator_identity.tftest.hcl`](../infra/modules/compliance_bridge/tests/operator_identity.tftest.hcl). Closes on the mainline squash-merge SHA plus a Gate W run exercising `compliance-bridge` startup and WORM custody with residency active in `staging` (detail section) | High | 2026-11-30 | | POAM-2026-222 | CA-7 / RA-3 / SI-4 / PM-9 | **Threshold Traceability Matrix monitoring column cites telemetry that was never emitted; RA-001–RA-004 held unsigned until every row has a disposition.** The `Langfuse Metric` column and Appendix B of [`THRESHOLD_TRACEABILITY_MATRIX.md`](../compliance/risk_acceptance/THRESHOLD_TRACEABILITY_MATRIX.md) (CAGE-TTM-001, dated `2026-03-06`, present in root commit `1902c92c`) name 26 `cage.*` metrics for 23 thresholds. None has ever been emitted by `src/` or `infra/`; the column describes a Langfuse custom-metric backend that was never built (the first metric emitter landed `2026-08-15`, and every emitted metric is a `cage_*` Prometheus series). `#769` withdrew one name under `POAM-2026-010`; the remaining 25 names across 22 rows each need an owner disposition (implement, re-point, or withdraw and reopen), and THR-LAT-001 also misnames the quantity it bounds: FIN-2 enforces quote age at receipt, not trade round-trip latency. Zero OSCAL statements or census rows cite the column, and RA-001–RA-004 are unsigned. **The Moderate rating rests on the signing hold, not on that timing:** RA-001–RA-004 and the CAGE-TTM-001 approval block must not be signed until every row has a recorded disposition, and a signature applied earlier raises this item to High. Owner: Authorizing Official, the signatory of RA-001–RA-004 (detail section) | Moderate | 2026-12-31 | +| POAM-2026-224 | CA-2 / CA-7 / PL-2 | **Gate G9's `KNOWN_DOCUMENTED_GAPS` waiver hid citations of artifacts that never existed, and `DOC_SCOPE` leaves most of `docs/` unverified.** The waiver in [`check_doc_references.py`](../scripts/check_doc_references.py) excused 160 targets: 129 never existed in any commit, 28 were retired, and 3 exist at HEAD. It was described as deliberate NIST RMF gap citations but applied to every document in scope, so it hid the 2026-Q1 SAR and the authorization boundary document citing a security assessment plan at a path that exists in no commit (re-pointed there by `a2a338b8` after the draft plan they first cited was deleted in `00442dec`), and the SSP outline citing never-existent OSCAL assessment-plan and POA&M files. A hand sweep of the `2026-03-06` document cohort found them; the gate did not. Without the waiver, at `eca0c553` the current `DOC_SCOPE` (69 documents) had 17 broken references in 9 documents, and all of `docs/` except `docs/partners/**` had 299 in 52. **Remediated in part (`2026-11-15`):** the waiver is deleted, the 17 references are corrected or pinned, the SAR is retracted under a verified notice ([`SAR_2026Q1.md`](../compliance/archive/retracted/SAR_2026Q1.md)), and `AGENTS.md` states the real scope. Open until `DOC_SCOPE` covers all of `docs/` except `docs/partners/**` (detail section) | Moderate | 2026-12-31 | ### EU ECB Region (EU_ECB) @@ -4609,3 +4610,35 @@ Making [`scripts/check_stpa_freshness.py`](../scripts/check_stpa_freshness.py) f 2. THR-LAT-001's row and threshold description name the enforced quantity, quote age at receipt. 3. The mechanism-reference gate passes on the matrix with zero waivers. 4. The approval-block history shows RA-001–RA-004 were not signed before criterion 1 held. + +### POAM-2026-224: Gate G9 Waiver Hid Citations of Artifacts That Never Existed, and `DOC_SCOPE` Leaves Most of `docs/` Unverified + +**Control:** NIST CA-2 (Control Assessments), CA-7 (Continuous Monitoring), PL-2 (System Security and Privacy Plans) +**Risk Level:** Moderate +**Status:** Open +**Owner:** Maintainer (Gate G9) +**Date Opened:** 2026-11-15 +**Target Closure:** 2026-12-31 + +**Description:** +Gate G9 ([`check_doc_references.py`](../scripts/check_doc_references.py)) carried a `KNOWN_DOCUMENTED_GAPS` waiver of 162 entries naming 160 targets (two entries repeat a path with a `:` suffix). Classified against git history on `2026-11-15`, 129 targets never existed in any commit, 28 were retired, and 3 exist at HEAD. The waiver's comment described its entries as deliberate NIST RMF gap citations, but the gate skipped them in every document in scope, in path resolution, file-path checks, and Python-symbol checks alike. It therefore hid the 2026-Q1 SAR and the authorization boundary document citing a security assessment plan at a path that exists in no commit (re-pointed there by `a2a338b8` after the draft plan they first cited was deleted in `00442dec`), and the SSP outline citing never-existent OSCAL assessment-plan and POA&M files. These citations were found by hand in the sweep of the `2026-03-06` document cohort, not by the gate. + +**Scope without the waiver (measured at `eca0c553`):** + +| Scope | Documents | Broken references | Documents affected | +|---|---|---|---| +| Current `DOC_SCOPE` | 69 | 17 | 9 | +| `DOC_SCOPE` plus `docs/compliance/**` | 101 | 237 | 24 | +| All of `docs/` except `docs/partners/**` | 179 | 299 | 52 | + +The widest scope's 299 references name 166 distinct targets: 134 never existed and 32 were retired. Until this change, `AGENTS.md` named only `CHANGELOG.md` and `docs/partners/**` as outside the scope, so every other document read as verified. + +**Remediation:** +1. ✅ **`2026-11-15` (this change):** The waiver is deleted with its three uses. The 17 references are corrected, or pinned to a mainline commit where the path existed. The SAR is retracted to [`compliance/archive/retracted/SAR_2026Q1.md`](../compliance/archive/retracted/SAR_2026Q1.md) under a dated notice, because no in-place correction yields a true record of an assessment that did not happen. G9 verifies a retracted record's first-line notice and its POAM row before skipping the preserved body, and verifies the examples in branch-naming tables against their branch patterns instead of the filesystem. `AGENTS.md` states the real scope. +2. Widen `DOC_SCOPE` area by area (reference counts at `eca0c553`): `docs/compliance/us_fed` (177 references), `docs/compliance/cross-region` (21), `docs/compliance/REGION_GUARD_AUDIT.md` (13), the rest of `docs/compliance`, then the rest of `docs/`. Each change adds its area and corrects that area's references. A citation of a never-existent artifact as current evidence gets its own POAM item, not just rewording. A citation of an artifact that is deliberately missing uses a marker G9 verifies: the path must not exist and must name an open POAM item. +3. ✅ **`2026-11-15` (this change):** A regression test fails if `KNOWN_DOCUMENTED_GAPS` returns ([`test_check_doc_references.py`](../tests/test_check_doc_references.py)). + +**Closure Criteria:** +1. `DOC_SCOPE` covers every Markdown document under the current roots plus all of `docs/` except `docs/partners/**`. +2. Retracted records carry verified notices. +3. G9 passes on `main` with no waiver. diff --git a/docs/architecture/EXTENSIBILITY_ARCHITECTURE.md b/docs/architecture/EXTENSIBILITY_ARCHITECTURE.md index c12374466..ef50c3e3f 100644 --- a/docs/architecture/EXTENSIBILITY_ARCHITECTURE.md +++ b/docs/architecture/EXTENSIBILITY_ARCHITECTURE.md @@ -716,7 +716,7 @@ the public repository), use the plugin escape hatch described below. ### Step 1: Build Plugin Loader (One-Time Setup) -If not already implemented, create `src/gateway/governance/provider_plugin_loader.py`: +CAGE ships no provider plugin loader. An adopter writes one, for example as a `provider_plugin_loader` module beside [`plugin_loader.py`](../../src/gateway/governance/plugin_loader.py), which loads domain plugins: 1. Implement allow-list validation from external Secret/ConfigMap 2. Implement signature verification (cosign/Sigstore or SHA-256 digest pinning) diff --git a/docs/architecture/NON_FORMATION_PROOF_SPEC.md b/docs/architecture/NON_FORMATION_PROOF_SPEC.md index 0feca8f3d..2010d743c 100644 --- a/docs/architecture/NON_FORMATION_PROOF_SPEC.md +++ b/docs/architecture/NON_FORMATION_PROOF_SPEC.md @@ -307,7 +307,7 @@ in §5-§8 is justified by exactly one row below. | **`containment_attestation` sub-object** | 🆕 New | Closes proof element 6 — new field referencing pinned `proof/model.py` digest + deployed commit SHA (§6.6) | | **`rule_snapshot` + `rule_lineage` chain** | 🆕 New | Closes proof element 9 — new chained structure mirroring `ProvenanceRecord` (§6.4) | | **Receipt WORM persistence path** (`refusal-receipts//.json`) | 🆕 New | Closes proof element 7 fully — a new object-key prefix in the existing retention-locked, CMEK-encrypted WORM bucket (`infra/modules/worm_bucket`) that the evidence cold store already writes to | -| **Replay verification procedure/tool** | 🆕 New | Closes proof element 8 — a new `scripts/replay_refusal_receipt.py`-class utility (design only, §7.1) | +| **Replay verification procedure/tool** | 🆕 New | Closes proof element 8 — a new replay utility under `scripts/` (design only, §7.1; not implemented) | | **`GovernanceRefusalReceipt` OSCAL cross-reference emitter** | 🆕 New (optional) | Feeds `oscal_ssp_exporter.py`-style ingestion for regulator-facing evidence bundles | --- diff --git a/docs/architecture/SUBSTRATE_MOAT_STRATEGY.md b/docs/architecture/SUBSTRATE_MOAT_STRATEGY.md index 175799725..6c022ada2 100644 --- a/docs/architecture/SUBSTRATE_MOAT_STRATEGY.md +++ b/docs/architecture/SUBSTRATE_MOAT_STRATEGY.md @@ -209,7 +209,7 @@ The following work items are ordered by strategic priority. Items marked **[BLOC | Python Substrate SDK | HIGH | ✅ Shipped (v0.2.0) | [`packages/cage-client/`](../../packages/cage-client/README.md) | | TypeScript Substrate SDK | MEDIUM | TBD | ❌ Not implemented | | Policy Version Pinning API | HIGH | TBD | Extend `src/gateway/server/hybrid_server.py` | -| Developer Quickstart (ACS path) | HIGH | TBD | `docs/QUICKSTART_ACS.md` (new) | +| Developer Quickstart (ACS path) | HIGH | TBD | A new quickstart document under `docs/` (not yet written) | ### Phase 3 — Semantic Context Integration (Q1 2027) diff --git a/docs/compliance/us_fed/PHASE4_LULA_VALIDATION_PLAN.md b/docs/compliance/us_fed/PHASE4_LULA_VALIDATION_PLAN.md index 7f3735d62..23187e700 100644 --- a/docs/compliance/us_fed/PHASE4_LULA_VALIDATION_PLAN.md +++ b/docs/compliance/us_fed/PHASE4_LULA_VALIDATION_PLAN.md @@ -345,7 +345,7 @@ See [`compliance/lula/README.md`](../../../README.md) for the full validation co | [`compliance/lula/README.md`](../../../README.md) | Lula validation coverage and activation guide | | `de5534ac:deployment/k8s/lula-cron.yaml` | CronJob manifest (lula-audit + lula-sc4-watch) | | `de5534ac:deployment/k8s/lula-rbac.yaml` | RBAC for lula-auditor ServiceAccount | -| [`compliance/sar/SAR_2026Q1.md`](../../../compliance/sar/SAR_2026Q1.md) | Security Assessment Report (pre-ATO gap assessment) | +| [`compliance/archive/retracted/SAR_2026Q1.md`](../../../compliance/archive/retracted/SAR_2026Q1.md) | Security Assessment Report, retracted 2026-11-15: the assessment it records did not take place (`POAM-2026-224`) | | [`compliance/continuous-monitoring/ISCM_STRATEGY.md`](../../../compliance/continuous-monitoring/ISCM_STRATEGY.md) | ISCM cadence and escalation procedures | --- diff --git a/scripts/check_doc_references.py b/scripts/check_doc_references.py index 88ad1db2e..d48ac467f 100755 --- a/scripts/check_doc_references.py +++ b/scripts/check_doc_references.py @@ -29,6 +29,17 @@ The label of a Markdown link with a local target is display text: the target is the reference, and it is what gets verified. +There is no waiver. A record withdrawn as untrue is preserved verbatim under +`compliance/archive/retracted/`, and its first non-empty line is a dated notice, +`> **RETRACTED YYYY-MM-DD** ...`, naming a POAM item that has a row in +`docs/POAM.md`. The gate verifies the notice and that row, then skips the +preserved body; a record there without both is reported and checked in full. + +In a branch-naming table row, `| Purpose | | |` with the +pattern a backticked branch name built from `` parts, the example +is a branch name, not a path: it is verified against its pattern, not the +filesystem. + Usage: uv run --active python scripts/check_doc_references.py uv run --active python scripts/check_doc_references.py --output /tmp/doc_references_report.txt @@ -45,8 +56,8 @@ import shutil import subprocess import sys -import tempfile from dataclasses import dataclass +from datetime import date from pathlib import Path from typing import Any @@ -87,8 +98,10 @@ def _construct_no_dup_yaml_mapping( # The gate's scope, and the only copy of it: CI gate G9 and `make docs-check` # run this script without --path. CHANGELOG.md and docs/partners/** are outside -# the scope by design (a changelog must be free to name removed paths; the -# partner tree carries a known backlog). +# it by design (a changelog must be free to name removed paths; the partner tree +# carries a known backlog). Most of the rest of docs/ is outside it until +# POAM-2026-224 widens it area by area, so a document outside DOC_SCOPE is +# unverified, not verified. DOC_SCOPE = ( "README.md", "AGENTS.md", @@ -109,6 +122,15 @@ def _construct_no_dup_yaml_mapping( ) DOC_ROOTS = [REPO_ROOT / entry for entry in DOC_SCOPE] +# A record withdrawn as untrue moves here verbatim. Its first non-empty line is a +# dated notice naming the POAM item that withdrew it; the gate verifies the +# notice and that item's row in docs/POAM.md, then skips the preserved body. +# This is the gate's only exclusion. +RETRACTED_ARCHIVE = "compliance/archive/retracted" +RETRACTION_NOTICE_PATTERN = re.compile( + r"^> \*\*RETRACTED (?P\d{4}-\d{2}-\d{2})\*\*.*?\b(?PPOAM-\d{4}-\d{3,})\b" +) + # Markdown link: [text](target) MD_LINK_PATTERN = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") # Inline code backticks: `code` @@ -138,173 +160,6 @@ def _construct_no_dup_yaml_mapping( } # Literal words or tokens to ignore - -# Deliberately documented compliance gap artifacts (cited in NIST RMF as missing) -KNOWN_DOCUMENTED_GAPS = { - "src.gateway.governance.ftra", - "src.compliance_bridge.evidence_stream", - "src.cage_finance.safety.cbf", - ".roo/rules-code/01-code-standards.md", - ".github/branch-protection-rules.json", - "compliance/authorization/PACKAGE_INDEX.md", - "compliance/governance/ROLES_MATRIX.md", - "compliance/lula/eu_ecb/component-validations.yaml", - "compliance/lula/apac_mas/component-validations.yaml", - "compliance/lula/lula-validation-ai600-supply-chain.yaml", - "compliance/lula/lula-validation-ai600-privacy.yaml", - "compliance/lula/sp80053/", - "compliance/lula/sp80053", - "compliance/evidence/", - "compliance/evidence", - "compliance/eu_ai_act/technical_documentation/", - "compliance/eu_ai_act/technical_documentation", - "compliance_bridge/reconciliation_worker.py", - "compliance_bridge/evidence_stream.py", - "deployment/TERRAFORM_MIGRATION.md", - "config/critics.yaml", - "compliance/ssp/INHERITED_CONTROLS.md", - "compliance/sar/SAR_TEMPLATE.md", - "compliance/oscal/us_fed/system-security-plan.yaml", - "compliance/oscal/us_fed/component-definition.yaml", - "compliance/oscal/sp80053-profile.yaml", - "compliance/oscal/sp80053-assessment-results.yaml", - "compliance/oscal/profiles/cage-moderate-tailored.yaml", - "compliance/oscal/profiles/", - "compliance/oscal/profiles", - "compliance/oscal/plan-of-action-milestones.yaml", - "compliance/oscal/eu_ecb/system-security-plan.yaml", - "compliance/oscal/eu_ecb/component-definition.yaml", - "compliance/oscal/components/cage_iso42001.json", - "compliance/oscal/components/", - "compliance/oscal/components", - "compliance/oscal/assessment-plan.yaml", - "compliance/oscal/apac_mas/system-security-plan.yaml", - "compliance/oscal/apac_mas/component-definition.yaml", - "compliance/lula/us_fed/component-validations.yaml", - "compliance/lula/sp80053/lula-au-12.yaml", - "compliance/lula/sp80053/lula-ac-3.yaml", - "deployment/cloudbuild.yaml", - "deployment/k8s/generated/debug-downloader.yaml", - "deployment/k8s/generated/debug-downloader-fix.yaml", - "deployment/k8s/falco-rules.yaml", - "deployment/k8s/falco-daemonset.yaml", - "deployment/k8s/compliance-bridge-servicemonitor.yaml", - "deployment/k8s/authz-policy-seal.yaml", - "deployment/k8s/network-policy-seal.yaml", - "deployment/k8s/mtls-policy.yaml", - "deployment/k8s/istio-peer-auth.yaml", - "deployment/k8s/trace-auditor-cron.yaml", - "docs/RED_TEAM_DATASET_EXTENSION.md", - "docs/QUICKSTART_AGW.md", - "docs/QUICKSTART_ACS.md", - "docs/ISO42001_MANAGEMENT_REVIEW.md", - "docs/INFRASTRUCTURE_POLICY.md", - "docs/GIT_HISTORY_SCRUBBING.md", - "docs/DATA_GOVERNANCE.md", - "docs/Configuration_Baseline.md", - "docs/CONTINGENCY_PLAN.md", - "docs/COMPLIANCE_POLICY.md", - "docs/CHANGE_MANAGEMENT_PROCESS.md", - "docs/Authentication_Architecture.md", - "docs/cage-arxiv-revision-plan", - "docs/cage-arxiv-reproducibility", - "docs/cage-arxiv-critical-fixes", - "docs/cage-arxiv-bibliography", - "docs/ai600-agentic-scope-statement", - "proof/compliance-trigger-evidence.yaml", - "oscal-artifacts/index.json", - "scripts/deploy_sw.py", - "scripts/check_region_guards.py", - "docs/paper/FPR_IMPROVEMENT_PLAN.md", - "docs/templates/COMPLIANCE_STATUS_REPORT.md", - "docs/TECHNICAL_DECOMMISSION_GUIDELINES.md", - "docs/System_Interconnection_Agreements.md", - "docs/SYSTEM_BOUNDARY_DIAGRAM.md", - "docs/SECURITY_POLICY.md", - "docs/SECRET_MANAGEMENT_OPTIONS.md", - "docs/security/INCIDENT_RESPONSE_PLAN.md", - "tests/infra/", - "src/evaluator_agent/", - "compliance/lula/sp800-53/", - "compliance/lula/sp800-53", - "plans/poam-framework-redesign.md", - "plans/domain_extraction_implementation_plan.md", - "plans/CAGE_RISK_MATRIX.md", - "src/cage_sdk/", - "src/cage_sdk", - "src/cage_sdk_ts/", - "src/cage_sdk_ts", - "scripts/vulnerability_scan.sh", - "scripts/generate_sbom.sh", - "scripts/generate_sar.py", - "scripts/replay_refusal_receipt.py", - "src/compliance_bridge/oscal_ssp_exporter.py", - "src/gateway/governance/agw_envelope.py", - "src/gateway/governance/provider_plugin_loader.py", - "src/gateway/governance/region_utils.py", - "test_results/final_summary_20260603T104718.json", - "src/protos/gateway.js", - "tests/test_fiscal_limit_guard.py", - "tests/test_agw_envelope.py", - "tests/red_team/README.md", - "tests/integration/test_defer_workflow.py", - "src/governed_financial_advisor/infrastructure/telemetry/nemo_exporter.py", - "src/governed_financial_advisor/infrastructure/mcp_client.py", - "src/governed_financial_advisor/infrastructure/llm_client.py", - "src/governed_financial_advisor/infrastructure/llm_client.py:23", - "src/governed_financial_advisor/governance/nemo_action_registry.py", - "src/governance/transpiler.py", - "/.well-known/nexart-node.json", - ".well-known/nexart-node.json", - "docs/DATA_RETENTION_POLICY.md", - "docs/Continuous_Monitoring_Plan.md", - "docs/AUTHORIZATION_BOUNDARY.md", - "docs/AI_600_1_IMPLEMENTATION_PLAN.md", - "docs/AGENT_OPS_ARCHITECTURE.md", - "docs/AC2_Account_Management_Procedure.md", - "compliance/risk_acceptance/RISK_ACCEPTANCE_STATEMENT.md", - "compliance/authorization/AUTHORIZATION_MEMO.md", - "compliance/poam/POA_AND_M.md", - "compliance/poam/poam.yaml", - "deployment/k8s/secret-rotation-cronjob.yaml", - "src/gateway/server/agw_service_extension.py", - "docs/Network_Architecture.md", - "docs/MODEL_OUTPUT_HANDLING.md", - "docs/MISSION_BUSINESS_PROCESS.md", - "docs/Flaw_Remediation_Policy.md", - "docs/EXTERNAL_DEPENDENCY_REGISTER.md", - "docs/DECOMMISSION_CHECKLIST.md", - "docs/INTERCONNECTION_AGREEMENTS", - "docs/INTERCONNECTION_AGREEMENTS/", - "docs/Supply_Chain_Risk_Assessment.md", - "docs/SUPPLY_CHAIN_RISK_ASSESSMENT.md", - "docs/SECURITY_AUDIT_REPORT.md", - "docs/SECURITY_AUDIT_REPORT.md:758", - "docs/security/IR_PLAN.md", - "docs/Incident_Response_Plan.md", - "docs/NIST_RMF_CHUNK3_SELECT_IMPLEMENT.md", - "src/governed_financial_advisor/requirements.txt", - "src/governed_financial_advisor/utils/privacy.py", - "src/governed_financial_advisor/utils/langfuse_utils.py", - "src/governed_financial_advisor/governance/policy/generated_rules.rego", - "src/compliance_bridge/requirements.txt", - "plans/layer_inversion_remediation_plan.md", - "docs/stpa-control-diagram", - "docs/SYSTEM_SECURITY_PLAN.md", - "docs/proposals/004_risk_remediation_plan.md", - "docs/compliance/us_fed/SECURITY_ASSESSMENT_PLAN.md", - "docs/compliance/us_fed/SP800-60_INFORMATION_TYPES.md", - "deployment/deploy_sw.py", - "docs/governance/ROLES_AND_RESPONSIBILITIES.md", - "docs/compliance/universal/AI_IP_POLICY.md", - "docs/compliance/us_fed/CONTINUOUS_MONITORING_STRATEGY.md", - "docs/MIGRATION_GUIDE_v3.md", - "docs/whitepapers/CAGE_TECHNICAL_PREPRINT.md", - str(Path(tempfile.gettempdir()) / "benign_dataset.json"), - str(Path(tempfile.gettempdir()) / "adversarial_dataset.json"), - "tests/governance/test_ir_drill.py", -} - IGNORED_TOKENS = { "true", "false", @@ -537,12 +392,6 @@ def _resolve_candidate_path(base_dir: Path, target_path_str: str) -> Path | None ): return None norm = clean_target[2:] if clean_target.startswith("./") else clean_target - if ( - clean_target in KNOWN_DOCUMENTED_GAPS - or norm in KNOWN_DOCUMENTED_GAPS - or stripped_target in KNOWN_DOCUMENTED_GAPS - ): - return None candidates = [ # 1. Direct from repo root @@ -629,14 +478,6 @@ def verify_file_path( if is_template_or_pattern(clean_target): return True, "" - norm = clean_target[2:] if clean_target.startswith("./") else clean_target - if ( - clean_target in KNOWN_DOCUMENTED_GAPS - or norm in KNOWN_DOCUMENTED_GAPS - or stripped_target in KNOWN_DOCUMENTED_GAPS - ): - return True, "" - resolved = _resolve_candidate_path(base_dir, target_path_str) if resolved is not None: return _validate_line_anchor( @@ -1039,12 +880,6 @@ def _get_yaml_uuid_span(yaml_path: Path, uuid_str: str) -> tuple[int, int] | Non def verify_python_symbol(symbol_str: str) -> tuple[bool, str]: """Verify qualified module.Symbol references (e.g. src.gateway.symbolic_governor.SymbolicGovernor).""" - clean_sym = symbol_str.strip() - if clean_sym in KNOWN_DOCUMENTED_GAPS or any( - clean_sym == g or clean_sym.startswith(g) for g in KNOWN_DOCUMENTED_GAPS - ): - return True, "" - if is_template_or_pattern(symbol_str): return True, "" @@ -1330,6 +1165,139 @@ def _audit_line_references_and_contextual_symbols( return issues +def _is_retracted_record(path: Path) -> bool: + """Return True if *path* lies under RETRACTED_ARCHIVE.""" + archive = (REPO_ROOT / RETRACTED_ARCHIVE).resolve() + return archive in path.resolve().parents + + +def _poam_has_table_row(poam_id: str) -> bool: + """Return True if docs/POAM.md has a table row `| |`.""" + try: + poam_text = (REPO_ROOT / "docs" / "POAM.md").read_text(encoding="utf-8") + except OSError: + return False + row = re.compile(rf"^\|\s*{re.escape(poam_id)}\s*\|", re.MULTILINE) + return row.search(poam_text) is not None + + +def _audit_retraction_notice(path: Path, lines: list[str]) -> Issue | None: + """Verify a retracted record's dated notice and its POAM row; None if both hold.""" + line_idx, first = next( + ( + (idx, line.strip()) + for idx, line in enumerate(lines, start=1) + if line.strip() + ), + (1, ""), + ) + missing = ( + "Retracted record lacks a dated retraction notice naming an existing POAM item" + ) + notice = RETRACTION_NOTICE_PATTERN.match(first) + if notice is None: + return Issue( + path, + line_idx, + "Retraction Notice", + first[:90] or path.name, + f"{missing}: its first non-empty line must read " + "'> **RETRACTED YYYY-MM-DD** ... POAM-YYYY-NNN ...'", + ) + try: + date.fromisoformat(notice["date"]) + except ValueError: + return Issue( + path, + line_idx, + "Retraction Notice", + notice["date"], + f"{missing}: '{notice['date']}' is not a calendar date", + ) + if not _poam_has_table_row(notice["poam"]): + return Issue( + path, + line_idx, + "Retraction Notice", + notice["poam"], + f"{missing}: {notice['poam']} has no table row in docs/POAM.md", + ) + return None + + +# Branch-naming table row: | Purpose | `` | `` | +BRANCH_PLACEHOLDER_PATTERN = re.compile(r"<([a-z][a-z0-9-]*)>") +BRANCH_LITERAL_PATTERN = re.compile(r"^[a-z0-9/-]*$") +BRANCH_SEMVER_PLACEHOLDERS = frozenset({"version", "semver"}) +BRANCH_DESCRIPTION_PLACEHOLDERS = frozenset({"short-description", "description"}) +BRANCH_DESCRIPTION_MAX_CHARS = 30 # AGENTS.md: description <= 30 characters +BRANCH_KEBAB_CASE = r"[a-z0-9]+(?:-[a-z0-9]+)*" +BRANCH_SEMVER = r"\d+\.\d+\.\d+" +TABLE_CELL_DELIMITER = re.compile(r"(? str: + """Return why *example* does not match branch *pattern*, or "" if it does.""" + parts = BRANCH_PLACEHOLDER_PATTERN.split(pattern) + regex = "".join( + re.escape(part) + if idx % 2 == 0 + else f"({BRANCH_SEMVER if part in BRANCH_SEMVER_PLACEHOLDERS else BRANCH_KEBAB_CASE})" + for idx, part in enumerate(parts) + ) + match = re.fullmatch(regex, example) + if match is None: + return ( + f"Branch-name example does not match pattern '{pattern}' ( and " + " are N.N.N; every other placeholder is lowercase kebab-case)" + ) + for name, value in zip(parts[1::2], match.groups(), strict=True): + if ( + name in BRANCH_DESCRIPTION_PLACEHOLDERS + and len(value) > BRANCH_DESCRIPTION_MAX_CHARS + ): + return ( + f"Branch-name example's <{name}> part is {len(value)} characters " + f"(at most {BRANCH_DESCRIPTION_MAX_CHARS})" + ) + return "" + + +def _audit_branch_examples( + path: Path, line_idx: int, line: str +) -> tuple[tuple[int, int] | None, list[Issue]]: + """Verify the example cell of a branch-naming table row against its pattern cell. + + A row qualifies when its second cell is a single backticked branch pattern: + `` parts and lowercase literal parts, at least one with a + letter (`feat/`, `rc-v`). Returns the third + cell's span, whose backticked tokens are branch names rather than paths, and + an Issue per mismatch. + """ + bars = [m.start() for m in TABLE_CELL_DELIMITER.finditer(line)] + if len(bars) < 4 or line[: bars[0]].strip(): + return None, [] + pattern_cell = re.fullmatch(r"`([^`\s]+)`", line[bars[1] + 1 : bars[2]].strip()) + if pattern_cell is None: + return None, [] + pattern = pattern_cell.group(1) + parts = BRANCH_PLACEHOLDER_PATTERN.split(pattern) + literals = parts[::2] + if ( + len(parts) < 3 + or not all(BRANCH_LITERAL_PATTERN.match(p) for p in literals) + or not re.search(r"[a-z]", "".join(literals)) + ): + return None, [] + cell = (bars[2] + 1, bars[3]) + issues = [ + Issue(path, line_idx, "Branch Example", cm.group(1).strip(), reason) + for cm in INLINE_CODE_PATTERN.finditer(line, *cell) + if (reason := _branch_example_reason(cm.group(1).strip(), pattern)) + ] + return cell, issues + + def audit_markdown_file(path: Path) -> list[Issue]: """Scan a single markdown file for broken file paths, links, and code symbols.""" issues: list[Issue] = [] @@ -1340,6 +1308,13 @@ def audit_markdown_file(path: Path) -> list[Issue]: lines = content.splitlines() + # 0. A retracted record: verify its notice and POAM row, then skip its body + if _is_retracted_record(path): + notice_issue = _audit_retraction_notice(path, lines) + if notice_issue is None: + return [] + issues.append(notice_issue) + for line_idx, line in enumerate(lines, start=1): # Collect any revision-pinned .tf addresses on this line for historical HCL lookups pinned_tf_addrs: set[str] = set() @@ -1371,7 +1346,10 @@ def audit_markdown_file(path: Path) -> list[Issue]: Issue(path, line_idx, "Markdown Link", link_target, reason) ) - # 2. Inline code backticks: `target` + # 2. Inline code backticks: `target`. A branch-naming table row's example + # is a branch name, verified against its pattern, not a path. + branch_cell, branch_issues = _audit_branch_examples(path, line_idx, line) + issues.extend(branch_issues) is_mutation_row = bool(re.search(r"\|\s*(?:Re-declare|Restore)\s+`", line)) for code_match in INLINE_CODE_PATTERN.finditer(line): if any( @@ -1379,6 +1357,11 @@ def audit_markdown_file(path: Path) -> list[Issue]: for start, end in local_link_labels ): continue + if branch_cell is not None and ( + branch_cell[0] <= code_match.start() + and code_match.end() <= branch_cell[1] + ): + continue code_snippet = code_match.group(1).strip() if ( @@ -1685,7 +1668,7 @@ def _audit_oscal_prose_paths( _find_line_1(raw_lines, cand), "OSCAL File Path", cand, - f"OSCAL narrative path '{cand}' does not exist on disk (no KNOWN_DOCUMENTED_GAPS waiver)", + f"OSCAL narrative path '{cand}' does not exist on disk", ) ) return issues @@ -1736,7 +1719,7 @@ def _walk_oscal_node( _find_line_1(raw_lines, href_val), "OSCAL Href Path", href_val, - f"OSCAL local href '{clean_href}' does not exist on disk (no KNOWN_DOCUMENTED_GAPS waiver)", + f"OSCAL local href '{clean_href}' does not exist on disk", ) ) elif k in {"description", "remarks"} and isinstance(v, str): diff --git a/tests/test_check_doc_references.py b/tests/test_check_doc_references.py index 476928c7f..fb92c92c2 100644 --- a/tests/test_check_doc_references.py +++ b/tests/test_check_doc_references.py @@ -722,7 +722,7 @@ def test_oscal_and_lula_yaml_rejects_duplicate_mapping_keys(tmp_path: Path) -> N def test_oscal_yaml_rejects_dangling_fragment_local_href_and_docs_proof_packages_paths( tmp_path: Path, ) -> None: - """OSCAL YAML hrefs and narrative paths cover docs/, proof/, and packages/ without KNOWN_DOCUMENTED_GAPS waivers.""" + """OSCAL YAML hrefs and narrative paths cover docs/, proof/, and packages/, with no waiver.""" oscal_dir = tmp_path / "compliance" / "oscal" src_dir = tmp_path / "src" / "gateway" docs_dir = tmp_path / "docs" @@ -735,10 +735,10 @@ def test_oscal_yaml_rejects_dangling_fragment_local_href_and_docs_proof_packages (proof_dir / "real_proof.py").write_text("x = 1\n", encoding="utf-8") (pkg_dir / "pyproject.toml").write_text("[project]\n", encoding="utf-8") + # Both were KNOWN_DOCUMENTED_GAPS entries; the waiver is deleted (POAM-2026-224). + assert not hasattr(g9, "KNOWN_DOCUMENTED_GAPS") waived_gap_path = "compliance/oscal/sp80053-profile.yaml" - assert waived_gap_path in g9.KNOWN_DOCUMENTED_GAPS waived_narrative = "scripts/generate_sar.py" - assert waived_narrative in g9.KNOWN_DOCUMENTED_GAPS (oscal_dir / "eu-ai-act-profile.yaml").write_text( "profile:\n" @@ -903,3 +903,160 @@ def test_ring3_census_claim_ids_and_summary_tables_enforced( (9, "Census Claim ID", "component-definition.yaml#ghost-999"), (10, "Census Claim ID", "system-security-plan.yaml#weak-002"), ] + + +# --------------------------------------------------------------------------- +# POAM-2026-224: retracted records and branch-name examples +# --------------------------------------------------------------------------- + +NOTICE = ( + "> **RETRACTED 2026-11-15** \u2014 Withdrawn in full under `POAM-2026-224`. " + "Nothing below is a current claim.\n" +) + + +@pytest.fixture() +def retracted_repo(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """Fixture repo: docs/POAM.md with a POAM-2026-224 row, and the retracted archive.""" + (tmp_path / "docs").mkdir() + (tmp_path / "docs" / "POAM.md").write_text( + "| ID | Control | Description | Severity | Target Date |\n" + "|----|---------|-------------|----------|-------------|\n" + "| POAM-2026-224 | CA-2 | Waiver deleted | Moderate | 2026-12-31 |\n", + encoding="utf-8", + ) + (tmp_path / "compliance" / "archive" / "retracted").mkdir(parents=True) + monkeypatch.setattr(g9, "REPO_ROOT", tmp_path) + return tmp_path + + +def _retracted(repo: Path, text: str) -> list[g9.Issue]: + doc = repo / "compliance" / "archive" / "retracted" / "SAR.md" + doc.write_text(text, encoding="utf-8") + return audit_markdown_file(doc) + + +def test_retracted_record_with_notice_and_poam_row_skips_its_body( + retracted_repo: Path, +) -> None: + body = "\n# Old report\n\n| Related SAP | `docs/no_such_plan_g9.md` |\n" + assert _retracted(retracted_repo, "\n" + NOTICE + body) == [] + + +def test_retracted_record_without_notice_is_reported_and_checked( + retracted_repo: Path, +) -> None: + issues = _retracted(retracted_repo, "# Old report\n\n`docs/no_such_plan_g9.md`\n") + assert [(i.line_number, i.reference_type) for i in issues] == [ + (1, "Retraction Notice"), + (3, "File Path"), + ] + assert ( + "lacks a dated retraction notice naming an existing POAM item" + in issues[0].reason + ) + + +def test_retracted_record_naming_a_poam_without_a_row_is_reported( + retracted_repo: Path, +) -> None: + notice = NOTICE.replace("POAM-2026-224", "POAM-2026-999") + issues = _retracted(retracted_repo, notice + "\n`docs/no_such_plan_g9.md`\n") + assert [(i.reference_type, i.target) for i in issues] == [ + ("Retraction Notice", "POAM-2026-999"), + ("File Path", "docs/no_such_plan_g9.md"), + ] + assert "has no table row in docs/POAM.md" in issues[0].reason + + +@pytest.mark.parametrize( + "opening", + [ + "# Old report\n\n" + NOTICE, # the notice is not the first non-empty line + NOTICE.replace("2026-11-15", "2026-11"), # undated + NOTICE.replace("2026-11-15", "2026-13-45"), # not a calendar date + NOTICE.replace("`POAM-2026-224`", "the POAM"), # names no POAM item + ], +) +def test_retracted_record_with_a_malformed_notice_is_reported( + retracted_repo: Path, opening: str +) -> None: + issues = _retracted(retracted_repo, opening + "\nBody.\n") + assert [i.reference_type for i in issues] == ["Retraction Notice"] + + +def test_retraction_notice_outside_the_archive_excludes_nothing( + retracted_repo: Path, +) -> None: + doc = retracted_repo / "docs" / "report.md" + doc.write_text(NOTICE + "\n`docs/no_such_plan_g9.md`\n", encoding="utf-8") + assert [i.reference_type for i in audit_markdown_file(doc)] == ["File Path"] + + +def test_retracted_records_are_discovered_through_doc_scope( + retracted_repo: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + assert any( + g9.RETRACTED_ARCHIVE == entry or g9.RETRACTED_ARCHIVE.startswith(entry + "/") + for entry in g9.DOC_SCOPE + ) + doc = retracted_repo / "compliance" / "archive" / "retracted" / "SAR.md" + doc.write_text("# Old report\n", encoding="utf-8") + monkeypatch.setattr(g9, "DOC_SCOPE", ("compliance",)) + assert doc.resolve() in g9.find_markdown_files() + + +BRANCH_TABLE = "| Purpose | Pattern | Example |\n|---|---|---|\n" + + +def test_branch_table_examples_are_verified_as_branch_names(tmp_path: Path) -> None: + doc = tmp_path / "doc.md" + doc.write_text( + BRANCH_TABLE + + "| Documentation | `docs/` | `docs/stpa-control-diagram` |\n" + "| Hotfix on release | `hotfix/-` | `hotfix/2.0.1-redis-timeout` |\n" + "| Release candidate | `rc-v` | `rc-v2.1.0` |\n", + encoding="utf-8", + ) + assert audit_markdown_file(doc) == [] + + +@pytest.mark.parametrize( + ("example", "reason"), + [ + ("docs/STPA-control-diagram", "does not match pattern"), + ("docs/" + "a" * 31, " part is 31 characters"), + ("fix/stpa-control-diagram", "does not match pattern"), + ], +) +def test_branch_table_example_mismatch_is_reported( + tmp_path: Path, example: str, reason: str +) -> None: + doc = tmp_path / "doc.md" + doc.write_text( + BRANCH_TABLE + + f"| Documentation | `docs/` | `{example}` |\n", + encoding="utf-8", + ) + issues = audit_markdown_file(doc) + assert [(i.line_number, i.reference_type, i.target) for i in issues] == [ + (3, "Branch Example", example) + ] + assert reason in issues[0].reason + + +def test_docs_path_outside_a_branch_table_is_still_checked(tmp_path: Path) -> None: + doc = tmp_path / "doc.md" + doc.write_text( + "See `docs/nonexistent-thing` for details.\n" + "| Area | Document | Example |\n|---|---|---|\n" + "| Docs | `main` | `docs/nonexistent-thing` |\n" + "| Plan | `docs//plan.md` | `docs/nonexistent-thing/plan.md` |\n", + encoding="utf-8", + ) + issues = audit_markdown_file(doc) + assert [(i.line_number, i.reference_type) for i in issues] == [ + (1, "File Path"), + (4, "File Path"), + (5, "File Path"), + ]