Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### Changed

- **Gate W closes `kubectl port-forward` before Phase 4's hermetic regional suites, and hermetic pytest sessions refuse or fail on `GATEWAY_URL`** (`fix(tests)`, [`staging_lifecycle.sh`](scripts/staging_lifecycle.sh), [`conftest.py`](tests/conftest.py), [`gateway_url_guard.py`](tests/fixtures/gateway_url_guard.py), [`test_gateway_url_guard.py`](tests/unit/test_gateway_url_guard.py), [`test_staging_lifecycle.py`](tests/infrastructure/test_staging_lifecycle.py)).
- `staging_lifecycle.sh` opened the staging `kubectl port-forward` tunnel during Phase 3 pre-Lula custody seeding (and again before Phase 4) without closing it before Phase 4's `us_fed`, `eu_ecb` and `apac_mas` hermetic posture suites, while `tests/conftest.py` defaulted `GATEWAY_URL` to `http://localhost:8080` in every session. Any hermetic test reading `GATEWAY_URL` would therefore hit live staging and write synthetic records into the deployed evidence chain and Lula counts.
- Phase 3 now stops its port-forward when Lula validation finishes, Phase 4 calls `ensure_no_staging_port_forward` and passes `env -u GATEWAY_URL` before running the three hermetic regional suites, and `start_staging_port_forward` runs only afterward (before live data-residency, live Langfuse isolation, and Phase 5 live sweeps).
- `tests/conftest.py` stops defaulting `GATEWAY_URL` in hermetic sessions, refuses to start a hermetic session when `GATEWAY_URL` is exported (`gateway_url_guard.py`), and asserts `"GATEWAY_URL" not in os.environ` before and after every hermetic test; live sessions (`--run-integration`, `--run-live-external`, `--run-partner-integration`, `--run-e2e`) keep the `http://localhost:8080` default.

- **Evidence cold-store residency is enforced in every enforcing posture (`staging` included) against the WORM bucket's immutable cloud location** (`fix(governance)!`, [`residency.py`](src/gateway/governance/evidence/residency.py), [`factory.py`](src/gateway/governance/evidence/factory.py), [`residency.json`](config/compliance/residency.json), [`worm_bucket/outputs.tf`](infra/modules/worm_bucket/outputs.tf), [`compliance_bridge/main.tf`](infra/modules/compliance_bridge/main.tf), [`POAM.md`](docs/POAM.md), [`gate_w_certification.json`](compliance/gate_w_certification.json)).
- `resolve_cold_store_bucket()` skipped geographic validation whenever `CAGE_ENV` was in a hardcoded list that included `staging`, `uat` and `preprod`. Gate W therefore never exercised runtime residency, and the bypass masked a latent production failure (`POAM-2026-220`): under `CAGE_ENV=prod` and `US_FED` with `<project>-evidence-worm-prod` and no `GOOGLE_CLOUD_LOCATION`, `compliance-bridge` failed lifespan startup with `ResidencyViolationError` while the identical inputs passed under `staging`.
- `resolve_cold_store_bucket()` now enforces residency whenever `is_enforcing()` is true (`production`, `staging`, `local`, or an unknown `CAGE_ENV`), skipping only in `dev`, `test` and `ci`. Bucket-name prefixes (`allowed_prefixes` / `allow_prefixes`), `GOOGLE_CLOUD_LOCATION` / `AWS_REGION` fallbacks, and the `LOCAL` rule / `default_bucket` are removed; residency checks only `location` or `EVIDENCE_COLD_STORE_LOCATION` against `residency.json`. `get_cold_store()` validates explicit `bucket=` overrides through `resolve_cold_store_bucket(bucket_override=bucket)` instead of short-circuiting.
Expand Down
2 changes: 1 addition & 1 deletion docs/EVALUATOR_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ Every run populates a self-contained audit bundle under `<archive-dir>`:

When an evaluator runs `make test-fast` (`uv run pytest tests/ -m "local or unit" -n auto --dist loadscope --no-cov -p no:langsmith -q` in [`Makefile`](../Makefile)), CAGE intentionally runs only hermetic offline tests and prints an explicit notice listing every live suite that did not execute.

1. **Fail-Closed Selection-Marker Contract**: `_assert_selection_markers()` in [`tests/conftest.py`](../tests/conftest.py) requires every collected test to carry at least one marker from `SELECTION_MARKERS` (`unit`, `local`, `integration`, `load`, `chaos`, `live_external`, `partner_integration`, `e2e`; `MARKER_GUARD_EXEMPT_MODULES` is empty). Any unmarked test aborts collection with `PYTEST MARKER CONTRACT VIOLATION` so tests cannot silently vanish from CI gates. Hermetic sessions also run `_refuse_reachable_dotenv()` ([`tests/fixtures/dotenv_guard.py`](../tests/fixtures/dotenv_guard.py)) and abort if a `.env` file is reachable on disk, preventing ambient stack credentials from altering offline test behavior.
1. **Fail-Closed Selection-Marker Contract**: `_assert_selection_markers()` in [`tests/conftest.py`](../tests/conftest.py) requires every collected test to carry at least one marker from `SELECTION_MARKERS` (`unit`, `local`, `integration`, `load`, `chaos`, `live_external`, `partner_integration`, `e2e`; `MARKER_GUARD_EXEMPT_MODULES` is empty). Any unmarked test aborts collection with `PYTEST MARKER CONTRACT VIOLATION` so tests cannot silently vanish from CI gates. Hermetic sessions also run `_refuse_reachable_dotenv()` ([`tests/fixtures/dotenv_guard.py`](../tests/fixtures/dotenv_guard.py)) and `_refuse_hermetic_gateway_url()` ([`tests/fixtures/gateway_url_guard.py`](../tests/fixtures/gateway_url_guard.py)), aborting if a `.env` file is reachable on disk or `GATEWAY_URL` is set in the environment and asserting `GATEWAY_URL` stays unset across every hermetic test.
2. **Why `make test-fast` Skips Live Suites — and How `pytest_terminal_summary` Surfaces Them**: Suites marked `integration`, `live_external`, `partner_integration`, `e2e`, or `chaos` require live Kubernetes services, Cloud KMS/WORM/CAS APIs, external partner sandboxes, or Redis failover injection. `make test-fast` deselects them via `-m "local or unit"`, and when pytest is invoked without `-m`, `pytest_collection_modifyitems()` in [`tests/conftest.py`](../tests/conftest.py) auto-skips them unless their opt-in CLI flag (`--run-integration`, `--run-live-external`, `--run-partner-integration`, `--run-e2e`, `--run-chaos`) is supplied. To prevent a green hermetic unit run from being misread as live infrastructure validation, `pytest_deselected()` and `pytest_terminal_summary()` (via `format_unexecuted_live_suites_notice()` in [`tests/conftest.py`](../tests/conftest.py)) emit a controller-level banner at the end of the session:
```text
--- live / opt-in suites not executed ---
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/GKE_TEST_RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ Asymmetric signing keys (`EC_SIGN_P256_SHA256`) live in keyring `cage-signing-<e
- **`binauthz-attestor`**: Used exclusively by the dedicated Cloud Build service account (`cage-cloudbuild-<env>@<project>.iam.gserviceaccount.com`) via [`deployment/docker/cloudbuild.image.yaml`](../../deployment/docker/cloudbuild.image.yaml), [`scripts/build_images.sh`](../../scripts/build_images.sh), [`scripts/mirror_and_attest_images.sh`](../../scripts/mirror_and_attest_images.sh), and [`scripts/attest_image.sh`](../../scripts/attest_image.sh) (`gcloud beta container binauthz attestations sign-and-create` + fail-closed `binauthz attestations list` verification) to sign Binary Authorization attestations for every container image digest in `var.image_digests`, which [`scripts/render_image_digests.sh`](../../scripts/render_image_digests.sh) renders per deployment for exactly the images the profile deploys (no committed digests, POAM-2026-131).

**Local Workstation KMS Verification:**
- In [`tests/conftest.py`](../../tests/conftest.py), running client test commands with `CAGE_ENV=test` allows the test harness to use software-backed signing for local assertions while directing HTTP traffic to live cluster endpoints (`BACKEND_URL=http://localhost:8081`, `GATEWAY_URL=http://localhost:8080`).
- In [`tests/conftest.py`](../../tests/conftest.py), running live client test commands (`--run-integration` / `--run-e2e`) with `CAGE_ENV=test` allows the test harness to use software-backed signing for local assertions while directing HTTP traffic to live cluster endpoints (`BACKEND_URL=http://localhost:8081`, `GATEWAY_URL=http://localhost:8080`; hermetic runs keep `GATEWAY_URL` unset).
- To verify Gateway responses signed by the live cluster's `gateway-seal` KMS key, fetch the public key via the Gateway JWKS endpoint (`curl -s http://localhost:8080/governance/.well-known/jwks.json`) or extract the cached PEM from the pod:
```bash
kubectl exec -n governance-stack deploy/gateway -c gateway -- cat /tmp/kms_governance_public.pem > /tmp/kms_governance_public.pem
Expand Down
21 changes: 16 additions & 5 deletions scripts/staging_lifecycle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -584,7 +584,7 @@ run_hermetic_region_pytest() {
local marker="$3"
if [[ -n "${CAGE_HERMETIC_PYTEST_WRAPPER:-}" ]]; then
run_pytest_require_passed "$label" \
"$CAGE_HERMETIC_PYTEST_WRAPPER" env CAGE_DEPLOYMENT_REGION="$region_val" ./.venv/bin/pytest tests/ -m "$marker" -v --tb=short
env -u GATEWAY_URL "$CAGE_HERMETIC_PYTEST_WRAPPER" env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION="$region_val" ./.venv/bin/pytest tests/ -m "$marker" -v --tb=short
return $?
fi
if [[ -f "$REPO_ROOT/.env" ]]; then
Expand All @@ -594,7 +594,7 @@ run_hermetic_region_pytest() {
local rc=0
set +e
run_pytest_require_passed "$label" \
env CAGE_DEPLOYMENT_REGION="$region_val" uv run pytest "$REPO_ROOT/tests/" -m "$marker" -v --tb=short
env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION="$region_val" uv run pytest "$REPO_ROOT/tests/" -m "$marker" -v --tb=short
rc=$?
set -e
restore_shielded_dotenv
Expand Down Expand Up @@ -1628,13 +1628,17 @@ if errors:
error "$LULA_FAILURES Lula validation check(s) failed"
error "POAM-024 is NOT satisfied — security posture incomplete"
fi
stop_staging_port_forward
else
warn "Lula validation skipped (--skip-lula flag)"
fi

if [[ "$SKIP_POSTURE" == "false" || "$SKIP_LIVE_SWEEP" == "false" ]]; then
start_staging_port_forward
fi
ensure_no_staging_port_forward() {
if [[ -x "$REPO_ROOT/scripts/port_forward_staging.sh" ]]; then
"$REPO_ROOT/scripts/port_forward_staging.sh" --stop >/dev/null 2>&1 || true
fi
PORT_FORWARD_ACTIVE=false
}

# ─── Phase 4: Region Posture, Live Residency & Live Langfuse (FAIL CLOSED) ─
if [[ "$SKIP_POSTURE" == "false" ]]; then
Expand All @@ -1644,6 +1648,10 @@ if errors:
echo -e "${CYAN}╚══════════════════════════════════════════════════════════════════╝${RESET}"
echo ""

# Hermetic regional suites must never run while kubectl port-forward exposes
# live staging on localhost:8080 (even if started externally before --skip-lula).
ensure_no_staging_port_forward

info "Running US_FED posture tests..."
if ! run_hermetic_region_pytest "US_FED posture tests" "US_FED" "us_fed"; then
((POSTURE_FAILURES++)) || true
Expand All @@ -1659,6 +1667,8 @@ if errors:
((POSTURE_FAILURES++)) || true
fi

start_staging_port_forward

info "Running live data-residency verification (test_data_residency.py --run-integration)..."
if ! run_pytest_require_passed "Live data-residency tests" \
env -u CAGE_DEPLOYMENT_REGION \
Expand Down Expand Up @@ -1702,6 +1712,7 @@ if errors:
echo ""

if [[ "$SKIP_LIVE_SWEEP" == "false" ]]; then
start_staging_port_forward
info "Running first-party GCP live_external sweep (-m 'live_external and not partner' --run-live-external)..."
if ! run_pytest_require_passed "Phase 5 live_external sweep" \
env \
Expand Down
14 changes: 9 additions & 5 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,17 +188,21 @@ valid selection markers. Do not work around it by excluding the file from collec

Tests read variables from the process environment only. No `.env` file is
loaded, and a hermetic session (no live opt-in flag) refuses to start while a
`.env` file can reach it: litellm loads the first `.env` above its install
directory when it is imported, so the file would change results partway through
the run ([`dotenv_guard.py`](fixtures/dotenv_guard.py)). For a live run, export
the variables or pass a file explicitly:
`uv run --env-file .env pytest tests/ --run-integration`.
`.env` file can reach it ([`dotenv_guard.py`](fixtures/dotenv_guard.py)) or
while `GATEWAY_URL` is set in the environment
([`gateway_url_guard.py`](fixtures/gateway_url_guard.py)): litellm loads the
first `.env` above its install directory on import, and a set `GATEWAY_URL`
would send any hermetic test that reads it into a live or port-forwarded
gateway. Every hermetic test also asserts `"GATEWAY_URL" not in os.environ`
before and after running. For a live run, export the variables or pass a file
explicitly: `uv run --env-file .env pytest tests/ --run-integration`.
[`.env.example`](../.env.example) lists the variable names.

### Service URLs

| Variable | Default | Description |
| ----------------------------- | --------------------------------- | ------------------------------------- |
| `GATEWAY_URL` | `http://localhost:8080` _(live runs only; unset & refused in hermetic runs)_ | Governance Gateway base URL (port-forward) |
| `BACKEND_URL` | `http://localhost:8081` | Governed Financial Advisor backend |
| `LANGFUSE_HOST` | `http://localhost:3001` | Self-hosted Langfuse instance |
| `COMPLIANCE_BRIDGE_URL` | `http://localhost:3002` | Compliance Bridge API (port-forward) |
Expand Down
54 changes: 46 additions & 8 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,17 @@

from tests.fixtures.deployment_region import REGION_MARKERS
from tests.fixtures.dotenv_guard import (
LIVE_RUN_OPTIONS,
reachable_dotenv_files,
refusal,
search_starts,
)
from tests.fixtures.gateway_url_guard import (
GATEWAY_URL_ENV_VAR,
is_live_session,
)
from tests.fixtures.gateway_url_guard import (
refusal as gateway_url_refusal,
)
from tests.fixtures.selection_flags import applied_marker_names

os.environ.setdefault("LANGCHAIN_TRACING_V2", "false")
Expand All @@ -98,10 +104,12 @@
"For local testing without KMS, set CAGE_ENV=test."
)

# Sanitize cluster-internal Kubernetes DNS URLs that cannot be resolved outside GKE
# Sanitize cluster-internal Kubernetes DNS URLs that cannot be resolved outside GKE.
# GATEWAY_URL is intentionally excluded here: hermetic sessions must keep
# GATEWAY_URL unset (tests/fixtures/gateway_url_guard.py), and live sessions
# populate the localhost:8080 default in pytest_configure.
_cluster_internal_replacements = {
"BACKEND_URL": "http://localhost:18080",
"GATEWAY_URL": "http://localhost:8080",
"LANGFUSE_HOST": "http://localhost:3001",
"LANGFUSE_BASEURL": "http://localhost:3001",
"COMPLIANCE_BRIDGE_URL": "http://localhost:3002",
Expand Down Expand Up @@ -177,8 +185,9 @@ def filter(self, record: logging.LogRecord) -> bool:

def pytest_configure(config: pytest.Config) -> None:
"""Set environment variable defaults that tests expect."""
# Before anything else: no .env file may reach a hermetic session.
# Before anything else: no .env file or GATEWAY_URL may reach a hermetic session.
_refuse_reachable_dotenv(config)
_refuse_hermetic_gateway_url(config)

# One region for the whole session: US_FED for hermetic runs, the
# deployment's ConfigMap value for live runs (fails closed on mismatch).
Expand All @@ -191,7 +200,8 @@ def pytest_configure(config: pytest.Config) -> None:
os.environ["GOOGLE_CLOUD_LOCATION"] = _region_locations[_region]

_setdefault("BACKEND_URL", "http://localhost:18080")
_setdefault("GATEWAY_URL", "http://localhost:8080")
if is_live_session(config):
_setdefault(GATEWAY_URL_ENV_VAR, "http://localhost:8080")
_setdefault("LANGFUSE_HOST", "http://localhost:3001")
_setdefault("VLLM_REASONING_API_BASE", "http://localhost:8000/v1")
# OTEL_EXPORTER_OTLP_ENDPOINT: default points at the Langfuse native OTLP
Expand Down Expand Up @@ -268,14 +278,42 @@ def pytest_configure(config: pytest.Config) -> None:

def _refuse_reachable_dotenv(config: pytest.Config) -> None:
"""Refuse a hermetic session that a ``.env`` file can reach (``dotenv_guard``)."""
live_run = any(
config.getoption(option, default=False) for option in LIVE_RUN_OPTIONS
)
live_run = is_live_session(config)
reason = refusal(reachable_dotenv_files(search_starts()), live_run=live_run)
if reason is not None:
raise pytest.UsageError(reason)


def _refuse_hermetic_gateway_url(config: pytest.Config) -> None:
"""Refuse a hermetic session started with ``GATEWAY_URL`` set."""
reason = gateway_url_refusal(
os.environ.get(GATEWAY_URL_ENV_VAR), live_run=is_live_session(config)
)
if reason is not None:
raise pytest.UsageError(reason)


@pytest.fixture(autouse=True)
def assert_hermetic_gateway_url_unset(request: pytest.FixtureRequest):
"""Assert ``GATEWAY_URL`` stays unset across every test in a hermetic session."""
if is_live_session(request.config):
yield
return
assert GATEWAY_URL_ENV_VAR not in os.environ, (
f"Hermetic test {request.node.nodeid} started with "
f"{GATEWAY_URL_ENV_VAR}={os.environ.get(GATEWAY_URL_ENV_VAR)!r}; "
"GATEWAY_URL must remain unset in hermetic runs."
)
yield
if GATEWAY_URL_ENV_VAR in os.environ:
leaked = os.environ.pop(GATEWAY_URL_ENV_VAR)
pytest.fail(
f"Hermetic test {request.node.nodeid} leaked "
f"{GATEWAY_URL_ENV_VAR}={leaked!r} into os.environ; "
"use monkeypatch.setenv so it is undone at teardown."
)


def _setdefault(key: str, value: str) -> None:
"""Set an env var only when it is not already present or is a placeholder or cluster-internal."""
current = os.environ.get(key)
Expand Down
Loading
Loading