diff --git a/CHANGELOG.md b/CHANGELOG.md index 1cc01eb0..962ab5b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,11 @@ Versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Changed +- **Gate W closes `kubectl port-forward` before Phase 4's hermetic regional suites, and hermetic pytest sessions refuse or fail on `GATEWAY_URL`** (`fix(tests)`, [`staging_lifecycle.sh`](scripts/staging_lifecycle.sh), [`conftest.py`](tests/conftest.py), [`gateway_url_guard.py`](tests/fixtures/gateway_url_guard.py), [`test_gateway_url_guard.py`](tests/unit/test_gateway_url_guard.py), [`test_staging_lifecycle.py`](tests/infrastructure/test_staging_lifecycle.py)). + - `staging_lifecycle.sh` opened the staging `kubectl port-forward` tunnel during Phase 3 pre-Lula custody seeding (and again before Phase 4) without closing it before Phase 4's `us_fed`, `eu_ecb` and `apac_mas` hermetic posture suites, while `tests/conftest.py` defaulted `GATEWAY_URL` to `http://localhost:8080` in every session. Any hermetic test reading `GATEWAY_URL` would therefore hit live staging and write synthetic records into the deployed evidence chain and Lula counts. + - Phase 3 now stops its port-forward when Lula validation finishes, Phase 4 calls `ensure_no_staging_port_forward` and passes `env -u GATEWAY_URL` before running the three hermetic regional suites, and `start_staging_port_forward` runs only afterward (before live data-residency, live Langfuse isolation, and Phase 5 live sweeps). + - `tests/conftest.py` stops defaulting `GATEWAY_URL` in hermetic sessions, refuses to start a hermetic session when `GATEWAY_URL` is exported (`gateway_url_guard.py`), and asserts `"GATEWAY_URL" not in os.environ` before and after every hermetic test; live sessions (`--run-integration`, `--run-live-external`, `--run-partner-integration`, `--run-e2e`) keep the `http://localhost:8080` default. + - **Evidence cold-store residency is enforced in every enforcing posture (`staging` included) against the WORM bucket's immutable cloud location** (`fix(governance)!`, [`residency.py`](src/gateway/governance/evidence/residency.py), [`factory.py`](src/gateway/governance/evidence/factory.py), [`residency.json`](config/compliance/residency.json), [`worm_bucket/outputs.tf`](infra/modules/worm_bucket/outputs.tf), [`compliance_bridge/main.tf`](infra/modules/compliance_bridge/main.tf), [`POAM.md`](docs/POAM.md), [`gate_w_certification.json`](compliance/gate_w_certification.json)). - `resolve_cold_store_bucket()` skipped geographic validation whenever `CAGE_ENV` was in a hardcoded list that included `staging`, `uat` and `preprod`. Gate W therefore never exercised runtime residency, and the bypass masked a latent production failure (`POAM-2026-220`): under `CAGE_ENV=prod` and `US_FED` with `-evidence-worm-prod` and no `GOOGLE_CLOUD_LOCATION`, `compliance-bridge` failed lifespan startup with `ResidencyViolationError` while the identical inputs passed under `staging`. - `resolve_cold_store_bucket()` now enforces residency whenever `is_enforcing()` is true (`production`, `staging`, `local`, or an unknown `CAGE_ENV`), skipping only in `dev`, `test` and `ci`. Bucket-name prefixes (`allowed_prefixes` / `allow_prefixes`), `GOOGLE_CLOUD_LOCATION` / `AWS_REGION` fallbacks, and the `LOCAL` rule / `default_bucket` are removed; residency checks only `location` or `EVIDENCE_COLD_STORE_LOCATION` against `residency.json`. `get_cold_store()` validates explicit `bucket=` overrides through `resolve_cold_store_bucket(bucket_override=bucket)` instead of short-circuiting. diff --git a/docs/EVALUATOR_GUIDE.md b/docs/EVALUATOR_GUIDE.md index 6899bdf9..fe6ccbaf 100644 --- a/docs/EVALUATOR_GUIDE.md +++ b/docs/EVALUATOR_GUIDE.md @@ -241,7 +241,7 @@ Every run populates a self-contained audit bundle under ``: When an evaluator runs `make test-fast` (`uv run pytest tests/ -m "local or unit" -n auto --dist loadscope --no-cov -p no:langsmith -q` in [`Makefile`](../Makefile)), CAGE intentionally runs only hermetic offline tests and prints an explicit notice listing every live suite that did not execute. -1. **Fail-Closed Selection-Marker Contract**: `_assert_selection_markers()` in [`tests/conftest.py`](../tests/conftest.py) requires every collected test to carry at least one marker from `SELECTION_MARKERS` (`unit`, `local`, `integration`, `load`, `chaos`, `live_external`, `partner_integration`, `e2e`; `MARKER_GUARD_EXEMPT_MODULES` is empty). Any unmarked test aborts collection with `PYTEST MARKER CONTRACT VIOLATION` so tests cannot silently vanish from CI gates. Hermetic sessions also run `_refuse_reachable_dotenv()` ([`tests/fixtures/dotenv_guard.py`](../tests/fixtures/dotenv_guard.py)) and abort if a `.env` file is reachable on disk, preventing ambient stack credentials from altering offline test behavior. +1. **Fail-Closed Selection-Marker Contract**: `_assert_selection_markers()` in [`tests/conftest.py`](../tests/conftest.py) requires every collected test to carry at least one marker from `SELECTION_MARKERS` (`unit`, `local`, `integration`, `load`, `chaos`, `live_external`, `partner_integration`, `e2e`; `MARKER_GUARD_EXEMPT_MODULES` is empty). Any unmarked test aborts collection with `PYTEST MARKER CONTRACT VIOLATION` so tests cannot silently vanish from CI gates. Hermetic sessions also run `_refuse_reachable_dotenv()` ([`tests/fixtures/dotenv_guard.py`](../tests/fixtures/dotenv_guard.py)) and `_refuse_hermetic_gateway_url()` ([`tests/fixtures/gateway_url_guard.py`](../tests/fixtures/gateway_url_guard.py)), aborting if a `.env` file is reachable on disk or `GATEWAY_URL` is set in the environment and asserting `GATEWAY_URL` stays unset across every hermetic test. 2. **Why `make test-fast` Skips Live Suites — and How `pytest_terminal_summary` Surfaces Them**: Suites marked `integration`, `live_external`, `partner_integration`, `e2e`, or `chaos` require live Kubernetes services, Cloud KMS/WORM/CAS APIs, external partner sandboxes, or Redis failover injection. `make test-fast` deselects them via `-m "local or unit"`, and when pytest is invoked without `-m`, `pytest_collection_modifyitems()` in [`tests/conftest.py`](../tests/conftest.py) auto-skips them unless their opt-in CLI flag (`--run-integration`, `--run-live-external`, `--run-partner-integration`, `--run-e2e`, `--run-chaos`) is supplied. To prevent a green hermetic unit run from being misread as live infrastructure validation, `pytest_deselected()` and `pytest_terminal_summary()` (via `format_unexecuted_live_suites_notice()` in [`tests/conftest.py`](../tests/conftest.py)) emit a controller-level banner at the end of the session: ```text --- live / opt-in suites not executed --- diff --git a/docs/operations/GKE_TEST_RUNBOOK.md b/docs/operations/GKE_TEST_RUNBOOK.md index a5c9ded8..d384f309 100644 --- a/docs/operations/GKE_TEST_RUNBOOK.md +++ b/docs/operations/GKE_TEST_RUNBOOK.md @@ -183,7 +183,7 @@ Asymmetric signing keys (`EC_SIGN_P256_SHA256`) live in keyring `cage-signing-@.iam.gserviceaccount.com`) via [`deployment/docker/cloudbuild.image.yaml`](../../deployment/docker/cloudbuild.image.yaml), [`scripts/build_images.sh`](../../scripts/build_images.sh), [`scripts/mirror_and_attest_images.sh`](../../scripts/mirror_and_attest_images.sh), and [`scripts/attest_image.sh`](../../scripts/attest_image.sh) (`gcloud beta container binauthz attestations sign-and-create` + fail-closed `binauthz attestations list` verification) to sign Binary Authorization attestations for every container image digest in `var.image_digests`, which [`scripts/render_image_digests.sh`](../../scripts/render_image_digests.sh) renders per deployment for exactly the images the profile deploys (no committed digests, POAM-2026-131). **Local Workstation KMS Verification:** -- In [`tests/conftest.py`](../../tests/conftest.py), running client test commands with `CAGE_ENV=test` allows the test harness to use software-backed signing for local assertions while directing HTTP traffic to live cluster endpoints (`BACKEND_URL=http://localhost:8081`, `GATEWAY_URL=http://localhost:8080`). +- In [`tests/conftest.py`](../../tests/conftest.py), running live client test commands (`--run-integration` / `--run-e2e`) with `CAGE_ENV=test` allows the test harness to use software-backed signing for local assertions while directing HTTP traffic to live cluster endpoints (`BACKEND_URL=http://localhost:8081`, `GATEWAY_URL=http://localhost:8080`; hermetic runs keep `GATEWAY_URL` unset). - To verify Gateway responses signed by the live cluster's `gateway-seal` KMS key, fetch the public key via the Gateway JWKS endpoint (`curl -s http://localhost:8080/governance/.well-known/jwks.json`) or extract the cached PEM from the pod: ```bash kubectl exec -n governance-stack deploy/gateway -c gateway -- cat /tmp/kms_governance_public.pem > /tmp/kms_governance_public.pem diff --git a/scripts/staging_lifecycle.sh b/scripts/staging_lifecycle.sh index 9cc8b0fa..9e2f30c5 100755 --- a/scripts/staging_lifecycle.sh +++ b/scripts/staging_lifecycle.sh @@ -584,7 +584,7 @@ run_hermetic_region_pytest() { local marker="$3" if [[ -n "${CAGE_HERMETIC_PYTEST_WRAPPER:-}" ]]; then run_pytest_require_passed "$label" \ - "$CAGE_HERMETIC_PYTEST_WRAPPER" env CAGE_DEPLOYMENT_REGION="$region_val" ./.venv/bin/pytest tests/ -m "$marker" -v --tb=short + env -u GATEWAY_URL "$CAGE_HERMETIC_PYTEST_WRAPPER" env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION="$region_val" ./.venv/bin/pytest tests/ -m "$marker" -v --tb=short return $? fi if [[ -f "$REPO_ROOT/.env" ]]; then @@ -594,7 +594,7 @@ run_hermetic_region_pytest() { local rc=0 set +e run_pytest_require_passed "$label" \ - env CAGE_DEPLOYMENT_REGION="$region_val" uv run pytest "$REPO_ROOT/tests/" -m "$marker" -v --tb=short + env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION="$region_val" uv run pytest "$REPO_ROOT/tests/" -m "$marker" -v --tb=short rc=$? set -e restore_shielded_dotenv @@ -1628,13 +1628,17 @@ if errors: error "$LULA_FAILURES Lula validation check(s) failed" error "POAM-024 is NOT satisfied — security posture incomplete" fi + stop_staging_port_forward else warn "Lula validation skipped (--skip-lula flag)" fi - if [[ "$SKIP_POSTURE" == "false" || "$SKIP_LIVE_SWEEP" == "false" ]]; then - start_staging_port_forward - fi + ensure_no_staging_port_forward() { + if [[ -x "$REPO_ROOT/scripts/port_forward_staging.sh" ]]; then + "$REPO_ROOT/scripts/port_forward_staging.sh" --stop >/dev/null 2>&1 || true + fi + PORT_FORWARD_ACTIVE=false + } # ─── Phase 4: Region Posture, Live Residency & Live Langfuse (FAIL CLOSED) ─ if [[ "$SKIP_POSTURE" == "false" ]]; then @@ -1644,6 +1648,10 @@ if errors: echo -e "${CYAN}╚══════════════════════════════════════════════════════════════════╝${RESET}" echo "" + # Hermetic regional suites must never run while kubectl port-forward exposes + # live staging on localhost:8080 (even if started externally before --skip-lula). + ensure_no_staging_port_forward + info "Running US_FED posture tests..." if ! run_hermetic_region_pytest "US_FED posture tests" "US_FED" "us_fed"; then ((POSTURE_FAILURES++)) || true @@ -1659,6 +1667,8 @@ if errors: ((POSTURE_FAILURES++)) || true fi + start_staging_port_forward + info "Running live data-residency verification (test_data_residency.py --run-integration)..." if ! run_pytest_require_passed "Live data-residency tests" \ env -u CAGE_DEPLOYMENT_REGION \ @@ -1702,6 +1712,7 @@ if errors: echo "" if [[ "$SKIP_LIVE_SWEEP" == "false" ]]; then + start_staging_port_forward info "Running first-party GCP live_external sweep (-m 'live_external and not partner' --run-live-external)..." if ! run_pytest_require_passed "Phase 5 live_external sweep" \ env \ diff --git a/tests/README.md b/tests/README.md index 298e969c..b922f9e5 100644 --- a/tests/README.md +++ b/tests/README.md @@ -188,17 +188,21 @@ valid selection markers. Do not work around it by excluding the file from collec Tests read variables from the process environment only. No `.env` file is loaded, and a hermetic session (no live opt-in flag) refuses to start while a -`.env` file can reach it: litellm loads the first `.env` above its install -directory when it is imported, so the file would change results partway through -the run ([`dotenv_guard.py`](fixtures/dotenv_guard.py)). For a live run, export -the variables or pass a file explicitly: -`uv run --env-file .env pytest tests/ --run-integration`. +`.env` file can reach it ([`dotenv_guard.py`](fixtures/dotenv_guard.py)) or +while `GATEWAY_URL` is set in the environment +([`gateway_url_guard.py`](fixtures/gateway_url_guard.py)): litellm loads the +first `.env` above its install directory on import, and a set `GATEWAY_URL` +would send any hermetic test that reads it into a live or port-forwarded +gateway. Every hermetic test also asserts `"GATEWAY_URL" not in os.environ` +before and after running. For a live run, export the variables or pass a file +explicitly: `uv run --env-file .env pytest tests/ --run-integration`. [`.env.example`](../.env.example) lists the variable names. ### Service URLs | Variable | Default | Description | | ----------------------------- | --------------------------------- | ------------------------------------- | +| `GATEWAY_URL` | `http://localhost:8080` _(live runs only; unset & refused in hermetic runs)_ | Governance Gateway base URL (port-forward) | | `BACKEND_URL` | `http://localhost:8081` | Governed Financial Advisor backend | | `LANGFUSE_HOST` | `http://localhost:3001` | Self-hosted Langfuse instance | | `COMPLIANCE_BRIDGE_URL` | `http://localhost:3002` | Compliance Bridge API (port-forward) | diff --git a/tests/conftest.py b/tests/conftest.py index 3448d503..e601c487 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -68,11 +68,17 @@ from tests.fixtures.deployment_region import REGION_MARKERS from tests.fixtures.dotenv_guard import ( - LIVE_RUN_OPTIONS, reachable_dotenv_files, refusal, search_starts, ) +from tests.fixtures.gateway_url_guard import ( + GATEWAY_URL_ENV_VAR, + is_live_session, +) +from tests.fixtures.gateway_url_guard import ( + refusal as gateway_url_refusal, +) from tests.fixtures.selection_flags import applied_marker_names os.environ.setdefault("LANGCHAIN_TRACING_V2", "false") @@ -98,10 +104,12 @@ "For local testing without KMS, set CAGE_ENV=test." ) -# Sanitize cluster-internal Kubernetes DNS URLs that cannot be resolved outside GKE +# Sanitize cluster-internal Kubernetes DNS URLs that cannot be resolved outside GKE. +# GATEWAY_URL is intentionally excluded here: hermetic sessions must keep +# GATEWAY_URL unset (tests/fixtures/gateway_url_guard.py), and live sessions +# populate the localhost:8080 default in pytest_configure. _cluster_internal_replacements = { "BACKEND_URL": "http://localhost:18080", - "GATEWAY_URL": "http://localhost:8080", "LANGFUSE_HOST": "http://localhost:3001", "LANGFUSE_BASEURL": "http://localhost:3001", "COMPLIANCE_BRIDGE_URL": "http://localhost:3002", @@ -177,8 +185,9 @@ def filter(self, record: logging.LogRecord) -> bool: def pytest_configure(config: pytest.Config) -> None: """Set environment variable defaults that tests expect.""" - # Before anything else: no .env file may reach a hermetic session. + # Before anything else: no .env file or GATEWAY_URL may reach a hermetic session. _refuse_reachable_dotenv(config) + _refuse_hermetic_gateway_url(config) # One region for the whole session: US_FED for hermetic runs, the # deployment's ConfigMap value for live runs (fails closed on mismatch). @@ -191,7 +200,8 @@ def pytest_configure(config: pytest.Config) -> None: os.environ["GOOGLE_CLOUD_LOCATION"] = _region_locations[_region] _setdefault("BACKEND_URL", "http://localhost:18080") - _setdefault("GATEWAY_URL", "http://localhost:8080") + if is_live_session(config): + _setdefault(GATEWAY_URL_ENV_VAR, "http://localhost:8080") _setdefault("LANGFUSE_HOST", "http://localhost:3001") _setdefault("VLLM_REASONING_API_BASE", "http://localhost:8000/v1") # OTEL_EXPORTER_OTLP_ENDPOINT: default points at the Langfuse native OTLP @@ -268,14 +278,42 @@ def pytest_configure(config: pytest.Config) -> None: def _refuse_reachable_dotenv(config: pytest.Config) -> None: """Refuse a hermetic session that a ``.env`` file can reach (``dotenv_guard``).""" - live_run = any( - config.getoption(option, default=False) for option in LIVE_RUN_OPTIONS - ) + live_run = is_live_session(config) reason = refusal(reachable_dotenv_files(search_starts()), live_run=live_run) if reason is not None: raise pytest.UsageError(reason) +def _refuse_hermetic_gateway_url(config: pytest.Config) -> None: + """Refuse a hermetic session started with ``GATEWAY_URL`` set.""" + reason = gateway_url_refusal( + os.environ.get(GATEWAY_URL_ENV_VAR), live_run=is_live_session(config) + ) + if reason is not None: + raise pytest.UsageError(reason) + + +@pytest.fixture(autouse=True) +def assert_hermetic_gateway_url_unset(request: pytest.FixtureRequest): + """Assert ``GATEWAY_URL`` stays unset across every test in a hermetic session.""" + if is_live_session(request.config): + yield + return + assert GATEWAY_URL_ENV_VAR not in os.environ, ( + f"Hermetic test {request.node.nodeid} started with " + f"{GATEWAY_URL_ENV_VAR}={os.environ.get(GATEWAY_URL_ENV_VAR)!r}; " + "GATEWAY_URL must remain unset in hermetic runs." + ) + yield + if GATEWAY_URL_ENV_VAR in os.environ: + leaked = os.environ.pop(GATEWAY_URL_ENV_VAR) + pytest.fail( + f"Hermetic test {request.node.nodeid} leaked " + f"{GATEWAY_URL_ENV_VAR}={leaked!r} into os.environ; " + "use monkeypatch.setenv so it is undone at teardown." + ) + + def _setdefault(key: str, value: str) -> None: """Set an env var only when it is not already present or is a placeholder or cluster-internal.""" current = os.environ.get(key) diff --git a/tests/fixtures/gateway_url_guard.py b/tests/fixtures/gateway_url_guard.py new file mode 100644 index 00000000..76dae543 --- /dev/null +++ b/tests/fixtures/gateway_url_guard.py @@ -0,0 +1,71 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Keep ``GATEWAY_URL`` unset across every hermetic test session. + +Hermetic suites (including the ``us_fed``, ``eu_ecb``, and ``apac_mas`` +regional posture suites) must never target a live or port-forwarded gateway. +If ``GATEWAY_URL`` is present in the environment—either exported in the caller's +shell or populated as a default by ``tests/conftest.py``—the first hermetic test +that reads ``GATEWAY_URL`` reaches whatever listens on that endpoint (such as a +``kubectl port-forward`` tunnel on ``localhost:8080``) and lands unmarked +synthetic traffic in the deployed evidence chain and Lula counts. + +Two rules hold hermetic runs closed by construction: + +1. Session startup (:func:`refusal`): a hermetic session (none of + :data:`~tests.fixtures.dotenv_guard.LIVE_RUN_OPTIONS` passed) refuses to + start if ``GATEWAY_URL`` is set in ``os.environ``. +2. Per-test invariant (``assert_hermetic_gateway_url_unset`` in + ``tests/conftest.py``): every test in a hermetic session asserts + ``"GATEWAY_URL" not in os.environ`` before running and after fixture + teardown, so any test that mutates ``GATEWAY_URL`` without ``monkeypatch`` + fails closed immediately. + +Live sessions (:func:`is_live_session`) are exempt and keep the +``http://localhost:8080`` port-forward default when ``GATEWAY_URL`` is unset or +holds a cluster-internal placeholder. +""" + +from __future__ import annotations + +import pytest + +from tests.fixtures.dotenv_guard import LIVE_RUN_OPTIONS + +GATEWAY_URL_ENV_VAR = "GATEWAY_URL" + + +def is_live_session(config: pytest.Config) -> bool: + """Return ``True`` when any live-target CLI flag is enabled on ``config``.""" + return any( + bool(config.getoption(option, default=False)) for option in LIVE_RUN_OPTIONS + ) + + +def refusal(gateway_url: str | None, *, live_run: bool) -> str | None: + """Return why a session must not start, or ``None`` when allowed.""" + if live_run or gateway_url is None: + return None + flags = ", ".join(LIVE_RUN_OPTIONS) + return ( + "Refusing to start a hermetic test session with " + f"{GATEWAY_URL_ENV_VAR}={gateway_url!r} set in the environment.\n" + "Hermetic and regional posture tests must run with GATEWAY_URL unset so\n" + "no test can reach a live or port-forwarded gateway (e.g. localhost:8080)\n" + "and record synthetic traffic in a deployed evidence chain.\n" + "Unset it for hermetic runs:\n" + " env -u GATEWAY_URL uv run pytest ...\n" + f"Or pass a live-run flag ({flags}) when targeting a deployed stack." + ) diff --git a/tests/infrastructure/test_staging_lifecycle.py b/tests/infrastructure/test_staging_lifecycle.py index 541b14f4..baa07c93 100644 --- a/tests/infrastructure/test_staging_lifecycle.py +++ b/tests/infrastructure/test_staging_lifecycle.py @@ -185,6 +185,7 @@ def _base_env(bin_dir: Path, **overrides: str) -> dict[str, str]: "CAGE_E2E_SUITE_SCRIPT", "DEPLOYER_EGRESS_IP", "COMPLIANCE_BRIDGE_URL", + "GATEWAY_URL", "CAGE_AUDIT_CORROBORATION_SCRIPT", "CAGE_BINAUTHZ_PROBE_SCRIPT", "CAGE_WORM_ANCHOR_SCRIPT", @@ -1777,15 +1778,15 @@ def test_staging_lifecycle_clean_room_dotenv_isolation_wrapper_and_e2e_suite_wir assert "--var registry_url=" in calls assert "operated-legacy-proj" not in calls assert ( - "wrapper env CAGE_DEPLOYMENT_REGION=US_FED ./.venv/bin/pytest tests/ -m us_fed" + "wrapper env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION=US_FED ./.venv/bin/pytest tests/ -m us_fed" in calls ) assert ( - "wrapper env CAGE_DEPLOYMENT_REGION=EU_ECB ./.venv/bin/pytest tests/ -m eu_ecb" + "wrapper env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION=EU_ECB ./.venv/bin/pytest tests/ -m eu_ecb" in calls ) assert ( - "wrapper env CAGE_DEPLOYMENT_REGION=APAC_MAS ./.venv/bin/pytest tests/ -m apac_mas" + "wrapper env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION=APAC_MAS ./.venv/bin/pytest tests/ -m apac_mas" in calls ) assert "--generate-ephemeral-jwks" in calls @@ -2593,4 +2594,189 @@ def test_staging_lifecycle_phase3_rejects_startup_grace_remarks_on_host_and_job_ ) +def _install_port_forward_and_env_probes( + repo: Path, bin_dir: Path, log_file: Path, pf_state: Path +) -> None: + """Install a stateful port_forward_staging.sh stub and record tunnel/env state on uv calls.""" + _write_executable( + repo / "scripts" / "port_forward_staging.sh", + f"""#!/usr/bin/env bash +echo "port_forward_staging.sh $*" >> "{log_file}" +if [[ "$*" == *"--stop"* ]]; then + rm -f "{pf_state}" + exit 0 +fi +touch "{pf_state}" +exit 0 +""", + ) + e2e_stub = repo / "tests" / "e2e" / "run_gke_e2e_suite.py" + e2e_stub.parent.mkdir(parents=True, exist_ok=True) + e2e_stub.write_text("# stub\n", encoding="utf-8") + + uv_path = bin_dir / "uv" + uv_body = uv_path.read_text(encoding="utf-8") + uv_body = uv_body.replace( + f'echo "uv $*" >> "{log_file}"\n', + f'_pf="closed"\n[[ -f "{pf_state}" ]] && _pf="open"\n' + '_gw="unset"\n[[ -n "${GATEWAY_URL+x}" ]] && _gw="set:${GATEWAY_URL}"\n' + f'echo "uv [pf=${{_pf}} gw=${{_gw}}] $*" >> "{log_file}"\n' + 'if [[ "$*" == *"run_gke_e2e_suite.py"* ]]; then\n' + " echo '{\"passed\": true}'\n" + " exit 0\n" + "fi\n", + ) + _write_executable(uv_path, uv_body) + + +def test_staging_lifecycle_closes_port_forward_and_unsets_gateway_url_for_phase4_hermetic_suites( + tmp_path: Path, +) -> None: + repo, bin_dir, log_file = _setup_fake_repo(tmp_path) + pf_state = tmp_path / "pf_active" + # Pre-seed an external tunnel and an exported GATEWAY_URL in the caller's shell. + pf_state.write_text("preexisting\n", encoding="utf-8") + _install_port_forward_and_env_probes(repo, bin_dir, log_file, pf_state) + + env = _base_env( + bin_dir, + GOOGLE_CLOUD_PROJECT="demo-staging-proj", + GATEWAY_URL="http://localhost:8080", + ) + proc = subprocess.run( + [ + "bash", + str(repo / "scripts" / "staging_lifecycle.sh"), + "--skip-telemetry", + "--skip-rebuild", + ], + env=env, + capture_output=True, + text=True, + check=False, + ) + assert proc.returncode == 0, proc.stderr + lines = log_file.read_text(encoding="utf-8").splitlines() + + def _find_uv_line(substr: str) -> tuple[int, str]: + for idx, line in enumerate(lines): + if line.startswith("uv [") and substr in line: + return idx, line + raise AssertionError( + f"Missing expected uv invocation matching {substr!r} in:\n" + + "\n".join(lines) + ) + + seed_idx, seed_line = _find_uv_line("--seed-only") + assert "[pf=open " in seed_line + + us_idx, us_line = _find_uv_line("-m us_fed") + eu_idx, eu_line = _find_uv_line("-m eu_ecb") + apac_idx, apac_line = _find_uv_line("-m apac_mas") + for reg_line in (us_line, eu_line, apac_line): + assert "[pf=closed gw=unset]" in reg_line, reg_line + + res_idx, res_line = _find_uv_line("test_data_residency.py --run-integration") + lf_idx, lf_line = _find_uv_line("verify_langfuse_posture.py --posture staging") + live_ext_idx, live_ext_line = _find_uv_line("--run-live-external") + integ_idx, integ_line = _find_uv_line("-m integration --run-integration") + partner_idx, partner_line = _find_uv_line("--run-partner-integration") + + assert ( + seed_idx + < us_idx + < eu_idx + < apac_idx + < res_idx + < lf_idx + < live_ext_idx + < integ_idx + < partner_idx + ) + for live_line in (res_line, lf_line, live_ext_line, integ_line, partner_line): + assert "[pf=open " in live_line, live_line + + +def test_staging_lifecycle_port_forward_isolation_across_skip_lula_and_skip_posture( + tmp_path: Path, +) -> None: + # 1. --skip-lula with a pre-existing external tunnel + CAGE_HERMETIC_PYTEST_WRAPPER: + # ensure_no_staging_port_forward still closes the pre-existing tunnel and strips + # GATEWAY_URL before the wrapper executes. + repo_a, bin_a, log_a = _setup_fake_repo(tmp_path / "skip_lula") + pf_a = tmp_path / "pf_a" + pf_a.write_text("external\n", encoding="utf-8") + _install_port_forward_and_env_probes(repo_a, bin_a, log_a, pf_a) + + wrapper_a = tmp_path / "wrapper_a.sh" + _write_executable( + wrapper_a, + f"""#!/usr/bin/env bash +_pf="closed" +[[ -f "{pf_a}" ]] && _pf="open" +_gw="unset" +[[ -n "${{GATEWAY_URL+x}}" ]] && _gw="set:${{GATEWAY_URL}}" +echo "wrapper [pf=${{_pf}} gw=${{_gw}}] $*" >> "{log_a}" +echo "collected 3 items" +echo "3 passed in 0.05s" +exit 0 +""", + ) + proc_a = subprocess.run( + [ + "bash", + str(repo_a / "scripts" / "staging_lifecycle.sh"), + "--skip-lula", + "--skip-live-sweep", + "--skip-telemetry", + "--skip-rebuild", + ], + env=_base_env( + bin_a, + GOOGLE_CLOUD_PROJECT="demo-staging-proj", + GATEWAY_URL="http://localhost:8080", + CAGE_HERMETIC_PYTEST_WRAPPER=str(wrapper_a), + ), + capture_output=True, + text=True, + check=False, + ) + assert proc_a.returncode == 0, proc_a.stderr + calls_a = log_a.read_text(encoding="utf-8") + for marker in ("us_fed", "eu_ecb", "apac_mas"): + assert ( + f"wrapper [pf=closed gw=unset] env -u GATEWAY_URL CAGE_DEPLOYMENT_REGION={marker.upper()} ./.venv/bin/pytest tests/ -m {marker}" + in calls_a + ) + assert ( + "uv [pf=open " in calls_a + and "test_data_residency.py --run-integration" in calls_a + ) + + # 2. --skip-posture (with Phase 5 live sweeps enabled): Phase 3 closes its tunnel + # after Lula and Phase 5 re-opens it before --run-live-external. + repo_b, bin_b, log_b = _setup_fake_repo(tmp_path / "skip_posture") + pf_b = tmp_path / "pf_b" + _install_port_forward_and_env_probes(repo_b, bin_b, log_b, pf_b) + proc_b = subprocess.run( + [ + "bash", + str(repo_b / "scripts" / "staging_lifecycle.sh"), + "--skip-posture", + "--skip-telemetry", + "--skip-rebuild", + ], + env=_base_env(bin_b, GOOGLE_CLOUD_PROJECT="demo-staging-proj"), + capture_output=True, + text=True, + check=False, + ) + assert proc_b.returncode == 0, proc_b.stderr + lines_b = log_b.read_text(encoding="utf-8").splitlines() + assert any( + line.startswith("uv [pf=open ") and "--run-live-external" in line + for line in lines_b + ) + + # Regression coverage binding: POAM-024, POAM-2026-159, POAM-2026-192, POAM-2026-197, POAM-2026-202, POAM-2026-203, POAM-2026-204, POAM-2026-205, POAM-2026-206, POAM-2026-207, POAM-2026-215, POAM-2026-216 diff --git a/tests/unit/test_gateway_url_guard.py b/tests/unit/test_gateway_url_guard.py new file mode 100644 index 00000000..a7da4e2e --- /dev/null +++ b/tests/unit/test_gateway_url_guard.py @@ -0,0 +1,196 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Contract tests for the hermetic ``GATEWAY_URL`` guard. + +Pins ``tests/fixtures/gateway_url_guard.py`` and its wiring in +``tests/conftest.py``: + +* ``GATEWAY_URL`` is unset across unmarked and region-marked (``us_fed``, + ``eu_ecb``, ``apac_mas``, ``each_region``) hermetic tests. +* Starting a hermetic pytest session with ``GATEWAY_URL`` exported fails closed + with ``pytest.ExitCode.USAGE_ERROR``. +* Live-flagged sessions remain exempt. +* A hermetic test that mutates ``os.environ["GATEWAY_URL"]`` without + ``monkeypatch`` fails closed at teardown and is cleaned up before the next + test runs. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from tests.fixtures.dotenv_guard import LIVE_RUN_OPTIONS +from tests.fixtures.gateway_url_guard import ( + GATEWAY_URL_ENV_VAR, + is_live_session, + refusal, +) + +pytestmark = [pytest.mark.unit, pytest.mark.local] + + +def _fake_config(*enabled_options: str) -> SimpleNamespace: + enabled = set(enabled_options) + return SimpleNamespace( + getoption=lambda opt, default=False: True if opt in enabled else default + ) + + +def test_is_live_session_matches_every_live_run_option() -> None: + assert is_live_session(_fake_config()) is False # type: ignore[arg-type] + assert is_live_session(_fake_config("--run-chaos")) is False # type: ignore[arg-type] + for opt in LIVE_RUN_OPTIONS: + assert is_live_session(_fake_config(opt)) is True, opt # type: ignore[arg-type] + + +def test_refusal_allows_unset_gateway_url_in_hermetic_session() -> None: + assert refusal(None, live_run=False) is None + + +@pytest.mark.parametrize( + "exported", ["http://localhost:8080", "https://gw.example", ""] +) +def test_refusal_rejects_set_gateway_url_in_hermetic_session(exported: str) -> None: + reason = refusal(exported, live_run=False) + assert reason is not None + assert f"GATEWAY_URL={exported!r}" in reason + assert "env -u GATEWAY_URL" in reason + + +def test_refusal_exempts_live_session() -> None: + assert refusal("http://localhost:8080", live_run=True) is None + + +def test_unmarked_hermetic_test_has_gateway_url_unset() -> None: + assert GATEWAY_URL_ENV_VAR not in os.environ + + +@pytest.mark.us_fed +def test_us_fed_hermetic_test_has_gateway_url_unset() -> None: + assert GATEWAY_URL_ENV_VAR not in os.environ + + +@pytest.mark.eu_ecb +def test_eu_ecb_hermetic_test_has_gateway_url_unset() -> None: + assert GATEWAY_URL_ENV_VAR not in os.environ + + +@pytest.mark.apac_mas +def test_apac_mas_hermetic_test_has_gateway_url_unset() -> None: + assert GATEWAY_URL_ENV_VAR not in os.environ + + +def test_each_region_hermetic_test_has_gateway_url_unset(each_region: str) -> None: + assert each_region in {"US_FED", "EU_ECB", "APAC_MAS"} + assert GATEWAY_URL_ENV_VAR not in os.environ + + +def test_monkeypatch_setenv_gateway_url_cleans_up_cleanly( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv(GATEWAY_URL_ENV_VAR, "http://example.test:8080") + assert os.environ[GATEWAY_URL_ENV_VAR] == "http://example.test:8080" + + +def _run_subsession( + args: list[str], *, cwd: Path, extra_env: dict[str, str] | None = None +) -> subprocess.CompletedProcess[str]: + env = { + k: v + for k, v in os.environ.items() + if not k.startswith("PYTEST_") and k != GATEWAY_URL_ENV_VAR + } + if extra_env: + env.update(extra_env) + return subprocess.run( + [ + sys.executable, + "-m", + "pytest", + *args, + "-q", + "-n", + "0", + "-p", + "no:randomly", + "-p", + "no:cacheprovider", + ], + cwd=cwd, + env=env, + capture_output=True, + text=True, + timeout=180, + check=False, + ) + + +def test_conftest_refuses_hermetic_session_when_gateway_url_exported( + tmp_path: Path, +) -> None: + result = _run_subsession( + [str(Path(__file__).resolve()), "--collect-only"], + cwd=tmp_path, + extra_env={GATEWAY_URL_ENV_VAR: "http://localhost:8080"}, + ) + assert result.returncode == pytest.ExitCode.USAGE_ERROR, ( + result.stdout + result.stderr + ) + assert "Refusing to start a hermetic test session" in result.stderr + assert "GATEWAY_URL='http://localhost:8080'" in result.stderr + + +def test_conftest_starts_hermetic_session_when_gateway_url_unset( + tmp_path: Path, +) -> None: + result = _run_subsession( + [str(Path(__file__).resolve()), "--collect-only"], + cwd=tmp_path, + ) + assert result.returncode == pytest.ExitCode.OK, result.stdout + result.stderr + + +def test_probe_leaks_gateway_url_when_enabled() -> None: + if os.environ.get("_CAGE_TEST_PROBE_LEAK_GATEWAY_URL") == "1": + os.environ[GATEWAY_URL_ENV_VAR] = "http://leaked.invalid:8080" + + +def test_probe_followup_sees_gateway_url_unset() -> None: + assert GATEWAY_URL_ENV_VAR not in os.environ + + +def test_conftest_fails_test_that_leaks_gateway_url_and_isolates_next_test( + tmp_path: Path, +) -> None: + this_file = str(Path(__file__).resolve()) + result = _run_subsession( + [ + f"{this_file}::test_probe_leaks_gateway_url_when_enabled", + f"{this_file}::test_probe_followup_sees_gateway_url_unset", + ], + cwd=tmp_path, + extra_env={"_CAGE_TEST_PROBE_LEAK_GATEWAY_URL": "1"}, + ) + combined = result.stdout + result.stderr + assert result.returncode == pytest.ExitCode.TESTS_FAILED, combined + assert "leaked GATEWAY_URL='http://leaked.invalid:8080' into os.environ" in combined + assert "2 passed" in combined + assert "1 error" in combined