fix: redact credentials in logs at the model layer

[Zivxx]

Context

Follow-up to #902. A customer reported credentials still leaking into logs after that fix — this time via opal_client/data/updater.py: logger.exception(...) serialized the full DataSourceEntry, whose config carries auth headers and data the payload.

Rather than patch one line and wait for the next path, this addresses the root cause at the model layer and then sweeps the codebase (with multi-agent adversarial review) for every related vector.

1. Central fix — model-layer redaction

A small RedactedReprMixin (opal_common/logging_utils/redaction.py) overrides __repr_args__ so repr()/str() — and therefore loguru's serialize path — mask secret-bearing fields. The redaction set is unioned across the MRO, so a subclass can only ever add protection, never silently drop it.

Model	Redacted fields
FetcherConfig / HttpFetcherConfig	headers, data
FastApiRpcFetchConfig	rpc_arguments
DataSourceEntry	config, data
FetchEvent	config

This protects every current and future log site at once, and propagates transitively to container models (DataUpdate, DataSourceConfig, DataUpdateReport).

Why it's safe (no functional impact)

__repr_args__ only feeds repr()/str(). Every functional path uses something else and is unaffected — verified empirically against pydantic 1.10 and 2.13:

• Transport: update.dict(by_alias=True) / .json(); event reconstruction via .dict().
• Hashing / dedup keys: calc_hash uses config.json(); repos_last_fetched keys on source id/url.
• Equality / copy / schema / pickle / jsonable_encoder: none call __repr_args__.

2. URL credential redaction

DataSourceEntry.url and git repo URLs can embed credentials (https://user:token@host). Added http_utils.redact_url() / redact_url_in_text() and applied them to every URL log/exception in data/updater.py, data_update_publisher.py, repo_cloner.py, and git_fetcher.py. The functional clone_repository() call still gets the real URL.

3. Traceback leak — LOG_DIAGNOSE default flipped to False

loguru's diagnose=True renders raw local-variable values in tracebacks (e.g. a bare headers dict, or opa_client._get_auth_headers()'s Bearer token), which bypasses model-level redaction entirely. Default flipped True→False (matches loguru's own production guidance); operators can opt back in for local debugging.

4. Verbose git tracing gated

git_utils/env.py forced GIT_TRACE=1 + GIT_CURL_VERBOSE=1 on every SSH clone, making git dump HTTP Authorization headers to captured stderr. Now only enabled when LOG_DIAGNOSE is on.

Spot fixes (belt-and-suspenders)

Sites that logged whole credential-bearing objects now log only safe, redacted fields: data/updater.py (×3), data_update_publisher.py, data/api.py (dropped the config model), fastapi_rpc_fetch_provider.py (stopped logging rpc_arguments).

⚠️ Reviewer note — one intentional visible change

The publish-data-update CLI (cli/commands.py:188) echoes str(update), which now renders config='<redacted>' data='<redacted>'. The published payload is unchanged (uses update.json()). Redacting the stdout echo is intentional — CLI output lands in shell history / CI logs.

Tests

opal-common/.../tests/redaction_test.py covers: repr/str redaction (incl. f-strings & nested DataUpdate), the RPC subclass, wire-payload integrity, loguru serialize=True and diagnose=True, the MRO-union footgun, and redact_url/redact_url_in_text. Verified locally against pydantic 1.10 + loguru: 58 tests pass (new + #902 reporter regression + opal-common + git_utils suites).

Review

Reviewed by a 4-agent adversarial panel (security/leak-completeness, pydantic-internals, behavioral-regression, code-quality); all findings above were surfaced and addressed.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for opal-docs ready!

Name	Link
🔨 Latest commit	5c54135
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a439fff61c03700085928df
😎 Deploy Preview	https://deploy-preview-921--opal-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR hardens credential redaction across OPAL by centralizing model-layer repr()/str() redaction for secret-bearing pydantic models, and by systematically redacting credentialed URLs in log messages/exceptions (plus reducing loguru/git diagnostic leakage vectors).

Changes:

• Introduces RedactedReprMixin to mask sensitive model fields in repr()/str() across the MRO (protecting log interpolation + serialize=True sinks).
• Adds URL redaction utilities (redact_url, redact_url_in_text) and applies them across git/data fetcher logging paths.
• Flips LOG_DIAGNOSE default to False and gates verbose git tracing env vars behind diagnosis mode; adds regression tests for the above.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/opal-server/opal_server/git_fetcher.py	Redacts repo URLs in debug/info/error logs to prevent credentialed URL leakage.
packages/opal-server/opal_server/data/data_update_publisher.py	Avoids logging full entries; logs redacted URL instead.
packages/opal-server/opal_server/data/api.py	Stops logging full configuration model on validation failure; logs safer message.
packages/opal-common/opal_common/tests/redaction_test.py	Adds regression tests for repr/str redaction, URL redaction, loguru serialize/diagnose, and git env gating.
packages/opal-common/opal_common/sources/git_policy_source.py	Redacts remote URLs when reporting repo mismatch / existing remotes.
packages/opal-common/opal_common/schemas/data.py	Applies RedactedReprMixin to DataSourceEntry and redacts config/data in repr/str.
packages/opal-common/opal_common/logging_utils/redaction.py	Adds the core RedactedReprMixin implementation.
packages/opal-common/opal_common/http_utils.py	Adds redact_url() and redact_url_in_text() utilities + sensitive query param handling.
packages/opal-common/opal_common/git_utils/repo_cloner.py	Redacts URLs and scrubs credentialed URLs from git error text before logging.
packages/opal-common/opal_common/git_utils/env.py	Gates GIT_TRACE/GIT_CURL_VERBOSE behind LOG_DIAGNOSE to avoid auth header leakage.
packages/opal-common/opal_common/fetcher/providers/http_fetch_provider.py	Redacts fetch URL in debug logs; marks headers/data as redacted fields on config model.
packages/opal-common/opal_common/fetcher/providers/fastapi_rpc_fetch_provider.py	Redacts RPC arguments in repr/str and stops logging them directly.
packages/opal-common/opal_common/fetcher/events.py	Applies RedactedReprMixin to FetcherConfig/FetchEvent; redacts config field in event repr/str.
packages/opal-common/opal_common/config.py	Changes default LOG_DIAGNOSE to False and clarifies credential-leak rationale.
packages/opal-client/opal_client/data/updater.py	Redacts URLs in logs/errors and avoids logging full DataSourceEntry objects.
packages/opal-client/opal_client/data/fetcher.py	Redacts URLs in fetcher logs/exceptions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeevmoney]

Review: credential-leak sweep is incomplete

The model-layer redaction (RedactedReprMixin) and redact_url helpers are sound and verified working (pydantic 1.10.26 + loguru, serialize/diagnose). Inline comments cover issues on changed lines. This comment lists credential-leak sites that remain outside this PR's diff — relevant because the PR's stated goal is to sweep every related vector. None of these can be posted as inline comments since they sit on unchanged lines.

HIGH — still leak credentials:

• opal_common/fetcher/engine/fetch_worker.py:42 and fetcher/fetch_provider.py:59 — logger.exception(...) runs while an aiohttp ClientResponseError is active. Its str() includes request_info.real_url, a yarl.URL that renders https://user:password@host/path?token=... in full. This fires under default config (backtrace=True, even with diagnose=False) and is the most direct leak for any failed authenticated data fetch. The model-layer strategy cannot catch it — the leaking object is a third-party exception, not an OPAL model. The robust fix is a loguru sink-level patcher that runs redact_url_in_text over the formatted record (including record["exception"]); that is the only thing that makes the "every vector" claim true for 3rd-party exceptions.

• opal_common/sources/api_policy_source.py:190-196 (and :89, :124) — the HTTP-bundle analogue of git_policy_source.py (which this PR did redact). Logs/raises the full bundle URL raw (full_url = f"{url}/{path}", url = self.remote_source_url) and renders the active aiohttp exception. Symmetric site, missed entirely. Wrap with redact_url(...).

• opal_server/scopes/service.py:132-134 — logger.debug(f"Sync scope: ... (remote: {source.url}, ...)") logs source.url raw on the scope-sync hot path (same GitPolicyScopeSource.url that git_fetcher.py redacts). Use redact_url(source.url).

• opal_server/scopes/loader.py:22-25 — logger.info("Adding default scope from env: {url}", url=opal_server_config.POLICY_REPO_URL) logs the policy repo URL raw at startup; POLICY_REPO_URL commonly embeds a token for HTTPS auth. Use redact_url(...).

• opal_client/data/updater.py:552, 561-564, 570-571 (unchanged lines in a file this PR edits) — store_transaction._update_remote_status(url=entry.url, ...) passes entry.url raw three times. That URL flows into the transaction-log JSON dump (opa_client.py, json.dumps(transaction, default=str)) and the OPA-transaction-failed error log. The PR redacted the adjacent log lines in this same function but missed these. Pass url=redact_url(entry.url) (changes only the logged/persisted status URL, not any functional fetch).

MEDIUM:

• opal_client/callbacks/reporter.py:65 (and :70, :84) — callback urls/url logged raw. Callback URLs are user-supplied and can embed creds; fix: stop logging FetcherConfig in callback reporter #902 hardened this file but left the URLs themselves unredacted. Use redact_url(...).
• opal_client/data/updater.py:450 — logger.exception(f"Failed to calculate hash for data {data}: {e}") logs the full inline data payload (a credential vector per this PR's own threat model; here it is the raw dict, so the mixin does not protect it).
• opal_client/data/updater.py (~line 230) — logger.info("Getting data-sources configuration from '{source}'", source=url) logs the data-sources config URL raw.

Operational note: flipping LOG_DIAGNOSE default to False means operators upgrading silently lose diagnose tracebacks and verbose git tracing — worth a release-note line (opt back in via OPAL_LOG_DIAGNOSE=true).

[zeevmoney]

See comments.

[Zivxx]

@zeevmoney thanks for the out-of-diff sweep — all of it is now addressed.

Redacted at the missed call sites (7cedb07):

• api_policy_source.py — full_url in the 404 log + HTTPException detail, and the "Fetching changes from remote" log.
• opal_server/scopes/service.py and scopes/loader.py — source.url / POLICY_REPO_URL.
• opal_client/data/updater.py — entry.url in all three _update_remote_status calls (it is persisted into the transaction log) plus the error=str(e) strings; calc_hash no longer logs the raw inline data payload; the data-sources config URL is redacted.
• opal_client/callbacks/reporter.py — the user-supplied callback URLs (:65/:70/:84).

Third-party exception leak (9c73269): rather than redacting each logger.exception site individually, I closed the vector at the sink layer, as you suggested. CredentialScrubbingStream wraps the stdout/stderr sink and runs redact_url_in_text over the fully-formatted record (message and exception traceback, including the JSON blob under serialize=True), so an aiohttp ClientResponseError rendering https://user:pw@host/... in its traceback is scrubbed regardless of which logger.exception("...") raised it. A loguru patcher additionally scrubs record["message"] for every record, so the optional rotating file sink (path-based, can't be stream-wrapped) still gets message-level scrubbing without losing rotation/retention. Covered by test_scrubbing_stream_redacts_thirdparty_exception_traceback.

LOG_DIAGNOSE default flip: documented in configuration.mdx (now Default: False, with a "default changed" note that operators opt back in via OPAL_LOG_DIAGNOSE=true).

One caveat worth flagging: the sink wrapper scrubs scheme://user:pw@ userinfo anywhere in a traceback, but a bare ?token=... query param that appears only inside a third-party exception string (with no known URL to match against) is not generically detectable — those remain best-effort via the per-call-site redact_url paths.