Capture LLM trajectories across OAuth and API-key runs

docs/agent-quickstart.md

@@ -138,15 +138,24 @@ and what it is:
     agent/                           — agent-side logs
     trajectory/acp_trajectory.jsonl  — the full agent trace (every ACP event:
                                        prompts, tool calls, outputs)
-    trajectory/llm_trajectory.jsonl  — raw provider requests/responses captured
-                                       by the usage-tracking proxy
+    trajectory/llm_trajectory.jsonl  — always-present LLM exchange log
+    trajectory/llm_trajectory.manifest.json
+                                     — source/fidelity/completeness for that log
     trainer/verifiers.jsonl          — trainer-ready scored trajectory record
     trainer/atif.json                — the trajectory in ATIF interchange
                                        format (omitted if the trajectory is empty)
     trainer/adp.jsonl                — the trajectory in ADP format
     verifier/reward.txt              — raw verifier reward
     verifier/test-stdout.txt         — verifier stdout (and ctrf.json when the
                                        test emits a CTRF report)
+
+`llm_trajectory.jsonl` is an audit artifact for every run, not an unconditional
+training claim. Only a `complete` manifest with `provider_wire` fidelity and
+positive provider token usage is training-ready. Native OAuth/session capture
+and continuation replay-proxy ingress stay available for audit but are marked
+lower-fidelity and excluded from training. A sandbox-local proxy that shares
+root custody with the agent records that additional custody limitation.
+
 Also note the job-level summary.json plus aggregated results.jsonl,
 verifiers.jsonl, and adp.jsonl in the job directory. The trainer/ files and the job-level
 aggregates are written by current BenchFlow releases; if they are missing,

docs/continue-runs.md

@@ -37,6 +37,15 @@ request/response pairs from the original run. `bench eval continue`:
    with a stitched `llm_trajectory.jsonl` (recorded prefix + live suffix) and
    `continued_from` provenance — a drop-in replacement for the timed-out entry.
 
+The stitched manifest preserves both capture boundary and custody. The live
+suffix is observed at replay-proxy ingress, before the forwarder and LiteLLM can
+replace, filter, or transform the provider request. It is therefore always
+labeled `replay_proxy` / `agent_session` with an incomplete provider request,
+even when the proxy is host-owned. If replay and the untrusted agent also share
+root custody in the sandbox, the manifest records that additional limitation.
+The suffix remains useful for audit and continuity, but any continued run with
+a live suffix is excluded from training-ready provider evidence.
+
 Because the agent rebuilds its own state by re-doing its own steps, no
 reverse-engineering of OpenHands internals is needed, and the result is a single
 continuous run rather than a fresh agent on a warm filesystem.

docs/getting-started.md

@@ -202,7 +202,8 @@ Each run writes under `--jobs-dir` (default `jobs/`):
       prompts.json                  # prompts sent to the agent
       trajectory/
         acp_trajectory.jsonl        # full agent trace (ACP events)
-        llm_trajectory.jsonl        # raw provider requests/responses (when the usage-tracking proxy captured exchanges)
+        llm_trajectory.jsonl        # always present; LLM exchanges at the fidelity described below
+        llm_trajectory.manifest.json # capture source, fidelity, completeness, and errors
       trainer/
         verifiers.jsonl             # trainer-ready scored trajectory (Verifiers/ORS record)
         atif.json                   # ATIF trajectory record (omitted if the trajectory is empty)
@@ -213,6 +214,36 @@ Each run writes under `--jobs-dir` (default `jobs/`):
         test-stdout.txt             # verifier stdout
 ```
 
+`llm_trajectory.jsonl` is created when the rollout directory is initialized,
+including for setup failures and tasks that make no model call. Its sidecar is
+the source of truth for interpreting the JSONL:
+
+| Agent/auth path | Primary source | `capture_fidelity` |
+|---|---|---|
+| API key through the BenchFlow gateway (including Azure) | LiteLLM provider request/response capture | `provider_wire` |
+| Claude Code subscription/OAuth | Claude Code raw API-body files correlated by local OTLP logs | `agent_session` |
+| Claude Code subscription/OAuth fallback | Claude Code native session JSONL | `agent_session` |
+| Codex subscription/OAuth | Codex native session JSONL | `agent_session` |
+| Continued-run live suffix | Replay-proxy ingress before provider transformation | `agent_session` |
+
+The manifest status is `complete`, `partial`, `no_model_call`, or
+`capture_failed`. Mixed-role rollouts merge API-key and native-subscription
+exchanges into the same JSONL; `role_captures` records each prepared role's
+scene role, agent, model, auth mode, source, fidelity, completeness, and exchange
+count. Continued runs retain the source role entries as the `recorded` leg and
+append a separate `live` leg, so a model or auth switch remains auditable. The
+live continuation request is captured before provider transformation and is
+therefore audit-only even when its replay proxy is host-owned.
+Missing or ambiguously attributed roles make the rollout-level capture
+`partial`. Reconstructed `agent_session` rows remain useful for audit and viewer
+workflows, but trainer exports fail closed unless the manifest says the capture
+is complete provider-wire data and every successful exchange has positive token
+usage. Claude's raw-body surface is deliberately audit-only because Claude Code
+writes it inside the agent-controlled sandbox; its API shape does not make its
+custody provider-trusted. It can also contain provider-redacted
+extended-thinking blocks. BenchFlow applies its normal secret redaction before
+publishing the JSONL.
+
 ### Reading results
 
 Exit code 0 means the pipeline completed — it is not a pass/fail signal. A

docs/running-any-benchmark.md

@@ -217,12 +217,20 @@ Every layer terminates at the *same* output contract, written per rollout under
 | `results.jsonl` | Verifiers/Prime-RL shaped rollout row |
 | `rewards.jsonl` | The reward record for the rollout (ORS / OpenReward shape) |
 | `trajectory/acp_trajectory.jsonl` | Full agent trace as ACP events |
-| `trajectory/llm_trajectory.jsonl` | Raw provider requests/responses (when captured) |
+| `trajectory/llm_trajectory.jsonl` | Always-present LLM exchange log (possibly empty for `no_model_call`) |
+| `trajectory/llm_trajectory.manifest.json` | Capture source, fidelity, completeness, and errors |
 | `trainer/verifiers.jsonl` | Trainer-ready scored trajectory (Verifiers record) |
 | `trainer/atif.json` | ATIF trajectory record |
 | `trainer/adp.jsonl` | ADP trajectory record |
 | `verifier/` | Raw verifier output (`reward.txt`, `ctrf.json`, stdout) |
 
+The JSONL file is always an audit artifact; its sidecar determines whether it
+is also training-ready. Training requires complete `provider_wire` capture with
+positive provider token usage. Native OAuth/session capture and sandbox-local
+capture that shares root custody with the agent are retained but marked
+lower-fidelity, so exporters reject them rather than treating agent-writable
+records as trusted provider evidence.
+
 Hosted runs share this artifact contract too (see the `hosted_env.py` module
 docstring), with `source.type="hosted_env"` / `trajectory_source="hosted_env"`
 marking the lineage.

hatch_build.py

@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+import tempfile
+from pathlib import Path
+from typing import Any
+
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
+
+_RUNTIME_RESOURCES = (
+    (
+        "src/benchflow/trajectories/redaction.py",
+        "benchflow/trajectories/resources/canonical_redaction.py.txt",
+    ),
+    (
+        "src/benchflow/continue_run/sandbox_replay_runtime.py",
+        "benchflow/continue_run/resources/sandbox_replay_runtime.py.txt",
+    ),
+)
+
+
+class CustomBuildHook(BuildHookInterface):
+    PLUGIN_NAME = "custom"
+
+    def initialize(self, _version: str, build_data: dict[str, Any]) -> None:
+        if self.target_name != "wheel":
+            return
+        self._generated_paths: list[Path] = []
+        for source_name, resource_name in _RUNTIME_RESOURCES:
+            source = Path(self.root, source_name)
+            with tempfile.NamedTemporaryFile(
+                prefix="benchflow-runtime-resource-",
+                suffix=".txt",
+                delete=False,
+            ) as generated:
+                generated.write(source.read_bytes())
+                generated_path = Path(generated.name)
+            self._generated_paths.append(generated_path)
+            build_data["force_include"][str(generated_path)] = resource_name
+
+    def finalize(
+        self,
+        _version: str,
+        _build_data: dict[str, Any],
+        _artifact_path: str,
+    ) -> None:
+        self._discard_generated()
+
+    def clean(self, _versions: list[str]) -> None:
+        self._discard_generated()
+
+    def _discard_generated(self) -> None:
+        for generated in getattr(self, "_generated_paths", []):
+            generated.unlink(missing_ok=True)
+        self._generated_paths = []

pyproject.toml

@@ -146,12 +146,16 @@ build-backend = "hatchling.build"
 only-include = [
     "src",
     "tests",
+    "hatch_build.py",
     "README.md",
     "CHANGELOG.md",
     "LICENSE",
     "pyproject.toml",
 ]
 
+[tool.hatch.build.hooks.custom]
+path = "hatch_build.py"
+
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
 addopts = "-m 'not live and not integration'"

src/benchflow/acp/container_transport.py

@@ -7,7 +7,7 @@
 
 from benchflow.sandbox.process import LiveProcess
 from benchflow.sandbox.process._base import _ANSI_CSI_RE, _ANSI_OSC_RE
-from benchflow.trajectories.types import redact_trajectory_text
+from benchflow.trajectories.redaction import redact_trajectory_text
 
 from .transport import Transport, decode_json_rpc_message
 

src/benchflow/acp/runtime.py

@@ -28,6 +28,7 @@
 from benchflow.acp.container_transport import ContainerTransport
 from benchflow.acp.selection import selected_acp_transport
 from benchflow.acp.types import McpServerSpec
+from benchflow.agents.opencode_config import opencode_provider_reset_command
 from benchflow.agents.protocol import ACPSessionAdapter
 from benchflow.agents.providers import (
     find_provider,
@@ -72,6 +73,41 @@
 _OPENHANDS_DISABLE_SUBAGENTS_ENV = "BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS"
 
 
+def _harden_proxy_agent_launch(
+    agent: str, agent_launch: str, agent_env: dict[str, str]
+) -> str:
+    """Insert agent-side config hardening after proxy selection."""
+
+    if not agent_env.get("BENCHFLOW_LITELLM_MODEL_ALIAS"):
+        return agent_launch
+    if agent == "opencode":
+        # OpenCode merges several global filenames, ~/.opencode, project
+        # configs, an arbitrary file/directory, and inline JSON. Proxy mode
+        # must expose exactly the manifest-owned config that the wrapper
+        # registers below. Pin XDG to the same canonical agent home, disable
+        # project discovery, and remove every supported alternate env source
+        # before the process imports its configuration flags.
+        isolate_sources = (
+            "unset OPENCODE_CONFIG OPENCODE_CONFIG_DIR "
+            "OPENCODE_CONFIG_CONTENT XDG_CONFIG_HOME && "
+            'export HOME="${BENCHFLOW_AGENT_HOME:-$HOME}" && '
+            'export XDG_CONFIG_HOME="$HOME/.config" && '
+            "export OPENCODE_DISABLE_PROJECT_CONFIG=1"
+        )
+        return (
+            f"{isolate_sources} && {opencode_provider_reset_command()} "
+            f"&& {agent_launch}"
+        )
+    if agent == "mimo":
+        # MiMo's manifest-owned launcher replaces both canonical config files
+        # in proxy mode, but the CLI also honors this arbitrary alternate path.
+        # Do not let an image-baked config bypass the capture proxy. Direct mode
+        # deliberately retains the override as part of the caller's provider
+        # configuration.
+        return f"unset MIMOCODE_CONFIG && {agent_launch}"
+    return agent_launch
+
+
 def _acp_handshake_timeout_sec() -> float:
     """Effective timeout for the pre-prompt ACP handshake, in seconds.
 
@@ -610,6 +646,8 @@ async def connect_acp(
             agent_launch = " ".join(parts)
             logger.info(f"Resolved agent path: {agent_launch}")
 
+    agent_launch = _harden_proxy_agent_launch(agent, agent_launch, agent_env)
+
     if sandbox_user:
         agent_launch = build_priv_drop_cmd(agent_launch, sandbox_user)
         logger.info(f"Agent sandboxed as: {sandbox_user}")

src/benchflow/agents/codex_config.py

@@ -20,52 +20,104 @@ def codex_provider_id(provider_name: str | None) -> str:
     return f"{_CODEX_PROVIDER_ID_PREFIX}{safe_name or 'provider'}"
 
 
-def apply_codex_provider_config(
+def apply_codex_custom_provider_config(
     agent_env: dict[str, str],
     *,
     base_url: str,
     model: str | None,
     provider_name: str,
-    strict: bool = False,
 ) -> None:
-    """Create or update Codex's model provider entry in ``agent_env``."""
+    """Update caller-owned Codex config for a direct custom provider."""
+
     raw_config = agent_env.get(CODEX_CONFIG_ENV)
     if not raw_config:
         config: dict[str, Any] = {}
     else:
         try:
             config = json.loads(raw_config)
         except json.JSONDecodeError as exc:
-            if strict:
-                raise ValueError(f"{CODEX_CONFIG_ENV} must be valid JSON") from exc
-            return
+            raise ValueError(f"{CODEX_CONFIG_ENV} must be valid JSON") from exc
     if not isinstance(config, dict):
-        if strict:
-            raise ValueError(f"{CODEX_CONFIG_ENV} must decode to a JSON object")
-        return
+        raise ValueError(f"{CODEX_CONFIG_ENV} must decode to a JSON object")
 
+    configured_provider_id = agent_env.get(CODEX_MODEL_PROVIDER_ENV) or config.get(
+        "model_provider"
+    )
     provider_id = (
-        agent_env.get(CODEX_MODEL_PROVIDER_ENV)
-        or config.get("model_provider")
-        or codex_provider_id(provider_name)
+        configured_provider_id
+        if isinstance(configured_provider_id, str) and configured_provider_id
+        else codex_provider_id(provider_name)
     )
-    providers = config.get("model_providers")
-    providers = {} if not isinstance(providers, dict) else dict(providers)
-    provider = providers.get(provider_id)
-    provider = dict(provider) if isinstance(provider, dict) else {}
+    providers_value = config.get("model_providers")
+    providers = {} if not isinstance(providers_value, dict) else dict(providers_value)
+    provider_value = providers.get(provider_id)
+    provider = dict(provider_value) if isinstance(provider_value, dict) else {}
     provider.setdefault("name", provider_name)
     provider["base_url"] = base_url
-    provider.setdefault("env_key", "OPENAI_API_KEY")
+    provider["env_key"] = "OPENAI_API_KEY"
     provider.setdefault("wire_api", "responses")
     provider.setdefault("supports_websockets", False)
 
+    _write_codex_provider_config(
+        agent_env,
+        config=config,
+        providers=providers,
+        provider_id=provider_id,
+        provider=provider,
+        model=model,
+        base_url=base_url,
+        provider_name=provider_name,
+    )
+
+
+def apply_codex_proxy_config(
+    agent_env: dict[str, str],
+    *,
+    base_url: str,
+    model: str | None,
+    provider_name: str,
+) -> None:
+    """Replace Codex config with one BenchFlow-owned proxy provider."""
+
+    provider_id = codex_provider_id(provider_name)
+    _write_codex_provider_config(
+        agent_env,
+        config={},
+        providers={},
+        provider_id=provider_id,
+        provider={
+            "name": provider_name,
+            "base_url": base_url,
+            "env_key": "OPENAI_API_KEY",
+            "wire_api": "responses",
+            "supports_websockets": False,
+        },
+        model=model,
+        base_url=base_url,
+        provider_name=provider_name,
+    )
+
+
+def _write_codex_provider_config(
+    agent_env: dict[str, str],
+    *,
+    config: dict[str, Any],
+    providers: dict[str, Any],
+    provider_id: str,
+    provider: dict[str, Any],
+    model: str | None,
+    base_url: str,
+    provider_name: str,
+) -> None:
+    """Serialize one already-resolved Codex provider configuration."""
+
     providers[provider_id] = provider
     config["model_providers"] = providers
     config["model_provider"] = provider_id
     if model:
         config["model"] = model
 
-    agent_env[CODEX_MODEL_PROVIDER_ENV] = str(provider_id)
+    agent_env[CODEX_MODEL_PROVIDER_ENV] = provider_id
     agent_env[CODEX_CONFIG_ENV] = json.dumps(config, separators=(",", ":"))
     _apply_codex_default_auth_request(
         agent_env,