Skip to content

Capture LLM trajectories across OAuth and API-key runs - #1057

Open
bingran-you wants to merge 74 commits into
mainfrom
bry/llm-trajectory-all-auth
Open

Capture LLM trajectories across OAuth and API-key runs#1057
bingran-you wants to merge 74 commits into
mainfrom
bry/llm-trajectory-all-auth

Conversation

@bingran-you

@bingran-you bingran-you commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Outcome

BenchFlow now creates trajectory/llm_trajectory.jsonl and trajectory/llm_trajectory.manifest.json at rollout initialization for Claude Code and Codex, across API-key and native OAuth/subscription authentication. Setup failures, no-call runs, capture failures, mixed-role runs, and continuations all retain a truthful terminal artifact.

Capture contract

  • API-key routes, including Codex through Azure, use LiteLLM provider-wire capture only when the gateway has trusted host or verified sandbox custody.
  • Codex proxy wiring reconstructs an exact minimal benchflow-litellm provider definition from allowlisted fields. Caller top-level fields, provider IDs, unselected providers, literal headers, and credential aliases are discarded; only the BenchFlow proxy URL, OPENAI_API_KEY bridge, selected model alias, Responses wire API, and disabled WebSocket flag survive.
  • Sandbox-local custody is proven after startup against the actual LiteLLM runtime directory. The root-owned runtime directory and real callback.jsonl and capture_state.json artifacts are forced to 0700 and 0600 independent of inherited umask, and must be unreadable and unmodifiable by the agent; passwordless sudo/doas, root-group membership, effective capabilities, writable container-control sockets, probe anomalies, and integrity mismatches all fail closed to audit-only capture.
  • Provider credential-file custody is a separate training gate: providers such as Vertex ADC that expose raw credentials in the agent home remain audit-only even when callback artifacts have verified root custody. ADC variables are stripped from the proxy-wired agent environment.
  • Proxy-wired agents receive only the BenchFlow master key: registered provider and agent credentials, Codex and Claude OAuth aliases, AWS and ADC forms, generic bridges, and custom credential-shaped environment names are stripped and post-wiring guarded.
  • Provider request capture uses LiteLLM's post-transform pre_api_call body—the payload handed to the provider—not proxy ingress. Missing post-transform bodies fail request_complete closed and downgrade the exchange to audit-only fidelity.
  • Provider response completeness requires an explicit LiteLLM success callback. Failure callbacks may represent local DNS/connect/timeout failures and therefore retain their rows but fail closed to audit-only fidelity.
  • Claude Code OAuth raw request/response bodies are correlated with local OTel events and retained as audit-only agent_session data because the files are agent-writable.
  • Claude and Codex native-session data are audit-only agent_session fallbacks; ACP projection is the final audit-only fallback.
  • Mixed role/auth runs retain per-role provenance. Continued runs preserve recorded role captures and append a separate live leg with its own model, auth, source, fidelity, completeness, and count.
  • Continuation live requests are observed at replay-proxy ingress before forwarding/provider transformation. They are always retained as audit-only replay_proxy / agent_session rows with request_complete=false; host ownership and sandbox shared-root custody remain separately recorded.
  • Successful provider-wire training rows require positive token usage. Prime-SFT and TRL-SFT exports fail closed for lower-fidelity data.
  • Schema-v2 training admission also requires payload_redacted=true; false or missing redaction custody fails closed before canonical results or trainer export.
  • Claude credentials are never routed through Azure.

Lifecycle and integrity

  • LiteLLM journals attempts durably before forwarding, applies the single canonical trajectory redactor before append-only persistence, and marks terminal callbacks only after callback JSONL flush plus fsync.
  • Canonical redaction is a packaged stdlib-only module plus a dedicated .txt data resource generated from that module at wheel-build time. Host and sandbox LiteLLM runtimes receive the exact resource beside the callback; no reflection or manually reconstructed dependency closure is used.
  • Sandbox continuation journals apply that same packaged structural redactor before durable append, so an abrupt sandbox termination cannot leave raw request or response secrets in live_exchanges.jsonl.
  • Host and sandbox shutdown close acceptance, drain in-flight work, import final capture, and reconcile attempts, terminal callbacks, parsed exchanges, roles, request completeness, response completeness, and token evidence.
  • Result completion is validated per role: non-OAuth roles require complete request and response evidence, OAuth roles must come from known native audit surfaces, replay roles must satisfy the exact suffix contract, and role counts plus active auth aggregation must match the manifest. Errors and missing fields remain strict-allowlisted.
  • Malformed provider JSONL is isolated so valid bound native evidence still produces an audit-only partial artifact.
  • Finalization failures preserve already-valid JSONL rows while failing the sidecar closed; malformed JSONL is repaired to the valid empty artifact.
  • Raw capture cleanup, result regeneration, continuation stitching, structural redaction, credential routing, and concurrent attempt ordering all have regression coverage.

Maintainability remediation

The exact-head thermo review findings are addressed:

  • actual runtime-artifact access probing replaces username/UID-only custody trust;
  • local provider failures no longer masquerade as complete provider responses;
  • audit-only completion is role-scoped, count-consistent, auth-consistent, and strict-allowlists both errors and intentional missing fields;
  • malformed provider input no longer suppresses native evidence;
  • sidecar write failure no longer erases assembled rows;
  • continuation role captures retain distinct recorded/live provenance;
  • LLMTrajectoryCapture is reduced from 965 to 586 lines; typed ClaudeOtelCollector and NativeSessionCollector own OTel lifecycle and native collection in a focused module;
  • proxy-ingress reconstruction is audit-only; training fidelity requires the post-transform provider body and an explicit source marker;
  • durable callback journaling and final artifacts consume one standalone canonical redactor;
  • continuation suffixes fail closed at their actual replay-proxy boundary while successful runs remain is_completed=true only when lifecycle provenance is internally consistent;
  • repeated continuation validates every recorded replay leg, including prior replay-proxy audit warnings, while preserving strict rejection for OAuth and live-leg provenance errors;
  • repeated and model-switching continuations derive their aggregate model from active preserved role captures, so mixed-model prefixes remain explicit instead of inheriting rewritten run-level config;
  • host and sandbox continuation exchanges are stitched by assigned attempt order, not response completion order;
  • direct Codex providers retain caller reasoning effort, service tier, wire API, headers, WebSocket setting, and unselected provider definitions; exact minimal reconstruction is isolated to the BenchFlow-owned LiteLLM route;
  • host-owned capture translates Docker-view host aliases to host loopback, preserving local custom-provider routing after provider capture became mandatory;
  • the 1,027-line native capture test module was decomposed into 941-line capture and 651-line resilience suites; the branch has zero under-1,000-to-over-1,000 file crossings.

Exact-head verification

Head: 272557493b0f056502b1a90bd8bbfeb749ba9316

Thermo-nuclear remediation

  • Provider custody is now a dedicated orchestration module plus a packaged, shellcheck-able provider_capture_custody.sh resource; the dead sandbox_local argument and duplicated custody block are gone.

  • Sandbox replay is a real stdlib-only Python module, packaged as an exact wheel resource and uploaded verbatim. Its quiesce/handler concurrency protocol is now linted, type-checked, and directly unit-tested.

  • payload_redacted defaults false and is derived from the final serialized JSONL after the canonical structural redactor runs. A skipped or ineffective redactor fails training admission.

  • Atomic writes, training-grade admission, private sandbox uploads, and sandbox-home resolution each have one canonical helper. The redaction compatibility re-export is removed.

  • Codex direct-provider configuration and BenchFlow proxy configuration are separate APIs; direct caller settings are preserved while proxy routes remain isolated.

  • Provider request paths now follow the actual LiteLLM call type (/v1/messages, /v1/responses, or /v1/chat/completions).

  • Healthy audit-only OAuth captures are reported as well-formed but non-training-grade, never malformed. bench train validate --require-llm-trajectory now rejects the non-training-grade category explicitly.

  • A prepared OAuth role with no observed model call now terminates as a clean zero-exchange no_model_call artifact that preserves canonical audit completion. Missing prepared roles still fail closed as soon as ACP reports a call or any captured row proves a call; regression coverage is tied to exact-head review finding r3888576789.

  • Mixed API/OAuth usage is now reported as trusted usage_source=mixed: provider and native token/cache totals are summed, source-specific detail and priced provider cost are preserved, and the aggregate cost remains unknown rather than pretending the unpriced OAuth leg was priced.

  • Native ACP usage is accumulated only for roles actually using subscription auth, and role transitions reset cumulative checkpoints so provider-proxied ACP counters cannot be double-counted.

  • Primary setup consumes the first role's scoped environment, so Claude OAuth and Codex Azure credentials never need to coexist in the parent/global environment.

  • Every native-subscription reconnect refreshes credential files, registry auth uploads, native MCP settings, and web policy even when the role reuses the primary agent/model with an empty role environment. This closes exact-head review r3888704109 for OAuth → API → OAuth sequences after proxy cleanup removes the earlier login file.

  • Proxy startup removes every registry-declared native-login route left by any prior role, not only the current harness's credential. Cleanup is no-follow, settings-aware, effective-home-aware, and fails closed when a runtime or safe path proof is unavailable.

  • Role teardown freezes and records the exact ACP wrapper process tree before termination, preventing its real CLI child from being orphaned under PID 1 while preserving unrelated task processes. Claude's OTel sink now runs as root-owned BenchFlow infrastructure with only the raw-body directory delegated to the agent, so it cannot invalidate the next API role's zero-process custody proof.

  • Training admission rejects contradictory complete/provider_wire manifests unless both errors and missing_fields are present and empty; direct-predicate and canonical-results regressions cover both gap collections.

  • OpenCode-family proxy mode replaces the full provider map, preventing image-baked literal keys or endpoints from bypassing the capture gateway while retaining unrelated settings.

  • OpenCode proxy launch now isolates every effective config authority from pinned OpenCode 1.18.11: alternate file/directory/inline env sources are unset; project discovery is disabled; HOME/XDG and test-home overrides are pinned; global, ~/.opencode, and redirected system-managed configs are removed; inline/file auth and remote well-known config are neutralized; and active-account state uses an empty in-memory DB. Regression r3888738113 seeds hostile values in every source.

  • Continuation run-level model attribution now comes from finalized active role captures. A requested-but-unused live model cannot overwrite replay-only config.json, result.json, or canonical/aggregate trainer rows; regression r3888738115 covers the zero-live-call path.

  • Audit-only completion now cross-checks the strictly loaded JSONL rows against sidecar per-role cardinality and role/agent/model/auth/source/fidelity. Truncated or misattributed JSONL fails completion even when the sidecar remains valid (r3888793594), and zero-call completion requires nonempty prepared-role evidence (r3888793598).

  • Pi proxy mode proves the agent UID quiescent, no-follow removes .pi/agent/models.json from every safe effective home, pins the canonical launch home, and fails capture trust closed on unsafe roots or cleanup errors before the unchanged manifest-owned launcher runs; direct-provider behavior remains unchanged.

  • Provider usage and provider-API failure evidence are retained and aggregated across every retired role-scoped LiteLLM runtime plus the final runtime. Token/cache counters sum across rotations; cost sums only when every contributing runtime is priced, so partial pricing cannot masquerade as a complete total.

  • The real agent launch uses setpriv --no-new-privs --reuid/--regid/--init-groups when supported. The custody probe is run through that same privilege boundary and requires Linux /proc/self/status to report NoNewPrivs: 1; the compatibility su fallback therefore fails capture trust closed to audit-only instead of claiming provider-wire custody.

  • MiMo proxy mode clears the alternate MIMOCODE_CONFIG override before its manifest-owned launcher writes both canonical configs; direct-provider mode remains unchanged.

  • Host and sandbox replay persist the recorded prefix actually consumed. Stitched artifacts include only that prefix, and an early stop fails closed with recorded_replay_prefix missing.

  • Schema-v2 training admission cross-checks every row against the sidecar for fidelity, request/response completeness, redaction, capture source, auth mode, role, agent, model, and exact per-role cardinality; contradictory or empty role captures fail closed.

  • Every training-grade row must independently assert role_attribution_complete=true; false or missing attribution evidence fails closed.

  • Before API-proxy capture can be trusted, every agent—including agents without subscription-auth support—must prove a non-root sandbox UID has no live processes. The guard accepts only pgrep exact no-match status 1; missing tooling, UID 0, a live process, or a probe error fails closed to audit-only, and no process is killed. For subscription-capable agents, registry-defined default/config-home/effective-home tokens are removed in the same root shell only after the guard passes and are verified absent through no-follow directory descriptors. Claude settings at every effective config home are atomically sanitized of env, apiKeyHelper, awsAuthRefresh, and awsCredentialExport while preserving unrelated policy, ownership, and mode; malformed JSON, symlinks, rewrite failures, and residual keys fail trust closed. HOME and BENCHFLOW_AGENT_HOME are pinned to the canonical sandbox home, unsafe roots and symlink traversal fail closed, and sandbox-home values are excluded from the host LiteLLM environment.

  • Continuation live rows redact once at their durable boundary; crash-safety and authoritative post-cleanup writes are documented; replay state cannot fabricate a balanced journal.

  • LLMTrajectoryCapture remains below 600 lines; the native capture suites are 967 and 715 lines; continuation trainer metadata was extracted so test_orchestrator.py is 897 lines; the branch has no under-1,000-to-over-1,000 crossing.

Local verification

  • pytest tests/: 6,067 passed, 55 skipped, 7 deselected in 200.58s on exact head 272557493b0f056502b1a90bd8bbfeb749ba9316.
  • ruff format src tests tools --check: all 631 files formatted.
  • ruff check ., ty check src/, and git diff --check: pass.
  • OpenCode/continuation remediation passes 68 focused tests; audit lifecycle/native/continuation remediation passes 97 focused tests. A Linux/amd64 container installed the exact wheel and passed 480 tests, 2 skipped across 18 auth, custody, lifecycle, native-capture, OpenCode, continuation, and redaction modules from a read-only source mount.

Real Docker authentication matrix

The authentication lanes used credential-isolated parent processes. Claude received only its OAuth subscription token; every Azure/OpenAI/Anthropic API-key variable was removed before launch. Codex received Azure credentials while every Claude credential was removed.

  • Exact-head Claude OAuth (27255749): reward 1 with two redacted claude_otel_raw_body / agent_session / oauth_subscription exchanges. Request/response completeness is true, the manifest has no errors or missing fields, canonical results report is_completed=true but training_ready=false, native ACP usage reports 43,562 tokens, and raw/base64/URL-encoded scans found zero credential occurrences. config.json and result.json contain no Azure marker. Both trainer exporters reject it with skipped_insufficient_capture_fidelity=1.
  • Exact-head Codex Azure (27255749): the raw Azure Responses preflight returned HTTP 200 with a valid usage envelope, then the task scored reward 1. Canonical results are training_ready=true and is_completed=true; the artifact contains two complete redacted litellm_proxy / provider_wire rows, 28,797 positive-usage tokens, priced cost, no capture errors or missing fields, and zero raw/encoded credential occurrences. No Claude credential name or value was present.
  • Claude OAuth → Codex Azure → Claude OAuth (5646fe83): reward 1 with seven complete schema-v2 exchanges and no manifest errors or missing fields. Both Claude sessions aggregate into four claude_otel_raw_body / agent_session / oauth_subscription exchanges; Codex contributes three litellm_proxy / provider_wire / api_key exchanges. The return to the original OAuth role succeeds after the API role's all-agent credential cleanup. Mixed usage reconciles exactly: 87,104 native + 43,708 provider = 130,812 total tokens. Raw, base64, and URL-encoded scans found zero Claude/Azure credential occurrences.
  • Mixed fidelity is explicit: the overall manifest is truthfully partial / mixed because it includes OAuth audit evidence. Codex rows retain provider_wire fidelity, but the mixed rollout is intentionally non-exportable as a whole; Prime-SFT and TRL-SFT each write zero rows with skipped_insufficient_capture_fidelity=1.
  • Role-transition custody (5646fe83): real Docker handoffs leave no agent-owned process from the outgoing role, keep the OTel sink in a root-owned BenchFlow infrastructure boundary, remove every registered native credential route before Codex proxy startup, and still restore OAuth configuration on the final Claude reconnect.

Supplemental failure-mode runs were produced on earlier reviewed heads of this branch and remain covered by final-head regressions:

  • Rejected Claude API-key and Bedrock credentials still emitted redacted, response-incomplete audit rows instead of false provider-wire success; Azure was never supplied to either Claude process.
  • Expired Codex OAuth still emitted the required empty JSONL plus truthful no_model_call / oauth_subscription manifests.
  • Root Codex/Azure retained complete audit rows but correctly failed strict training admission as non-training-grade.
  • Abrupt continuation-proxy termination preserved durable request/response rows with structural secret redaction.

Trainer and package proof

  • The final-head Codex Azure artifact converts to exactly 2 Prime-SFT rows and 2 TRL-SFT rows. Both validators pass --expected-rows 2 --source-jobs ... --require-llm-trajectory with non_training_grade_llm_trajectory=0.
  • --require-tool-calls is intentionally not claimed for these converted rows: the source ACP trajectory records one custom shell action, but Codex's provider requests declare no trainer-format tool definitions. The validator correctly rejects that optional gate rather than fabricating tool schemas.
  • A fresh benchflow-0.7.6.dev0 wheel built from exact head 27255749 installed successfully in a clean Linux Python 3.12 container; packaged custody resources were present and the bench, evaluation, and trainer CLI surfaces passed smoke checks.
  • Claude OAuth remains audit-only and is deliberately rejected by training-grade export while retaining its complete audit artifact.

Artifacts

  • /tmp/benchflow-pr1057-272557-live-claude-oauth.QnlVVk/2026-08-30__01-03-55/hello-world-task__862fa3cb
  • /tmp/benchflow-pr1057-272557-live-codex-azure.P1aY13/2026-08-30__01-05-45/hello-world-task__b64cc03c
  • /tmp/benchflow-pr1057-272557-claude-audit-export.KQsclo/
  • /tmp/benchflow-pr1057-272557-trainer.i7HLkW/
  • /tmp/benchflow-pr1057-272557-package.GdPwd1/
  • /tmp/benchflow-pr1057-final22-oauth-api-oauth._z4_k7ha/2026-08-30__00-04-19/hello-world-task__eb91faa1
  • /tmp/benchflow-pr1057-final22-mixed-trainer.J04Xsx/

Exact-head CI

  • test, pip-audit, parity, rollout-smoke, fixture-scenarios, and both detect-scope jobs are green on exact head 272557493b0f056502b1a90bd8bbfeb749ba9316.
  • Matrix/release jobs are skipped by their normal scope gates.

Artifacts

  • /tmp/benchflow-pr1057-remediation-claude-oauth/2026-08-29__16-43-43/hello-world-task__f37883b1
  • /tmp/benchflow-pr1057-remediation-codex-azure/2026-08-29__16-43-43/hello-world-task__fff6b39b
  • /tmp/benchflow-pr1057-final15-codex-azure/2026-08-29__22-51-02/hello-world-task__f9043e20
  • /tmp/benchflow-pr1057-final7-root-codex-azure/2026-08-29__20-35-47/hello-world-task__6a6a8648
  • /tmp/benchflow-pr1057-remediation-claude-api/2026-08-29__16-45-34/hello-world-task__7b910a48
  • /tmp/benchflow-pr1057-remediation-codex-oauth-json/2026-08-29__16-50-11/
  • /tmp/benchflow-pr1057-exact-claude-bedrock/2026-08-29__17-10-14/
  • /tmp/benchflow-pr1057-final15-trainer.zt29BT/
  • /tmp/benchflow-pr1057-final15-package.iJMIK4/
  • /tmp/benchflow-pr1057-final3-parity.cFCRow/
  • /var/folders/s8/dpnknm2j0kj9ww1r_nvtxdrr0000gq/T/benchflow-pr1057-abrupt-stop-in3z8w4d/

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-30T08:15:06.873390Z 2725574 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 05:30 — with GitHub Actions Active
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 05:33 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread src/benchflow/trajectories/llm_capture.py Outdated
Comment thread src/benchflow/trajectories/llm_capture.py Outdated
Comment thread src/benchflow/trajectories/export_prime_sft.py
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 05:35 — with GitHub Actions Active

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22861954d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture.py
Comment thread src/benchflow/trajectories/native_capture_parsers.py
Comment thread src/benchflow/trajectories/llm_capture_manifest.py Outdated
Comment thread tests/trajectories/test_native_llm_capture.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 06:26 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c3ae11b1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture_manifest.py Outdated
Comment thread src/benchflow/trajectories/llm_capture.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 06:40 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f905374331

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture.py Outdated
Comment thread src/benchflow/trajectories/native_capture_parsers.py
Comment thread src/benchflow/trajectories/llm_capture.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 06:54 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 744d43c58e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 07:08 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ffe3dcab87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/rollout/__init__.py
@bingran-you
bingran-you deployed to pypi-internal-preview August 29, 2026 07:26 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: effc13256a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture.py
Comment thread src/benchflow/trajectories/export_prime_sft.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa078e09c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/providers/litellm_runtime.py
Comment thread src/benchflow/providers/resources/provider_capture_custody.sh
@bingran-you
bingran-you deployed to pypi-internal-preview August 30, 2026 05:28 — with GitHub Actions Active
@bingran-you
bingran-you deployed to pypi-internal-preview August 30, 2026 05:29 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 200b1075a6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/llm_capture_records.py Outdated
@bingran-you
bingran-you deployed to pypi-internal-preview August 30, 2026 05:44 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3ac7f2dc4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/rollout/__init__.py
@bingran-you
bingran-you had a problem deploying to pypi-internal-preview August 30, 2026 06:07 — with GitHub Actions Failure
@bingran-you
bingran-you deployed to pypi-internal-preview August 30, 2026 06:12 — with GitHub Actions Active
@bingran-you
bingran-you deployed to pypi-internal-preview August 30, 2026 06:17 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b21b1aaeed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/agents/credentials.py
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5646fe8383

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/acp/runtime.py Outdated
Comment thread src/benchflow/continue_run/orchestrator.py
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd768e05d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/trajectories/results.py Outdated
Comment thread src/benchflow/trajectories/llm_capture_manifest.py
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 272557493b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant