Record ordinary judge call provenance#2136
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review Schema change (TRACE_VERSION 4→5) with new JudgeCall data structure, removal of extra_usage field, and refactored judge provenance recording. These structural changes to core trace infrastructure warrant human review. You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e90dc26940
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fb9a3e337
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0dbd0844d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8e9cfa58d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf9b15b471
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c3400c469
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f451009546
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb637f402e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fb637f4. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7ae41181d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1 participant
Overview
Make ordinary LLM judges more robust and observable with compact typed evidence while retaining the existing SDK client behavior.
Details
Trace.judge_calls; each record carries judge identity, response/refusal/parser evidence, and one existingModelCallfor model, sampling, endpoint, timing, usage, and provider errors.ProviderErrorpath.Note
Record judge call provenance with timing, usage, and errors in traces
JudgeCallin trace.py to capture one judge decision per call, including a nestedModelCallwith timing, token usage, finish reason, and any errors.Judge.completein judge.py to append aJudgeCalltotrace.judge_callsbefore the provider call, then populate it with the response or error on completion.Trace.extra_usageand replaces alltrace.record_judge/extra_usagerecording paths withjudge_calls; bumpsTRACE_VERSIONfrom 4 to 5.judge_callsinstead ofextra_usage; exported samples includejudge_callsin the info block.TRACE_VERSIONor readextra_usagewill need to be updated.Macroscope summarized 9193c6b.
Note
Medium Risk
Schema bump to trace v5 and removal of
extra_usagebreak compatibility with saved v4 traces and any external readers ofinfo["judge"]; judge scoring behavior changes slightly around structured output and error attribution onJudgeCall.Overview
Judge LLM calls are now first-class trace data instead of opaque blobs in
trace.info["judge"]plus a separateextra_usagelist. Each invocation appends aJudgeCall(judge class id, response text, refusal, parse/judge-level error) wrapping the sameModelCallshape used for agent turns (model, sampling, endpoint, timing, usage, provider error)—without persisting judge prompts.Judge.completewas reworked to go throughchat.completions.create(still via the OpenAI SDK), pre-append theJudgeCall, mapOpenAIErrorthrough the existing provider error path, and treat refusals, content filters, and length-truncated structured output as scoring failures recorded on the call.RubricJudgefunnels plain-text and structured grading through onecomplete+parsepath.Downstream consumers were updated: eval dashboard judge token/cost totals read
judge_calls, replay clearsjudge_calls(and skips failed traces viatrace.ok), platform push embeds judge calls underinfo["judge_calls"], andError.from_exceptionis shared with interception.TRACE_VERSIONis 5 andextra_usageis removed—older v4 traces are incompatible.Reviewed by Cursor Bugbot for commit 9193c6b. Bugbot is set up for automated code reviews on this repo. Configure here.