Skip to content

fix(judge): parse native edit targets - #1069

Open
kywch wants to merge 2 commits into
mainfrom
fix/judge-native-edit-scratch-paths
Open

fix(judge): parse native edit targets#1069
kywch wants to merge 2 commits into
mainfrom
fix/judge-native-edit-scratch-paths

Conversation

@kywch

@kywch kywch commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Parse exact trailing Editing <path> targets from native OpenHands ACP edit titles.
  • Keep suffix parsing limited to edit events; malformed, trailing-prose, and unrecognized titles retain conservative full-title scanning.
  • Normalize POSIX paths before applying scratch-root exemptions, and make shell-path masking use the same normalized check.

Why

A confirmed live rollout earned reward 1.0 and passed the LLM judge, but the mechanical tamper gate rejected native edits to /tmp/test_logistic.py and /tmp/test_rnn.py. OpenHands recorded those targets in titles such as Create test script: Editing /tmp/test_rnn.py; scanning the whole title matched the test-like filename before the scratch-path exemption could apply.

Failed job: https://github.com/benchflow-ai/benchflow/actions/runs/31622440593/job/94200401493

Behavior

The scanner now extracts only an exact trailing Editing <path> target for native ACP edit events. Protected targets remain flagged. Non-edit events and malformed titles remain fail-closed. Lexical normalization prevents paths such as /tmp/../verifier/test.sh from receiving the scratch exemption, including during shell-command masking.

Tests

  • observed benign native /tmp edit title
  • protected native verifier target
  • malformed trailing title fallback
  • non-edit title fallback
  • scratch-root traversal through a shell command

Focused result: 59 passed, 1 deselected. Ruff check and format checks pass.

Provenance

Extracts and hardens the judge fix originally authored in #979 commit bc9a1c3. This separates judge behavior from #979 and #1030 Gemini model-routing scope while preserving the original author through the existing repository author identity.


Devin Review

@kywch
kywch deployed to pypi-internal-preview August 30, 2026 21:35 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread tests/integration/agent_judge.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant