docs: survey code agent benchmarks and their headless interface requirements - #20
Merged
Conversation
- fix the terminal.py isatty citation (:20, not :22) - mark Opus 5's Terminal-Bench and DeepSWE figures as third-party, and drop the AutomationBench check for GLM-5.2, which never reported it itself - add a labelled example to every benchmark subsection - correct the FrontierSWE dominance range against the percentages in the tables - flag three unresolved source conflicts: GLM-5.2's Toolathlon score, GLM's two Terminal-Bench rows, and AA's 84-task Terminal-Bench v2 subset - align four wording drifts between the Chinese source and the English version
Add a source-level survey of the headless interface required by Terminal-Bench 2.1 / Harbor, SWE-bench Verified, and NL2Repo-Bench, in both Chinese (source) and English. - Terminal-Bench: the BaseInstalledAgent adapter interface, both official invocation examples, and eight constraints read out of the Harbor source -- most notably that a non-zero exit code marks the whole trial as an agent failure. - SWE-bench: the predictions format plus the runner-side essentials (image naming, /testbed, budgets, patch collection). - NL2Repo-Bench: scoring is decoupled from the agent, so the only requirement is running one prompt headlessly in a given directory. Revise code_agent_benchmark.md against those conclusions: invert the exit-code semantics (turn-limit overrun and timeout now exit 0), move the working-directory item from P0 to P2 and renumber P1/P2, simplify the retry requirement in favour of Harbor's own error classification, and correct the DeepSWE leaderboard wording -- it pins the harness to mini-swe-agent rather than gating submissions on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two research notes on code-agent benchmarks — bilingual, hand-written Chinese
source plus generated English — and the 0.8.x changelog they open.
What's here
code_agent_benchmark.md— a survey of the code-agent benchmarks reportedacross six 2026 model releases (DeepSeek-V4-Flash-0731, Claude Opus 5, GPT-5.6
Sol, Qwen3.8-27B, Kimi-K3, GLM-5.2), flattened into one table. Terminal-Bench
2.1 and DeepSWE turn out to be the only two every release has a number for. It
closes with a recommendation for this project — Terminal-Bench 2.1 first, a
SWE-bench Verified subset second, NL2Repo-Bench third — and a P0/P1/P2 list of
what nanoPyCodeAgent still lacks.
benchmark_headless_interface.md— a follow-up that reads the integrationsurface of those three benchmarks down to the source level:
BaseInstalledAgentadapter interface,both official invocation examples (Claude Code pipes the instruction over
stdin, mini-swe-agent passes
--task=), and eight constraints that are inthe source but not in the documentation.
format. The runner-side essentials (image naming,
/testbed, step/costbudgets, patch collection) come from mini-swe-agent.
is running one prompt headlessly in a given directory; integrating means
forking its OpenHands runner.
The most important finding: a harness reads a non-zero exit code as its own
failure, so an agent that runs out of turns without solving the task must
still exit 0. The first draft of the survey had this backwards.
Revisions to the earlier survey
it through the container's default WORKDIR — with P1/P2 renumbered.
retries via
--retry-include, so the agent only needs basic backoff plusprinting errors verbatim.
mini-swe-agent for consistency rather than gating submissions on it, which
means a self-built harness has no place on that leaderboard.
leaderboard entries are two-dimensional (harness + model), so this project
has a legitimate place on it.
Docs bookkeeping
docs/changelogs/0.8.x.md, with both surveys under[Unreleased].docs/research/README.md.No code changes.