Skip to content

docs: survey code agent benchmarks and their headless interface requirements - #20

Merged
minixalpha merged 4 commits into
mainfrom
claude/benchmark_research
Aug 18, 2026
Merged

docs: survey code agent benchmarks and their headless interface requirements#20
minixalpha merged 4 commits into
mainfrom
claude/benchmark_research

Conversation

@minixalpha

Copy link
Copy Markdown
Owner

Two research notes on code-agent benchmarks — bilingual, hand-written Chinese
source plus generated English — and the 0.8.x changelog they open.

What's here

code_agent_benchmark.md — a survey of the code-agent benchmarks reported
across six 2026 model releases (DeepSeek-V4-Flash-0731, Claude Opus 5, GPT-5.6
Sol, Qwen3.8-27B, Kimi-K3, GLM-5.2), flattened into one table. Terminal-Bench
2.1 and DeepSWE turn out to be the only two every release has a number for. It
closes with a recommendation for this project — Terminal-Bench 2.1 first, a
SWE-bench Verified subset second, NL2Repo-Bench third — and a P0/P1/P2 list of
what nanoPyCodeAgent still lacks.

benchmark_headless_interface.md — a follow-up that reads the integration
surface of those three benchmarks down to the source level:

  • Terminal-Bench 2.1 / Harbor: the BaseInstalledAgent adapter interface,
    both official invocation examples (Claude Code pipes the instruction over
    stdin, mini-swe-agent passes --task=), and eight constraints that are in
    the source but not in the documentation.
  • SWE-bench Verified: no agent interface at all — only the predictions
    format. The runner-side essentials (image naming, /testbed, step/cost
    budgets, patch collection) come from mini-swe-agent.
  • NL2Repo-Bench: scoring is decoupled from the agent, so the only requirement
    is running one prompt headlessly in a given directory; integrating means
    forking its OpenHands runner.

The most important finding: a harness reads a non-zero exit code as its own
failure
, so an agent that runs out of turns without solving the task must
still exit 0. The first draft of the survey had this backwards.

Revisions to the earlier survey

  • Exit-code semantics inverted (P0-2).
  • The working-directory item moved from P0 to P2 — all three benchmarks deliver
    it through the container's default WORKDIR — with P1/P2 renumbered.
  • The retry requirement simplified: Harbor classifies API errors itself and
    retries via --retry-include, so the agent only needs basic backoff plus
    printing errors verbatim.
  • The DeepSWE leaderboard wording corrected: it pins the harness to
    mini-swe-agent for consistency rather than gating submissions on it, which
    means a self-built harness has no place on that leaderboard.
  • Terminal-Bench's reasons reordered — the strongest one is that its
    leaderboard entries are two-dimensional (harness + model), so this project
    has a legitimate place on it.

Docs bookkeeping

  • New docs/changelogs/0.8.x.md, with both surveys under [Unreleased].
  • Both notes indexed in docs/research/README.md.
  • The English version of each note is in sync with its Chinese source.

No code changes.

- fix the terminal.py isatty citation (:20, not :22)
- mark Opus 5's Terminal-Bench and DeepSWE figures as third-party, and drop
  the AutomationBench check for GLM-5.2, which never reported it itself
- add a labelled example to every benchmark subsection
- correct the FrontierSWE dominance range against the percentages in the tables
- flag three unresolved source conflicts: GLM-5.2's Toolathlon score, GLM's two
  Terminal-Bench rows, and AA's 84-task Terminal-Bench v2 subset
- align four wording drifts between the Chinese source and the English version
Add a source-level survey of the headless interface required by
Terminal-Bench 2.1 / Harbor, SWE-bench Verified, and NL2Repo-Bench, in
both Chinese (source) and English.

- Terminal-Bench: the BaseInstalledAgent adapter interface, both
  official invocation examples, and eight constraints read out of the
  Harbor source -- most notably that a non-zero exit code marks the
  whole trial as an agent failure.
- SWE-bench: the predictions format plus the runner-side essentials
  (image naming, /testbed, budgets, patch collection).
- NL2Repo-Bench: scoring is decoupled from the agent, so the only
  requirement is running one prompt headlessly in a given directory.

Revise code_agent_benchmark.md against those conclusions: invert the
exit-code semantics (turn-limit overrun and timeout now exit 0), move
the working-directory item from P0 to P2 and renumber P1/P2, simplify
the retry requirement in favour of Harbor's own error classification,
and correct the DeepSWE leaderboard wording -- it pins the harness to
mini-swe-agent rather than gating submissions on it.
@minixalpha
minixalpha merged commit 07cac13 into main Aug 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant