[SeraphimSerapis]

Problem

The benchmark only measures tool use in English. QwenLM BabelArena shows that models lose ground when the user, the tool documentation, and the error strings speak another language, even when every identifier stays the same, and that models often start in the user's language and drift back to English after a few turns of English tool output. Neither effect is visible here today.

Change

--locale de|es|zh|ja (and locale= on run_benchmark) runs the scenarios whose grading is language-neutral with translated user messages, follow-up messages, and tool descriptions.

• Packs: src/tool_eval_bench/evals/locales/<locale>.json, one per locale, checked in and deterministic. They were drafted by a model and still need native-speaker review (see below).
• What stays fixed: tool and parameter names, types, enum values, identifiers, code, fenced JSON schemas, and per-scenario invariant anchors (ANCHORS) such as Tokyo, Project Alpha, or INC-710, which every translation must contain verbatim. Mock tool results stay in English.
• Scope: 26 scenarios. A scenario is excluded when its evaluator reads English prose (refusals, clarifications, explanations, success claims), when its mock matches an ordinary English noun in text the model writes (search queries, email bodies, titles), when it tests language itself, or when a fixture variant bypasses the locale wrapper (TC-61). The CLI applies the filter during selection, so --dry-run and progress totals match the run. An explicitly named ineligible ID is an error, not a silent skip.
• Grading: answers are normalized before the unchanged evaluators run: full-width digits, de/es decimal commas and dot-grouped thousands, a trailing $, and locale negation words mapped to not.
• Drift diagnostic: each result records requested=… first=… final=… under the language diagnostic, with (drifted) when the model started in the requested language and ended in another. It appears in the report's capability diagnostics. Scores are not affected.
• Cohorts: locale and pack version join the scenario variant metadata, so locale runs get their own config_fingerprint and resume rejects a different locale. The locale composes with --variant-seed: the locale applies first, then seeded fixtures rewrite the translated prompt.

Adapted from BabelArena's localized task families and its language-consistency audit.

Known limits

• Japanese places negation after the value (30ではありません), outside the shared helpers' look-back, so a negated Japanese number can still read as affirmed.
• The system prompt and the benchmark date line stay in English.
• Language detection is coarse (script, then stopwords) and returns unknown for short or mixed replies.

Review needed before merge

The packs need a native-speaker pass. de.json is the first priority.

Regression coverage

tests/test_locales.py (248 tests):

• Each pack covers exactly the eligible scenarios and tool strings, with nothing stale. Every translation keeps its anchors, fenced blocks, and follow-up count, is not left in English, and is not detected as English.
• Every eligible scenario's reference trace passes under every locale.
• Number and negation normalization, including negative cases ("nicht 30", "不是30") and the ambiguous 1,050.
• Language detection, plus a drift test showing the grade is unchanged.
• Composition with --variant-seed, fingerprint metadata, and CLI selection (filtering, rejecting a named ineligible ID, an empty selection, an unsupported locale). The CLI flag reaches the service and the recorded config.

The CLI golden test and the compatibility snapshots are updated (snapshots via scripts/update_compat_snapshots.py). A locally built wheel includes the four packs.

Local: ruff, ruff format, mypy, and 10,215 tests pass with seed 104729.

Changelog and documentation

• changelog.d/+locale-runs.added.md
• New docs/locales.md. README and docs/cli-reference.md gain short sections that link to it.

Contributor checklist

• This PR contains one focused, logically related change.
• Regression tests were added or a maintainer applied tests-not-needed.
• A changelog.d/ fragment was added or updated, or a maintainer applied skip-changelog.
• Fork PRs enable Allow edits from maintainers.
• No credentials, live endpoints, generated run artifacts, or unrelated formatting changes are included.

Written with AI assistance; reviewed before opening.