feat(locales): run language-neutral scenarios in de, es, zh, and ja

README.md

@@ -128,6 +128,17 @@ clarification followed by action, room capacities, transaction outcomes, paginat
 and changed identifiers. Private YAML packs remain unchanged. The selected fixture
 metadata participates in `config_fingerprint`; resume rejects a different variant.
 
+### Run in another language
+
+```bash
+tool-eval-bench run --locale de
+```
+
+`--locale` (`de`, `es`, `zh`, `ja`) sends translated user messages and tool descriptions while
+identifiers, schemas, and mock data stay fixed, and runs only the 26 scenarios whose grading is
+language-neutral. Each result also records whether the model drifted back into English. See
+[locale runs](https://github.com/SeraphimSerapis/tool-eval-bench/blob/main/docs/locales.md).
+
 See [methodology](https://github.com/SeraphimSerapis/tool-eval-bench/blob/main/docs/methodology.md#controlled-fixture-variants) for coverage and limits, and
 [capability diagnostics](https://github.com/SeraphimSerapis/tool-eval-bench/blob/main/docs/methodology.md#capability-diagnostics) for what reports record about toolset
 size, reasoning transport, and structured output.

changelog.d/+locale-runs.added.md

@@ -0,0 +1,9 @@
+**Locale runs with `--locale de|es|zh|ja`.** The user messages and tool descriptions of 26
+language-neutral scenarios can now be sent in German, Spanish, Simplified Chinese, or Japanese
+from checked-in translation packs, while identifiers, schemas, invariant anchors, and English mock
+data stay fixed. Answers are normalized for locale number formats and negation before the usual
+evaluators run, and each result records the first and final reply language so drift back into
+English shows up in the report. Scenarios that grade English prose are skipped. The locale is part
+of the run's fingerprint, so locale runs form their own leaderboard cohort and are not comparable
+with English runs. The Python API takes `locale=`. Adapted from QwenLM BabelArena's localized task
+families and its language-consistency audit. See `docs/locales.md`.

docs/cli-reference.md

@@ -609,3 +609,11 @@ else:
 of the sampling `--seed`. Unvaried scenarios remain controls. Variants are persisted in
 run configuration and comparison fingerprints; resume requires the same variant seed
 for affected scenarios. See [variant coverage](methodology.md#controlled-fixture-variants).
+
+### Locale runs
+
+`--locale {de,es,zh,ja}` translates the user messages and tool descriptions and runs only the 26
+scenarios whose grading is language-neutral; the rest are skipped. Identifiers, code, schemas, and
+each scenario's invariant anchors stay unchanged, and mock tool results stay in English. The locale
+is part of the run's fingerprint, and each result records first-reply and final-reply language as a
+diagnostic. See [locale runs](locales.md).

docs/locales.md

@@ -0,0 +1,79 @@
+# Locale runs
+
+`--locale` runs the benchmark with the user's side of the conversation in another language:
+German (`de`), Spanish (`es`), Simplified Chinese (`zh`), or Japanese (`ja`).
+
+```bash
+tool-eval-bench run --locale de
+tool-eval-bench run --hardmode --locale ja --variant-seed 1
+```
+
+The Python API takes `locale="de"` on `run_benchmark`.
+
+## What changes
+
+- The user message and any follow-up messages come from a checked-in translation pack,
+  `src/tool_eval_bench/evals/locales/<locale>.json`.
+- Every natural-language description in the tool schemas, for both the tool and its parameters,
+  is translated too.
+- Everything a mock or an evaluator checks stays the same: tool and parameter names, types, enum
+  values, identifiers, tickers, file names, code, and JSON schemas in fenced blocks. Each scenario
+  also lists invariant anchors in `ANCHORS` (`evals/locales/__init__.py`), the proper nouns its mock
+  matches, such as `Tokyo` or `Project Alpha`. Every translation must contain those verbatim.
+- Mock tool results stay in English. The model reads a translated request, translated tool
+  documentation, and English data, which is what it sees against a real English backend.
+- The system prompt and the benchmark's date line stay in English.
+
+## Which scenarios run
+
+A locale run includes only scenarios whose grading does not depend on English prose. Everything
+else is skipped, so locale scores cover a smaller set than English runs and are not comparable with
+them. The locale and pack version are part of the scenario variant metadata, so each locale gets
+its own `config_fingerprint` cohort, and resume rejects a different locale.
+
+A scenario is left out when:
+
+- its evaluator reads English discourse: refusals, clarification questions, explanations, claims
+  of success, or Markdown prose structure;
+- its mock or evaluator matches an ordinary English noun inside an argument the model writes, such
+  as a search query (`population iceland`), an email body, or a reminder text, which a model writing
+  in German would reasonably translate;
+- it tests language itself (the translation and German-response scenarios);
+- a fixture variant replaces its evaluator in a way that skips the locale wrapper (TC-61).
+
+The current list is the 26 IDs in `ANCHORS`. Adding a scenario means adding its anchors and a
+translation to all four packs; `tests/test_locales.py` checks that every pack covers every eligible
+scenario and tool string, keeps every anchor and fenced block, and that each scenario's reference
+trace still passes under every locale.
+
+## How answers are graded
+
+The scenario's own evaluator grades the run. Before it sees the answer, the locale layer rewrites
+two things the shared helpers parse with English rules:
+
+- **Numbers.** Full-width digits become ASCII. For `de` and `es`, dot-grouped thousands
+  (`4.250.000`) lose their dots and a decimal comma (`187,42`) becomes a point. A comma followed by
+  exactly three digits (`1,050`) is ambiguous and is left alone, which the helpers read as a
+  thousands separator. A trailing dollar sign (`1.020 $`) moves to the front.
+- **Negation.** Locale negation words become `not`, so "nicht 30" or "不是30" does not count as an
+  answer of 30. Japanese puts negation after the value (`30ではありません`), where the helpers'
+  look-back does not reach, so a negated Japanese number can still read as affirmed. This is a known
+  limit.
+
+## Language drift
+
+Models often start in the user's language and slide back into English after a few turns of English
+tool output. Each locale result records the detected language of the first and the last reply under
+the `language` diagnostic, for example `requested=de first=de final=en (drifted)`. It appears in the
+report's capability diagnostics and does not change the score.
+
+Detection is deliberately coarse: kana means Japanese, Han characters mean Chinese, and Latin text is
+scored against short English, German, and Spanish stopword lists after identifiers, code, and numbers
+are stripped. Short or mixed replies come back as `unknown` rather than a guess.
+
+## Maintaining the packs
+
+The packs were drafted by a model and need a native-speaker review before their results carry
+weight. When an English prompt or tool description changes, the pack test fails with the missing or
+stale string; update all four packs in the same change and bump `LOCALE_VERSION` when a translation's
+meaning changes, so old and new runs do not share a cohort.

pyproject.toml

@@ -144,6 +144,7 @@ package-dir = {"" = "src"}
 # glob keeps them together.
 tool_eval_bench = [
   "evals/yaml_scenarios/*.yaml",
+  "evals/locales/*.json",
   "plugins/decision/vendor/typed_decisions/*",
   "web/static/*",
   "py.typed",

src/tool_eval_bench/api.py

@@ -99,6 +99,7 @@ async def run_benchmark(
     seed: int | None = None,
     reference_date: str | None = None,
     variant_seed: int | None = None,
+    locale: str | None = None,
     concurrency: int = 1,
     error_rate: float = 0.0,
     alpha: float = 0.7,
@@ -147,6 +148,8 @@ async def run_benchmark(
         max_turns: Maximum conversation turns per scenario.
         seed: Random seed passed to the server.
         variant_seed: Select versioned scenario fixtures independently of sampling.
+        locale: Translate prompts and tool text into ``de``, ``es``, ``zh``, or ``ja``
+            and run only the scenarios whose grading is language-neutral.
         reference_date: Override benchmark reference date (``YYYY-MM-DD``).
         concurrency: Number of scenarios to run in parallel.
         error_rate: Inject random tool errors at this rate (0.0–1.0).
@@ -257,6 +260,7 @@ async def run_benchmark(
             seed=seed,
             reference_date=reference_date,
             variant_seed=variant_seed,
+            locale=locale,
             on_scenario_start=on_scenario_start,
             on_scenario_result=on_scenario_result,
             concurrency=concurrency,

src/tool_eval_bench/application/service.py

@@ -165,6 +165,7 @@ async def run_benchmark(
         max_turns: int = 8,
         seed: int | None = None,
         variant_seed: int | None = None,
+        locale: str | None = None,
         reference_date: str | None = None,
         on_scenario_start: OnScenarioStart | None = None,
         on_scenario_result: OnScenarioResult | None = None,
@@ -260,11 +261,16 @@ async def run_benchmark(
         # real score of 0.  A resume may legitimately have nothing left to run.
         if not resolved and resume_run_id is None:
             raise ValueError("No scenarios to run")
-        from tool_eval_bench.evals.variants import apply_variants
+        from tool_eval_bench.evals.variants import prepare_scenarios
 
-        resolved = apply_variants(resolved, variant_seed)
+        before_locale = resolved
+        resolved = prepare_scenarios(resolved, variant_seed=variant_seed, locale=locale)
+        if locale is not None and before_locale and not resolved:
+            raise ValueError(f"None of the selected scenarios support --locale {locale}.")
         if resume_scenarios is not None:
-            resume_scenarios = apply_variants(resume_scenarios, variant_seed)
+            resume_scenarios = prepare_scenarios(
+                resume_scenarios, variant_seed=variant_seed, locale=locale
+            )
         report_scenarios = resolved
 
         # Build metadata from RunContext (preferred) or legacy probe
@@ -383,7 +389,7 @@ async def capture_audit(
             # resume_scenarios. It ranks below the run's own definitions and
             # gets the run's variants, or preserved outcomes would be recorded
             # as unvarianted and move the run's fingerprint.
-            registry = apply_variants(ALL_SCENARIOS, variant_seed)
+            registry = prepare_scenarios(ALL_SCENARIOS, variant_seed=variant_seed, locale=locale)
             scenario_by_id = {s.id: s for s in [*registry, *(resume_scenarios or []), *resolved]}
             missing_ids = {r.scenario_id for r in merged_results} - scenario_by_id.keys()
             if missing_ids:

src/tool_eval_bench/cli/command_registry.py

@@ -57,6 +57,7 @@
     "hardmode",
     "hardmode_only",
     "variant_seed",
+    "locale",
     "scenario_pack",
     "pack_only",
     "include_held_out",

src/tool_eval_bench/cli/commands.py

@@ -85,13 +85,36 @@ def resolve_scenarios(args: argparse.Namespace) -> list[ScenarioDefinition]:
         unknown = sorted(requested - known_ids)
         if unknown:
             raise ValueError(f"Unknown scenarios: {', '.join(unknown)}")
-        return [s for s in base if s.id in requested]
-
-    if args.categories:
+        selected = [s for s in base if s.id in requested]
+    elif args.categories:
         cats = {c.upper() for c in args.categories}
-        return [s for s in base if s.category.value in cats]
+        selected = [s for s in base if s.category.value in cats]
+    else:
+        selected = base
+    return _locale_subset(args, selected)
+
 
-    return base
+def _locale_subset(
+    args: argparse.Namespace, selected: list[ScenarioDefinition]
+) -> list[ScenarioDefinition]:
+    """Keep the scenarios ``--locale`` supports, so previews and totals match the run.
+
+    Explicitly named IDs that cannot run in a locale are an error rather than a
+    silent skip.
+    """
+    locale = getattr(args, "locale", None)
+    if locale is None:
+        return selected
+    from tool_eval_bench.evals.locales import ineligible_ids, is_eligible
+
+    if args.scenarios and (named := ineligible_ids(selected)):
+        raise ValueError(
+            f"--locale {locale} does not support {', '.join(named)}; see docs/locales.md."
+        )
+    kept = [s for s in selected if is_eligible(s.id)]
+    if not kept:
+        raise ValueError(f"None of the selected scenarios support --locale {locale}.")
+    return kept
 
 
 def resolve_all_scenarios_for_ids(

src/tool_eval_bench/cli/legacy_parser.py

@@ -200,6 +200,13 @@ def _make_parser() -> argparse.ArgumentParser:
         default=None,
         help="Use versioned deterministic fixture variants; unchanged scenarios remain controls.",
     )
+    select.add_argument(
+        "--locale",
+        choices=("de", "es", "zh", "ja"),
+        default=None,
+        metavar="LANG",
+        help="Prompt language: de, es, zh, ja (language-neutral scenarios only)",
+    )
     select.add_argument(
         "--hardmode",
         action="store_true",

src/tool_eval_bench/cli/scored_run.py

@@ -77,6 +77,7 @@ class ScoredRun:
     reference_date: str | None
     seed: int | None
     variant_seed: int | None
+    locale: str | None
     concurrency: int
     error_rate: float
     alpha: float
@@ -140,6 +141,7 @@ def from_args(
             reference_date=args.reference_date,
             seed=args.seed,
             variant_seed=getattr(args, "variant_seed", None),
+            locale=getattr(args, "locale", None),
             concurrency=args.parallel,
             error_rate=args.error_rate,
             alpha=args.alpha,
@@ -200,6 +202,7 @@ def service_kwargs(self) -> dict[str, Any]:
             "reference_date": self.reference_date,
             "system_prompt": self.system_prompt,
             "variant_seed": self.variant_seed,
+            "locale": self.locale,
             "seed": self.seed,
             "concurrency": self.concurrency,
             "error_rate": self.error_rate,
@@ -228,9 +231,13 @@ def config_scenarios(self) -> list[ScenarioDefinition]:
         A resumed run records its complete original protocol, not the rerun
         subset, with the run's variants applied, exactly as the service does.
         """
-        from tool_eval_bench.evals.variants import apply_variants
+        from tool_eval_bench.evals.variants import prepare_scenarios
 
-        return apply_variants(self.resume_scenarios or self.scenarios, self.variant_seed)
+        return prepare_scenarios(
+            self.resume_scenarios or self.scenarios,
+            variant_seed=self.variant_seed,
+            locale=self.locale,
+        )
 
     def run_settings(self) -> RunSettings:
         """The persisted-config view of this run; resume compares it with the stored one."""

src/tool_eval_bench/evals/helpers.py

@@ -215,7 +215,8 @@ def positive_argument_contains(value: Any, expected: str) -> bool:
     """
     text = as_str(value).strip().lower()
     target = expected.strip().lower()
-    if not text or not re.search(rf"(?<!\w){re.escape(target)}(?!\w)", text):
+    boundary = r"[a-z0-9_]" if target.isascii() else r"\w"
+    if not text or not re.search(rf"(?<!{boundary}){re.escape(target)}(?!{boundary})", text):
         return False
     return not re.search(
         rf"\b(?:not|no|without|except|exclude|excluding)\s+(?:the\s+)?{re.escape(target)}\b",
@@ -236,8 +237,8 @@ def answer_contains_number(answer: str, value: str) -> bool:
 # the few words immediately before the value.  A wider window rejects correct
 # answers, which is exactly as damaging as accepting negated ones.
 _CLAUSE_BOUNDARY = re.compile(
-    r"[.!?;:,()\[\]\n–—]|\s-\s|"
-    r"\b(?:but|however|instead|and|or|while|whereas|though|although|yet)\b"
+    r"[.!?;:,()\[\]\n–—，。！？；：、（）【】「」『』]|\s-\s|"
+    r"\b(?:but|however|instead|and|or|while|whereas|though|although|yet|sondern|aber|pero|sino)\b"
 )
 _NEGATION = re.compile(r"\b(?:not|never|no|neither|nor|without)\b|n't\b")
 # Function words carry no distance ("could not find a price of 187" is still a
@@ -258,7 +259,7 @@ def answer_contains_number(answer: str, value: str) -> bool:
 _SCOPE_BREAKERS = frozenset(
     "at in on for to from by with near around before after until past over under into onto than as "
     "is are was were be been being am has have had will would shall should can could may might must "
-    "do does did".split()
+    "do does did ist sind war waren es son fue fueron".split()
 )
 
 
@@ -397,7 +398,10 @@ def answer_affirms_pattern(
 
 def answer_affirms_text(answer: str, value: str) -> bool:
     """Return whether a textual value is asserted rather than negated."""
-    return answer_affirms_pattern(answer, re.compile(rf"\b{re.escape(value)}\b", re.IGNORECASE))
+    boundary = r"[a-z0-9_]" if value.isascii() else r"\w"
+    return answer_affirms_pattern(
+        answer, re.compile(rf"(?<!{boundary}){re.escape(value)}(?!{boundary})", re.IGNORECASE)
+    )
 
 
 # ---------------------------------------------------------------------------