openenv/tbench2: adapter-driven TB2 fidelity (no OpenEnv fork/vendor)

[Shi-Dong]

Summary

• Drives full Terminal-Bench-2 fidelity from the miles adapter instead of patching OpenEnv: agent commands and scoring run in the task image's real /app workdir, and scoring uses the task's canonical tests/test.sh -> /logs/verifier/reward.txt harness, driven through plain exec steps.
• Keeps upstream OpenEnv (huggingface/OpenEnv) unmodified and unvendored — the env server is used purely as a docker-exec transport.

Why

Upstream's Tbench2DockerEnvironment docker mode is low-fidelity: it runs the container in /task (a copy of the task source), drops the agent there instead of /app (the task image WORKDIR), and scores via bare pytest tests/ rather than TB2's canonical test.sh (which pins the pytest toolchain, copies test.py into /app, runs test_outputs.py, and writes reward.txt). That mis-drives and mis-scores essentially all 89 TB2 tasks. Rather than fork or vendor OpenEnv to fix it, this moves the TB2-specific fidelity logic into the adapter, where the rest of the TB2 integration already lives.

What changed (openenv_agent_function.py)

• OPENENV_TASK_WORKDIR (default /app): every agent command is prefixed to run in the real task workdir.
• Scoring replaces the built-in evaluate action with an exec that stages the task tests to /tests, runs the canonical test.sh in /app, and parses reward.txt.
• OPENENV_TB2_TESTS_SRC (default /task/tests): where the unmodified upstream env stages the task's tests inside the container.

Validation

Prototyped end-to-end against a pristine origin/main Tbench2DockerEnvironment on a real docker daemon: reset -> apply reference solution in /app -> canonical eval -> reward = 1.0 (task cancel-async-tasks).

Caveat / deployment

This intentionally couples the adapter to unmodified upstream OpenEnv (which copies the task dir, tests included, into /task). It is incompatible with the earlier docker-patched snapshot path. Deployment therefore hosts the unmodified upstream env-server on a real-docker host (e.g. agent-server) rather than the serverless Daytona snapshot.

Test plan

• Full-89 docker-mode smoke on a real-docker host, compared against a base reward baseline
• 1-node training smoke run

[gemini-code-assist]

Code Review

This pull request introduces adapter-driven Terminal-Bench-2 fidelity support to the OpenEnv agent function. It configures the agent to run commands in the correct task working directory and executes the canonical test.sh harness via an exec step rather than using the built-in non-canonical evaluate action. Feedback was provided to address potential stale reward leakage, ensure correct workdir fallback behavior when OPENENV_TASK_WORKDIR is disabled, and avoid shared temporary directory conflicts on multi-user clusters by appending the user's UID to the log path.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.