Model hidden sandbox as a persistent agent

[xeophon]

Overview

Model iterative hidden-sandbox evaluation as a pair of long-lived agents. The solver and hidden judge now alternate through the existing interaction lifecycle, so ACP-backed harness continuation, runtime ownership, limits, traces, and teardown are shared with the multi-agent framework.

Details

• Add a reusable sandbox-judge environment with independent solver and judge AgentConfig roles.
• Copy a bounded, validated workspace snapshot into the persistent offline judge runtime for each submission.
• Relay only validated score, feedback, and numeric metrics back to the solver while retaining a trace-visible submission ledger.
• Reuse agentic-judge reward composition and select the best submission across the episode.
• Add narrow interaction support for accessing the live runtime and suppressing task-specific stop decorators during environment-controlled iterative exchanges.
• Validate runtime topology and artifact shape at the environment boundary.

Example

A private judge image can contain the hidden tests and grader while the solver uses its own long-lived workspace:

num_tasks = 10

[env]
id = "sandbox-judge"

[env.taskset]
id = "my-coding-taskset-v1"

[env.solver]
model = "openai/gpt-5.4"
max_turns = 10

[env.solver.harness]
id = "codex"

[env.solver.runtime]
type = "docker"

[env.judge]
model = "openai/gpt-5.4"
max_turns = 5

[env.judge.harness]
id = "bash"

[env.judge.runtime]
type = "docker"
image = "ghcr.io/example/private-grader:latest"
allow = []

[env.submission]
submit_paths = ["."]
max_submissions = 5
threshold = 1.0

Run it with uv run eval @ sandbox-judge.toml. The solver produces a submission in its persistent workspace; the environment copies the bounded artifact into the judge agent's persistent private runtime; the judge returns validated score and feedback; and the solver resumes through the same harness session.

Note

Add SandboxJudgeEnv to evaluate solver submissions via a hidden judge agent

• Introduces verifiers/v1/envs/sandbox_judge/env.py with a two-agent loop where the solver submits work and a hidden judge iterates verdicts, with early stopping on score threshold or submission cap.
• Archives solver workspaces inside the solver sandbox using a bounded shell pipeline (_snapshot), then validates and installs them into the judge runtime (_install) before each evaluation round.
• Validates tar archives pre-extraction to reject path-traversal attacks, excluded paths, non-regular files, and oversized submissions.
• Enforces isolation at config time: the solver must not use SubprocessConfig, and the judge runtime must be a NetworkPolicyConfig with an empty allow list.
• Adds respect_task_stops=False support to Agent.interaction and RolloutRun so the solver's task stop markers are ignored during the multi-turn loop.
• Risk: the judge agent is marked non-trainable; existing ScoreConfig weights (task, judge) now reject non-finite values, which is a breaking validation change.

^{Macroscope summarized bd6c005. (Automatic summaries will resume when PR exits draft mode or review begins).}