fix(rollout): keep completed rollouts when publish-path execs time out - #1043
Open
tulerfeng wants to merge 1 commit into
Open
fix(rollout): keep completed rollouts when publish-path execs time out#1043tulerfeng wants to merge 1 commit into
tulerfeng wants to merge 1 commit into
Conversation
`_publish_trajectory_for_verifier` ran `mkdir -p /logs/agent` at a hardcoded `timeout_sec=10`. Publishing happens after the agent has run to completion, so on a loaded host — where Docker exec setup alone can exceed 10s — the RuntimeError propagated and discarded a rollout whose expensive work had already succeeded. On mounted backends that exec cannot help: `agent_dir.mkdir()` two lines above already created the directory Docker bind-mounts to `/logs/agent`. On backends that don't mirror, a genuinely missing directory still surfaces through the `upload_file` immediately after, which stays fatal. `_scrape_agent_trajectory` carries the same 10s exec and is awaited unguarded by `verify()` before the publish call, so it reaches the failure first. Every other failure in that fallback already degrades to "no scraped trajectory"; the timeout was the one case that propagated instead. Both guards log a warning rather than suppressing silently — the exec failing is a real signal about host health. Fixes benchflow-ai#948.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #948.
Summary
Makes both publish-path
execcalls non-fatal, so a container too slow for theirhardcoded 10s budget no longer discards a rollout whose work already succeeded.
Root cause
_publish_trajectory_for_verifierrunsmkdir -p /logs/agentattimeout_sec=10.Publishing happens after the agent has run to completion, so on a loaded host —
where Docker exec setup alone can exceed 10s — the
RuntimeErrorpropagates andthrows away a ~40 minute rollout over a bookkeeping step.
On mounted backends that exec cannot help:
agent_dir.mkdir()two lines abovealready created the directory Docker bind-mounts to
/logs/agent. On backendsthat don't mirror, a genuinely missing directory still surfaces through the
upload_fileimmediately after._scrape_agent_trajectorycarries the same 10s exec and is awaited unguarded byverify()before the publish call, so it hits the failure first. Every otherfailure in that fallback already degrades to "no scraped trajectory" (non-zero
return code, unparseable JSON); the timeout was the one case that propagated
instead.
Safety
upload_filestays fatal — suppressing it too would let a rollout claim atrajectory the verifier never received.
contextlib.suppress(Exception)asthe issue suggests. The exec failing is a real signal about host health, and the
visible
RuntimeErroris what made the reported incident diagnosable; silencingit removes the only trace. Happy to switch to
suppressfor a smaller diff.End-to-end Validation
Docker sandbox, oracle agent, single task, injecting a slow exec at the
mkdircall site:
sleep 30 && mkdir -p /logs/agentmainRuntimeError: Command timed out after 10 secondssleep 30 && mkdir -p /logs/agentsleep 600 && mkdir -p /logs/agentRegression tests (the first and third fail with their fix reverted; the second
guards against over-suppression and passes either way):
test_publish_trajectory_survives_mkdir_timeoutmkdirexec times outtest_publish_trajectory_still_raises_on_upload_failureupload_filestays fataltest_scrape_agent_trajectory_survives_exec_timeout[]instead of abortingverify()uv run python -m pytest tests/test_rollout_upload.py tests/test_capture_trajectory.pyuv run python -m pytest— 5667 passed; the only failures are the 11 intests/test_cli_live_progress.py, which reproduce unchanged on a clean checkout(terminal-width-dependent assertions)
uv run ruff check .uv run ruff format --check .git diff --checkFixes #948.