Skip to content

fix(rollout): keep completed rollouts when publish-path execs time out - #1043

Open
tulerfeng wants to merge 1 commit into
benchflow-ai:mainfrom
tulerfeng:fix/948-publish-path-exec-timeout
Open

fix(rollout): keep completed rollouts when publish-path execs time out#1043
tulerfeng wants to merge 1 commit into
benchflow-ai:mainfrom
tulerfeng:fix/948-publish-path-exec-timeout

Conversation

@tulerfeng

@tulerfeng tulerfeng commented Aug 20, 2026

Copy link
Copy Markdown

Fixes #948.

Summary

Makes both publish-path exec calls non-fatal, so a container too slow for their
hardcoded 10s budget no longer discards a rollout whose work already succeeded.

Root cause

_publish_trajectory_for_verifier runs mkdir -p /logs/agent at timeout_sec=10.
Publishing happens after the agent has run to completion, so on a loaded host —
where Docker exec setup alone can exceed 10s — the RuntimeError propagates and
throws away a ~40 minute rollout over a bookkeeping step.

On mounted backends that exec cannot help: agent_dir.mkdir() two lines above
already created the directory Docker bind-mounts to /logs/agent. On backends
that don't mirror, a genuinely missing directory still surfaces through the
upload_file immediately after.

_scrape_agent_trajectory carries the same 10s exec and is awaited unguarded by
verify() before the publish call, so it hits the failure first. Every other
failure in that fallback already degrades to "no scraped trajectory" (non-zero
return code, unparseable JSON); the timeout was the one case that propagated
instead.

Safety

  • upload_file stays fatal — suppressing it too would let a rollout claim a
    trajectory the verifier never received.
  • Both guards log a warning rather than using contextlib.suppress(Exception) as
    the issue suggests. The exec failing is a real signal about host health, and the
    visible RuntimeError is what made the reported incident diagnosable; silencing
    it removes the only trace. Happy to switch to suppress for a smaller diff.

End-to-end Validation

Docker sandbox, oracle agent, single task, injecting a slow exec at the mkdir
call site:

Injection Branch Outcome
none (control) this branch reward 1.00
sleep 30 && mkdir -p /logs/agent main rollout lost — RuntimeError: Command timed out after 10 seconds
sleep 30 && mkdir -p /logs/agent this branch warning logged, reward 1.00, both trajectory copies written
sleep 600 && mkdir -p /logs/agent this branch identical to the 30s row — the client-side budget aborts at 10s either way

Regression tests (the first and third fail with their fix reverted; the second
guards against over-suppression and passes either way):

Test Guards
test_publish_trajectory_survives_mkdir_timeout publish completes and writes both trajectory copies when the mkdir exec times out
test_publish_trajectory_still_raises_on_upload_failure a failing upload_file stays fatal
test_scrape_agent_trajectory_survives_exec_timeout scrape degrades to [] instead of aborting verify()
  • uv run python -m pytest tests/test_rollout_upload.py tests/test_capture_trajectory.py
  • uv run python -m pytest — 5667 passed; the only failures are the 11 in
    tests/test_cli_live_progress.py, which reproduce unchanged on a clean checkout
    (terminal-width-dependent assertions)
  • uv run ruff check .
  • uv run ruff format --check .
  • git diff --check

Fixes #948.

`_publish_trajectory_for_verifier` ran `mkdir -p /logs/agent` at a hardcoded
`timeout_sec=10`. Publishing happens after the agent has run to completion, so
on a loaded host — where Docker exec setup alone can exceed 10s — the
RuntimeError propagated and discarded a rollout whose expensive work had already
succeeded.

On mounted backends that exec cannot help: `agent_dir.mkdir()` two lines above
already created the directory Docker bind-mounts to `/logs/agent`. On backends
that don't mirror, a genuinely missing directory still surfaces through the
`upload_file` immediately after, which stays fatal.

`_scrape_agent_trajectory` carries the same 10s exec and is awaited unguarded by
`verify()` before the publish call, so it reaches the failure first. Every other
failure in that fallback already degrades to "no scraped trajectory"; the
timeout was the one case that propagated instead.

Both guards log a warning rather than suppressing silently — the exec failing is
a real signal about host health.

Fixes benchflow-ai#948.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardcoded 10s timeout on mkdir -p /logs/agent discards completed rollouts under I/O load

1 participant