fix(runtime): reap the sandbox container when Caido bootstrap fails#831
fix(runtime): reap the sandbox container when Caido bootstrap fails#831thejesh23 wants to merge 2 commits into
Conversation
`create_or_reuse` starts a container, then resolves the Caido port and runs `bootstrap_caido` before storing the bundle in `_SESSION_CACHE`. If either of those raises, the exception propagates with the container still running and nothing tracking it: `cleanup` reaps only cached bundles, so `cleanup(scan_id)` returns early and the container is orphaned. `bootstrap_caido` giving up after its 10 guest-login attempts is a routine failure on a loaded machine, so each failed scan start leaked one container along with its exposed port and bind mounts. Wrap the post-start bootstrap in `try/except BaseException`, delete the session best-effort, and re-raise the original error. The delete path is now shared with `cleanup` as `_delete_quietly`, so both teardown sites release the docker client the same way. Adds `tests/test_session_bootstrap_failure.py` covering both failure points; both tests fail on main with "started container was never reaped".
Greptile SummaryThis PR cleans up sandbox containers when Caido startup fails. The main changes are:
Confidence Score: 4/5The cancellation cleanup path can still leave an uncached container running.
strix/runtime/session_manager.py Important Files Changed
Prompt To Fix All With AIFix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
strix/runtime/session_manager.py:149
**Second Cancellation Interrupts Teardown**
If another `Task.cancel()` arrives while `_delete_quietly` awaits `client.delete(session)`, its `CancelledError` escapes the helper and deletion can stop midway. Because this session was never cached, later `cleanup(scan_id)` is a no-op, leaving the container running and replacing the original bootstrap failure with the second cancellation.
Reviews (1): Last reviewed commit: "fix(runtime): reap the sandbox container..." | Re-trigger Greptile |
| "Sandbox bootstrap failed for scan %s; deleting the container it started", | ||
| scan_id, | ||
| ) | ||
| await _delete_quietly(client, session, scan_id) |
There was a problem hiding this comment.
Second Cancellation Interrupts Teardown
If another Task.cancel() arrives while _delete_quietly awaits client.delete(session), its CancelledError escapes the helper and deletion can stop midway. Because this session was never cached, later cleanup(scan_id) is a no-op, leaving the container running and replacing the original bootstrap failure with the second cancellation.
Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/runtime/session_manager.py
Line: 149
Comment:
**Second Cancellation Interrupts Teardown**
If another `Task.cancel()` arrives while `_delete_quietly` awaits `client.delete(session)`, its `CancelledError` escapes the helper and deletion can stop midway. Because this session was never cached, later `cleanup(scan_id)` is a no-op, leaving the container running and replacing the original bootstrap failure with the second cancellation.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
Confirmed and fixed in 0893e65.
You're right that this reopens the exact leak the PR closes: _delete_quietly catches Exception, not CancelledError, and because the session was never cached cleanup() can't retry.
with contextlib.suppress(BaseException):
await asyncio.shield(_delete_quietly(client, session, scan_id))
raiseshield lets the delete run to completion, and suppressing the escaping cancellation means the original bootstrap error is still what propagates rather than being replaced by the second cancellation.
Regression test is event-gated rather than timing-based: client.delete blocks on an asyncio.Event, the task is cancelled while inside the delete, then the gate is released. Fails without the shield.
_delete_quietly catches Exception, not CancelledError, so a second Task.cancel() landing while it awaited client.delete() would escape and abandon the container mid-teardown - the exact leak this change exists to close, and unrecoverable because the session was never cached, leaving cleanup() a no-op. Wrap the teardown in asyncio.shield so the delete runs to completion, and suppress the escaping cancellation so the original bootstrap error is still what propagates. Adds a deterministic regression test that gates client.delete on an event, cancels the task while it is inside the delete, then releases the gate; it fails without the shield.
|
Good catch — that's a real hole in this very change, fixed in 0893e65.
The teardown now runs under Added a deterministic regression test rather than a timing-based one: (Edited: this comment originally cited a wrong commit hash. The correct one is 0893e65.) |
Fixes #830
What
create_or_reusenow deletes the sandbox container it just started if the remaining bootstrap steps fail, instead of letting the exception escape with the container still running and untracked.Why
The container is live from
backend(...)(:119), but it is only registered in_SESSION_CACHEat:145. Between those two points sitsession.resolve_exposed_port(:129) andbootstrap_caido(:134), either of which can raise.cleanupreaps only cached bundles:so after a bootstrap failure
cleanup(scan_id)is a no-op and nothing else holds a reference toclient/session— the container is orphaned for the lifetime of the daemon, keeping its exposed port and bind mounts.bootstrap_caidoexhausting its 10 guest-login retries on a loaded machine is a routine failure, so this leaks one container per failed scan start.How
Wrap the post-start steps in
try/except BaseException, delete the session best-effort, and re-raise the original exception unchanged.BaseExceptionrather thanExceptionso aKeyboardInterruptorasyncio.CancelledErrorduring bootstrap also reaps.The delete path is now shared with
cleanupas_delete_quietly, so both teardown sites also release the docker client identically.cleanup's behaviour is unchanged.Tests
New
tests/test_session_bootstrap_failure.py, driving the realcreate_or_reusewith a stub backend:test_container_is_deleted_when_port_resolution_failstest_container_is_deleted_when_caido_bootstrap_failstest_cleanup_is_a_noop_for_an_unknown_scan— pins the behaviour that made the leak invisibleBoth leak tests fail on
mainwithAssertionError: started container was never reaped, and pass with this change. Each also asserts the original exception still propagates and that nothing is left in_SESSION_CACHE.The 2 failures are pre-existing on
main(tests/test_runner_root_prompt.py— a staleSimpleNamespacesettings stub missingruntime) and unrelated; separate PR for those.ruff format --check,ruff checkandmypy strix/runtime/session_manager.pyare clean.Scope
One file plus a new test file. No signature or behaviour change on the success path.