fix(agent,control): make interrupted turns recoverable without repeating side effects / 中断回合可恢复且不重复副作用 - #9868
Open
SivanCola wants to merge 13 commits into
Open
Conversation
Problem: interrupted tool results could be treated as not started even after execution began.\nRoot cause: cancellation and recovery metadata only had a coarse interrupted state.\nFix: add explicit pending, running, cancelled, failed, unknown states, preserve local argument receipts, and surface silent interruption facts without exposing raw arguments to providers.\nVerification: focused provider, event, agent, and control tests pass.
Problem: frontends could not distinguish cancelled, unknown, and silent interrupted work from ordinary tool results.\nRoot cause: execution evidence stayed in provider-local messages and was not projected into durable turn events.\nFix: add local tool run states, argument receipts, recovery metadata, and explicit recovery-required terminal status while preserving old wire fields.\nVerification: focused eventwire, agent, control, and provider tests pass.
Problem: interrupted side-effecting calls could be proposed again before their outcome was known.\nRoot cause: recovery tracked file writes only and generic tool calls had no durable replay guard.\nFix: retain all unknown calls in turn runtime, allow read-only inspection, and block matching non-read-only replays; dispatches persist pending state.\nVerification: go test ./internal/agent
Problem: desktop wire types could not represent durable recovery-required terminal turns.\nRoot cause: the frontend status union only covered interrupted and failed states.\nFix: add recovery_required and unknown statuses plus recovery identity fields for future action cards.\nVerification: TypeScript types updated without changing older wire payloads.
Problem: provider-facing history had no explicit final structural validator.\nRoot cause: normalization repaired common cases but callers could not assert the resulting transcript was safe.\nFix: add a non-mutating validator for JSON arguments and tool-call pairing.\nVerification: go test ./internal/provider
Problem: a dispatched call could be cancelled after invocation began without a durable start marker.\nRoot cause: ToolDispatch represented proposal and execution with one event.\nFix: add wire-stable ToolStarted event immediately before executeOne; cancellation can now distinguish started work.\nVerification: targeted event, wire, and agent tests pass.
Problem: checkpoints could not identify the interrupted tool attempt they protected.\nRoot cause: existing checkpoint metadata only addressed turn and workspace files.\nFix: add optional turn, attempt, tool call, argument digest, provider digest, and workspace reference fields while preserving legacy reads.\nVerification: go test ./internal/checkpoint
Complete the interrupted turn recovery design started in issue esengine#9825 with a concrete three-phase implementation plan: Phase 1: Tool-level budget checkpoints - check budget before each tool dispatch (not just at round boundaries) to prevent mid-tool interruptions Phase 2: Interrupt signal handling - gracefully handle SIGINT/timeout/ context cancellation with deferred state persistence and recovery sidecars Phase 3: Completed result injection - restore tool results that finished before interruption into model context on recovery The current branch has completed the infrastructure (tool state classification, write intent verification, recovery identity tracking). This plan maps the remaining work to close esengine#9825's core UX issues: completed work being lost, missing retry arguments, and frequent mid-execution interruptions in long sessions. Related: esengine#9825 Documentation-impact: added - TURN_INTERRUPTION_IMPLEMENTATION.md describes the three-phase delivery plan, root cause analysis, and success criteria Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Detailed task breakdown for tool-level budget checkpoints: - Core implementation (nearLimit, checkpoints, partial batch) - Testing strategy (unit + integration) - Desktop integration for partial completion UX - Acceptance criteria and manual validation scenarios Tracks progress toward closing esengine#9825 core issue: mid-tool interruptions causing completed work to be lost. Documentation-impact: added - PHASE1_CHECKLIST.md tracks implementation tasks Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comprehensive summary document covering: - Current branch implementation status (completed infrastructure) - Three-phase delivery plan (checkpoints, signals, result injection) - Documentation structure and next action items - Success criteria and key decision points Serves as the top-level overview for the recovery refactoring effort, linking together the design doc, implementation plan, and Phase 1 checklist. Related: esengine#9825 Documentation-impact: added - 回合可靠性与中断恢复重构总结.md (Chinese summary) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate provider transcripts Problem: a call interrupted after its start barrier was stored as not_started, an unconfirmed write let later writers run, explicit failures read as unknown in recovery, cancellation with unproven side effects ended as interrupted, and no final transcript gate ran before provider requests. Root cause: the stored-message and event classifiers diverged and checked the executed flag before interruption evidence; the mutation barrier ignored unknown outcomes; terminalTurnStatus ignored recovery facts; ValidateTranscript was never wired and rejected shapes normalization allows. Fix: one toolOutcomeState classifier, start-barrier-aware cancellation, unknown engages the batch barrier, failed_tools bucket, ledger TurnID stamping, recovery_required escalation, positional transcript gate in buildSamplingRequest. Extract helpers so touched files stay within their repolint budgets; add the missing tool_started frontend kind. Verification: go test ./internal/... and the desktop module, golangci-lint v2.12.2, repolint clean, tsc --noEmit, cache and docs guard scripts. Related: esengine#9825 esengine#9805 esengine#9683 esengine#9566 Cache-impact: none - system-prompt prefix, tool schemas, and memory block untouched; changes affect tool-result classification, the recovery tail block, terminal turn status, and a pre-send gate on the already-normalized wire view Cache-guard: go test ./internal/agent/ -run 'SamplingGate|ReplayRecovery|InterruptedBatch' and the prefix-stability effect tests under internal/boot Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md; CHANGELOG Unreleased/Fixed entry added
…arrier Problem: reopening a ledger whose turn never terminated answered every unanswered call with one interrupted result, so a call that may already have committed was indistinguishable from one that never ran; the recovered turn always landed as interrupted; the restart handoff could not tell the two apart either; and checkpoints named only the user turn. Root cause: reopen predated the durable tool_started barrier and never read it, the synthesized evidence was not durable enough to survive a second reopen, and no component carried the barrier into the session-side recovery record. Fix: classify orphaned calls by the persisted start barrier, tag the synthesized results with a reopen source so re-reopen reads the same evidence, escalate the recovered turn to recovery_required when an effect is unproven, thread that evidence into the controller's tail strip (cause runtime_restart, silent interruption), stamp the turn's first side-effecting call onto its checkpoint, and carry Recovery on the TurnDone wire payload. Verification: go test ./internal/... and the desktop module, golangci-lint v2.12.2, repolint clean, tsc --noEmit, cache and docs guard scripts. Related: esengine#9825 esengine#9805 esengine#9683 Cache-impact: none - system-prompt prefix, tool schemas, and memory block untouched; changes affect ledger reopen classification, the recovery tail block, checkpoint metadata, and the TurnDone wire payload Cache-guard: go test ./internal/turnevent/ ./internal/checkpoint/ and go test ./internal/agent/ -run 'InterruptedRecoveryBlock' Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md gain restart-recovery and checkpoint-identity sections; CHANGELOG Unreleased/Fixed entry added
SivanCola
force-pushed
the
fix/interrupted-turn-recovery-v2
branch
from
September 6, 2026 18:15
e79d5e8 to
6073d7e
Compare
…terruption Problem: an interrupted call whose outcome the host could not prove blocked the next turn with no way out but guessing, and a session reload silently masked the proof the ledger had recorded: normalization backfills a placeholder result for every unanswered call, and recovery read that placeholder as a genuine unknown outcome, so restart classification never consulted the start barrier. Root cause: no host action could retire an unknown outcome, and the placeholder was indistinguishable from a real result at the recovery boundary. Fix: add a user attestation that records who resolved the call and when, keeps a distinct completed_by_user_confirmation state so the host never claims it saw the result, refuses calls already proved and turns still running, and synthesizes no tool result. Teach recovery to ignore backfilled placeholders so ledger evidence wins. Project the handoff into a structured Desktop card carrying each call's proven state and its local argument receipt. Verification: go test ./internal/... and the desktop module, golangci-lint v2.12.2, repolint clean, gofmt, cache and docs guard scripts. Related: esengine#9825 esengine#9805 Cache-impact: low - the model-facing recovery block gains a user_confirmed_effects_do_not_repeat line and one sentence of guidance; it is appended to the mutable user-turn tail, never the cached prefix Cache-guard: go test ./internal/agent/ -run 'InterruptedRecoveryBlock|ConfirmedEffects' asserts the block's exact contents and that arguments never enter it Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md gain a resolution section; CHANGELOG Unreleased/Fixed entry added
SivanCola
force-pushed
the
fix/interrupted-turn-recovery-v2
branch
from
September 6, 2026 18:28
6073d7e to
eee2c5d
Compare
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turn reliability and interrupted-turn recovery: work that already happened is
never repeated, a result the host never saw is never invented, and every turn
ends in a durable terminal state.
Closes the core of #9825; also covers #9805, #9683, and the replay half of #9566.
Built on the existing
turns.jsonllifecycle ledger, session events,InterruptedTurnRecovery, and theexecuteBatchexecutor. No parallelpersistence system is introduced.
Phase 1 — state correctness
toolOutcomeStateclassifier behind both the ledger event and the storedtool message, so the two can never disagree. Interruption evidence now wins
over the
executedflag; an explicit refusal still reads as never-run.tool_startedbarrier: started calls become
unknown, the restcancelled.unknownoutcome engages the batch mutation barrier, so later writers areskipped while read-only diagnosis still runs.
failed_toolsbucket instead of being demotedinto the bucket that forbids retrying.
producing anything, ends as
recovery_requiredrather thaninterrupted.view the adapters send ([Bug]: Truncated assistant turn with unterminated tool_calls is replayed verbatim, causing 400 on strict OpenAI-compatible backends (MiMo) / 截断响应携带未闭合 tool_calls 原样回传,严格后端(MiMo)400 #9566).
Phase 2 — persistence and recovery
from its persisted start barrier, tags the synthesized results with a reopen
source so a second reopen reads the same evidence, and escalates the recovered
turn when an effect is unproven. The results still carry no arguments or output.
the cause
runtime_restartand marking a turn that produced nothing as silent.call ID, argument digest, and transcript digest.
TurnDonecarriesRecoveryon the wire, so Desktop can actually see therecovery_requiredstate Phase 1 introduced.Phase 3a — resolving an unproven effect
unknowncall alreadyhappened. Recorded with provenance under a distinct
completed_by_user_confirmationstate — never plaincompleted, because thehost never saw the result — with no fabricated tool result. Refused for calls
the host already proved and while a turn is running.
for every unanswered call, and recovery read that placeholder as a genuine
unknown outcome. That silently bypassed the ledger evidence on exactly the
path that needs it — a real restart. Placeholders are now distinguishable, so
proof wins.
proven state plus its local argument receipt, which is what makes a long
edit_file/write_fileinspectable by hand. Those arguments are display stateonly and never enter the model-facing block.
session; forking stays reserved for a genuine cross-process file conflict.
Not in this PR
The recovery card UI itself, the read-only "inspect workspace" action, and
"re-run tool" (new attempt ID, original call ID never reused). Re-run adds a new
execution path outside a model turn and is worth landing on its own.
The Wails
Appbinding for the confirmation deliberately lands with that UIrather than here: an unconsumed binding would grow
bridge.tspast its repolintratchet and the raw bundle ceiling for no user-visible gain. The controller API
and the card projection are here and tested, so the UI change stays small.
Verification
go test ./internal/...and the desktop module, golangci-lint v2.12.2 (0issues),
go run ./tools/repolintclean,gofmt,go vet,tsc --noEmit.Cache-impact: low - the model-facing recovery block gains failed_tools, user_confirmed_effects_do_not_repeat, cause and silent_interruption lines plus one sentence of guidance. It is appended to the mutable user-turn tail, never the cached system-prompt prefix; tool schemas, base prompt and memory block are untouched.
Cache-guard: go test ./internal/agent/ -run 'InterruptedRecoveryBlock|ConfirmedEffects|SamplingGate|ReplayRecovery' asserts the block's exact contents, that arguments never enter it, and that the provider request view is unchanged for healthy turns.
Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md document tool run states, batch rules, the recovery handoff, restart recovery, checkpoint identity, resolution, terminal status and the transcript gate; CHANGELOG has Unreleased/Fixed entries.