Skip to content

fix(agent,control): make interrupted turns recoverable without repeating side effects / 中断回合可恢复且不重复副作用 - #9868

Open
SivanCola wants to merge 13 commits into
esengine:main-v2from
SivanCola:fix/interrupted-turn-recovery-v2
Open

fix(agent,control): make interrupted turns recoverable without repeating side effects / 中断回合可恢复且不重复副作用#9868
SivanCola wants to merge 13 commits into
esengine:main-v2from
SivanCola:fix/interrupted-turn-recovery-v2

Conversation

@SivanCola

@SivanCola SivanCola commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Turn reliability and interrupted-turn recovery: work that already happened is
never repeated, a result the host never saw is never invented, and every turn
ends in a durable terminal state.

Closes the core of #9825; also covers #9805, #9683, and the replay half of #9566.

Built on the existing turns.jsonl lifecycle ledger, session events,
InterruptedTurnRecovery, and the executeBatch executor. No parallel
persistence system is introduced.

Phase 1 — state correctness

Phase 2 — persistence and recovery

  • Reopening a ledger whose turn never terminated answers each unanswered call
    from its persisted start barrier, tags the synthesized results with a reopen
    source so a second reopen reads the same evidence, and escalates the recovered
    turn when an effect is unproven. The results still carry no arguments or output.
  • The controller threads that evidence into the dead turn's tail strip, naming
    the cause runtime_restart and marking a turn that produced nothing as silent.
  • The first side-effecting call of a turn stamps its checkpoint with the turn ID,
    call ID, argument digest, and transcript digest.
  • TurnDone carries Recovery on the wire, so Desktop can actually see the
    recovery_required state Phase 1 introduced.

Phase 3a — resolving an unproven effect

  • A user who inspected the workspace can attest that an unknown call already
    happened. Recorded with provenance under a distinct
    completed_by_user_confirmation state — never plain completed, because the
    host never saw the result — with no fabricated tool result. Refused for calls
    the host already proved and while a turn is running.
  • Bug found while testing: loading a session backfills a placeholder result
    for every unanswered call, and recovery read that placeholder as a genuine
    unknown outcome. That silently bypassed the ledger evidence on exactly the
    path that needs it — a real restart. Placeholders are now distinguishable, so
    proof wins.
  • Desktop history gains structured interruption-card data carrying each call's
    proven state plus its local argument receipt, which is what makes a long
    edit_file/write_file inspectable by hand. Those arguments are display state
    only and never enter the model-facing block.
  • A regression test locks in that ordinary tool interruption never forks a
    session; forking stays reserved for a genuine cross-process file conflict.

Not in this PR

The recovery card UI itself, the read-only "inspect workspace" action, and
"re-run tool" (new attempt ID, original call ID never reused). Re-run adds a new
execution path outside a model turn and is worth landing on its own.

The Wails App binding for the confirmation deliberately lands with that UI
rather than here: an unconsumed binding would grow bridge.ts past its repolint
ratchet and the raw bundle ceiling for no user-visible gain. The controller API
and the card projection are here and tested, so the UI change stays small.

Verification

go test ./internal/... and the desktop module, golangci-lint v2.12.2 (0
issues), go run ./tools/repolint clean, gofmt, go vet, tsc --noEmit.

Cache-impact: low - the model-facing recovery block gains failed_tools, user_confirmed_effects_do_not_repeat, cause and silent_interruption lines plus one sentence of guidance. It is appended to the mutable user-turn tail, never the cached system-prompt prefix; tool schemas, base prompt and memory block are untouched.
Cache-guard: go test ./internal/agent/ -run 'InterruptedRecoveryBlock|ConfirmedEffects|SamplingGate|ReplayRecovery' asserts the block's exact contents, that arguments never enter it, and that the provider request view is unchanged for healthy turns.
Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md document tool run states, batch rules, the recovery handoff, restart recovery, checkpoint identity, resolution, terminal status and the transcript gate; CHANGELOG has Unreleased/Fixed entries.

SivanCola and others added 12 commits September 7, 2026 01:58
Problem: interrupted tool results could be treated as not started even after execution began.\nRoot cause: cancellation and recovery metadata only had a coarse interrupted state.\nFix: add explicit pending, running, cancelled, failed, unknown states, preserve local argument receipts, and surface silent interruption facts without exposing raw arguments to providers.\nVerification: focused provider, event, agent, and control tests pass.
Problem: frontends could not distinguish cancelled, unknown, and silent interrupted work from ordinary tool results.\nRoot cause: execution evidence stayed in provider-local messages and was not projected into durable turn events.\nFix: add local tool run states, argument receipts, recovery metadata, and explicit recovery-required terminal status while preserving old wire fields.\nVerification: focused eventwire, agent, control, and provider tests pass.
Problem: interrupted side-effecting calls could be proposed again before their outcome was known.\nRoot cause: recovery tracked file writes only and generic tool calls had no durable replay guard.\nFix: retain all unknown calls in turn runtime, allow read-only inspection, and block matching non-read-only replays; dispatches persist pending state.\nVerification: go test ./internal/agent
Problem: desktop wire types could not represent durable recovery-required terminal turns.\nRoot cause: the frontend status union only covered interrupted and failed states.\nFix: add recovery_required and unknown statuses plus recovery identity fields for future action cards.\nVerification: TypeScript types updated without changing older wire payloads.
Problem: provider-facing history had no explicit final structural validator.\nRoot cause: normalization repaired common cases but callers could not assert the resulting transcript was safe.\nFix: add a non-mutating validator for JSON arguments and tool-call pairing.\nVerification: go test ./internal/provider
Problem: a dispatched call could be cancelled after invocation began without a durable start marker.\nRoot cause: ToolDispatch represented proposal and execution with one event.\nFix: add wire-stable ToolStarted event immediately before executeOne; cancellation can now distinguish started work.\nVerification: targeted event, wire, and agent tests pass.
Problem: checkpoints could not identify the interrupted tool attempt they protected.\nRoot cause: existing checkpoint metadata only addressed turn and workspace files.\nFix: add optional turn, attempt, tool call, argument digest, provider digest, and workspace reference fields while preserving legacy reads.\nVerification: go test ./internal/checkpoint
Complete the interrupted turn recovery design started in issue esengine#9825
with a concrete three-phase implementation plan:

Phase 1: Tool-level budget checkpoints - check budget before each tool
dispatch (not just at round boundaries) to prevent mid-tool interruptions

Phase 2: Interrupt signal handling - gracefully handle SIGINT/timeout/
context cancellation with deferred state persistence and recovery sidecars

Phase 3: Completed result injection - restore tool results that finished
before interruption into model context on recovery

The current branch has completed the infrastructure (tool state
classification, write intent verification, recovery identity tracking).
This plan maps the remaining work to close esengine#9825's core UX issues:
completed work being lost, missing retry arguments, and frequent
mid-execution interruptions in long sessions.

Related: esengine#9825
Documentation-impact: added - TURN_INTERRUPTION_IMPLEMENTATION.md describes
the three-phase delivery plan, root cause analysis, and success criteria

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Detailed task breakdown for tool-level budget checkpoints:
- Core implementation (nearLimit, checkpoints, partial batch)
- Testing strategy (unit + integration)
- Desktop integration for partial completion UX
- Acceptance criteria and manual validation scenarios

Tracks progress toward closing esengine#9825 core issue: mid-tool interruptions
causing completed work to be lost.

Documentation-impact: added - PHASE1_CHECKLIST.md tracks implementation tasks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comprehensive summary document covering:
- Current branch implementation status (completed infrastructure)
- Three-phase delivery plan (checkpoints, signals, result injection)
- Documentation structure and next action items
- Success criteria and key decision points

Serves as the top-level overview for the recovery refactoring effort,
linking together the design doc, implementation plan, and Phase 1 checklist.

Related: esengine#9825
Documentation-impact: added - 回合可靠性与中断恢复重构总结.md (Chinese summary)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate provider transcripts

Problem: a call interrupted after its start barrier was stored as not_started,
an unconfirmed write let later writers run, explicit failures read as unknown
in recovery, cancellation with unproven side effects ended as interrupted, and
no final transcript gate ran before provider requests.
Root cause: the stored-message and event classifiers diverged and checked the
executed flag before interruption evidence; the mutation barrier ignored
unknown outcomes; terminalTurnStatus ignored recovery facts; ValidateTranscript
was never wired and rejected shapes normalization allows.
Fix: one toolOutcomeState classifier, start-barrier-aware cancellation, unknown
engages the batch barrier, failed_tools bucket, ledger TurnID stamping,
recovery_required escalation, positional transcript gate in
buildSamplingRequest. Extract helpers so touched files stay within their
repolint budgets; add the missing tool_started frontend kind.
Verification: go test ./internal/... and the desktop module, golangci-lint
v2.12.2, repolint clean, tsc --noEmit, cache and docs guard scripts.

Related: esengine#9825 esengine#9805 esengine#9683 esengine#9566
Cache-impact: none - system-prompt prefix, tool schemas, and memory block untouched; changes affect tool-result classification, the recovery tail block, terminal turn status, and a pre-send gate on the already-normalized wire view
Cache-guard: go test ./internal/agent/ -run 'SamplingGate|ReplayRecovery|InterruptedBatch' and the prefix-stability effect tests under internal/boot
Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md; CHANGELOG Unreleased/Fixed entry added
…arrier

Problem: reopening a ledger whose turn never terminated answered every
unanswered call with one interrupted result, so a call that may already have
committed was indistinguishable from one that never ran; the recovered turn
always landed as interrupted; the restart handoff could not tell the two apart
either; and checkpoints named only the user turn.
Root cause: reopen predated the durable tool_started barrier and never read it,
the synthesized evidence was not durable enough to survive a second reopen, and
no component carried the barrier into the session-side recovery record.
Fix: classify orphaned calls by the persisted start barrier, tag the synthesized
results with a reopen source so re-reopen reads the same evidence, escalate the
recovered turn to recovery_required when an effect is unproven, thread that
evidence into the controller's tail strip (cause runtime_restart, silent
interruption), stamp the turn's first side-effecting call onto its checkpoint,
and carry Recovery on the TurnDone wire payload.
Verification: go test ./internal/... and the desktop module, golangci-lint
v2.12.2, repolint clean, tsc --noEmit, cache and docs guard scripts.

Related: esengine#9825 esengine#9805 esengine#9683
Cache-impact: none - system-prompt prefix, tool schemas, and memory block untouched; changes affect ledger reopen classification, the recovery tail block, checkpoint metadata, and the TurnDone wire payload
Cache-guard: go test ./internal/turnevent/ ./internal/checkpoint/ and go test ./internal/agent/ -run 'InterruptedRecoveryBlock'
Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md gain restart-recovery and checkpoint-identity sections; CHANGELOG Unreleased/Fixed entry added
@SivanCola
SivanCola requested a review from esengine as a code owner September 6, 2026 18:01
@github-actions github-actions Bot added desktop Wails desktop app (desktop/**) skills Skill system (internal/skill, internal/tool) agent Core agent loop (internal/agent, internal/control) provider Model providers & selection (internal/provider) v2 Go rewrite (1.x) — main-v2 branch, active development labels Sep 6, 2026
@SivanCola
SivanCola force-pushed the fix/interrupted-turn-recovery-v2 branch from e79d5e8 to 6073d7e Compare September 6, 2026 18:15
…terruption

Problem: an interrupted call whose outcome the host could not prove blocked the
next turn with no way out but guessing, and a session reload silently masked the
proof the ledger had recorded: normalization backfills a placeholder result for
every unanswered call, and recovery read that placeholder as a genuine unknown
outcome, so restart classification never consulted the start barrier.
Root cause: no host action could retire an unknown outcome, and the placeholder
was indistinguishable from a real result at the recovery boundary.
Fix: add a user attestation that records who resolved the call and when, keeps a
distinct completed_by_user_confirmation state so the host never claims it saw the
result, refuses calls already proved and turns still running, and synthesizes no
tool result. Teach recovery to ignore backfilled placeholders so ledger evidence
wins. Project the handoff into a structured Desktop card carrying each call's
proven state and its local argument receipt.
Verification: go test ./internal/... and the desktop module, golangci-lint
v2.12.2, repolint clean, gofmt, cache and docs guard scripts.

Related: esengine#9825 esengine#9805
Cache-impact: low - the model-facing recovery block gains a user_confirmed_effects_do_not_repeat line and one sentence of guidance; it is appended to the mutable user-turn tail, never the cached prefix
Cache-guard: go test ./internal/agent/ -run 'InterruptedRecoveryBlock|ConfirmedEffects' asserts the block's exact contents and that arguments never enter it
Documentation-impact: updated - docs/TURN_RELIABILITY.md and docs/TURN_RELIABILITY.zh-CN.md gain a resolution section; CHANGELOG Unreleased/Fixed entry added
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) desktop Wails desktop app (desktop/**) provider Model providers & selection (internal/provider) skills Skill system (internal/skill, internal/tool) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant