Skip to content

Commit 7bc9a86

Browse files
oldsjclaude
andauthored
Flag turns that go silent as stalled instead of spinning forever (#273)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 3aae27a commit 7bc9a86

34 files changed

Lines changed: 1837 additions & 526 deletions

‎api/openapi.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4615,6 +4615,11 @@ components:
46154615
type: boolean
46164616
title: Turn In Flight
46174617
default: false
4618+
stalled_turn:
4619+
description: The session's turn, when it is stalled (no task activity past its allowance); null otherwise.
4620+
anyOf:
4621+
- $ref: '#/components/schemas/TurnStall'
4622+
- type: 'null'
46184623
restartable:
46194624
type: boolean
46204625
title: Restartable
@@ -10054,6 +10059,44 @@ components:
1005410059
- looks
1005510060
- native_deleted
1005610061
additionalProperties: false
10062+
TurnStall:
10063+
description: >-
10064+
A delivered turn whose task has shown no activity for longer than its allowance. It may still
10065+
finish; the next task event clears it. Nothing is replayed: stop or restart the session to
10066+
recover.
10067+
type: object
10068+
properties:
10069+
message_id:
10070+
description: The delivery (the user message) whose turn is stalled.
10071+
type: string
10072+
session_id:
10073+
type: string
10074+
task_id:
10075+
anyOf:
10076+
- type: string
10077+
- type: 'null'
10078+
main_thread:
10079+
description: Whether the session is the owner's main thread.
10080+
type: boolean
10081+
last_activity_at:
10082+
description: The last task event Mainloop saw, or when the turn was delivered if none.
10083+
type: string
10084+
format: date-time
10085+
stalled_at:
10086+
type: string
10087+
format: date-time
10088+
tool_running:
10089+
description: The last event started a tool call with no result yet, which has the longer allowance.
10090+
type: boolean
10091+
required:
10092+
- message_id
10093+
- session_id
10094+
- task_id
10095+
- main_thread
10096+
- last_activity_at
10097+
- stalled_at
10098+
- tool_running
10099+
additionalProperties: false
1005710100
HealthView:
1005810101
type: object
1005910102
properties:
@@ -10062,6 +10105,12 @@ components:
1006210105
items:
1006310106
$ref: '#/components/schemas/HealthRow'
1006410107
maxItems: 3000
10108+
stalled_turns:
10109+
description: Delivered turns marked stalled (no task activity past their allowance), oldest stall first. Always sent; absent only from older servers.
10110+
type: array
10111+
items:
10112+
$ref: '#/components/schemas/TurnStall'
10113+
maxItems: 200
1006510114
stops:
1006610115
description: Requested stops that stayed unconfirmed past the resolver's bound, oldest first.
1006710116
type: array

‎docs/specs/chat.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,4 +178,5 @@ MSW fixtures typed from `api/openapi.yaml`.
178178
**Start a task in <first project>**, **What's running?**, **Pause a schedule**). Each fills the
179179
composer, unsent. They go once the thread has a message.
180180
- The native identity strip in the header details is supplied by the sessions slice.
181+
- A stalled turn (no task activity past its allowance, see [orchestration health](orchestration-health.md#stalled-turns)) shows the turn recovery panel instead of leaving only the spinner: "No activity for 25m. It may still finish. Stop or restart?" (with "while a tool runs" when the last event started a tool call), a **stop** button, and **restart session** when the thread is restartable. A stalled turn stays separate from a failed one: the next task event clears the panel, and nothing is resent. The owner also gets an attention item for it. Tested with MSW only.
181182
- A failed stop on the main thread shows the turn recovery panel below the conversation, with **restart session** behind an in-app confirmation that says the native context is lost (see [sessions](sessions.md#react-status)). The stop error now carries the server's reason. Tested with MSW only.

‎docs/specs/orchestration-health.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -473,3 +473,52 @@ the section rather than hidden.
473473
Not built: a push notification for an unresolved stop. Not proven: the signal against a live
474474
runtime that refuses to die, only against the fake in `server/internal/tasks/stop_force_test.go`
475475
and the database-backed `TestViewListsUnresolvedStops`.
476+
477+
## Stalled turns
478+
479+
Status: **implemented**, 2026-10-18 (issue #265). Tested against the fake kagent and a scratch
480+
PostgreSQL with a test clock; **not measured** against a live kagent.
481+
482+
On 2026-10-18 the main thread showed "Searching the web…" for over two hours, and four tasks were
483+
in the same state: each turn's task had gone silent inside kagent and nothing in Mainloop treated
484+
that as abnormal. A stalled turn is that signal for one delivered turn. It is separate from task
485+
liveness above (which blocks a `running` task) and from a failed delivery: a stalled turn may still
486+
finish.
487+
488+
- **What is tracked.** For every `delivered` delivery, the delivery loop records when it last saw a
489+
live event of the turn's task (`deliveries.activity_at`) and what that event left the turn doing
490+
(`activity_kind`): `event` (thinking or writing), `tool` (a tool call started with no result yet)
491+
or `parked` (waiting for the owner's input or auth). Live events are the send stream and every
492+
event of a subscription after its first. The first event of a subscription is the task as it
493+
stands, so a resubscribe to a silent task does not reset the clock; it counts only when its status
494+
was recorded after the activity already stored. Writes are throttled (at most one activity time
495+
per 30 seconds per turn; a change of kind or the end of a stall is written at once), fenced on the
496+
delivery's claim, and never touch `updated_at`, `state` or `outcome_version`.
497+
- **When a turn is stalled.** A `mark-stalled-turns` sweep (every 30 seconds) sets `stalled_at` on a
498+
delivered turn whose last activity (or, before any, the time it was marked delivered) is older
499+
than its allowance: `SESSION_STALL_SECONDS` (1500, 25 minutes), or `SESSION_STALL_TOOL_SECONDS`
500+
(3600, an hour; never less than the first) while a tool call is running, since a web search or a
501+
long build emits nothing until it ends. A parked turn is never stalled. The sweep publishes
502+
`session:message` for the turn's session so an open page re-reads it, and logs a warning.
503+
- **Clearing.** The next live event clears `stalled_at` in the same write that records it and
504+
publishes `session:message` again. A turn that ends is no longer counted, whatever its
505+
`stalled_at`.
506+
- **Where it shows.** `NativeSessionInfo.stalled_turn` (`GET /main-thread` and
507+
`GET /sessions/{id}/native`) carries the turn, its last activity, when it was marked and whether a
508+
tool was running; `null` otherwise. `HealthView.stalled_turns` lists every stalled turn of the
509+
owner, oldest stall first (at most 200). The `task`, `session` and `attempt` filters narrow it;
510+
the time window does not. The Health page shows a "Stalled turns" section above the rows when the
511+
list is not empty, and counts an entry it cannot read rather than hiding it.
512+
- **Attention.** The attention sweep opens one `owner_decision` item per stalled turn of the main
513+
thread or a standalone session (`source_kind = turn_stalled`, `reason_code = stalled`, linking to
514+
the main thread or the session) and resolves it once the turn moves again or ends. A task-owned
515+
session is left to the task liveness sweep, which already blocks the task as `stalled` and opens
516+
its own card.
517+
- **Never a replay.** The watch only surfaces the stall. It changes no delivery state, sends
518+
nothing and retries nothing. Recovery is the owner's stop, or the restart of
519+
[sessions](sessions.md#go-server-status-g6a-g6b), which deletes the native session and fails the turn.
520+
521+
Not proven: behavior against a live kagent. The fake-clock tests in
522+
`server/internal/sessions/stall_test.go` cover a turn stalled past its allowance and cleared by
523+
the next event, no stall while a tool runs within its longer allowance, and no stall for a parked
524+
turn.

‎docs/specs/sessions.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ History before `e776537` describes the retired Python backend. Where other text
3131
task attempt.
3232
- Stop turn cancels the A2A task with `CancelTask` (the kagent client and `kagenttest` both have it). It takes no lock (cancel is the one that takes the per-session send lock); the one-open-turn index and the fenced writes keep it consistent with a send. A delivery still `recorded` is cancelled without any kagent call. The task is cancelled outside any transaction; the delivery ends `cancelled`, the queue is held, and the partial reply and the stop note go into one message in one transaction (the note's id is derived from the delivery, so the delivery loop and the stop call settle the same turn once). Results: `stopped`, `finished` (the task had already ended, and its real outcome is recorded), `no_open_turn`. A task that is not visible yet or still running after the cancel is `409` and nothing changes; a kagent error is `502`. An `uncertain` delivery counts as open while it is inside its hold; one past the hold is skipped, since it no longer blocks sends and the delivery loop fails it, so it cannot mask a newer running turn. The stop is not repeated blindly.
3333
- Restart (`POST /sessions/{id}/restart`, issue #262) is the way out of a turn that stop cannot confirm, such as a runtime wedged inside kagent that never answers `CancelTask` (kagent will not suspend a Session with a non-terminal task either). The body carries a caller-chosen `request_id`, the idempotency key. Under the session's send lock Mainloop first writes the intent (`session_restarts`, state `requested`, naming the kagent Session to delete), then, with no transaction open, reads the Session and deletes it only if kagent still shows it, and reads it again when the delete reply still shows it running. kagent answering NotFound, `DELETED` or `DELETING` (a delete it has accepted, which cannot be undone and admits nothing) counts as gone. One transaction under the send lock then makes the binding forget that Session (fenced on its id, so a newer Session is never touched; a new create request id, and the standing context cleared), settles every `sending`, `delivered` or `uncertain` delivery sent to it as `failed` with "the native session was restarted before this turn finished; it was not sent again", admits the oldest `queued` message, and marks the intent `done` with the failed ids. Failed, not uncertain: Mainloop knows the Session is gone, so the message cannot still run, and nothing waits out `SESSION_UNCERTAIN_HOLD_SECONDS`. A `recorded` message has not been sent and is left for the new Session. The next message creates a new kagent Session that carries the recent conversation (the replacement path above); the native context is lost. The response is `restarted` (or `no_native_session` when the binding had none) with `failed_message_ids`. Repeating the same `request_id` returns the recorded outcome and deletes nothing; a different id while a restart is open continues the open one. A delete kagent does not confirm is `502` and keeps the intent `requested`; a repeat, or the `resume-session-restarts` sweep (every 30 seconds, for intents idle a minute), continues it the same way. Only the main thread and standalone native sessions can be restarted: a workspace session (its runtime belongs to the workspace service), a task-owned session (the task's stop path force-deletes a runtime that never answers, see [orchestration health](orchestration-health.md#unconfirmed-stops)), an archived, cancelled or failed session, and a main thread whose provider switch is pending are `409`. `NativeSessionInfo.restartable` says whether it applies. Nothing is restarted automatically and no prompt is replayed. Tested against the fake kagent only; not measured against a live kagent.
34+
- A delivered turn whose task shows no activity past its allowance is marked stalled; it is shown with stop and restart and raised as attention, and the next event clears it. It is an observation, not a delivery state: see [orchestration health](orchestration-health.md#stalled-turns).
3435
- A delivered or uncertain turn whose task kagent no longer has (`GetTask` not found, or no task shows the message) is checked against its kagent Session: when kagent shows the Session gone (NotFound, deleting or deleted), the turn fails at once with "the kagent session no longer exists, so this turn cannot finish; it was not sent again" instead of waiting out the uncertain hold, and the queue moves on. A task that is missing while its Session still exists keeps the uncertain path.
3536
- Cancel closes the session's open deliveries except task-owned `sending` and `uncertain` rows,
3637
whose claims and unresolved outcomes are retained for reconciliation without replay. In the same
@@ -213,7 +214,7 @@ Labels: **implemented** means built in `web/` and covered by component or route
213214

214215
- **Implemented:** session rows (on the project page); the `/agents` form, reachable by URL only (Claude Code or Codex, Ctrl/Cmd+Enter submits, blocked while offline); and the session page. The page offers **Stop turn** only while the native state reports a turn in flight, and never on archived, failed or cancelled sessions, **Cancel session** through an in-app confirmation (never `window.confirm`), and **Clear** for completed, failed and cancelled sessions, which returns to the main chat. It also shows the task link, workspace badges, summary or error, the identity strip, and the not-found, server error and unreachable states.
215216
- **Implemented:** after any session action, the page re-reads the session, its native state and attention before it shows the result. A `409` therefore shows the current state.
216-
- **Implemented:** the turn recovery panel in every chat (main thread and session chat). While a turn is in flight, the session is `restartable` and the last stop failed, it says the turn could not be stopped and offers **restart session**. The in-app confirmation (never `window.confirm`) says the native context is lost, the open turn ends as failed and is not sent again, and the next message starts a new native session with the recent conversation. One request id is kept per restart until it succeeds, so a retry after a lost reply or a `502` continues the same restart. Tested with MSW only.
217+
- **Implemented:** the turn recovery panel in every chat (main thread and session chat). While a turn is in flight it shows for a stalled turn (how long it has been quiet, with **stop**, and **restart session** when the session is `restartable`), and for a restartable session whose last stop failed (it says the turn could not be stopped and offers **restart session**). The in-app confirmation (never `window.confirm`) says the native context is lost, the open turn ends as failed and is not sent again, and the next message starts a new native session with the recent conversation. One request id is kept per restart until it succeeds, so a retry after a lost reply or a `502` continues the same restart. Tested with MSW only.
217218
- **Implemented:** while the session page first loads, placeholder rows with a row's height sit under the loading text. The text is what a screen reader announces; the placeholders are hidden from it.
218219
- **Wording:** the identity strip's live word is the agent's own state, shown verbatim. It shows `live` when the agent is ready, otherwise whatever state string the agent reports (for example `busy` or `suspended`, and any other word it sends), or `unknown` when there is no state yet. A failed or unconfirmed last delivery takes priority and shows as `last message failed` or `last message unconfirmed`, so it never reads `live`. A cancelled session shows `cancelled` the same way, even when the agent still reports ready. When the backend cannot be reached the word is `unreachable`, which the page adds itself; an agent never reports it.
219220
- **Implemented:** the session chat in the page's chat slot (`SessionChatSlot`). It shows the session's own conversation, with the same timeline, delivery notices, retry and composer as the main thread. It reads `GET /sessions/{id}/native` and `GET /sessions/{id}/conversation`, and sends to `POST /sessions/{id}/message`. Its unsent draft is kept for that session only. While a turn is in flight the header shows the turn's state and a **stop** button. A cancelled session's header reads `cancelled`, never the agent's own `ready`, and the **stop** button is hidden when the native state is unknown or malformed. Its pending requests show as cards at the end of the conversation (below); it does not show the topic index.

‎server/cmd/mainloop/main.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,7 @@ func serveAPI(ctx context.Context, cfg *config.Config, log *slog.Logger) error {
231231
runner.AddSweep(svc.CancelSweep())
232232
runner.AddSweep(svc.PromoteStrandedQueuedSweep())
233233
runner.AddSweep(svc.RestartSweep())
234+
runner.AddSweep(svc.StallSweep())
234235
runner.Add(hitlSvc.ResponseLoop())
235236
runner.AddSweep(hitlSvc.ObserverSweep())
236237
runner.AddSweep(attn.Sweep())

0 commit comments

Comments
 (0)