fix(history): early user messages vanish or jump mid-run when pagination and context compaction overlap - #4696
Open
rayhpeng wants to merge 9 commits into
Open
fix(history): early user messages vanish or jump mid-run when pagination and context compaction overlap#4696rayhpeng wants to merge 9 commits into
rayhpeng wants to merge 9 commits into
Conversation
…d page window Two independent paths made a user's own message disappear from a long thread (#4666, #4508, #4363). Both are reproduced by a real two-round run: once the thread passes the 50-row `/messages/page` window AND context compaction fires, the two sources of truth stop overlapping at the head. 1. Middleware-answered tool results never reached the event store. A middleware that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked write) returns a user-visible ToolMessage, but LangChain never emits `on_tool_end`, so RunJournal never persisted it — the user saw it during the run and it vanished on reload. RunJournal already reconciles final-output tool messages, but only for an `ask_clarification` allowlist. The allowlist is removed; scope stays bounded by the three conditions that actually matter (visible, this run's lead agent, not already persisted), so subagent results still stay in their own step feed. 2. mergeMessages discarded the checkpoint prefix before the first shared anchor. #4065 correctly established that a summarization-rescued early message must not be appended to the tail, and suppressed it instead. That suppression is what deletes the message when the first history page no longer reaches back to it. It is now woven in before the first shared anchor — the one position both the checkpoint and seq-sorted history agree on — so #4065's invariant (never the tail) still holds. A collapsed unloaded gap is recoverable by paging; a dropped message is not. Verified against real captured payloads from the reproducing run: the first user message returns to the transcript. Its exact position is still approximate — after compaction the live window carries too few anchors to place it precisely, which only seq-based ordering can close. Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Groundwork for placing checkpoint messages in the seq-ordered thread feed (#4666). A checkpoint carries no seq of its own and loses messages to summarization, so once the feed's 50-row page window no longer reaches back to a surviving old message, a client has nothing to place it by. The seq already exists in run_events keyed by the message id — this exposes it without paging the whole feed. `message_identity` is the backend half of the identity rule the frontend applies in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity. The two halves must stay in sync — a mismatch is silent, degrading placement rather than raising. `get_message_seqs` is implemented for all three stores. Misses are absent from the result rather than an error, so callers degrade to their own placement rule; the earliest seq wins when one identity resolves to several rows, so a re-persisted message keeps the position it first occupied. The DB store decodes rows in Python because `content` is a TEXT column holding a JSON string, not a JSON column — the identity fields cannot be projected in SQL. Nothing consumes this yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame that the thread feed already holds, so a client can place a message the checkpoint kept but its loaded history page window no longer reaches (#4666). Nothing is written back to the checkpoint: the seq is added when the frame is serialized and belongs to that frame only. Cost is bounded to frames introducing identities the run has not resolved yet. Messages this run produces are not in the feed while streaming, so they are looked up once, recorded as misses, and never retried — in a real run the only frame that pays for a query is the one where compaction brings older messages back into view. Measured on a reproducing two-round run: 1 lookup across 25 values frames. The stamper is built once per run rather than per `_stream_once`, or a goal continuation would discard the resolved seqs. Subgraph frames are not stamped: a subagent's snapshot is not part of this thread's feed ordering. A lookup failure logs and leaves the frame unstamped rather than failing it — placement is an enhancement and clients fall back to their own ordering rule. Frontend does not read the field yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`deerflow_seq` is display metadata the Gateway attaches when it serializes a values frame. A client replaying messages (regenerate / edit-and-rerun) would otherwise write it into the checkpoint, where it becomes wrong the moment the thread is forked — a branch re-seeds its feed and reassigns seq (#4380). Joins the existing server-owned key set, so it follows the same trusted-internal rule as the dynamic-context and view-image markers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arest anchor Completes #4666. Weaving a compaction-rescued message before the first shared anchor keeps it in the transcript, but not in the right place: after compaction the live window carries too few anchors, and the nearest one can sit deep inside the loaded page window — measured at row 25 of 50 on a reproducing run, which is why the first user turn rendered mid-transcript instead of at the head. Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages` copies each row's `seq` onto the message (same shape as the existing `run_id`), and the Gateway stamps it onto `values` frame messages it has already persisted. A live message whose seq is below the loaded window's lower bound is placed ahead of everything on screen rather than before the nearest anchor. A message with no seq — still streaming, so not in the feed yet — keeps the weaving path, since the tail is already its correct position. Verified against the captured payloads of the reproducing run: the first user message goes from absent, to #13 (behind the second question), to #0. Frontend: 988 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hor is shared Also #4666. Placing a compaction-rescued message by its feed seq was gated on reaching a shared anchor, because the split ran inside the anchor walk. When the loaded page and the live checkpoint share no identity at all, that walk never runs and the message fell through to `[...canonical, ...live]` — appended after the entire window, the one arrangement #4065 proved wrong, with its seq known the whole time. That is not a corner case. Open an old, already-summarized conversation and send a message: the page on screen is the newest rows from before that turn, while the checkpoint holds the rescued first user turn plus steps of the new run that are not in the feed yet. On a reproducing run the two sides shared zero anchors and the user's own first question rendered at row 50 of 50 — the reported "first message jumps to the bottom". Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`, and use it for the no-anchor branch as well, so a message routed ahead of the window is not re-appended at the tail by dedup. Measured on captured payloads of a reproducing run (real gateway, real compaction), first user message position: no shared anchor: row 50 -> row 0, seq order monotonic again shared anchors: row 0 -> row 0 (unchanged) paged to the top: row 0 -> row 0 (unchanged) Regression test verified red-green: reverting the fix fails it with the message rendered after the window. Frontend: 989 passed, eslint + tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…y on stream frames Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a client that joins a live run learns where a summarization-rescued turn belongs while a client that merely opens the conversation does not — and opening is the common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned the checkpoint with no seq at all, so the merge fell back to the nearest shared anchor, which after summarization sits deep inside the loaded page. Reproduced in a browser against a real gateway, on a thread that had already compacted: the user's first question rendered at row 320 of 389, behind the newest question instead of at the head. Both reads showed 0 of 13 messages carrying a seq. That is the reported symptom, still present after the streaming fix. Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper: everything a checkpoint still holds is already persisted, so one batched lookup resolves the whole list and there is nothing to retry later. Resolve the store through `_optional_run_event_store` rather than `get_run_event_store`, because seq is placement metadata — a deployment without a feed must still be able to read a thread. After the fix, on the same thread in the same browser: 13 of 13 messages carry a seq and the first question renders at the head, ahead of the newest one. Backend: ruff clean, 326 passed across the touched suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…messages to break an import cycle message_identity imported strip_injected_user_message_id_suffix from the dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime -> worker -> events -> middleware) that only stayed hidden while an earlier import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the strip helper in deerflow.utils.messages and re-export them from the middleware so existing importers keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
area:agents
Collaborator
Author
概要长任务运行中,用户较早的消息(通常是第一条)会消失或窜到步骤流中部,两种形态交替;任务结束后重载自愈,因此长期难定位。触发需两个条件同时成立:feed 超过一页(>50 行, 根因:消息流由两个源合并——feed( 修复在出口给每条已持久化消息贴上服务端权威位置:worker 序列化根 选型对比(A/B/C 为何不行):
Commit 明细
安全边界:只处理根帧;查不到 seq 不挂(流式中的消息本就该追加尾部);查询失败只告警、帧照发。identity 规则在 验证
Reviewer 注意
范围外: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try: | ||
| found = await store.get_message_seqs(thread_id, sorted(wanted)) | ||
| except Exception: | ||
| logger.warning("Failed to resolve message seqs for thread %s", thread_id, exc_info=True) |
Open
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4606, #4508, #4363
Summary
While a long task runs, the user's earlier message (usually the thread's first) disappears from the transcript or jumps into the middle of the step stream, alternating between the two; a full reload after the run heals it, which is why this stayed hard to pin down. It triggers only when both hold: the feed has grown past one history page (>50 rows, so
GET /messages/pageno longer contains the message) and context compaction has fired (so the checkpoint keeps only[hidden reminder, user message, recent tail]). Uploading a file is not part of the mechanism — it just makes the task heavy enough to hit both quickly.Root cause: the transcript is merged from two sources — the feed (
run_events, append-only, seq-ordered) and the checkpoint (valuesframes, compacted, no position info). Once the message falls out of the loaded feed window, its checkpoint copy has no anchor and no seq, so the frontend can only guess: guess fails → dropped (vanishes, the #4065 "suppress" path); heuristics re-correct → lands mid-stream (#4660's reconnect fix detects a real misplacement signal but, with no ground truth, moves steps to the wrong side). The backend was never at fault — across 900+ captured checkpoints the message order is correct in every one.Fix
Stamp the server-authoritative feed position onto every persisted message at the exit: when the worker serializes a root
valuesframe (and on/state,/historyreads), it resolves each message's feed seq by identity and attachesadditional_kwargs.deerflow_seq. The frontend then places below-window messages by seq instead of guessing.flowchart TB ES[("run_events — (seq, content.id) mapping already exists")] --> W W["worker: serialize root values frame"] --> CK{"ids not in run cache?"} CK -->|"no (incremental)"| SKIP["zero queries"] CK -->|"yes (compaction frame)"| Q["one batch query per run"] SKIP --> STAMP["stamp additional_kwargs.deerflow_seq"] Q --> STAMP STAMP --> FE["frontend: seq below window → head ✅"] style FE fill:#e1f5e1Alternatives considered:
REMOVE_ALL— no answer to copyadditional_kwargslikerun_id/hide_from_ui); first step toward replacing ~500 lines of anchor heuristics with a pure seq sortCommits
ab99337emergeMessagesstops dropping the pre-anchor prefix1aeee4b6RunEventStore.get_message_seqs()+ shared identity rule (all three stores)8eec0c2b_MessageSeqStamperstamps rootvaluesframes442fec89deerflow_seq(server-owned field)62bfbac7be72b9d8f2cbf8da/stateand/historyreads stamp seq too, not only stream frames2a9c4d16__user-suffix helpers toutils.messages, breaking the import cycle1aeee4b6exposedSafety boundaries: root frames only; no seq resolved → no stamp (streaming messages are correctly tail-appended anyway); query failure → warn and publish unstamped. The identity rule is deliberately mirrored between
runtime/events/message_identity.pyandhooks.ts::messageIdentity— a mismatch fails silently.Verification
#13(stop-the-bleed only) →#0with the full chain. Frame-by-frame SSE decoding confirms all 25valuesframes were correct — the bug is 100% in the frontend merge.has_more=true).tsc+ eslint clean); backend 10848 passed / 23 failed — all 23 reproduce identically on the base commit (env-dependent), unrelated to this branch.Reviewer notes
ab99337erewrites an explicit fix(context): resolve context compress bug #4065 test contract (the "unloaded gap before first anchor" message was asserted to be dropped). Dropping fixed "old message appended to tail"; this PR takes the third path — keep, insert before the anchor, never tail-append — preserving fix(context): resolve context compress bug #4065's invariant. Flagging for @AnoobFeng / @Vanzeren._MessageSeqStamperwiring is unit-tested at_publish_stream_itemlevel; the wiring itself relies on e2e coverage.Out of scope: thread-title pollution from the
UploadsMiddlewarerewrite ("<current_uploads>…"as title — reproduced in both captures) is an independent bug with its own fix branch, follow-up PR.