openclaw: context lens crash recovery, retry dedupe, and ship-sync replay - #6155
openclaw: context lens crash recovery, retry dedupe, and ship-sync replay#6155wca4a wants to merge 4 commits into
Conversation
Retries reuse the original message id, and the SDK inbound dedupe tracker keys on MessageSid and remembers failed runs — so a manual retry of a failed run was silently swallowed. Suffix retry dispatches with the lens id to bust dedupe, keeping the canonical id in MessageSidFull. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Persist non-terminal run snapshots to an inflight checkpoint sidecar and, on the first store init per process, mark any runs the gateway abandoned (SIGTERM/SIGKILL) as aborted so they finalize and become retryable instead of hanging in a non-terminal state forever. Terminal events save the durable record before removing the checkpoint entry (and keep the entry when the save fails), and boot recovery removes checkpoint entries individually, so a crash or write failure between the two writes can no longer lose the run entirely. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A run-final poke that failed (ship unreachable, monitor disconnected, or the process died with the poke in flight) left the ship's copy of the run frozen at its last milestone forever. Track synced finals in an atomically-written ledger, replay unsynced terminal runs on boot (24h window, 50-run cap, idempotent ship-side), and arm a single exponential-backoff retry (30s base, 10min cap) on mid-session final failures so the replay re-runs once connectivity returns. The ledger write happens before the backoff reset so a failed write leaves the retry armed to repair it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Keeps .lens-panel-snapshot.txt (may contain private conversation data) out of commits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 636e755cc6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| api.logger.warn( | ||
| `[tlon] Context lens store write failed for ${lens.lensId}: ${String(error)}` | ||
| ); | ||
| return; |
There was a problem hiding this comment.
When a terminal event fails to append to the durable store (for example a transient ENOSPC/EACCES on the JSONL), this early return leaves the previous non-terminal checkpoint intact. On the next boot recoverInterruptedRuns() treats that stale checkpoint as an interrupted run and writes an aborted record, even though the run may already have completed and may even have been synced to %steward, which corrupts the lens status and can make a successful reply look retryable. Consider persisting the terminal snapshot to the checkpoint or marking/clearing the checkpoint so recovery can distinguish a store-write failure from a crashed in-flight run.
Useful? React with 👍 / 👎.
| !synced.has(lens.lensId) && | ||
| lensFinalizedAt(lens) > cutoff | ||
| ) | ||
| .slice(-REPLAY_MAX_RUNS); |
There was a problem hiding this comment.
When more than 50 terminal runs are unsynced (for example after a long ship outage or the first boot after adding the ledger), this selects only the newest 50 and no follow-up replay is scheduled after those are recorded. The remaining recent runs stay unsynced/frozen on the ship until another restart or unrelated failure happens to trigger replay. Treat the cap as a batch size and schedule another pass while misses remain, or explicitly mark skipped entries if they are intentionally never replayed.
Useful? React with 👍 / 👎.
| if (Date.now() - startedAt > REPLAY_GIVE_UP_MS) { | ||
| stop(); |
There was a problem hiding this comment.
If the gateway starts while the bot ship/API params are unavailable for longer than REPLAY_GIVE_UP_MS (for example the Urbit ship is still down during recovery), this branch stops the only boot replay poll and never hands off to the retry backoff. Any finals recovered from the previous crash then remain unsynced/frozen on the ship until another restart or an unrelated final failure happens to schedule a retry. Consider keeping this path on the same backoff loop until params become available, at least while the store still has unsynced finals.
Useful? React with 👍 / 👎.
Live production verification (2026-07-19)Deployed a branch build (tip 636e755) to a production openclaw gateway (bearclawd, bot ship Crash recovery: pass
New finding: Retry fails silently on desk version skewThe lens UI's Retry button did nothing. Isolation testing showed:
So the UI acks locally while the bot ship nacks silently — same failure class as the rube-27 version-skew note already in this PR, but worse UX because the button appears functional. Fix for this instance is a hosting desk update, not plugin code. It strengthens the case for the version-handshake follow-up (gateway/UI checking the bot steward's capabilities before enabling sync / offering retry) — filing that as a separate issue. |
Follow-up proposal: steward version handshake(Issues are disabled on this repo, so parking the write-up here.) Version/capability skew between owner and bot steward desks now has two observed silent-failure modes: missing agent entirely (rube-27 — pokes queue in gall forever) and partial skew (above —
|
1 participant
Summary
Hardens the context lens (per-run flight recorder) in
packages/openclawagainst gateway crashes and connectivity outages: interrupted runs are recovered asabortedon boot, run-finals that never reached the owner's ship are replayed with backoff, and lens-driven retries no longer get swallowed by the SDK's inbound dedupe tracker. Ported from the archivedopenclaw-tlonrepo and adapted to the%stewardagent plumbing.Changes
fix: retry dedupe — retried runs sendMessageSid: ${messageId}#retry:${lensId}(canonical id preserved inMessageSidFull) so the SDK inbound dedupe tracker, which remembers failed runs by MessageSid, doesn't silently drop the retry.feat: boot recovery — non-terminal run snapshots checkpoint to a.inflight.jsonlsidecar (atomic tmp+rename writes, mode 0600). On boot, abandoned entries are markedaborted(open tool runs closed as errors) and saved to the durable store; entries whose save fails stay checkpointed for the next boot. Terminal events save to the store before their checkpoint entry is removed. Bounded at 200 in-flight entries with a warning on eviction.feat: ship-sync replay + backoff — a.synced.jsonledger records which run-finals were acked by the ship. On boot, unsynced terminal runs from the last 24h (max 50) are re-poked (idempotent ship-side). Failed finals retry with exponential backoff (30s → 10min, single unref'd timer). A ledger write failure after a successful poke re-arms the retry instead of stranding the run as unsynced.chore: gitignore a local debug artifact (.lens-panel-snapshot.txt).How did I test?
pnpm test(894 tests, 51 files),pnpm tsc --noEmit,pnpm lint, Prettier — all clean.abortedwith"Gateway stopped before this run finished", the checkpoint drained, the replay fired 2s after boot, and the synced ledger recorded exactly the crashed run's lensId with no duplicate on a second pass.Note: lens mirroring requires a
%steward-bearing desk on the ship. On older desks gall queues the pokes until the agent starts (observed on the archived rube-27 test piers); the gateway's durable store retains all runs regardless.Risks and impact
Rollback plan
Revert the four commits. New sidecar files (
.inflight.jsonl,.synced.json) are ignored by older code and can be deleted freely.Screenshots / videos
N/A (gateway-side behavior; see crash-test log excerpts in the test plan).