bug(runs): preserve work through Memory source-check outages #3456

Activity

1. justinhelmer

   added

   bugSomething isn't workingSomething isn't working

   area:runs

   on Oct 22, 2026
2. justinhelmer commented on Oct 22, 2026

   

   justinhelmer

   on Oct 22, 2026

   Last edited by justinhelmer

   ContributorAuthor

   More actions

   Current live evidence, 2026-10-22 11:55 a.m. PDT:

   • Review run 15fcd0fa-f4b9-4e26-a7ab-a0992cf9c02c for PR fix(core): hold reclaimed browser runs until source verifies #3459 recorded a 10-second /runs/deleted transport timeout at event 214. It was a live-view poll read with request ID 672b83f7-0f7a-408c-b089-8bb8bb0ca713. Events 215-216 recorded /runs/append write request 23328c0e-3505-427c-b579-15ad9283f6a8 and /runs/heartbeat write request 44022241-d167-4f25-8020-42a3fdd14e98 timing out.
   • Review run a34f8cde-ebc1-4b5d-9e6a-f83dd9278161 for PR fix(core): hold Review turns through Memory read failures #3460 recorded the same live-view poll failure and request ID at event 84. Its own /runs/append write request 16b4d051-725f-4a35-8777-fee14b6cf361 and /runs/heartbeat write request 1799babf-39e3-49ce-8352-31900234f6f4 timed out at events 85-86.
   • Both runs were still working at the subsequent runs_get read. Neither had a Review verdict. The shared poll request and close timing show a common Memory availability interval. They do not identify the underlying provider cause or prove the write outcomes.

   PR #3460 handles a typed temporary observational read at the model-source gate before provider forwarding. It does not change live-view polling, run append, or heartbeat writes. Keep the four write effects unknown until exact original receipts establish them. Do not replay them from a timeout alone. This issue remains open for those paths and the separate OpenCode gap.

   Update: Review run eea6bfb5-d2ea-4462-a92b-c9ae99cd6b37 for PR #3461 recorded the same /runs/deleted poll request 672b83f7-0f7a-408c-b089-8bb8bb0ca713 at event 111. Its separate /runs/heartbeat write request 00cfa3a2-ddfe-42eb-a088-bfdfbcf0f237 and /runs/append write request 7375997a-586f-486e-bd75-c8886628dd05 timed out at events 112-113. The run then continued to Review tools; no verdict was present at the read. This is a third correlated run in the same interval. The write outcomes remain unknown.

   Update: Review run 8b24ad26-39d8-49bb-b12b-a56f367d917d for PR #3450 recorded that same /runs/deleted poll request 672b83f7-0f7a-408c-b089-8bb8bb0ca713 at event 300. Its separate /runs/session/write request 87c3f87f-7953-4e75-89c7-5595aeaaea2a timed out at event 301. The run was still working at the subsequent runs_get read. This is a fourth run with the shared poll note, but the session-write result and the underlying cause remain unknown. PR #3460 does not cover this session-write path.

   Update: Review run 4bf7e983-f3cb-4bb5-b57b-1d2969f764c8 for PR #3445 ended failed at 11:55:27 a.m. PDT without a Review verdict. Its run.model_source_check span at event 104 reported UncertainStoreError for effectful /runs/context-checkpoint transport request 292c4bd2-c4de-43b7-af11-f60c5700749a. Event 105 said the model did not receive the request. Event 106 recorded the same /runs/deleted poll timeout request 672b83f7-0f7a-408c-b089-8bb8bb0ca713. This is a fifth run with the shared poll note. The checkpoint effect is unknown and must not be retried as a read. PR #3460 does not change this effectful path.

   Update: PR #3450 Review run 8b24ad26-39d8-49bb-b12b-a56f367d917d later lost confirmation for original run_check call call_aJ3QRynpaL60FQHHLGPOMHyZ|fc_09af781f44b78950016ada5d37d2e487d1a7f9f7db3152b8c9. Event 428 reports persistence_failed with no completion receipt; event 430 is a cut tool result; event 431 says the recovery checkpoint is unconfirmed and original work remains held. runs_check returned unavailable/source for both metadata and command. runs_get include shutdown showed finished=false, recoveryPending=true, runtimeShutdown=unknown, and no runtime contact. This is original check-result uncertainty, separate from this run's earlier /runs/session/write timeout. Do not run the check again or infer a Review verdict from the interrupted analysis.
3. justinhelmer commented on Oct 22, 2026

   

   justinhelmer

   on Oct 22, 2026

   Last edited by justinhelmer

   ContributorAuthor

   More actions

   New Memory interval, October 22 at 1:10 p.m. PDT: PR #3458 Review run d439c854-aacf-43e0-a622-475a6afbb82e recorded four distinct storage failures under owner generation 20261022T160828Z-04942b9d:

   • Event 508: /runs/heartbeat write timed out after 10 seconds, request 82f315a3-58b9-44eb-9365-afc520ac536c.
   • Event 509: observational /runs/deleted read timed out after 10 seconds, request 0edb1909-13e1-4174-8702-214571b8f928. This is a different request from the shared 11:55 a.m. poll recorded above.
   • Event 510: /runs/append write timed out after 10 seconds, request 39c006eb-543e-4db8-80b8-ce9620f960a9.
   • Event 511: /runs/state write returned HTTP 500, request 8a2170af-5b20-4816-a0b7-9c35eafd3acd. The server diagnostic names a state-guard read timeout; the response body was not read.

   The later source check at events 516-517 was allowed. It does not settle any earlier write. Event 521 says the recovery checkpoint could not be confirmed and original work remains held. At the later runs_get include shutdown read, the run was unfinished with recoveryPending=true, producerUnconfirmed=true, runtime shutdown unknown, and no runtime contact. There is no Review verdict or authority to replay, settle or stop an original from these observations. The three write effects remain unknown. This interval is separate from the earlier PR #3450 original run_check hold. PR #3460 only covers typed Pi model-source reads; it does not repair these storage effects.

   Correlated run in the same 1:10 p.m. interval: Slack Review run 7ef063fd-0d55-4a72-a7a6-3f42f5285d0f has a structural run_meta event 15 binding Nominal PR #5229 at head f11cba57a9c3a901f0864aa24e40a99b875ad161; its parsed PR artifact at event 70 agrees. Its saved top-level run record omits that PR/head, so the event binding matters. Events 229-233 recorded /runs/heartbeat write timeout 426f5a4b-de67-4734-8a3e-8e696cb23d8c, /runs/inbox/read read timeout 07b5b877-4049-4211-a688-95d82152f48a, /runs/append write timeout e4ec7f5b-d687-4e0d-aed3-3431ef41c4e1, /runs/deleted read timeout c99d618a-e9c1-4523-87cc-25c23568973a, and the shared live-view /runs/deleted poll timeout 0edb1909-13e1-4174-8702-214571b8f928 also seen in the PR #3458 run above. Event 335 deferred the checkpoint. The run is unfinished, recoveryPending and producerUnconfirmed, with runtime shutdown unknown. It has no verdict, and the write effects remain unknown.

   Keep a separate earlier Nominal PR #5229 Review run 34e80395-ebf2-4c4c-a026-b9fc8eba59ab distinct: it completed and has confirmed runtime shutdown at old head a7893b00866748214832b185a7a29fd55a20c018. Its answerOutcome.output is unknown, but GitHub Review 5619433952 proves a COMMENTED changes-requested F1 was posted at that old head at 12:03:35 p.m. PDT. Publication is proved for that review. The current f11cba57 head still has no completed current-head Review. Neither run authorizes replay of an uncertain storage effect.

   Update at 2:45 p.m. PDT: a third, separate Nominal PR #5229 Review run d764ba28-bc21-436f-80c9-c9aab4386ba7 completed at exact head f11cba57a9c3a901f0864aa24e40a99b875ad161, posted an approve verdict, and has confirmed runtime shutdown. GitHub Review 5621232389 posted LGTM at 2:42:18 p.m. PDT; auto-approval 5621234097 followed at 2:42:29 p.m. PDT. The PR merged at 2:45:11 p.m. PDT as 890cff767c0d5d962e73a41abe7e2e22b267a87e. This proves current-head Review and merge for that later run. The earlier original 7ef063fd-0d55-4a72-a7a6-3f42f5285d0f is still unfinished, recoveryPending and producerUnconfirmed with shutdown unknown at a fresh read. The later Review and merge do not settle its original write effects or physical ending.

   October 23, 7:45 a.m. PDT custody read: original Nominal PR #5229 run 7ef063fd-0d55-4a72-a7a6-3f42f5285d0f is still unfinished, recoveryPending and producerUnconfirmed. runs_get include shutdown now names live-ledger owner generation 20261023T085034Z-d3b795f9, but runtime shutdown remains unknown with no stop intent or runtime contact. runs_original on the prior generation 20261022T160828Z-04942b9d returned unavailable/fenced; the new-generation selector returned not_found, and runs_events had no event after seq335. The old fence is the expected generation check in originalEvidenceReader.ts; runs.original maps unavailable source evidence to not_found. Neither response proves that the old native process ended, that the original moved, or that its writes settled. The remaining read-only ending proof is an exact retained runtime-shutdown acknowledgment in runs_get include shutdown, or a request-bound original result if later exposed. No replay, Stop, reset or settlement follows from the generation change.
4. justinhelmer commented on Oct 22, 2026

   

   justinhelmer

   on Oct 22, 2026

   Last edited by justinhelmer

   ContributorAuthor

   More actions

   New Review custody and Memory interval, October 22 at 2:04 p.m. PDT: Nominal PR #5314 was at head 4a6f7d9c2412481d105115adaedf275f9ce0bc1a. Original Review run 257bd361-ed99-4c99-8383-c9a2cff76692 has a structural run_meta event 15 binding that PR and head. Events 62-68 recorded six 10-second transport timeouts:

   • /runs/append write f6a2f445-b5ca-45c5-8c71-26216e4a0d21
   • /runs/deleted read 5a2074c0-ea52-4c56-be7b-cfffe0b530cc
   • /runs/heartbeat write d2663d9f-9552-44a9-a024-434bb4addb62
   • /runs/append write c49e0c28-a3dc-422e-ae76-2d794cce5cdd
   • /runs/session/write write 1c1913cc-71d8-41c3-be9a-c12f515e4a61
   • /runs/state write 901c8d12-ed0f-4deb-9eb4-111a815b955b

   Event 82 deferred the checkpoint because original work could not be confirmed. At the runs_get include shutdown read, this run was unfinished, recoveryPending and producerUnconfirmed, with runtime shutdown unknown, no stop intent, no runtime contact, and an eligible target. The five write results remain unknown.

   A second Review run, 97434215-d535-4076-82b0-4d4a26a18381, was active in a separate Slack thread. Its own run_meta event 15 binds the same PR and head. That run does not prove the original ended or transfer its pending work. GitHub had no current-head Review at the read; its visible LGTM was on older head 05dc146601a610e29851e72a4191390333606772. Keep both run identities separate. No replay, Stop, reset or settlement follows from this overlap.

   Update at 2:38 p.m. PDT: the second run 97434215-d535-4076-82b0-4d4a26a18381 finished, recorded an approve verdict, posted GitHub Review 5621187384 with LGTM at exact head 4a6f7d9c2412481d105115adaedf275f9ce0bc1a, and has confirmed runtime shutdown. GitHub auto-approval 5621189238 followed at that same head; required CI was terminal with no failure or pending check at the read. These facts establish a current-head verdict for that second run. The original 257bd361-ed99-4c99-8383-c9a2cff76692 remains unfinished, recoveryPending and producerUnconfirmed with runtime shutdown unknown. The later verdict does not settle its five uncertain write results or prove its original process ended.
5. justinhelmer

   added

   P1

   on Oct 23, 2026
6. justinhelmer commented on Oct 23, 2026

   

   justinhelmer

   on Oct 23, 2026

   ContributorAuthor

   More actions

   October 23 production example for the Pi source-read slice: Nominal PR #5313 Review run 6e5023f8-0042-4d30-adfe-219d205b46bc failed at exact head 2d0a3ec9feb1e95be8212af8defb98a9d2652eae. The run is sealed with confirmed runtime shutdown and no GitHub verdict at that head.

   The run.model_source_check span at event 218 ended with sourceCheckStage=github-execution, sourceErrorClass=TransientStoreError, observational storageOperation=/runs/session/read, storagePhase=transport, request 0559476a-2ca0-4c94-be83-3c4f4edc8bd7, and sourceReadStop=deadline. Event 219 says the model did not receive the request. Event 220 records source_unavailable, no provider stop and zero content parts; it does not establish a provider failure. The Memory route calls stub.read for /runs/session/read. This event shows no uncertain write.

   PR #3460 covers this typed temporary read class only while the same run owner is active and access is not revoked. Its proxy would send a signed local retry result before provider forwarding and Pi would recheck within the saved one-minute window and original lease. An outage past that bound still fails closed. Production for this original served the older bot build 48efe094290e865f7efd9f186fcb332b991a09db; the PR is not deployed and cannot recover this finished run or prove a Review verdict. This is a matching failure shape, not live acceptance.
7. justinhelmer commented on Oct 23, 2026

   

   justinhelmer

   on Oct 23, 2026

   Last edited by justinhelmer

   ContributorAuthor

   More actions

   October 23 at 8:10 a.m. PDT: Nominal PR #5230 Review run be767780-4ce7-4da3-9963-e6baf535deb9 failed at exact head 0b837d1e2edb8bbb4ef6f24416aac416544f831e before a verdict. It is finished and has confirmed runtime shutdown. The model received no request, and no provider failure was established.

   The run.model_source_check span at event 43 ended with sourceCheckStage=history, sourceErrorClass=TransientStoreError, observational /runs/live transport request ed9d174f-8ce4-4fba-ae94-a46d1cec9c31, one source-read attempt and a deadline stop. Event 44 says the model did not receive the request. Event 45 reports local source_unavailable, provider stop missing and zero content parts. Earlier events 40-41 recorded /runs/deleted read timeouts, requests 9d313424-3354-4316-9a21-ba8836427c32 and 09491b50-a60c-42c4-b974-aa338362b572. Event 42 recorded a separate /runs/heartbeat write timeout, request 8a9ddaa2-369c-4813-8b14-254c3e1728ff; its effect remains unknown.

   PR #3460 covers the typed temporary model-source-read class only while the same owner and audience remain valid. It would hold the Pi turn and recheck source within the original lease and one-minute window; an outage past that bound still fails closed. It does not settle the heartbeat write or recover this completed run. Production still served older bot commit 48efe094290e865f7efd9f186fcb332b991a09db at the read. This is a matching failure shape, not deployment or live acceptance.

   Second matching read failure, October 23 at 9:16 a.m. PDT: Switchboard PR #3477 Review run a6591d01-4df2-4d11-8fc9-016a747002c1 failed at exact head 0409eaed601cc9a0b2daf2dd4c151b27a2e29b11. Events 314-315 show run.model_source_check ending at stage history with TransientStoreError on observational /runs/live transport request 09b38b05-379a-4fbb-b7e7-9b4ae4c308d7, one source-read attempt and deadline stop. Event 316 says the model did not receive this request; event 317 says no provider failure was established and reports local source_unavailable. The run ended failed with confirmed runtime shutdown. GitHub has no Switchboard Review verdict at that head; an internal approve draft is not publication proof. This is another occurrence of the #3460-eligible typed read class, subject to the same active-owner and audience checks. It does not validate #3460 in production or recover the failed Review.
8. justinhelmer commented on Oct 23, 2026

   

   justinhelmer

   on Oct 23, 2026

   ContributorAuthor

   More actions

   Bounded production impact inventory, October 23 at 3:12 p.m. PDT: four more Nominal Review originals are finished:false, recoveryPending:true, producerUnconfirmed:true and paused for storage recovery. Each has run-active/runtimeShutdown=unknown with no ending basis under owner generation 20261023T085034Z-d3b795f9. Structural run_meta binds each PR and head below. The earlier PR #5314 original is already recorded separately; do not count it again here.

   • PR #5328, run 764051be-584c-4dbd-ba31-9a41237a4bb7, head 8f8dbd52c38fc915b1e45c2eea37ee8145ea629e: /runs/deleted read request 9b10b2b9-ab9f-47d0-a239-da9bf067c1cf, /runs/append write 2d3fcf56-2704-40ba-a17d-c09e8501beac, and /runs/session/write write 4e1cb887-287d-4b2a-b0ff-0647292a8d96 timed out. Event 132 says recorded run_check call call_iEShZnLF6kV6OB1eCSfF07LI|fc_0488dbdf8e9f708c016adad7b68e4487d19dbc523ba27c1a89 was unavailable after persistence failure; its cut result and a cut review_read result do not prove completion. Event 136 deferred the checkpoint.
   • PR #5341, run 88587642-bf52-41c7-97e8-f0e8449ca6d4, head 8e0a1e1b9d6f02967749f253482d8b434c3d166a: /runs/heartbeat write requests 8435a869-98ee-41a9-b248-e9f15a058ad2 and e7a5dccd-84ff-4d71-b5f1-9e1ea772da35, /runs/append write 9fbba0b6-e65f-49a2-aba3-806440246bdb, and /runs/session/write write 3d7ce2f8-9ec8-4f55-8561-ab94655abfd2 timed out.
   • PR #5346, run f9fa2114-deb4-4bab-bdda-3b2f5d4c005f, head fbcf3e3ec672da88a5a5cdde88bc6103d1b30b0f: /runs/session/write write c81b95e1-92c9-42ac-95dd-31c3a7330853, /runs/state write bfeb6d30-fcb0-470f-8c3a-c335323d75e9, and /runs/deleted read b7082214-220a-4b5a-9cd0-167c81899004 timed out. The submit_verdict tool result at event 242 was cut; event 243 deferred the checkpoint. Its original publication result needs exact GitHub or tool receipt before classification.
   • PR #5379, run 9009c38f-383a-45ae-860b-77f955cdf0f3, head 8ab61119dc88d45a6fa5fe6ef7f06d26b60aecfb: /runs/session/write write 77e42df0-f6e6-4299-8ddf-a354f7d6beed, /runs/deleted read eef4f758-c9ea-4f81-bb96-2dbf696e53e0, and /runs/heartbeat write 9e31d0d3-48a2-459a-9b91-d03fb3358dcf timed out; event 279 deferred the checkpoint.

   All four PRs later merged according to GitHub, but a PR merge does not settle its unfinished original Review or uncertain write. This is an impact inventory, not a claim of one provider cause or four distinct incidents. Keep each original under its existing custody. Do not replay, Stop, reset or settle from the list state or bound expiry.
9. justinhelmer commented on Oct 24, 2026

   

   justinhelmer

   on Oct 24, 2026

   Last edited by justinhelmer

   ContributorAuthor

   More actions

   October 24 custody checkpoint for five unresolved Nominal Review originals. The existing #3456 Memory source lane owns this evidence record only. Each original keeps its own requester, Slack thread and saved owner generation in Switchboard; no new recovery operator or writer is assigned here. The GitHub authors below identify the PR source writers, not authority to control an uncertain run. No unattended watcher was created.

   PR source writer	Original Review run	Next evidence that can change custody
   #5379, boristane	9009c38f-383a-45ae-860b-77f955cdf0f3	Exact /runs/session/write and /runs/heartbeat write receipts, plus a retained original runtime-ending acknowledgment.
   #5346, matanyall	f9fa2114-deb4-4bab-bdda-3b2f5d4c005f	Exact /runs/session/write and /runs/state write receipts; request-bound submit_verdict publication evidence; original runtime-ending acknowledgment. A later PR approval alone cannot classify this original tool call.
   #5341, boristane	88587642-bf52-41c7-97e8-f0e8449ca6d4	Exact heartbeat, append and session-write receipts, plus original runtime-ending acknowledgment.
   #5328, boristane	764051be-584c-4dbd-ba31-9a41237a4bb7	Exact append and session-write receipts, original run_check/review_read completion evidence, and original runtime-ending acknowledgment.
   #5314, nwm-54	257bd361-ed99-4c99-8383-c9a2cff76692	Exact two append, heartbeat, session-write and state-write receipts, plus original runtime-ending acknowledgment. The separate current-head LGTM run does not supply these.

   At the October 24 read, each runs_get include shutdown still said finished:false, recoveryPending:true, producerUnconfirmed:true, run-active/runtimeShutdown=unknown, with no stop intent under generation 20261023T085034Z-d3b795f9. All five PRs later merged, but merge is not an original-run ending receipt. The detailed request IDs are in the earlier four-run inventory and #5314 record. Until one of the row-specific receipts appears, keep the original outcome unknown. Do not replay, Stop, reset, settle or retire based on this checkpoint.

   October 24 read discrepancy for original PR #5346 run f9fa2114-deb4-4bab-bdda-3b2f5d4c005f: a 1:12 p.m. PDT portfolio plane.show listed it unfinished/recoveryPending/producerUnconfirmed and paused for storage recovery, while a direct runs_get returned not_found. At 1:14 p.m. PDT, my independent runs_get include shutdown, runs.list(active) and plane.show all found that same ID. The direct record still said finished:false, recoveryPending:true, producerUnconfirmed:true, owner generation 20261023T085034Z-d3b795f9, run-active/runtimeShutdown=unknown, no stop intent, no runtime contact and target eligible. The earlier not_found is an inconsistent read, not proof of deletion, ending or settled effects. The next material receipt for this row remains request-bound results for /runs/session/write, /runs/state and submit_verdict, plus a retained original runtime-shutdown acknowledgment. Do not replay or settle from either projection alone.
10. justinhelmer commented on Oct 24, 2026

    

    justinhelmer

    on Oct 24, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 24, 3:50 p.m. PDT: I ran local pinned-binary probes for the OpenCode source-hold gap at Switchboard head c07f95da3946e02b862b74faa2b6f5495acfd334. The isolated checkout used @opencode/cli 2.0.27, the real OpenCode server and model proxy, and a local fake upstream. The source guard returned the typed temporary read result retryable. Both named Vitest probes passed. This is local proof only; it is not production acceptance or a source fix.

    • The proxy returned HTTP 403 with signed source_check_retryable. The pinned binary emitted session.execution.started, then session.execution.failed for ses_run-source-refusal-probe. The failed event carried error.type=provider.auth, error.status=403, and error.response.body with the signed source result. proxySourceRetryIsAuthenticated accepted that exact body. Neither event exposed an execution ID.
    • The guard ran once. consumeTurn was not called. The fake upstream received zero requests, and the feed had zero session.tool.called events and zero session.retry.scheduled events.
    • The current bridge reduces this event to the message and ends the run with the model call failed: The run's source check is temporarily unavailable; the model did not receive this request. It does not hold or reissue the OpenCode turn.
    • A second probe read the failed input ID from the same live native session. Reposting /prompt with that ID returned the original input and made no provider call; it added an idle marker. POST /interrupt?resume=true answered interrupted:false and made no provider call. The pinned protocol lists no session retry endpoint. A fresh, unique continuation ID with metadata naming the original input made exactly one provider call as turn 1. The store kept the original user row once and added one distinct continuation user row; the upstream request included both. This is an explicit new input, not an exact replay of the failed input. The fixture ended its own process after the read.

    Explicit new-input contract: A future hold may send one uniquely identified continuation only after the signed terminal source refusal, exact same-session and original-input readback, a fresh observational source success, and unchanged run owner and original lease. Its new input ID must bind the original input. A transport-uncertain continuation POST needs exact native inbox or message readback; neither an absent response nor a repeated POST grants another provider call. Revocation, deadline expiry and hard cancellation must prevent admission. The bridge must keep the live native process, source-hold state and input IDs across a bot resume; a native process exit stays outside this contract.

    Owner and next dependency: This existing Memory source chat retains the #3456 OpenCode source gap. The current bridge discards error.response.body when it reduces the failure, then ends the process. No source hold, saved input ID, continuation acknowledgment or same-run restart proof exists. The signed failed event names the session but no execution ID. Those are blockers to a source patch that can be reviewed as safe. The next source work is a persisted same-process hold and one continuation path with focused end-to-end tests for temporary loss/recovery, revocation, the original lease deadline, hard cancellation, and unknown continuation admission. Existing proxy source-guard tests passed 15 focused cases, and existing OpenCode initial-deadline/hard-stop tests passed 2 focused cases at this checkout; these component checks do not prove the new hold. No OpenCode source patch or paid Review was opened. Process replacement and persistent native storage remain the separate, held #3472 task; production OpenCode source intake stays disabled.

    Bot-resume blocker: providerFailureAuth.ts creates a new signing key in each bot process. OpenCodeHarnessFacts has no source-hold or input-ID field. The current reattach path reads the native store, then posts continuePrompt before it can use a typed source-hold fact; it treats the previous generation's feed as catch-up history. A new bot cannot authenticate the prior generation's marker from that feed. A required facts acknowledgment uses ledgerRun.setStateAndFlush against Memory, the source that is unavailable in this failure case. If the bot dies before that acknowledgment, neither its local hold nor a verifiable signed result survives. The native process can remain alive, but an automatic continuation after reattach would have no trusted source-failure receipt. The #3456 Memory/model-proxy source owner must first define an acknowledged cross-generation source receipt or a fail-closed reattach state that retains the native process without continuation. A later source patch must prove the bounded continuation instruction, answer and tool-effect count through public dispatch, not just the proxy request count. This gap does not authorize a native process replacement or a retry of an uncertain original.

    4:00 p.m. PDT implementation experiment: In an isolated, unpushed checkout, the signed-failure classifier and bounded same-process continuation produced the expected real-binary answer with one upstream request and no tool call. TypeScript passed. This proves that live-path mechanism only. A broad reattach rule that refused every native idle row with outcome:failed caused 13 failures in the focused existing reattach tests: their recorded server intentionally has that marker after a bot loss and many cases must continue without a second server. The pinned binary stores the source marker in the failed assistant row once that row exists, but a bot can die before it is written or before Memory acknowledges a source-hold fact. A marker-only refusal misses that window; a broad refusal changes the settled reattach contract. I reverted the experimental source edits and left the branch unpushed. The next #3456 dependency is a trusted cross-generation pre-provider/source-outcome receipt, or an explicit decision to fail closed for ambiguous bot-loss resumes with the affected reattach behavior reviewed and changed. This source chat keeps ownership. #3472 remains on its memory-only hold.

    4:23 p.m. PDT current local checkpoint: I chose a narrower fail-closed rule for source-guarded OpenCode runs. A bot reattach retains their original native process and sends no continue input because this build has no verified cross-generation model-effect receipt. Ordinary OpenCode reattach remains on its existing path; all 44 focused reattach tests passed. The same-process bridge now verifies the signed refusal, reads the exact failed input and idle marker, holds pending follow-ups, rechecks source inside the original lease, requires an acknowledged hold fact, and admits one uniquely identified continuation. Exact native readback resolves a lost POST response without a second send; an absent or unacknowledged result retains the original process. Ten pinned-binary tests passed for recovery, one tool effect after and before the failure, revocation, deadline, hard cancellation, uncertain POST and failed hold acknowledgment. A public runLoop test passed with one continuation, one tool result and a completed answer. Focused bridge and Pi source tests, TypeScript, consistency and required changed-path spec coverage also passed. The work is two clean local commits on codex/opencode-source-hold, head 99a7bc300, based on #3460 head c07f95da; it is not pushed, reviewed, merged, deployed or live accepted. #3460 remains unchanged. The next delivery step stays behind #3460's current-head Review and production live gate; #3472 storage and OpenCode production source intake remain held. Issue #3456 stays open for uncertain Memory effects and the unproved deployed path.

    4:31 p.m. PDT remote draft checkpoint: PR #3481 is a draft child of PR #3460 in managed stack #3482. Its head is 6fe537805c9d8743c152cb606f6cf384317b1087; the parent remains at c07f95da3946e02b862b74faa2b6f5495acfd334. The first child CI attempt found a foreign-run custody error leak in contract.test.ts: my broad HarnessEndingUnconfirmedError handler returned a foreign run's error unchanged. I narrowed the handler to the current run. At the new head, 10 pinned-binary cases, 44 reattach cases, 14 bridge cases, the public run case and both foreign-run custody cases passed locally. TypeScript, scoped lint and format, consistency and required spec coverage passed. The new-head CI has no failure at this read but still has pending checks. The PR has no Review, merge, deployment or live acceptance. It stays draft behind the production Review gate; #3472 and OpenCode production intake remain held. The 4:23 p.m. local-only state above is historical.

    4:34 p.m. PDT CI result: Draft PR #3481 at exact head 6fe537805c9d8743c152cb606f6cf384317b1087 has 28 successful checks, four planned skips, and no failed or pending check. Its title and body now describe this head. It has no current-head Review. Parent PR #3460 remains open at c07f95da3946e02b862b74faa2b6f5495acfd334, without a current-head Review. Production status_show still reports v1.308.2 / 48efe094290e865f7efd9f186fcb332b991a09db; neither local proof nor green child CI establishes deployment or live acceptance. The production Review gate and #3472 memory-only hold remain in force. Issue #3456 stays open for uncertain Memory effects and unproved live source recovery.
11. justinhelmer commented on Oct 25, 2026

    

    justinhelmer

    on Oct 25, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 25, 10:03 a.m. PDT custody read: direct runs_get(include: shutdown) found all five Nominal Review originals in the existing custody record: 9009c38f-383a-45ae-860b-77f955cdf0f3, f9fa2114-deb4-4bab-bdda-3b2f5d4c005f, 88587642-bf52-41c7-97e8-f0e8449ca6d4, 764051be-584c-4dbd-ba31-9a41237a4bb7, and 257bd361-ed99-4c99-8383-c9a2cff76692.

    Each direct record still says finished:false, recoveryPending:true, producerUnconfirmed:true, run-active/runtimeShutdown=unknown, with no ending basis. The retained owner generation is 20261023T085034Z-d3b795f9. The live ledger says stopIntent:null, runtimeContacted:false, and confirmation:not-checked. Its working state and expired bound do not prove that the runtime is alive or ended.

    #3456 continues to own the evidence record only. Each original keeps its existing requester, thread, and run owner. The next event is an exact request-bound write or publication result and original runtime-ending acknowledgment, as listed for each run in the linked custody record. The portfolio owner made a separate 10:01 a.m. PDT spot-check; this lane made the 10:03 a.m. direct read. The existing 30-minute portfolio watch now covers change detection for these five originals and will relay new request-bound ending evidence to this #3456 custody record. It adds no run control, new schedule, or writer. This read does not authorize replay, Stop, reset, settlement, or retirement.
12. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    ContributorAuthor

    More actions

    October 26, 9:57 a.m. PDT: I am driving the end-to-end #3456 resolution from the existing Recover Switchboard live Review acceptance chat. Ownership is explicit:

    1. Memory source owner: the existing Repair Memory run-ledger availability chat keeps the sole writer for #3460 at c07f95da3946e02b862b74faa2b6f5495acfd334 and draft #3481 at 6fe537805c9d8743c152cb606f6cf384317b1087. fix(core): hold Review turns through Memory read failures #3460 has green checks but lacks a current-head Review. The owner is preparing the next source and review gate on the original branch.
    2. Release and live acceptance owner: the existing Coordinate Switchboard v1.309.0 release chat owns the production smoke route. Controller 40361683839 deployed e1fcac8a9cb6a52ccaaf5fdc8ab84f26e487e2b2 successfully, but smoke=false; the ordinary request and acceptance artifact were skipped. A separate Answer run and Workspace run are finished, sealed, and have confirmed shutdown. The Workspace run failed to clone the private release-smoke fixture through Git Door with a 404. Neither run is a live Review receipt. The owner is diagnosing that fixture access and naming one supported, bounded acceptance route before any new Review is admitted.
    3. Native custody owner: the existing Resolve native operation release holds chat keeps the three retained records with pendingAttempts=1. Their original effects and process endings remain unknown. This gate is separate from the successful deployment and from fix(core): hold Review turns through Memory read failures #3460.
    4. End-to-end gate: after a valid live Review acceptance receipt, obtain one current-head Review and address findings for fix(core): hold Review turns through Memory read failures #3460, then confirm required checks and a person-authorized merge. Prove deployment of the fix by exact build and repeat the affected live Review path. Qualify fix(core): keep OpenCode work during source outages #3481's OpenCode path separately. Preserve the original uncertain requests and runs; no replay, Stop, reset, new Review, merge, or deployment follows from this status note.

    Next update: the release owner's fixture/acceptance route and the Memory owner's exact #3460 next action. A healthy /healthz and the v1.309.0 upload do not close this issue.
13. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    ContributorAuthor

    More actions

    October 26, 10:00 a.m. PDT, next dependency: the release owner found no existing smoke-only CI entrypoint. The available production workflow couples smoke=true to another deployment; that path can meet the three unresolved native-operation holds. The prior release smoke used the dedicated http:release-smoke identity and passed Answer and Workspace. The later independent Workspace 404 used mcp:default, so it does not prove the dedicated identity lost fixture access.

    The existing #3456 source writer is now scoping a separate, small smoke-only workflow PR on current main. Its proposed contract is an exact main/build fence, the production environment and deploy lock, existing dedicated smoke credentials and config, a --check --check-reader preflight, one unique-thread three-scenario run, and an artifact saved even on incomplete execution. The release owner will verify the route and keep acceptance custody. No smoke run, deploy, or Review was started by this update.
14. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 10:10 a.m. PDT: the existing Memory source owner opened separate draft smoke-only PR #3489 at exact head 0d8b0e8ff5c7d6a32680ec1b41e3efcbb8bee22d, based on deployed main e1fcac8a9cb6a52ccaaf5fdc8ab84f26e487e2b2. Local focused workflow tests passed 162/162, with TypeScript, consistency, scoped lint/format and spec checks reported green. CI is running; there is no current-head formal Review or production smoke yet.

    Independent read confirms the PR changes only the workflow, its spec row and its focused tests. Its admission fence names the successful exact-build controller 40361683839; the release tag v1.309.0 points to e1fcac8. It checks the four live Worker builds, Bot drain, Resident registry and dedicated reader before one unique-thread smoke, and always uploads the receipt. The release owner is auditing that contract at this head. #3460 and #3481 remain unchanged.

    PR #3476 is a separate reader reliability follow-up: the released smoke script can mark RPC -32004 incomplete without a second admission. It is not a prerequisite for one bounded attempt; if that read or the Workspace Git Door clone fails, preserve the partial receipt and stop. No Review, deploy, reset or smoke run was started by this checkpoint.
15. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 10:11 a.m. PDT correction to the draft #3489 checkpoint: release-owner source audit found three pre-admission defects at head 0d8b0e8. The released plan gives Memory and Resident wakeUrl rather than the health URL shape used by the draft workflow, so its fleet check would refuse before smoke. The job is missing the Cloudflare read token needed by the released Sandbox observation path. The mutable smoke config is not yet pinned to the intended disposable fixture before effectful Review. The source owner is fixing all three on the same PR branch and will rerun focused checks before requesting current-head Review. The draft is not acceptance-ready. No production smoke was started.

    Additional verified gate: the Resident predicate must reject an empty registry, since jq all([]) succeeds. The source owner has this repair.

    Further verified gate: the released tree has a Sandbox Wrangler template but no rendered wrangler.jsonc; the released Sandbox smoke observer runs Wrangler in that directory without an explicit config path. The smoke-only job must render the profile-bound Worker configs locally before plan/observation, without uploading Workers. The source owner is repairing this on the same branch.

    October 26, 10:27 a.m. PDT: source repairs are pushed at #3489 head 9a16345. The PR is now READY/open/mergeable. Prior exact-head CI finished green with planned skips; the ready transition started new authorize and staging Review E2E checks, still in progress. Both owner chats confirmed the release owner alone will request one current-head PR Review after these settle and an active-review check. No Review or production smoke has started.

    October 26, 10:29 a.m. PDT: release owner admitted ONE #3489 current-head Review request for 9a16345. Original run 0d441979-00c1-4c40-aaf9-016df2a5bbb8, receipt organization:coreplanelabs:61905, thread mcp:default:pr3489review. A direct runs_get read confirms agent=review, repo=coreplanelabs/switchboard and unfinished; target PR/head are not yet in that projection. Keep this original ID and wait for target/head, findings and terminal GitHub verdict. No duplicate request, smoke or deploy.

    October 26, 10:38 a.m. PDT: original #3489 Review run 0d441979-00c1-4c40-aaf9-016df2a5bbb8 is completed, sealed and has confirmed runtime shutdown. GitHub review 5656383732 is COMMENTED at exact head 9a16345 with one valid F1, not approval: the Resident readiness filter uses unique repository resources and omits pending copies, unlike the released physical resident check. The source owner has a same-branch fix and focused proof to make. #3489 needs new-head CI and one re-review after that push; merge and production smoke remain closed.

    October 26, 10:43 a.m. PDT: F1 is fixed and replied on the same PR at new head 2db2555. The readiness filter now uses physical residentId (resource fallback), refuses pendingReplicas, and keeps the other readiness guards. A red-before-fix healthy two-copy case and focused checks passed locally. New-head CI is running; old GitHub review 5656383732 is pinned to 9a16345. The release owner remains the sole requester for one same-thread re-review after terminal checks, F1 disposition and no-active-run check. No smoke or merge.

    October 26, 10:51 a.m. PDT: release owner admitted ONE same-thread #3489 re-review request at fixed head 2db2555. Original run 9dd518a0-3325-4343-8f1f-efb2d67b373a, receipt organization:coreplanelabs:62009, thread mcp:default:pr3489review. Preflight found 22 successful checks, 8 planned skips, F1 reply and no active same-head Review. Immediate runs_get returned not_found; this is an evidence/materialization gap, not authority to repeat. Preserve this ID until target/head, ending, findings and GitHub verdict read back. No smoke, merge or deploy.

    October 26, 10:58 a.m. PDT: re-review original 9dd518a0-3325-4343-8f1f-efb2d67b373a materialized at exact head 2db2555, then failed/sealed with confirmed runtime shutdown before review_read or submit_verdict. Event 91 records history_tail_unavailable during run.model_source_check; GitHub has no review at this head. Its lack of a verdict does not approve #3489. The release owner checked terminal custody, unchanged head/green CI and no active same-head Review, then admitted one new request using review-context:fresh: run bc6f0342-d58d-4fd7-8746-2979f5855c47, receipt organization:coreplanelabs:62045, same PR Review thread. My direct runs_events read now confirms run_meta seq 11 binds PR #3489, branch codex/smoke-only-acceptance and exact head 2db2555. The run remains unfinished. Preserve both IDs; no duplicate admission, smoke or merge.

    October 26, 11:18 a.m. PDT: #3489 received same-head LGTM review 5656737151 and GitHub APPROVED review 5656746655 at 2db2555, then merged as ed0df1e at 11:14:58 a.m. PDT. One production capability smoke workflow dispatch, 40376300155, is now in progress at verifier main ed0df1e. No smoke artifact exists yet; do not classify Answer, Workspace or Review as accepted from workflow admission. #3460 remains READY/OPEN at c07f95d with only an old-head COMMENTED review; its current-head Review waits for the release owner's exact smoke artifact and acceptance classification. Native-operation custody remains separate.
16. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 11:27 a.m. PDT: production v1.309.0 capability acceptance passed. The single guarded smoke-only workflow 40376300155, attempt 1 at verifier main ed0df1ea1c0dd65ed5898b7059ecce985d2d290d, is terminal SUCCESS. Uploaded artifact smoke-only-40376300155-1 (ID 12675506229) records capabilityOutcome=passed against deployed build e1fcac8a9cb6a52ccaaf5fdc8ab84f26e487e2b2 / v1.309.0.

    • Answer: passed, original a7094ebd-467a-41d5-9f0b-43f90484c955.
    • Workspace: passed, original 9535415b-587e-4de2-a681-6fbd9b3ea69c.
    • Review: passed, original ab10828e-8d68-42e6-bbc3-23e823fa9b33; GitHub review 5657016991 posted LGTM at fixture head 9a8e770b7ebdc245e251f01257d2bb8091da5317.

    Direct runs_get(include: shutdown) reads confirm all three originals completed, sealed and have recorded runtime-shutdown acknowledgments. This receipt clears the production capability and live Review acceptance prerequisite for this issue. The artifact separately marks productAcceptance=incomplete: the private question → Fix it → draft PR journey is still unproven.

    The existing Memory source owner now owns the next gate: reconcile PR #3460 at current head c07f95da3946e02b862b74faa2b6f5495acfd334, confirm no active same-head Review, then request one current-head Review and address its findings. #3460 merge, deployment and live proof remain separate. Draft #3481 follows #3460. The three retained native-operation records and historical uncertain writes remain with their custody owners; this smoke did not settle them. No reset, replay or second smoke was started.

    October 26, 11:29 a.m. PDT: the Memory source owner made ONE #3460 current-head Review request after the capability PASS. The MCP admission response returned no run ID, so we held the original request. Read-only plane_show, runs_get and runs_events reconciled it to live original run e2e856b1-2c73-4d96-8487-8b673bdfe562, receipt organization:coreplanelabs:62223, in thread mcp:default:pr3460-memory-source-recovery. run_meta seq 11 binds PR #3460, ref codex/memory-source-recovery and exact head c07f95d. It remains unfinished/admitted. Preserve this run ID; no duplicate Review, merge or deployment follows from admission.

    October 26, 11:35 a.m. PDT: #3460 current-head Review original e2e856b1-2c73-4d96-8487-8b673bdfe562 failed/sealed with confirmed shutdown at the history source check (history_tail_unavailable), before a verdict or GitHub post. The old-head COMMENTED review remains the only GitHub review. After confirming no active same-head run, the source owner made ONE distinct fresh-context request with idempotencyKey pr3460-c07f95da3946e02b862b74faa2b6f5495acfd334-fresh-history-tail. Its admission returned no ID, then read-only plane_show/runs_get/runs_events reconciled it to live original d37b360f-af02-4033-8db7-3eb7272c93c6. run_meta binds PR #3460 and exact head c07f95d. It is unfinished/admitted; preserve both IDs and do not dispatch again. Merge and deployment remain held.

    October 26, 11:52 a.m. PDT: #3460's fresh-context Review original d37b360f-af02-4033-8db7-3eb7272c93c6 completed/sealed with runtime shutdown confirmed. It approved exact head c07f95d and marked prior F1 fixed; GitHub LGTM review 5657264279 and APPROVED review 5657266426 are pinned to that head. PR #3460 then merged as main a6865cd at 11:51:55 a.m. PDT. This clears source, current-head Review and merge only. Production deployment of the fix and live proof remain with the release owner and its native-operation safety gates; the earlier v1.309.0 capability smoke was on deployed e1fcac8. Draft #3481 auto-restacked on main to head 96e255d and has new CI pending.

    October 26, 11:53 a.m. PDT: #3460's merge produced release PR #3490 (v1.310.0), OPEN/ready at exact head 8263209 on base a6865cd. Its sole automatic deploy-staging controller 40381616713 is IN_PROGRESS and current PR checks are pending. The existing release coordinator chat owns #3490 read-only staging, Review, merge-readiness, deployment and live-acceptance tracking; this ownership does not grant merge or deploy authority. Fresh Switchboard status still reports production v1.309.0/e1fcac8, so the earlier capability smoke does not validate the new fix. Preserve the three native-operation holds and do not duplicate the staging controller. Draft #3481 is now on main a6865cd at head 800aa27 with CI pending.

    October 26, 12:03 p.m. PDT: draft OpenCode source PR #3481 was restacked and marked READY at exact head 800aa27, base main a6865cd. After terminal current-head CI (32 success, 6 planned skips), no GitHub reviews and no active same-head Review, the sole source owner admitted ONE fresh-context Review request: original run 17a0d077-1e78-456c-81a6-ce4428b7a341, receipt organization:coreplanelabs:62400, thread mcp:default:pr3481-opencode-source-hold. Direct runs_get shows it unfinished, and run_meta seq 11 binds PR #3481, branch codex/opencode-source-hold and exact head 800aa27. Preserve the run ID; findings, merge, deployment, native storage and OpenCode production intake remain separate gates.

    October 26, 12:11 p.m. PDT: #3481 first Review original 17a0d077-1e78-456c-81a6-ce4428b7a341 failed/sealed during sandbox prepare_checkout after a worker /exec connection closed. It made no model turn, Review tool call or GitHub post, but retained shutdownEvidence is not-recorded/runtimeShutdown unknown; the checkout effect and physical ending are not proven absent. The source owner admitted ONE replacement request, run 5ee7b254-da56-43e8-b53c-0fa8bf6919a6, receipt organization:coreplanelabs:62436, at the same exact head 800aa27. Direct run_meta binds PR #3481/ref/head; it remains unfinished/preparing. The two runs have distinct sandbox keys (review:17a0d077 and review:5ee7b254). Preserve both original IDs and the old unknown; no third request, merge, deployment or intake follows from a setup refusal.

    October 26, 12:42 p.m. PDT: #3481's F1 repair was pushed at current head cc97767 with local pinned-binary proof and CI green. The source owner sent ONE same-thread re-review request, ID cd420009-b559-4479-8fe6-649be31fb8a8, receipt organization:coreplanelabs:62593. Direct runs_request reports ownerStatus=uncertain, handlerOutcome=uncertain, failure stage=execute, code=history_context_unavailable. runs_get has no run record and GitHub has no current-head Review. Preserve this accepted/uncertain original; absence from the run list does not prove nonexecution or authorize a second request. Source, merge, deployment, native storage and production OpenCode intake remain distinct.
17. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    ContributorAuthor

    More actions

    October 26, 11:32 a.m. PDT custody read for the original Nominal PR #5229 Review run 7ef063fd-0d55-4a72-a7a6-3f42f5285d0f: runs_get(include: shutdown) now reports finished:true, status:interrupted, and finishedAt 9:48:03 a.m. PDT. It reports eventCount:335, storedEventCount:331, and truncated:true. The last retained event is seq 335 checkpoint_deferred; runs_events(afterSeq:330) returned no terminal event.

    Retained shutdown evidence still says status:not-recorded, runtimeShutdown:unknown, basis:none. The finished-run shutdown check is unavailable. This is a terminal run-record projection. It does not prove physical runtime ending, a Review verdict, or the effects of the timed-out writes. The later exact-head Review and PR merge remain separate evidence.

    The original events supply exact request-bound selectors under owner generation 20261022T160828Z-04942b9d:

    Event	Write	Request ID	Digest	Read-only receipt result
    229	/runs/heartbeat (live-state)	426f5a4b-de67-4734-8a3e-8e696cb23d8c	e583acaeacda6bda192d001066daab5ae368fa691ebe43a776310223f5f6d4d7	runs_receipt: not_found
    231	/runs/append (conversation-write)	e4ec7f5b-d687-4e0d-aed3-3431ef41c4e1	0fc2d7ad7f57d9818d4da2e5b731dcd36f3f16184ed27bcd168a82d37c15ba04	runs_receipt: not_found

    Both receipt reads used this original run ID, owner generation, request ID, digest and operation. not_found is an unavailable read, not proof that either write did not commit. runs_original also returned not_found for the saved generation; it does not establish absence or ending. Keep the original outcome and write effects unknown. The next material evidence is an exact request-bound completion/readback receipt or a retained original runtime-shutdown acknowledgment. The original run owner keeps custody. No replay, Stop, reset, settlement or retirement follows from this status.
18. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 12:32 p.m. PDT: v1.310.0 staging capability acceptance failed. The sole #3490 staging controller 40381616713 is terminal FAILURE at candidate head 8263209dd338d24f11cc50096b854a397a8d2ff1. Artifact staging-deploy-40381616713-1 (ID 12682240212) records Answer passed, Workspace original 6ed41546-ba80-4e48-ba15-efa287181f74 failed with run_not_completed, and Review skipped because the prior scenario was unproven.

    The release owner's staging events classify the Workspace failure as tool_recovery_exhausted. Two model Bash calls for a simple directory read each supplied timeout:120000. The pinned Pi Bash API treats this value as seconds; src/core/harness/pi/toolRules.ts refused both calls before shell execution because each exceeded the remaining run loop. The run did not read math.mjs. This is not evidence of a Git Door refusal or a #3460 Memory failure. The Workspace run is failed/sealed in the staging artifact, but physical runtime shutdown is not independently proved by a production MCP not_found.

    The existing Memory source writer now owns one separate, narrow source fix for the staging Workspace prompt/tool contract. The release coordinator retains the original controller and future release gates. No replacement staging controller, replay, merge or production deploy follows from this failure. Production still serves v1.309.0/e1fcac8; the earlier capability smoke qualifies only that build. Native-operation and private product-journey gaps remain separate.

    Release order, 12:34 p.m. PDT: keep ready #3484 at f4cec01 out of the current v1.310.0 candidate. It protects uncertain seed workspace cleanup but does not fix the observed Bash timeout-unit refusal. The order is one narrow timeout source fix with its own Review and merge, then one new-head #3490 staging receipt, then separate current-head Review, native safety, production authority and live acceptance gates. #3484 follows in a later release unless new safety evidence changes priority. No #3484 merge or new controller follows from this plan.

    October 26, 12:42 p.m. PDT: the sole source writer opened draft PR #3491 at exact head cbd800a on main a6865cd. It adds a Pi Bash timeout-is-seconds cue with 120 seconds for two minutes to the Workspace smoke request, plus matching public request-output test and spec row. The release owner found no source-diff objection. A red/green test proves the emitted instruction, not model compliance. Current-head CI and Review are still pending; only a later terminal new-head #3490 staging Workspace/Review artifact can prove runtime acceptance. No controller rerun or deployment has followed.

    October 26, 12:44 p.m. PDT: #3491 is now READY/OPEN/MERGEABLE at unchanged head cbd800a. CI has 25 successes, 8 planned skips, only CodeQL analysis still running and no failures. The sole source owner sent ONE fresh-context Review request: run acf98df3-eb22-480c-88e0-271f21811af9, receipt organization:coreplanelabs:62608, thread mcp:default:pr3491-staging-workspace-timeout. Direct plane_show/runs_get show it unfinished/admitted; run_meta binds PR #3491/ref codex/staging-workspace-timeout/exact head. Preserve this original ID and wait for verdict; no duplicate Review, merge, controller retry or deploy.

    October 26, 12:54 p.m. PDT: #3491 received current-head LGTM review 5658061286 and APPROVED review 5658063116, both pinned to cbd800a. Justin merged it at 12:53:30 p.m. as main 807d686. Immediate #3490 read still shows old failed head 8263209 and base a6865cd; release-please has not yet reheaded it. The release owner will follow that existing mechanism for one automatic new-head staging controller. The reviewed source merge is not staging acceptance, production deployment or live proof. Preserve old failed controller 40381616713 and Workspace original 6ed41546; no manual retry.

    October 26, 12:54 p.m. PDT: release-please reheaded #3490 to exact candidate 2071197 on base 807d686. ONE automatic deploy-staging controller 40390839387 exists at that head; authorize passed, validate is in progress. There is no terminal Answer, Workspace or Review artifact yet. The old controller 40381616713 remains failed at superseded 8263209 and cannot qualify the new head. The release owner follows only 40390839387; no manual dispatch, duplicate Review or production deploy.

    October 26, 1:16 p.m. PDT: new-head controller 40390839387 attempt 1 is terminal FAILURE at #3490 head 2071197. Artifact staging-deploy-40390839387-1 (ID 12685311187) has setup, fleet and deploy logs but no staging-e2e scenario file. Memory and Resident reached exact staging build 2071; Bot remained old 8263209, Sandbox was not attempted. Bot Docker build failed before upload when the runner fetched bowser@2.15.0 from npm and received E404. The same tarball now answers HEAD 200/range GET 206; a transient registry/CDN fault is plausible but unproved. No Answer, Workspace or Review scenario was admitted, so #3491's prompt fix was not exercised. This is a partial staging rollout, not capability acceptance. The release owner is evaluating the workflow's supported same-run failed-jobs recovery; no new controller or production action follows from this receipt. Preserve attempt 1 artifact and the old Workspace original 6ed41546 separately.

    October 26, 1:17 p.m. PDT: GitHub now reports run_attempt=2, status in_progress, head 2071197, on the SAME controller ID 40390839387. The release owner used the workflow's failed-jobs recovery after classifying the partial staging state. Attempt 1 remains its own failed artifact; attempt 2 has no terminal staging or smoke receipt yet. No new controller or production deployment was started.
19. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    ContributorAuthor

    More actions

    October 26, 1:16 p.m. PDT: #3492 Review original custody is held read-only by this live recovery chat. Run 89bb52d7-26c0-4ee4-b5e2-b10df5b7cf6e belongs to PR #3492 at head fc10c51c67751bb81c4454dde31c93c5a30af2a2, original owner generation 20261026T164740Z-fb82325f. The source chat retains the sole PR writer and Review requester.

    Direct runs_get says finished:false, recoveryPending:true, producerUnconfirmed:true, with runtime shutdown unknown. Events 268–272 show an attempted submit_verdict, a tool result “the run ended,” and a note that the recovery checkpoint could not be confirmed. runs_request is uncertain at execute. runs_original records state, step, context checkpoint and producer through event 272, but appendAcknowledgement=not-recorded, failedRequestBinding=unknown, and physicalEnding=not-assessed. GitHub had no review pinned to the current head at the read. Absence of a post is not proof of no effect.

    Next evidence that can change custody: an exact original-generation append or verdict receipt, producer/runtime-ending acknowledgment, or a matching pinned GitHub review. There is no supported MCP same-ID resume control exposed in this client. Preserve this original; do not dispatch a replacement, Stop, settle, reset or merge from its current records. This custody is separate from the #3490 staging failure and #3481 OpenCode source work.
20. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 1:56 p.m. PDT: v1.310.0 exact-head staging capability acceptance passed. Controller 40390839387 attempt 2 is terminal SUCCESS at candidate 2071197e176f25abc9af81327b89ec941943b86b. Uploaded artifact staging-deploy-40390839387-2 (ID 12688186531) records capabilityOutcome=passed and build v1.310.0/2071197e.

    • Answer original e22de19d-c9b3-4714-82ba-9c3766460d13: passed, activity 4.
    • Workspace original cd15d3a1-dbc8-4003-b281-e44a90b0b318: passed, read the staging fixture math.mjs.
    • Review original b78bac7d-e4f2-4212-8ee2-6e5ce07be9cb: passed, posted the expected finding on the intentionally broken staging fixture PR Authorization: per-agent user allowlists + restricted channel config #1 at head bfe2d4db936ec136e946943ad62c87118a946dcd; GitHub review 5658778321 is pinned to that head.

    The artifact records all three runs finished, sealed and persisted. It does not independently prove their physical runtime shutdown. It also marks the private question → Fix it → draft PR product journey incomplete. Attempt 1 remains a separate partial-staging failure; its artifact 12685311187 is retained.

    Release PR #3490 is OPEN/MERGEABLE at this same head with current checks settled green/planned skip (Polylane NEUTRAL). The release owner admitted ONE current-head Switchboard PR Review, run a689b734-ea3d-438c-9b7b-1eb15caef177, receipt organization:coreplanelabs:62920, thread mcp:default:pr3490review. Direct run metadata binds PR #3490 and exact head 2071197e; it is unfinished. Preserve this run. The next gates are its terminal findings and pinned GitHub verdict, then separate merge, native-operation safety, production deployment and exact-live acceptance authority. Fresh production status still serves v1.309.0/e1fcac8; staging is not shipment.

    October 26, 2:01 p.m. PDT: #3490's sole current-head Review a689b734-ea3d-438c-9b7b-1eb15caef177 completed/sealed with runtime shutdown confirmed, verdict approve and zero findings. GitHub LGTM review 5658906673 and APPROVED review 5658909025 are both pinned to exact head 2071197. Required checks are terminal green/planned skip (Polylane NEUTRAL); source, staging and PR Review gates are clear. Production still serves v1.309.0/e1fcac8.

    The release remains an operational HOLD. Exact-head deploy targets include all four Workers, including Resident. In the merged source, residentEarlyHoldProblem reads native-operation rows before upload and refuses any closed row without original ending evidence. Production retains three such Switchboard rows; --force changes that refusal to an explicit warning but does not prove the original endings or write effects. Earlier v1.309 force authorization is exact-release scope and does not authorize v1.310. Native and release owners are preparing per-row custody and a concrete force/hold plan. Do not merge #3490, deploy, reset, retire or replay from this reviewed/staged receipt. Private product acceptance remains incomplete.
21. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 2:03 p.m. PDT: v1.310.0 production release decision: HOLD. Release PR #3490 is open/clean/mergeable at exact head 2071197e176f25abc9af81327b89ec941943b86b. Required checks are green or planned skips, the current-head Review a689b734-ea3d-438c-9b7b-1eb15caef177 approved with zero findings and confirmed shutdown, and staging controller 40390839387 attempt 2 passed the Answer, Workspace and fixture Review capability gate. Production still serves v1.309.0/e1fcac8; the private question → Fix it → draft PR journey remains unproven.

    The ordinary production path is blocked before Worker upload: the #3490 deploy target selects all four Workers, including Resident, and residentEarlyHoldProblem refuses closed native-operation rows without original ending evidence. Production Switchboard retains three such rows with pendingAttempts=1 and no process IDs. The current workload is a best-effort inventory, not physical ending proof. Nominal and Polylane rows are separate custody. Merging the release PR would cut v1.310.0 and normally start an automatic deploy into this known hold.

    Preferred path: the existing native owner provides exact per-row original-ending/recovery receipts. Any effectful closure needs its own authority. After the ordinary preflight is clear, the release owner can refresh the PR head, checks, grants, fleet and deployment plan and bring the normal merge/deploy decision forward.

    Force alternative requires new, exact human authority. If Justin chooses to roll v1.310.0 despite unknown native effects, the release owner has prepared this reviewable sequence: set and verify the exact-tag automatic-deploy hold SWITCHBOARD_RELEASE_DEPLOY_SKIP_TAG=v1.310.0 before merge; merge #3490 at its reviewed head; verify the tag and image publications; then run one manual deploy-production from the exact release commit with force=true, all four targets and smoke=false. Force bypasses the early native hold with a warning. It does not prove prior processes ended, clear records, or guarantee readiness. The variable change, merge and force deployment each need the scoped authority for this release. No step in this branch has run.

    After any authorized deployment, require terminal controller and exact live-build/readiness receipts for all four Workers, Bot and Resident undrained state, Sandbox rollout/probe, then one dedicated smoke-only Answer → Workspace → Review run pinned to the release tag, deployed commit, successful deploy run ID and verified disposable fixture head. Preserve every partial or uncertain original and stop on failure; no blind retry, reset, rollback or replay. Native custody and private product acceptance remain separate even if capability smoke passes.

    Next trigger: the native owner's per-row receipt or a later explicit v1.310.0 release decision. No merge, deploy or native control from this comment.

    October 26, 2:20 p.m. PDT: Justin chose HOLD for native ending proof. This is the v1.310.0 release decision. Do not set SWITCHBOARD_RELEASE_DEPLOY_SKIP_TAG, merge #3490, start a production deploy or smoke run, or use a native control under this decision. #3490 remains open at head 2071197e176f25abc9af81327b89ec941943b86b; production remains v1.309.0/e1fcac8. The existing native owner continues exact per-row read-only ending/recovery proof. The #3456 live Review recovery chat owns this release decision and next release gate because the former v1.309 release coordination chat is archived; the portfolio owner has the decision receipt. Any effectful closure or later release path needs a separate, current decision and scope.

    October 26, 2:32 p.m. PDT: #3451 merged as main 69483748e7560bf449c8435be24e2e833e030d1e. #3490 moved to bf94cc22649663168193096a9eb0e2cb7ed46a5c on that base. CI is in progress; the old 2071197e GitHub Review and staging receipts do not qualify this new head. Justin’s explicit v1.310.0 HOLD persists; no merge, deploy, force or new smoke. Requalify checks, one current-head Review and staging before a later separate release decision.

    October 26, 2:51 p.m. PDT: #3494 merged as main 5e128ed457fade81681f7c0e458ef6fbca498d75. #3490 moved to OPEN/MERGEABLE head 1b9565135e815c749eb0ba9f148604535ff1c285; CI is in progress, and its only GitHub Review/approval is pinned to old 2071197e. No current-head Review or staging receipt qualifies 1b95651. #3493 is merge-ready source at d54c60b but remains open pending Justin’s exact merge decision; settle that choice before one final #3490 head qualification. The v1.310.0 production HOLD and native custody remain unchanged. No release action occurred.

    October 26, 2:54 p.m. PDT: #3493 merged from reviewed head d54c60b as main eefc63f6a61d9eed8d24c40f3f65fbbb01e6163b at 2:53:10 p.m. PDT. #3490 moved to OPEN/MERGEABLE but BLOCKED head 17156bf942ec41be0e455f8d62f6fe5bd3274624 on that base; checks remain in progress and its only GitHub Review/approval is pinned to old 2071197e. Qualify one final head after checks settle and any further main movement. The explicit production HOLD remains: no release merge, tag skip, deploy, force or native control. Production remains v1.309.0/e1fcac8.

    October 26, 4:18 p.m. PDT: Justin asked to close the dedicated #3456 recovery chat. The existing portfolio coordination chat accepted read-only release/native/original-run custody and the sole #3490 controller watch; it recorded the exact owner map and action limits in the work queue. #3490 remains OPEN/BLOCKED at handoff head c38f30d03519e48b25128fd6097b7c15622c47e3 on base 03727550d947e66833451369a78a4b584b6b8338; production remains v1.309.0/e1fcac8. The explicit v1.310.0 HOLD and all separate effectful-action gates remain unchanged.
22. justinhelmer commented on Oct 26, 2026

    

    justinhelmer

    on Oct 26, 2026

    ContributorAuthor

    More actions

    October 26, 2:04 p.m. PDT: native owner’s exact per-row release safety receipt for #3490. Production Bot and Resident still serve v1.309.0/e1fcac8; the complete Switchboard Resident native inventory has exactly three closed rows, each `pendingAttempts=1), no saved process identity or retirement. This is best-effort custody evidence, not physical ending proof.

    Original row	Hash	Missing ending or cleanup evidence
    native-operation:mcp:default:pr3477review:7797213a-aa3a-4e15-a922-bde951e32308:314c5693-851c-471e-a5a0-b40260d0ed18	1c5ab288c87f09348682b7104bdd2ce69d51aac77b2c46abaa9e167a7fb9afb6	Pending process attempt a492860f-b45d-4347-8288-72f4d6261462 has no process identity.
    native-operation:mcp:default:pr3477review:7797213a-aa3a-4e15-a922-bde951e32308:44b662d9-6544-4824-9cfe-4d9b33558fd6	c87a67babc0e2a2222979fbeac00451fb13813bc13bdc4d8da5f06f07bb1e3d2	Pending process attempt 33b5e76f-64e4-4b81-ade4-6b59196b6afe has no process identity.
    native-operation:op:da026e68-bd7e-4d03-87b1-2f9907141eab	9829c4e9af441ba5a88ab967562a0458e5b75e9f6e73461562951693d68fa6db	Opaque checkout restore attempt a219705c-5fa0-4692-8758-a03db4d7eb81 lacks its own archive cleanup receipt; a saved mirror receipt belongs to a different attempt.

    The two #3477 rows share the original run 7797213a-aa3a-4e15-a922-bde951e32308, owner generation 20261023T085034Z-d3b795f9, fence 15323 and scope d4a58922-f840-437b-accf-7e56934b497c. Its run record is failed/sealed, but runtime shutdown is not recorded. An original-generation runs_original read returned not_found; this does not prove an ending. With no saved process identities, ordinary process recovery is unavailable. A scoped terminal retirement would require a Memory sealed-run permit and matching original owner/scope/unchanged hashes, plus separate effectful-control authority. The hydration row has scope f6d9adca-cec0-4d55-b2c2-60cabb784b78 on old boot 9385b324-b167-454b-9684-bb7cc5aeb85f; same-original archive cleanup may refuse and also needs separate authority. PR #3451’s restore-only selector cannot clear this mixed set. No supported same-ID read now supplies the missing receipts.

    For the exact #3490 all-four deploy, the ordinary Resident preflight refuses these closed rows before upload. force would warn and bypass that refusal, but it would not clear records, prove old processes or writes ended, or protect current in-flight work from interruption. It also affects the Resident Worker serving Nominal and Polylane cohorts, whose separate native rows this Switchboard receipt does not clear. Keep the v1.310.0 merge/deploy HOLD pending exact original evidence and scoped recovery authority, or Justin’s new explicit decision on the prepared force path. No native control or production action was taken in this read.
23. justinhelmer commented on Oct 27, 2026

    

    justinhelmer

    on Oct 27, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 1:10 p.m. PDT: #3492's Review lost a reply after a committed session write. This is an exact-request finding, not a general cause for every Memory timeout.

    Original Review run 89bb52d7-26c0-4ee4-b5e2-b10df5b7cf6e, generation 20261026T164740Z-fb82325f, sent /runs/session/write request 49db71e2-520b-4925-8e30-54c2f562a328 (digest 5ecff382a3cba74ff1a315289e6a0bf867e7430188075693624cf536bf276290). The Memory Worker completed the receipt-owner read in 62 ms and logged session-write-rpc started at 20:10:29.487Z. Its target was SessionLogDO object d8b0d44772b57fdb9f013cd6dfa0c324bc8e6cefeb3f4797f6ead7979486b0f6. Cloudflare Workers Logs record that object's write RPC as outcome=canceled, wallTimeMs=9911, cpuTimeMs=21 at 20:10:39.299Z. Object stage logs for the same request show transaction and result ok:true. The exact runs_receipt says state=recorded, rowsWritten=6, attachmentsWritten=0. The Bot timed out at 20:10:39.322Z, about 10 seconds after the request began. The Worker has no recorded session-write-rpc returned event for this request.

    The object stage logs report elapsed time of 0 ms for the write method. Earlier reads for the same object used incarnation 564d1963-4ac3-4c29-a201-cd10d3b4e7b4; this write used b081cfe6-7e20-4b33-ba82-a6aa72c2f214. A new object instance ran the write. Cloudflare can recreate an instance after idle eviction, deployment or runtime decisions, but these logs do not identify which occurred here. There is no recovered span tree or constructor timing. The long wait lies in or before the object RPC completion, but its precise cause, including any startup, scheduling or storage wait, remains unproved. The transaction itself completed; its reply did not reach the Bot before the deadline.

    In the same one-minute Workers Logs sample, the other 33 SessionLogDO.jsrpc invocations finished ok in 35–105 ms; this request was the only sampled canceled invocation at 9,911 ms. This sample shows an outlier, not an authoritative failure rate.

    The confirmed application failure path is: the Bot treated that effectful write response as uncertain, held its next step boundary, and the later submit_verdict attempt was cut with the run ended. The Review had attempted approve with zero findings at PR #3492 head fc10c51c67751bb81c4454dde31c93c5a30af2a2; no accepted current-head GitHub review is recorded. The original remains recovery-held. Do not replay its write, verdict or Review.

    PR #3496 is merged on main as 6daf979c0ad63fbe818172d6c5328b99b93c6dd9. For a future lost session-write reply, it reads the exact saved receipt and advances only when recorded success matches the row and attachment counts; it never resends that batch. This fix is not yet deployed. It covers this application failure mode, not the underlying long object RPC or the other effectful routes in this issue. Production release #3490 remains under Justin's explicit HOLD. To close the infrastructure cause, obtain request-bound object startup/queue/storage timing or a Cloudflare trace for a recurrence and compare the session-write tail latency and canceled RPCs before and after deployment. No release or duplicate run follows from this comment.
24. justinhelmer commented on Oct 27, 2026

    

    justinhelmer

    on Oct 27, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    October 26, 6:03 p.m. PDT: The October 22 shared timeout burst has one identified blocking call. This narrows the cause of the cascade, but the reason that call took 14 seconds remains unproved.

    At 11:55:11.228 a.m. PDT on October 22, original /runs/context-checkpoint request 292c4bd2-c4de-43b7-af11-f60c5700749a entered RunHistoryDO.readContextCheckpoint gate 682e6e00-d43b-49a5-8984-12c74a0184a8 on object 1f82993dbbbe5796d8c6d736c927018c3d444b2094479c6417d1e1416b4248db, incarnation ac503a9c-0c28-4121-857f-151637f5fb40. Its source-read stage took 14,194 ms and ended at 11:55:25.422 a.m.; the gate reported queueMs=0, heldMs=14194, then released completed. The deployed code awaits SessionLogDO.read inside this blockConcurrencyWhile gate. Cloudflare logged one long SessionLogDO.read invocation in that window: object adceaa97eb44e52e17da97f5583b5a003007d5d3d77fb76e2a277d98327413c5, event 01M5JFFMNG000000000000000N, outcome ok, wall 14,385 ms, CPU 9 ms. Its trace ID differs from the gate's trace, and the old read method did not log the caller request ID. The method association is supported by the code path and the sole long session read in the narrow window, not by an exact cross-object request-ID receipt.

    While the RunHistory gate awaited that read, other calls to the same object exhausted their 10-second clients. The shared /runs/deleted poll request 672b83f7-0f7a-408c-b089-8bb8bb0ca713 has a canceled RunHistory invocation 01M5JFFEJK0000000000000001 with 0 RPC methods and 0 ms CPU. PR #3460's /runs/append request 16b4d051-725f-4a35-8777-fee14b6cf361 has the same pattern at invocation 01M5JFFK7K0000000000000001. Its heartbeat request 1799babf-39e3-49ce-8352-31900234f6f4 was canceled at invocation 01M5JFFMV80000000000000003, about 10 seconds and 0 ms CPU. A later RunHistory invocation 01M5JFFMX600000000000000Y4 processed 45 RPC methods in 1,227 ms wall after the long source-read stage ended. The timeout notes shared by several Reviews do not represent separate /runs/deleted polls.

    The supported causal chain is: one long session source read held the shared RunHistory input gate; queued sibling calls missed their client deadlines; some later work ran after callers had timed out. The 14,385 ms SessionLog invocation used little CPU, but this alone cannot distinguish delayed object activation, SQLite/storage wait or another platform wait. Issue #3501 now has the narrow diagnostic goal. PR #3502 is being revised to add construction and request-bound method-stage markers without numeric durations for synchronous work: Cloudflare documents that deployed Worker clocks can stay frozen. No production trace flag, original write, Review or release was changed by this read-only investigation.
25. justinhelmer commented on Oct 27, 2026

    

    justinhelmer

    on Oct 27, 2026

    Last edited by justinhelmer

    ContributorAuthor

    More actions

    Oct 26, 2026, 6:42 p.m. PDT: two Review source checks timed out

    Both Reviews reached the model and used tools. Each failed later, before its next model turn, when a source check called /runs/context-checkpoint. Neither produced a Review verdict. Both runs are terminal with runtime shutdown confirmed. Do not replay either admission from a timeout.

    Review	Checkpoint request	Memory receiver	Original admission receipt
    #3475, run df3ddb5e-d0d7-48f9-a01a-783e1f53556b	11c6011d-3af1-4fe8-b0cf-1b2f10d3d249	fetch invocation 01M5XGBQFH000000000000000D canceled after 9,996 ms, 0 ms CPU	organization:coreplanelabs:64429
    #3502, run 3f2bfd3e-c627-40fa-b055-89efba62b2a2	d2b0a4ba-9156-41d4-8b9f-52d23dc4b913	fetch invocation 01M5XGBQRF000000000000000D canceled after 9,992 ms, 1 ms CPU	organization:coreplanelabs:64441

    For #3475, RunHistoryDO.readContextCheckpoint entered input gate 33023afd-0b86-4c93-af37-2c9a76c259e8 on object 1f82993dbbbe5796d8c6d736c927018c3d444b2094479c6417d1e1416b4248db at 01:42:24.472Z. Its source-read stage held the gate for 13,571 ms and released with completed at 01:42:38.043Z, after the caller timed out. The gate recorded ownerGen=null and no recovery stage. The code's archived-source path only reads, so this observed checkpoint call did not install a pending checkpoint.

    In the same narrow interval, native SessionLogDO.read invocation 01M5XGC4NY000000000000000N on object 4c3ba5cb9262d10eacba5b0ad4898b32fbc7fb30b6485763926263b0aacdfc9b took 13,565 ms wall time, 1 ms CPU, and ended ok at 01:42:38.014Z. The deployed callee did not log the caller's request ID. The association is from the code path and the matching interval, not a direct ID join. The wait inside this read remains unknown.

    The #3502 Memory request arrived at 01:42:24.783Z, 311 ms after that gate entered. Its receiver canceled about 3.3 seconds before the gate released. No #3502 gate entry was retained. Because the route uses the same default run-store object, waiting behind that gate is the leading explanation, but there is no direct callee receipt for #3502. Its checkpoint effect stays unknown.

    runs_request still reports ownerStatus=uncertain, handlerOutcome=uncertain, ownerStage=execute for both admission receipts above. That admission state is separate from the checkpoint RPC outcome. Both Reviews ran and then failed without verdict. The production Memory build was e1fcac8a9cb6a52ccaaf5fdc8ab84f26e487e2b2 at this event. PR #3502 adds narrow construction and session-read stage markers, but they were not deployed for this event. Do not call the 13.565-second wait a Cloudflare fault or an application fault without its internal stage evidence.

    Next safe trigger and source decision. Each admission remains held until an authoritative original owner/effect receipt resolves its uncertain status. Reconcile that receipt with the already terminal run and its missing verdict before any separately authorized fresh exact-head Review. The terminal run alone does not permit a second dispatch. The source owner should watch the two existing receipt IDs above; no duplicate admission or new writer is needed.

    There is no supported small source fix for the 13,565 ms read now. The deployed SessionLogDO.read body has no application await: it queries indexed rows, loads referenced attachments and projects the result. The same object had 22 reads near 53–98 ms, one at 680 ms and one at 4,245 ms in the preceding nine minutes. The 1 ms CPU outlier does not distinguish object activation, SQLite/storage wait or other platform scheduling. Moving the external read outside RunHistoryDO.blockConcurrencyWhile would change checkpoint serialization. Raising the 10-second caller timeout would hide the cause and could extend uncertain outcomes.

    The root-cause trigger is a qualified deployment of PR #3502 under the release owner's separate hold, followed by a naturally occurring long read with correlated constructor/read-stage markers and native invocation wall/CPU records. Those markers identify the last reached stage. They do not divide synchronous duration or, alone, prove a Cloudflare fault. Do not enable broader tracing for this incident without a separate measured need.

    Merge and deployment status. Justin merged #3502 from b54b0f7e0 as 184e9001399759ded20ea6611d907540b0e747a9 at 7:30 p.m. PDT and #3492 from d998c281b as 82b72ebc253046c82592f013f6c224872983be4c at 7:37 p.m. PDT. Main now matches 82b72ebc. These human merges do not supply either missing Review verdict or settle the original admissions. Production still served v1.309.0 build e1fcac8a on the last read. The v1.310.0 release PR #3490 remains open under its separate Resident native-operation HOLD; the new diagnostic markers are not yet live.