Repository navigation
A save's backups get what a Durable Object's alarm leaves them - #1020
Conversation
A save runs inside the alarm whose events asked for it (an idle sleep's hold, save and stop; an awake save's hold, save and unhold), and Cloudflare ends an alarm at 15 minutes. So the bound on a save's backups is sized from that wall, as a start's restore is from its deadline: 15 minutes, less a minute for the alarm's other events and the bounds of the sleep's other steps, is 11 minutes 10 seconds (it was 8). Measured on production (577 saves over 1 GB, 2026-10-27 to 11-03): saves moved 9 to 156 MB/s, median 45. A 20 GB disk holds about 17 GB of a person's data, about 10.8 GB saved at the 1.6 to 1 the largest save showed. At 8 minutes, 18% of such a save's tries would be cut; at 11 minutes 10 seconds, 6%, and each is tried again. Nothing longer is possible inside an alarm, so a bound that follows the save's progress would let no save run longer (it would only cut a stalled one sooner, which is not this risk). The lifecycle's save deadline, which only catches a report its isolate lost, goes from 10 to 13 minutes to stay past the cell's own bounds, and an owner's sleep now answers within 14 minutes (docs/api.md). The arithmetic is checked as the cell compiles: the sleep's steps plus the alarm's other events fit the wall, the backups get at least 11 minutes, and the sleep answers within 14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
futurepaul
left a comment
There was a problem hiding this comment.
Codex review @ 22eff52: CHANGES
- [P1] The alarm budget omits the start that can follow the sleep.
cell/src/computer.rs:200budgets throughStop, but a record/tab/owner wake arriving while that sleep saves setswake_after_sleep;Lifecycle::gonethen emitsAction::StartonAsleep.drive_eventsimmediately awaits that start in the original alarm (cell/src/computer.rs:926,:976). A no-wait probe against this head confirmed a permitted sleep finishing at +680 seconds emitsStart { generation: 2 }; a four-minute restore then needs +920 seconds, beyond the 900-second alarm wall, despite fitting its own start deadline. The handler can be cut during restore, before it reports Ready, with its lifecycle still Starting. Give the follow-on start a fresh invocation with a durable handoff, rather than spending the remainder of the sleep's alarm on it, and add a deterministic long-save-plus-wake test that checks this invocation boundary.
Non-blocking:
cell/src/computer.rs:188:ALARM_OTHER_MSis reserved headroom, not an enforced bound. For example,try_flush_awakeawaitsledger::askwithout a timeout. The compile-time assertion proves the allocated durations add up, not that these ancillary calls finish within the reserved minute; describe that limit explicitly or enforce a shared elapsed-time budget.
Checked the 55 core computer tests (all passed), shared backup timeout propagation, unchanged save-failure/backoff paths, and callers for stale 8/11-minute assumptions. No Docker/e2e run.
Review of #1020: a wake during an idle sleep's save (or a restart, or a crash while wanted, or a start falling back past a bad save) made the next start in what was left of the same alarm. An alarm ends at 15 minutes, and a sleep's steps can now take 14 of them, so that start would be cut mid-restore and left starting. The lifecycle no longer starts at once when a container goes (`gone`). It owes the start (`wake_after_sleep`, now kept until a start is made), marks it due now (`retry_at_ms`) and asks for an alarm at once. The next alarm is an invocation of its own and makes it, even if the wake's hold ran out during the save. A request, which nothing ends while it waits, makes a due start at once (`drive_with`), so a restart still answers with the view once started again. Every start an alarm makes now fits that alarm: `RESTORE_MS` also leaves the alarm's other events their minute (10.5 to 9.5 minutes). That still restores a full disk's save, 17 GB at 30 MB/s against the 38.6 measured, and covers a failed start's retry too, which had no slack before. Checked as the cell compiles. `ALARM_OTHER_MS` says plainly that it is reserved, not enforced: the ledger's `ask` and `tell_changed` take no bound of their own. Test: an idle sleep whose save runs eleven minutes, with a presence wake whose hold runs out during it. The sleep's end makes no start but asks for an alarm at once, and that alarm makes the start once. A request that comes first makes it instead. Tests that expected the start in the same step now take it from the alarm's step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
futurepaul
left a comment
There was a problem hiding this comment.
Codex review @ 626340c: CHANGES
-
[P2] A request follows only one owed start, so Restart can return before fallback finishes.
cell/src/computer.rs:844calls the follow-updrive_events(Alarm)only once. For a Restart, the first drive finishes the sleep; the follow-up starts the image, but if its save is unusable and an older one exists,RestoreFailedleaves another start due. The request then returns an asleep view instead of waiting for the older save's start, unlike the previous behavior and the stated request contract. Drain immediately due starts on request invocations until none remains (with the existing bounded fallback policy), while keeping alarm invocations limited to their own start. Add a request-path regression with an unusable first restore and a usable older save. -
[P2] An owner's Sleep no longer cancels a start owed after the stop. The new deferred state produced at
crates/core/src/computer.rs:963isAsleepwithwake_after_sleepandretry_at_msset, butEvent::Sleepin that phase still does nothing (:1279). If Sleep arrives after the prior stop and before its alarm,drive_withsees the start still due and starts the computer inside the Sleep request itself. Clear the pending wake/retry and the prior wake's hold when the owner sleeps an asleep computer; test that neither the request's follow-up nor a replayed alarm starts it.
The original same-alarm P1 is addressed. All 57 core computer tests passed; additional no-wait probes confirmed that the owed start survives serialization, expired presence, and late Asleep replay, and is consumed once, while reproducing both request-side cases above. The restore budget is now 570 seconds; ancillary-call headroom is explicitly documented as reserved rather than enforced. Docker/e2e were not run.
…e go Review of #1020: 1. A request made only one owed start. When a restart's start found its save unusable and fell back to an older one, the next start was left due, and the request answered with an asleep view. A request now makes each start that is due now, one after another, until none is. `STARTS_OWED_MAX` (new in core) bounds them: one owed after the container went, one past a snapshot, and one past each kept save. If the bound were ever reached, the alarm makes the next and the cell logs it. Alarms still make only their own start. 2. An owner's sleep of an asleep computer did nothing, so a start owed after a sleep that a wake came during was made inside the sleep request. Now that sleep lets go of the owed start, a failed start's retry, and the earlier wake's hold. A computer that is always on keeps its start, as after an owner's sleep of it awake. The lifecycle simulation holds this, since an asleep, wanted computer always has a start due. Tests: a restart whose newest save won't restore goes on to the older one in the same request and ends awake on it, after 2 starts. The longest chain the fallbacks allow is exactly `STARTS_OWED_MAX`, then a strike. An owner's sleep before the owed start leaves nothing for the request's follow-up or the replayed alarm to start, and it cancels a failed start's retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-alarm # Conflicts: # cell/src/computer.rs
#1018 reads a computer's disk once a start is ready, and after each awake save, inside the same invocation (`DISK_READ_MS`, 5 s). So a start an alarm makes must leave room for that read too. The merge of master carries the cell's side of this in its conflict resolution: - `RESTORE_MS` also leaves the disk's read, so it goes to 9 minutes 25 seconds. That still restores a full disk's 17 GB at 31 MB/s, against the 38.6 measured. - The start's alarm check counts the read. - An awake save's hold, save, unhold and read now have an alarm check of their own (820 s with the alarm's other events, of 900). This commit updates the core doc and computers.md to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
futurepaul
left a comment
There was a problem hiding this comment.
Codex review @ 9846dca: PASS
Both findings are addressed: requests drain immediately due starts within the five-start fallback bound, while alarms retain the separate-invocation handoff; owner Sleep cancels the pending start for ordinary computers and preserves it for always-on computers. The merge keeps Ready-before-disk-read ordering and counts the read in both budgets: restore gets 565 seconds, and an awake save totals 820 seconds plus 60 seconds of reserved headroom (880 of 900). All 62 core computer tests passed, including the longest fallback chain; additional no-wait probes verified ordinary and always-on cancellation behavior across serialization and alarm replay without duplicate starts; Docker/e2e were not run.
A computer's disk is now 20 GB (#1001), so a person can hold far more than before. The bound on a save's backups was 8 minutes, so a full disk's save would have started failing. This sizes that bound from the true ceiling instead.
The ceiling. A save runs inside the Durable Object alarm whose events asked for it:
Cloudflare ends an alarm handler at 15 minutes (limits). So no save can run past that, whatever its bound says.
The change.
BACKUPS_MS=ALARM_WALL_MS(15 min) −ALARM_OTHER_MS(1 min for the alarm's other events) − the bounds of the sleep's other steps = 11 min 10 s (was 8). This mirrors howRESTORE_MSis sized from the start's deadline.SAVE_DEADLINE_MSgoes from 10 to 13 minutes, so it stays past the cell's own bounds. It only catches a report its isolate lost.docs/api.md). No client in the repo times that call out shorter: the shell and the CLI don't call it, and e2e calls it against fakes.The arithmetic, checked as the cell compiles:
Measured on production (577 saves over 1 GB on 6 computers, 2026-10-27 to 11-03):
A cut try is tried again a minute later, then at doubling pauses. A sleep keeps trying for up to 30 minutes.
A start owed after a container goes takes an alarm of its own (from review).
gonenever starts at once. It owes the start (wake_after_sleep, kept until a start is made), marks it due now (retry_at_ms) and asks for an alarm at once. The next alarm, a fresh invocation, makes it, even after the wake's own hold ran out during the save.drive_with). A restart whose newest save won't restore goes on to the older one and still answers with the view once started again. The run is bounded bySTARTS_OWED_MAX=SAVES_KEPT+ 2: one owed start, one past a snapshot, one past each kept save. If that bound were ever reached, the alarm makes the next.RESTORE_MSnow also leaves the alarm's other events their minute and A computer's disk is read once it is ready and after each awake save, and its owner told as it fills #1018's disk read its 5 seconds, so it goes from 10.5 minutes to 9 min 25 s. That still restores a full disk's save: 17 GB needs 31 MB/s, and production measured 38.6 (5.7 GB in 147 s). An awake save, its unhold and the disk read after it have their own alarm check too.ALARM_OTHER_MSis reserved, not enforced, and the comment beside it now says so: the ledger'saskandtell_changedtake no bound of their own.STARTS_OWED_MAX, then a strike;Why not a bound that follows the save's progress. It would let no save run past the alarm's 15 minutes. All it adds is cutting a stalled save sooner, which is a separate issue: two 0.43 GB saves on production took 370 s where that computer's usual is 6–10 s.
What this does not fix. A save bigger or slower than one alarm allows can't fit, such as a full disk of incompressible data on a slow day. That needs saves that are not whole copies each time, which belongs to the storage design (FIN-375).
🤖 Generated with Claude Code