Skip to content

A save's backups get what a Durable Object's alarm leaves them - #1020

Merged
futurepaul merged 6 commits into
masterfrom
claude/saves-fit-the-alarm
Nov 3, 2026
Merged

futurepaul merged 6 commits into
masterfrom
claude/saves-fit-the-alarm

Conversation

@futurepaul

@futurepaul futurepaul commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

A computer's disk is now 20 GB (#1001), so a person can hold far more than before. The bound on a save's backups was 8 minutes, so a full disk's save would have started failing. This sizes that bound from the true ceiling instead.

The ceiling. A save runs inside the Durable Object alarm whose events asked for it:

  • an idle sleep's hold, save and stop run in one alarm;
  • an awake save's hold, save and unhold do too.

Cloudflare ends an alarm handler at 15 minutes (limits). So no save can run past that, whatever its bound says.

The change.

  • BACKUPS_MS = ALARM_WALL_MS (15 min) − ALARM_OTHER_MS (1 min for the alarm's other events) − the bounds of the sleep's other steps = 11 min 10 s (was 8). This mirrors how RESTORE_MS is sized from the start's deadline.
  • The lifecycle's SAVE_DEADLINE_MS goes from 10 to 13 minutes, so it stays past the cell's own bounds. It only catches a report its isolate lost.
  • An owner's sleep answers within 14 minutes, not 11 (docs/api.md). No client in the repo times that call out shorter: the shell and the CLI don't call it, and e2e calls it against fakes.

The arithmetic, checked as the cell compiles:

  • the sleep's steps plus the alarm's other events fit the 15-minute wall;
  • the backups get at least 11 minutes;
  • the sleep answers within 14 minutes.

Measured on production (577 saves over 1 GB on 6 computers, 2026-10-27 to 11-03):

  • saves moved 9 to 156 MB/s compressed (median 45, p5 15.4, p1 11.7);
  • a fresh computer uses 2.4 GB of its disk, so a 20 GB disk holds about 17 GB of a person's data;
  • the largest save showed about 1.6 to 1 compression, so that is about 10.8 GB saved.
a save of needs at 8 min tries cut needs at 11 min 10 s tries cut
8 GB 16.7 MB/s 6.6% 11.9 MB/s 1.2%
10.8 GB (full disk at 1.6:1) 22.5 MB/s 17.7% 16.1 MB/s 5.7%
17.3 GB (full disk, incompressible) 36.0 MB/s 36.9% 25.8 MB/s 22.7%

A cut try is tried again a minute later, then at doubling pauses. A sleep keeps trying for up to 30 minutes.

A start owed after a container goes takes an alarm of its own (from review).

  • The problem: a wake during an idle sleep's save made the next start in what was left of the same alarm. So did a restart asked during the save, a crash while wanted, and a start falling back past a bad save. With the sleep taking up to 14 minutes, that start would be cut mid-restore.
  • Now: the lifecycle's gone never starts at once. It owes the start (wake_after_sleep, kept until a start is made), marks it due now (retry_at_ms) and asks for an alarm at once. The next alarm, a fresh invocation, makes it, even after the wake's own hold ran out during the save.
  • Requests are unchanged: nothing ends a request while it waits, so a request makes each start due now, one after another, as each falls back to the next (drive_with). A restart whose newest save won't restore goes on to the older one and still answers with the view once started again. The run is bounded by STARTS_OWED_MAX = SAVES_KEPT + 2: one owed start, one past a snapshot, one past each kept save. If that bound were ever reached, the alarm makes the next.
  • An owner's sleep while asleep lets a start owed go, along with a failed start's retry and the earlier wake's hold, so neither its request nor the alarm starts it. A computer that's always on keeps its start, as after an owner's sleep of it awake.
  • Every start an alarm makes fits that alarm, including a failed start's retry, which had no slack before. RESTORE_MS now also leaves the alarm's other events their minute and A computer's disk is read once it is ready and after each awake save, and its owner told as it fills #1018's disk read its 5 seconds, so it goes from 10.5 minutes to 9 min 25 s. That still restores a full disk's save: 17 GB needs 31 MB/s, and production measured 38.6 (5.7 GB in 147 s). An awake save, its unhold and the disk read after it have their own alarm check too.
  • ALARM_OTHER_MS is reserved, not enforced, and the comment beside it now says so: the ledger's ask and tell_changed take no bound of their own.
  • Tests:
    • an idle sleep whose save runs 11 minutes, with a presence wake whose hold runs out during it. The sleep's end makes no start and asks for an alarm at once; that alarm makes the start, once. A request that comes first makes it instead;
    • a restart whose newest save won't restore makes both starts in its request and ends awake on the older save;
    • the longest fallback chain is exactly STARTS_OWED_MAX, then a strike;
    • an owner's sleep before the owed start leaves nothing to start, and cancels a failed start's retry;
    • tests that expected the start in the same step now take it from the alarm's step.

Why not a bound that follows the save's progress. It would let no save run past the alarm's 15 minutes. All it adds is cutting a stalled save sooner, which is a separate issue: two 0.43 GB saves on production took 370 s where that computer's usual is 6–10 s.

What this does not fix. A save bigger or slower than one alarm allows can't fit, such as a full disk of incompressible data on a slow day. That needs saves that are not whole copies each time, which belongs to the storage design (FIN-375).

🤖 Generated with Claude Code

A save runs inside the alarm whose events asked for it (an idle sleep's
hold, save and stop; an awake save's hold, save and unhold), and
Cloudflare ends an alarm at 15 minutes. So the bound on a save's backups
is sized from that wall, as a start's restore is from its deadline:
15 minutes, less a minute for the alarm's other events and the bounds of
the sleep's other steps, is 11 minutes 10 seconds (it was 8).

Measured on production (577 saves over 1 GB, 2026-10-27 to 11-03): saves
moved 9 to 156 MB/s, median 45. A 20 GB disk holds about 17 GB of a
person's data, about 10.8 GB saved at the 1.6 to 1 the largest save
showed. At 8 minutes, 18% of such a save's tries would be cut; at
11 minutes 10 seconds, 6%, and each is tried again. Nothing longer
is possible inside an alarm, so a bound that follows the save's progress
would let no save run longer (it would only cut a stalled one sooner,
which is not this risk).

The lifecycle's save deadline, which only catches a report its isolate
lost, goes from 10 to 13 minutes to stay past the cell's own bounds, and
an owner's sleep now answers within 14 minutes (docs/api.md). The
arithmetic is checked as the cell compiles: the sleep's steps plus the
alarm's other events fit the wall, the backups get at least 11 minutes,
and the sleep answers within 14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@futurepaul futurepaul left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review @ 22eff52: CHANGES

  1. [P1] The alarm budget omits the start that can follow the sleep. cell/src/computer.rs:200 budgets through Stop, but a record/tab/owner wake arriving while that sleep saves sets wake_after_sleep; Lifecycle::gone then emits Action::Start on Asleep. drive_events immediately awaits that start in the original alarm (cell/src/computer.rs:926, :976). A no-wait probe against this head confirmed a permitted sleep finishing at +680 seconds emits Start { generation: 2 }; a four-minute restore then needs +920 seconds, beyond the 900-second alarm wall, despite fitting its own start deadline. The handler can be cut during restore, before it reports Ready, with its lifecycle still Starting. Give the follow-on start a fresh invocation with a durable handoff, rather than spending the remainder of the sleep's alarm on it, and add a deterministic long-save-plus-wake test that checks this invocation boundary.

Non-blocking:

  • cell/src/computer.rs:188: ALARM_OTHER_MS is reserved headroom, not an enforced bound. For example, try_flush_awake awaits ledger::ask without a timeout. The compile-time assertion proves the allocated durations add up, not that these ancillary calls finish within the reserved minute; describe that limit explicitly or enforce a shared elapsed-time budget.

Checked the 55 core computer tests (all passed), shared backup timeout propagation, unchanged save-failure/backoff paths, and callers for stale 8/11-minute assumptions. No Docker/e2e run.

futurepaul and others added 2 commits November 3, 2026 14:09
Review of #1020: a wake during an idle sleep's save (or a restart, or a
crash while wanted, or a start falling back past a bad save) made the
next start in what was left of the same alarm. An alarm ends at 15
minutes, and a sleep's steps can now take 14 of them, so that start
would be cut mid-restore and left starting.

The lifecycle no longer starts at once when a container goes (`gone`).
It owes the start (`wake_after_sleep`, now kept until a start is made),
marks it due now (`retry_at_ms`) and asks for an alarm at once. The next
alarm is an invocation of its own and makes it, even if the wake's hold
ran out during the save. A request, which nothing ends while it waits,
makes a due start at once (`drive_with`), so a restart still answers
with the view once started again.

Every start an alarm makes now fits that alarm: `RESTORE_MS` also leaves
the alarm's other events their minute (10.5 to 9.5 minutes). That still
restores a full disk's save, 17 GB at 30 MB/s against the 38.6 measured,
and covers a failed start's retry too, which had no slack before. Checked
as the cell compiles.

`ALARM_OTHER_MS` says plainly that it is reserved, not enforced: the
ledger's `ask` and `tell_changed` take no bound of their own.

Test: an idle sleep whose save runs eleven minutes, with a presence wake
whose hold runs out during it. The sleep's end makes no start but asks
for an alarm at once, and that alarm makes the start once. A request
that comes first makes it instead. Tests that expected the start in the
same step now take it from the alarm's step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@futurepaul futurepaul left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review @ 626340c: CHANGES

  1. [P2] A request follows only one owed start, so Restart can return before fallback finishes. cell/src/computer.rs:844 calls the follow-up drive_events(Alarm) only once. For a Restart, the first drive finishes the sleep; the follow-up starts the image, but if its save is unusable and an older one exists, RestoreFailed leaves another start due. The request then returns an asleep view instead of waiting for the older save's start, unlike the previous behavior and the stated request contract. Drain immediately due starts on request invocations until none remains (with the existing bounded fallback policy), while keeping alarm invocations limited to their own start. Add a request-path regression with an unusable first restore and a usable older save.

  2. [P2] An owner's Sleep no longer cancels a start owed after the stop. The new deferred state produced at crates/core/src/computer.rs:963 is Asleep with wake_after_sleep and retry_at_ms set, but Event::Sleep in that phase still does nothing (:1279). If Sleep arrives after the prior stop and before its alarm, drive_with sees the start still due and starts the computer inside the Sleep request itself. Clear the pending wake/retry and the prior wake's hold when the owner sleeps an asleep computer; test that neither the request's follow-up nor a replayed alarm starts it.

The original same-alarm P1 is addressed. All 57 core computer tests passed; additional no-wait probes confirmed that the owed start survives serialization, expired presence, and late Asleep replay, and is consumed once, while reproducing both request-side cases above. The restore budget is now 570 seconds; ancillary-call headroom is explicitly documented as reserved rather than enforced. Docker/e2e were not run.

futurepaul and others added 3 commits November 3, 2026 14:43
…e go

Review of #1020:

1. A request made only one owed start. When a restart's start found its
   save unusable and fell back to an older one, the next start was left
   due, and the request answered with an asleep view. A request now makes
   each start that is due now, one after another, until none is.
   `STARTS_OWED_MAX` (new in core) bounds them: one owed after the
   container went, one past a snapshot, and one past each kept save. If
   the bound were ever reached, the alarm makes the next and the cell
   logs it. Alarms still make only their own start.

2. An owner's sleep of an asleep computer did nothing, so a start owed
   after a sleep that a wake came during was made inside the sleep
   request. Now that sleep lets go of the owed start, a failed start's
   retry, and the earlier wake's hold. A computer that is always on keeps
   its start, as after an owner's sleep of it awake. The lifecycle
   simulation holds this, since an asleep, wanted computer always has a
   start due.

Tests: a restart whose newest save won't restore goes on to the older
one in the same request and ends awake on it, after 2 starts. The
longest chain the fallbacks allow is exactly `STARTS_OWED_MAX`, then a
strike. An owner's sleep before the owed start leaves nothing for the
request's follow-up or the replayed alarm to start, and it cancels a
failed start's retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1018 reads a computer's disk once a start is ready, and after each
awake save, inside the same invocation (`DISK_READ_MS`, 5 s). So a start
an alarm makes must leave room for that read too. The merge of master
carries the cell's side of this in its conflict resolution:
- `RESTORE_MS` also leaves the disk's read, so it goes to 9 minutes 25
  seconds. That still restores a full disk's 17 GB at 31 MB/s, against
  the 38.6 measured.
- The start's alarm check counts the read.
- An awake save's hold, save, unhold and read now have an alarm check
  of their own (820 s with the alarm's other events, of 900).

This commit updates the core doc and computers.md to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@futurepaul futurepaul left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review @ 9846dca: PASS

Both findings are addressed: requests drain immediately due starts within the five-start fallback bound, while alarms retain the separate-invocation handoff; owner Sleep cancels the pending start for ordinary computers and preserves it for always-on computers. The merge keeps Ready-before-disk-read ordering and counts the read in both budgets: restore gets 565 seconds, and an awake save totals 820 seconds plus 60 seconds of reserved headroom (880 of 900). All 62 core computer tests passed, including the longest fallback chain; additional no-wait probes verified ordinary and always-on cancellation behavior across serialization and alarm replay without duplicate starts; Docker/e2e were not run.

@futurepaul
futurepaul added this pull request to the merge queue Nov 3, 2026
Merged via the queue into master with commit 0efe9a8 Nov 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant