Skip to content

fix(ocap-kernel): record a vat's death in one synchronous step - #1093

Merged
sirtimid merged 5 commits into
mainfrom
sirtimid/retire-vat-synchronously
Oct 1, 2026
Merged

sirtimid merged 5 commits into
mainfrom
sirtimid/retire-vat-synchronously

Conversation

@sirtimid

@sirtimid sirtimid commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

A vat's death is four store writes: reject the promises it was deciding, unpin its root, drop its config and vat store, set the terminated mark. They were spread across VatManager.stopVat and VatHandle.terminate, with awaits between them, so a crank could land in the middle and see a half-dead vat.

A new private #retireVat does all four in one synchronous step, before the worker is touched. The worker is torn down even if a write fails, and the caller hears about the store failure rather than a channel that would not close.

Having the death in one place leads to these changes:

  • stopVat no longer needs a live handle to terminate. A failed relaunch leaves a vat persisted but not running. terminateVat threw VatNotFoundError for one, so it could not be retired at all, and terminateSubcluster stopped part-way on reaching one. A vat that is neither running nor persisted still throws.
  • A worker whose launch fails is stopped. platformServices.launch spawns the worker before the handshake, and a failed handshake leaves no handle to stop it through. runVat now stops it, so initializeAllVats and restartVat no longer strand workers (only launchVat cleaned up before). A launch that failed before a worker existed no longer asks the runtime to stop one, which logged a misleading second error.
  • VatHandle.terminate only does the handle's own work: reject pending commands, then close the channel.
  • removeVatFromSubcluster tolerates a vat in no subcluster, so deleteVat can be retried.

A channel that fails on its own

The drain's catch in VatHandle used to call terminate(true, …) and get the store teardown for free. With that moved out, the handle takes a required onStreamFailure. It rejects its own pending commands, and VatManager.#retireLostVat retires the vat the same way a requested termination would. Two details matter:

  • It waits for the current crank. Run inside a delivery savepoint, an aborting crank would roll the retirement back while the handle stays dropped from #vats.
  • It matches the report to the handle that made it, not the vat id. A restart puts a new handle under the same id, and a vat the kernel is stopping also breaks its own drain. Before this PR that second teardown ran in full, rejecting the decider promises and calling deleteVat twice.

There is nobody to retry this path, so a store failure ends in a log line saying to terminateVat by hand.

"Channel fails" is narrower than it sounds. Nothing listens for a worker that dies outright (NodeWorkerReader registers only 'message', and PlatformServices.launch drops 'error'/'exit' once the worker is online), so the kernel only sees a worker that is still sending bad frames. Wiring up 'exit'/'error' is left for later.

Write order

The mark goes after the rejections and before deleteVat:

  • After the rejections, because the mark schedules nextTerminatedVatCleanup, which deletes the decider promises' c-list entries assuming they were already rejected. stopVat records a vat's death before killing its worker, so a failed record strands a live worker #1067 asked for the mark in a finally; all three review agents showed that marking after failed rejections hangs those promises' subscribers for good.
  • Before deleteVat, because deleteVat drops the vatConfig row that a retry uses to find the vat. With the mark last, a failure on that write left a vat that could never be marked.

As a result, every failure leaves either the mark set or the row that allows a retry. The idempotence guard covers only the writes that must happen once, and deleteVat reruns. deleteVat itself now drops vatConfig last, so a failure on the vat store or subcluster write leaves the row a retry needs. If deleteVat itself throws, the vat is marked and its vatConfig row survives. cleanupTerminatedVat now discards that row, so the sweep cannot bring the vat back to life.

Synchronous is not atomic. terminateVat runs between cranks with no open transaction, so the four writes are four commits. This PR removes the interleaving and the dead ends, not the partial-failure window. Closing that window is the Tier 3 question of routing control-plane termination through the run loop.

Superseding #1030

Supersedes #1030 by @grypez, which fixed the same "persisted but not running" defect with a dedicated #retirePersistedVat branch. This PR routes it through the one path instead. Carried over: his defect narrative, his removeVatFromSubcluster change and test, and his persisted-but-not-running test cases.

Changes

  • VatManager.#retireVat records a vat's death in one synchronous step
  • stopVat tolerates a missing handle when terminating, and reports the store failure over the channel failure
  • VatHandle.terminate keeps only the handle's own work; a new required onStreamFailure reports a failed channel
  • VatManager.#retireLostVat retires such a vat between cranks, ignoring reports from a handle it no longer keeps
  • runVat stops the worker a failed handshake leaves behind; launchVat drops its own teardown and its now-redundant trailing mark
  • cleanupTerminatedVat discards the vat's vatConfig row
  • deleteVat drops the vatConfig row last
  • Kernel's in-crank termination lambda drops its now-redundant trailing mark
  • removeVatFromSubcluster tolerates a vat in no subcluster

Testing

VatManager.test.ts:

  • All four writes land in stopVat's synchronous prefix. They are read out of the mocks before the returned promise is awaited, so the test checks that no yield point comes between them, not just their order.
  • A failure before the mark leaves the vat unmarked. A failure on deleteVat leaves it marked, and a retry finishes the job without repeating the one-time writes.
  • The store failure wins over a channel failure.
  • A failed handshake stops the worker; a launch that never had a worker does not try to.
  • A reported channel failure retires the vat with the channel's error and waits for the open crank first. It is ignored after the kernel has stopped the vat or a restart has replaced its handle, and a failed retirement logs rather than rejects.
  • @grypez's five persisted-but-not-running cases.

VatHandle.test.ts checks that pending commands are rejected before the channel closes, using a stream whose end never settles. It also checks that a failed channel is reported with its commands already rejected, and that a restart leaves them alone.

store/index.test.ts checks that a deleteVat failing on the vat store keeps the vatConfig row, and that a second call finishes.

kernel-test's vat-lifecycle.test.ts adds two cases against a real kernel and SQLite store:

  • No vatConfig record survives termination and cleanup.
  • A vat whose worker emits a frame the reader rejects is fully retired, using the real NodejsPlatformServices trigger.

Each fix was mutation-checked on its own, and each mutation failed only its own cases.

Full @metamask/ocap-kernel suite green. @ocap/kernel-test's vat-lifecycle, subclusters, persistence, garbage-collection, resume and orphaned-ephemeral-exo were green against a fresh dist before the rebase.

Closes #1063
Closes #1067

🤖 Generated with Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c9cc6fb. Configure here.

Comment thread packages/ocap-kernel/src/vats/VatManager.ts
Comment thread packages/ocap-kernel/src/vats/VatHandle.ts
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 73.34%
⬆️ +0.07%
9973 / 13598
🔵 Statements 73.13%
⬆️ +0.06%
10102 / 13813
🔵 Functions 73.75%
⬆️ +0.03%
2332 / 3162
🔵 Branches 67.72%
⬆️ +0.11%
4115 / 6076
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
packages/ocap-kernel/src/Kernel.ts 89.84%
⬇️ -0.08%
79.54%
🟰 ±0%
85.41%
🟰 ±0%
89.84%
⬇️ -0.08%
322-324, 395, 419, 494-504, 592, 660, 736-739, 752, 762-763, 816, 839
packages/ocap-kernel/src/store/index.ts 98.87%
🟰 ±0%
95.23%
🟰 ±0%
100%
🟰 ±0%
98.86%
🟰 ±0%
416
packages/ocap-kernel/src/store/methods/subclusters.ts 98.82%
⬆️ +0.02%
90.62%
⬆️ +0.62%
96.15%
🟰 ±0%
98.78%
⬆️ +0.02%
263
packages/ocap-kernel/src/store/methods/vat.ts 98.51%
⬆️ +0.01%
90%
🟰 ±0%
100%
🟰 ±0%
98.5%
⬆️ +0.01%
308-309
packages/ocap-kernel/src/vats/VatHandle.ts 90%
⬇️ -0.14%
86.66%
🟰 ±0%
100%
🟰 ±0%
90%
⬇️ -0.14%
176-179, 375-380, 389-395
packages/ocap-kernel/src/vats/VatManager.ts 98.33%
⬇️ -1.67%
97.61%
⬇️ -2.39%
96.42%
⬇️ -3.58%
98.33%
⬇️ -1.67%
208-211, 238
Generated in workflow #5054 for commit 4dd8e8a by the Vitest Coverage Report Action

@rekmarks-consensys-1 rekmarks-consensys-1 self-assigned this Sep 25, 2026

@rekmarks-consensys-1 rekmarks-consensys-1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good except for two essay-length docstrings, resulting from—I suspect—a common Claudish failure mode where it dumps its train of thought into documentation instead of trying to protect the reader's attention.

Comment thread packages/ocap-kernel/src/vats/VatManager.ts Outdated
Comment thread packages/ocap-kernel/src/vats/VatManager.ts Outdated
sirtimid and others added 4 commits October 1, 2026 10:19
A vat's death is four writes: the promises it was deciding rejected, its root
unpinned, its config and store dropped, the terminated mark set. They were
spread across `stopVat` and `VatHandle.terminate` with awaits between them, so
a failure part-way left a state nothing recovers from. The sharpest is marked
terminated while `vatConfig` survives — only `deleteVat` removes that row,
while the cleanup the mark schedules sweeps `${vatId}.` keys, which never match
`vatConfig.${vatId}` — and such a vat reads as active again the moment cleanup
drops the mark.

`#retireVat` does all four with no await between them, before the worker is
touched, and the mark happens even if an earlier write throws. Killing the
worker is deliberately not part of it: that can fail, and a store that says the
vat is dead is worth more than one still waiting to find out. It runs in a
`finally`, so a record that fails cannot leave a worker running either.

`stopVat` no longer insists on a live handle when terminating, which is what
lets a vat left behind by a failed relaunch be retired at all, and `launchVat`
tears its worker down when the kernel-side registration fails.

`VatHandle.terminate` is now only the handle's own business: reject the callers
waiting on commands the worker will never answer, then close the channel.

Supersedes #1030 by @grypez.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mark makes the vat eligible for `nextTerminatedVatCleanup`, which deletes
its decider promises' c-list entries on the stated understanding that its
caller has already rejected them. Marking in a `finally` therefore turned a
failed rejection loop into subscribers that hang for good, and the retry that
would have finished the job hit the already-terminated guard and did nothing.
The mark is now simply the last step, so a failure leaves a vat that has not
been retired, with its c-list intact and the step available to be tried again.
`deleteVat` sits immediately before it: the two must not be separated.

`stopVat` no longer lets a channel that will not close stand in for a store
left half-written, and `launchVat`'s first catch stops the worker that
`platformServices.launch` has already spawned — `runVat` records no handle to
reach it by, so nothing else could.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moving the store side of a death out of `VatHandle.terminate` left the one
caller that had nothing else to fall back on: the catch on the handle's own
drain, which is how the kernel hears that a channel has failed. Left as it was,
such a vat kept its handle, its `vatConfig` row and the promises it was
deciding, so the kernel went on delivering over a channel that was gone and the
next boot brought it back.

The handle now takes an `onStreamFailure` and does the part that is its own —
rejecting the callers waiting on commands the worker will never answer, since a
crank blocked on one of them is what the retirement would wait behind — and
`VatManager` retires the vat exactly as a requested termination would. Between
cranks, like `terminateVat`: `#retireVat` writes synchronously, so run inside an
open delivery savepoint the whole death is undone by any crank that goes on to
abort, while the handle it dropped stays dropped. Reports are matched against
the handle that made them, since a restart puts a new handle under the same vat
id, and a vat the kernel is itself stopping reports the same way.

The terminated mark moves ahead of `deleteVat`, which drops the `vatConfig` row
`stopVat` reads to decide there is still a vat here to retire: a mark not
reached by then could never be set, and the c-list it gates never swept. Every
way a retirement can now fail leaves either the mark set, or the row that lets
the whole step be tried again — so the idempotence guard covers only the writes
that may happen once, and `deleteVat` runs again to carry an interrupted
retirement to the end. `cleanupTerminatedVat` discards the `vatConfig` row too,
so a vat marked but not discarded cannot read as active again once the sweep
drops its mark.

`runVat` stops a worker left over from a failed handshake, which `launchVat`
alone used to do and only for its own launches, and which asked both runtimes
to stop a worker they had never registered. Its unconditional
`markVatAsTerminated` goes: `stopVat` sets the mark early enough that asserting
it afterwards could only add it where `stopVat` stopped short of `deleteVat`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cannot say

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sirtimid
sirtimid force-pushed the sirtimid/retire-vat-synchronously branch from 95824dc to 2125267 Compare October 1, 2026 08:42
…be retried

`#stopVat` finds a handle-less vat by its `vatConfig` row, and `deleteVat`
dropped that row first. A failure on the vat store or subcluster write after
it left those behind for good: a retry threw `VatNotFoundError`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@rekmarks-consensys-1 rekmarks-consensys-1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sirtimid
sirtimid added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit c82c5ce Oct 1, 2026
27 checks passed
@sirtimid
sirtimid deleted the sirtimid/retire-vat-synchronously branch October 1, 2026 16:38
sirtimid added a commit that referenced this pull request Oct 6, 2026
#1100's change, ported onto #1098 and the stream-failure mechanism #1093
put on main.

- A channel that closes without an error counts as lost, as one that fails
  already did: a worker that exits closes the channel, and a worker thread
  emits no port event at all. The Node runtime keeps an `exit` listener once
  the worker is online, which closes the channel, and `terminate` tolerates a
  worker that has already gone.
- A lost vat's handle comes off the books at once, so #1098's lookup skips
  deliveries to it, and its death goes through the queued `terminateVat` in
  a crank of its own. A dead run loop gets the death recorded directly.
- `Kernel.stop` closes every channel through the platform services, so it
  tells the manager first rather than have that read as every vat dying.
- One vat that will not start no longer fails `initializeAllVats`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sirtimid added a commit that referenced this pull request Oct 6, 2026
The original commits were written against a #1097 from before #1093
landed on main. Their intent is ported onto #1098 in the second parent;
this commit records that tree over the old head so the branch moves
forward without a force-push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants