fix(manager)!: leave managed agents running on a stack stop - #1301
fix(manager)!: leave managed agents running on a stack stop#1301davidfarah2003 wants to merge 17 commits into
Conversation
Bare Manager.stop() and cotal down no longer reap live seats. The previous
reap is stop({ withAgents: true }) / cotal down --with-agents. A live
manager that cannot list seats is refused. Leftover seats are printed
after a spare down.
Refs #964. A PTY child may still die when the manager process exits and
the PTY master closes; that survival is not claimed here.
Review — BLOCKHead: Verdict: BLOCK — the PR breaks its own test suite at this head.A stale in-process test still asserts the OLD destructive default, was not updated with the flip, and is not in the diff: That cell (comment: "Regression: active-mode stop remains the existing destructive shutdown path") calls bare
Independently built reap-path enumeration (built from the other end, then checked against the diff)Method: grepped every call that ends a managed agent, independent of the diff —
No hidden mass-reap path found. Findings per numbered questionPrimary (reap-path completeness): complete for 1. Smoke teardown leakage — real, and no mechanism guards it. Many in-process suites spawn real pty seats and tear down with bare 2. Mutation fixtures — sound. All 6 anchors resolve exactly once at this head; all 3. Docs + bundle — clean. 4. The refusal is reachable, correct, covered. 5. Design contradiction resolved as "refused loud." 6. 7. No overclaim. PR body states both caveats (PTY child may die on manager-process exit; successor 8. 9. Changeset level justified. Standards / what I could not check
Worktree left clean. |
Merge hold at
|
Correction to my review — the
|
|
Two of the findings above sharpen on inspection, and both sharpen in a useful direction — each turns Finding 4: the manager already has the mechanism, and this path opts out of it
So The useful part: Neither function is touched by this diff (0 hunks against the merge-base), so this is a pre-existing Finding 8: the leakage is not hypothetical on the machine this runs onRefined inventory: 24 suites construct a Manager, populate slots and call a bare Two of the 24 are concrete rather than theoretical, and neither is in this diff:
Both bare-stop, both swallow the result. After this change they leave live PTY children behind, and This does not argue against the change — the default is right. It argues that flipping a default which |
Finding 5 has two concrete instances outside the five docs pages, and an adjudicationBoth verified at this head; neither file is in the diff.
Adjudication: the two reviews do not actually disagree about the enumerationOne review reports the reap-path enumeration COMPLETE. The other then found
I checked which of the two the new code guards: Worth noting for whoever fixes that later, not here: at This does not block this PR. It is |
BLOCK —
|
Reading this PR's checks: they are present, and they are stale#1301 is CONFLICTING against main, on one file — Two consequences worth stating explicitly, because they pull in opposite directions: The runs currently showing on this PR were built before the base moved, so their conclusions That does not make the red ones go away. Every failure cited in the findings above was
So: a green here is worthless until the conflict is resolved, and a red here is real. And after the conflict is resolved, expect no runs at all until it is. While a PR conflicts, |
Plain Manager.stop leaves managed seats; stop({ withAgents: true }) is the reap. Live-PTY smokes must not spare-stop except named coverage. Spare down always signals and lists seats as honesty, including --with-agents --dry-run.
…gents # Conflicts: # extensions/connector-core/src/docs-bundle.generated.ts
The planted control is matched by regex, never imported, but bin's smoke tsconfig still typechecks it. Import Manager from the workspace package. README and embedding.md now name the spare stop default.
Re-review of the repaired head — BLOCK (one new red gate), but the prior grounds are resolvedHead: CI presence at the full sha (checked before conclusions): all four required workflows PRESENT by name — Docs Verdict: BLOCK — a new self-inflicted typecheck failure, introduced by the follow-up commit.
The new planted-control fixture The prior BLOCK grounds are genuinely fixed — credit where due
New findings the repair created (none blocking on its own; F1 above is the blocker)F2 — the spare-guard is a real mechanism, but a name-allowlist heuristic, not airtight. This is the direct answer to "does it catch a future author, or only the fixed suites?" It catches a future author: it walks all 613 smoke sources, flags any F3 — F4 — F5 — README and embedding docs still describe the old behavior (confirmed; not in the diff). Standards
What I could not check
Bottom line: the design change is sound and the two prior BLOCK grounds are resolved. The delta earns a BLOCK only on F1, the planted-fixture typecheck failure — a one-line path fix. F2–F5 are worth addressing but do not require a block on their own. |
Correction to F3 — its mechanism was overstated; the finding narrows (and survives)In my delta re-review I wrote that So the The finding narrows to two real things, both verified in code at this head:
Net: F3 is not "external runtimes break today" — it is "a disposition was coupled to a proof requirement, changing hard-despawn behaviour on unreviewed paths and making an optional adapter capability load-bearing." The fix is to decouple, passing |
The setup finale printed "stop everything" for `cotal down` on both the default and --demo paths. Bare down leaves managed agents running.
|
BLOCK on The original safety defects are substantially repaired: bare stop preservation and explicit-reap counterparts pass, hard-stop replies now require authoritative exit, dry-run is supported, the teardown sentinel is wired into CI, and the stale README, embedding, generated bundle, and first-run setup claims are corrected. One destructive false-success remains. In That is acceptable for bare Suggested shape: retain warning-only behavior for bare down and dry-run, but remember a failed list/reap proof under non-dry-run Validation at ancestor heads remains applicable because the reviewed SHAs are ancestors. |
Re-review of the settled head — BLOCK on one ground (F4), everything else is resolved or an approvable residualHead: CI presence at the full sha (checked before conclusions): all four required workflows PRESENT by name — Docs Everything from the prior passes is resolved — re-verified at this head, not read off a run
The single BLOCK ground: F4 —
|
|
Follow-up direct reproduction at the same SHA strengthens the block. I invoked the exported |
|
One final integration observation closes the remaining gap in the reproduction: the same exported |
Correction and escalation to my verdict — the BLOCK is broader than F4, and the refusal removal did not actually deliverI posted my last verdict while Correction 1 — F3 is a FAILING TEST, not an approvable residual. Retract "non-blocking".
The suite declares a local mirror of the signature — three params, synchronous, I graded F3 "no in-tree regression". That was true of the four runtime handles I checked and false of this signature-mirror suite — a mirror is invisible to an import search, and Correction 2 — the refusal removal did not deliver its own central claim. The unstoppable-manager defect is NOT fixed.
So at this head, This revises what I told you last pass. I graded the refusal removal "the right trade" on the reasoning that the CLI listing was only honesty and the non-reap guarantee lives in Third failing gate —
|
Listing and reap throw on an unreachable broker so a live manager is not stranded. --with-agents that cannot list seats reaps none, still stops the stack, and fails the process. Mass reap sends waitForExit instead of deriving proof from graceful. Refs #964.
The unreaped cell now prints its completion marker after an assertion throw. The down-manager cell fails if the process.exit stub ran, so restoring connectOrExit cannot be caught as a listing error.
| const broker = spawn("nats-server", ["-p", String(port), "-js", "-sd", store], { stdio: "ignore" }); | ||
| spawnedChildren.push(broker); | ||
| let up = false; | ||
| for (let i = 0; i < 100 && !up; i++) { |
|
BLOCK on Three repair grounds validate. One blocking regression remains in ground 1. Exact-head CI shard 0 failed This is not an inferred or stale red. The exact-head job log shows the registry assertion fail after the manager EPERM path, then reports 92 of 134 partitioned smokes never ran. Local Suggested fix: listing/reap need a throwing control resolution that does not prune the selected mesh record as a side effect. Preserve the registry until the normal successful whole-stack cleanup removes it; a failed stop must retain it. Keep the preservation sites unchanged as documented. Add or retain a non-root-capable regression cell binding failed listing plus failed manager stop to registry preservation. Exact-SHA workflows present: Docs success; Code Quality success; Windows is active with required and shards 1/3 already successful; Mutation reproof active; CI active with shard 0 already failed, shards 1/2/3 and unit active, live queued. Pending/in-progress are unmeasured. Mutation was dispatched at 19:47:54Z, so it has the fresh 150-minute budget. |
Independent review —
|
Adjudication — the registry-prune finding at
|
| recorded origin | entry after failed-listing down |
|---|---|
up |
DELETED |
| default (no origin) | DELETED |
manual |
SURVIVED |
manual is protected by pruneMesh (mesh-registry.ts:256: an operator-registered record is never auto-pruned — the remote-mesh / VPN-blip protection). up and default are deleted. The stop itself is pidfile-based (stopLocalProcess, registry-independent), so it runs whether or not the entry survives — and it can fail (EPERM on the kill, the marker-contention throws at down.ts:424, the very domain shard 0's clean exercises). The two-failure case is therefore reachable: unreachable-broker listing prunes the entry, then a failing stop leaves an operator with a half-stopped stack and, for an up/default mesh, no registry entry to reconnect or diagnose with until the broker returns and cotal up rewrites it.
Severity, without softening
- Real for
up/default meshes, void formanual. The most dangerous case (a manual record for a mesh on another machine, unreconstructible locally) is safe by design. - The lost
upentry is recreatable bycotal up— but only once the broker is reachable again, which is exactly not the case in the window this fires. - The PR does not assert entry survival in
down-target.smoke.ts, does not re-record after the prune, and the PR body does not mention the interaction. So it is undisclosed and untested, not a documented trade.
This is not a data-loss-forever bug, and it does not touch the safety property I verified (the manager still stops; the process no longer exits). It is a recovery-evidence regression on a newly-reachable path, and it is the kind of thing that is invisible until someone looks after the catch — which is what the second seat did.
Disposition
BLOCK on this ground. Minimal repair options, any one sufficient: (a) make the listing preflight non-pruning (pass a probe variant that classifies unreachable without prune, since listing is best-effort honesty and has no business deleting durable state); or (b) re-record the entry after a prune when the stop is about to proceed; or (c) at minimum, scope the prune to not fire on the down listing path and cover it with a cell that asserts the up-origin entry survives a failed-listing down. I did not run mutation-proof and made no code change; the probe was scratch and has been removed. Worktree clean at f6f2d33e.
My earlier block-lift on the four original grounds is unchanged and not withdrawn. I now hold a block on this fifth, independent ground. Two seats, one sha, explicit disagreement reduced to a table: the entry is deleted for up/default, preserved for manual, and the path is new to this PR.
Seat listing is honesty, not a stale-registry sweep. Match recorded roots canonical-root-wise, and restore the entry if preflight deleted it, so a failed dependent still blocks later cleanup.
Fifth-ground fix review —
|
| recorded origin | before | after fix |
|---|---|---|
up |
deleted | survives, restored as up |
| default (no origin) | deleted | survives, restored as default |
manual |
survived | survives (never pruned) |
Round-trip is faithful: recordMesh(mesh) re-serializes the exact loaded MeshEntry, so server/root/mode/origin are all preserved — no default-shaped record replacing an up-origin one (the quieter-regression I checked for does not occur). down-target 20/20; smoke:clean 70/70 (was 69, one new cell), with ✓ down: a failed dependent preserves the mesh registry and ✓ down: listing a failed dependent names the control plane, not a missing record both green. pnpm typecheck exit 0, Scope 27 of 28.
The three seams you flagged
Skip-connect rejection is correct. The down-with-agents-unreaped.json fixture is unchanged, and its mutation 2 mutates the listing connect back to connectOrExit expecting the process.exit stub to fire. If listing skipped the connect, that mutation would SURVIVE (nothing to exit). So listing must connect for the honesty-under-process.exit guarantee, and once it connects it prunes on an unreachable broker, so restore is the honest remaining option. The two guarantees genuinely coexist only because listing connects. Verified, not accepted.
Restore is faithful — settled above (origin preserved for all three kinds).
The restore window is real but far smaller than the defect it replaces. The on-disk delete is pruneMesh→removeMesh at connect.ts:511 (synchronous rm) inside the awaited connect; the restore is a synchronous recordMesh as the first statement of the catch. Between them: the throw unwinding plus one microtask hop, no I/O, no timer, no network await. A SIGINT would have to land in that sub-millisecond microtask gap to strand the entry, versus the original defect which deleted it for the entire remainder of down (including the multi-second stop) and permanently on a failed stop. Materially different in kind and duration. I record it as a residual, not a blocker: a crash-consistent write would need the delete and the re-add to be one atomic op, which is beyond the scope of this fix and not the property under review.
New mutation fixture is sound
down-failed-dependent-preserves-mesh.json (suite clean.smoke.ts): both find anchors resolve exactly once. Both expectRed cells exist verbatim and are driven by the real exported down({positionals:[],values:{},raw:[]}) at clean.smoke.ts:352.
- Mutation 2 (drop the restore) →
expectRed"a failed dependent preserves the mesh registry" (:364). Earlier cells :357 (broker not stopped) and :358 (names control plane) stay green under this mutant — only the registry cell notices. Earliest-red confirmed. Itswhystates it "goes red only if listing actually connected and pruned", which is exactly the skip-connect constraint. - Mutation 1 (raw
===root match) →expectRed"names the control plane, not a missing record" (:358). Under the mutant the canonically-recorded mesh is missed, listing never connects, and :364 stays green (nothing pruned); :358 is the first to notice. Earliest-red confirmed.
(Graded by reading; I did not runmutation-proof.)
On the shifted shard-0 cell (your caution)
The failing cell moved between heads: at 42e34534 it was the pid-1 EPERM signal cell; at f6f2d33e the EPERM cells are visibly green in the log (✓ down: a failed stop sets a failing exit code, ✓ down: a failed dependent prevents the broker stop) and the failure is the new registry cell (69→70). So the registry cell is not a regression from the F1 fix — it is a newly-added cell catching the real defect, consistent with unmasking (earlier red hid later suites). And I am not claiming the EPERM cell fixed by inference: it is run-and-green at f6f2d33 specifically. At the new head it is unmeasured (shard 0 in_progress).
Unmeasured — do not read as pass
CI keyed to 138e1a1b: Docs + Code Quality success; CI (all shards + live), Windows, Mutation reproof all in_progress/null. So shard-0 clean on this sha (the cell that carries this fix) and the mutation reproof are unmeasured now. The lane honestly did not claim otherwise.
Disposition
The fifth ground is cleared on the merits: the entry survives a failed-listing down, faithfully, and the design choice is justified. My four earlier grounds remain discharged. No further code change requested. Recommend merge once CI on this sha goes green, specifically shard-0 clean, Windows, and Mutation reproof reaching a conclusion; the residual restore-window seam is noted for the record and does not block. No code change made by me; probe was scratch and removed; worktree clean at 138e1a1b.
|
BLOCK on The runtime state fix is correct.
The block is the mutation fixture gate. Exact-head CI shard 0 fails This also corrects my previous causal citation: the deletion occurred in Required fix: replace all three Exact-SHA workflows present: Docs success, Windows success, Code Quality success; CI is active with live/unit success but shard 0 failed at mutation-fixtures and shards 1/2/3 active; Mutation reproof active. Active checks remain unmeasured. |
smoke:mutation-fixtures refuses a find that spans a comment. Move the listing comments out of the mutated windows so the three down anchors stay unique without being disarmed by a docs-only edit.
|
APPROVE on The anchor block is discharged without changing shipped behavior. After removing comments and whitespace, All four mutation The preceding runtime fix remains validated: at Named residuals remain outside this delta: process-exit survival and successor-terminalization are unproven; #1310 carries the conventional-name spare-guard residual. Exact-SHA workflows are present. Docs and Code Quality are green. CI is pending; Windows and Mutation reproof are queued, so their conclusions are unmeasured. |
Anchor-repair review —
|
Spare-down leftover listing interpolates manager process status and control-plane errors. The presence-render census now classifies those three sinks as non-render/control.
|
BLOCK on The product-surface approval transfers. The delta is fixture-only and The three new candidates are honestly outside the presence-progress contract. The two error templates are control-operation failure data. The seat row formatter is human-facing, but its The block is rationale quality. All three new entries carry the exact same text: Please replace the three rationales with candidate-specific statements, for example:
Exact local and PR SHAs match and the worktree is clean. Exact-head workflows are present: Docs green; CI queued; Windows has two smoke shards green with required/other shards active or queued; Mutation reproof active; Code Quality active. Active/queued conclusions remain unmeasured. The previous shard stopped at presence-render-census, leaving 15 of 132 partition smokes unrun. |
|
Mutation reproof failed at Ruled out against the base rather than argued. Main's own Mutation reproof run The mechanism is structural, and worth fixing rather than waiving. The fixture has three cells: D2a and D2b were both KILLED, each naming A single-cell file has no internal positive control, so this verdict is reachable on any diff that selects this fixture. It is a fixture-quality defect that will keep charging the reproof budget on unrelated PRs. The remedy is the one the runner names: add a mutation to Also in that run, and also not this PR's: Gate status: not merging while a required workflow reads failure. The exoneration explains the red, it does not clear it. Sequencing the fixture repair first, after which this head's reproof can be re-run on its own merits. |
A plain Manager.stop() leaves node-pty waitpid handles, so Linux CI shards hung after attach-auth-root and persona-role-capability passed.
DELTA review — APPROVEScope: this is a delta review of the five changed files at The delta flips four teardown §1 — Masking hazard (does anything still prove
|
A source-checkout attach without COTAL_SKIP_CONNECTOR_SEED now refuses the operator-global seed store instead of writing payloads, so the seed-skip mutant stayed WRONG-RED on a green named cell.
Resolve the four-file conflict set against 74f7eeb: - regenerate the docs bundle so it stamps 0.46.0 - manager cluster document revision 13 for optional waitForExit - recount the presence-render census and keep the M8 mutation at -1
Regenerate the docs bundle against d6a61b5. Contract revision and presence-render census are unchanged.
|
BLOCK on
The three previously unreviewed-file fixes otherwise pass my review. Ran:
Read:
Not run: attach-auth-root, mutation-proof, any build, any |
|
Acceptance-loop addendum for |
|
Whole-result evidence audit, still bound to live clean HEAD
Result: BLOCK is unchanged. The three target-file fixes have the strongest safe evidence available, and the census defect is reproduced through its real acceptance command. |
|
Cold review — adversarial coverage-integrity lens. Verdict: BLOCK at Graded SHA verified: Blocking findingUndocumented The receiver-name boundary in the same file is a fair contrast: it is documented in-file and measurement confirms it, so it is not a defect. The difference between a declared boundary and a silent filter is the whole finding. Repair (no census/validator/count change): either drop the spawn-shape prefilter so any file with Items verified clean
Method / RAN vs READ
|
|
Numeric correction to my BLOCK, reproduced first-hand and accepted. My earlier "~48 skipped" figure was 74 minus 26 — the count of skipped files containing |
…s census reasons Keep the spawn-shape gate but print every skipped live-PTY spare-stop, pin manager-deregister as the real-file control, and floor the examined set so a later shrink reds. Two already-examined teardowns from main pass withAgents so the visible hits check is green. Census totals and the M8 anchor are unchanged.
A new spawn-shape-skipped live-PTY spare-stop now reds instead of printing and exiting green. A stale frozen path reds so the 29-file baseline can only shrink.
|
Cold review — adversarial coverage-integrity lens. Verdict: APPROVE at The
|
|
APPROVE My rationale BLOCK is cleared. The three PR-added I also independently graded the repaired livePty ratchet. Ran:
Read/static:
Not run: mutation-proof, builds, install, CI observed, not graded green: docs passes; Analyze, Windows, smoke shards including shard 2, unit, live, and changed are pending. The merge gate must still require shard-2 guard output and changed's mutation selection/fatal verdicts to complete within 145 minutes. |
|
APPROVE at Both prior blocks are cleared. Ran
Read and checked
Not run CI |
Summary
Bare
cotal down/Manager.stop()no longer reap live managed seats. The previous reap iscotal down --with-agents/stop({ withAgents: true }).One 2026-08-26 teardown took six live seats, some with uncommitted work. The operator-visible default is now spare: leftover seats are listed (name, mode, pid, agent, cwd, status) as unmanaged OS processes, with
cotal down --with-agentsas the reap. Spare down always signals the manager; listing is honesty, never a refuse-to-signal. Listing and reap pass{ onRefusal: "throw" }intoresolveControlTargetso an unreachable broker is catchable instead ofprocess.exit(1), and the stop loop still runs.--with-agents --dry-runprints the seats that would be reaped and mutates nothing.--preserve-state --with-agentsremains refused.Preservation /
stopRetainedAgentsOnExitis unchanged. No new manager RPC.--with-agentssendswaitForExit: trueon each seatstopso proof of exit is not derived fromgraceful: false. Ordinarycotal stop/cotal_despawnstay acceptance-not-exit.--preserve-statekeeps the defaultconnectOrExitat its threeresolveControlTargetsites (retry, prepare, commit). A half-cut that cannot talk to the broker must abort rather than continue; that is not listing honesty.Listing seats matches recorded roots canonical-root-wise (
meshesForRoot). If preflight still deletes the entry as a stale unreachable broker, listing and reap restore it. A failed dependent must keep that record so later cleanup cannot proceed under a live process.Cluster revision
This PR claims manager cluster document revision 11, measured against
origin/main(still revision 10) at review time, not against this branch's merge base. Revision 11 is the optionalwaitForExitinput ondespawn/stop: a changed input contract, so a changed described surface, even though the command names are unchanged.#1302 also needs revision 11, for a different surface change (status output digest). If #1302 merges first, this PR must move to 12 and re-derive its digest against the merged surface. Do not treat 11 as surviving a later merge by number alone.
Named gaps
reconcileStaticLifecyclescan still terminalize durable slots with no live owner. Out of this PR.cotal down. A failedcotal upteardown SIGTERMs the manager without a seat snapshot.stop()still reaps will still reap on SIGTERM.--with-agentsthat cannot list seats (no recorded mesh, unreachable control plane, or a non-okps) still stops the stack, reaps none, and exits non-zero naming that no seats were reaped. It does not claim a completed reap.Because those remain, #964 stays open for the process-exit survival and successor-terminalize halves.
Follow-up: #1310 (spare-guard conventional-name allowlist).
Signature-mirror SET
The SET is one suite. A mirror is invisible to an import search, so the claim is the shape search: a local re-declaration of
despawnAuthorized(...). That search returns onlyimplementations/manager/smoke/renewal-terminal-race.smoke.tsplus the real method inimplementations/manager/src/manager.ts. Twelve manager smokes castmgr as unknown as { ... }; only that one redeclaresdespawnAuthorized.requireAuthoritativeExithas the same two-file set. That suite now awaitsPromise<{ ok: boolean }>.CI / smoke:clean
Shard 0 at
138e1a1b5ranpnpm smoke:cleantoCLEAN SMOKE OK (70 passed), includingdown: listing a failed dependent names the control plane, not a missing recordanddown: a failed dependent preserves the mesh registry. The shard then failed atpnpm smoke:mutation-fixtures: threefindanchors spanned comments (down-failed-dependent-preserves-mesh.json[1],down-with-agents-unreaped.json[0] and [1]).83ab72640re-anchors those on unique code-only windows (pnpm smoke:mutation-fixtureslocal OK, both mutation-proof fixtures 2/2 killed). CI green at83ab72640is still unmeasured.Tests
pnpm smoke:manager-stop-reap(authed broker + delivery daemon + real managed PTY seats): spare leaves seat A running and undeprovisioned; a second manager starts on a different root so it is not A's successor;stop({ withAgents: true })reaps seat B; seat A survives B's reap (18 cells).pnpm smoke:preserve-state: named spare cells plusstop({ withAgents: true })counterpart. Mutation-proofpreserve-state-spare.json: restoring always-reap reddensnormal stop leaves managed agents running.pnpm smoke:manager-stop-spare-guard: planted-control walk; live-PTY smokes must not spare-stop except named coverage.pnpm smoke:start-modelcell 9b: in-process spare path. Mutation-proof: restoring always-reap reddensspare: a plain stop does not hard-stop the child.pnpm smoke:lease-loss-keeps-servingCONTROL retargeted ontostop({ withAgents: true }); all 4 mutations killed.pnpm smoke:down-target: hermetic--with-agentscombination refusals, including--with-agents --dry-runallowed; live-broker unreaped cell (stack stops, exit 1, loud unreaped line);down manageragainstnats://127.0.0.1:1still SIGTERMs and does notprocess.exit. Mutation-proofdown-with-agents-unreaped.json(2/2 killed after the code-only re-anchor).pnpm smoke:clean: a failed dependent with a spelled-different recorded root still names the control plane (not a missing record) and keeps the mesh registry. Mutation-proofdown-failed-dependent-preserves-mesh.json(2/2 killed after the code-only re-anchor).pnpm smoke:mutation-fixtures: 0 dead / 0 ambiguous / 0 prose-spanning anchors locally at83ab72640.pnpm smoke:renewal-terminal-race: signature-mirror awaitsdespawnAuthorized.pnpm smoke:flag-inventory:with-agents:booleanondown.stop({ withAgents: true })(freeslot, int2-revoke-hold, attach-reconnect/stdin, cli-seat-locality, console-ws-duplex, lifecycle-e2e, manager-multi-live, manager-reconcile-startup, user-spawn F, probes).Refs #964