fix(manager): retry failed static reconciliation

[davidfarah2003]

Summary

• report startup and post-adoption static lifecycle reconciliation as an aggregate with attempted, succeeded, and failed counts plus greppable alias, phase, and disposition values
• expose current reconciliation state through manager status and cotal status --components
• re-drive failed coordinates in process through a fresh durable slot read, planStaticSlotResume, and the existing deterministic exact terminal
• bound each manager process to one initial attempt plus retries after 1s, 5s, and 30s, with keyed single-flight across the durable read and terminal entry
• add a real authenticated broker suite, CI fragment, mutation proofs, operator docs, and a patch changeset

Convergence and safety

The status snapshot is not authority. Before every re-drive the manager re-reads the slot from broker KV and runs the same planner again. The retry reuses retireOpId(lifecycleUid), so runStaticTerminal passes its own same-op frozen retirement and resumes the idempotent head transition. A different retirement op or another frozen operation is reported as refused-foreign and is never force-cleared.

The delays are scheduling only and never release a fence. The durable non-retired slot and exact-op state keep refusing alias reuse between attempts. No new lifecycle or second terminal operation is minted.

The 1s retry covers a one-shot control-plane or eviction blip. The 5s and 30s spacing gives a recovering broker more time without hammering a persistently unavailable authority. Four total entries are a finite per-process budget. A failure lasting beyond about 36 seconds becomes retry-exhausted; the alias stays unavailable and needs a manager restart. The next process receives a fresh budget because its counters are in memory and the durable row remains the sole authority.

recovered remains visible until the next static reconciliation sweep, then clears. The literal dispositions, next retry time, error, and remedy are visible in status and component health.

Scope of #774

This PR addresses the reported static-lifecycle reconciliation wedge: failed terminals now produce a failed aggregate, remain visible in current health, and receive bounded same-process re-drives. It does not implement general desired-fleet convergence from a last-applied user manifest or relaunch missing declared seats on manager boot. That broader boot-convergence scope remains open, so #774 stays open and this PR only references it.

Relationship to #1274

This change reports whether a reconciliation attempt is running, waiting, recovered, refused, or exhausted. It does not independently project whether static footprint cleanup completed. The durable cleanup marker and a general read verb remain #1274's scope.

Changeset

Patch. This adds recovery and operator visibility without changing lifecycle wire semantics. The fixed package group will version every package together; the changeset names the manager, CLI, and connector-core docs bundle that changed directly.

Verification

• pnpm typecheck completed across the 27 TypeScript workspace projects. The 28th project, @cotal-ai/example-04-frontier-faces, is JS-only and skipped by the recursive typecheck.
• pnpm smoke:manager-reconcile-redrive passed 30 checks against a real authenticated nats-server and in-process Manager. Its shutdown control gates the first exact terminal, proves stop() waits for it, proves the serial sweep does not start the later orphan, proves startup leaves no manager service registered after stop completes, and proves a later sweep is refused by the stop fence.
• pnpm smoke:manager-reconcile-startup passed 25 checks.
• The reanchored startup availability mutant was KILLED at its named cell, status serves while later orphan rows remain ACTIVE: the baseline completed with 27 progress marks, while the mutant reached 25 marks and failed that cell rather than an earlier one. Progress marks are ✓ matches across combined stdout and stderr, so these totals include the manager's own reconcile progress lines as well as suite cells. The earliest-red result also follows independently from the non-fail-fast check order: the only earlier cell still passes because the awaited sweep terminalizes orphan-0, while the named cell requires the status service to answer with the final orphan still active. The anchor targets the structural startupReconcile declaration and branch, not the catch message text.
• After merging current main, manager-service-contract.ts preserves both the custody field added on main and this PR’s staticReconciliation field.
• pnpm smoke:manager-service passed 25 checks.
• pnpm smoke:manager-service-invoke passed 12 checks, including generic resolution of manager cluster document revision 11 after the status output digest changed.
• The redrive mutation proof used a 34-mark green baseline and killed all five mutations at their named cells. The three shutdown mutants reached 31 or 32 marks and died at stop waits for an accepted startup reconciliation terminal, the shutdown fence prevents the serial sweep from starting a later terminal, and startup cannot publish the manager service after stop completes.
• The shipped-surface mutation proof used a 14-mark green baseline and killed revision 10 at static reconciliation status advances the manager cluster revision with 13 marks.
• node scripts/mutation-proof.mjs --config implementations/manager/smoke/fixtures/manager-reconcile-redrive.mutations.json killed both the aggregate-report and re-drive mutations at their named cells.
• A focused proof of reconcile-startup.json mutation 0 killed the serial-start mutant at status serves while later orphan rows remain ACTIVE, with 25 progress marks against a 27-mark green baseline. Progress marks are ✓ matches across combined stdout and stderr, so these totals include the manager's own reconcile progress lines as well as suite cells. The non-fail-fast check order and the passing earlier orphan-0 cell independently establish that this named cell is the earliest red. The anchor targets the structural startup reconciliation assignment and branch rather than the operator message text.
• pnpm smoke:mutation-fixtures checked 1,758 anchors with zero dead or ambiguous anchors.
• pnpm smoke:gate-inventory passed.
• pnpm smoke:ci-fragments passed.
• pnpm smoke:required-arg-seam passed after updating the attributed 119/88 to 124/92 call-site count.
• pnpm smoke:flag-inventory passed.
• pnpm check:docs-voice passed.
• pnpm check:docsbundle passed.
• pnpm changeset status reports the fixed package group patch plan.

pnpm smoke:component-health was attempted after its build. Under concurrent local validation it first reached the final delivery cell and observed the holder before its ready lease appeared, then a serial rerun reached the web cell after its subprocess exceeded the command timeout. The manager component cells passed locally, including the new static-reconciliation fallback and the distinction between present-not-serving and absent. Exact-head CI then completed all four Linux smoke shards green. Its mutation reproof ran component-health with a 17-mark green baseline and killed all four mutants at their named cells.

The broader changed-source mutation workflow did not complete: the configured 60-minute job limit cancelled it while it was still traversing the large manager.ts fixture intersection, before it reached reconcile-startup.json. Its log already contained unrelated timed-out, inconclusive, and ungradable fixture verdicts. The focused committed-head startup proof above is the issue-specific mutation result; the aggregate changed-source workflow is not claimed green.

Refs #774

[davidfarah2003]

Review — independent cold review at ec6f1f7b2425fdf3b075dbce75015f7ed67a3c03

Verdict: APPROVE (code). The one open item is a merge gate, not a code defect: two CI workflows were still in_progress at review time and must land green before merge (detail in Q8). Reviewed as the sole reviewer under the fleet CPU gate; depth was weighted over breadth.

Local HEAD and gh pr view 1302 --json headRefOid both read the sha above.

Q1 — the hand-resolved merge (manager-service-contract.ts)

PASS. Verified against both parents, not by reading the final file alone.
Head ec6f1f7b2 is itself the merge commit ("chore: merge main into reconciliation fix"). Parents: ^1 = bcd8fcb5a (feature branch, the #774 work) and ^2 = f3bad8772 (main, which carried #1294's custody seam); merge-base 5b19238f7.

• git diff <base|P1|P2|merge> on the required[] array shows the exact clean union:
  • base: [instanceId, runtime, agentCount, uptimeMs, connectors]
  • P1 (feat) added staticReconciliation; P2 (main) added custody
  • resolved: [instanceId, runtime, custody, agentCount, uptimeMs, connectors, staticReconciliation] — both additions survive.
• git diff -c <merge> -- <file> (combined diff, what the merge changed vs both parents) is empty → no evil-merge content: every resolved line traces to at least one parent, and nothing was deleted by the merge that neither parent deleted.
• Both required fields are actually populated on the single status path managerStatusData() (manager.ts:4972 custody: "legacy", :4976 staticReconciliation: this.staticReconciliationStatus()). No second status path omits either. The full staticReconciliation schema (state/lastSweep/failures) and its three interfaces survive intact.

Q2 — the mutation re-anchor (reconcile-startup.json, named cell "status serves while later orphan rows remain ACTIVE")

PASS (graded by reading + pnpm smoke:mutation-fixtures; mutation-proof was NOT run).

1. Anchor resolves exactly once and is code-only. The re-anchored find is the two-line code block const startupReconcile = … reconcileStaticLifecycles() : undefined; + if (startupReconcile) (manager.ts:1249-1250) — pure code, no comment or call signature spanned. pnpm smoke:mutation-fixtures reports reconcile-startup.json — 17 anchor(s) present and unique, and the specific first line matches exactly once.
2. The named cell is the EARLIEST red. The suite's check() is NOT fail-fast (it counts, never throws), and its only ✓-mark producer is a passing check, so "marks" = passing checks. Under the mutant (inserting await), manager.start() blocks registration until the whole sweep completes. The only earlier check (line 190, "a real orphan terminal began") polls the broker directly and still PASSES because the awaited sweep still terminalizes orphan-0. The named cell at line 213 uses an independent 20s until requiring status.ok && phase(last)==="active" simultaneously — impossible once the service registers only after every orphan is retired — so it is the first cell to redden. Fewer marks than baseline is consistent with dying there, and I traced that it dies there specifically, not upstream.
   • Note: the brief's "25 marks vs baseline 27" is a reporting imprecision — this suite has 25 check() calls, so a clean run is 25 marks, not 27. The fixture and suite are sound regardless; the delta direction (fewer marks) and the earliest-red trace are what matter.
3. Dependency AND reach. The fixture writes real durable ACTIVE orphan rows against a real JWT broker and drives a live Manager.start() (not a hand-built call to reconcileStaticLifecycles), then calls the registered status service. The mutant reddens because the overlap guarantee (serve-while-sweeping) genuinely breaks, reached through the real entry point.

Q3/Q4 — the +314 in manager.ts (idempotence)

PASS. Read as a design. It is a bounded, per-coordinate retry engine:

• Cannot run twice per coordinate: attemptStaticReconcile coalesces on item.flight and refuses while item.timer is armed; retryStaticReconcile dedups on item.retryFlight; the timer joins item.flight before consuming budget; scheduleStaticReconcileRetry won't arm a second timer.
• Never acts on the stale view: every retry does a fresh durable re-read (readStaticReconcileSlot) and refuses if the row vanished, changed actor/uid, or changed ownerInstanceId — "reconciliation never crosses incarnations."
• Never terminalizes a live owner's slot: adoption check (live agent at same uid → refused), cross-instance ownership left untouched (the historical all-agents-kill hazard), and the boot sweep defers an un-adopted active row to the post-adoption sweep while resumeRequired.
• Bounded + idempotent terminal: budget = 4 attempts (delays [1s,5s,30s]+initial), consumed even on read failure so a broker outage can't reschedule forever; the destructive terminal driveStaticRetirement uses a deterministic opId = retireOpId(lifecycleUid) under withLifecycleExecutor, re-reads the slot, and a foreign op is refused-foreign, never force-cleared. stop() sets staticReconcileStopping and clears all (unref'd) timers, so no post-stop double-fire.

Q5 — scope discipline

PASS. Body ends Refs #774 and states "#774 stays open and this PR only references it." Scan for any clos*/fix*/resolv* token immediately followed by #774 (negation-agnostic): no match. The body's "This PR closes the reported … wedge" has "closes" followed by prose, not #774, so GitHub's closing parser will not fire.

Q6 — changeset

JUSTIFIED. File declares patch for @cotal-ai/manager, @cotal-ai/cli, @cotal-ai/connector-core — the three packages with first-party changes. This is a bug fix with additive contract fields (no removal), so patch is the correct semantic level. pnpm changeset status shows all 21 packages bumped at minor; that is the fixed-group lockstep in a 0.x repo (per AGENTS.md), not a mismatch in the changeset.

Q7 — inventory sentinels

PASS, verified empirically.

• package.json gained smoke:manager-reconcile-redrive (matches the new suite file).
• The new ci-suites.d/*.txt golden is content-addressed: sha256("smoke:manager-reconcile-redrive") = 4fac1584…993a = the exact filename. Integrity confirmed.
• required-arg-seam sentinel moved sites 119→124, untypecheckedSites 88→92 with an attributing comment (+1 typechecked provisioner re-read, +4 smoke-side connections for the isolated broker fixture). pnpm smoke:required-arg-seam passes (253/0) and confirms exactly 124/92 by scan — the count changed because the code did.

Standards

• pnpm typecheck at this head: exit 0. JS-only @cotal-ai/example-04-frontier-faces has no .ts and no typecheck task — the reported 28th, legitimately skipped (Scope 27 of 28).
• CI at the full sha (workflow-runs API, name as a quantity separate from conclusion): 5 workflow runs present.
  • Docs — completed / success
  • Windows — completed / success
  • CI — in_progress (smoke shards 0-3/4 and changed still running) → unmeasured
  • Mutation reproof — in_progress → unmeasured
  • (also Code Quality: PR #1302 — success)
    This is no longer the "zero runs looks green" trap — real runs exist. But CI and Mutation reproof have not concluded; per this fleet's rule I treat in_progress as unmeasured, so merge must wait for both to land green.

What I could not check and why

• Did not run mutation-proof (forbidden — it writes a broken tree and restores with git checkout HEAD --). The fixture was graded by reading + smoke:mutation-fixtures.
• Did not run the live reconcile smoke suites end-to-end (CPU gate; and -live suites are forbidden); the redrive design was reviewed statically and the seam/fixture smokes were run targeted.
• CI and Mutation reproof conclusions are unknown (still in_progress at review time) — flagged above as the merge gate.

[davidfarah2003]

Addendum — withdrawing one Q2 sub-finding (the mark-count remark)

Correcting the durable record: my Q2 note claimed the brief's "25 marks vs baseline 27" was "a reporting imprecision" because "this suite has 25 check() calls, so a clean run is 25 marks, not 27." That inference is unsound; I withdraw it. The rest of Q2 stands.

The mechanism I missed. A "mark" is not a suite check() call. scripts/mutation-proof.mjs counts the pattern over the combined transcript:

• mutation-proof.mjs:130 — const output = \${r.stdout ?? ""}${r.stderr ?? ""}`;`
• mutation-proof.mjs:145 — const re = new RegExp(pattern ?? "✓", "gm");

reconcile-startup.json declares no progressPattern (only a completionMarker), so the counted pattern is the bare ✓ over stdout AND stderr from every source in the run — not the suite's cells. The suite drives a real in-process Manager.start(), and manager.ts prints ✓ lines to stderr on exactly this path: ✓ static reconcile completed: … (manager.ts:6864) and ✓ static reconcile recovered … (:6671, :6721), among others. So the suite's 25 check() sites are a floor; cells + manager output is the total. 25 cells + 2 manager ✓ lines = 27 is a coherent green baseline, and the author's published 27/25 figure is most likely correct.

What still holds: the earliest-red conclusion for the named cell status serves while later orphan rows remain ACTIVE. It was established without the counts — non-fail-fast checks, the only earlier cell (orphan-0 terminal began) still passes because the awaited sweep terminalizes it, and the named cell needs the status service to answer while the final orphan is still active, which is impossible once registration waits for the whole sweep. The anchor is structural (the startupReconcile assignment/branch), not the catch message text. The verdict is unchanged: APPROVE at ec6f1f7b2425fdf3b075dbce75015f7ed67a3c03.

General lesson for the next re-derivation: a progress counter that scans a whole transcript measures the transcript, not the test. When marks and cells disagree, the gap is usually the code under test printing in the same ✓ convention, not an error in the figure.

[davidfarah2003]

BLOCK on reviewed sha ec6f1f7b2425fdf3b075dbce75015f7ed67a3c03.

Independent review of that exact head. Verdict was formed before reading any other review comment on this PR. I disagree with the existing APPROVE on this sha: the two items below are product defects, not merge-gate weather.

The PR-owned patch at live head 913e9db67b2f5e875fa027fdab788ada8fafce5b is identical by stable git patch-id 18d1917929984ec172457b5debbad610b0da5dca (f3bad8772..ec6f1f7b2 equals 99d50f510..913e9db67; same 13-file set). This comment still names ec6f1f7b2 as the sha that was executed against.

1. Redrive engine (attacked, not only read)

Four attempts is real. STATIC_RECONCILE_MAX_ATTEMPTS = delays.length + 1 produced 4 exact-terminal entries in pnpm smoke:manager-reconcile-redrive (25/25). Duplicate retryStaticReconcile triggers joined one flight. Exhaustion stayed at 4 and did not busy-loop. A later same-process sweep did not reset the budget.

Fresh durable re-read is real on the retry path. driveStaticReconcileRetry always calls readStaticReconcileSlot before planStaticSlotResume. A vanished or moved slot becomes refused and does not cross incarnations. Foreign frozen retirement is refused-foreign and never reached eviction (same suite).

Live owner / sibling. planStaticSlotResume(active, adopted=true) is none. Unnamed suite pnpm smoke:manager-reconcile-ownership (author did not name it) passed 6/6: sibling ownerInstanceId rows stayed active; own and legacy dead-active rows retired.

BLOCK: stop() does not drain an in-flight sweep, and the serial loop never consults staticReconcileStopping. Timers are cleared. That is not enough. staticReconcileSweepsInFlight is a counter, not a lock, and stop() does not await it.

Adversarial probe against a real JWT nats-server and in-process Manager: two ACTIVE orphans, first eviction gated, start() in flight, then stop().

• stop() returned in 74ms with staticReconcileSweepsInFlight still 1.
• After releasing the gated eviction, the sweep continued and drove hold-last through driveStaticRetirement after stop() had returned.
• start() then continued and registered the manager service (manager service endpoint registered after the post-stop terminals).
• Failed items were left retry-scheduled with timer: false because scheduleStaticReconcileRetry bails on staticReconcileStopping.

That is work after shutdown, including a second exact-op terminal, plus boot continuing to re-register after stop(). A timer that fires after stop is guarded. An in-flight sweep is not.

2. Hand-merged status contract (coherence, not parent-text)

Both custody and staticReconciliation are required. managerStatusData() is the single emit path and always populates both. custody stays "legacy". This diff does not need another custody value.

Reached states in the redrive run: running/retrying while serving, retry-wait, recovered, failed (retry-exhausted, refused-foreign). idle is the empty map. No required field was missing on a served status in the suites I ran.

BLOCK: cluster document revision stayed at 10 while the status output digest changed.

• main: sha256:171c1ea6bd518efc106491a69624d6dd19bf3f1ca221a9293eb8cec4abaedbbd
• this head: sha256:fac47815f91955f9c721d88dbeca166850b2e7090fc7b98c6d11b887e28afd5d

The same file already treats a changed status output contract as a new described surface (revision 9: connectors). Mixed-version callers pin outputDigest. Revision 10 now names two incompatible status contracts. That is not saved by the fields being textually merged.

3. Suites

pnpm install --frozen-lockfile was run in this worktree (there was no node_modules at checkout).

Ran:

• pnpm typecheck: Scope 27 of 28. Skipped JS-only @cotal-ai/example-04-frontier-faces. Not a clean 28.
• pnpm smoke:manager-reconcile-redrive: 25/25
• pnpm smoke:manager-reconcile-startup: 25/25
• pnpm smoke:manager-service-invoke: 11/11
• pnpm smoke:manager-service: 25/25
• pnpm smoke:required-arg-seam: 253/253, attributed 124/92 held
• pnpm smoke:mutation-fixtures: 343 files, 1758 anchors, 0 dead / 0 ambiguous
• pnpm changeset status: this PR's changeset is patch. The reported minor is custody-seam.md already on the merged main side, not this patch file.

Unnamed suites I picked:

• pnpm smoke:manager-reconcile-ownership (6/6), because it is the live-owner / sibling-kill proof the redrive suite does not run.
• pnpm smoke:manager-deregister (30/30), because it is a real Manager.stop() path.

Also ran pnpm smoke:legacy-pty-custody: 4 passed, 1 failed (M1 red: legacy manager death ends the manager-owned counter PTY, child still live after SIGKILL). This PR does not touch that fixture or PTY adopt. Treated as environment flake, not a #774 block.

Did not run mutation-proof, cotal up/down, any *:live suite, or pnpm check.

4. Mutation mark count

Startup suite has 25 check() cells. The named expectRed cell status serves while later orphan rows remain ACTIVE is check 2, the first cell that can notice a serial await this.reconcileStaticLifecycles(). Check 1 still passes under that mutant. The suite is not fail-fast, but that cell is still the earliest one that notices.

mutation-proof.mjs counts ✓ matches over stdout+stderr combined (progressCount, default pattern ✓). A green startup run therefore prints 25 check ticks plus manager console.error lines that also contain ✓ (static reconcile completed and deregistered manager instance on this path). 25 cells plus those log lines is 27 marks. The author's original figure matches that counting rule. I did not run mutation-proof.

5. Scope and hygiene

Body ends with Refs #774. GraphQL closingIssuesReferences was empty at review time, and #774 was still open. The paragraph that says "This PR closes the reported static-lifecycle reconciliation wedge" does not put closes immediately before #774. Still a bad word to leave next to that issue.

Inventory sentinels: 119/88 -> 124/92 is attributed in required-arg-seam.smoke.ts and matched a live scan.

CI at ec6f1f7b2425fdf3b075dbce75015f7ed67a3c03

Presence by workflow name (separate from conclusion): CI, Docs, Mutation reproof, and Windows were each present.

Conclusions at review time are not a green gate. cancelled and in_progress are unmeasured. Docs had a completed success. The Mutation reproof run on this sha had job changed cancelled (60-minute cap on a dispatch before the 150-minute budget). Later CI/Windows runs on this sha were cancelled. I am not treating this sha as CI-green.

Could not check

• mutation-proof (forbidden here).
• Whether GitHub would still auto-close manager startup reconciliation reports N/N after a failed terminal and never re-drives it in-process #774 after a future body edit.

Worktree left clean. Reviewed sha remains ec6f1f7b2425fdf3b075dbce75015f7ed67a3c03.

[davidfarah2003]

Transfer proof for the current-main fold to ab9e52606207af6b96c62119843a11e29f4d475d:

• old base 99d50f5100616571fab5ecdf9b9552213fa163ba → old head 2a2fb7f0b8acc516f6b74dd238c5f6c7f0665790: stable patch ID ddc889156b2473cc0d7488eaa1058af6cff74baf
• new base 2528ec1f05570e58a08a4ac1341211a7062bb868 → new head ab9e52606207af6b96c62119843a11e29f4d475d: stable patch ID ddc889156b2473cc0d7488eaa1058af6cff74baf
• file sets are identical when excluding only extensions/connector-core/src/docs-bundle.generated.ts
• the PR-owned diff contains zero mutation-reproof workflow or smoke files
• regenerating the docs bundle was byte-stable: SHA-256 26b4df5d3d27cea5a4d4fc7b8de85cac64540b8a073ca3f5c81b2aa4b9a7519a before and after, with a clean worktree

[davidfarah2003]

APPROVE 2a2fb7f0b8acc516f6b74dd238c5f6c7f0665790

Independent re-review of the authorized SHA after the two blockers at ec6f1f7b2. Both product defects are gone on this SHA. Live PR head at publish time is ab9e52606207af6b96c62119843a11e29f4d475d (chore: merge current main). The only file delta from the reviewed SHA is .github/workflows/mutation-reproof.yml. I did not re-point.

Finding 1 (stop drain)

The shipped shutdown control is the same gated-first-eviction shape as the original probe: two owned orphans, first exact terminal held, stop() requested, then the gate released.

On a real JWT nats-server, all four shutdown cells passed:

• stop waits for an accepted startup reconciliation terminal
• the shutdown fence prevents the serial sweep from starting a later terminal
• startup cannot publish the manager service after stop completes
• shutdown refuses a new static reconciliation sweep

stop() now sets the fence, drains accepted sweeps and exact-terminal / retry flights, then joins the captured startTask (rejected start is swallowed). The serial loop checks the fence at the start of each alias, before attempted++ and before attemptStaticReconcile. That is the right boundary for "do not start the later alias." A fence tested only after the current await would be the off-by-one.

Synchronous admit: sweepsInFlight++ is immediately after the fence check, before minting the provisioner creds. The original race (stop returned in 74ms with sweepsInFlight=1, then the sweep drove hold-last and start() re-registered) does not reproduce.

Plain residual, not reproduced, not a block:

• There is still a one-statement window between if (staticReconcileStopping) return and sweepsInFlight++. I could not turn that into a later-alias terminal or a post-stop registration with the gated-first shape.
• await starting?.catch(() => {}) swallows a failed start. If start() never settles after the fence checks (a hang inside registration, not a throw), stop() waits with it. That is a worse failure mode than returning early, and I did not reproduce it.

Finding 2 (revision vs digest)

managerClusterDocument() is revision 11. The status output digest on this SHA is sha256:fac47815f91955f9c721d88dbeca166850b2e7090fc7b98c6d11b887e28afd5d. That equals MANAGER_CONTRACTS.status.output.closureDigest. It is not main's revision-10 digest sha256:171c1ea6bd518efc106491a69624d6dd19bf3f1ca221a9293eb8cec4abaedbbd. Generic invoke 12/12 includes a live status reply that passed the fetched output contract, so the number is not carrying a stale digest.

The invoke revision cell itself only asserts shipped.revision === 11. The digest match above is from compiling the document, not from that cell.

Mutation cells (not run: mutation-proof forbidden)

From check order in manager-reconcile-redrive.smoke.ts, each shutdown expectRed is the first cell that would notice its mutant. Marks are ✓ matches over combined stdout+stderr (manager log lines count). I am not inferring cell counts from mark counts, and I did not re-run the proofs.

• drain+join removal: first red is stop waits for an accepted startup reconciliation terminal (asserted before the gate is released)
• serial fence removal: stop waits… still passes, first red is the shutdown fence prevents the serial sweep from starting a later terminal
• start-join removal only: first two still pass, first red is startup cannot publish the manager service after stop completes

shipped-surface-pin.json M2 restoring revision: 10 is pinned to static reconciliation status advances the manager cluster revision.

Suites actually run

• pnpm typecheck: 27 of 28 (JS-only @cotal-ai/example-04-frontier-faces skipped)
• pnpm smoke:manager-reconcile-redrive: 30/30
• pnpm smoke:manager-reconcile-startup: 25/25
• pnpm smoke:manager-service-invoke: 12/12
• pnpm smoke:manager-service: 25/25
• unnamed pnpm smoke:manager-reconcile-ownership: 6/6
• unnamed pnpm smoke:manager-deregister: 30/30
• pnpm smoke:required-arg-seam: 253/253 (124/92)
• pnpm smoke:mutation-fixtures: 343 files, 1762 anchors, 0 dead / 0 ambiguous
• pnpm install --frozen-lockfile was run

closingIssuesReferences is empty. Body ends with Refs #774. Changeset calm-ravens-reconcile.md is patch. pnpm changeset status reports a minor for the fixed group because of other open changesets on main, same as last review of this PR.

CI at 2a2fb7f0b8acc516f6b74dd238c5f6c7f0665790

Present by name (separate from conclusion): CI, Docs, Mutation reproof, Windows.

• Docs: success
• Windows: success (windows-ok plus required + four shards + soak)
• CI run 33990634604: unit success, live success, smoke (shard 0/4) failure, shards 1–3 later cancelled, ci-ok queued. Shard 0 failed at pnpm smoke:card-host with ENOTEMPTY under /tmp/cotal-smoke-broker-6505-…, not at the redrive suite.
• Mutation reproof run 33990634742: full skipped, changed still in_progress (conclusion null). I did not treat that as green.

Merge hazard the body does not state

Main is still cluster revision 10. This PR claims 11 because the status output digest changed. #1301 also claims 11, for optional waitForExit on despawn / stop. #1301's body names this collision. This PR's body does not. Whichever merges second must move to 12 and re-derive its digest against the merged surface.

[davidfarah2003]

Independent re-review after re-point. This verdict binds only ab9e52606207af6b96c62119843a11e29f4d475d.

APPROVE

Confirmed live head with gh pr view 1302 at start and again at publish: OPEN, not draft, MERGEABLE, base main. GraphQL closingIssuesReferences is empty. The body uses Refs #774 and states that #774 stays open.

This SHA is the fold of 2a2fb7f0b8acc516f6b74dd238c5f6c7f0665790 onto current main 2528ec1f05570e58a08a4ac1341211a7062bb868. Versus 2a2fb7f0, the only changed file is .github/workflows/mutation-reproof.yml. implementations/manager/src/manager.ts and manager-service-contract.ts blob ids match that earlier SHA. Versus origin/main, the mutation-reproof workflow file is unchanged, so this PR no longer owns that workflow.

Independently recomputed git patch-id --stable over each side's merge-base..head diff, excluding extensions/connector-core/src/docs-bundle.generated.ts. Both sides are ddc889156b2473cc0d7488eaa1058af6cff74baf with the same 14-file set.

Shutdown control still holds on this tree. pnpm smoke:manager-reconcile-redrive printed MANAGER RECONCILE REDRIVE OK (30 checks), including stop waits for an accepted startup reconciliation terminal, the shutdown fence prevents the serial sweep from starting a later terminal, startup cannot publish the manager service after stop completes, and shutdown refuses a new static reconciliation sweep. pnpm smoke:manager-service-invoke printed MANAGER SERVICE INVOKE SMOKE OK with 12 passed. Cluster document revision is 11. Compiled MANAGER_CONTRACTS.status output digest is sha256:fac47815f91955f9c721d88dbeca166850b2e7090fc7b98c6d11b887e28afd5d. origin/main is still revision 10 and still requires status without staticReconciliation.

CI at this exact SHA. Workflow presence is listed separately from job conclusion. Incomplete work is unmeasured, not green.

• CI (https://github.com/Cotal-AI/Cotal/actions/runs/33991522377): present, pending, jobs list empty
• Mutation reproof (https://github.com/Cotal-AI/Cotal/actions/runs/33991522339): present, in_progress. Jobs: changed in_progress, full completed skipped
• Windows (https://github.com/Cotal-AI/Cotal/actions/runs/33991522343): present, queued. Jobs: Windows / required in_progress, Windows / smoke (shard 2/4 — non-blocking) completed success, soak queued, shards 0/1/3 queued
• Docs (https://github.com/Cotal-AI/Cotal/actions/runs/33991522337): present, queued. Job docs queued
• Code Quality: PR fix(manager): retry failed static reconciliation #1302 (https://github.com/Cotal-AI/Cotal/actions/runs/33991520501): present, in_progress. Analyze (python) completed success, Analyze (javascript-typescript) in_progress

I did not run mutation-proof.

Residuals, not blockers:

• One statement remains between the staticReconcileStopping check and staticReconcileSweepsInFlight++ in reconcileStaticLifecycles.
• stop() joins startTask after the drain. A start that never settles would hang stop.
• Open fix(manager)!: leave managed agents running on a stack stop #1301 also sets manager cluster revision: 11 on a different surface. This PR body does not name that collision.