cotal up leaves no persisted mesh record — a dead stack erases the mesh's existence

[davidfarah2003]

Two coupled behaviors, measured 2026-08-26 on macOS with 0.30.0→0.30.1:

1. An up-started mesh is only discoverable while its processes live. cotal meshes rows for meshes registered with meshes add persist in ~/.cotal/meshes/ and carry the registered tag. A mesh started by cotal up shows a row with no tag, derived from runtime state. When the stack's processes die, the row vanishes: the operator gets ✗ no mesh named "X" is running - see cotal meshes, and cotal meshes shows nothing — no record the mesh ever existed, no root path, no restart affordance. To someone who did not start the stack themselves this reads as data loss ("my mesh is not even registered, wtf" — verbatim operator reaction).

2. cotal up runs the whole stack as children of the invoking process, with no daemonize option. Broker, delivery daemon and manager all die with the invoking process tree. When the invoker is an agent session — increasingly the normal case, an agent setting a mesh up on a person's behalf — a session restart silently kills the broker, the manager, and every seat mid-work.

Together they compound: the session dies → the mesh dies with it → and the CLI then denies the mesh was ever there, so the person has nothing to restart from except memory of a directory path.

Proposal:

• Persist the mesh record at provision time (same store as meshes add, marked self-hosted with its root), so a dead stack yields mesh "X" is recorded at <root> but not running — run cotal up there to restart instead of an existence denial.
• Give cotal up a --daemon mode (setsid/reparent + pid files), or make detaching the default with the current behavior behind --foreground. An agent-invoked up should produce a stack that outlives the agent.

[davidfarah2003]

Operator directive raising the bar on this: the right path should be the only way — cotal must be resilient to a naive agent launching it. Not a --daemon flag an agent can forget, but the default and only behavior: up always detaches the stack from the invoking process (setsid/reparent), always persists the mesh record, and treats the invoking environment as hostile (an agent session's env, an ssh hangup, a CI runner — none of them may become a load-bearing parent). --foreground can exist for debugging; nothing else should depend on the invoker's lifetime.

[davidfarah2003]

Reproduced live at exact main SHA 229f69b054cf781aabdbf4f08ec471482ec201b1. In the fully sandboxed real-stack smoke, after a successful up/down, cotal meshes printed no meshes registered; the assertion that the up-owned mesh remained recorded offline failed. Source matches: bare up only calls startMeshDetached when --detach is supplied, and foreground broker exit/down removes matching non-manual records. Open PR #880 owns the intended detach-by-default + persistent self-hosted-record fix. However, independent validation of its current head failed at its own up persists a self-hosted mesh record cell; the PR is conflict-dirty and separately blocked by Windows process identity #969 plus a dead-current routing review finding. Recommend bug, triage:confirmed, area:cli, severity:critical, triage:owned. Do not duplicate; refold and repair #880, then require current live persistence/detach proof and native Windows identity acceptance.

[davidfarah2003]

Code reading, not a live reproduction. I have not run cotal up to reproduce this, and I say that plainly because what follows corrects part of the diagnosis rather than confirming it. The symptom described here is real. The stated cause does not look right, and the difference makes the fix considerably smaller.

up does persist a mesh record today. implementations/cli/src/commands/up.ts:2316:

recordMesh({ ...m, origin, ...(unencryptedOverlay !== undefined ? { unencryptedOverlay } : {}) });

So the record is written at provision time already. What removes it is a later prune. packages/workspace/src/connect.ts:505-511:

const r = await preflightTarget(target, probeCreds);
if (r.ok) return;
// The classifier says whether this failure is a stale-entry signal; `pruneMesh` says whether the
// record is one an automatic sweep may delete (an operator-registered mesh is not).
const pruned = r.prune ? pruneMesh(target.space) : false;

That is the asymmetry in the report, and it is deliberate rather than an oversight: a record written by up is sweepable, a record from meshes add is exempt. So the sequence an operator sees is not "the mesh was never recorded". It is "the mesh was recorded, the stack died, and the first command that found it unreachable deleted the record as a stale entry". The row is present right up until something looks for the mesh, which is also why it is hard to catch in the act.

Both of those landed in 531d37d8f on 2026-08-09, so they predate the 2026-08-26 measurement here and this is current behaviour rather than something that has since changed.

Why this matters for the fix. The proposal reads "persist the mesh record at provision time", and that part is already done. The work is in the prune deciding that an unreachable self-hosted mesh is a stale entry worth erasing, when it is the only remaining evidence of where the mesh lives. Keeping the record, or leaving a tombstone carrying the root path and the restart affordance, is a much narrower change than adding persistence, and it lands exactly where the operator experience in this report goes wrong.

Worth knowing for whoever takes it: the prune has prior form for over-deleting. The comment immediately above that call records a case where it removed a healthy mesh's record, found live, with foreground spawn triggering it and every later command falling into raw-path copy. So this would be tightening a classifier that has already been wrong in the same direction once.

I have not looked at part 2, the absence of a daemonize option, and nothing above speaks to it. That half of the issue stands as filed.

[davidfarah2003]

Correcting the framing of my earlier comment, and adding the dependency it missed.

This is not unowned. PR #880, feat(cli)!: detach and persist meshes on up, is open and covers both asks here, the persisted record and the detach. I missed it because the scan I used filtered cross-referencing PRs to closed ones, so anything with a fix in flight read as untouched. My analysis above was written as though nobody was on this.

That analysis is still worth having, but it belongs to #880's author and reviewer rather than to this thread, and one part of it may change what that PR needs to do. cotal up already calls recordMesh at implementations/cli/src/commands/up.ts:2316. What removes the record is pruneMesh in packages/workspace/src/connect.ts, on the first command that finds the mesh unreachable, with operator-registered meshes exempt. If #880 is written against the premise stated in this issue, that no record is persisted, then the persistence half may already be done and the real work is stopping the sweep from erasing it. Worth checking before that PR is reviewed on its stated scope.

The more useful thing to record is why this is stuck. #880 is conflicted against current main across 7 files:

bin/smoke/up-stack-live.smoke.ts
docs/cli.md
extensions/connector-core/src/docs-bundle.generated.ts
implementations/cli/src/commands/up.ts
implementations/cli/src/lib/delivery-proc.ts
implementations/cli/src/lib/manager-proc.ts
package.json

measured with merge-tree against main, and it is the same 7 recorded on that PR hours earlier, so nothing has moved. A conflicted pull request mints zero pull_request workflow runs, which means #880's checks are not red, they are absent. It cannot be evaluated by anyone until the merge is done.

So this issue is not waiting on a decision, a triage, or an owner. It is waiting on a semantic merge across 7 files, two of which are generated or pinned (docs-bundle.generated.ts, package.json) and three of which are production code. That is the whole blockage, and it is worth stating plainly because a critical sitting behind an invisible CI state looks like neglect and is actually a merge nobody has picked up.

I am not doing that merge. It is someone else's PR and a semantic conflict in up.ts and the process-lib files is exactly the kind of change that should be resolved by whoever wrote the intent.

[davidfarah2003]

Reproduced both behaviours against main at ffa1501748d59d45234c1208a3322fbf14ceadd9 (cotal-ai 0.48.2, Linux). Two of this issue's premises turn out to be wrong in ways that change what needs building, so putting them here before anyone writes code.

The command the error recommends is the command that destroys the record

pruneStaleMeshes() is invoked by meshes (per its own docstring, alongside spawn/use). The failure text tells the operator to run cotal meshes. So the record is not lost when the stack dies — it is lost the moment the operator follows the instruction they were just given to go looking for it. The first cotal meshes after the stack dies is what erases it, and every one after that reports it never existed.

Correction 1: the record IS written at provision time; it is deleted afterwards

The issue title says up "leaves no persisted mesh record". It does write one. A cotal up-provisioned mesh on this machine carries a durable record with its root:

{ "space": "cloudbuild", "server": "nats://127.0.0.1:4422",
  "root": "/home/cotal/hack-lanes/cloudmesh", "mode": "auth",
  "ts": "2026-09-09T20:57:48.470Z", "origin": "up" }

The record is created and then deleted after death, at packages/workspace/src/mesh-registry.ts:254:

export function pruneMesh(space: string): boolean {
  const m = findMesh(space);
  if (!m || m.origin === "manual") return false;   // operator-registered survives
  removeMesh(space);                                // origin:"up" is deleted
  return true;
}

This matters because the proposal here ("persist the mesh record at provision time") describes a mechanism that already exists. Acting on the issue as written would mean building a persistence layer next to a working one. The actual gap is that survival is withheld from origin:"up" records.

Evidence — controlled pair. Two registry records, identical in every field (same dead broker, same root, same timestamp) except origin, then a single read-only cotal meshes:

record	origin	outcome
probe864up	up	deleted from disk; absent from the listing entirely
probe864man	manual	survives; renders its root, tagged registered · offline

Follow-up command against each:

$ cotal endpoints --space probe864up
✗ no mesh named "probe864up" is running - see `cotal meshes`

$ cotal endpoints --space probe864man
✗ no broker answered at nats://127.0.0.1:1 - "probe864man" is registered here but its
  mesh is not up; start it where it runs, or `cotal meshes rm probe864man` to unregister it

The affordance this issue asks for already exists and already works. It is withheld from exactly the meshes up creates.

Correction 2: the delivery daemon is already detached — only the broker matches

The issue states "Broker, delivery daemon and manager all die with the invoking process tree." The delivery daemon does not: it is spawned detached: true and unref()ed (implementations/cli/src/lib/delivery-proc.ts:147,149). Process topology of a live up-started stack:

    PID    PPID    PGID     SID  COMMAND
3569481 3565639 3569481 3569481  cotal up --space cloudbuild
3569539 3569481 3569481 3569481  nats-server     <- shares up's PGID and SID: not detached
 732215 3565639  732215  732215  cotal deliver   <- own session leader: already detached

The broker does match the claim: implementations/cli/src/commands/up.ts:1002 spawns it stdio: "inherit" with no detached, and up.ts:1041-1045 installs SIGINT/SIGTERM handlers that child.kill("SIGTERM") it — so an ordinary session teardown takes the broker down deliberately. There is no --daemon or --foreground affordance anywhere in the CLI.

So the blast radius is one component, not three. Worth correcting because it makes the daemonization half a smaller change than this issue implies.

Note for whoever picks this up

The current behaviour is test-locked as correct in packages/workspace/smoke/preflight.smoke.ts:206-212, which asserts that an origin:"up" record is pruned and an origin:"manual" one goes offline. Changing the behaviour necessarily turns those assertions red. That redness is the intended outcome, not a regression — but it needs to be called out loudly in any change, because a reviewer meeting a silently-flipped assertion should block on it.

The two halves are also separable, and are better separated: the record's survival is a narrow change to pruneMesh; daemonizing the broker is a different change with a different risk profile.

Not verified here, and stated so it is not assumed: the daemonization behaviour was graded from the signal handlers and live process topology, never by killing a stack end-to-end; up's own write path was read rather than run; the manager component was not checked; and all of the above is Linux 0.48.2, whereas this issue was measured on macOS 0.30.0→0.30.1.