feat(manager,runtime)!: the manager hosts workflow runs - #1296
feat(manager,runtime)!: the manager hosts workflow runs#1296davidfarah2003 wants to merge 21 commits into
Conversation
… records walk A run's driver used to connect as `admin`, which on an authed broker cannot read its own run record, so `cotal run` only ever worked on an open mesh. This adds the per-run credential SPEC 14.6 names. - `run-driver` profile: minted per run and per takeover attempt (`MintOpts.runDriver` with the endpoint, run id, takeover id and the driving instance's id and epoch). Its rows are the journal subject and replay durable, the run's own records, the checkpoint plane at its own timer coordinates, the channel and presence reads a wait needs, the conclave registries, and the manager's lifecycle commands as the run's derived caller. No consumer verb on the records store, no read of the journal stream at large, no channel publish. - `program` record kind: the source a run was started from, written create-only by the driver that pins the run (and by a fork for its child). `cotal run resume` reads it back, so the run id alone resumes a run; a `--file` that differs from the record is refused as a migration. - `walkKvEntries`: a consumer-free enumeration of a key family by a forward `STREAM.MSG.GET` walk. Notice, migration and `ps` listings use it, so a principal with no consumer verb on the records store can list its own keys. - The settle watcher polls the checkpoint settle fact instead of binding an EPF consumer, so a driver credential needs no consumer on that stream either. - D32 matrix section 6 pins the profile's rows exactly; the grant sweep covers it; a new authed suite (`smoke:runtime-run-driver-auth`) drives a program with a sleep and a channel wait on the credential and proves the pins the broker enforces, with a mutation config.
Parse { restarts, window? } at spawn, refuse a policy this host cannot honour,
and restart the process under the same lifecycle until the budget is spent.
Mutation-proof needs the completion marker even when a cell fails, so these suites grade with named cells and still print their exit line.
…arts Drive supervise-restart through a real Manager on a scratch broker: SIGKILL the pty pid, keep identity/lifecycle, re-yield the pending turn from the replacement, retire on a spent budget, and refuse user-mode/non-pty at accept. Parse also refuses restarts:"2".
Stale presence of a SIGKILL'd child still matches principal+uid, so recovery now waits for a later heartbeat and retires a relaunch that never joins. The live suite forces that miss with a missing binary instead of wrapping Manager.runtime.
A composition-root cotal-lang program spawn()s with supervise, SIGKILLs the real join seat mid-turn, and asserts the replacement pulls and yields the same goal before a second kill spends the budget as L4002.
The live supervise cell named the recorded reason but only checked the code. Require kind turn and the handler's agent-down message.
Top-level return is L1024, so the program cannot yield e.code as the run value. completed with no wrapper error is the catch; uncaught L4002 is re-thrown by startRun after noteFinal(failed).
The manager serves run-start, run-resume, run-answer, run-status and run-ps on its endpoint rails (cluster document revision 12). A program is validated before anything is recorded and a refusal carries every problem as an ai.cotal.lang.problem detail; the run is driven in the manager's process on its own connection under a per-run, per-takeover run-driver credential, re-minted on the renewal loop; a served read or answer rides a one-shot run-operator credential. run-start and run-resume answer once the attempt's status record is a fact, or with why it never became one. At boot the manager takes back every run recorded running from its recorded program under the next epoch, so a parked checkpoint survives a manager restart. A remote-authority manager refuses the family loud. The runtime registers the cotal-lang run host on the core run-host contract, so the manager resolves it by name and never imports the runtime. cotal run is a client of the manager by default; --local keeps the in-process drive for a bare broker or a run with no recorded program. resolveControlTarget moves to the workspace tier so the runtime shares it with the CLI. A run capability mints the family plus the spawn set into an agent credential, and the connector injects a cotal_run tool for it, so an agent can write a cotal-lang program and start it from a session. The privileged instrument carries the five commands too. bin/smoke/run-host-live proves the family against the real manager and runtime on a JWT broker under the run capability alone (validation refusal, a pure run, a parked checkpoint answered from outside, resume refusals, a restart takeback) and on an open broker through the shipped cotal run client, with a mutation config aimed at validation, the answer-after-record wait and the boot reconcile filter.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| const kids: ChildProcess[] = []; | ||
| const scratch: string[] = [home]; | ||
| let rc = 1; |
| authDir, endpointAuth, findCotalRoot, isWorkspaceTargetError, loadSpaceAuth, resolveMeshTarget, | ||
| pruneStaleMeshes, renderWorkspaceError, soleSpaceOf, type MeshTarget, type MeshTargetErrorCode, | ||
| } from "@cotal-ai/workspace"; | ||
| import { controlCaller, loadSpaceAuth, renderWorkspaceError, type ControlAuth } from "@cotal-ai/workspace"; |
# Conflicts: # bin/smoke/required-arg-seam.smoke.ts # extensions/connector-core/src/docs-bundle.generated.ts
|
Two measured CI failures at 1.
|
|
Correcting my own finding #2 above before anyone acts on it. The diagnosis was right; the remedy I proposed was wrong and would have had someone re-register a suite that is already registered.
So it already runs in the sharded chain, on a job that installs nats-server. It is registered correctly and its CI fragment exists. The defect is only that it is also appended to the manager package's My earlier wording — "register it as a smoke suite so it runs in a sharded job, or add the broker install to Both fixes as an applicable patch, at
|
Review round on the manager-hosted workflow runs. - Boot gate: run-start and run-resume are refused `unavailable` until the host exists and its boot reconcile has returned, so a resume can no longer race the reconcile into a second attempt of the same run. A remote-authority manager still refuses `unimplemented`; a booting one no longer borrows that sentence. - One attempt per run, held synchronously: the run's slot is claimed before the first await of a launch and freed only by that attempt's end, and each attempt activates under its own holder id (endpoint id plus takeover id) so two attempts never present the tuple the activation barrier admits as one process. A launch that never activates is released and its connection closed rather than left orphaned. - Client deadlines outlive the manager's activation wait (`RUN_LAUNCH_DEADLINE_MS` over `RUN_ACTIVATION_WAIT_MS`), so a slow launch reads as the manager's own refusal and never as "no manager answered" pointing at `--local`. - `run-answer` records the answerer from the caller's authenticated principal (a managed seat by its name), and takes no `by`; the tool and the hosted CLI send none. `--by` stays on `answer --local`. - Admission cap on concurrent launches (`resource-exhausted`), and the validation refusal drops the rendered source frame. - Grants: the driver profile loses its answer-record write (it never files one); the operator profile splits into a read form with no write row and an answering form that adds the answer record, checkpoint status and settle. - `cotal run --local` mints the run's own `run-driver` / `run-operator` credentials instead of `admin`, so it drives on an auth broker; a user-auth mesh refuses it by name. Hosted `start`/`resume` refuse `--endpoint`, and every refusal that hands back a resume command carries the run id. - CI: the `run` flag golden gains `local:boolean`; supervise-restart leaves the manager package's unit script (it already runs as a CI shard suite). Suites: run-host-live grows to 42 cells (boot gate, per-attempt holder, caller-derived answerer, frame strip, local drive on an auth mesh, deadline relation); d32-matrix pins the operator profile's two forms; run-command-usage grades the hosted-path flag refusals.
… resume's slot, refuse hosting on a user mesh A run-answer is now two operator credentials: a read that replays the run's journal to the open pause, then an answering one minted for that pause's token alone, so its answer record, checkpoint status and settle fact reach no other pause on the endpoint. The RunHost contract splits accordingly into locate and answer, and cotal run answer --local takes the same two steps. A resume refused at the admission cap, or during a stop, gave nothing back: the slot it had claimed stayed in the live map with no drive, so every later resume of that run was a conflict on a run nobody was driving. launch() now owns the slot from its first line and frees it on every refusal before a drive exists. A user-auth mesh stands no run host up and refuses the family as unimplemented by name: a hosted run's seats would be spawned under the static owner, which the user-mode spawn door refuses. The driver profile's residual reach (endpoint-wide checkpoint rows, store point reads, per-step chat durables, conclave registries) is now stated in the docs as it is minted. run-host-live grows to 45 cells with three new mutants; d32-matrix pins the token-pinned answering form.
The usage line's ps segment and the resolver's presenter moved with the hosted-run work, and the presence-render census total moved by the two run-status strings cotal_run renders; each anchor follows its code, and the two strings are classified as run-record state, not presence.
…osted path Both doors refuse on a user-auth mesh; the connect sentence now says so and names the static-auth mesh as the way to run programs, as the manager's refusal does.
1 participant
What
The manager hosts workflow runs.
cotal run starthands a cotal-lang program to the manager of the resolved mesh, which validates it, mints the run id, drives it in its own process and answers once the run is recorded. The terminal is free the moment the id prints; the run continues through every pause, and a manager restart takes every run it had recorded running back from the journal under the next epoch. Agents get the same surface as a tool.spawn({ supervise })is the restart policy the manager enforces in place, which the same programs lean on.Scope
Core
run-hostextension contract (packages/core/src/run-host.ts): validate, drive, answer, status, list. The runtime registerscotalLangRunHoston it; the manager resolves it by name and never imports the runtime.LANG_PROBLEM_DETAIL_KIND(ai.cotal.lang.problem): the reverse-DNS detail kind a validation refusal carries each language problem under.RUN_ACTIVATION_WAIT_MS(the manager's answer-after-record wait) andRUN_LAUNCH_DEADLINE_MS(the client deadline a start or resume uses, which outlives that wait by construction).runcapability:runCallerCapabilities(owner)mints the five untargeted run commands plus the spawn set a program's own spawns need. The privileged instrument carries the run family too.run-driver(per run, per takeover attempt; its rows hold no answer-record write, since a driver never files one; its endpoint-wide checkpoint rows, store point reads, per-step chat durables and conclave registries are named in the docs as the profile's residual) andrun-operator(one-shot, per served call) in two forms: a read form with no write row, and an answering form minted for ONE checkpoint token (answers: { token }) that holds that pause's answer record, checkpoint status and settle fact and no other pause's.programrecord kind and the consumer-freewalkKvEntriesrecords walk.Manager
run-start,run-resume,run-answeronmanager.run;run-status,run-psonmanager.read.run-starttakes no endpoint (the manager hosts under its own).run-answertakes noby: the manager records the answerer from the caller's authenticated principal, a managed seat by its persona name and anyone else by their principal. An answer is two operator credentials: a read that replays the journal to the open pause, then an answering one pinned to that pause's token. The run-host contract splits intolocateandansweraccordingly.RunHosting: each drive on its own standing connection under a per-run, per-takeoverrun-drivercredential, re-minted on the renewal loop so a run parked for days outlives the credential's TTL; each served read or answer on a one-shotrun-operatorcredential in the form the call needs.run-startandrun-resumeanswer once the attempt's status record is a fact, or with why it never became one; a launch that never activates in the wait is released and its connection closed.run-startandrun-resumeare refusedunavailableuntil the host exists and its boot reconcile has returned. A remote-authority manager refuses the family asunimplemented; a booting one no longer borrows that sentence. A user-auth mesh stands no host up and refuses the family asunimplementedby name: a hosted run's seats would be spawned under the static owner, which the user-mode spawn door refuses. Boot reconcile resumes every run recorded running from its recorded program (epoch + 1); a run without a recorded program is logged and left.resource-exhaustedpast eight in their wait), and a refused launch frees the slot it claimed, so a refused resume never leaves its run reading as held. A validation refusal drops the rendered source frame.stop()closes a parked drive's connection rather than draining it, so its record staysrunningfor the successor.spawn({ supervise: { restarts, window? } }): the manager relaunches a seat in place within the budget (heartbeat after spawn, retirement past the budget so the nextturnis L4002); a policy this host cannot honour is refused at accept. The capability filters (manager andcotal spawn) acceptrun.Runtime / CLI
cotal runis a client of the manager by default:startprints the minted id and returns,resume <runId>takes no--file(the manager reads the recorded program),ps/journal/answerride the served commands. Hostedstartandresumerefuse--endpoint, hostedanswerrefuses--by, and every refusal that hands back a resume command carries the run id. A validation refusal is rendered as the validator would print it.--localkeeps the in-process drive, now under the run's ownrun-driver(a drive) andrun-operator(a read or answer) credentials minted from the project folder's trust material instead ofadmin, so it drives on an auth broker; a user-auth mesh refuses it by name.answer --localkeeps--byand takes the same two credentials as the hosted answer.resolveControlTargetandcontrolCallermove to@cotal-ai/workspaceso the runtime shares them with the CLI (implementations never import each other); the CLI re-exports them.connectOrExittakes amintoption for a profile's own pins.Connector
cotal_runtool (verbsstart,resume,answer,status,ps), injected only for personas declaringcapabilities: [run]in auth mode;answersends no name, the manager records the agent from its credential. Tool docs and the docs bundle are regenerated (22 tools).Docs
workflows.md"Operating a run" and the wire section,cli.mdrun section,agent-files.mdcapabilities row,identity-and-auth.md"Declared capabilities" and the profile table,lang-card.mdpointer.check:docs-voicegreen.Tests
New suites, each with a CI fragment (five fragments in
bin/smoke/ci-suites.d/):bin/smoke/run-host-live(45 cells): a JWT broker under anagentcredential holdingcapabilities: [run]alone: validation refusal with frame-stripped detail records; a pure run completes and is listed; a checkpoint parks,run-resumeis a conflict,run-answerat a wrong key is not-found, a request naming its ownbyis refused at the contract, the answer is recorded under the caller's principal; resume of an unknown run is not-found; a manager restart takes a parked run back under epoch 2 with each attempt under its own holder id; the boot gate refuses a start and a mid-reconcile resume asunavailable, then a post-reconcile resume asconflict; an answering credential re-minted for the answered pause is refused by the broker on any other token; a resume refused at the admission cap isresource-exhaustedtwice and holds no slot; a manager on a user-marked workspace refusesrun-startasunimplementednaming the space;--localdrives, reads and answers on the auth broker under its own credentials; the client deadline outlives the activation wait. Then an open broker through the shippedcotal runclient. Mutation config with thirteen killed mutants (validation gate, answer-after-record wait, reconcile filter, boot gate on start, boot gate on resume, slot conflict, per-attempt holder, caller-derived answerer, frame strip, deadline relation, the pre-claimed slot a cap refusal kept, the token pin on the answering credential, the user-mesh refusal).bin/smoke/lang-supervise-live(11 cells): a program-drivenspawn({ supervise })against the real manager with a SIGKILL mid-turn and the replacement completing the goal.implementations/manager/smoke/supervise-restart(21 cells): the in-place restart budget, window, heartbeat and retirement.implementations/runtime/smoke/run-driver-auth(13 cells): therun-drivercredential drives across every plane its rows name, andadmincannot.implementations/runtime/smoke/spawn-policy: thesupervisepolicy's parse and refusal sentences.Grown suites:
d32-matrixpins the driver profile exactly and the operator profile's two forms, the answering one to its token;endpoint-grantspins the 13-command privileged instrument and the 12-row run set;eps-grant-sweepcoversrun-driverandrun-operator;orientationcovers thecotal_rungating;run-command-usagegrades the hosted-path flag refusals;flag-inventorycarrieslocal:boolean;required-arg-seampin 127/94 with its ledger;presence-render-censusclassifies the two run-status strings; three mutants re-anchored where their code moved.Re-run green after the two review rounds: run-host-live 45/45 with 13/13 mutants killed, runtime run-command 23/23, run-driver-auth 13/13, mesh-checkpoint 42/42, mesh-ask 42/42, manager service-ops 76/76, service-invoke 11/11, supervise-restart 21/21, d32-matrix 78/78, lang-supervise-live 11/11, ep-grants 45/45, eps-grant-sweep 11/11, required-arg-seam 253/253, run-command-usage 15/15, presence-render-census, mutation-fixtures, flag-inventory, orientation, ci-suites-freeze, ci-declarations, the manager package's unit script, typecheck and build.
Not in this PR (proposed spec text, for David's gate)
SPEC 14.3 could state the hosting in one sentence: "The manager hosts the run driver:
run-startvalidates and records the program, drives it in the manager's process under a per-runrun-drivercredential, and a manager restart resumes every run it recorded running." SPEC.md and spec/cotal-lang.md are untouched here.Breaking
cotal runneeds a manager on the mesh by default; the previous in-process behaviour is--local.cotal run answerno longer takes--byon the hosted path. A user-auth mesh refuses the run family. The manager's cluster document moves to revision 12.