Repository navigation
Commit d7a4bd3
feat(watch): opt-in work claims for multi-machine squad watch (ADR-002, gh-aw work-queue readiness) (#2337)
* docs(spec): ADR-002 work claims and external work queues
Add ADR-002 defining how Squad-run executors claim work and how Squad
relates to external work queues such as the gh-aw work queue:
- Split roles: WorkClaimProvider for executors Squad runs (watch, pods,
dev boxes) vs a future external work dispatcher (gh-aw submit-work /
submit-graph, cross-squad) where Squad only produces and observes.
- Git ref compare-and-swap primitive refs/squad/claims/<sha1(workKey)>
with a JSON claim record commit: acquire if absent, renew by
fast-forward with epoch+1, takeover after expiry plus skew margin,
release by CAS delete; fallback refs/heads/squad-claims/*.
- At-least-once semantics with fencing (kill the spawned agent) and
idempotent effects; opt-in via --claims git-ref / watch.claims,
default none; in-process hybrid dedupe always on.
- Normative TypeScript contract (WorkKey, ExecutorIdentity, ClaimRecord,
WorkClaimProvider, resolveExecutorIdentity, Noop/InMemory providers).
- gh-aw status evidence pinned to v0.93.3 and adapter gates G1-G7.
Amend the GitHub binding section 2 to map work.claim to the claim ref
and state that assignee/label mirrors are informational only.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(sdk): WorkClaimProvider contract and executor identity (ADR-002)
Add @bradygaster/squad-sdk/ralph/claims implementing the ADR 002 claim
contract: WorkKey + canonical workKeyToString, ExecutorIdentity with
resolveExecutorIdentity (SQUAD_MACHINE_ID > capabilities.machine >
SQUAD_POD_ID > hostname; one randomUUID instanceId per process),
ClaimRecord/AcquireResult/FinishOutcome, WorkClaimProvider with declared
guarantees, NoopClaimProvider (no exclusion) and InMemoryClaimProvider
(injectable clock, epochs, expiry takeover with skew margin, handle fencing,
exactly one winner under concurrent acquire).
Add a reusable guarantee-gated contract suite in
test/helpers/claim-provider-contract.ts for later providers. getPodId()
accepts an optional env so identity resolution is injectable.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(sdk): GitRefClaimProvider — Git-ref CAS work claims (ADR-002)
Add @bradygaster/squad-sdk/ralph/claims-git: a distributed WorkClaimProvider
(exclusion 'distributed', leases, fencing). Each claim is an empty-tree commit
whose message is the JSON record, at refs/squad/claims/<sha1(work key)> on a
shared remote (configurable; fallback refs/heads/squad-claims/). Acquire,
renew (fast-forward, epoch + 1), expiry takeover with skew margin, and release
(CAS delete) all push with --force-with-lease, so the remote decides every
race. No working tree, index, or HEAD changes; remote claims are fetched into
refs/squad-claims-cache/. Corrupt records yield 'unavailable'. Git runs via an
injectable GitTransport; the default spawns git with argument arrays (no
shell), and remotes/ref prefixes are validated against option injection.
Records carry a random nonce so one holder writing twice in the same second
never reproduces its own commit SHA (found by the contract suite).
Tests run the shared contract suite against a real temp bare repo with one
repository per executor, plus a mutation control that swaps in plain --force
and must observe more than one winner.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(cli): opt-in work claims for squad watch execute and fleet-dispatch (ADR-002)
- `--claims <none|git-ref>` flag and `watch.claims` config (string or
{ provider, ttlMs, skewMarginMs, remote, refPrefix }); flag wins; default
none keeps today's behavior. `claims` is a reserved watch key; invalid
values fail instead of silently disabling claims.
- execute and fleet-dispatch acquire a claim per issue and dispatch only
won issues; held/unavailable are skipped (fail closed). Claims renew every
ttlMs/3; a fenced renewal, or renewals failing until expiresAt - skew,
kills the agent process tree. Claims are released in finally with
completed/cancelled/abandoned.
- execute applies needs:* capability routing like fleet-dispatch.
- Per-round dispatched set: hybrid execute + fleet-dispatch never dispatch
the same issue in one round, whatever provider is configured.
- fleet-dispatch launches copilot asynchronously so it can be renewed/killed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(cli): claimed-work prompt contract for Ralph watch agents (ADR-002)
When an exclusive claim provider (exclusion !== 'none') is active, execute and fleet-dispatch prompts tell the agent to work only its claimed issues, not self-assign or take over others' work, and stop on issues it cannot complete (the watcher releases the claim). Prompts are unchanged under 'none'. Adds a Claimed Work section to the ralph-instructions.md template.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix(claims): lease bounds, record plausibility, capped git output, fixed claim identity (ADR-002 security review)
- claims.ts: MIN/MAX_CLAIM_TTL_MS, MAX_CLAIM_SKEW_MARGIN_MS, MAX_CLAIM_EPOCH,
claimTimingError(), claimRecordImplausibility(); assertClaimTtl caps ttl at 24h;
providers cap skewMarginMs at 1h; InMemoryClaimProvider takes over implausible
records (parity with git-ref); hostname fallback for machineId is published
as a salted pseudonym h-<sha256(hostname, salt)[:12]>.
- claims-git.ts: transport caps stdout (1 MiB default, 64 KiB for claim
commits), never throws in 'close'; claim commits with a non-empty tree, more
than one parent, oversized bodies, or implausible epoch/times are taken over
with CAS on the observed sha and a WARN naming the ref; commit-tree runs with
a fixed squad-claims <squad-claims@invalid> author/committer and a +0000
date from the injected clock.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix(watch): enforce lease bounds, kill agent process groups, fail closed on malformed config (ADR-002 security review)
- config: watch.claims timing must satisfy claimTimingError (30s <= ttlMs <=
24h, skewMarginMs <= 1h, ttlMs > 2 * skewMarginMs); a .squad/config.json
that exists but is not valid JSON throws WatchConfigError; unknown
watch.claims keys are warned about.
- claims: acquireClaims fails closed and holdClaimsWhile refuses (releasing
its claims, never running work) when timing would make the renew timer spin.
- agent-spawn / fleet-dispatch: agents that may be fenced run as POSIX
process-group leaders; killProcessTree kills the group (-pid, ESRCH
fallback) and skips children that already exited (PID-reuse guard);
detached groups are killed on watch exit; PidTracker kills groups too.
- watch: hostname fallback identity is salted with the team root path.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* docs(claims): ADR-002 plausibility rule, lease bounds, claim identity; changeset (security review)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* feat(cli): work-claim telemetry for squad watch (ADR-002)
Per-round and since-startup claim counters (acquired, contended, unavailable, fenced, lost, released by outcome, implausible takeover) surfaced in the round board, watch --health, and OTel counters squad.watch.claims.* (claim.provider, claim.outcome). Only when the provider excludes other executors; 'none' output unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* test(watch): double-pick regression suite with mutation controls (ADR-002)
Adds end-to-end git-ref coverage over real clones of one bare repository: 8-clone acquire race plus late retries, two ExecuteCapability executors dispatching disjoint issues (with a provider-none overlap control), cross-clone fencing with injected clocks, and hybrid execute + fleet-dispatch same-round dedupe under none, in-memory, and git-ref claims.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* docs(watch): multi-machine work claims guide; mark cross-machine skill unimplemented (ADR-002)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix(watch): renew claims that went stale during acquisition; de-flake double-pick suite (ADR-002)
Root cause of the double-pick flakes under the full parallel suite:
1. "two executors dispatch disjoint issue sets" (A [1,2,3,4], B [4]) was not
a CAS hole. Instrumented runs under load show B's claim on #4 was a fresh
epoch-1 root claim pushed after A's release had deleted the ref: A's whole
round (4 acquires, 300 ms agent, release) finished while B's contended
acquires (~3 s each) were still walking its stale issue list. Claims only
promise that no two executors hold an issue at the same time. The race now
keeps each agent (and so its claims) alive until both executors have
finished every acquire, so disjointness is a strict exclusion check again.
2. The fencing test bounded total runtime from test start (< 15 s) and its
stub agent lived 30 s from spawn; on a loaded host B's takeover alone can
take longer. The agent now lives 30 s past B's takeover and writes a
marker only if it does, so the test proves the fence killed it without a
wall-clock threshold. TTL 9 s (renew every 3 s) gives B's takeover a wide
CAS window; per-test timeout raised to 120 s for real git under load.
Product gap found while analysing (1): acquisition is sequential, so a slow
or hanging remote can carry an early claim past expiresAt - skewMarginMs (or
past takeover) before dispatch, and the agent started on an issue another
watcher held until its first renewal fenced it. acquireClaims now renews such
claims before returning and drops them as held (fenced) or unavailable (lost).
Regression tests in watch-claims.test.ts are red without the fix
("A dispatched issues: ... expected [ 1, 2 ] to deeply equal [ 2 ]").
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* test(watch): keep run-bound mocks compatible with claims
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
* fix(watch): close claim fencing review gaps
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
* fix(watch): terminate escaped POSIX descendants
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
* test(release): classify noop claim contract skips
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
* fix(watch): release late renewed claims
When cleanup races a renewal that already won remotely, inspect the fenced claim, advance to the newer epoch owned by this executor, and release that current handle. Late renewal responses also run idempotent compensating release without blocking forever.
Adds a deterministic delayed-response regression proving no live claim remains after work completes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: B'Elanna (Squad) <tamirdresher@users.noreply.github.com>
Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba91 parent c713a82 commit d7a4bd3
53 files changed
Lines changed: 7018 additions & 202 deletions
File tree
- .changeset
- .squad-templates
- skills/cross-machine-coordination
- docs
- specification
- src
- content/docs
- features
- reference
- scenarios
- packages
- squad-cli
- src
- cli
- commands/watch
- capabilities
- core
- templates
- skills/cross-machine-coordination
- squad-sdk
- src/ralph
- templates
- skills/cross-machine-coordination
- scripts/release
- templates
- skills/cross-machine-coordination
- test
- cli
- helpers
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
55 | 69 | | |
56 | 70 | | |
57 | 71 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
15 | 34 | | |
16 | 35 | | |
17 | 36 | | |
| |||
53 | 72 | | |
54 | 73 | | |
55 | 74 | | |
56 | | - | |
| 75 | + | |
57 | 76 | | |
58 | 77 | | |
59 | 78 | | |
| |||
64 | 83 | | |
65 | 84 | | |
66 | 85 | | |
67 | | - | |
| 86 | + | |
68 | 87 | | |
69 | | - | |
| 88 | + | |
70 | 89 | | |
71 | 90 | | |
72 | | - | |
| 91 | + | |
73 | 92 | | |
74 | 93 | | |
75 | 94 | | |
| |||
96 | 115 | | |
97 | 116 | | |
98 | 117 | | |
99 | | - | |
| 118 | + | |
100 | 119 | | |
101 | 120 | | |
102 | 121 | | |
| |||
204 | 223 | | |
205 | 224 | | |
206 | 225 | | |
207 | | - | |
| 226 | + | |
208 | 227 | | |
209 | 228 | | |
210 | 229 | | |
| |||
387 | 406 | | |
388 | 407 | | |
389 | 408 | | |
390 | | - | |
| 409 | + | |
391 | 410 | | |
392 | 411 | | |
393 | | - | |
| 412 | + | |
394 | 413 | | |
395 | 414 | | |
396 | 415 | | |
| |||
402 | 421 | | |
403 | 422 | | |
404 | 423 | | |
405 | | - | |
| 424 | + | |
406 | 425 | | |
407 | 426 | | |
408 | 427 | | |
| |||
414 | 433 | | |
415 | 434 | | |
416 | 435 | | |
417 | | - | |
| 436 | + | |
418 | 437 | | |
419 | 438 | | |
420 | 439 | | |
| |||
0 commit comments