Skip to content

Commit d7a4bd3

Browse files
feat(watch): opt-in work claims for multi-machine squad watch (ADR-002, gh-aw work-queue readiness) (#2337)
* docs(spec): ADR-002 work claims and external work queues Add ADR-002 defining how Squad-run executors claim work and how Squad relates to external work queues such as the gh-aw work queue: - Split roles: WorkClaimProvider for executors Squad runs (watch, pods, dev boxes) vs a future external work dispatcher (gh-aw submit-work / submit-graph, cross-squad) where Squad only produces and observes. - Git ref compare-and-swap primitive refs/squad/claims/<sha1(workKey)> with a JSON claim record commit: acquire if absent, renew by fast-forward with epoch+1, takeover after expiry plus skew margin, release by CAS delete; fallback refs/heads/squad-claims/*. - At-least-once semantics with fencing (kill the spawned agent) and idempotent effects; opt-in via --claims git-ref / watch.claims, default none; in-process hybrid dedupe always on. - Normative TypeScript contract (WorkKey, ExecutorIdentity, ClaimRecord, WorkClaimProvider, resolveExecutorIdentity, Noop/InMemory providers). - gh-aw status evidence pinned to v0.93.3 and adapter gates G1-G7. Amend the GitHub binding section 2 to map work.claim to the claim ref and state that assignee/label mirrors are informational only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(sdk): WorkClaimProvider contract and executor identity (ADR-002) Add @bradygaster/squad-sdk/ralph/claims implementing the ADR 002 claim contract: WorkKey + canonical workKeyToString, ExecutorIdentity with resolveExecutorIdentity (SQUAD_MACHINE_ID > capabilities.machine > SQUAD_POD_ID > hostname; one randomUUID instanceId per process), ClaimRecord/AcquireResult/FinishOutcome, WorkClaimProvider with declared guarantees, NoopClaimProvider (no exclusion) and InMemoryClaimProvider (injectable clock, epochs, expiry takeover with skew margin, handle fencing, exactly one winner under concurrent acquire). Add a reusable guarantee-gated contract suite in test/helpers/claim-provider-contract.ts for later providers. getPodId() accepts an optional env so identity resolution is injectable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(sdk): GitRefClaimProvider — Git-ref CAS work claims (ADR-002) Add @bradygaster/squad-sdk/ralph/claims-git: a distributed WorkClaimProvider (exclusion 'distributed', leases, fencing). Each claim is an empty-tree commit whose message is the JSON record, at refs/squad/claims/<sha1(work key)> on a shared remote (configurable; fallback refs/heads/squad-claims/). Acquire, renew (fast-forward, epoch + 1), expiry takeover with skew margin, and release (CAS delete) all push with --force-with-lease, so the remote decides every race. No working tree, index, or HEAD changes; remote claims are fetched into refs/squad-claims-cache/. Corrupt records yield 'unavailable'. Git runs via an injectable GitTransport; the default spawns git with argument arrays (no shell), and remotes/ref prefixes are validated against option injection. Records carry a random nonce so one holder writing twice in the same second never reproduces its own commit SHA (found by the contract suite). Tests run the shared contract suite against a real temp bare repo with one repository per executor, plus a mutation control that swaps in plain --force and must observe more than one winner. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(cli): opt-in work claims for squad watch execute and fleet-dispatch (ADR-002) - `--claims <none|git-ref>` flag and `watch.claims` config (string or { provider, ttlMs, skewMarginMs, remote, refPrefix }); flag wins; default none keeps today's behavior. `claims` is a reserved watch key; invalid values fail instead of silently disabling claims. - execute and fleet-dispatch acquire a claim per issue and dispatch only won issues; held/unavailable are skipped (fail closed). Claims renew every ttlMs/3; a fenced renewal, or renewals failing until expiresAt - skew, kills the agent process tree. Claims are released in finally with completed/cancelled/abandoned. - execute applies needs:* capability routing like fleet-dispatch. - Per-round dispatched set: hybrid execute + fleet-dispatch never dispatch the same issue in one round, whatever provider is configured. - fleet-dispatch launches copilot asynchronously so it can be renewed/killed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(cli): claimed-work prompt contract for Ralph watch agents (ADR-002) When an exclusive claim provider (exclusion !== 'none') is active, execute and fleet-dispatch prompts tell the agent to work only its claimed issues, not self-assign or take over others' work, and stop on issues it cannot complete (the watcher releases the claim). Prompts are unchanged under 'none'. Adds a Claimed Work section to the ralph-instructions.md template. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(claims): lease bounds, record plausibility, capped git output, fixed claim identity (ADR-002 security review) - claims.ts: MIN/MAX_CLAIM_TTL_MS, MAX_CLAIM_SKEW_MARGIN_MS, MAX_CLAIM_EPOCH, claimTimingError(), claimRecordImplausibility(); assertClaimTtl caps ttl at 24h; providers cap skewMarginMs at 1h; InMemoryClaimProvider takes over implausible records (parity with git-ref); hostname fallback for machineId is published as a salted pseudonym h-<sha256(hostname, salt)[:12]>. - claims-git.ts: transport caps stdout (1 MiB default, 64 KiB for claim commits), never throws in 'close'; claim commits with a non-empty tree, more than one parent, oversized bodies, or implausible epoch/times are taken over with CAS on the observed sha and a WARN naming the ref; commit-tree runs with a fixed squad-claims <squad-claims@invalid> author/committer and a +0000 date from the injected clock. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(watch): enforce lease bounds, kill agent process groups, fail closed on malformed config (ADR-002 security review) - config: watch.claims timing must satisfy claimTimingError (30s <= ttlMs <= 24h, skewMarginMs <= 1h, ttlMs > 2 * skewMarginMs); a .squad/config.json that exists but is not valid JSON throws WatchConfigError; unknown watch.claims keys are warned about. - claims: acquireClaims fails closed and holdClaimsWhile refuses (releasing its claims, never running work) when timing would make the renew timer spin. - agent-spawn / fleet-dispatch: agents that may be fenced run as POSIX process-group leaders; killProcessTree kills the group (-pid, ESRCH fallback) and skips children that already exited (PID-reuse guard); detached groups are killed on watch exit; PidTracker kills groups too. - watch: hostname fallback identity is salted with the team root path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(claims): ADR-002 plausibility rule, lease bounds, claim identity; changeset (security review) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(cli): work-claim telemetry for squad watch (ADR-002) Per-round and since-startup claim counters (acquired, contended, unavailable, fenced, lost, released by outcome, implausible takeover) surfaced in the round board, watch --health, and OTel counters squad.watch.claims.* (claim.provider, claim.outcome). Only when the provider excludes other executors; 'none' output unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(watch): double-pick regression suite with mutation controls (ADR-002) Adds end-to-end git-ref coverage over real clones of one bare repository: 8-clone acquire race plus late retries, two ExecuteCapability executors dispatching disjoint issues (with a provider-none overlap control), cross-clone fencing with injected clocks, and hybrid execute + fleet-dispatch same-round dedupe under none, in-memory, and git-ref claims. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(watch): multi-machine work claims guide; mark cross-machine skill unimplemented (ADR-002) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(watch): renew claims that went stale during acquisition; de-flake double-pick suite (ADR-002) Root cause of the double-pick flakes under the full parallel suite: 1. "two executors dispatch disjoint issue sets" (A [1,2,3,4], B [4]) was not a CAS hole. Instrumented runs under load show B's claim on #4 was a fresh epoch-1 root claim pushed after A's release had deleted the ref: A's whole round (4 acquires, 300 ms agent, release) finished while B's contended acquires (~3 s each) were still walking its stale issue list. Claims only promise that no two executors hold an issue at the same time. The race now keeps each agent (and so its claims) alive until both executors have finished every acquire, so disjointness is a strict exclusion check again. 2. The fencing test bounded total runtime from test start (< 15 s) and its stub agent lived 30 s from spawn; on a loaded host B's takeover alone can take longer. The agent now lives 30 s past B's takeover and writes a marker only if it does, so the test proves the fence killed it without a wall-clock threshold. TTL 9 s (renew every 3 s) gives B's takeover a wide CAS window; per-test timeout raised to 120 s for real git under load. Product gap found while analysing (1): acquisition is sequential, so a slow or hanging remote can carry an early claim past expiresAt - skewMarginMs (or past takeover) before dispatch, and the agent started on an issue another watcher held until its first renewal fenced it. acquireClaims now renews such claims before returning and drops them as held (fenced) or unavailable (lost). Regression tests in watch-claims.test.ts are red without the fix ("A dispatched issues: ... expected [ 1, 2 ] to deeply equal [ 2 ]"). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(watch): keep run-bound mocks compatible with claims Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9 * fix(watch): close claim fencing review gaps Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9 * fix(watch): terminate escaped POSIX descendants Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9 * test(release): classify noop claim contract skips Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9 * fix(watch): release late renewed claims When cleanup races a renewal that already won remotely, inspect the fenced claim, advance to the newer epoch owned by this executor, and release that current handle. Late renewal responses also run idempotent compensating release without blocking forever. Adds a deterministic delayed-response regression proving no live claim remains after work completes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: B'Elanna (Squad) <tamirdresher@users.noreply.github.com> Copilot-Session: 995eea17-c356-4912-90ea-5e5b144e7ba9
1 parent c713a82 commit d7a4bd3

53 files changed

Lines changed: 7018 additions & 202 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@bradygaster/squad-sdk': patch
3+
'@bradygaster/squad-cli': patch
4+
---
5+
6+
Mark the `cross-machine-coordination` skill template as a design proposal that is not implemented: no Squad release reads `.squad/cross-machine/` task files, the `cross_machine` config block, or `squad:machine-*` labels. The skill now points to `squad watch` work claims (`--claims git-ref`, ADR-002) for keeping several watchers off the same issue, and to `needs:*` capability routing for hardware-specific work.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@bradygaster/squad-cli': patch
3+
---
4+
5+
`squad watch` work-claim telemetry (ADR-002). With an excluding claim provider (`git-ref`), each round report adds a compact line such as `🔒 Claims (git-ref): won 2 · contended 1 · released 2 (completed 2)` when any claim counter is non-zero: acquired, contended (held elsewhere), unavailable, fenced, lost (not renewed before expiry), released by outcome (completed / cancelled / abandoned), and implausible-record takeovers. `squad watch --health` shows the totals since startup from the watch PID file, and the counters are emitted as OpenTelemetry counters `squad.watch.claims.*` (attributes `claim.provider`, `claim.outcome`) when `OTEL_EXPORTER_OTLP_ENDPOINT` or `SQUAD_TELEMETRY=1` is set. No issue numbers, hostnames, or actors are recorded. With `none`, watch output is unchanged.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@bradygaster/squad-cli': patch
3+
'@bradygaster/squad-sdk': patch
4+
---
5+
6+
When `squad watch` runs with an exclusive claim provider (for example `--claims git-ref`), the `execute` and `fleet-dispatch` prompts now tell the agent that the listed issues are claimed for it: work only on those issues, do not pick up, self-assign, or take over other issues, and stop on an issue it cannot complete (the watcher releases the claim when the session ends). With `--claims none` (the default) the prompts are unchanged. The `ralph-instructions.md` template gains a matching "Claimed Work" section stating that claims are authoritative and assignees and labels are informational mirrors (ADR-002). Existing `.squad/ralph-instructions.md` files are user-owned and are not changed by `squad upgrade`.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@bradygaster/squad-cli': patch
3+
---
4+
5+
`squad watch` work claims (ADR-002): never dispatch on a claim that went stale while the rest of the batch was being acquired. Claims are acquired one issue at a time, so a slow or unreachable remote could carry an early claim past its renew deadline (`expiresAt - skewMarginMs`) — or past the point where another watcher may take it over — before the agent started, and the agent briefly worked an issue another watcher held until its first renewal fenced it. `execute` and `fleet-dispatch` now renew such claims before dispatch and skip the issue when the renewal is fenced (counted as `fenced`) or fails (counted as `lost`).

‎.changeset/watch-work-claims.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@bradygaster/squad-cli': minor
3+
---
4+
5+
`squad watch` can claim issues before dispatching them, so several watchers (machines, pods, dev boxes) never work on the same issue (ADR-002). Opt in with `--claims git-ref` or `watch.claims` in `.squad/config.json` (`"git-ref"` or `{ "provider": "git-ref", "ttlMs", "skewMarginMs", "remote", "refPrefix" }`); the flag overrides the config provider. The default, `none`, keeps today's behavior. With claims on, `execute` and `fleet-dispatch` dispatch only issues they won, skip issues held elsewhere or when the claim remote is unreachable, renew every `ttlMs / 3` (default 15 min lease, 2 min skew), kill the agent when its claim is fenced, and release claims when the agent finishes. `execute` now also applies `needs:*` machine capability routing, and one watch round never dispatches an issue twice across `execute` and `fleet-dispatch` (hybrid mode), whatever provider is configured. `claims` is now a reserved `watch` config key; invalid values stop watch with an error instead of being ignored.

‎.changeset/work-claims-contract.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@bradygaster/squad-sdk': minor
3+
---
4+
5+
Add the work claim contract from ADR 002 as `@bradygaster/squad-sdk/ralph/claims`: `WorkClaimProvider` (with declared `guarantees`), `WorkKey` and canonical `workKeyToString`, `ExecutorIdentity` with `resolveExecutorIdentity` (`SQUAD_MACHINE_ID` > capabilities machine > `SQUAD_POD_ID` > hostname, one `instanceId` per process), `NoopClaimProvider` (no exclusion, today's behavior), and `InMemoryClaimProvider` (single-process leases, epochs, expiry takeover, and handle fencing). `getPodId()` now accepts an optional `env` argument.

‎.changeset/work-claims-git-ref.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@bradygaster/squad-sdk': minor
3+
---
4+
5+
Add `GitRefClaimProvider` as `@bradygaster/squad-sdk/ralph/claims-git`: a distributed `WorkClaimProvider` (ADR 002) that stores each claim as a JSON record in an empty-tree commit at `refs/squad/claims/<sha1(work key)>` on a shared remote and decides every race with `git push --force-with-lease` compare-and-swap. Acquire, renew (fast-forward, epoch + 1), expiry takeover with skew margin, and release (CAS delete) never touch the working tree or index. The remote, ref namespace (including the `refs/heads/squad-claims/` fallback), clock, and git transport are configurable; the default transport spawns `git` with argument arrays and no shell.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@bradygaster/squad-sdk': patch
3+
'@bradygaster/squad-cli': patch
4+
---
5+
6+
Harden ADR-002 work claims after security review. Lease timing is bounded (30 s ≤ `ttlMs` ≤ 24 h, `skewMarginMs` ≤ 1 h, `ttlMs` > 2 × `skewMarginMs`) so the renew timer can never spin; `squad watch` rejects other values at startup and refuses to run work under them. Claim records with an impossible epoch or time, a non-empty tree, several parents, or a body over 64 KiB are taken over as expired with a warning naming the ref, and git output read from claim refs is capped. Claim commits use a fixed `squad-claims <squad-claims@invalid>` identity and a UTC date, and a hostname-derived `machineId` is published as a salted pseudonym. Fenced agents are killed with their whole POSIX process group, never after they have exited, and a `.squad/config.json` that does not parse now stops `squad watch` instead of silently disabling claims.

‎.squad-templates/ralph-instructions.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,20 @@ MAXIMIZE PARALLELISM — spawn agents for ALL actionable issues simultaneously.
5252
Work on every open, unblocked, unassigned issue labeled `squad` or `squad:{member}`.
5353
Skip issues that are assigned to a human, blocked, or marked `status:on-hold`.
5454

55+
### Claimed Work
56+
57+
When `squad watch` runs with a claim provider (for example `--claims git-ref`), the
58+
issues listed in your prompt are claimed for this session (ADR-002):
59+
- Work ONLY on the claimed issues in your prompt. Do not pick up other issues, even if
60+
they look actionable.
61+
- Claims are authoritative; assignees and labels are informational mirrors. Do not
62+
self-assign or take over issues claimed or assigned elsewhere.
63+
- If you cannot complete a claimed issue, stop and comment why. The watcher releases
64+
the claim when your session ends.
65+
66+
With `--claims none` (the default) there is no cross-watcher exclusion and Issue
67+
Selection above applies as written.
68+
5569
### Post-Task Actions
5670

5771
<!-- Uncomment and customize to add post-task hooks, e.g. Teams notifications:

‎.squad-templates/skills/cross-machine-coordination/SKILL.md‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
---
22
name: "cross-machine-coordination"
3-
description: "Enables squad agents on different machines to share work via git-based task queuing"
3+
description: "Design proposal, not implemented: git-based task queuing between squad machines. For competing watchers, use squad watch work claims (ADR-002)."
44
domain: "orchestration"
5-
confidence: "medium"
5+
confidence: "low"
66
source: "manual"
77
---
88

@@ -11,7 +11,26 @@ source: "manual"
1111
**Skill ID:** `cross-machine-coordination`
1212
**Owner:** Ralph (Work Monitor)
1313
**Squad Integration:** All agents
14-
**Status:** Specification (ready for implementation)
14+
**Status:** Design proposal — NOT implemented
15+
16+
> ⚠️ **Not implemented.** No Squad release reads `.squad/cross-machine/`,
17+
> the `cross_machine` config block, or `squad:machine-{name}` labels. Ralph
18+
> and `squad watch` do not poll task files, run whitelisted commands, or
19+
> write result files. Everything below describes a proposed design, not
20+
> current behavior. Do not tell users it works, and do not create task files
21+
> expecting another machine to pick them up.
22+
>
23+
> **Use these instead:**
24+
>
25+
> - **Several machines or pods running `squad watch --execute` on one
26+
> repository:** enable work claims (`squad watch --execute --claims
27+
> git-ref`, or `watch.claims` in `.squad/config.json`). Claims are Git-ref
28+
> leases that make sure only one watcher works on each issue. See ADR-002
29+
> (`docs/specification/adr-002-work-claims-and-external-queues.md`) and the
30+
> "Multi-Machine Watch with Work Claims" docs scenario
31+
> (`docs/src/content/docs/scenarios/multi-machine-watch.md`).
32+
> - **Sending work to a machine with specific hardware:** use capability
33+
> routing with `needs:*` issue labels and a machine capabilities file.
1534
1635
---
1736

@@ -53,7 +72,7 @@ git commit -m "Cross-machine task: GPU voice cloning [squad:machine-devbox]"
5372
git push origin main
5473
```
5574

56-
Ralph on DevBox will:
75+
In this proposal (not implemented), Ralph on DevBox would:
5776
1. Pull the task on next cycle (5-10 min)
5877
2. Validate schema & command whitelist
5978
3. Execute the GPU workload
@@ -64,12 +83,12 @@ Ralph on DevBox will:
6483

6584
### For Task Executors (DevBox, Azure VMs)
6685

67-
Ralph automatically watches `.squad/cross-machine/tasks/` for work targeted at this machine.
86+
In this proposal (not implemented), Ralph would watch `.squad/cross-machine/tasks/` for work targeted at this machine. Ralph does not do this today.
6887

69-
**On each cycle (5-10 min):**
88+
**On each cycle (5-10 min), as proposed:**
7089

7190
```python
72-
# Pseudo-code (Ralph implementation)
91+
# Pseudo-code (proposed Ralph implementation; does not exist)
7392
1. git pull origin main
7493
2. Load all .yaml files in .squad/cross-machine/tasks/
7594
3. Filter for status=pending AND target_machine=HOSTNAME
@@ -96,7 +115,7 @@ gh issue create \
96115
--label "urgent"
97116
```
98117

99-
Ralph on DevBox will:
118+
In this proposal (not implemented), Ralph on DevBox would:
100119
1. Detect issue with `squad:machine-devbox` label
101120
2. Parse task from issue body
102121
3. Execute task
@@ -204,7 +223,7 @@ All tasks go through:
204223

205224
## Configuration
206225

207-
Ralph reads config from `.squad/config.json`:
226+
Proposed `.squad/config.json` block. No Squad release reads it:
208227

209228
```json
210229
{
@@ -387,10 +406,10 @@ tail -f .squad/log/ralph-watch.log | grep "cross-machine"
387406

388407
## Integration with Ralph Watch
389408

390-
Ralph automatically includes this pattern in its watch loop:
409+
Not implemented: `squad watch` does not include this pattern. The proposed watch loop was:
391410

392411
```
393-
Ralph Watch Cycle (every 5-10 min):
412+
Proposed Ralph Watch Cycle (every 5-10 min):
394413
1. Fetch GitHub issues with squad:machine-* labels
395414
2. Poll .squad/cross-machine/tasks/
396415
3. For each matching task:
@@ -402,7 +421,7 @@ Ralph Watch Cycle (every 5-10 min):
402421
5. Sleep until next cycle
403422
```
404423

405-
No manual Ralph configuration needed — just create task files or issues with the right labels.
424+
Until this is built, task files and `squad:machine-*` labels do nothing. To stop several watchers from picking the same issue, use work claims (`--claims git-ref`, ADR-002).
406425

407426
---
408427

@@ -414,7 +433,7 @@ No manual Ralph configuration needed — just create task files or issues with t
414433
- user copies output back
415434
- user pastes result manually
416435

417-
**After (with this pattern):**
436+
**After (if this proposal is built):**
418437
- Laptop Ralph writes task file → git push
419438
- DevBox Ralph auto-executes → git push result
420439
- Laptop Ralph auto-reads result

0 commit comments

Comments
 (0)