Record the headless trial traps from #400, #404 and #405 in tm-ab-test

[sv-tmueller]

Part of batch #417

What to build

The three trials of 2026-09-28 lost runs, money, or fix rounds to the same operational traps, and tm-ab-test mentions none of them. Add a headless-run checklist to the tm-ab-test skill. Each item gives the trap, its fix, and the report section it came from, so the next trial avoids it up front. The next batch's main work is two such trials (a second native-vs-kickoff pair, and a paired /code-review run), so this lands first.

Sources to extract from:

• A/B: native plan-and-execute vs the kickoff pipeline on real issues #400: docs/reviews/2026-09-28-ab-native-vs-kickoff.md ("The 11 deviations", "Cost-measurement gap") and its protocol file.
• A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404: docs/reviews/2026-09-28-lead-effort-comparison.md (sections 3, 4 and 8) and its protocol file.
• A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405: docs/reviews/2026-09-28-ultracode-arm-379.md (sections 3 and 5) and its protocol file ("Amendment 1", "Amendment 2").

Acceptance criteria

• tm-ab-test's SKILL.md has a headless-run checklist. Each item states the trap, the fix, and its source report and section.
• It covers at least:
  • every plugin copy disabled (including orchestrai@synced) and verified in the init event
  • dontAsk silently denying writes under .claude/
  • the 600 s background-task ceiling cutting off a workflow
  • total_cost_usd and modelUsage being cumulative across a --resume chain (take the last result event per session, don't sum)
  • modelUsage vs token-report.mjs's 3.8-4.4x-low estimate
  • a judge checkout that reproduces the runs' filesystem layout
  • the Glob allowlist gap
• The 600 s item names the documented way to keep a headless session waiting for a background workflow, with a docs link. If none is documented, it says so with the check date.
• npm test is green, and version in .claude/.claude-plugin/plugin.json is bumped (the change touches .claude/skills/).

Non-goals

• A new script or code.
• Re-running any trial.
• Editing the three reports or any frozen protocol file.

[sv-tmueller]

Sub-plan (architect, batch #417)

SUB_PLAN

Plan source: none. #418's body only has scope and acceptance criteria, and batch #417's P2 section repeats the same text. There are no signed-off decisions, no docs/plans/ file and no linked spec, so this is the full checkpoint. Nothing conflicts with docs/architecture/. The provider-portability spec already marks this SKILL.md as host-bound, so Claude Code CLI details belong in it. Size check: fits size:S (details at the end).

Approach

• Add one unnumbered section, ## Headless claude -p checklist, to .claude/skills/tm-ab-test/SKILL.md. Put it after ## 6. Cleanup and before ## Report to the user. Do not renumber anything.
• Open the section with one scope sentence: it applies whenever an arm, a probe or a judge pass runs as its own claude -p process, as in all three trials (A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 protocol §11 item 3). It must not redefine the skill's existing headless mode (the lead driving the arm in its own session) or add a new mode. Leave the frontmatter description and argument-hint alone.
• Add two one-line pointers that lead into the section:
  • Step 3: before launching any arm or judge as a separate claude -p process, walk the checklist below.
  • Step 4: for a claude -p arm, record cost per the checklist's cost items.
• Group the items as Before launch (1-3, 7-10), Recording cost (4-5) and Judging (6). Each item is Trap, then Fix (including the gate that proves the fix held), then Source (report file and section). Keep each to 3-5 lines.
• Use placeholders ($TRIAL, $CLAUDE_CONFIG_DIR, <reference merge sha>), not this machine's paths or settings values. No real provider host.
• Word the token-report workflow blind spot so it stays true whichever of Token report: include workflow subagents and add a per-agent wall-clock table #416 and Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418 merges first.
• Leave the templates alone. templates/recording-checklist.md uses token-burn-analyze.mjs, which gives an estimate. The SKILL.md cost items stand on their own; a measured-cost field in the template would be a separate follow-up.

Items. The 7 required ones come first, then 3 extras. They are listed here so the reviewer reads them as planned, not as scope creep.

1. Plugin copies.
   • Trap: disabling only orchestrai@orchestrai left the account-synced copy orchestrai@synced loaded. Either key alone failed in diagnostics.
   • Fix: one --settings JSON that sets every orchestrai key the init event's plugins[] reports to false, merged with any other settings keys. Gate: the init event (type: system, subtype: init) lists zero orchestrai: skills or agents. If the gate fails, fix and relaunch before the run counts.
   • Source: A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 report "The 11 deviations" items 1 and 5; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §3 item 1; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Probe result", "Amendment 1" item 1, and "Arm run 1" (gates).
2. dontAsk and .claude/.
   • Trap: .claude/ is a built-in protected path. dontAsk denies every write there, no allow rule overrides it, and the run still ends RC 0 completed. A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 run 1 had 7 of 9 Writes and its only Edit denied, committed nothing, and was invalid ($3.91).
   • Fix: use --permission-mode auto for any session that may write .claude/, and keep dontAsk for read-only sessions. Never use bypass flags. Gate: zero Write or Edit denials on .claude/ paths in the result event's permission_denials. Docs: https://code.claude.com/docs/en/permission-modes
   • Source: A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §1 and §3 item 3; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Arm run 1" and "Amendment 2". Related: A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 "The 11 deviations" item 8 (the judges needed auto).
3. 600 s background ceiling.
   • Trap: Workflow runs as a background task. When the main turn ends with a background task still running, -p waits 600 s and then terminates.
     • In A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 run 2 the review workflow was killed at 643 s. The lead never read its result, so one confirmed finding and the planned fix-and-commit were lost. About 600 of the run's 1,996 s were idle.
     • Signature: the stderr line Background tasks still running after 600s; terminating. Set CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 to wait indefinitely., plus a task_notification with status: stopped just before the final result event.
     • Judge passes are exposed too: tm-review-changes took 651 s in A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405, and Verify stage in tm-review-changes, with spec-format item transforms #406 adds a verify stage.
   • Fix: set the ceiling variable inside the env -i wrapper. Anything set outside it is dropped, which is the trap A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 deviation 2 hit with GH_CONFIG_DIR. When the value is 0, keep an outer timeout. Report a run that shows the stderr line as cut short.
   • Source: A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §5; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Arm run 2 result"; A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 "The 11 deviations" item 5 (a headless session does wait for background subagents, and the ceiling caps that wait).
4. Cost is cumulative across --resume.
   • Trap: each invocation's result event carries total_cost_usd and modelUsage for the whole session so far, so adding them up across invocations double-counts.
   • Fix: take the last result event per session id, and add only across distinct ids. A kickoff resume after a limit death starts a new id; A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 arm A had two. The top-level usage block is per invocation and covers the lead only. modelUsage also covers subagents.
   • Source: A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 report "Per-pair comparison" (the paragraph after the "Cost, three ways" table, and "How adjusted R was computed" step 1) and "Cost-measurement gap"; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Arm run 2 result", cost bullets.
5. modelUsage vs token-report.mjs.
   • Trap: the script estimates output as visible characters divided by 4 and leaves out thinking. Measured output was 3.76x the estimate on A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 arm A and 4.40x on arm B, so the cost ratio came out 0.62 on the estimate vs 0.71 measured. A version that does not read subagents/workflows/ also misses workflow agents entirely: $4.28 reported vs $12.77 measured in A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405.
   • Fix: the measured figure from item 4 is the cost of record. Report the script's number only as a labeled estimate, and never compare one arm's estimate with the other arm's measured figure. Before a workflow run, check that the script in use reads subagents/workflows/. Optional, my own inference: enforce spend caps against measured cost, since A/B: native plan-and-execute vs the kickoff pipeline on real issues #400's $35 stop rule (protocol §10) used the estimate.
   • Source: A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 report "Cost-measurement gap" and "How adjusted R was computed"; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §4 "Cost cross-check".
6. Judge checkout layout.
   • Trap: A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404's judge copy nested _snapshot/ inside judge-repo/, but in the runs it sat next to repo/. The judge marked H6's true path claim false, which cost H6 its only lost point.
   • Fix: build the judge's checkout so every relative path the runs saw resolves the same way. Otherwise, normalize those paths during redaction and log each replacement.
   • Source: A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 report §8, "Judge-checkout layout mismatch".
7. Glob allowlist gap.
   • Trap: --allowedTools grants Glob by name, with no path limit (unlike Bash(...) entries). In A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 round 1, H2 and H3 globbed one level above the trial root and reached a sibling root, leaving 0 of 3 runs valid. Worse, from A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405: a bare Read or Grep in --allowedTools bypasses permissions.blockReadsOutsideWorkingDirectories, and a diagnostic read the reference solution that way.
   • Fix: set blockReadsOutsideWorkingDirectories: true in --settings. Leave Read and Grep out of --allowedTools, since in-directory reads are still approved; Glob can stay because it respects the block. As a second layer, keep the trial root's parent directory empty except for this run. Expect some Bash safetyCheck denials on computed paths (19 in A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 run 2); they are friction, not a failure.
   • Source: A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 report §3 and §8; A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 protocol "Amendment 1" ("Not applied"); A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §3 item 2; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol §3 "Known gap" and "Amendment 1" item 2.
8. Extra: reference reachable through refs.
   • Trap: A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405's clone kept local main at a commit after the merge, so git show main:<path> exposed the reference. Path-based gates cannot see that.
   • Fix: keep only the detached base. Gate: git rev-list --all | grep -qx <reference merge sha> must find nothing.
   • Source: A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 report §3 item 4; A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 protocol §2 (clone recipe and gate).
9. Extra: spend-limit 429 mid-run.
   • Trap: it happened in all three trials (A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 arm A run 2, A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 H1, one A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 diagnostic). The restart overhead biases the arm that died.
   • Fix: decide in advance whether a 429 run is invalid-not-replaced or resumed. Record the reset time, leave the wait out of wall-clock, name the bias in the report, and commit each run's record before the next run starts.
   • Source: A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 "The 11 deviations" item 7 and "Limit event"; A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 report §3; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Amendment 1", last paragraph.
10. Extra: effort override.
    • Trap: the config dir's settings.json can keep a per-model effortLevel, and stream-json has no effort field.
    • Fix: pass --effort explicitly and pin modelSettings in --settings. Gate on perTurnEffort in $CLAUDE_CONFIG_DIR/projects/<slug>/<sid>.jsonl: every value matches and at least one exists. Ultracode records as xhigh.
    • Source: A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 protocol "New gates per high run" and report §8 "Context"; A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Documented ultracode syntax" and §7.

The 600 s item: docs lookup

• What the repo has: the only source is the CLI's own stderr line (build 2.1.280), recorded in A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 protocol "Arm run 2 result". No file in the repo cites a docs page for it.
• What I checked on 2026-10-01: the installed CLI (2.1.286) still contains CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS and the same message. Its code accepts any finite value of 0 or more. I have no WebFetch, so I could not check the docs site.
• The developer should WebFetch these and search each for PRINT_BG_WAIT, background and 600:
  • https://code.claude.com/docs/en/env-vars (if that page 404s, the "Environment variables" section of https://code.claude.com/docs/en/settings)
  • https://code.claude.com/docs/en/headless
  • https://code.claude.com/docs/en/workflows
  • https://code.claude.com/docs/en/cli-reference
• If one of them documents a way: cite it and quote the line.
• If none does: write "No documented way found (checked YYYY-MM-DD: env-vars, headless, workflows, cli-reference)". Still give the variable as the fix, label it undocumented and taken from stderr, and pair it with an outer timeout.
• Do not claim that a prompt line such as "wait for the workflow" works. No trial tested it.

Files and order

1. WebFetch the docs first, because the wording of item 3 depends on the result.
2. .claude/skills/tm-ab-test/SKILL.md: the new section and the two pointers.
3. .claude/.claude-plugin/plugin.json: bump from 2.7.0 (the author picks the level). P1 and P3 also bump, so rebase on the version line if either merges first.

Tests and checks that constrain the change

• No test reads the body of tm-ab-test.
• effort-policy.test.mjs reads every SKILL.md's frontmatter model: and fails if it names fable. tm-ab-test has no model: line; do not add one.
• The hermes-prompts, role-contract-sync and prompts-sync tests do not read skills.
• No writing-style lint exists, so checking for em dashes and banned phrases is manual.
• scripts/check-version-bump.mjs (the CI job version-guard, PR-only) fails a change under .claude/skills/ that has no version bump.

Verification

• npm test exits 0.
• node scripts/check-version-bump.mjs origin/main HEAD exits 0.
• git diff --stat origin/main lists only the two files, and git diff origin/main -- docs/reviews/ is empty.
• The new section has no U+2014 and no banned phrases.
• Each of the 7 AC bullets maps to one item with a trap, a fix and a source.
• Item 3 has a docs URL or the dated "not documented" line.

Size check: fits size:S. The work is one section of roughly 45-60 lines in one file, a one-line version bump, and 3-4 WebFetch calls; the sources are already extracted above. It stops fitting S if the developer re-reads the roughly 2,950 report lines or edits the templates.

[sv-tmueller]

Package summary (batch #417): PR #420 ready for review

• Pipeline: architect sub-plan; developer; tester PASS (a3e01d4); reviewer CHANGES_REQUESTED at spec stage, fix round 1/3 (1 must-fix, 2 should-fix, 2 nits, all applied verbatim, see batch Decision 4); tester re-test PASS (117bc1f); reviewer re-review APPROVE at quality stage (117bc1f).
• Checks at 117bc1f: npm test 383 pass; node scripts/check-version-bump.mjs origin/main HEAD exit 0 (2.7.0 -> 2.7.1).
• Should-fix for the human review: none open.
• Nits for the human review (not applied, verbatim replacement text in the re-review comment on PR docs: add headless claude -p checklist to tm-ab-test #420):
  1. SKILL.md:254-255: the Source bullet of the modelUsage item should also cite "A/B: native plan-and-execute vs the kickoff pipeline on real issues #400 protocol section 10 ($35 stop rule)".
  2. SKILL.md:177-178: "Set it inside the env -i wrapper" names a wrapper the skill never introduces; the reviewer gives a conditional rewording.
• Untested claims: none for the acceptance criteria. The skill is prose, so no test pins its content; testers and the reviewer checked each cited section by hand, and the 600 s docs pages were fetched on 2026-10-01.
• Merge note: plugin.json is also bumped by PR feat: verify stage in tm-review-changes with spec-format item transforms #419 (Verify stage in tm-review-changes, with spec-format item transforms #406) and PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 (Token report: include workflow subagents and add a per-agent wall-clock table #416). Whichever merges first, the others need a one-line rebase on the version.