Batch: measurement-tools

[sv-tmueller]

Approved contract (sign-off: dispatch, 2026-10-01). Raw need: "advise on the next issues to tackle. Consider the latest changes."

Refinement answers (owner, 2026-10-01)

• Token report: include workflow subagents and add a per-agent wall-clock table #416 speed: wall-clock per agent, drop tokens per second. Interactive transcripts carry only streaming placeholders for output_tokens, so a rate would rest on the chars/4 estimate, which leaves out thinking.
• Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415 effort: the owner typed /effort in a session, which wrote modelSettings.claude-opus-5-5.effortLevel: xhigh into ~/.claude-work/settings.json (2026-10-01 09:29). So /effort does write a per-model override back.
• Hermes: the provider credential is still blocked. Hermes adapter: swap spawn from delegate_task to the hermes subprocess #351, Make SKILL.hermes.md a thin wrapper over the pipeline driver #355 and PR feat: spawn Hermes seats via subprocess instead of delegate_task #354 stay out of this batch.

Proposal: Batch: measurement-tools

Why this batch: every recent study had to work around broken measuring tools. This batch fixes the token report, the A/B harness and the review workflow, plus one small doc cleanup. The next batch can then run the two measurements that could change policy.

Four packages, all independent of each other. P1, P3 and P4 are existing issues. On dispatch their bodies are updated to the scope below and get the batch line. P2 is new.

P1 (#416, rescoped): Token report includes workflow subagents and adds a per-agent wall-clock table

Scope: token-report.mjs reads the lead transcript and subagents/agent-*.jsonl, but never subagents/workflows/<wf-id>/. So a workflow's whole cost goes missing without a warning ($4.28 reported vs $12.77 measured in #405). This package makes the report read those transcripts too, and say which transcript directories it found but couldn't read. It also adds a per-agent table so slow seats are visible. Speed is shown as measured wall-clock, not tokens per second. Interactive transcripts carry only streaming placeholders for output_tokens (checked 2026-10-01: values 5/8/16 across 32 messages), so a tokens-per-second figure would rest on an estimate that leaves out thinking.

Acceptance criteria:

• The report includes workflow subagents under subagents/workflows/, grouped by role and by model like every other agent. A fixture test covers a session with a workflow.
• The report names every subagent transcript directory it found but could not read.
• A per-agent table lists role, model, calls, tool uses, estimated output tokens, and wall-clock time for each agent (the lead, each dispatched seat, each workflow subagent).
• The report states where each wall-clock value comes from (for example, duration_ms in a task notification, or the first-to-last timestamp span in a transcript), and marks any estimated value.
• Tests are written first, against fixture transcripts. They include an agent with no output and an agent with a zero or missing duration, which shows as "n/a" (no division by zero, no made-up value).
• /tm-advisor section 5 and /tm-kickoff's wave-end report still post the report on every run, with the new table.
• npm test is green, and plugin.json version is bumped.

Size: M. Track: full (executable code under .claude/skills/).
Non-goals: a tokens-per-second column; pricing headless claude -p child sessions; real billing figures; live or streaming throughput.

P2 (new): Record the headless trial traps from #400, #404 and #405 in tm-ab-test

Scope: The three trials of 2026-09-28 lost runs, money, or fix rounds to the same operational traps, and tm-ab-test mentions none of them. This package adds a checklist section to the skill. Each item gives the trap, its fix, and the report section it came from, so the next trial avoids it up front. These trials are the next batch's main work, so this lands first.

Acceptance criteria:

• tm-ab-test's SKILL.md has a headless-run checklist. Each item states the trap, the fix, and its source report and section.
• It covers at least:
  • every plugin copy disabled (including orchestrai@synced) and verified in the init event
  • dontAsk silently denying writes under .claude/
  • the 600 s background-task ceiling cutting off a workflow
  • total_cost_usd and modelUsage being cumulative across a --resume chain (take the last result event per session, don't sum)
  • modelUsage vs token-report.mjs's 3.8-4.4x-low estimate
  • a judge checkout that reproduces the runs' filesystem layout
  • the Glob allowlist gap
• The 600 s item names the documented way to keep a headless session waiting for a background workflow, with a docs link. If none is documented, it says so with the check date.
• npm test is green, and plugin.json version is bumped.

Size: S. Track: full (touches .claude/skills/).
Non-goals: a new script or code; re-running any trial; editing the three reports or any frozen protocol file.

P3 (#406, as filed plus one line): Verify stage in tm-review-changes, with spec-format item transforms

Scope: Unchanged from the issue. It adds an items_transform field with the must_fix_deduped reducer in both renderers, and fixes the items_cap lookup bug. It also adds an adversarial worker-tier verify stage between review and consolidate, a refuted array and appendix in the report, and over-cap findings reported rather than dropped. All seven surfaces change together. The one live /tm-review-changes run on the PR's own diff is included in this dispatch, as the owner pre-approved on 2026-09-28. The single addition: that run also records its wall-clock time. tm-review-changes already ran past 600 s in #405, and a verify phase makes it longer, which matters for headless judging.

Acceptance criteria: all eight from the issue, plus:

• The PR body records the live run's wall-clock time beside the confirmed count, refuted count, and cost.

Size: M. Track: full.
Non-goals: as filed (no P2 port to tm-review-codebase, no expression language, no severity floor, no A/B rerun).

P4 (#415, two criteria settled): Tidy up after batch #403

Scope: Unchanged edits to the team-guide note and the lead-effort report nits. Two criteria change to match what is now known:

• The owner step is done. settings.json shows modelSettings.claude-opus-5-5.effortLevel: xhigh as of 2026-10-01, which has the same effect as removing it.
• The owner confirmed that typing /effort writes a per-model override into settings.json. So the team-guide note must say that a refine-phase /effort high persists across sessions until /effort xhigh is typed again.

Acceptance criteria:

• The first eight criteria from the issue (team-guide "dispatch" or "file only", plus the seven report and data.json nits).
• Owner step: already done. The developer confirms that a session log written after 2026-10-01 09:29 shows perTurnEffort xhigh, or says none exists yet.
• The team-guide note says that /effort high writes a persistent per-model override (owner-confirmed 2026-10-01), and that typing /effort xhigh before "dispatch" or "file only" is what undoes it.

Size: S. Track: lean (only team-guide.md and docs/reviews/, no guarded paths, no code; the reviewer runs the check suite).
Non-goals: as filed (no frozen protocol edit, no tiers.lead, agent or test change, no new trials, no score changes).

How the run goes

• Up to 3 packages run at once. P3 (the longest) starts first, alongside P1 and P2. P4 starts when a slot frees up.
• P1, P2 and P3 all bump plugin.json. Whichever merges first, the other two each need a one-line rebase on the version.
• Planned decision, logged here: the installed token-report.mjs can't see workflows, and P3's live run is a workflow. If P1's PR is ready by report time, the batch token report runs from P1's branch so that run is counted, and the comment says so.
• Cost: roughly $30-50 at list price for the pipeline (for reference, batch Batch: lead-effort-and-ultracode #403's two packages cost $37.71), plus a few dollars for P3's live run.

Deferred to the next batch, and why

• Auto-must-fix severity floor in review prompts #407 severity floor: edits the same prompt files as Verify stage in tm-review-changes, with spec-format item transforms #406, and its body says to merge Verify stage in tm-review-changes, with spec-format item transforms #406 first.
• review agent orchestration #344 P2 (verify stage in tm-review-codebase): waits for Verify stage in tm-review-changes, with spec-format item transforms #406 to merge.
• Second native-vs-kickoff pair (T1 replay): waits for P1 and P2, so the cost and harness are fixed first. Two pairs are what the frozen rule needs before it can give a verdict.
• Paired run of local /code-review vs the reviewer quality pass, on seeded known-flaw diffs. This is the audit's "thin" reviewer row. Same reason as above.

Left out, with a recommendation

• Lead-effort run-phase measurement: not filed (the lead's share of calls is small, and Opus is back on xhigh).
• Desk study: Jev as a typed decision oracle for routing #370 (Jev desk study): the Batch: lead-effort-and-ultracode #403 numbers already point to no-go on routing savings. Recommended: close it with that comment. Owner's call.
• Audit's HEAD-repair thinning: keep the step. It's a two-command backstop, and the case for removing it rests on docs, not observation.
• Hermes (Hermes adapter: swap spawn from delegate_task to the hermes subprocess #351, Make SKILL.hermes.md a thin wrapper over the pipeline driver #355, PR feat: spawn Hermes seats via subprocess instead of delegate_task #354): parked until the credential works.
• Owner housekeeping: mark PR docs: use a placeholder provider URL in the hermes config example #353 ready and merge it (the real host is still on main). Also mark PR docs: survey Claude-of-Tanks orchestration and plan the adoption #345 ready and merge it, so the plan that Verify stage in tm-review-changes, with spec-format item transforms #406 and Auto-must-fix severity floor in review prompts #407 cite is on main.

Packages

• P1 Token report: include workflow subagents and add a per-agent wall-clock table #416 - Token report includes workflow subagents and adds a per-agent wall-clock table (PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 merged as 74fc2b0)
• P2 Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418 - Record the headless trial traps from A/B: native plan-and-execute vs the kickoff pipeline on real issues #400, A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 and A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 in tm-ab-test (PR docs: add headless claude -p checklist to tm-ab-test #420 merged as 12e8c0f)
• P3 Verify stage in tm-review-changes, with spec-format item transforms #406 - Verify stage in tm-review-changes, with spec-format item transforms (PR feat: verify stage in tm-review-changes with spec-format item transforms #419 merged as 43cee0b)
• P4 Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415 - Tidy up after batch Batch: lead-effort-and-ultracode #403 (PR docs: tidy lead-effort note and report nits after batch 403 #422 merged as 2bdf754)

Decision log

Decisions made during the run are posted as comments below.

[sv-tmueller]

Decision 1 (lead, 2026-10-01): run under main's 2.7.0 rules and agents.

The installed plugin is 2.4.0, while main is at 2.7.0. The repo's tm-kickoff and tm-advisor skills are newer than the installed copies: they add the plan-source pointer in architect dispatches (#395), the empty-or-stalled-return resume rule (#397), and the LESSONS: channel (#398). The repo's architect, tester and reviewer agent files also differ from the installed ones. This run follows main's .claude/skills/tm-kickoff/SKILL.md and .claude/skills/tm-advisor/SKILL.md, and dispatches the project-level architect, developer, tester and reviewer agents (main's versions), not the orchestrai: plugin copies. Owner follow-up, outside this batch: run /plugin update so the installed copy matches main.

Run start (token report anchor): 2026-10-01T08:04:28.000Z.

[sv-tmueller]

Decision 2 (lead, 2026-10-01): how the #406 live run loads the workflow.

The #418 architect flagged a finding recorded in docs/reviews/2026-09-28-ultracode-arm-379-protocol.md ("tm-review-changes result"): the Workflow runtime refused tm-review-changes.js both by name and by path, because its export const meta is computed from SPEC rather than written as a literal. The same pattern is in all three production workflows (tm-map-codebase.js, tm-review-changes.js, tm-review-codebase.js). No issue covers it yet.

For the #406 live run (lead step, after tester PASS), the lead first calls the Workflow tool on the #406 branch checkout's .claude/workflows/tm-review-changes.js by path, with the branch checkout as cwd and base: origin/main (sub-plan L1, L2). If the runtime refuses it for the computed meta, the lead runs a scratch copy with meta hoisted to a literal first statement and every other byte unchanged, the same adaptation #405 used, and says so in the PR body. This measures the branch's stages and prompts unchanged, so it stays within the live-run acceptance criterion.

Not fixed in this batch: changing the meta pattern touches all three workflows and their spec tests, which is outside every package's scope. It goes in the batch report as the first follow-on candidate.

[sv-tmueller]

Decision 3 (lead, 2026-10-01): #416 wall-clock fallback, and a correction to a sign-off fact.

1. Fallback rule, accepted. The Token report: include workflow subagents and add a per-agent wall-clock table #416 architect's sub-plan uses duration_ms from task notifications as the measured source, and falls back to the transcript's run-segment span, marked est., when a seat's notifications carry a zero or missing duration. It shows n/a only when no source gives a positive value. Reasoning: the acceptance criteria require that every value's source is stated, that estimates are marked, and that a zero or missing duration never produces a division by zero or a made-up value. A transcript span is a derived measurement, not a made-up value, and a rule using only duration_ms would show n/a for every seat killed by a limit, which are the slow seats the table exists to show. The AC fixture (an agent with zero duration and no span) still shows n/a.
2. Correction. The sign-off said "interactive transcripts carry only streaming placeholders for output_tokens". That holds for subagent transcripts (what the lead checked). The architect found that lead transcripts appear to carry real values (115 distinct values up to 4,886 across 118 messages in one session, not yet checked against a result event). No scope change: the per-agent output column stays an estimate, as the AC says. Pricing the lead from its real output_tokens goes in the batch report as a follow-on candidate.

[sv-tmueller]

Decision 4 (lead, 2026-10-01): #418 fix round 1 carries all five reviewer findings, not only the must-fix.

The reviewer returned CHANGES_REQUESTED at STAGE: spec on PR #420: 1 must-fix, 2 should-fix, 2 nits, each with verbatim replacement text. The fix round forwards all five unchanged. Reasoning: a spec-stage round never qualifies as lead-verified, so a re-test and re-review follow regardless and including the extra four adds no round. Should-fix 2 and 3 correct factual claims (the #404 round-1 cause, and the split of the #405 $4.28 vs $12.77 gap) in a checklist whose whole value is that its claims are accurate. Within the signed-off scope; no AC change.

[sv-tmueller]

Decision 5 (lead, 2026-10-01): where the #406 live run executes.

The branch's workers get the diff with git diff ${base}...HEAD in the session's working directory, and the workflow takes no path argument. To review PR #419's own diff with the branch's own workflow file (sub-plan L1, L2), the lead detaches its own main checkout at the PR head 57416579 for the run, calls the Workflow tool on that checkout's .claude/workflows/tm-review-changes.js with args: { base: "origin/main" }, checks afterward that HEAD did not move during the run, then returns the checkout to main. EnterWorktree was not used: its rules allow it only when the user or project instructions ask for a worktree. The other in-flight agents work in their own isolated worktrees.

[sv-tmueller]

Decision 6 (lead, 2026-10-01): token-prices.json has no Sonnet 5.5 entry; handled in reporting, not in #416.

While pricing the #406 live run, the token report listed claude-sonnet-5-5 as unpriced and left it out of the dollar total. .claude/skills/tm-kickoff/token-prices.json (main, and PR #421's branch) prices only claude-opus-5-5 and claude-sonnet-5, but the sonnet alias now resolves to claude-sonnet-5-5, so every developer, tester and workflow-worker call since that change is unpriced. The report does name the model under Limitations, so the gap is not silent, but the total understates spend.

Not folded into #416: a price-table change is outside its signed-off scope (workflow subagents and the per-agent table). Handling for this batch: the live-run cost on PR #419 and this batch's token report price Sonnet 5.5 at the published list rates (https://platform.claude.com/docs/en/about-claude/pricing, retrieved 2026-10-01: $2 input, $2.50 5m cache write, $4 1h cache write, $0.20 cache read, $10 output per MTok, identical to the Sonnet 5 entry), and each says so. The batch report lists adding the entry as a follow-on.

[sv-tmueller]

Decision 7 (lead, 2026-10-01): #415 fix round 1 carries reviewer findings 1-5; finding 6 is met in the new commit instead of a history rewrite.

The lean reviewer returned CHANGES_REQUESTED on PR #422: 1 must-fix (the team-guide note no longer instructs the switch back to /effort xhigh), 2 should-fix, 3 nits. Same reasoning as Decision 4: a re-review follows regardless (a spec-stage round is never lead-verified), and findings 2-3 correct text the acceptance criteria cover. Finding 6 asks for a why in commit 2e1179c's body; rewriting a pushed commit is not worth a force push, so the fix commit's body carries the why instead, and the squash-merge message can too.

[sv-tmueller]

How this report was produced (read first):

• Script: token-report.mjs from PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421's branch at a8e2393 (unmerged, run read-only), per the contract's planned decision, so the Verify stage in tm-review-changes, with spec-format item transforms #406 live workflow run is counted. Main's script does not read subagents/workflows/.
• Price table: the branch's token-prices.json plus one entry the lead added for this run only, claude-sonnet-5-5 ($2 input, $2.50 5m cache write, $4 1h cache write, $0.20 cache read, $10 output per MTok, from the same source page retrieved 2026-10-01; batch Decision 6). Without it, every Sonnet 5.5 call is unpriced and the total drops to the Opus figure.
• Window: from the batch issue's creation (2026-10-01T08:05:00Z). One session, no resume. Refinement before the batch issue is not included.
• Not covered: nothing. No headless child sessions ran in this batch.

Token report

• Session: aaf4a664-3ff2-4c66-bc90-c2eeee2051d1
• Window: from 2026-10-01T08:05:00Z
• Price source: https://platform.claude.com/docs/en/about-claude/pricing (retrieved 2026-09-27)

By role

Role	Calls	Input	Cache read	Cache write (5m)	Cache write (1h)	Output (est.)
lead	87	200	22,539,029	0	193,495	40,779
architect	84	168	6,565,339	419,965	0	19,841
developer	170	340	11,351,476	459,311	0	59,298
reviewer	126	252	6,044,866	389,686	0	14,071
tester	115	230	4,734,773	258,830	0	16,065
workflow:tm-review-changes	48	96	2,350,084	396,304	0	5,838

By model

Model	Calls	Input	Cache read	Cache write (5m)	Cache write (1h)	Output (est.)	Cost
claude-opus-5-5	308	642	35,860,595	897,864	193,495	77,235	$14.76
claude-sonnet-5-5	322	644	17,724,972	1,026,232	0	78,657	$6.90
Total	630	1,286	53,585,567	1,924,096	193,495	155,892	$21.66

By agent

Start (UTC)	Role	Model	Calls	Tool uses	Output (est.)	Wall-clock	Source
2026-10-01 08:05:01	lead	claude-opus-5-5	87	82	40,779	42m 3s (est.)	lead first-to-last span
2026-10-01 08:06:41	architect	claude-opus-5-5	17	40	4,331	7m 9s	task notification duration_ms
2026-10-01 08:06:41	architect	claude-opus-5-5	32	43	9,501	12m 40s	task notification duration_ms
2026-10-01 08:06:41	architect	claude-opus-5-5	35	51	6,009	8m 55s	task notification duration_ms
2026-10-01 08:14:47	developer	claude-sonnet-5-5	56	81	21,246	6m 54s	task notification duration_ms
2026-10-01 08:16:36	developer	claude-sonnet-5-5	16	28	6,318	1m 46s	task notification duration_ms
2026-10-01 08:18:37	tester	claude-sonnet-5-5	23	30	2,494	1m 51s	task notification duration_ms
2026-10-01 08:20:30	developer	claude-sonnet-5-5	50	83	22,585	7m 8s	task notification duration_ms
2026-10-01 08:20:53	reviewer	claude-opus-5-5	30	35	3,010	4m 58s	task notification duration_ms
2026-10-01 08:21:58	tester	claude-sonnet-5-5	36	43	5,452	3m 11s	task notification duration_ms
2026-10-01 08:26:44	developer	claude-sonnet-5-5	9	14	1,286	40s	task notification duration_ms
2026-10-01 08:27:43	tester	claude-sonnet-5-5	12	15	1,159	40s	task notification duration_ms
2026-10-01 08:28:09	tester	claude-sonnet-5-5	28	40	5,393	2m 41s	task notification duration_ms
2026-10-01 08:28:42	workflow:tm-review-changes (review:bugs)	claude-sonnet-5-5	7	8	604	47s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:scope)	claude-sonnet-5-5	6	7	985	36s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:security)	claude-sonnet-5-5	4	4	479	21s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:tests)	claude-sonnet-5-5	5	6	364	22s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:docs)	claude-sonnet-5-5	8	8	612	27s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:perf)	claude-sonnet-5-5	3	3	69	7s (est.)	transcript span
2026-10-01 08:28:43	workflow:tm-review-changes (review:style)	claude-sonnet-5-5	4	5	181	13s (est.)	transcript span
2026-10-01 08:29:04	reviewer	claude-opus-5-5	14	17	1,648	2m 22s	task notification duration_ms
2026-10-01 08:29:30	workflow:tm-review-changes (consolidate)	claude-opus-5-5	11	14	2,544	2m 26s (est.)	transcript span
2026-10-01 08:31:22	developer	claude-sonnet-5-5	11	12	2,888	1m 2s	task notification duration_ms
2026-10-01 08:32:53	tester	claude-sonnet-5-5	16	18	1,567	1m 4s	task notification duration_ms
2026-10-01 08:35:29	reviewer	claude-opus-5-5	22	25	2,413	4m 59s	task notification duration_ms
2026-10-01 08:35:29	reviewer	claude-opus-5-5	26	38	2,714	4m 35s	task notification duration_ms
2026-10-01 08:35:29	developer	claude-sonnet-5-5	17	21	2,921	1m 24s	task notification duration_ms
2026-10-01 08:37:15	reviewer	claude-opus-5-5	17	21	2,410	4m 32s	task notification duration_ms
2026-10-01 08:42:38	developer	claude-sonnet-5-5	11	13	2,054	1m 1s	task notification duration_ms
2026-10-01 08:43:57	reviewer	claude-opus-5-5	17	18	1,876	2m 0s	task notification duration_ms

Limitations

• Output token counts are not in the transcripts; the "Output (est.)" column is estimated from visible text and tool-call input, divided by 4.
• The estimate excludes thinking tokens, so it undercounts real output token usage.
• Prices are Anthropic public list prices, not your actual billing (discounts, batch pricing and negotiated rates are not reflected).
• Wall-clock: "task notification duration_ms" is the time the harness measured for a dispatched seat. "transcript span" is the first-to-last timestamp span of each run in a transcript, used for workflow subagents and for a seat with no positive measured duration. "est." marks a span. The lead row spans its first to last line in the window, so it includes idle and waiting time. "n/a" means no source gave a positive value.
• Tool uses are counted from tool_use blocks in each transcript.

[sv-tmueller]

Batch report: measurement-tools

All four packages reached a ready PR. None parked. No limit events, no run-long Sonnet fallback: every architect and reviewer verdict ran on Opus, so no Opus re-review is owed.

Packages

• Token report: include workflow subagents and add a per-agent wall-clock table #416 token report: PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 ready. It now reads workflow subagents under subagents/workflows/ and adds a per-agent table. Wall-clock is measured from task notifications where available, else a transcript span marked est., else n/a (Decision 3). Tester round 1 found agent hex ids leaking through the "Found but not read" line, and a newline splitting a table row; both fixed with tests. Reviewer APPROVE, both passes clean, 2 nits. Its first real use is this batch's token report.
• Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418 tm-ab-test checklist: PR docs: add headless claude -p checklist to tm-ab-test #420 ready. Adds a headless claude -p checklist with 10 items (the 7 required plus 3 extras), each with trap, fix and source. The 600 s ceiling turned out to be documented: CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS, with the env-vars and headless docs cited. Reviewer round 1 (spec) caught an operator rule nobody asked for, plus two factual over-claims; all five findings were applied verbatim (Decision 4). Re-test PASS, re-review APPROVE with 2 nits.
• Verify stage in tm-review-changes, with spec-format item transforms #406 verify stage: PR feat: verify stage in tm-review-changes with spec-format item transforms #419 ready. This is the must_fix_deduped transform plus the items_cap fix in both renderers, an adversarial verify stage, the refuted and unverified report fields, and over-cap findings reported. Tester PASS with no fix rounds; its mutations confirm that the tests pin the dead-verifier, overflow and refuted-strip behavior. The live run on its own diff: approve, confirmed 0, refuted 0, $1.76, 195,901 ms. No must-fix came up, so verify did not fire; the verify path is proven by tests only. Reviewer APPROVE with one should-fix recommended before merge (below).
• Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415 tidy-up: PR docs: tidy lead-effort note and report nits after batch 403 #422 ready (lean track). Reviewer round 1 caught that the rewrite had dropped the instruction to switch back to /effort xhigh, the very step that was forgotten before. Fixed (Decision 7), then the re-review approved with both passes clean. Owner step confirmed: sessions after the settings change show only perTurnEffort: xhigh.

Decisions made during the run

1. Ran under main's 2.7.0 skills and the project-level agents. The installed plugin is 2.4.0.
2. Verify stage in tm-review-changes, with spec-format item transforms #406's live run loads a hoisted-meta copy if the runtime refuses the computed meta. It did refuse; the copy differs from the branch file only in the moved meta block.
3. Token report: include workflow subagents and add a per-agent wall-clock table #416 wall-clock fallback accepted. Correction: lead transcripts appear to carry real output_tokens; only subagent transcripts carry placeholders.
4. Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418 fix round 1 carried all five reviewer findings, not only the must-fix.
5. Verify stage in tm-review-changes, with spec-format item transforms #406 live run executed in the lead checkout, detached at the PR head; HEAD was checked unchanged afterward.
6. token-prices.json has no claude-sonnet-5-5 entry. Handled in reporting only (live-run cost and token report), not in Token report: include workflow subagents and add a per-agent wall-clock table #416.
7. Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415 fix round 1 carried findings 1-5. Finding 6 (commit body) was met in the new commit instead of a history rewrite.

Should-fix before merge (recommended)

• PR feat: verify stage in tm-review-changes with spec-format item transforms #419: finalizeReport strips refuted findings from mustFix but keeps the critic's verdict, so a report can read mustFix: [] with changes-requested. The verdict still depends on the model, which the sub-plan set out to remove. Both the live run and the reviewer raised it. The reviewer's exact change, 1 line plus 2 tests, is in its comment on PR feat: verify stage in tm-review-changes with spec-format item transforms #419.

Merge order and version bumps

PRs #419, #421 and #420 each bump plugin.json from 2.7.0 (to 2.8.0, 2.8.0 and 2.7.1), so the second and third merges each need a one-line rebase. Suggested order: #422 any time (no bump); then #419 (2.8.0); then #421 (rebase to 2.9.0); then #420 (rebase to 2.9.1).

Follow-on candidates for the next batch

1. Bug, highest priority: the Workflow runtime refuses all three tm- workflows (tm-review-changes, tm-review-codebase, tm-map-codebase) by name and path, because their export const meta is computed from SPEC. First seen in A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405, reconfirmed on 2026-10-01 on Verify stage in tm-review-changes, with spec-format item transforms #406's branch. As shipped, the slash commands likely fail to load.
2. Add claude-sonnet-5-5 to token-prices.json (Decision 6), and consider pricing the lead from its real output_tokens (Decision 3).
3. Auto-must-fix severity floor in review prompts #407 (severity floor) and review agent orchestration #344 P2 (verify stage in tm-review-codebase), once feat: verify stage in tm-review-changes with spec-format item transforms #419 merges.
4. The two measurements this batch prepared, once docs: add headless claude -p checklist to tm-ab-test #420 and feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 merge: a second native-vs-kickoff pair, and a paired local /code-review vs reviewer-quality-pass run.
5. Untested on a live host: Hermes's fact-checker role returning a parseable {confirmed, note} for verify.

Flags for the owner

• Run /plugin update: the installed plugin is 2.4.0, while main is 2.7.0 (2.9.x after these merges).
• From the proposal: merge PR docs: use a placeholder provider URL in the hermes config example #353 (the real provider host is still on main) and PR docs: survey Claude-of-Tanks orchestration and plan the adoption #345 (the plan that Verify stage in tm-review-changes, with spec-format item transforms #406 and Auto-must-fix severity floor in review prompts #407 cite). The recommendation to close Desk study: Jev as a typed decision oracle for routing #370 stands.
• Cleanup: every worktree from this batch is removed and the lead checkout is clean on main. A stale worktree-agent-* branch from 2026-09-29, fully on origin, was deleted. The local branches docs/352-placeholder-provider-url and feat/351-hermes-subprocess-spawn were kept; they back your open draft PRs docs: use a placeholder provider URL in the hermes config example #353 and feat: spawn Hermes seats via subprocess instead of delegate_task #354.

Lessons (tester and reviewer)

• Lesson (Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418, tester): check each cited number against the source's own definition (an estimate versus a measurement) before copying it into a checklist item.
• Lesson (Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418, reviewer): a sub-plan's authoring constraints can leak into the deliverable as rules for its readers; check that instructions meant for the developer did not end up in the output.
• Lesson (Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418, tester): a re-test of a verbatim fix round is cheapest as a line-by-line diff against the reviewer's quoted replacement texts, then a spot-check of each cited source.
• Lesson (Verify stage in tm-review-changes, with spec-format item transforms #406, tester): a worktree-isolated tester cannot run heredoc or node -e scripts; write scratch scripts to a file with printf and run the file.
• Lesson (Token report: include workflow subagents and add a per-agent wall-clock table #416, tester): a "never print identifier X" rule needs a test per output path (unread paths, labels, errors), not only the main table.
• Lesson (Verify stage in tm-review-changes, with spec-format item transforms #406, reviewer): a sub-plan that moves ownership of a list field from the model to the script should also name the fields derived from it (here the verdict).
• Lesson (Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415, reviewer): when two acceptance criteria are merged into one sentence, check each one on its own afterwards; rewording an instruction as a description quietly drops the instruction.

What happened

• PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 ready - Token report includes workflow subagents and adds a per-agent wall-clock table (Token report: include workflow subagents and add a per-agent wall-clock table #416)
• PR docs: add headless claude -p checklist to tm-ab-test #420 ready - Record the headless trial traps from A/B: native plan-and-execute vs the kickoff pipeline on real issues #400, A/B: lead effort, Opus 5.5 at high vs xhigh, on the advisor refine task #404 and A/B arm: live ultracode on Opus 5.5 on a replayed merged issue #405 in tm-ab-test (Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418)
• PR feat: verify stage in tm-review-changes with spec-format item transforms #419 ready - Verify stage in tm-review-changes, with spec-format item transforms (Verify stage in tm-review-changes, with spec-format item transforms #406)
• PR docs: tidy lead-effort note and report nits after batch 403 #422 ready - Tidy up after batch Batch: lead-effort-and-ultracode #403 (Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415)
• Token report: Batch: measurement-tools #417 (comment) ($21.66 list price, output estimated)
• Lesson (Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415, reviewer): rewording an instruction as a description quietly drops the instruction (6 more lessons above)

Next steps

1. Review & merge: docs: tidy lead-effort note and report nits after batch 403 #422, feat: verify stage in tm-review-changes with spec-format item transforms #419, feat: token report covers workflow subagents and adds a per-agent wall-clock table #421, docs: add headless claude -p checklist to tm-ab-test #420, in that order, with the version rebases above (merging closes each package issue via its Closes #N). Consider the feat: verify stage in tm-review-changes with spec-format item transforms #419 should-fix first.
2. Close this batch issue by hand once everything is merged (it is the run record), or run /tm-advisor to confirm the merges, close it, and propose the next batch (follow-on 1, the computed-meta bug, first).

[sv-tmueller]

Owner request (2026-10-01): apply the #419 should-fix (recompute the verdict in finalizeReport, the reviewer's exact change), then merge all four PRs (#422, #419, #421, #420, with the plugin.json version rebases) and close this batch.

[sv-tmueller]

Closing: all packages merged

Owner request (2026-10-01): apply the #419 should-fix, merge all four, close the batch.

• PR docs: tidy lead-effort note and report nits after batch 403 #422 merged as 2bdf754 (closes Tidy up after batch #403: team-guide effort note, report nits, Opus back on xhigh #415).
• PR feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 merged as 74fc2b0 (closes Token report: include workflow subagents and add a per-agent wall-clock table #416), plugin 2.8.0.
• PR docs: add headless claude -p checklist to tm-ab-test #420 merged as 12e8c0f (closes Record the headless trial traps from #400, #404 and #405 in tm-ab-test #418). The lead merged main into the branch and resolved the version to 2.8.1 (1cd07fa); CI green before merge.
• PR feat: verify stage in tm-review-changes with spec-format item transforms #419 merged as 43cee0b (closes Verify stage in tm-review-changes, with spec-format item transforms #406). Before merge: reviewer fix round 1/3 for the should-fix (69800c2: finalizeReport recomputes the verdict to approve when only refuted findings were removed, plus two tests; the new test fails on the old code). Then the lead merged main and resolved the version to 2.9.0 (93bb0b1). Re-test PASS and re-review APPROVE at 93bb0b1, CI green. Two nits left: one more guard test (finalizeReport with empty mustFix and nothing refuted keeps the critic's verdict; checked by hand, not pinned by a test), and PR-body staleness, applied by the lead before merge.
• main is at 2.9.0. No package parked, so no Part of batch lines needed stripping.

Follow-on candidates for the next batch (from the report above, updated):

1. Bug: the Workflow runtime refuses all three tm- workflows because their export const meta is computed from SPEC.
2. token-prices.json lacks claude-sonnet-5-5; optionally price the lead from its real output_tokens.
3. Auto-must-fix severity floor in review prompts #407 (severity floor) and review agent orchestration #344 P2 (verify stage in tm-review-codebase): both are unblocked now that feat: verify stage in tm-review-changes with spec-format item transforms #419 has merged.
4. The second native-vs-kickoff pair and the paired /code-review run: both are unblocked now that docs: add headless claude -p checklist to tm-ab-test #420 and feat: token report covers workflow subagents and adds a per-agent wall-clock table #421 have merged.
5. Small: the finalizeReport guard test (nit 1 of the feat: verify stage in tm-review-changes with spec-format item transforms #419 re-review); Hermes fact-checker {confirmed, note} on a live host.

Cleanup: the lead checkout is on main at 43cee0b, clean. Two scratch agent worktrees were kept by the origin-safety rule: .claude/worktrees/agent-a626d5b68e2009dc6 (69800c2) and .claude/worktrees/agent-ac44e4ae0c397f8f1 (93bb0b1). Their commits left origin when the PR branch was deleted after the squash merge. Both are clean, and 93bb0b1's tree is identical to main's 43cee0b, so nothing is lost by removing them by hand.