fix(ci): select mutation fixtures when suites change

[davidfarah2003]

Summary

• retain each mutation fixture suite path while loading the corpus
• select fixtures when their config, suite, or guarded source intersects the changed set
• add positive and negative suite-only selector controls and mutation coverage

Validation

• minimal public CLI repro: 1 fixture selected and KILLED
• selector smoke: 12 passed, 0 failed
• mutation proof: 7/7 KILLED with tree restored
• historical accounting: PR test(manager,connector-core): make the mutation reproof gate grade what it claims #1329 head missed 8 suite-only fixtures; merged main adds the separate parent fixture and misses 9

[davidfarah2003]

Retrospective verdict: BLOCK

Exact head: 4d43c9799e240b6940562c74c39ad87b732c29ec.

I re-confirmed the live PR head immediately before the verdict. Current main 5e635a22c66c1efbe5921c1ebd112d6dd351b386 is its parent. The PR changes exactly three files, +55/-7, and git diff --check passes.

What passed

The selector implementation and focused acceptance behavior are coherent:

• loadCorpus() retains the configured suite path.
• Selection is an exact set membership check on config path, suite path, or mutation.file, not a substring match.
• The selector smoke invokes the shipped scripts/mutation-reproof.mjs through its SCAN path.
• The exact-head smoke passes 12/12.
• In a scratch public-CLI repro, the old selector exited 0 with 0 of 2 fixtures selected after a suite-only change.
• The fixed selector selected only the fixture naming that suite, excluded the unrelated fixture and an unrelated config-shaped file, killed the selected mutant, and reported one discriminated fixture.
• All seven mutation anchors are unique. Removing suite selection reddens the named suite-only selector cell.
• In the historical projection, user-spawn-signal-readiness.json is selected through both the new suite arm and its pre-existing guarded-file arm.

Blocking evidence gap

The exact-head CI run does not prove the claimed 16-fixture repaired projection and does not close the live-only events-remedy-user.json path.

Completed mutation-reproof job 101443500333 selected exactly one fixture:

mutation reproof: 1 fixture(s) selected from 351
selected fixture paths:
  bin/smoke/mutations/mutation-reproof.json

It killed that fixture's seven mutations. implementations/auth/smoke/mutations/events-remedy-user.json was not selected and has no terminal verdict at this head.

That omission is decisive for this review because events-remedy-user exercises the user-spawn throw-summary arm that #1329's own exact CI skipped. Its command ends in -live, which this review is prohibited from running locally. The acceptance requirement therefore required exact CI to close it.

Count reconciliation

Applying the original #1329 eight changed paths to the current-main corpus produces:

• 6 fixtures selected by the old config/guard selector
• 9 additional suite-only fixtures
• 15 total

The nine suite-only fixtures are:

• bin/smoke/mutations/boot-self-heal-gate.json
• bin/smoke/mutations/user-endpoint-cli.json
• implementations/auth/smoke/mutations/events-delegation.json
• implementations/auth/smoke/mutations/events-remedy-user.json
• implementations/auth/smoke/mutations/scope-default-full.json
• implementations/auth/smoke/mutations/user-spawn-second-name.json
• implementations/auth/smoke/mutations/view-regrant-whole-row.json
• implementations/manager/smoke/mutations/events-grant-acl.json
• implementations/manager/smoke/mutations/events-own-channel.json

The reported sixteenth is bin/smoke/mutations/mutation-reproof.json. It appears only when the three PR #1341 repair paths are unioned with the original #1329 changed paths. That is a valid projected set, but it is not what the exact PR changed job ran.

Required evidence

Add an exact CI job or checked-in acceptance case that invokes the fixed selector against the original #1329 changed-path set on the current-main corpus and executes all 16 projected fixtures. events-remedy-user.json must reach a terminal clean verdict. A green changed job over only the selector fixture does not establish this.

I did not read the peer verdict. I did not run mutation-proof in a shared tree, a local live or stack suite, cotal up/down, the dev-tree CLI, pnpm check, an install, or a build.

[davidfarah2003]

Independent first-hand APPROVE of PR #1341 at 4d43c97.

Head re-confirmed via git ls-remote. Parent main 5e635a2 is an ancestor. Three files. Isolated archive and scratch worktree, not the shared checkout.

Selector is exact Set.has against git --name-status -M paths: config path OR suite string OR mutation.file. No substring or path-prefix match.

Ran:

• Shipped selector smoke on the exact-head copy: 12 passed, 0 failed. scan() invokes scripts/mutation-reproof.mjs.
• Mutated away the suite arm in scratch. First red was a suite-only change selects exactly the fixture that names that suite. Config-path and mutation.file cells still passed. Restored; 12/12 again.
• Exact CI Mutation reproof / changed job 101443500333 SUCCESS: 1 fixture(s) selected from 351 (diff 3 record(s), 3 changed path(s), corpus 351); selected bin/smoke/mutations/mutation-reproof.json; 7/7 KILLED including the suite-arm mutant. Green alone was not treated as evidence.

Path probes:

• Prefix traps selected 0.
• Shared suite selected both fixtures.
• user-spawn.smoke.ts alone selects 7 fixtures. Six are suite-only. user-spawn-signal-readiness.json has suite === mutation.file, so both arms fire; a decoupled scratch copy of that name selected via suite intersection alone.
• ./ suite spelling vs git a.suite.mjs selected 0. Corpus at this SHA has 0 dotted/abs/backslash suite fields.

Historical projection on the #1329 file set: 6 old + 9 suite-only = 15 unique.

No -live, cotal up, or pnpm check.

[davidfarah2003]

Retrospective verdict update: APPROVE

Exact head: 4d43c9799e240b6940562c74c39ad87b732c29ec.

This supersedes my earlier evidence-hold BLOCK at the same SHA. I re-confirmed immediately before this update that both PR #1341 and proof PR #1342 point to the unchanged head above. Proof PR #1342 used temporary historical base d6a61b58a90d306f085f0fa5b6bb4f7a5a36a42b; that base is an ancestor of the head and recreates the intended 11-path historical changed set.

Hosted job 101445886135 completed successfully. I read its terminal log rather than inferring from the green check:

• exact job head: 4d43c9799e240b6940562c74c39ad87b732c29ec
• exact base: d6a61b58a90d306f085f0fa5b6bb4f7a5a36a42b
• exactly 16 fixtures selected from the 351-fixture corpus
• every selected path reached a terminal classification
• 14 fixtures discriminated
• 2 fixtures were explicitly PRE-RED
• 0 fixtures were inconclusive
• elapsed time: 2318 seconds within the 150-minute budget

The previously missing historical proof is now present: implementations/auth/smoke/mutations/events-remedy-user.json was selected, its baseline was green, and both mutations were killed.

The two pre-red fixtures were:

• implementations/auth/smoke/mutations/user-spawn-signal-readiness.json
• implementations/auth/smoke/mutations/view-regrant-whole-row.json

For each, the runner refused to grade mutations because pnpm smoke:user-spawn:live was already red before mutation. That is the intended nonfatal classification for an existing suite failure, not a false kill or an inconclusive result.

Together with my prior first-hand checks at this unchanged head, the completed historical proof closes the only remaining evidence hold. I did not rerun mutation-proof locally, install, build, start a stack, invoke a live suite, use the dev-tree CLI, or run pnpm check for this update.

[davidfarah2003]

Independent first-hand review of draft proof PR #1342 against frozen #1341 at 4d43c97.

I downloaded and parsed GitHub Actions logs for run 34018253649 / changed job 101445886135 (pull_request, full skipped, BASE d6a61b5). I did not run -live, cotal up, or pnpm check, and did not move the shared checkout.

Printed: 16 fixture(s) selected from 351 (diff 11 / 11 changed paths). Path list includes implementations/auth/smoke/mutations/events-remedy-user.json. That fixture ran: baseline green, 2/2 KILLED.

Job conclusion success. MUTATION REPROOF OK (16 selected; 14 discriminated, 2 pre-red, 0 inconclusive). 66 padded KILLED verdict lines. No SURVIVED / UNGRADABLE / WRONG-RED / ERROR verdict tokens. No MUTATION REPROOF FAILED.

PRE-RED (selected, then refused as already-red live suite, mutants not graded):

• implementations/auth/smoke/mutations/user-spawn-signal-readiness.json (pnpm smoke:user-spawn:live exit 1 before mutation)
• implementations/auth/smoke/mutations/view-regrant-whole-row.json (pnpm --filter @cotal-ai/auth build && pnpm smoke:user-spawn:live exit 1 before mutation)

Selector arithmetic on the exact candidate tree holds. Those two already-red suites are a #1329/main quantity, not a #1341 selector miss.

Independent APPROVE of #1341 at 4d43c97 still holds. Do not merge #1342.

[davidfarah2003]

Retrospective verdict correction: BLOCK

Exact merged head: 4d43c9799e240b6940562c74c39ad87b732c29ec.

I retract my approval at this SHA. The unchanged-head historical proof closed the original selection-evidence hold, but a subsequent whole-result feedback-loop pass found a composition failure between the new suite selector and the existing PRE-RED policy.

Reproduction

In clean, committed synthetic repositories running the shipped scripts/mutation-reproof.mjs, I tested two one-variable diffs:

1. Change only the configured suite so its baseline exits red.
2. Delete only the configured suite.

Both runs correctly selected the fixture through config.suite, but both exited 0:

mutation reproof: 1 fixture(s) selected from 1
REFUSING: `node a.suite.mjs` is red BEFORE any mutation (exit 1).
PRE-RED (1 fixture(s)) — suite already red before mutation, not this diff's defect
MUTATION REPROOF OK (1 fixture(s) selected; 0 discriminated, 1 pre-red, 0 inconclusive)

In these cases the suite failure is caused by the changed set itself. Calling it “not this diff's defect” and returning success is therefore a false green for the exact class this PR newly adds to selection.

Why existing proof missed it

The exact-head selector smoke passes 12/12 and proves:

• a green suite-only change selects its fixture
• an unrelated suite selects none
• a pre-red fixture selected because its guarded source changed remains nonfatal

It does not test a fixture selected because the configured suite itself changed and became red or disappeared. The seven mutation controls likewise remove suite selection entirely, but do not mutate the attribution/classification boundary after suite selection succeeds.

The hosted historical proof selected all 16 projected fixtures and reached 66 killed mutations, 2 pre-red fixtures, and no fatal or inconclusive verdicts. That proves the selector reaches the historical population. It does not exercise a suite-only diff that creates its own baseline failure.

Required repair

A selected fixture whose configured suite changed and is red at head must not be unconditionally dismissed as an unrelated pre-red. Attribute that state to the changed suite and fail, or compare base and head strongly enough to prove the red predates the diff. Add regression cells for both a changed suite becoming red and a configured suite being deleted, plus a mutation that removes the attribution check.

For this whole-result rerun I also verified the public suite-only CLI path selects and kills its fixture, shared-suite fanout selects both fixtures, rename/delete old paths enter the changed set, unrelated suite names do not select, the real 351-fixture corpus parses, package dry-run includes all three changed files, release metadata is unchanged, and the hosted Linux, Windows, unit, smoke, live, analysis, and 16-fixture historical jobs completed successfully. None covers this newly reproduced false-green boundary.

[davidfarah2003]

Independent retrospective BLOCK of merged PR #1341 at exact head 4d43c9799e240b6940562c74c39ad87b732c29ec. I retract my APPROVE at this SHA (comment 5557636594).

First-hand on an isolated archive of that commit, running the shipped scripts/mutation-reproof.mjs against committed synthetic two-fixture repos (suite + command, a vs b):

• Changing ONLY a.suite.mjs to fail selected smoke/mutations/a.mutations.json and exited 0 as PRE-RED (1 fixture(s)) — suite already red before mutation, not this diff's defect / MUTATION REPROOF OK.
• Deleting ONLY a.suite.mjs selected the same fixture and also exited 0 as PRE-RED / MUTATION REPROOF OK.
• Control: a still-green suite-only edit selected the same fixture and discriminated (exit 0).

The new suite selector is what admits these diffs. PRE-RED still treats a red baseline as not this diff's defect, including when the suite file is the only change. Shipped smoke covers suite-only green selection, unrelated-suite exclusion, and source-triggered PRE-RED. It does not cover suite-triggered PRE-RED or suite deletion.

Ran, not only read. No merge. Corrective follow-up needed: classify a changed-suite baseline failure as attributable/fatal, or compare base vs head, with both regression cells and mutation.