Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 42 additions & 3 deletions bin/smoke/mutation-reproof.smoke.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ function makeRepo(mutate: (root: string) => void): { root: string; base: string;
"",
].join("\n"));
writeFileSync(join(root, "smoke", "mutations", `${name}.mutations.json`), JSON.stringify({
suite: `${name}.suite.mjs`,
command: `node ${name}.suite.mjs`,
mutations: [{
name: `the ${name} cap is removed`,
Expand Down Expand Up @@ -228,7 +229,45 @@ try {
);
}

// 5. A selected fixture whose suite is ALREADY RED before any mutation. a.mjs changes (so a's
// 5. A suite-only change must select the fixture that names that suite, even though neither its
// config nor guarded source changed. The other suite is the negative arm: changing a file whose
// name merely looks like a suite must not select either fixture.
{
const { root, base, head } = build((r) => {
writeFileSync(join(r, "a.suite.mjs"), [
"// suite-only change; a.mjs and the fixture config are untouched",
"import { capped_a } from './a.mjs';",
"if (capped_a(100) !== 32) { console.error('✗ FAIL: the a cap holds'); process.exit(1); }",
"console.log('✓ the a cap holds');",
"",
].join("\n"));
git(r, ["add", "a.suite.mjs"]);
git(r, ["commit", "--quiet", "-m", "change only a's suite"]);
});
const { status, out } = scan(root, base, head);
check(
"a suite-only change selects exactly the fixture that names that suite",
status === 0
&& eq(selectedPaths(out), ["smoke/mutations/a.mutations.json"])
&& JSON.stringify(okCounts(out)) === JSON.stringify({ discriminated: 1, preRed: 0, inconclusive: 0 }),
`status=${status} selected=${JSON.stringify(selectedPaths(out))} counts=${JSON.stringify(okCounts(out))}\n${out}`,
);
}
{
const { root, base, head } = build((r) => {
writeFileSync(join(r, "unrelated.suite.mjs"), "// not named by any fixture\n");
git(r, ["add", "unrelated.suite.mjs"]);
git(r, ["commit", "--quiet", "-m", "change an unrelated suite"]);
});
const { status, out } = scan(root, base, head);
check(
"an unrelated suite selects no fixture",
status === 0 && selectedPaths(out).length === 0,
`status=${status} selected=${JSON.stringify(selectedPaths(out))}\n${out}`,
);
}

// 6. A selected fixture whose suite is ALREADY RED before any mutation. a.mjs changes (so a's
// fixture is selected) but a's suite fails unconditionally, so mutation-proof refuses at its
// baseline (exit 4). That is the suite's own defect, not this diff's — the #1279/sandbox-guard
// shape the reviewer reproduced. The gate must report it as PRE-RED, name exactly that fixture,
Expand Down Expand Up @@ -263,7 +302,7 @@ try {
);
}

// 6. INCONCLUSIVE (not a timeout — deterministic): a mutation that leaves the suite exiting 0 but
// 7. INCONCLUSIVE (not a timeout — deterministic): a mutation that leaves the suite exiting 0 but
// never printing its named assertion. mutation-proof grades that INCONCLUSIVE, "a green status
// is not a pass". The gate must report it as INCONCLUSIVE, not fail, and NOT collapse it into
// SURVIVED (false blocker) or KILLED (false clearance). This is the outcome the reviewer flagged
Expand Down Expand Up @@ -303,7 +342,7 @@ try {
);
}

// 7. A real all-clear that reaches the OK summary (a fixture whose mutant is genuinely killed).
// 8. A real all-clear that reaches the OK summary (a fixture whose mutant is genuinely killed).
// Its counts must read `discriminated > 0, pre-red 0, inconclusive 0` — the positive contrast to
// cases 5 and 6, so "the gate worked and everything was clean" is not the same output as "every
// selected fixture was pre-red or inconclusive".
Expand Down
10 changes: 9 additions & 1 deletion bin/smoke/mutations/mutation-reproof.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"suite": "bin/smoke/mutation-reproof.smoke.ts",
"guard": "changed guarded source re-proves its fixture; a deleted or renamed-away source is a loud dangling failure naming the exact fixture; both sides of a rename enter the changed set",
"guard": "changed fixture config, suite, or guarded source re-proves its fixture; a deleted or renamed-away source is a loud dangling failure naming the exact fixture; both sides of a rename enter the changed set",
"command": "pnpm smoke:mutation-reproof",
"grades": "tool",
"proveWith": "pnpm mutation-proof --config bin/smoke/mutations/mutation-reproof.json",
Expand All @@ -21,6 +21,14 @@
"expectRed": "the gate FAILS on a renamed-away guarded source and names exactly the dangling fixture",
"cell": "the gate FAILS on a renamed-away guarded source and names exactly the dangling fixture"
},
{
"name": "a changed configured suite is ignored by changed-set selection",
"file": "scripts/mutation-reproof.mjs",
"find": " || (typeof suite === \"string\" && changed.has(suite))\n",
"replace": "",
"expectRed": "a suite-only change selects exactly the fixture that names that suite",
"cell": "a suite-only change selects exactly the fixture that names that suite"
},
{
"name": "a dangling fixture is no longer refused, so a deleted source stops being a loud failure",
"file": "scripts/mutation-reproof.mjs",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"suite": "extensions/connector-claude-code/smoke/boot-wake-race.smoke.ts",
"guard": "a wake dropped in the connector's boot window (message already pending before the claude/channel handshake activates) is re-fired without a human turn, so the session is never permanently deaf",
"guard": "a wake dropped in the connector's boot window (message already pending before the claude/channel handshake activates) is re-fired by activation itself, so the session is never permanently deaf",
"command": "pnpm --filter @cotal-ai/connector-claude-code... build && pnpm smoke:claude-boot-wake",
"proveWith": "node scripts/mutation-proof.mjs --config extensions/connector-claude-code/smoke/mutations/boot-wake-race.json",
"why": [
Expand All @@ -9,34 +9,34 @@
"while proving nothing about the boot window, so the recovery has to be traced to the one line",
"that performs it.",
"",
"The connector itself has no reconcile and the cell asserts that too: activation only flips the",
"flag (hooks.ts setChannelActive), nudge() returns early while inactive, and the retry is",
"unreachable in that window because it is armed only from a REJECTED notify, which cannot happen",
"when nudge returns before notifying. So the whole recovery rests on ingest re-announcing the",
"still-pending item when JetStream redelivers it, and B1 deletes exactly that.",
"THAT LINE IS THE ACTIVATION RECONCILE, not the durable re-announce. This fixture used to mutate",
"the re-announce in `connector-core/src/agent.ts`, from a time when the suite's recovery ran",
"through JetStream redelivery. It does not any more: the suite settles each scenario in a 500ms",
"window chosen so that no redelivery can have happened, and its header says the redelivery guard",
"lives in `smoke:cross-path-dedup` and `smoke:claude-wake`. So the old mutation changed nothing",
"this suite observes, the suite passed with it applied, and it graded UNGRADABLE rather than",
"either verdict. The re-announce is now graded where it is driven, in",
"`extensions/connector-core/smoke/mutations/durable-redelivery.json`.",
"",
"REBUILD IS PART OF THE EXPERIMENT, NOT AN OPTIMIZATION. The suite loads the connector's own",
"wake policy by relative source path (../src/hooks.js) but reaches MeshAgent through the package",
"name, which resolves to dist/. Editing src/agent.ts alone therefore mutates bytes the run never",
"reads: measured here, the first attempt at B1 SURVIVED with the source visibly mutated, purely",
"because dist/ still held the pre-mutation build. Hence the build in `command` and the rebuild in",
"`afterRestore` — dist/ is gitignored, so putting the source back is not enough to leave the tree",
"honest.",
"What is left here is the recovery this suite really does drive: `setChannelActive(false → true)`",
"reconciling one buffered wake, with nothing else in the window able to produce a nudge. The",
"mention branch above it is left in place, so the #917 scenario still passes and this mutation",
"reddens only the buffered-wake cell it names.",
"",
"One mutation, deliberately: the guard is a single line, and B1 is its exact removal — the state",
"of this branch before the re-announce landed. A second contrived mutant would grade nothing the",
"first does not."
"REBUILD IS PART OF THE EXPERIMENT, NOT AN OPTIMIZATION. `command` builds the mutated source into",
"dist before each run, so `afterRestore` has to rebuild from the restored source; dist is",
"gitignored, and putting the source back is not enough to leave the tree honest."
],
"mutations": [
{
"name": "B1 a durable redelivery of a still-pending item refreshes the ack handle without re-announcing it",
"file": "extensions/connector-core/src/agent.ts",
"find": " existing.ack = delivery.ack;\n this.emit(\"incoming\", existing.item);",
"replace": " existing.ack = delivery.ack;",
"expectRed": "a wake lost in the boot window is re-fired without a human turn",
"cell": "a wake lost in the boot window is re-fired without a human turn (JetStream redelivery re-announces the still-pending item)",
"name": "B1 activation flips the flag without reconciling the wake that was already buffered",
"file": "extensions/connector-claude-code/src/hooks.ts",
"find": " else if (agent.pendingWake() > 0) nudge();",
"replace": "",
"expectRed": "the claude/channel handshake reconciles a buffered wake that was already pending",
"cell": "the claude/channel handshake reconciles a buffered wake that was already pending",
"afterRestore": "pnpm --filter @cotal-ai/connector-claude-code... build",
"note": "Restores the swallow this branch had before the re-announce: the redelivery arrives, the ack handle is refreshed, and nothing tells the wake policy the message is still there. The session then sits idle with the DM buffered and un-acked for as long as the cell waits."
"note": "Restores the #226 defect exactly: activation sets `channelActive` and stops there. `nudge()` returns early while inactive, and the retry timer is armed only from a REJECTED notify, which cannot happen when nudge returns before notifying. With this line gone nothing in the boot window wakes the session at all."
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@
"guard": "cotal_connection_status reports MeshAgent's live state, and its five states stay distinct",
"command": "pnpm smoke:connection-status",
"progressPattern": " ✓ ",
"completionMarker": "SUITE COMPLETE: 15 cells",
"completionMarker": "SUITE COMPLETE:",
"proveWith": "node scripts/mutation-proof.mjs --config extensions/connector-core/smoke/fixtures/connection-status.mutations.json",
"why": [
"The state a caller acts on is derived from three facts, so each collapse of that derivation gets its own mutation rather than one mutation standing in for the whole getter. A single mutation on connectionState would be killed by whichever cell ran first and would leave the other states ungraded.",
"M1 and M6 mutate the two liveness getters, which proves the tool reports live MeshAgent state rather than deriving or assuming it. M2 to M4 collapse one state into another, which proves each state is separately observable. M5 mutates the issue scoping, which is the only thing keeping a stopped session's post-mortem from being read as a current fault.",
"The full predicted kill set, including every exclusion, is recorded in the suite header before this config is run.",
"The suite imports connector-core source relatively, so mutation-proof executes the changed source directly and needs no build or afterRestore command."
"The suite imports connector-core source relatively, so mutation-proof executes the changed source directly and needs no build or afterRestore command.",
"The completion marker carries no cell count on purpose. Each of these suites prints `SUITE COMPLETE: ${ran} cells` and separately refuses a run whose `ran` is not its EXPECTED_CELLS, so a count in the marker duplicates a check the suite already makes and rots the moment a cell is added. It did: a cell landed in the suite, the marker still said 15, and every mutant here graded INCONCLUSIVE for a week because a finished run printed a line the marker could never match."
],
"mutations": [
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"guard": "real NATS transport edges reach CotalEndpoint and MeshAgent without flapping readiness",
"command": "pnpm --filter @cotal-ai/core build && pnpm smoke:transport-liveness:broker",
"progressPattern": " \u2713 ",
"completionMarker": "SUITE COMPLETE: 13 cells",
"completionMarker": "SUITE COMPLETE:",
"proveWith": "node scripts/mutation-proof.mjs --config extensions/connector-core/smoke/fixtures/transport-liveness-broker.mutations.json",
"why": [
"The unit transport-liveness suite proves the stop-versus-bind race by replacing connectAndBind",
Expand All @@ -23,7 +23,8 @@
"doRebuild's tearDownIfStopped. That helper is what prevents a rebuild that already emitted",
"connection:true (the shared tail ran before stop completed) from leaving nc+heartbeat+supervisor",
"live and from supervising a stopped endpoint. A shared-tail mutation is deliberately NOT used to",
"grade the rebuild cell."
"grade the rebuild cell.",
"The completion marker carries no cell count on purpose. Each of these suites prints `SUITE COMPLETE: ${ran} cells` and separately refuses a run whose `ran` is not its EXPECTED_CELLS, so a count in the marker duplicates a check the suite already makes and rots the moment a cell is added. It did: a cell landed in the suite, the marker still said 15, and every mutant here graded INCONCLUSIVE for a week because a finished run printed a line the marker could never match."
],
"mutations": [
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@
"guard": "raw NATS transport liveness is epoch-safe and separate from full endpoint readiness",
"command": "pnpm --filter @cotal-ai/core build && pnpm smoke:transport-liveness",
"progressPattern": " \u2713 ",
"completionMarker": "SUITE COMPLETE: 20 cells",
"completionMarker": "SUITE COMPLETE:",
"proveWith": "node scripts/mutation-proof.mjs --config extensions/connector-core/smoke/fixtures/transport-liveness.mutations.json",
"why": [
"The deterministic status queues grade the exact nats.js lifecycle contract without timing a broker outage. The companion broker smoke separately proves real public disconnect and reconnect events reach these paths.",
"Core is rebuilt before each run and after every restore because connector-core resolves @cotal-ai/core through dist. Connector-core's agent source is imported relatively by the suite.",
"The full predicted kill sets, including every exclusion, are recorded in the suite header before this config is run.",
"M16 is defensive-only, not a real-entry claim. Temporary instrumentation around the real watchStatus catch fired zero times across five complete broker-companion runs on pinned nats.js 3.4.0, each covering loss, reconnect, manual epoch replacement, and terminal close. The client's status iterator closes normally on those paths. The controlled rejection cell and mutation keep catch behavior epoch-consistent if a runtime or future client version can reject, but no real cell is claimed because no reachable rejection was measured."
"M16 is defensive-only, not a real-entry claim. Temporary instrumentation around the real watchStatus catch fired zero times across five complete broker-companion runs on pinned nats.js 3.4.0, each covering loss, reconnect, manual epoch replacement, and terminal close. The client's status iterator closes normally on those paths. The controlled rejection cell and mutation keep catch behavior epoch-consistent if a runtime or future client version can reject, but no real cell is claimed because no reachable rejection was measured.",
"The completion marker carries no cell count on purpose. Each of these suites prints `SUITE COMPLETE: ${ran} cells` and separately refuses a run whose `ran` is not its EXPECTED_CELLS, so a count in the marker duplicates a check the suite already makes and rots the moment a cell is added. It did: a cell landed in the suite, the marker still said 15, and every mutant here graded INCONCLUSIVE for a week because a finished run printed a line the marker could never match."
],
"mutations": [
{
Expand Down
Loading
Loading