Installer matrix pinned-version case is broken and the gate has been dormant 28 days

[davidfarah2003]

The pinned-version case in the onboarding matrix fails: installing cotal-ai@0.14.1 produces a cotal that reports no version at all.

  upgrade        node:22-bookworm-slim PASS
  pinned-version node:22-bookworm-slim FAIL
    installed cotal did not report a version:

29 of 30 cases pass. upgrade, which installs the current version and then upgrades, passes. So install.sh works; what has rotted is specifically the old version this one case pins.

Where it comes from

install/test-matrix.sh:61:

pinned-version~node:22-bookworm-slim~useradd -m -s /bin/bash tester~--version 0.14.1~installs_ok && version_is 0.14.1

The pin is a hardcoded 0.14.1 while the package is now at 0.35.0. The case asserts that installing that exact published version yields a working binary that reports 0.14.1. It installs from the registry, so what it grades is a four-week-old published artifact rather than anything in the tree.

Why nobody saw it

The Installer workflow only fires on changes to install.sh, install/**, or .github/workflows/installer.yml. Its previous run was 2026-08-01, twenty-eight days ago, and it passed. Nothing between then and now touched those paths, so the gate has been dormant while the thing it grades drifted.

It surfaced only because #980 edits .github/workflows/installer.yml and therefore matched the path filter for the first time in a month.

This is not caused by #980

#980 changes eight files:

.github/workflows/changesets.yml
.github/workflows/ci.yml
.github/workflows/installer.yml
.github/workflows/windows.yml
bin/smoke/ci-suites.txt
bin/smoke/mutations/workflow-concurrency.json
bin/smoke/workflow-concurrency.smoke.ts
package.json

Neither install.sh nor install/test-matrix.sh is among them, and its only change to installer.yml is the concurrency block. The failing case runs install.sh inside a Docker container against a published npm package, so nothing in that diff is reachable from it.

What to decide

Two separable questions, and the second is the one that matters more.

The pin itself. A case that hardcodes one old version is asserting that a specific historical release stays installable forever. That may be the intent, in which case the break is real and worth diagnosing. If the intent was only "a --version flag pins what it says", the case should pin a version derived from the registry or from a recent release rather than a literal that ages.

The dormancy. A gate that runs only when its own inputs change cannot notice its subject rotting underneath it, and this one went twenty-eight days. Same shape as #976: the badge was green because nothing ran, not because anything passed. Worth considering a scheduled run, or widening the path filter so a release also exercises it.

Diagnosis of the actual 0.14.1 breakage is not attempted here; this issue records that it is broken, that it is dormant, and that it is unrelated to the PR that revealed it.

[davidfarah2003]

Additional evidence that this is head-independent: the same two checks (installer-ok, matrix) have now failed identically at three consecutive heads of #980, while the diff between those heads changed substantially.

547221f52   installer-ok, matrix  fail
37eb117a3   installer-ok, matrix  fail
ad9c75e29   installer-ok, matrix  fail

Between the first and last of those, the PR gained a committed mutation fixture, changed the suite's completion-marker behaviour, added a doc comment, merged main twice, and moved a ci-suites.txt block. None of it altered the outcome, which is what you would expect from a case whose subject is a published npm package rather than anything in the tree.

Worth noting for whoever picks this up: because installer.yml is in its own path filter, any PR that edits that workflow will now carry these two red checks until the pin is dealt with. That is a small tax on an unrelated change, and it is the reason it is filed separately rather than folded in.

[davidfarah2003]

Narrowing this with evidence, and ruling out the obvious explanation so the next person does not spend the time on it.

What the failure actually is

From the Installer run at ad9c75e29:

pinned-version node:22-bookworm-slim FAIL
  installed cotal did not report a version:
    + Added ~/.local/bin to your PATH
    Cotal 0.14.1
    ~/.local/bin/cotal - installed in 7s

The install succeeds. The installer prints Cotal 0.14.1 and writes the launcher. What fails is the check afterwards, install/test-matrix.sh:418:

echo "___VERSION=\$(${user_prefix} '\$HOME/.local/bin/cotal --version 2>/dev/null | head -1' || echo none)"

and its assertion at :81 and :106:

grep -q 'cotal-ai ' <<<"$version_out" || { echo "installed cotal did not report a version: $version_out"; ... }
version_is() { grep -q "cotal-ai $1" <<<"$version_out"; }

$version_out came back empty, so the installed binary produced nothing on stdout when asked for its version.

Ruled out: "0.14.1 predates the --version flag"

This was my first theory and it is wrong.

• cotal -v / --version was added by d9c54b5d3, 2026-07-20
• cotal-ai@0.14.1 was published to npm 2026-07-24T05:28:43Z

The flag is four days older than the release, so 0.14.1 contains it. The output format is not the problem either: implementations/cli/src/command.ts:133 builds cotal-ai ${cliVersion()}, and that line was introduced by the same commit that added the flag, so it has not changed since before 0.14.1.

Also worth stating: the upgrade case in the same matrix passes, and it installs the current published version twice. So the harness, the launcher path and version_is all work against a current binary. Whatever this is, it is specific to running the 0.14.1 artifact.

Why nobody can see the cause

The probe discards the only evidence that would identify it:

• 2>/dev/null throws away stderr, which is where a module-resolution failure, an engine refusal or a stack trace would go
• | head -1 masks the exit status, so a binary that crashed and one that printed nothing are indistinguishable
• the || echo none fallback never fires, because the pipeline's status is head's

The result is a failure that can only say "did not report a version" and can never say why. Capturing stderr and the exit code into the failure message would very likely identify this in one run, and would be worth doing regardless of what this specific cause turns out to be.

Not fixed, and why

I could not reproduce it. This needs Docker with node:22-bookworm-slim and there is no daemon running on the machine I am on, so I have no way to see the stderr the probe throws away. Guessing at a fix for a failure I cannot observe would be the wrong move, so I am leaving it and recording what is established.

Open question worth someone's attention: if cotal --version genuinely fails on the published 0.14.1 rather than this being a harness artifact, then customers who installed 0.14.1 have a binary that cannot report what it is. That is a customer-facing fact rather than a test problem, and it is the reason this is worth diagnosing properly rather than just re-pinning to a newer version.

Impact meanwhile

This is currently the only red on #980 and #981, and it reds any PR that touches install.sh, install/** or installer.yml. It is not a merge blocker: nothing on main is a required status check (#996).

[davidfarah2003]

Reproduced and root-caused. This is not a rotted test pin, and the fix I suggested earlier (re-pin to something newer) would have hidden a real defect.

Reproduced

./install/test-matrix.sh --keep pinned-version locally reproduces CI exactly, same message. Then, testing the published package directly and keeping stderr, which the matrix throws away:

$ docker run --rm node:22-bookworm-slim bash -c 'npm i -g cotal-ai@0.14.1; cotal --version'

Error: Cannot find package
  '/usr/local/lib/node_modules/cotal-ai/node_modules/json-canonicalize/bundles/index.umd.js'
  imported from .../node_modules/@cotal-ai/core/dist/canonical.js
  code: 'ERR_MODULE_NOT_FOUND'
rc=1

The failure is in @cotal-ai/core/dist/canonical.js at import time, so it is not confined to --version. Every command on that install fails.

Root cause

cotal-ai@0.14.1 -> @cotal-ai/core declares "json-canonicalize": "^2.0.0"
npm resolves      json-canonicalize@2.0.1
2.0.1 package.json main = "./bundles/index.umd.js"
2.0.1 ships no bundles/ directory at all

json-canonicalize@2.0.1 is a broken publish: its main points at a file the tarball does not contain. Any cotal-ai release whose core carries the floating ^2.0.0 resolves to it and is permanently broken on a fresh install.

Current releases are safe, by design rather than luck

latest (0.35.0) installs and runs: cotal-ai 0.35.0, rc=0. Its core declares json-canonicalize as an exact 2.0.0, which does ship bundles/.

That pin is deliberate: 758e1e37d, "fix(core): pin json-canonicalize so a published install cannot resolve a broken tarball", 2026-08-16. So this was already found and fixed for new releases.

The pin is load-bearing and unprotected. packages/core/package.json:60 carries a bare "json-canonicalize": "2.0.0" among neighbours that all use ^. JSON cannot hold a comment, so the only record of why it must not be widened is that commit message. Anything that normalizes dependency ranges reintroduces this.

What that means for the failing case

The first release carrying the pin is 0.19.0 (published 2026-08-17, first after the 2026-08-16 fix). Verified in the same image:

$ docker run --rm node:22-bookworm-slim bash -c 'npm i -g cotal-ai@0.19.0; cotal --version'
cotal-ai 0.19.0
core declares json-canonicalize: 2.0.0

So the matrix case is correctly detecting a real breakage, not rotting. Everything published before 0.19.0 is broken on fresh install today.

Two separate decisions, and I have not taken either

1. The test. Re-pointing install/test-matrix.sh:61 from 0.14.1 to 0.19.0 makes it pass and keeps testing what it is for, which is that install.sh --version X installs exactly X. It needs a comment recording that the version must be at or after 0.19.0 and why, or the next person re-pins it to something older and this returns.

I did not make that change on its own, because doing so turns a true red green while the underlying fact is unchanged, and that should be a deliberate choice rather than a side effect of tidying CI.

2. The published versions, which is the bigger half. Every cotal-ai before 0.19.0 currently installs a CLI that cannot run any command. Anyone who pinned an old release, or installs one from a lockfile-free environment, gets a binary that fails at import with no indication of why. npm deprecate 'cotal-ai@<0.19.0' with a message pointing at the cause would at least warn them at install time. That is a publishing action on the public registry and is an owner's call.

[davidfarah2003]

Both halves are now addressed, each with live evidence:

The pin: repinned to 0.19.0 in 697edbe, with the number's load-bearing reason recorded beside it in test-matrix.sh - it is the oldest release that still installs from a blank machine, because every earlier core floats json-canonicalize ^2.0.0 and 2.0.1 stopped shipping the bundle its main points at (the diagnosis this issue deferred; #1001 has the full chain). The case passes on the current matrix.

The dormancy: #1021 (merged as 66d3f88) adds a weekly scheduled run (23 6 * * 1) and workflow_dispatch to the Installer workflow, with the reason in a comment above the triggers: the matrix installs PUBLISHED artifacts that rot while the tree holds still, so a path-filtered gate cannot notice its subject changing underneath it. Dormancy is now bounded at seven days instead of unbounded, and a re-check no longer needs a path-touching commit.

Live oracle, both trigger paths, on the merged commit: the push-triggered run (33277888121) and a manual workflow_dispatch run (33277892054) both completed green - 30/30 cases including pinned-version.