Visual suite: a whole shard fails with "Could not capture a stable screenshot" on half of all main runs

[mfal]

One shard of the visual suite goes red with every test in it reporting

Could not capture a stable screenshot within 5000ms.

Nothing is ever compared, so there are no diff images to look at and nothing in the log says what the shard would have rendered. It hits 5 of the last 10 main runs, a different shard each time, and it is the failure behind #3090 and #3104.

Evidence

Five main runs, all with the same signature — the stable screenshot message, zero pixels (ratio …) differ, and the whole shard down:

Run	Shard	Result
33631694282	visual (3)	whole shard
33626556173	visual (3)	whole shard
33608121174	visual (1)	85 failed, 1 passed
33607745081	visual (2)	73 failed, 1 passed
33599165776	visual (1)	85 failed, 1 passed

The other three to five shards of the same run pass in 3-6 minutes.

What is known about it

• It is not a diff. Zero pixel comparisons happen. toMatchScreenshot never clears its stability gate, so it writes neither reference, actual nor diff — the visual-diffs-* artifact is empty.
• It is not browser-specific. In 33633352648 shard 5, both Firefox and WebKit failed on the same runner (118 of 120 tests). ci: retry a hung visual shard on the label workflow #3090 observed 63 of 64 blocks on firefox-linux, which reads as browser-specific but is one sample.
• It starts at the first test, not after warm-up. In 33633338781 shard 1, the first file to finish (Button, 8 tests) took 41966ms — 8 × the 5s budget. No degradation curve.
• A re-run in the same job does not help. The attempts retry in .github/actions/run-visual-shard was pointed at exactly this. Its first real exercise ran three attempts and produced 85 failed / 1 passed each time, at 474s, 472s and 472s. The attempts share the runner. ci: retry a hung visual shard on the PR run too #3104 documents this; the label-workflow half is being reverted.
• A tiny scenario survives. The single pass in each affected shard is a small one — CopyButton at 656ms, LoadingSpinner (Local) at 411ms — while everything larger times out on the same runner in the same attempt.

Lead

toMatchScreenshot's stability budget is vitest's default 5000ms; nothing in the repo raises it. Right beside it, waitForPaintedContent (packages/remote-react-components/src/tests/lib/environments.tsx) deliberately carries 20s, with a comment that CI hardware is slower than the machine its numbers came from. The same argument applies to the screenshot's stability window, and #3090 already noted it.

The option is per-call ScreenshotMatcherOptions.timeout, so it can be set once for the visual project rather than per test.

Two things to settle before raising it:

1. Does a longer window actually stabilize on the affected runner? The failure is "two consecutive captures never matched", and nothing measured so far says whether the gap is 6s or unbounded. If it is unbounded, the budget is the wrong knob and something keeps repainting.
2. The interaction with the job timeout. A budget of 15s over 86 tests is 21 minutes per attempt in the worst case, against a 35-minute job with up to three attempts. Raising the budget means revisiting attempts and timeout-minutes together.

Why it is worth fixing rather than living with

A reviewer meets a red required check with no diff images. The cheapest-looking next move is the update-screenshots label — which runs --update over the whole suite and git add -A, and is exactly how #2945 committed a tooltip-less frame as a baseline and kept the scheduled run red (#2985). Half of all main runs currently offer that invitation.

🤖 Generated with Claude Code

[mfal]

The retry mentioned above is reverted on the label path in #3107, and #3104 (which would have wired it into test.yml) is closed. Nothing now retries a visual shard on either PR path, so this failure reports as a plain red shard with no diff artifacts until it is fixed here.

🤖 Generated with Claude Code

[mfal]

#3092 is the same phenomenon from a smaller sample — I am closing it as a duplicate of this one. Three things from it that are not here yet.

1. test.yml's visual job is the only caller without the retry. .github/actions/run-visual-shard has an attempts input built for exactly this class of hang, and the scheduled workflow plus (since #3090) the label workflow pass it. But the visual job in test.yml — the one that gates every PR and every push to main, the one going red on half the runs — calls pnpm --filter @mittwald/flow-remote-react-components test:visual --shard=… directly, with no retry at all. Routing that job through the composite action would make the interim gh run rerun unnecessary.

2. There is a SECOND mode with the same outward symptom. On #3098, visual (2) burned the whole 20-minute job timeout inside playwright install --with-deps webkit, without running a single test — job 100191902405, 20m18s:

09:27:42  ##[group]Run pnpm exec playwright install --with-deps webkit
09:45:16  Get:59 …/libflite1 amd64 2.2-6build3 [13.6 MB]
09:46:40  Get:60 …/libserd-0-0 …            ← 84s for one package
09:46:58  ##[error]The operation was canceled.

19m16s of apt crawling azure.archive.ubuntu.com. Zero occurrences of Could not capture a stable screenshot in that log — so the signature in this issue distinguishes the two correctly, but a fix for the stability window leaves this one open. prepare-browser-tests caches ~/.cache/ms-playwright (the browser binaries); --with-deps installs system libraries through apt, which live outside that path and are re-fetched every run. Splitting the step lets it fail fast instead:

- run: timeout 300 pnpm exec playwright install-deps webkit
       || timeout 300 pnpm exec playwright install-deps webkit
- run: pnpm exec playwright install webkit

(timeout-minutes is not available on composite-action steps, hence coreutils timeout.)

3. The evidence gap is worth closing first. The visual job in test.yml uploads no artifact. .vitest-attachments is collected only by test-visual-label.yml and test-visual-scheduled.yml, and a stability failure writes no diff there anyway — it never reached a comparison. So today there is nothing to inspect after such a failure. Uploading the unstable frame, or two consecutive frames, would make "what did not settle" answerable, and logging which scenario gave up first would show whether it is always the shard's first capture.

Two more observations from the #3092 sample: three of the seven failing runs were chore(release): version-bump commits carrying no code at all, and each matrix shard gets its own ubuntu-latest runner — so the shards are not competing with each other for one machine.