Skip to content

ci: e2e shards give new links no IPv6, so Chrome sees no network change (the module stalls) - #796

Merged
futurepaul merged 1 commit into
masterfrom
claude/e2e-no-ipv6-on-new-links
Oct 31, 2026
Merged

futurepaul merged 1 commit into
masterfrom
claude/e2e-no-ipv6-on-new-links

Conversation

@futurepaul

Copy link
Copy Markdown
Collaborator

Flake: the module stalls: a page whose later imports never reach the node, so it never starts. It shows as shell-ui's "Still loading…" (shell.rs:1145), billing-page's window.__held.release is not a function (three times in an hour today: queue runs 41151869271 and 41151870056, #779's 41153845292), and the chat's fit checks. It's been the queue's main blocker. It happened through wrangler's proxy and through #743's front door alike.

Cause (Chrome and Docker on one host, not our code): Chrome takes a link's IPv6 address coming or going for a network change. On one, it fails every request still waiting for a connection with net::ERR_NETWORK_CHANGED. A page's module imports go out at once, six at a time per host on HTTP/1.1, so the rest wait in Chrome's pool. Docker adds a veth for each container a computer starts and removes it when the container stops, and the veth's IPv6 link-local address becomes valid after DAD. That's exactly the CI signature: the first imports answered, the rest never reaching the port, with no restart and no error on the node.

Evidence:

  • CDP, a repro of page reloads (20 imports each) through the front door: 11,000 loads, 6 stalls, every one ERR_NETWORK_CHANGED on the imports still queued (usually 14 of 20, after the first 6 answered). All 20 had requestWillBeSent at once. Answered ones went out at once on reused connections; the failed ones never got a connection.
  • Docker's own activity: a container's connect on the bridge came about 2 s before one stall. Starting a container on purpose while a page's 7th import waits behind six held ones fails it with ERR_NETWORK_CHANGED in 4 of 4 runs.
  • Isolated, in a network namespace of its own (Chrome, the page and nothing else): bringing up a new link fails the queued import in 3 of 3 runs. With net.ipv6.conf.default.disable_ipv6=1, the same link fails it in 0 of 3. A request already sent on a connection survives either way.

Fix: each e2e shard's first step, before any container, sets net.ipv6.conf.default.disable_ipv6=1 (every link made from then on gets no IPv6) and turns IPv6 off on docker0. Containers then change nothing Chrome watches. Nothing in the e2e uses IPv6 on Docker's links. The runner is a throwaway VM, and it covers every browser test at once.

Not covered: local Linux runs (xtask e2e, xtask dev) still hit it unless that sysctl is set; it's the developer's machine, so not ours to set. macOS is unaffected (Docker's links are in its VM). Options for local parity are a hint from xtask e2e, or Chrome in a network namespace of its own; that's for later.

Proven vs not: proven in the repro, the targeted probe and the isolated namespace above. Not yet proven in CI, where the stall rate is a few a day; watch the queue for module stalls after this lands.

🤖 Generated with Claude Code

The module stalls: Chrome takes a link's IPv6 address coming or going for
a network change and fails every request still waiting for a connection
(net::ERR_NETWORK_CHANGED). Docker adds a link for each container a
computer starts, so a page's module loads queued behind its six
connections failed and the page never started. Each shard now first sets
net.ipv6.conf.default.disable_ipv6=1 (and docker0's), before any
container, so containers change nothing Chrome watches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@alexlwn123 alexlwn123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass at fac4d130999c6f71e63c1136d2fbd456dea09dc6.

The runner-only change is appropriately small: it disables IPv6 on future links and the existing Docker bridge before BuildKit, containers, or Chrome start. The local harness uses IPv4, and the workflow keeps its existing test and deployment gates. I found no blocking regression or unnecessary implementation machinery.

Validation: the new sysctl step succeeded on all four CI shards, and check passed; the e2e shards are still running. git diff --check passed locally. I did not reproduce the Linux network-change behavior on this macOS host; whether the intermittent module stalls disappear still needs CI observation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants