Repository navigation
ci: e2e shards give new links no IPv6, so Chrome sees no network change (the module stalls) - #796
Merged
Merged
Conversation
The module stalls: Chrome takes a link's IPv6 address coming or going for a network change and fails every request still waiting for a connection (net::ERR_NETWORK_CHANGED). Docker adds a link for each container a computer starts, so a page's module loads queued behind its six connections failed and the page never started. Each shard now first sets net.ipv6.conf.default.disable_ipv6=1 (and docker0's), before any container, so containers change nothing Chrome watches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alexlwn123
approved these changes
Oct 31, 2026
alexlwn123
left a comment
Collaborator
There was a problem hiding this comment.
Pass at fac4d130999c6f71e63c1136d2fbd456dea09dc6.
The runner-only change is appropriately small: it disables IPv6 on future links and the existing Docker bridge before BuildKit, containers, or Chrome start. The local harness uses IPv4, and the workflow keeps its existing test and deployment gates. I found no blocking regression or unnecessary implementation machinery.
Validation: the new sysctl step succeeded on all four CI shards, and check passed; the e2e shards are still running. git diff --check passed locally. I did not reproduce the Linux network-change behavior on this macOS host; whether the intermittent module stalls disappear still needs CI observation.
This was referenced Oct 31, 2026
e2e flake: shell-ui's first-run page never comes back after its reload (#sidebar-agent missing)
#792
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flake: the module stalls: a page whose later imports never reach the node, so it never starts. It shows as shell-ui's "Still loading…" (shell.rs:1145), billing-page's
window.__held.release is not a function(three times in an hour today: queue runs 41151869271 and 41151870056, #779's 41153845292), and the chat's fit checks. It's been the queue's main blocker. It happened through wrangler's proxy and through #743's front door alike.Cause (Chrome and Docker on one host, not our code): Chrome takes a link's IPv6 address coming or going for a network change. On one, it fails every request still waiting for a connection with
net::ERR_NETWORK_CHANGED. A page's module imports go out at once, six at a time per host on HTTP/1.1, so the rest wait in Chrome's pool. Docker adds a veth for each container a computer starts and removes it when the container stops, and the veth's IPv6 link-local address becomes valid after DAD. That's exactly the CI signature: the first imports answered, the rest never reaching the port, with no restart and no error on the node.Evidence:
ERR_NETWORK_CHANGEDon the imports still queued (usually 14 of 20, after the first 6 answered). All 20 hadrequestWillBeSentat once. Answered ones went out at once on reused connections; the failed ones never got a connection.connecton the bridge came about 2 s before one stall. Starting a container on purpose while a page's 7th import waits behind six held ones fails it withERR_NETWORK_CHANGEDin 4 of 4 runs.net.ipv6.conf.default.disable_ipv6=1, the same link fails it in 0 of 3. A request already sent on a connection survives either way.Fix: each e2e shard's first step, before any container, sets
net.ipv6.conf.default.disable_ipv6=1(every link made from then on gets no IPv6) and turns IPv6 off ondocker0. Containers then change nothing Chrome watches. Nothing in the e2e uses IPv6 on Docker's links. The runner is a throwaway VM, and it covers every browser test at once.Not covered: local Linux runs (
xtask e2e,xtask dev) still hit it unless that sysctl is set; it's the developer's machine, so not ours to set. macOS is unaffected (Docker's links are in its VM). Options for local parity are a hint fromxtask e2e, or Chrome in a network namespace of its own; that's for later.Proven vs not: proven in the repro, the targeted probe and the isolated namespace above. Not yet proven in CI, where the stall rate is a few a day; watch the queue for module stalls after this lands.
🤖 Generated with Claude Code