Skip to content

e2e: a run's computers start with internet off, so it never depends on public DNS - #1021

Merged
futurepaul merged 2 commits into
masterfrom
claude/e2e-computers-no-internet
Nov 3, 2026
Merged

futurepaul merged 2 commits into
masterfrom
claude/e2e-computers-no-internet

Conversation

@futurepaul

Copy link
Copy Markdown
Collaborator

Fixes #1017

Cause: local workerd's networking sidecar (cloudflare/proxy-everything) answers a container's DNS. With the internet on, it first forwards every lookup to public DNS, which workerd hard-codes to 1.1.1.1 and 8.8.8.8. That includes the hosts the platform intercepts (backups.sandbox.internal, api.fragment.internal). It answers them itself (11.0.0.1) only once public DNS says NXDOMAIN.

  • In merge-queue run 41672731370, public DNS didn't answer from the runner for a whole node boot. Every lookup failed with failed to lookup address information: Try again, after 5.1 s, so every save failed, a brand-new computer's included.
  • It wasn't the restart. workerd's recovery keeps the sidecar, the cell's adopt() re-arms the intercepts, and the DNS path never touches workerd.

Fix: the e2e's fleet sets FRAGMENT_COMPUTER_INTERNET=off (devstack::Fleet::computer_internet), and cell/entry.mjs reads it at a computer's start. The sidecar then answers intercepted hosts at once, without asking public DNS. Deployments and cargo xtask dev keep the internet on. The reason sits beside the var. The ledger entry "The e2e's computers have no internet" is deleted when the sidecar answers intercepted names locally (an upstream issue drafted for Paul).

The check, in the computers lane: from the crash of the platform on, the adopted computer's sidecar drops any query it would send public DNS. That's an iptables rule in its network namespace (devstack::containers::drop_public_dns, on the sidecar's mark for those queries). Then: "after the crash its computer asked public DNS nothing … its sleep's save went on with public DNS dropped".

  • Before (this head with the e2e fleet's internet on): the check fails with {"asked":496,"sidecars":1}, and with it CI's checks: "it sleeps at the end", the ledger refusals, and the routine.
  • After: cargo xtask e2e --only computers gives 280 passed, 0 failed, on this head (rebased onto 8500481).

Nothing needed the internet: I grepped every lane and image for what a container fetches.

  • The lanes' container commands reach intercepted swap hosts (answered by the e2e's swap upstream fake) or api.fragment.internal.
  • The hermes lane's install is an offline .deb, and Claude Code's install is the stub's local stand-in.
  • Outside URLs in images/ appear only in tests.
  • With internet off: the full local cargo xtask e2e gave 3305 passed, 0 failed; --only hermes (real Hermes) gave 70 passed, 0 failed. cargo xtask check passes.

🤖 Generated with Claude Code

futurepaul and others added 2 commits November 3, 2026 14:25
…n public DNS

Local workerd's networking sidecar answers a container's DNS. With the
internet on, it asks public DNS (workerd's 1.1.1.1 and 8.8.8.8) first for
every name, the platform's intercepted hosts included, and answers them
itself only once public DNS says there is no such name. When public DNS
did not answer, every lookup of backups.sandbox.internal failed ("Try
again") and every save of the boot with it (#1017). With the internet off
it answers them at once, asking no one.

The e2e's fleet sets FRAGMENT_COMPUTER_INTERNET=off, which cell/entry.mjs
reads at a computer's start; deployments and cargo xtask dev keep it on.
Its computers reach only hosts the platform intercepts.

The computers lane's check: from the crash of the platform on, its
computer's sidecar drops any query it would send public DNS (an iptables
rule in its network namespace, devstack::containers), and the sleep's save
after goes on, the rule having dropped nothing.

Fixes #1017

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its removal: the sidecar answering an intercepted name without asking
public DNS (an upstream issue drafted for Paul).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@futurepaul futurepaul left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review @ d0907f6: PASS

The setting is read only at ContainerHost.start, once per launch, and only exact "off" disables internet; missing and unknown values stay on. Only the local e2e fleet emits it, dev stays on, and the deployment renderer constructs its own vars without this setting; the DNS-drop check is scoped by the run's project label and refuses an empty sidecar set, with the dependency and removal condition recorded in the debt ledger. Validation: 41 devstack host tests passed, plus a probe of the actual launch method covering 10 values and read count; Docker/e2e were not run.

@futurepaul
futurepaul added this pull request to the merge queue Nov 3, 2026
Merged via the queue into master with commit 34a1ea0 Nov 3, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e flake: after the computers lane restarts workerd in CI, containers can't resolve intercepted hosts

1 participant