fix: reject empty resolver state during LocalDNS startup

[Saewon Kwak (saewoni)]

Problem

When LocalDNS exits, cleanup removes the 70-localdns.conf drop-in and runs networkctl reload. During that reload the active resolver state (/run/systemd/resolve/resolv.conf) can transiently contain no nameservers at all — or still contain the LocalDNS listener 169.254.10.10 while the listener is already gone.

The LocalDNS startup path waits for the LocalDNS listener address to disappear from resolv.conf before reading the upstream VNET DNS servers into the Corefile. The old guard only checked that 169.254.10.10 was absent:

if ! grep -qwF "$LOCALDNS_NODE_LISTENER_IP" <<< "$current_dns"; then
    echo "DNS configuration refreshed successfully. Current DNS: ${current_dns}"
    return 0
fi

An empty nameserver list satisfies that condition. So wait_for_localdns_removed_from_resolv_conf returned success on an empty resolver, logged "DNS configuration refreshed successfully" against an empty upstream, and startup proceeded. replace_azurednsip_in_corefile then found no upstream to substitute and exited 216 (ERR_LOCALDNS_FAIL).

Because LocalDNS serves both listeners — the node listener 169.254.10.10 and the pod/cluster listener 169.254.10.11 — from a single Corefile, this aborted startup before either listener was configured. Any scenario with LocalDNS enabled was affected (node and pod resolution), not just the LocalDNS tests; distros only differed by whether they won the timing race.

Change

wait_for_localdns_removed_from_resolv_conf now requires both:

• at least one nameserver is present, and
• the LocalDNS listener address is absent.

if [ -n "$current_dns" ] && ! grep -qwF "$LOCALDNS_NODE_LISTENER_IP" <<< "$current_dns"; then

If the resolver is empty, comment-only, or missing, the function keeps polling until a real upstream appears (or times out) instead of falsely reporting success. The existing asynchronous networkctl reload behavior is retained — this PR does not call networkctl reconfigure, which live testing showed clears resolv.conf and reproduces the same exit-216 one step later.

cleanup_iptables_and_dns also now re-initializes DEFAULT_ROUTE_INTERFACE alongside the drop-in path variables, so cleanup invoked from a trap or watchdog restart has a complete network context.

Behavior

resolv.conf state during startup	Before (shipped VHD)	After (this PR)
Empty (no nameservers)	wait returns 0, logs "refreshed successfully" on empty upstream → replace exits 216	wait keeps waiting / times out; startup does not proceed on empty upstream
Comment-only (# nameserver ...)	wait returns 0 (false success) → 216	wait keeps waiting / times out
Missing file	wait returns 0 (false success) → 216	wait keeps waiting / times out
Healthy upstream present	wait returns 0, replace succeeds	wait returns 0, replace succeeds (unchanged)

Validation

• bash -n on localdns.sh and localdns_spec.sh; git diff --check.
• ShellSpec now asserts empty, comment-only, and missing resolver files do not report successful recovery (previously they asserted success).
• go-test, shellcheck, shellspec, cue, lint all green.

Live AKS before/after

Validated on a live Ubuntu 24.04 AKS cluster (the distro where the failure reproduces), running the shipped VHD localdns.sh as the baseline and the PR localdns.sh applied to a node and restarted via systemctl restart localdns.service.

Subscription: 359833f5-8592-40b6-8175-edc664e2196a
Resource group:  sakwa-localdns-repro-scu-0710
Cluster:         sakwa-localdns-repro-0710
Baseline node:   aks-npd16-25269873-vmss000000  (shipped VHD, unmodified)
Patched node:    aks-npd16-25269873-vmss000001  (PR localdns.sh, unit restarted, then restored)

The shipped node's localdns.sh has no non-empty guard (grep -c 'n "$current_dns"' = 0), confirming the baseline is the buggy version.

Startup wait — before vs after (empty-resolver repro)

Running the real wait_for_localdns_removed_from_resolv_conf and replace_azurednsip_in_corefile functions from each script against the injected resolver states:

Case (resolv.conf)	Baseline wait rc	Baseline result	Fixed wait rc	Fixed result
A. empty	0 (false success)	replace → rc=1 "No Upstream VNET DNS servers found" → exit 216	1 (keep waiting)	correctly refuses to proceed
B. comment-only	0 (false success)	would exit 216	1 (keep waiting)	correct
C. missing	0 (false success)	would exit 216	1 (keep waiting)	correct
D. healthy upstream 168.63.129.16	0	replace rc=0	0	replace rc=0 (unchanged)

Baseline, Case A (verbatim):

Waiting for localdns (169.254.10.10) to be removed from resolv.conf...
DNS configuration refreshed successfully. Current DNS:          <-- empty upstream, false success
No Upstream VNET DNS servers found in /tmp/resolv.empty.        <-- replace_azurednsip_in_corefile fails -> exit 216

Fixed, Case A (verbatim):

Waiting for localdns (169.254.10.10) to be removed from resolv.conf...
Timed out waiting for localdns to be removed from resolv.conf after 2 seconds.
Current DNS:                                                    <-- correctly refuses to declare success

Node-level and pod-level resolvers — after the fix

The PR script was installed on the patched node and localdns.service restarted. Both listeners come up healthy (a dig health-check.localdns.local against each IP succeeds):

                                        Fixed node
localdns.service is-active              active
node resolver   169.254.10.10:53        OK
pod  resolver   169.254.10.11:53        OK
/etc/resolv.conf nameserver             169.254.10.10
/run/systemd/resolve/resolv.conf        169.254.10.10

Because both the node listener (169.254.10.10) and the pod/cluster listener (169.254.10.11) are served from the one Corefile that startup was previously aborting before writing, the fix restores both node-level and pod-level resolution. The patched node was restored to the shipped baseline afterward (localdns.service active, both listeners OK); the cluster was left in its original state.

Source

The networkctl reconfigure approach was explored in the earlier #9360 branch but is intentionally not included here — reviewer E2E showed it caused the same empty-resolver failure on Ubuntu 24.04, Azure Linux V3, and ACL. This PR is based on main and contains only the resolver-wait correction.

[github-actions]

Windows Unit Test Results

  3 files   13 suites   49s ⏱️
404 tests 404 ✅ 0 💤 0 ❌
407 runs  407 ✅ 0 💤 0 ❌

Results for commit c3c511b.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Restores upstream DNS after LocalDNS cleanup by reapplying network configuration to the live interface.

Changes:

• Reconfigures the default-route interface after network reload.
• Adds success and failure ShellSpec coverage.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
parts/linux/cloud-init/artifacts/localdns.sh	Reapplies DNS configuration during cleanup.
spec/parts/linux/cloud-init/artifacts/localdns_spec.sh	Tests reconfiguration success and failure.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Ye Wang [msft] (yewmsft)]

don't merge yet — the e2e gate is red because of this pr, not flake. networkctl reconfigure on the default-route link wipes resolv.conf instead of reapplying upstream dns, so localdns hits the same exit 216 one step later. details inline on localdns.sh:720.

scenario	this pr	9362	9328
localdns hosts plugin ubuntu 2404	❌ 216	✅	✅
localdns hosts plugin azure linux v3	❌ 216	✅	✅
localdns hosts plugin ACL	❌ 216	✅	✅
localdns hosts plugin ubuntu 2204 / 2604 minimal	✅	✅	✅

Test_DCGM_Exporter_Compatibility/Ubuntu2404 shows the same empty-dns trace, so this hits any scenario with localdns on, not just the localdns tests. 2204/2604 only pass because they win the race.

the wait is also wrong, and this pr is what exposes it. wait_for_localdns_removed_from_resolv_conf (localdns.sh:607) only checks that 169.254.10.10 is absent. an empty nameserver list satisfies that, so it returns success and logs "DNS configuration refreshed successfully" on an empty resolver, then replace_azurednsip_in_corefile finds nothing and exits 216. it should require a non-empty upstream list that does not contain the listener ip. worth fixing regardless of what happens to this pr.

the repro in the description is circular — the mock was wired to write the corrected resolver file, then the test observed the corrected resolver file. the real reconfigure does the opposite.

suggested direction: if networkctl reload really isn't converging on some nodes, fix the wait (require a non-empty upstream list, extend the 5s budget) instead of reconfiguring the default-route link. please get a live ubuntu 24.04 repro first to confirm reload-only convergence is actually the problem.

nit: commit body says (cherry picked from abcca13) — link the source pr in the description.

[Copilot]

🟢 Approval recommended

The startup guard correctly rejects unusable resolver states and is covered by focused regression tests.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Saewon Kwak (saewoni)]

Replying to Ye Wang [msft] (@yewmsft)'s review #pullrequestreview-5083056681 — all points addressed as of c3c511b8cb.

don't merge yet — the e2e gate is red because of this pr, not flake. networkctl reconfigure on the default-route link wipes resolv.conf instead of reapplying upstream dns, so localdns hits the same exit 216 one step later.

Fixed — networkctl reconfigure was removed entirely. There is no reconfigure anywhere in localdns.sh on this commit (grep -n reconfigure returns nothing). The shutdown/cleanup path keeps the prior networkctl reload-only behavior.

the wait is also wrong, and this pr is what exposes it. wait_for_localdns_removed_from_resolv_conf (localdns.sh:607) only checks that 169.254.10.10 is absent. an empty nameserver list satisfies that ... it should require a non-empty upstream list that does not contain the listener ip.

Fixed exactly as suggested — localdns.sh:616:

if [ -n "$current_dns" ] && ! grep -qwF "$LOCALDNS_NODE_LISTENER_IP" <<< "$current_dns"; then

Now requires a non-empty upstream that does not contain the listener IP. Empty / comment-only / missing resolv.conf all keep waiting instead of false-succeeding, with ShellSpec coverage at localdns_spec.sh:1341 (no nameservers), :1348 (comments only), :1364 (does not exist).

the repro in the description is circular — the mock was wired to write the corrected resolver file, then the test observed the corrected resolver file. the real reconfigure does the opposite.

Fixed — the circular mocked repro was removed from the description. It's replaced with a real live before/after (below).

suggested direction: if networkctl reload really isn't converging on some nodes, fix the wait ... instead of reconfiguring the default-route link. please get a live ubuntu 24.04 repro first to confirm reload-only convergence is actually the problem.

Done. Validated live on an Ubuntu 24.04 AKS node — shipped-VHD baseline vs the PR's localdns.sh applied and restarted via systemctl restart localdns.service:

• Baseline (shipped VHD): empty resolv.conf → wait returns 0 (false success), logs "DNS configuration refreshed successfully. Current DNS:" (empty), then replace_azurednsip_in_corefile fails "No Upstream VNET DNS servers found" → exit 216.
• Patched (this PR): empty / comment-only / missing resolv.conf → wait keeps waiting and does not proceed; healthy upstream still succeeds. After restart, both the node listener 169.254.10.10:53 and the pod listener 169.254.10.11:53 are healthy (dig health-check.localdns.local OK on both).

Full before/after tables (node-level and pod-level) are in the PR description.

nit: commit body says (cherry picked from abcca13) — link the source pr in the description.

Fixed — no cherry-pick line remains in the commit body, and the description now has a ## Source section linking #9360.

Re-review appreciated 🙏

[aks-node-assistant]

Failed gate run

• Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=180380308
• Failed job/stage/task: build / build2004fipsgen2containerd / Build VHD

Detective summary

Ubuntu 20.04 FIPS VHD failed while enabling Ubuntu Pro FIPS updates. The first failing step was ua enable fips-updates; one attempt hit 503 Service Unavailable on Ubuntu Pro context/machines and the final attempt timed out, then Packer exited 184 and the job surfaced script exit code 2.

Likely cause

Signature: vhd-ubuntu-pro-machine-token-service-unavailable. Classification: VHD/build dependency service flake. Infra/environment and external Ubuntu Pro service dependency are more likely than product or test-code regression; this PR only changes LocalDNS startup resolver handling, while the failure occurs during image-generation FIPS/UA attach before PR LocalDNS behavior is exercised.

Recommended owner/action

Node Lifecycle/VHD owner: track as the existing Ubuntu Pro machine-token/fips-updates flake; no PR action unless this reproduces consistently outside Ubuntu Pro service failures.

Strongest alternative

PR-change-caused LocalDNS regression is the strongest alternative because PR 9361 changes localdns.sh and related specs; it is less likely because the failed operation is Ubuntu Pro/FIPS repository enablement inside Packer, not LocalDNS startup or E2E DNS validation.

Evidence

• Timeline: build2004fipsgen2containerd failed; Build VHD log 453 and cleanup/perf logs also failed/skipped after the VHD failure.
• Log: ua enable fips-updates followed by Ubuntu Pro context/machines 503 Service Unavailable on an earlier attempt and The read operation timed out on the final attempt; Packer exited 184.
• Build metadata: PR 9361, source branch refs/pull/9361/merge, finish 2026-09-09T20:52:43Z.
• Changed files are LocalDNS shell/spec files, not FIPS/Ubuntu Pro attach logic.

Wiki signature

vhd-ubuntu-pro-machine-token-service-unavailable

[aks-node-assistant]

Failed gate run

• Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=179240201
• Failed job/stage/task: e2e / Run AgentBaker E2E / VHDCaching VMSS creation

Detective summary

VHDCaching scenarios failed while creating VMSS from custom gallery images. The first failing step was VMSS PUT for Ubuntu2604Minimal/Ubuntu2404 VHDCaching; Compute returned repeated GalleryImageNotFound and cleanup surfaced ParentResourceNotFound.

Likely cause

Signature: e2e-vhdcaching-gallery-image-notfound. Classification: E2E/test SIG gallery image propagation or VMSS provisioning flake. The failure occurs after custom image creation/lookup and before node validation, so image propagation is more likely than the PR's LocalDNS resolver change.

Recommended owner/action

Node Lifecycle/E2E owner: track image-version availability before VHDCaching VMSS create, or add targeted wait/retry around gallery image availability.

Strongest alternative

PR-change-caused LocalDNS regression is the strongest alternative because PR 9361 changes LocalDNS startup; it is less likely because the VMSS never reaches LocalDNS/node validation and fails on Compute gallery image lookup.

Evidence

• Timeline: Run AgentBaker E2E failed, log 812.
• Logs: failed to create VMSS after 10 retries with ERROR CODE: GalleryImageNotFound and later ParentResourceNotFound for VHDCaching scenarios.
• Test results: many scenarios marked failed from the same E2E run, with log showing VHDCaching create failures as the concrete root error.
• Build metadata: PR 9361, source branch refs/pull/9361/merge.

Wiki signature

e2e-vhdcaching-gallery-image-notfound