fix: recover LocalDNS after repeated unexpected exits

[Saewon Kwak (saewoni)]

TL;DR

Problem: If LocalDNS crashes repeatedly, systemd can exhaust its default restart limit and leave localdns.service permanently failed. Pods still have nameserver 169.254.10.11 in their existing /etc/resolv.conf, but nothing is listening on that address, so pod DNS remains broken until LocalDNS is manually restored.

Fix: Give systemd a controlled recovery budget and restart backoff (StartLimitIntervalSec=300, StartLimitBurst=30, RestartSec=2) so transient LocalDNS crash storms do not immediately wedge the service. When LocalDNS restarts, CoreDNS rebinds the pod-facing 169.254.10.11 listener and existing pods recover DNS.

Scope: This is bounded recovery, not an infinite-retry or zero-downtime fallback. A persistently broken LocalDNS process can still exhaust the larger budget; a standby DNS fallback would be a separate follow-up.

Problem

When LocalDNS crashes repeatedly, systemd can exhaust its restart limit and leave localdns.service permanently in failed. LocalDNS owns the DNS listener used by pods (169.254.10.11). Existing pods keep that address in /etc/resolv.conf, so once the service is failed and nothing listens on .11:53, pod DNS remains broken until LocalDNS is manually restored.

The failure sequence is:

LocalDNS crashes repeatedly
  -> systemd reaches the default start limit (5 starts in 10 seconds)
  -> localdns.service becomes failed
  -> 169.254.10.11:53 stops answering
  -> existing pods continue querying 169.254.10.11
  -> pod DNS remains unavailable

PR #9360 separately restores node-level DNS by removing the 169.254.10.10 node-resolver drop-in after an unexpected exit. This PR addresses the pod-level recovery gap by keeping systemd retrying LocalDNS long enough for the pod-facing .11 listener to come back.

LocalDNS has two listeners:

Listener	Audience	Wiring
169.254.10.10	node/host resolver	70-localdns.conf networkd drop-in
169.254.10.11	pod resolver	kubelet --cluster-dns, baked into pod /etc/resolv.conf

A running pod cannot be repointed from the node, so pod recovery requires the .11 listener to become available again.

Fix

Increase the systemd restart budget and add a restart backoff:

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=30

[Service]
Restart=on-failure
RestartSec=2
KillMode=mixed
KillSignal=SIGTERM

This changes the behavior from “a short crash storm can permanently wedge the unit” to “systemd retries with a controlled backoff and the service can recover.” When LocalDNS restarts, CoreDNS rebinds 169.254.10.10 and 169.254.10.11, allowing existing pods to resume DNS.

This is a bounded recovery improvement, not an infinite-retry or zero-downtime guarantee. A persistent failure can still exhaust the larger start budget, and a zero-gap standby responder would require a separate design.

PR #9439 is stacked on PR #9360, so ExecStopPost from #9360 and the recovery settings here work together.

Reproduction — pod DNS blackhole (shipped VHD, before this fix)

Live LocalDNS-enabled AKS node (Ubuntu 24.04), a busybox pod scheduled on it.

Baseline — LocalDNS healthy. The pod resolves through the cluster listener:

# pod /etc/resolv.conf
nameserver 169.254.10.11

# in-pod: nslookup mcr.microsoft.com          -> Server 169.254.10.11:53 -> resolves
#         nslookup kubernetes.default...       -> 10.0.0.1 (works)

Fault injection — supervisor driven into the terminal dead state (shipped VHD):

The node was first restored to active/running, then a transient Restart=no
drop-in was installed so that the test could force the terminal state without
waiting for systemd's restart policy. The supervisor PID was read from
systemd and killed with SIGKILL:

sudo mkdir -p /run/systemd/system/localdns.service.d
printf '[Service]\nRestart=no\n' | sudo tee \
  /run/systemd/system/localdns.service.d/99-repro-no-restart.conf >/dev/null
sudo systemctl daemon-reload
main=$(sudo systemctl show -p MainPID --value localdns.service)
sudo kill -9 "$main"

The shipped unit then settled in failed; its ExecStopPost was absent, so
the node listener and the pod listener were both left unusable. The pod remained
unable to resolve until LocalDNS was manually restored:

ActiveState=failed
ExecStopPost: absent
169.254.10.11:53 -> connection refused
drop-in 70-localdns.conf -> still present

The same failure was also reproduced with a rapid-kill harness matching the
production crash-storm shape. The harness made up to eight attempts, reading
MainPID before each attempt, sending SIGKILL when a live PID existed, and
waiting 300 ms between attempts. On the shipped unit, six live PIDs were
observed and killed before systemd reached Start request repeated too quickly.

Pod DNS after the kill — remained broken until LocalDNS was restored:

# pod /etc/resolv.conf still: nameserver 169.254.10.11   (baked in, unchanged)

# in-pod: nslookup mcr.microsoft.com
nslookup: write to '169.254.10.11': Connection refused
;; connection timed out; no servers could be reached

# in-pod: nslookup kubernetes.default.svc.cluster.local
nslookup: write to '169.254.10.11': Connection refused
;; connection timed out; no servers could be reached

The node-level fix (#9360) does nothing here — the pod never uses 169.254.10.10.

Change

Keep the unit recovering so 169.254.10.11 is rebound instead of wedging in failed:

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=30

[Service]
RestartSec=2
KillMode=mixed
KillSignal=SIGTERM

• RestartSec=2 backs off between restarts so a crash loop cannot exhaust the start-limit burst in a couple of seconds, and each restart has time to re-bind the node/cluster listeners and re-apply config.
• StartLimitIntervalSec=300 / StartLimitBurst=30 increase the restart budget so transient crash storms are less likely to exhaust the start limit immediately. A persistent failure can still exhaust this finite budget.
• KillMode=mixed keeps service-process cleanup under systemd control so the next ExecStart can re-bind the listeners cleanly. KillSignal=SIGTERM documents the existing/default graceful termination signal; it is not itself the child-reaping or recovery mechanism.

Stacked on #9360, so on each recovery ExecStopPost + startup restore both listeners together.

Fixed behavior — pod DNS recovers (same live node, after this fix)

The same live node was tested with the repeated-kill harness, which attempts up to
eight SIGKILLs when a live MainPID is available. The observed run recovered
through the restart/backoff cycle and the pod DNS subsequently recovered:

	Shipped (unfixed)	This PR
Unit after crash storm	Start request repeated too quickly → terminal failed	recovers to active/running (Result=success)
169.254.10.11:53 listener	dead until manual restore	back up
dig @169.254.10.11 from node	connection refused	resolves
In-pod DNS	failed until manual restore	down ~30 s, then recovers and stays up

In-pod DNS timeline under this fix (polled from inside the pod during the
restart/backoff window; the recovery run observed the service return to
active/running and the .11 listener return):

t=3–9s    Address: 150.171.70.10        # healthy before / at storm start
t=12–39s  (blank)                        # localdns crash-restarting through backoff
t=42–60s  Address: ...                    # recovered and stable

So this converts the reproduced permanent pod-DNS outage into a brief, self-healing (~30 s) interruption for a transient crash storm. The residual gap is the crash-loop-through-backoff window, and the finite restart budget means a persistently broken process can still eventually enter failed. A zero-gap guarantee would require an always-on standby responder on 169.254.10.11; a persistent-failure fallback is a separate follow-up. This PR addresses the reproduced transient crash-storm failure, not every possible permanent LocalDNS failure.

Validation

• Manual live A/B: performed on a LocalDNS-enabled AKS node via az vmss run-command + in-pod kubectl exec. The repro cluster's discovered kube-dns Service IP (10.0.0.10) was confirmed functional throughout; this address is cluster-specific and is not a universal constant.
• Automated pod-level E2E: not yet included in this PR. The follow-up test should schedule a pod on the LocalDNS node, confirm 169.254.10.11, drive LocalDNS through the crash/recovery cycle, assert in-pod nslookup recovers, and restore the node. This complements the node-level assertions in fix: restore node-level DNS after unexpected LocalDNS exit #9360.

Relationship to #9360

• fix: restore node-level DNS after unexpected LocalDNS exit #9360: node-level DNS restoration (revert 169.254.10.10 on unexpected exit).
• This PR: pod-level DNS restoration (keep 169.254.10.11 answering by guaranteeing LocalDNS recovery).

Together, #9360 and this PR address the reproduced node- and pod-DNS failure paths for transient LocalDNS crash storms. Persistent LocalDNS failure and zero-gap fallback remain out of scope and require additional work.

[github-actions]

Windows Unit Test Results

  3 files  ±0   14 suites  ±0   53s ⏱️ -1s
415 tests ±0  415 ✅ ±0  0 💤 ±0  0 ❌ ±0 
418 runs  ±0  418 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 123d242. ± Comparison against base commit 1f8e146.

♻️ This comment has been updated with latest results.

[Saewon Kwak (saewoni)]

Node SIG review requested

Could Node SIG please review AgentBaker PR #9439?

This PR addresses a LocalDNS pod-DNS outage after repeated LocalDNS crashes. When localdns.service exhausts systemd's default restart limit, it enters failed and stops serving the pod-facing DNS listener at 169.254.10.11. Existing pods continue using that address in /etc/resolv.conf, so pod DNS remains unavailable until LocalDNS is manually restored.

The PR adds a controlled systemd recovery policy:

StartLimitIntervalSec=300
StartLimitBurst=30
RestartSec=2

When LocalDNS restarts, CoreDNS rebinds 169.254.10.11 and existing pods can recover DNS.

Live validation

We reproduced the failure and the recovery on the same live LocalDNS-enabled AKS node:

• Shipped behavior: a crash storm exhausted the default restart limit, left localdns.service in failed, stopped the .11 listener, and left pod DNS broken until manual restoration.
• With this PR: the same type of crash storm recovered the service to active/running, brought .11 back, and pod DNS recovered after the restart/backoff window.

The pod continued using the same nameserver 169.254.10.11; no pod recreation or resolver-file rewrite was needed.

PR #9439 is stacked on AgentBaker PR #9360, which handles node-level DNS restoration through ExecStopPost. #9439 handles service recovery so the pod-facing listener comes back.

Questions / current findings

1. Should systemd be first-line recovery?

   We believe yes. Systemd sees the service failure immediately and can retry without depending on NPD polling, Kubernetes API availability, node-condition propagation, or another remediation service.

2. Are the restart values appropriate?

   The values successfully recover the reproduced crash storm. They are a bounded recovery policy, not an infinite guarantee: a persistently broken service can still exhaust 30 attempts in five minutes. We would appreciate Node SIG guidance on whether these values are appropriate or whether persistent failure should hand off to another recovery/fallback path.

3. Is KillMode=mixed appropriate?

   It appears appropriate for the localdns.sh supervisor plus its CoreDNS child: systemd signals the main process first and cleans up remaining service processes so the next start can bind the listeners cleanly. The explicit KillSignal=SIGTERM documents the normal graceful-stop signal; it is not itself the recovery or child-reaping mechanism.

4. Does LocalDNS NPD already recover the failed service?

   We inspected the aks-vm-extension NPD implementation. check_dns_to_localdns.sh detects the service/listener failure and emits LocalDNSError / LocalDNSProblem. The inspected remediate_dns.sh only repairs generic network state and can restart systemd-networkd; it does not reset or restart localdns.service. We found no LocalDNS-specific service restart path in that repository. If another internal component consumes LocalDNSProblem and performs remediation, please point us to it so the ownership can be coordinated with systemd.

This PR is intended to improve recovery from transient crash storms. It does not add an infinite retry policy or a zero-downtime DNS standby fallback. Persistent-failure handling and a zero-gap fallback remain separate design questions.

[Copilot]

🟡 Changes recommended

The crash-storm recovery behavior lacks automated coverage, and the new comments inaccurately describe finite retries and KillMode=mixed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Improves LocalDNS recovery after crash storms to restore pod DNS without manual intervention.

Changes:

• Expands systemd’s restart budget and adds a two-second backoff.
• Documents service termination and listener cleanup behavior.

File summaries

File	Description
parts/linux/cloud-init/artifacts/localdns.service	Configures bounded LocalDNS restart recovery.

Review details

Suppressed comments (2)

parts/linux/cloud-init/artifacts/localdns.service:32

• This describes KillMode=mixed incorrectly: KillSignal=SIGTERM is sent only to the main process, while the remaining control-group processes receive the subsequent SIGKILL. Please document the actual cleanup sequence so future changes do not assume CoreDNS receives SIGTERM here.

# On stop, SIGTERM the control group; on an unexpected exit / SIGKILL of the
# supervisor, systemd still reaps orphaned coredns children so the next
# ExecStart can re-bind the listeners cleanly.

parts/linux/cloud-init/artifacts/localdns.service:18

• 🟡 Medium Risk — The existing validateLocalDNSLifecycle test kills the supervisor only three times and waits for each recovery, so it passes under the old default five-start limit and never validates this new recovery budget or pod-facing listener. Add an E2E case that asserts these systemd properties, drives more than five unexpected exits, and verifies DNS through 169.254.10.11 recovers; otherwise a typo or ineffective directive can reintroduce the outage undetected.

StartLimitIntervalSec=300
StartLimitBurst=30

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.