Skip to content

fix: recover LocalDNS after repeated unexpected exits #9439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Saewon Kwak (saewoni) wants to merge 4 commits into from
+17 −0
Open

fix: recover LocalDNS after repeated unexpected exits #9439

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
96adef0
fix: restore node DNS after unexpected LocalDNS exit
saewoni Sep 9, 2026
d2cbaa7
test: require a non-empty resolver snapshot before accepting DNS revert
saewoni Sep 9, 2026
2392c5c
fix: keep LocalDNS recovering to restore pod DNS after unexpected exit
saewoni Sep 9, 2026
123d242
Merge commit '1f8e14672e' into HEAD
saewoni Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions parts/linux/cloud-init/artifacts/localdns.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,30 @@ Before=kubelet.service
Before=containerd.service
# don't run on old images; we're not compatible
ConditionKernelVersion=>=5.15
# Keep recovering from repeated unexpected exits instead of wedging the unit in
# a terminal 'failed' state. LocalDNS binds the pod cluster listener
# (169.254.10.11); if the unit stops being restarted, that listener stays dead
# and every pod on the node black-holes DNS (its /etc/resolv.conf nameserver is
# baked to 169.254.10.11 and cannot be repointed). A wide interval with a high
# burst, combined with the RestartSec backoff below, means a crash storm slows
# restarts but never gives up.
Comment on lines +15 to +16
StartLimitIntervalSec=300
StartLimitBurst=30

[Service]
Type=notify
NotifyAccess=all
WatchdogSec=60
Restart=on-failure
# Back off between restarts so a crash loop does not exhaust the start-limit
# burst in a couple of seconds, and so each restart has time to re-bind the
# node (169.254.10.10) and cluster (169.254.10.11) listeners and re-apply config.
RestartSec=2
KillMode=mixed
# On stop, SIGTERM the control group; on an unexpected exit / SIGKILL of the
# supervisor, systemd still reaps orphaned coredns children so the next
# ExecStart can re-bind the listeners cleanly.
KillSignal=SIGTERM
# Revert node DNS configuration even when the supervisor exits unexpectedly.
ExecStopPost=/opt/azure/containers/localdns/localdns.sh cleanup
TimeoutStopSec=30
Expand Down
Loading