Add LocalDNS Corefile patch DaemonSet

examples/localdns/README.md

@@ -0,0 +1,230 @@
+# DaemonSet to deliver private patches to the LocalDNS configuration
+
+This directory contains a DaemonSet that applies temporary LocalDNS Corefile
+patches on existing AKS Linux nodes.
+
+It also contains a separate rollback DaemonSet that restores the pre-patch
+backups created by the patch DaemonSet.
+
+## Why this is needed
+
+The LocalDNS Corefile is originally generated by AgentBaker during node
+bootstrap. For normal bootstrap config generation, the template comes from the
+AgentBaker service, not from a template file on the already-created node.
+
+The simplified flow is:
+
+```text
+AgentBaker service template
+  -> bootstrap/CSE payload
+  -> node writes /etc/localdns/environment
+  -> localdns.sh decodes /etc/localdns/environment into localdns.corefile when localdns starts
+  -> localdns.sh copies/updates it into updated.localdns.corefile
+  -> LocalDNS (the CoreDNS binary run by the node's localdns systemd unit) starts with updated.localdns.corefile
+```
+
+After a node is created, restarting the node's `localdns` systemd unit does not
+call AgentBaker again. Instead, `localdns.sh` reads the Corefile payload already
+stored on that node. Because of that, this DaemonSet patches the node-local files
+that `localdns.sh` actually uses at runtime.
+
+More specifically, `/etc/localdns/environment` contains environment variables
+whose values are base64-encoded Corefile text. On `localdns` service start,
+`localdns.sh` selects one of those payloads, decodes it, and writes:
+
+```text
+/opt/azure/containers/localdns/localdns.corefile
+```
+
+Then `localdns.sh` copies/processes that file into:
+
+```text
+/opt/azure/containers/localdns/updated.localdns.corefile
+```
+
+Finally, the node's `localdns` systemd unit starts the CoreDNS binary with:
+
+```text
+coredns -conf /opt/azure/containers/localdns/updated.localdns.corefile
+```
+
+This is why patching `/etc/localdns/environment` is required for restart
+persistence. If only `updated.localdns.corefile` is patched, a later
+`systemctl restart localdns` can recreate it from an older base64 value in the
+environment file.
+
+## Files patched by the DaemonSet
+
+### 1. `/etc/localdns/environment`
+
+This is the most important persistence point for an existing node.
+
+`localdns.sh` regenerates `/opt/azure/containers/localdns/localdns.corefile`
+from base64 values stored in this file whenever `localdns` starts. The DaemonSet
+patches these variables when present:
+
+```text
+LOCALDNS_COREFILE_BASE
+LOCALDNS_COREFILE_WITH_HOSTS
+LOCALDNS_BASE64_ENCODED_COREFILE
+```
+
+If this file is not patched, a later `systemctl restart localdns` can decode the
+previous base64 payload and recreate a Corefile without the intended patch.
+
+### 2. `localdns.corefile` and `updated.localdns.corefile`
+
+These are the materialized Corefile files on the node:
+
+```text
+/opt/azure/containers/localdns/localdns.corefile
+/opt/azure/containers/localdns/updated.localdns.corefile
+```
+
+`localdns.corefile` is the decoded base Corefile.
+
+`updated.localdns.corefile` is the file passed to LocalDNS, which is the
+CoreDNS binary run by the node's `localdns` systemd unit:
+
+```text
+coredns -conf /opt/azure/containers/localdns/updated.localdns.corefile
+```
+
+The DaemonSet patches these files so the node-local files on disk are correct
+immediately and so the next LocalDNS/CoreDNS start uses the fixed content. This
+does not by itself guarantee that an already-running CoreDNS process has loaded
+the new content; see [Restart behavior](#restart-behavior).
+
+## Directives added
+
+The DaemonSet adds the following CoreDNS directives where needed:
+
+```text
+reload 10s
+prefer_udp
+health_check 5s
+failfast_all_unhealthy_upstreams
+```
+
+`reload 10s` is added to each default `.:53` server block so CoreDNS can
+automatically reload future Corefile changes.
+
+`prefer_udp` is added inside the `forward` plugin for each default `.:53` server
+block when `ADD_DEFAULT_SERVER_PREFER_UDP` is set to `"true"`. Set that variable
+to `"true"` only when the customer requested `Protocol: PreferUDP` for the
+default LocalDNS overrides. The DaemonSet does not add it to blocks that already
+use `force_tcp`.
+
+`health_check 5s` is added inside the `forward` plugin for each default `.:53`
+server block. The DaemonSet does not add it to `cluster.local` /
+ClusterCoreDNS-forwarded blocks, custom suffix blocks such as `wmt:53`, or
+blocks that already use `force_tcp`.
+
+`failfast_all_unhealthy_upstreams` is added inside the `forward` plugin for each
+default `.:53` server block. It is intentionally not added to `cluster.local` /
+CoreDNS-forwarded blocks or custom suffix blocks such as `wmt:53`.
+
+## Restart behavior
+
+The DaemonSet runs:
+
+```text
+systemctl restart localdns
+```
+
+only once on a node after the first reconcile pass that actually changes
+LocalDNS files.
+
+Patching `updated.localdns.corefile` alone is not enough for an already-running
+CoreDNS process to pick up changes unless that process was already started with
+the CoreDNS `reload` plugin present in its Corefile.
+
+Therefore:
+
+- The first successful patch pass restarts `localdns` once so the running
+  CoreDNS process immediately loads the patched Corefile.
+- The DaemonSet writes a node-local restart marker after that successful restart.
+- Later reconcile loops do not restart `localdns` again on that node, even if
+  they repair regenerated files.
+- After the restart, `reload 10s` can pick up future Corefile edits without
+  another restart.
+
+The DaemonSet keeps reconciling files every 60 seconds.
+
+## Idempotency
+
+Each reconcile pass writes patched output to a temporary file and compares it
+with the existing file. It only writes when content differs.
+
+When a file is changed, the previous content is backed up with a
+`.pre-localdns-patch` suffix.
+
+## Rollback
+
+Use `localdns-corefile-rollback-ds.yaml` to undo the patch on nodes where the
+patch DaemonSet created `.pre-localdns-patch` backups.
+
+Delete the patch DaemonSet before applying the rollback DaemonSet:
+
+```bash
+kubectl -n kube-system delete ds localdns-corefile-patch --ignore-not-found=true
+kubectl apply -f localdns-corefile-rollback-ds.yaml
+```
+
+Do not run the patch DaemonSet and rollback DaemonSet at the same time. If both
+are running, they can fight over the same LocalDNS files.
+
+Both DaemonSets also write node-local active markers and refuse to reconcile if
+the opposite DaemonSet is active on the same node:
+
+```text
+/opt/azure/containers/localdns/.localdns-corefile-patch-active
+/opt/azure/containers/localdns/.localdns-corefile-rollback-active
+```
+
+The markers contain the host PID of the active DaemonSet shell. If a marker is
+stale because that process exited, the next reconcile pass removes it.
+
+The rollback DaemonSet requires all three `.pre-localdns-patch` backups before
+it restores anything, so it does not partially roll back a node:
+
+```text
+/etc/localdns/environment
+/opt/azure/containers/localdns/localdns.corefile
+/opt/azure/containers/localdns/updated.localdns.corefile
+```
+
+This restores the original pre-patch content exactly, including removing
+`reload 10s` if it was introduced by the patch DaemonSet.
+
+If it restores any file, it restarts `localdns` so the running CoreDNS process
+loads the restored Corefile. Rollback does not use a one-time restart marker:
+because rollback restores the original Corefile and may remove `reload 10s`,
+each later repair must restart `localdns` again to ensure the running process
+loads the restored file.
+
+Rollback also removes the patch DaemonSet's one-time restart marker:
+
+```text
+/opt/azure/containers/localdns/.localdns-corefile-patch-restarted
+```
+
+This ensures a later patch reapply on the same node performs its required
+one-time `localdns` restart.
+
+Each reconcile pass also verifies rollback by checking that all three files
+match their backups exactly and that the restored `updated.localdns.corefile`
+does not contain patch directives:
+
+```text
+reload 10s
+prefer_udp
+health_check 5s
+failfast_all_unhealthy_upstreams
+```
+
+After rollback is verified, delete the rollback DaemonSet:
+
+```bash
+kubectl -n kube-system delete ds localdns-corefile-rollback --ignore-not-found=true
+```