feat: add LocalDNS forward health knobs

.github/copilot-instructions.md

@@ -17,9 +17,9 @@ Windows VHD is configured through [VHD](./vhdbuilder/packer/windows/windows-vhd-
 
 [apiserver](./apiserver/) is `go` based webserver. It receives request from external client and generates CSE and CustomData to be used on the VHD when a new node is created / provisioned.
 
-windows generates its CSE package using [script](./parts/windows/kuberneteswindowssetup.ps1).
+windows generates its CSE package using [script](./parts/windows/kuberneteswindowssetup.ps1.template).
 
-Both CSE scripts run in two phases — Windows `BasePrep`/`NodePrep` (`kuberneteswindowssetup.ps1`) and Linux `basePrep`/`nodePrep` (`cse_main.sh`). `basePrep` is gated only by the `base_prep.complete` marker, so it is **skipped on a PIS-cached VHD** (the marker is baked in); `nodePrep` runs whenever `PreProvisionOnly` is false — every real node. So node-specific, secret, or expiring data (e.g. the TLS bootstrap token / kubeconfig) must be written in `nodePrep`, not `basePrep`, or the baked copy goes stale. On PIS the phases are separate VM runs — variables re-initialize from the real node's live CustomData, so `nodePrep` can't rely on `basePrep` state. (Non-PIS: both run sequentially in one execution, no reboot between phases.)
+Both CSE scripts run in two phases — Windows `BasePrep`/`NodePrep` (`kuberneteswindowssetup.ps1.template`) and Linux `basePrep`/`nodePrep` (`cse_main.sh`). `basePrep` is gated only by the `base_prep.complete` marker, so it is **skipped on a PIS-cached VHD** (the marker is baked in); `nodePrep` runs whenever `PreProvisionOnly` is false — every real node. So node-specific, secret, or expiring data (e.g. the TLS bootstrap token / kubeconfig) must be written in `nodePrep`, not `basePrep`, or the baked copy goes stale. On PIS the phases are separate VM runs — variables re-initialize from the real node's live CustomData, so `nodePrep` can't rely on `basePrep` state. (Non-PIS: both run sequentially in one execution, no reboot between phases.)
 
 The webserver is also used to determine the latest version of Linux VHDs available for provisioning within AKS clusters.
 
@@ -145,7 +145,7 @@ Analyze PRs for these compatibility scenarios:
   - Hardcoded paths that differ between deployment modes
 
 **4. PIS / VHD Caching — basePrep vs nodePrep split (Windows + Linux)**
-- **Context**: PIS bakes a VHD from a temporary VM, then boots many real nodes from it. Same model in Windows `parts/windows/kuberneteswindowssetup.ps1` (`BasePrep`/`NodePrep`) and Linux `parts/linux/cloud-init/artifacts/cse_main.sh` (`basePrep`/`nodePrep`):
+- **Context**: PIS bakes a VHD from a temporary VM, then boots many real nodes from it. Same model in Windows `parts/windows/kuberneteswindowssetup.ps1.template` (`BasePrep`/`NodePrep`) and Linux `parts/linux/cloud-init/artifacts/cse_main.sh` (`basePrep`/`nodePrep`):
   - `basePrep` — gated only by the `base_prep.complete` marker (`C:\AzureData\` Windows, `/opt/azure/containers/` Linux). **Skipped on PIS real nodes** because the marker is baked into the VHD.
   - `nodePrep` — runs whenever `PreProvisionOnly` is false (every real node); skipped only on the bake VM.
   - The bake run sets `PreProvisionOnly=true` (`{{GetPreProvisionOnly}}`) and writes the marker after `basePrep` succeeds (Windows `finally`; Linux `cse_start.sh`).

.github/renovate.json

@@ -58,7 +58,8 @@
         "custom.deb2004",
         "custom.deb2204",
         "custom.deb2404",
-        "custom.deb2404-test"
+        "custom.deb2404-test",
+        "custom.deb2604"
       ],
       "matchUpdateTypes": [
         "patch"
@@ -845,6 +846,20 @@
       "versioningTemplate": "deb",
       "autoReplaceStringTemplate": "\"renovateTag\": \"name={{{packageName}}}, repository=test, os=ubuntu, release=24.04\",\n                \"latestVersion\": \"{{{newValue}}}\"{{#if depType}},\n                \"previousLatestVersion\": \"{{{currentValue}}}\"{{/if}}"
     },
+    {
+      "customType": "regex",
+      "description": "auto update packages for OS ubuntu 26.04 in components.json",
+      "managerFilePatterns": [
+        "/parts/common/components.json/"
+      ],
+      "matchStringsStrategy": "any",
+      "matchStrings": [
+        "\"renovateTag\":\\s*\"name=(?<packageName>[^\"]+), repository=production, os=ubuntu, release=26\\.04\",\\s*\"latestVersion\":\\s*\"(?<currentValue>[^\"]+)\"(?:[^}]*\"previousLatestVersion\":\\s*\"(?<depType>[^\"]+)\")?"
+      ],
+      "datasourceTemplate": "custom.deb2604",
+      "versioningTemplate": "deb",
+      "autoReplaceStringTemplate": "\"renovateTag\": \"name={{{packageName}}}, repository=production, os=ubuntu, release=26.04\",\n                \"latestVersion\": \"{{{newValue}}}\"{{#if depType}},\n                \"previousLatestVersion\": \"{{{currentValue}}}\"{{/if}}"
+    },
     {
       "customType": "regex",
       "description": "auto update packages for OS Mariner 2.0 in components.json",
@@ -996,6 +1011,13 @@
       "transformTemplates": [
         "{\"releases\": $map(($index := releases#$i[version=\"Package: {{packageName}}\"].$i; $map($index, function($i) { $substringAfter(releases[$i + 1].version, \"Version: \") })), function($v) { {\"version\": $v} })[]}"
       ]
+    },
+    "deb2604": {
+      "defaultRegistryUrlTemplate": "https://packages.microsoft.com/ubuntu/26.04/prod/dists/resolute/main/binary-amd64/Packages",
+      "format": "plain",
+      "transformTemplates": [
+        "{\"releases\": $map(($index := releases#$i[version=\"Package: {{packageName}}\"].$i; $map($index, function($i) { $substringAfter(releases[$i + 1].version, \"Version: \") })), function($v) { {\"version\": $v} })[]}"
+      ]
     }
   }
 }

.github/workflows/codeql-analysis.yml

@@ -48,7 +48,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v4.37.3
+      uses: github/codeql-action/init@v4.37.6
       with:
         languages: ${{ matrix.language }}
         # If you wish to specify custom queries, you can do so here or in a config file.
@@ -62,7 +62,7 @@ jobs:
     # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v4.37.3
+      uses: github/codeql-action/autobuild@v4.37.6
 
     # ℹ️ Command-line programs to run using the OS shell.
     # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -75,4 +75,4 @@ jobs:
     #   ./location_of_script_within_repo/buildscript.sh
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v4.37.3
+      uses: github/codeql-action/analyze@v4.37.6

.pipelines/.vsts-vhd-builder.yaml

@@ -183,6 +183,32 @@ stages:
           parameters:
             artifactName: azurelinuxv3-gen2
 
+    # Kata VHDs are built here so that VHD-side Kata changes (kata packages, erofs tooling,
+    # kata configuration files) are covered by the E2E stage below. Without this job no Kata
+    # image carries this build's buildId tag, so Test_AzureLinuxV3Gen2Kata silently skips
+    # (the E2E stage sets IgnoreScenariosWithMissingVhd: true).
+    - job: buildAzureLinuxV3gen2kata
+      timeoutInMinutes: 360
+      steps:
+        - bash: |
+            echo '##vso[task.setvariable variable=OS_SKU]AzureLinux'
+            echo '##vso[task.setvariable variable=OS_VERSION]V3kata'
+            echo '##vso[task.setvariable variable=IMG_PUBLISHER]MicrosoftCBLMariner'
+            echo '##vso[task.setvariable variable=IMG_OFFER]azure-linux-3'
+            echo '##vso[task.setvariable variable=IMG_SKU]azure-linux-3-gen2'
+            echo '##vso[task.setvariable variable=IMG_VERSION]latest'
+            echo '##vso[task.setvariable variable=HYPERV_GENERATION]V2'
+            echo '##vso[task.setvariable variable=AZURE_VM_SIZE]Standard_D16ads_v5'
+            echo '##vso[task.setvariable variable=FEATURE_FLAGS]kata'
+            echo '##vso[task.setvariable variable=ARCHITECTURE]X86_64'
+            echo '##vso[task.setvariable variable=ENABLE_FIPS]false'
+            echo '##vso[task.setvariable variable=ENABLE_TRUSTED_LAUNCH]False'
+            echo '##vso[task.setvariable variable=ENABLE_CGROUPV2]True'
+          displayName: Setup Build Variables
+        - template: ./templates/.builder-release-template.yaml
+          parameters:
+            artifactName: azurelinuxv3-gen2-kata
+
     - job: buildAzureLinuxV3ARM64gen2fips
       timeoutInMinutes: 360
       steps:

.pipelines/e2e-gpu.yaml

@@ -37,5 +37,5 @@ jobs:
     parameters:
       name: Ubuntu GPU Tests
       IgnoreScenariosWithMissingVhd: false
-      jobTimeoutInMinutes: 120
+      jobTimeoutInMinutes: 180
       failedTestsRetryCount: ${{ variables.E2E_FAILED_TESTS_RETRY_COUNT }}

AGENTS.md

@@ -17,18 +17,16 @@ Windows VHD is configured through [VHD](./vhdbuilder/packer/windows/windows-vhd-
 
 [apiserver](./apiserver/) is `go` based webserver. It receives request from external client and generates CSE and CustomData to be used on the VHD when a new node is created / provisioned.
 
-windows generates its CSE package using [script](./parts/windows/kuberneteswindowssetup.ps1).
+windows generates its CSE package using [script](./parts/windows/kuberneteswindowssetup.ps1.template).
 
-Both CSE scripts run in two phases — Windows `BasePrep`/`NodePrep` (`kuberneteswindowssetup.ps1`) and Linux `basePrep`/`nodePrep` (`cse_main.sh`). `basePrep` is gated only by the `base_prep.complete` marker, so it is **skipped on a PIS-cached VHD** (the marker is baked in); `nodePrep` runs whenever `PreProvisionOnly` is false — every real node. So node-specific, secret, or expiring data (e.g. the TLS bootstrap token / kubeconfig) must be written in `nodePrep`, not `basePrep`, or the baked copy goes stale. On PIS the phases are separate VM runs — variables re-initialize from the real node's live CustomData, so `nodePrep` can't rely on `basePrep` state. (Non-PIS: both run sequentially in one execution, no reboot between phases.)
+Both CSE scripts run in two phases — Windows `BasePrep`/`NodePrep` (`kuberneteswindowssetup.ps1.template`) and Linux `basePrep`/`nodePrep` (`cse_main.sh`). `basePrep` is gated only by the `base_prep.complete` marker, so it is **skipped on a PIS-cached VHD** (the marker is baked in); `nodePrep` runs whenever `PreProvisionOnly` is false — every real node. So node-specific, secret, or expiring data (e.g. the TLS bootstrap token / kubeconfig) must be written in `nodePrep`, not `basePrep`, or the baked copy goes stale. On PIS the phases are separate VM runs — variables re-initialize from the real node's live CustomData, so `nodePrep` can't rely on `basePrep` state. (Non-PIS: both run sequentially in one execution, no reboot between phases.)
 
 The webserver is also used to determine the latest version of Linux VHDs available for provisioning within AKS clusters.
 
 ## Code Structure
 
 [parts](./parts/) serves both AgentBaker Service and VHD build. AgentBaker service and VHDs are coupled because of this shared component. When building VHD, packer maps and renames scripts from [parts](./parts/)  depending on the OS / versions. The mappings can be found at [packer](./vhdbuilder/packer/).
 
-> **IMPORTANT**: When making changes to files in the `parts` or `pkg` directories, you must run `make generate` afterward to regenerate the snapshot test data. This ensures consistency between the code and tests and prevents regressions.
-
 Windows uses a different folder [cse](./staging/cse/windows/) for almost the same purpose. There are subtle differences as windows CSEs can be downloaded as a zip file during provisioning time due to restrictions on the file size on Windows system, while for linux based systems the cse/custom data are dropped in during provisioning time.
 
 ## Deployment and Release
@@ -61,7 +59,7 @@ The operational goals of this project are:
 
 When making changes, reason whether the file is used in VHD building stage, or provision stage, or both. Make sure the changes are valid in its life stage. as an example, [windows-vhd-configuration.ps1](./vhdbuilder/packer/windows/windows-vhd-configuration.ps1) defines container images to be cached in VHD, while [configure-windows-vhd.ps1](./vhdbuilder/packer/windows/configure-windows-vhd.ps1) executes commands at provision time.
 
-One way to debug / explore / just for fun is to run [e2e](./e2e/) tests. To run locally, follow the readme file under that folder. 
+One way to debug / explore / just for fun is to run [e2e](./e2e/) tests. To run locally, follow the readme file under that folder.
 
 The SRE guidelines ground other coding guidelines and practices.
 
@@ -94,6 +92,7 @@ When reviewing pull requests, perform breaking change analysis to prevent regres
 Analyze PRs for these compatibility scenarios:
 
 **1. Linux Provisioning Script Changes**
+
 - **Context**: Scripts in `parts/linux/cloud-init/artifacts/` run during critical VM bootstrap and are used in both:
   - VHD build (uploaded via packer configs in `vhdbuilder/packer/*.json`)
   - VM provisioning (CSE - embedded in Go service via `pkg/agent/const.go`)
@@ -118,6 +117,7 @@ Analyze PRs for these compatibility scenarios:
   - **ANC hotfix entry removal**: If a PR removes or modifies the `hotfix-scripts: auto-generated` block in `parts/linux/cloud-init/nodecustomdata.yml`, or resets `parts/linux/cloud-init/artifacts/aks-node-controller-hotfix.json` to `{}`, **always confirm with the PR owner** that all affected VHDs have been republished with the fix baked in or are out of the 6-month support window. Premature removal means nodes provisioned via scale-up on the old buggy VHD will no longer receive the hotfix. These files are auto-generated by `hotfix/hotfix_generate.py` (via the `hotfix-generate` GH Action) — see that script for how `version`/`scripts_version` are computed.
 
 **2. Windows Bidirectional Compatibility**
+
 - **Context**: Windows VHD and CSE scripts release on different cadences with no guaranteed order
 - **What to check**: Changes to `staging/cse/windows/` (CSE scripts) or `vhdbuilder/packer/windows/` (VHD scripts)
 - **Breaking signals**:
@@ -127,6 +127,7 @@ Analyze PRs for these compatibility scenarios:
   - Removing PowerShell functions or cmdlets that the other component might call
 
 **3. aks-node-controller Migration (Dual-Mode Support)**
+
 - **Context**: Transitioning from uploading scripts during both VHD build and CSE to only uploading aks-node-controller during VHD build
 - **What to check**: Any changes must work in BOTH deployment modes
 - **Breaking signals**:
@@ -136,7 +137,8 @@ Analyze PRs for these compatibility scenarios:
   - Hardcoded paths that differ between deployment modes
 
 **4. PIS / VHD Caching — basePrep vs nodePrep split (Windows + Linux)**
-- **Context**: PIS bakes a VHD from a temporary VM, then boots many real nodes from it. Same model in Windows `parts/windows/kuberneteswindowssetup.ps1` (`BasePrep`/`NodePrep`) and Linux `parts/linux/cloud-init/artifacts/cse_main.sh` (`basePrep`/`nodePrep`):
+
+- **Context**: PIS bakes a VHD from a temporary VM, then boots many real nodes from it. Same model in Windows `parts/windows/kuberneteswindowssetup.ps1.template` (`BasePrep`/`NodePrep`) and Linux `parts/linux/cloud-init/artifacts/cse_main.sh` (`basePrep`/`nodePrep`):
   - `basePrep` — gated only by the `base_prep.complete` marker (`C:\AzureData\` Windows, `/opt/azure/containers/` Linux). **Skipped on PIS real nodes** because the marker is baked into the VHD.
   - `nodePrep` — runs whenever `PreProvisionOnly` is false (every real node); skipped only on the bake VM.
   - The bake run sets `PreProvisionOnly=true` (`{{GetPreProvisionOnly}}`) and writes the marker after `basePrep` succeeds (Windows `finally`; Linux `cse_start.sh`).
@@ -150,6 +152,7 @@ Analyze PRs for these compatibility scenarios:
 - **Don't flag**: plain variable reads (live on the real node from CustomData); cached binaries/packages/images; the `base_prep.complete` marker; cluster-wide non-secrets (CA cert, apiserver FQDN, service CIDR); pre-existing `basePrep` writes unless the PR newly depends on them.
 
 **5. Cross-OS Compatibility**
+
 - **What to check**: Changes work on Ubuntu, Azure Linux/Mariner, and Windows
 - **Breaking signals**:
   - Linux commands that don't work on both Ubuntu and Azure Linux/Mariner
@@ -158,6 +161,7 @@ Analyze PRs for these compatibility scenarios:
   - Systemd differences between distributions
 
 **6. Package/Dependency Update PRs (Renovate)**
+
 - **Context**: Renovate bot automatically creates PRs to update component versions in `parts/common/components.json`. These components are cached on VHDs during build and directly affect node stability, GPU workloads, networking, and security. Updated packages are downloaded from `packages.aks.azure.com` or upstream registries during VHD build.
 - **What to check**: Every version bump—even patch versions—can introduce regressions that affect production nodes.
 - **Analysis steps for every package update PR**:
@@ -181,6 +185,7 @@ Analyze PRs for these compatibility scenarios:
 - **Review output for package update PRs must include a detailed version diff analysis**:
 
   **Header:**
+
   ```
   ## Package Update Analysis: <component-name>
   **Version change**: X.Y.Z → A.B.C (<major|minor|patch> update)
@@ -215,13 +220,15 @@ Analyze PRs for these compatibility scenarios:
   **If upstream changelog is unavailable**, explicitly state: _"Upstream changelog not found for this version range. Manual testing recommended before merge."_
 
   **Overall risk assessment:**
+
   ```
   ### Overall Risk: 🟢 Low / 🟡 Medium / 🔴 High
   **Justification**: <1-2 sentence summary of why this risk level was chosen>
   **Recommendation**: Approve / Request more info / Flag for manual testing
   ```
 
   **Example** (for a PR like dcgm-exporter 4.7.1 → 4.8.0):
+
   ```
   ## Package Update Analysis: dcgm-exporter
   **Version change**: 4.7.1 → 4.8.0 (minor update)
@@ -244,6 +251,7 @@ Analyze PRs for these compatibility scenarios:
 ### Analysis Approach
 
 **Dynamic Dependency Tracing**:
+
 1. For each changed file, identify what depends on it
 2. Follow `source` statements in bash scripts to trace dependency chains
 3. Check for function calls, variable references across files
@@ -255,10 +263,12 @@ Analyze PRs for these compatibility scenarios:
    - Flag downloads from unauthorized sources (only packages.aks.azure.com and sources in components.json allowed)
 
 **Historical Context**:
+
 - Look for related changes that previously caused issues
 - Identify patterns of fragile areas that break frequently
 
 **Test Coverage Assessment**:
+
 - Note if changed code has e2e test coverage
 - Flag changes to untested areas as higher risk
 - Mention if new behavior lacks corresponding test additions
@@ -268,6 +278,7 @@ Analyze PRs for these compatibility scenarios:
 Provide targeted inline comments on specific lines where you detect issues:
 
 **For each breaking change or risk:**
+
 - Comment directly on the problematic line or code block
 - Explain why this is risky (e.g., "This removes function X which may be called by VHDs built in the last 6 months")
 - Suggest specific mitigations or alternatives
@@ -296,9 +307,14 @@ Provide targeted inline comments on specific lines where you detect issues:
 ### Review Philosophy
 
 Think like an experienced reviewer who "eyeballs" PRs for subtle risks. Look beyond pattern matching:
+
 - Understand the architecture and how components interact
 - Consider timing of releases and deployment sequences
 - Reason about implicit dependencies and assumptions
 - Flag changes that "feel risky" even without obvious red flags
 - Balance thoroughness with actionable feedback
-- Focus on high-impact issues that could break production VM provisioning
+- Focus on high-impact issues that could break production VM provisioning
+- Reason about implicit dependencies and assumptions
+- Flag changes that "feel risky" even without obvious red flags
+- Balance thoroughness with actionable feedback
+- Focus on high-impact issues that could break production VM provisioning