[DO NOT REVIEW]feat(anc): verify and extract repository hotfix packages

[Abigail Liang (abigailliang-aks-sig-node)]

What this PR does

Keeps the LPS hotfix contract version-only and implements the ANC repository fast path on the node.

After the configured hotfix resolves to a strictly newer patch for the current ANC release, ANC now:

• downloads the package and authenticated repository metadata concurrently;
• derives repository endpoints from the node's configured apt/yum sources, including RepoDepot rewrites;
• verifies Ubuntu InRelease and Azure Linux/Mariner repomd.xml signatures using installed repository keys;
• validates metadata size, SHA-256, package location, version, OS, and architecture;
• verifies the downloaded .deb/.rpm SHA-256 against authenticated metadata;
• extracts and atomically stages only usr/bin/aks-node-controller, never package bytes;
• falls back to apt/dnf for unsupported configurations and operational download failures;
• hard-fails integrity violations and removes any stale staged hotfix binary.

Repository traffic starts only after the existing hotfix version and newer-patch gates pass, so inactive or non-matching hotfix pointers do not download metadata or packages.

Mooncake Bootstrap Registry is not treated as PMC; unsupported repository layouts continue through the existing package-manager fallback.

Prerequisite: the hotfix must be published to PMC first

The fast path derives one deterministic URL from name+version+arch+codename, so it can only
find a build that has already been published to packages.microsoft.com. This is a
precondition, not a new constraint: an unpublished version 404s, which is an operational
failure rather than an integrity one, so it falls through to the package manager -- which
cannot install that version either. Neither path can install what has not shipped.

Worth noting for Azure Linux/Mariner: the AzureLinux ms-oss repositories currently publish
aks-node-controller RPMs and the Mariner Microsoft repository does not, so Mariner will
keep taking the fallback until that changes, regardless of the discovery fix in this PR.

Tracks work item 39535914.

Performance

Joined-node measurements on a Ready Ubuntu 24.04 AKS node
(aks-nodepool1-27955216-vmss000000), target ANC 202608.26.2-ubuntu24.04u1, 5 iterations
each. Test plan.

Path	p50	p95-ish/max
full apt-get update + reinstall	7990 ms	9494 ms
scoped ANC repo update + reinstall	1630 ms	1839 ms
serial direct resolve + download + SHA + extract + stage	184 ms	373 ms

Scoped update alone only measures metadata refresh, not install/dpkg cost, so the end-to-end scoped apt row is the relevant comparison. The direct row is a serial shell approximation, but it includes the important non-download work (dpkg-deb -x and binary staging) and is still much faster than scoped apt.

Joined-node timings are expected to be faster than pre-join/bootstrap because the node is already settled: package caches/network paths may be warm, kubelet/containerd/image pull work is no longer competing for CPU/IO, and dpkg is less likely to be contending with other bootstrap operations. For real pre-join/bootstrap timing, use the marker PR (#9424) once merged into this branch and compare route=direct-http vs route=package-manager terminal durations.

Security trade-off

Bypassing apt-get install also bypasses what dpkg provides: dependency resolution, the
package database, maintainer scripts, and file-conflict checks. The fast path is only sound
because ANC is a standalone binary with no maintainer-script setup; if that stops being
true, the fast path is no longer sufficient and the apt fallback is required.

The digest is not taken on trust from a config or an LPS response. It is derived at runtime
from PMC's signed metadata, so the trust model matches apt's:

gpgv verifies InRelease
  -> InRelease carries the digest of Packages.gz
    -> Packages.gz carries the digest of the .deb
      -> the downloaded .deb is verified against it

A SHA-256 only proves the bytes match an expected digest; it cannot prove the digest is
authentic. Chaining to InRelease is what makes it authentic, and it is why the .deb digest
must still be checked: GPG protects the metadata, not the package. If PMC's signing chain
itself were compromised, apt would be equally affected — this path is no weaker than apt, and
no stronger.

Alongside that: the package path is derived deterministically and cross-checked against the
signed Filename, decompression is bounded, only usr/bin/aks-node-controller is extracted,
and any validation failure falls back to scoped apt with a pinned version.

Testing

• cd aks-node-controller && go build -mod=readonly ./...
• cd aks-node-controller && go test ./...
• cd aks-node-controller && go test ./... -race

Fast-path behaviour is pinned by tests that fail against the pre-change code:

• TestUbuntuFastPathPrefersCompressedPackagesIndex — asserts Packages.gz is fetched and
  the plain index is not.
• TestUbuntuFastPathFallsBackToPlainPackagesIndex — repositories without a .gz entry still work.
• TestUbuntuFastPathRejectsTamperedCompressedIndex — a checksum mismatch on the compressed
  index is an integrity failure, and nothing is staged.
• TestUbuntuFastPathRequestsExactlyTheExpectedURLs — asserts the complete request log
  (InRelease, one index, one .deb), so any extra fetch fails rather than passing unnoticed.
• TestUbuntuRepositoryFastPathParallelSuccessExtractsBinary — both branches must start
  concurrently or the test times out.
• TestRepositoryFastPathCancelsPeerBranchOnFailure — a fast failure cancels its peer instead
  of waiting out gpgv's 60 s timeout; fails in ~10 s against the pre-change code.

[Copilot]

🟢 Approval recommended

The focused removal is internally consistent, backward-compatible with stale JSON fields, and adequately tested.

Pull request overview

Removes ANC’s unsafe direct-download path, retaining package-manager installation and binary staging.

Changes:

• Removes artifact descriptors, HTTP download logic, and test hooks.
• Ignores legacy artifacts fields and removes them during pointer rewrites.
• Updates tests to verify package-manager use and artifact cleanup.

File summaries

File	Description
aks-node-controller/hotfix.go	Removes direct artifact downloads.
aks-node-controller/hotfix_test.go	Updates package-manager behavior tests.
aks-node-controller/checkhotfix.go	Removes artifacts from parsing and persistence.
aks-node-controller/checkhotfix_test.go	Verifies stale artifacts are dropped.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Windows Unit Test Results

  3 files   14 suites   48s ⏱️
415 tests 415 ✅ 0 💤 0 ❌
418 runs  418 ✅ 0 💤 0 ❌

Results for commit 2bdbe0b.

♻️ This comment has been updated with latest results.

[Copilot]

🔵 Needs a closer look

Azure Linux thin-image repository resolution, verifier timeout handling, and fallback timing reporting need correction.

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

aks-node-controller/hotfix.go:114

• 🟡 Medium Risk — ⚡ Performance: This timer starts only after the repository attempt fails, but the new bootstrap scenario reports it as the hotfix path duration. A network or verifier timeout can consume 30–60 seconds before this line, and all of that provisioning latency is omitted from the reported fallback result. Start an overall timer before tryRepositoryDownload (or emit a separate total-duration field) and have the E2E parser use that total.
  aks-node-controller/repository_hotfix.go:408
• 🟡 Medium Risk — 🔄 Backward Compatibility: Every gpgv failure except “not found” is classified as an integrity violation, but runRepositoryCommand returns context.DeadlineExceeded when the verifier times out. Under bootstrap CPU/IO contention, that operational timeout therefore skips the documented apt/dnf fallback and disarms an otherwise valid staged hotfix. Reserve integrityError for an actual *exec.ExitError from verification; treat timeout/cancellation or process-launch failures as operational so fallback remains available.
  aks-node-controller/repository_hotfix.go:1085
• 🟡 Medium Risk — 🖥️ Cross-OS: Substituting the raw VERSION_ID breaks the fast path on Azure Container Linux 3 thin images. Those images report values such as 3.0.20260809 (vhdbuilder/release-notes/AKSAzureContainerLinux/gen2tl/latest.txt:161-168), while PMC is rooted at /azurelinux/3.0/ (parts/common/components.json:977-982), so every direct request uses a nonexistent repository and falls back. Normalize Azure Linux 3 to its repository release (3.0) before expanding $releasever, and add a dated-VERSION_ID regression case.

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Azure Linux repository resolution and bootstrap timing reporting contain correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Azure Linux thin-image repository resolution is incorrect, and fallback timing omits repository-attempt latency.

Review details

Suppressed comments (2)

aks-node-controller/repository_hotfix.go:1083

• 🟡 Medium Risk — Cross-OS: Substituting the raw VERSION_ID breaks the fast path on Azure Container Linux thin images. Those images report values such as 3.0.20260809 (vhdbuilder/release-notes/AKSAzureContainerLinux/gen2tl/latest.txt:161-168), while PMC publishes this repository under azurelinux/3.0 (parts/common/components.json:973-978). The generated URL therefore 404s and every thin-image hotfix falls back to dnf. Resolve $releasever using the distro/package-manager release value (or normalize supported Azure Linux 3 variants to 3.0) and cover the dated VERSION_ID in a plan test.

	baseURL := strings.ReplaceAll(repository.BaseURL, "$releasever", info.VersionID)
	baseURL = strings.ReplaceAll(baseURL, "${releasever}", info.VersionID)

aks-node-controller/hotfix.go:114

• 🟡 Medium Risk — Performance: This timer starts only after tryRepositoryDownload returns. A package or metadata timeout can add tens of seconds before apt/dnf starts, yet the completion line—and validateHotfixBootstrapTiming—reports only the package-manager portion as the bootstrap hotfix duration. That hides the main fallback latency this measurement is intended to expose. Start an overall timer before the repository attempt, or emit a separate overall duration and have the scenario report it.

	pmcStart := time.Now()

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Verifier timeouts are misclassified as integrity failures, performance measurements can be misleading, and formatting currently fails the repository formatter.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

aks-node-controller/repository_hotfix.go:408

• 🟡 Medium Risk — 🏗️ Architecture: runRepositoryCommand returns context.DeadlineExceeded when the 60-second verifier timeout fires, but this branch converts that operational timeout into an integrity failure. The caller then removes the staged hotfix and skips apt/dnf fallback even though no invalid signature was established. Preserve cancellation/deadline errors as ordinary operational errors, and reserve integrityError for an actual nonzero verification result.

aks-node-controller/hotfix.go:114

• 🟡 Medium Risk — ⚡ Performance: This timer starts only after tryRepositoryDownload has failed, while the fast-path timer starts before its network work. A fallback caused by a 30–60 second timeout will therefore report only the subsequent apt/dnf time, so parseHotfixTiming underreports bootstrap cost and the two route measurements are not comparable. Start an overall timer before the fast-path attempt (or emit separate overall and package-manager phase durations).

	pmcStart := time.Now()

• Files reviewed: 10/10 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

Proxy-aware repository access and meaningful hotfix activation in the timing scenarios must be addressed.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

aks-node-controller/repository_hotfix.go:345

• 🟡 Medium Risk — ⚡ Performance: NewBaseTransport deliberately sets Proxy: nil (aks-node-controller/common/httpclient.go:37-45). Reusing it here means every repository request bypasses the node's HTTP(S)_PROXY settings. On proxy-only clusters the new fast path waits for a direct-connect timeout before falling back, adding provisioning latency and never exercising the advertised fast path. Repository traffic should use a transport that honors the node proxy configuration (including NO_PROXY) while retaining these timeouts and redirect checks.

e2e/scenario_hotfix_bootstrap_perf.go:90

• 🟡 Medium Risk — 🧪 Test Coverage: These scenarios are registered specifically to collect bootstrap timing, but they succeed when no hotfix ran. This branch does not stage a hotfix pointer by default, and the scenarios neither inject one nor enable the LPS refresh, so a normal run reports success without measuring either path. Make the scenario arrange an active, matching hotfix target and treat a missing completion line as a failed/explicitly skipped run; otherwise the new E2E coverage cannot detect that the fast path was never exercised.

	if errors.Is(err, errNoHotfixCompletionLine) {
		// Not a failure of the code under test: if no hotfix was configured for this run
		// there is nothing to time. Log it plainly and pass, rather than failing on absent
		// data or reporting a misleading zero. Validators no longer control test outcome
		// (see 70d6199c3e), so this cannot skip the test from here.
		s.Logger.Logf("NO BOOTSTRAP HOTFIX TIMING: %v (no hotfix ran on this node)", err)
		return nil

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

RPM extraction lacks equivalent safety checks, and the new timing scenarios can pass without exercising a hotfix path.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

e2e/scenario_hotfix_bootstrap_perf.go:90

• 🟡 Medium Risk — 🧪 Test Coverage: This makes both new timing scenarios pass when they collected no timing at all. Neither scenario injects aks-node-controller-hotfix.json nor enables the LPS hotfix feature, and this tree has no generated hotfix JSON, so the launcher will not run download-hotfix; both scenarios can therefore succeed without exercising either new path. Provision a known published target (or a deterministic LPS fixture) and treat a missing completion line as failure.

	if errors.Is(err, errNoHotfixCompletionLine) {
		// Not a failure of the code under test: if no hotfix was configured for this run
		// there is nothing to time. Log it plainly and pass, rather than failing on absent
		// data or reporting a misleading zero. Validators no longer control test outcome
		// (see 70d6199c3e), so this cannot skip the test from here.
		s.Logger.Logf("NO BOOTSTRAP HOTFIX TIMING: %v (no hotfix ran on this node)", err)
		return nil

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🔵 Needs a closer look

RPM extraction is unbounded, stale binaries can remain active, and the E2E scenarios can pass without exercising a hotfix.

Review details

Suppressed comments (3)

aks-node-controller/repository_hotfix.go:627

• The RPM path bypasses repositoryBinaryMaxBytes: cpio writes the selected member directly to disk, and copyBinaryAlongside then reads the entire result into memory. A highly compressible RPM can therefore expand a binary far beyond the 128 MiB limit enforced for .deb, exhausting disk or memory during provisioning. Stream the selected member through a bounded writer (and reject non-regular members) before staging it.

	cpio := exec.CommandContext(commandCtx, "cpio", "-idmu", "--quiet", "./usr/bin/aks-node-controller")

aks-node-controller/hotfix.go:172

• This still permits the launcher to execute the stale hotfix after an integrity failure when both unlink and chmod fail. The examples in the comment—an immutable file or read-only mount—typically prevent both operations, and the launcher ignores download-hotfix's exit status before selecting any executable hotfix. Use a durable disarm marker or make the launcher select the hotfix only after a successful download instead of relying solely on best-effort file mutation.

	if chmodErr := os.Chmod(path, 0o600); chmodErr != nil {
		slog.Error("stale hotfix binary remains executable after repository integrity failure",
			"path", path, "removeError", err, "chmodError", chmodErr)
		return

e2e/scenario_hotfix_bootstrap_perf.go:90

• These scenarios currently succeed without exercising a hotfix. Their configs neither inject the optional hotfix JSON nor enable ENABLE_PROVISIONING_HOTFIX, and this branch does not ship the static pointer, so the expected completion line is absent and this branch returns success. Make the scenario arrange a deterministic hotfix target and treat a missing completion line as a failure; otherwise a green run records no bootstrap timing and cannot validate either route.

	if errors.Is(err, errNoHotfixCompletionLine) {
		// Not a failure of the code under test: if no hotfix was configured for this run
		// there is nothing to time. Log it plainly and pass, rather than failing on absent
		// data or reporting a misleading zero. Validators no longer control test outcome
		// (see 70d6199c3e), so this cannot skip the test from here.
		s.Logger.Logf("NO BOOTSTRAP HOTFIX TIMING: %v (no hotfix ran on this node)", err)
		return nil

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[github-actions]

PR Title Lint Failed ❌

Current Title: [DO NOT REVIEW]feat(anc): verify and extract repository hotfix packages

Your PR title doesn't follow the expected format. Please update your PR title to follow one of these patterns:

Conventional Commits Format:

• feat: add new feature - for new features
• fix: resolve bug in component - for bug fixes
• docs: update README - for documentation changes
• refactor: improve code structure - for refactoring
• test: add unit tests - for test additions
• chore: remove dead code - for maintenance tasks
• chore(deps): update dependencies - for updating dependencies
• ci: update build pipeline - for CI/CD changes

Guidelines:

• Use lowercase for the type and description
• Keep the description concise but descriptive
• Use imperative mood (e.g., "add" not "adds" or "added")
• Don't end with a period

Examples:

• ✅ feat(windows): add secure TLS bootstrapping for Windows nodes
• ✅ fix: resolve kubelet certificate rotation issue
• ✅ docs: update installation guide
• ❌ Added new feature
• ❌ Fix bug.
• ❌ Update docs

Please update your PR title and the lint check will run again automatically.

[aks-node-assistant]

Failed gate run

• Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=180421906
• Failed job/stage/task: build / buildacltlgen2 / Build VHD

Detective summary

The first concrete failure was Packer ARM template validation before VM provisioning. WestUS3 standardDADSv5Family quota was exhausted: current usage 144 of 150, with 16 additional cores required, so the Packer VM deployment failed with InvalidTemplateDeployment / QuotaExceeded and Azure Pipelines surfaced script exit code 2.

Likely cause

Signature: vhd-westus3-standarddadsv5family-quota-exceeded. Classification: VHD/build infrastructure capacity quota. This is a recurring quota signature and is more likely shared regional capacity pressure than product, test-code, CSE, dependency, Packer-template, or PR-change-caused behavior.

Recommended owner/action

Node Lifecycle/VHD infrastructure owner: reduce concurrent DADSv5 demand in WestUS3 or request a quota increase; PR owner likely does not need code changes for this gate failure.

Strongest alternative

PR-caused ANC hotfix change increasing VHD resource usage is the strongest alternative; it is less likely because the failure occurs before provisioning on a fixed Packer VM SKU quota preflight, not in the PR's ANC hotfix package logic.

Evidence

• Timeline: buildacltlgen2 failed with Build VHD log 225; cleanup/publish steps then failed due to missing artifacts.
• Log: ARM preflight returned QuotaExceeded for WestUS3 standardDADSv5Family before Packer VM creation.
• Build metadata: PR 9375, source branch refs/pull/9375/merge, finish 2026-09-10T00:50:10Z.
• Changed files are ANC hotfix package extraction/E2E timing files, not Packer VM SKU or quota config.

Wiki signature

vhd-westus3-standarddadsv5family-quota-exceeded