Repository navigation
fix: validate cache-only Ubuntu GPU images and preserve prerequisites - #9693
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 4 commits into
Conversation
Disable shared-image host prebaking, retain installer prerequisites and fail final-image validation on NVIDIA registration or installed host residue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 1
Open (3)
What changed in this PR
Updates Ubuntu shared VHD GPU handling to cache the CUDA-LTS installer container without prebaking/installing the NVIDIA driver, and adds stricter validation to prevent publishing “contaminated” images.
Changes:
- Switch shared Ubuntu 22.04/24.04 amd64 Gen2 pipeline defaults to
FEATURE_FLAGS=None(cache-only). - Add build-time prerequisites + nouveau blacklist/initramfs refresh for cache-only images.
- Add content-test invariant checks (no DKMS registration/modules/artifacts; required prerequisites and installer cache present) plus new ShellSpec fixtures.
| File | Description |
|---|---|
| vhdbuilder/packer/test/linux-vhd-content-test.sh | Adds a new final-image invariant test to enforce “GPU cache-only” policy on shared Ubuntu images. |
| vhdbuilder/packer/install-dependencies.sh | Adds cache-only prerequisites setup (toolchain + nouveau blacklist + initramfs refresh) during image build. |
| parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh | Introduces shared predicate isUbuntuGPUCacheOnlyImage() used by both build and validation logic. |
| spec/vhdbuilder/packer/gpu_cache_only_spec.sh | Adds fixtures covering the new predicate, build prerequisite behavior, and invariant enforcement. |
| .pipelines/.vsts-vhd-builder.yaml | Changes shared-image jobs to set FEATURE_FLAGS=None (no prebake). |
| .pipelines/.vsts-vhd-builder-release.yaml | Same as above for release pipeline jobs. |
| vhdbuilder/packer/README.md | Documents the new shared Ubuntu GPU installer cache-only policy and validation expectations. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Windows Unit Test Results 3 files 18 suites 1m 11s ⏱️ Results for commit 387b780. ♻️ This comment has been updated with latest results. |
Exclude only the kernel platform backlight path from cache-only disk and initramfs checks. Keep rejecting DKMS-path copies and test compressed modules plus pipefail behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
Preserve merged PR 9694 pipeline defaults exactly; retain only cache-only prerequisites and image invariants as follow-up changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
changed the title
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 2
Open (7)
ctr images lstypically requires--filter "name==..."(or equivalent) rather than passing the… · New.downloadURLto the literal string…testUbuntuGPUCacheOnlyImagerequiresdkmsandinitramfs-toolsto be… · Newrootand writes the modprobe config under${root}, butupdate-initramfs… · New${root}/usr/bin/lib64appears inconsistent with typical Linux filesystem layouts… · New${root}/usr/bin/lib64is a surprising path to check for NVIDIA artifacts (lib directories are…rootbut only applies it to the modprobe config path;…
| err "$test" "Cannot resolve cached GPU installer reference" | ||
| return 1 | ||
| fi | ||
| if ! cached=$(ctr -n k8s.io images ls -q "name==$image") || [ "$cached" != "$image" ]; then |
| local root="${1:-}" | ||
|
|
||
| # Keep the ordinary node-time installer independent of the optional host prebake. | ||
| apt_get_install 10 2 300 gcc make libc6-dev || return 1 |
| apt_get_install 10 2 300 gcc make libc6-dev || return 1 | ||
| mkdir -p "${root}/etc/modprobe.d" || return 1 | ||
| printf 'blacklist nouveau\noptions nouveau modeset=0\n' > "${root}/etc/modprobe.d/blacklist-nouveau.conf" || return 1 | ||
| update-initramfs -u -k all || return 1 |
| fi | ||
| # Include dangling links and residue with a missing marker; do not silently sanitize the image. | ||
| for path in "${root}"/var/lib/dkms/nvidia* "${root}"/usr/src/nvidia* \ | ||
| "${root}"/usr/bin/nvidia* "${root}"/usr/sbin/nvidia* "${root}"/usr/bin/lib64 \ |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 3
Open (10)
jqis doing string equality against a value containing*(downloadURL == "…:*"), which won’t… · Newctr images lstypically requires--filter "name==..."(or equivalent) rather than passing the….downloadURLto the literal string…rootargument and writes the blacklist under${root}, but… · New${root}/usr/bin/lib64, which is an unusual/likely incorrect location… · New${root}/usr/bin/lib64appears inconsistent with typical Linux filesystem layouts…rootand writes the modprobe config under${root}, butupdate-initramfs…testUbuntuGPUCacheOnlyImagerequiresdkmsandinitramfs-toolsto be…${root}/usr/bin/lib64is a surprising path to check for NVIDIA artifacts (lib directories are…rootbut only applies it to the modprobe config path;…
| fi | ||
| done | ||
| if ! image=$(jq -er ' | ||
| [.GPUContainerImages[] | select(.downloadURL == "mcr.microsoft.com/aks/aks-gpu-cuda-lts:*") | .gpuVersion.latestVersion] |
| local root="${1:-}" | ||
|
|
||
| # Keep the ordinary node-time installer independent of the optional host prebake. | ||
| apt_get_install 10 2 300 gcc make libc6-dev || return 1 | ||
| mkdir -p "${root}/etc/modprobe.d" || return 1 | ||
| printf 'blacklist nouveau\noptions nouveau modeset=0\n' > "${root}/etc/modprobe.d/blacklist-nouveau.conf" || return 1 |
| fi | ||
| # Include dangling links and residue with a missing marker; do not silently sanitize the image. | ||
| for path in "${root}"/var/lib/dkms/nvidia* "${root}"/usr/src/nvidia* \ | ||
| "${root}"/usr/bin/nvidia* "${root}"/usr/sbin/nvidia* "${root}"/usr/bin/lib64 \ |
AgentBaker Linux gate detectiveRun: 183950052 Detective summary: Test/Scan retried the Ubuntu 24.04 GPU cache-only content validation twice and both attempts failed Likely cause / signature: Confidence: Medium for PR-change-caused test/spec expectation drift; the failure is deterministic within the run. Recommended owner/action: PR owner should confirm whether Strongest alternative: A host image or kernel package drift could introduce the module independently of this PR, but the changed files include GPU cache-only spec and content-test logic. Evidence: build run, failed task log/timeline, build metadata, PR changed files, and wiki tracker. |
Pass an explicit empty filesystem prefix at the two production call sites. Preserve the existing default-root behavior while satisfying CI ShellCheck SC2119/SC2120 for helpers with optional fixture roots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
|
|
|
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Current-head VHD validation and exact-image GPU, PIS, and kernel-update acceptance remain unconfirmed.
Review effort: Balanced
Findings: 2
Open (7)
jqis doing string equality against a value containing*(downloadURL == "…:*"), which won’t…ctr images lstypically requires--filter "name==..."(or equivalent) rather than passing the…${root}/usr/bin/lib64, which is an unusual/likely incorrect location…rootargument and writes the blacklist under${root}, but…${root}/usr/bin/lib64appears inconsistent with typical Linux filesystem layouts…rootand writes the modprobe config under${root}, butupdate-initramfs…testUbuntuGPUCacheOnlyImagerequiresdkmsandinitramfs-toolsto be…
AgentBaker Linux gate detectiveRun: 183969331 Detective summary: E2E completed 192 scenarios with 116 passed, 0 flaky, 74 skipped, and 2 failed; AzureLinuxV3Gen2Kata failed on the recurring containerd migration-warning validator, and LocalDNSHostsPlugin/Ubuntu2204 also failed restart-budget validation. Likely cause / signature: Confidence: High for recurring known flakes over a new PR-specific break. Recommended owner/action: Node/AgentBaker owners should continue repairs #39969892 and #39909321; PR owner only needs to act if touching Kata/containerd or LocalDNS restart-budget behavior. Strongest alternative: PR 9693 changes GPU/VHD content-test areas, but the fatal signatures are recurring Kata/containerd and LocalDNS lifecycle validations across unrelated PRs. Evidence: build run, failed timeline records, focused Run AgentBaker E2E log, build metadata, PR changed files, and wiki tracker. Wiki: |
What this PR does / why we need it:
Follow up on merged #9694, which already stopped CUDA host prebaking in non-FIPS
Ubuntu 22.04/24.04 amd64 Gen2 shared VHDs. This PR adds explicit installer/boot
prerequisites and final-image cleanliness checks. It retains #9694's pipeline
defaults and CIS-report condition without additional pipeline changes.
This removes the need for CPU/opt-out CSE cleanup to prevent NVIDIA DKMS work
during kernel updates. It deliberately defers precompiled-module reuse rather
than adding a parking/restoration lifecycle or a coordinated aks-gpu producer
format change.
FEATURE_FLAGS=None.independently of host prebaking.
host residue, installed modules and initramfs copies.
ARM64/Grace-Blackwell, FIPS, CVM and other OS image policies are unchanged.
Existing cleanup, container caching, PIS and nodePrep activation remain in place.
No aks-gpu image update is required.
Validation:
Coverage includes all four pipeline defaults, image-family exclusions,
dangling links, missing markers, compressed modules, alternate initramfs,
prerequisite/cache failures and failed inspection.
built both target images but failed the new content check in both attempts
per image. The matcher incorrectly classified Ubuntu's stock
kernel/drivers/platform/x86/nvidia-wmi-ec-backlight.ko(compressed on 24.04)as CUDA/GRID residue. The corrected disk/initramfs checks exclude only that
stock-kernel path. DKMS-path copies and GPU modules remain failures.
Four new regression cases first reproduced the failure, then passed;
negative cases include compressed driver residue and a large initramfs listing
with pipefail enabled.
both Ubuntu target jobs passed, including content validation, on merge
a993a34. The four target test/scan resource groups are independently
confirmed deleted; the produced images are retained for GPU verification.
assessment exit 122/tool exit 1 despite the successful wrapper jobs. This is
not evidence of completed CIS compliance scans.
finished 2 passed / 187 skipped / 5 failed against these images and
fix: install absent or incomplete managed Ubuntu GPU drivers #9692's consumer source. Both westus3 GRID cases passed first attempt with
driver 570.237. Five T4 cases failed before VM creation because Shallow
images cannot replicate to westus2; the helper retried the permanent Azure
error for 3000 seconds. Both created VMSS identities are confirmed absent.
This is not CUDA, opt-out or validation-only coverage.
roots, although local ShellCheck did not reproduce those warnings. Commit
387b780 passes
""at both production call sites instead of relying onthe identical empty default. All 75 examples still pass; no root-path or
image-policy behavior changes. The images above came from preceding
5c77850. Current-head VHD gate 183969331,
merge 5f9c38e, has passed all 12 image build/content jobs, including both
intended Ubuntu images. Downstream general E2E finished 116 passed /
74 skipped / 2 failed: the known AzureLinuxV3Gen2Kata migration warning,
plus LocalDNSHostsPlugin/Ubuntu2204's fault-injection check missing the
expected start-limit journal entry (
Result=exit-code,NRestarts=5).The latter's cause is not established; the PR does not change LocalDNS
source, but that alone does not prove it unrelated. The overall gate is red.
All 121 logged downstream VMSS identities and the four target test/scan
groups are independently absent. Both current target CIS
assessments remain license-limited (122/tool exit 1), not completed scans.
Replacement CI ShellCheck 37080778665 and ShellSpec 37080778755 both passed.
succeeded: 6 passed, 1 flaky, 187 skipped, 0 failed on isolated test-only
source 8bc42be, pairing fix: install absent or incomplete managed Ubuntu GPU drivers #9692 with current 183969331 images
2204gen2containerd/1.1790986591.27907and2404gen2containerd/1.1790986591.7556. All seven required cases eventuallypassed with unchanged GPU SKUs and validators. Only test-branch T4 placement
changes to westus3; current CSE, no capacity skips and one retry remain pinned.
Neither product PR changes placement; no image copies or shared-image
updates were needed.
proving the missing
nvidia-modprobedispatch into normal installation andsubsequent healthy SMI 580.159.04/CUDA 13.0. Both opt-out cases, 22.04 GRID,
and 24.04 GRID/device-plugin also passed first attempt.
(17m38s), then passed unchanged on attempt 2 (9m22s). Serial-only evidence
shows NVIDIA module loading, but the precise timeout cause is unresolved;
it is not dismissed as platform-only. All eight explicitly created
GPU-attempt VMSS are independently absent. This targeted success does not
make broad CI green or establish distinct-VM PIS/kernel-update acceptance.
warnings across the changed builder/content scripts; base comparison confirms
no introduced diagnostics.
results do not establish real-image boot, PIS or latency behavior.
Rollout and remaining operational work:
Release the companion missing-driver consumer change #9692 before broad rollout where
validation-only callers otherwise spend 115 seconds retrying an absent binary.
Verify changed Ubuntu images on CPU/opt-out, CUDA and GRID nodes, distinct-VM PIS,
and kernel-update paths, using explicit source/VHD identities.
Changing future image production does not repair existing security-patched
derivatives, PIS caches or deployed nodes. Refresh affected image lineages and
nodes through the supported lifecycle; retain legacy cleanup. Re-enabling active
shared-image prebaking is not a safety-preserving rollback.
Which issue(s) this PR fixes:
Follow-up to merged #9694. This PR does not close or modify the host-prebake
parking/no-DKMS drafts #9545, #9551 or #9568.