Repository navigation
fix: install absent or incomplete managed Ubuntu GPU drivers - #9692
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 4 commits into
Conversation
Preserve the CSE budget on cache-only VHDs while retaining readiness retries for existing drivers and the normal installation fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
Reuse the existing T4 scenario with the distinct ConfigGPUDriverIfNeeded=false input on both supported shared images, preserving existing unconditional-install coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Reduces GPU validation latency on managed Ubuntu amd64 VHD nodes by skipping futile retries when nvidia-modprobe is not installed and immediately running the normal driver installer, plus adds tests and e2e coverage for the validation-only path.
Changes:
- Update
validateGPUDriversto detect missingnvidia-modprobeon Ubuntu amd64 VHDs and install drivers before validation retries. - Add shell spec coverage for cache-only Ubuntu VHD validation behavior and legacy behavior preservation.
- Add e2e “validation-only” GPU scenarios for Ubuntu 22.04 and 24.04 VHDs.
| File | Description |
|---|---|
| vhdbuilder/packer/README.md | Documents the new Ubuntu amd64 VHD GPU validation behavior and rationale. |
| spec/parts/linux/cloud-init/artifacts/cse_config_gpu_spec.sh | Adds spec cases covering cache-only Ubuntu VHD behavior, failure propagation, and legacy paths. |
| parts/linux/cloud-init/artifacts/cse_config_gpu.sh | Implements early installer selection when nvidia-modprobe is absent on Ubuntu amd64 VHDs. |
| e2e/scenario/scenario.go | Adds validation-only GPU scenarios to exercise the new behavior in e2e. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
The reused constructor currently has two 22.04-specific values: Config.VHD and Description; this wrapper replaces both. The T4 hardware, node settings and validators are otherwise shared, and the existing unconditional-install cases remain unchanged. This intentionally reuses the established scenario rather than copying its bootstrap/validator implementation. There is no current 24.04 behavior depending on 22.04-specific logic; keeping this limited reuse for the focused fallback change.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 3
Handle NVIDIA status 12 without changing other readiness retries. Deliver current CSE in validation-only E2E scenarios instead of baked-in scripts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
changed the title
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 6
Open (6)
errors.Joinwill runValidateNvidiaSMIInstalledeven whenvalidateGPUfails. This can add… · NewIS_VHDis lowercased via${IS_VHD,,}, which is Bash-specific. If this script can ever be… · Newnvidia-smiis onPATH, but later validation can use an… · Newretrycmd_if_failurereturns a non-zero status for any command other than…retrycmd_if_failurereturns the exit status of the last[test when$4is not…ubuntuGPUValidationScenariois built onubuntu2204GPUScenario, but it is also used for Ubuntu…
| } | ||
| validateGPU := s.Config.Validator | ||
| s.Config.Validator = func(ctx context.Context, s *Scenario) error { | ||
| return errors.Join(validateGPU(ctx, s), ValidateNvidiaSMIInstalled(ctx, s)) |
| validateGPUDrivers() { | ||
| retrycmd_if_failure 24 5 25 nvidia-modprobe -u -c0 && echo "gpu driver loaded" || configGPUDrivers || exit $ERR_GPU_DRIVERS_START_FAIL | ||
| local driver_payload_missing=false | ||
| if [ "$OS" = "$UBUNTU_OS_NAME" ] && [ "${IS_VHD,,}" = "true" ] && |
| if ! command -v nvidia-modprobe >/dev/null 2>&1; then | ||
| driver_payload_missing=true | ||
| elif command -v nvidia-smi >/dev/null 2>&1; then | ||
| local smi_status=0 | ||
| timeout 30 nvidia-smi >/dev/null 2>&1 || smi_status=$? | ||
| # NVIDIA exit 12 means NVML cannot be loaded, even if a prebaked kernel module loads. | ||
| if [ "$smi_status" -eq 12 ]; then | ||
| driver_payload_missing=true | ||
| fi | ||
| fi |
Windows Unit Test Results 3 files 18 suites 1m 7s ⏱️ Results for commit edc4339. ♻️ This comment has been updated with latest results. |
Honor the final NBC scriptless flag after scenario overrides. Exercise real VMSS artifact generation across eight Ubuntu legacy/scriptless cases without cloud resource creation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61d3dd43-5cc2-49b3-9ce5-fbe943764363
|
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
An incomplete payload with nvidia-modprobe present but no usable nvidia-smi still exits without invoking the installer.
Review effort: Balanced
Findings: 4
Open (4)
nvidia-smiis onPATH, but later validation can use an…IS_VHDis lowercased via${IS_VHD,,}, which is Bash-specific. If this script can ever be…errors.Joinwill runValidateNvidiaSMIInstalledeven whenvalidateGPUfails. This can add…ubuntuGPUValidationScenariois built onubuntu2204GPUScenario, but it is also used for Ubuntu…
|
|
AgentBaker Linux gate detectiveRun: 183962012 Detective summary: E2E completed 192 scenarios with 117 passed, 0 flaky, 74 skipped, and 1 failed; AzureLinuxV3Gen2Kata failed because the validator rejected containerd output containing the v2-to-v4 migration warning. Likely cause / signature: Confidence: High for recurring E2E validator/image dependency drift rather than a new build break. Recommended owner/action: Node/AgentBaker owners should continue repair #39969892; PR owner only needs to act if this PR intentionally changes Kata/containerd config generation or validation inputs. Strongest alternative: PR 9692 changes GPU/CSE config and VMSS test code, but the fatal scenario is the same Kata/containerd validation signature recurring across unrelated PRs. Evidence: build run, failed timeline records, focused Run AgentBaker E2E log, build metadata, PR changed files, and wiki tracker. Wiki: |
What this PR does / why we need it:
On managed Ubuntu amd64 VHD nodes,
validateGPUDriverscurrently retriesnvidia-modprobe24 times before falling back to the normal installer. When ashared VHD only caches the installer and has no installed host driver, retrying
the missing executable adds 23 x 5 = 115 seconds of sleeps before installation.
Select the existing normal installer immediately when
nvidia-modprobeisabsent on an Ubuntu amd64 VHD, or a bounded
nvidia-smiprobe returns NVIDIAstatus 12 (NVML cannot be found or loaded). The latter covers old partial
prebakes: a loadable kernel module is not sufficient when host userspace is
missing. Other probe statuses retain readiness retries; normal-install failures
and final validation failures still propagate. ARM64/non-Ubuntu/non-VHD behavior,
alternate SMI paths, and infoROM corruption handling are preserved. GPU
eligibility and opt-out policy remain in nodePrep.
Add
Ubuntu2204_GPUNC_ValidateDriverandUbuntu2404_GPUNC_ValidateDriverusing the existing T4 GPU scenario and validators. These need separate nodes
because
ConfigGPUDriverIfNeeded=falseis a different bootstrap input from theexisting unconditional-install scenarios, which remain unchanged. Both verify
module loading and successful nvidia-smi after validation/fallback. Both
explicitly disable the two scriptless CSE delivery flags so automatic CI tests
this checkout's changed scripts, not the default VHD's baked-in CSE. The VMSS
artifact guard now follows the final NBC's
EnableScriptlessCSECmd, matchingthe existing on-node validator, rather than treating the run-wide default as
the effective mode after a scenario override.
This is the consumer-first companion to #9693, the cache-only shared-VHD producer
change. It introduces no image layout, parking/restoration protocol, driver
container version change or skip-build path.
Validation:
cse_config_gpu_spec.sh: 80 examples passed, including 19 new cases.go test ./scenario ./runnerine2e: passed with both new scenarios compiled.scriptless CSE and scriptless NBC across both Ubuntu images. The two legacy
overrides reproduced the cloud harness error before the one-line guard fix;
all cases pass afterward. Image resolution is stubbed, without cloud creation.
warning; comparison with the base commit confirms no introduced diagnostics.
VHDs; its results will be recorded separately. Unit results are not presented
as changed-image or performance evidence.
Findings from the first automatic cloud run:
16 passed, 175 skipped, 3 failed. The two new validation-only cases each failed
all three attempts on main-tag prebaked images: modprobe succeeded, then SMI
returned 12 for missing NVML. Actual traces used the old baked-in validation
function, not this PR's code. The changes above address both discovered gaps.
The third failure was an RTX Pro6000 Bastion SSH timeout; an A100 quota skip
is not counted as a pass.
133 passed, 60 skipped, 1 failed (
AzureLinuxV3Gen2Kata, containerd's version-2migration warning). That non-GPU path is outside this change; fix fixAzureLinuxV3Gen2Kata #9690 addresses
the Kata/containerd area separately.
will be recorded separately, not inferred from local coverage.
15 passed, 176 skipped, 3 failed. Both new cases were rejected before VM
creation by the run-wide scriptless artifact guard; they did not exercise
driver recovery. The effective-mode correction above fixes that harness
mismatch. The third failure remained the RTX Bastion timeout; A100 and H100
quota skips are not passes.
failure, with 133 passed and 60 skipped. No unrelated product fix is included.
Latest cloud verification:
19 passed, 0 flaky, 174 skipped, 1 failed. Both new validation-only scenarios
passed on their first attempt against the same main-tag prebakes that failed
initially. Their VM traces execute the changed function: SMI status 12,
normal installation, then working
nvidia-smiwith driver 580.159.04.The remaining RTX failure starts with Azure
AllocationFailedin southeastasia,before the secondary Bastion/SSH failures; it is not a driver-validation result.
non-GPU Kata/containerd migration warning.
finished 2 passed / 187 skipped / 5 failed against producer 183958924.
Both westus3 GRID cases passed first attempt with driver 570.237. Five T4
cases failed before VM creation: the PR images use Shallow replication,
which cannot be extended to westus2. The image helper retried this permanent
Azure error for 3000 seconds; these failures provide no driver result.
succeeded: 6 passed, 1 flaky, 187 skipped, 0 failed. All seven required
scenarios eventually passed on current producer 183969331 images
2204gen2containerd/1.1790986591.27907and2404gen2containerd/1.1790986591.7556. Isolated test-only 8bc42be retainsthis consumer source, scenario definitions/SKUs/validators, current CSE,
no capacity skips and one retry; only T4 placement moves to the images'
westus3 region. Neither product PR changes regional placement.
show
command -v nvidia-modprobefailing,driver_payload_missing=true,immediate
configGPUDrivers, then healthy SMI 580.159.04/CUDA 13.0.Both opt-out cases, 22.04 GRID, and 24.04 GRID/device-plugin passed first
attempt. These are changed-image results, separate from old-prebake NVML
recovery above; whole-scenario times are not isolated installer benchmarks.
Ubuntu2204_GPUNCrequired its configured retry: first VMSS provisioningtimed out after 17m38s; the unchanged source/image passed attempt 2 in 9m22s.
Only serial diagnostics survived the first attempt, showing NVIDIA module
loading but not a complete CSE result. Its precise timeout cause remains
unresolved; this is not claimed as a clean first-attempt or platform-only
result. All eight explicitly created GPU-attempt VMSS are confirmed absent.
from consumer VHD gate 183962012 are independently confirmed absent. That
gate's downstream E2E finished 117 passed / 76 skipped / 1 failed, again on
the same non-GPU Kata migration warning; its image build/content jobs passed.
Rollout:
Release this consumer change before broad cache-only image rollout where
validation-only callers need the saved retry budget. Existing CSE still has its
normal-install fallback, but its latency must be accounted for.
Targeted GPU success is not full release approval. Broad CI still has the
documented Kata/RTX failures; the companion producer gate also has an unresolved
LocalDNS fault-injection assertion. Distinct-VM GPU PIS, kernel-update/reboot,
and existing-image/fleet remediation remain separate acceptance work.
Which issue(s) this PR fixes:
No automatic issue or alternative-PR closure.