Skip to content

feat(rocm): add a GPU vendor concept and AMD device reporting - #1883

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-1805-gpu-vendor
Sep 15, 2026
Merged

inureyes merged 1 commit into
mainfrom
feature/issue-1805-gpu-vendor

Conversation

@inureyes

@inureyes inureyes commented Sep 14, 2026

Copy link
Copy Markdown
Member

Closes #1805. Part of #1801, phase 1.

What was wrong

An AMD GPU resolved to silicon_gen: Unknown with every other field at its default, so the startup diagnostics, the device listing and the server metadata had nothing to report. Worse, cuda_arch_startup_summary printed CUDA compute capability 11.5 (sm_115) for a gfx1151 card, because MLX's ROCm backend fills compute_capability_major/minor from the HIP device properties and cuda_compute_capability read them as if they were CUDA's.

Decisions worth reviewing

Vendor is a new field, not an AppleSiliconGen variant. Twelve call sites across the VLM loaders, models/sanitize.rs and drafter/dflash/drafter.rs read silicon_gen != Unknown as "this is not Apple Silicon" to gate a bf16 to f16 weight conversion. An Amd variant would have flipped all twelve and enabled that conversion on a backend where bf16 is native. They now call is_apple_silicon(), whose body is the identical comparison, and gpu_vendor_does_not_imply_apple_silicon fails if a future vendor ever makes it true.

The vendor comes from the resolved backend, not from which device_info() keys exist. Every backend publishes architecture, each in its own vocabulary: gfx1151 on ROCm, sm_89 on CUDA, an Apple GPU family string on Metal. Deciding from key presence would say "AMD is CUDA", which is the same shape of defect #1803 removed one layer down. gpu_backend_kind() exposes mlxcel::GpuKernelBackend to Rust and everything reads that.

rocm_arch is a new module, not a reuse of cuda_arch. CUDA coverage is an ordering: a cubin runs on a higher minor revision and PTX JITs forward across majors, so a list can cover a device it does not name. HIP has neither property, so coverage is set membership over normalized gfx targets. Modelling it with cuda_arch's ArchVariant ladder would invent a compatibility rule the runtime does not have.

A generic target makes the check decline rather than refuse. ROCm 6.3 added gfx11-generic-style targets whose code objects do load on several specific targets, so equality would report "not covered" for a build that runs. Any list this module cannot judge reports nothing, the same direction an unrecorded list takes: a wrong refusal is a binary that will not run at all, a missed one is the opaque HIP error this check was going to improve on.

resolve_available_memory is untouched in behavior. The order is now a pure function with tests, including one asserting the allocator cap is not read behind an env limit, because reading it forces the MLX allocator singleton and inspect and serve --estimate-memory estimate before runtime bring-up on purpose. unified_memory_gb stays Apple-only so the CUDA and ROCm estimate cannot move; device memory is a separate field.

hardware_label() is deliberately left alone. It collapses to Unknown-0c on both CUDA and ROCm, so those two share an MTP policy cache key and a profile measured on one is reused on the other. That is a real defect, but the label is persisted in the hint files and any new spelling discards every profile recorded under the old one, which the "CUDA unchanged" constraint forbids. The vendor, device name and architecture are reported beside it in the /v1/internal/mtp-policy body instead (new fields, no schema bump). Worth its own issue.

Validation

Run on the gfx1151 spike host (Radeon 8060S, ROCm 10.0.0). Both tests #1805 names pass, run directly because the full gate run still aborts on the nvfp4 case in #1806 before printing its failure list:

cuda_arch_tests::the_capability_probe_agrees_with_the_backend ... ok
streams::tests::gpu_backend_available_does_not_track_the_default_device ... ok

cargo run -p mlxcel-core --features rocm --example device_memory_probe reproduces the issue's measurement table from source, with the misreport gone:

vendor:                      Amd
device_name:                 "AMD Radeon 8060S Graphics"
device_architecture:         "gfx1151"
device_memory_bytes:         103079215104 (96.00 GiB)
memory_limit():              82463372083 (76.80 GiB)
CUDA compute capability:     None
HIP gfx target:              Some("gfx1151")

rocm_arch (9), hardware (33), memory_estimate (49), streams, cuda_arch (26), the two runtime agreement tests and the mtp-policy routes (65) all pass. fmt and clippy are clean.

Not verified

The cross-architecture refusal is covered by unit tests and by a test asserting the refusal tracks rocm_arch_mismatch(), but not by a binary actually built for another gfx target: that needs a full MLX rebuild in each direction. Say the word and I will run it.

The full make verify-test-rocm run was killed twice by the host's memory watchdog while executing the 8499-test mlxcel lib binary, so the targets were run individually. That is a host limit, not a regression from this branch.

Gate state on this branch

Four targets still fail, all pre-existing and none introduced here. Each is now traced to a cause and filed:

  • mlxcel-core --lib: the nvfp4 abort of feat(quant): backend quantization capability table and load-time convert-or-reject policy #1806, Unsupported group_size for QuantizedMatmul: 16, in compiled_qgelu_mlp_global_scale_native_nvfp4_prefill_matches_reference. This is the abort that hides feat(rocm): memory estimation, device info and hardware detection on AMD UMA hosts #1805's own two failures from the gate output, which is why they are run directly above.
  • mlxcel --test sampling_gumbel_kill_switch and --test sampling_rejection_kill_switch: [metal_kernel] No Metal back-end. The dispatch at src/lib/mlx-cpp/turbo/sampling.cpp:413 and sampling_rejection.cpp:780 branches on gpu_kernel_backend() == Cuda, so the false arm still means Metal rather than "no port for this backend". The routing gate is correct (custom_kernels_available() is false on ROCm, so production sampling takes the categorical fallback); it is the direct entry point the tests call that is unguarded.
  • mlxcel --lib: 5 WebUI catalog tests. runnable_on_backend at src/server/webui/catalog_metadata.rs:309 resolves to BackendStatus::Unsupported whenever the build is neither macOS+Metal nor CUDA, so the catalog reports that no model on this host can run. That is the same two-backend assumption this PR removes in hardware detection, surfacing one layer higher.

Apart from those, the mlxcel lib target passes 8350 of 8355.

An AMD GPU resolved to `silicon_gen: Unknown` with every other field at its default, so nothing could report the device, and `cuda_arch_startup_summary` printed `sm_115` for it because MLX's ROCm backend fills `compute_capability_major`/`minor` from the HIP device properties.

Vendor is a new `GpuVendor` field rather than an `AppleSiliconGen` variant. Twelve call sites read `silicon_gen != Unknown` as "not Apple Silicon" to gate a bf16 to f16 weight conversion, so a new variant there would have enabled that conversion on a backend where bf16 is native. They now call `is_apple_silicon()`, whose body is the same comparison, and a test pins that no non-Apple vendor makes it true.

The vendor comes from the resolved MLX backend, exposed as `gpu_backend_kind()`, not from which `device_info()` keys are present: every backend publishes `architecture` in its own vocabulary (`gfx1151`, `sm_89`, an Apple GPU family string), so key presence alone says "AMD is CUDA". `cuda_compute_capability` is gated on that backend, and the two tests whose premise was "Metal, CUDA, or CPU-only" now ask the backend instead.

New `rocm_arch` module, deliberately not a reuse of `cuda_arch`: CUDA coverage is an ordering with a PTX JIT fallback, HIP coverage is set membership with none. A list containing a generic target (`gfx11-generic`) makes the check report nothing rather than refuse, since those do load on several specific targets.

`resolve_available_memory` keeps its order and its laziness, both now pinned by tests. `unified_memory_gb` stays Apple-only so the CUDA and ROCm estimate cannot move; device memory is a separate field. `hardware_label()` is unchanged: it collapses to `Unknown-0c` on both CUDA and ROCm, but it is persisted in the MTP hint files and any new spelling discards every profile recorded under the old one, so the vendor is reported alongside it in the `/v1/internal/mtp-policy` body instead.

Validated on the gfx1151 spike host; the two tests this issue names are run directly because the gate run still aborts on the nvfp4 case in #1806. The cross-architecture refusal is covered by unit tests, not by a binary built for another `gfx` target.

Closes #1805. Refs #1801, #1803, #1806.
@inureyes
inureyes force-pushed the feature/issue-1805-gpu-vendor branch from 6c66508 to c683b91 Compare September 15, 2026 00:40
@inureyes
inureyes merged commit 9457ec5 into main Sep 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:review Under review type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(rocm): memory estimation, device info and hardware detection on AMD UMA hosts

1 participant