Skip to content

Latest commit

 

History

History
122 lines (99 loc) · 6.09 KB

testing.md

File metadata and controls

122 lines (99 loc) · 6.09 KB

Testing and interpreting results

Validation has three separate levels. A passing host test does not establish that a firmware transaction ran, and a successful transaction does not predict the speedup of an application.

Level What it checks Current evidence
Host regression tests Asset pins, rejection/unwind paths, mocked transactions, benchmark helpers 29 tests passed with pinned inputs; eight C components and full GA102/AD102 cases passed ASan/UBSan
Actual boot and driver handoff Selected-card transaction completed; NVIDIA initialized; rate getter changed Independent RTX 4090 run of 8855e7b on Linux 7.2.11-arch1-1, NVIDIA 615.78.08
Workload checks Exercised outputs and measured throughput in both states Six seeded FP16/BF16 numerical cases, six large cuBLAS GEMMs, instruction/control probes; one matched pair

The matched RTX 4090 report contains the numeric evidence and limitations. Longer stability runs, additional boot reproductions and representative inference/training workloads remain useful follow-up tests.

Host tests without touching a GPU

make test

Missing generated assets/source are explicit skips of the full transaction tests. For strict validation, generate the pinned inputs and require them:

make assets CACHE=build/test-inputs
make test-full CACHE=build/test-inputs

The first command downloads the pinned NVIDIA inputs and generates assets locally. The second must fail if required inputs are absent or fail their hash checks. Use ASSETS= and NVIDIA_SOURCE= to point at existing inputs. The full tests still model hardware and firmware execution. The C harness models the SHA-256 primitive; Python independently hashes the real assets.

The recorded sanitizer run disabled LeakSanitizer because process tracing was restricted in the test environment. ASan/UBSan covered the C harness, not the live kernel module or GPU firmware. CUDA numerical tests require actual GPU access and are separate from this suite.

Build a matched set of workload probes

Use a persistent build directory so the same binaries survive the restart:

mkdir -p build/matched-bench
nvcc -O3 -arch=sm_89 -std=c++17 -o build/matched-bench/mma_bench \
  diagnostics/bench/mma_bench.cu
nvcc -O3 -arch=sm_89 -std=c++17 -o build/matched-bench/validate_gemm \
  diagnostics/bench/validate_gemm.cu -lcublas
nvcc -O3 -arch=sm_89 -std=c++17 -o build/matched-bench/gemm_bench \
  docs/evidence/rtx4090-pr1-stock/gemm_bench.cu -lcublas
sha256sum build/matched-bench/mma_bench build/matched-bench/validate_gemm \
  build/matched-bench/gemm_bench > build/matched-bench/SHA256SUMS

The standalone large GEMM source is exactly the one used in the matched report. Its simple inputs allow every output to be checked cheaply; use the separate seeded validator for varied numerical inputs. Neither establishes general application correctness. The tensor-mode probe adds FP16/TF32 and unsigned-INT4 controls with exact device/cubin/power checks.

Collect stock first, then unlocked

Start in a normal stock boot with the NVIDIA driver loaded. Record uname -r, /proc/cmdline, GPU UUID, driver, existing power cap, and source/binary hashes. Use the same exact GPU, kernel, driver, binaries, launch settings, clock policy and cap for the unlocked run. Keep comparable background activity; record whether a desktop is active. Do not change clock or power policies mid-pair.

For a single-GPU host, set CUDA_VISIBLE_DEVICES to that GPU's exact UUID before the CUDA probes. The RM getter separately requires --expect-uuid. Example collection after filling in your own UUID:

export CUDA_VISIBLE_DEVICES=GPU-PASTE-YOUR-EXACT-UUID
mkdir -p build/matched-bench/stock
uname -r > build/matched-bench/stock/kernel.txt
cat /proc/cmdline > build/matched-bench/stock/cmdline.txt
nvidia-smi --query-gpu=uuid,name,pci.bus_id,driver_version,power.limit \
  --format=csv > build/matched-bench/stock/gpu.csv
python3 tools/read_sm_rates.py --expect-uuid "$CUDA_VISIBLE_DEVICES" \
  > build/matched-bench/stock/sm-rates.json
sha256sum -c build/matched-bench/SHA256SUMS
build/matched-bench/validate_gemm > build/matched-bench/stock/numerical.txt
build/matched-bench/mma_bench 20000 5 > build/matched-bench/stock/mma.txt
build/matched-bench/gemm_bench > build/matched-bench/stock/gemm.jsonl

Stop if a command fails or any output reports mismatches. Capture clocks, temperature and power while benchmarking, for example with nvidia-smi sampling at 100 ms. Samples describe this short run; they do not establish maximum power, long-term thermals or electrical safety.

Follow the Quick Start for the firmware test, or use a separate test image/entry containing the module and pinned assets. Keep the normal entry available. With GRUB, inspect the actual entry's linux and initrd paths; selecting ordinary Arch instead of a separate test entry starts the ordinary boot. Our local test entry had a text-console COLD confirmation; that is not a prompt installed by the standard Quick Start.

After the test boot, first confirm all of these:

  • The running kernel and driver match the stock run.
  • /proc/cmdline names the exact intended target.
  • The current kernel journal contains that card's gpu_unlock: unlocked line.
  • The RM getter reports the expected full rate (fmla32=0 in this 4090 run; stock was 1).
  • NVIDIA initializes successfully and no NVRM: Xid errors are logged.

Repeat the same commands into an unlocked output directory, resetting the GPU UUID environment variable after login. Check the binary hashes again. Check the current kernel journal for Xid after the workloads as well. Do not rerun the firmware transaction manually against a driver-owned card.

Report each shape/type and all timing repetitions. Distinguish instruction throughput from GEMM medians and from application throughput. Include unchanged or slower cases, output mismatch counts, telemetry and repeat counts. Before publishing logs, remove UUIDs, PCI locations, hostnames, boot/root identifiers and private paths while preserving numerical results.