Repository navigation
Jetson AGX Xavier benchmark: APEX-MTP performance expectations? #14
Description
Device: NVIDIA Jetson AGX Xavier (Volta 512-core, ~135 GB/s unified memory, 32 GB RAM, 30W TDP)
Context: I'm running the Qwen3.6-35B-A3B-Native-MTP-Preserved model on llama.cpp (master) via the Jetson platform. Currently getting 23.14 t/s with MTP (spec-max=2, 97.7% acceptance) on the original Q3_K_M quant (17.3 GB).
Question: I just downloaded the APEX-MTP-I-Mini (13.3 GB) and APEX-MTP-Compact (16.1 GB) variants. On a Jetson Xavier bandwidth-limited setup (~135 GB/s), the I-Mini should theoretically do ~28 t/s (17.3/13.3 * 23.14). Has anyone tested APEX quantized models on similar embedded/edge hardware? Any known gotchas with llama.cpp + APEX on UMA/integrated GPU systems?
Plan: I'll benchmark both and report back here with numbers. Happy to provide any calibration/speed data that would help the project.
Activity
localai-org-maint-bot commentedon Jul 24, 2026
Thanks for following through with the measurements. PR #15 now contains the revised multi-run Jetson results, has addressed the review concerns, and is approved with @mudler tagged. Further discussion of the accepted benchmark interpretation should continue on that PR.