Skip to content

[ROCm][Kimi-K3] add VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4 for the a8w4 … #738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fanxingran wants to merge 1 commit into
base: main
Choose a base branch
from
+7 −2
Open

[ROCm][Kimi-K3] add VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4 for the a8w4 … #738

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions models/moonshotai/Kimi-K3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ meta:
provider: "Moonshot AI"
description: "Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window"
date_added: 2026-07-27
date_updated: 2026-08-03
date_updated: 2026-08-04
difficulty: hard
tasks:
- multimodal
Expand Down Expand Up @@ -193,6 +193,11 @@ hardware_overrides:
extra_env:
VLLM_ROCM_USE_AITER: "1"
SAFETENSORS_FAST_GPU: "1"
# a8w4 (fp8 activation) SiTU MoE. vllm-project/vllm#50582 renamed the
# switch; the pinned kimi-k3 image still reads the AITER_* pair, so set
# all three — and flip them together, a mixed state silently corrupts
# MoE output.
VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4: "1"
AITER_SITUV2_A8W4: "1"
AITER_BF16_FP8_MOE_BOUND: "0"
VLLM_USE_BREAKABLE_CUDAGRAPH: "0" # REQUIRED on ROCm: the build auto-enables =1
Expand Down Expand Up @@ -355,7 +360,7 @@ guide: |
- **MoE backend**: Recommend to use deep_gemm_mega_moe for any DEP environment.
- **Model Runner v2 and Rust Frontend**: `VLLM_USE_V2_MODEL_RUNNER=1` and `VLLM_USE_RUST_FRONTEND=1`: Model Runner v2 and Rust Frontend fully supports this model and can be enabled if needed.
- **Tool calling**: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.
- **AMD (MI355X / MI350X, CDNA4 gfx950)**: set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path.
- **AMD (MI355X / MI350X, CDNA4 gfx950)**: the AITER a8w4 MoE path is selected by `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1` since vllm-project/vllm#50582, and by `AITER_SITUV2_A8W4=1` + `AITER_BF16_FP8_MOE_BOUND=0` on older builds such as the pinned `kimi-k3` image; the recipe sets all three. Clear all of them for the a16w4 path — a mixed state leaves vLLM and AITER disagreeing on the weight layout and silently returns wrong output.
- **max-model-len**: Adjust max-model-len for different benchmark scenarios for best performance.
- **RDMA**: If RDMA is enabled, set `UCX_TLS="rc,cuda_copy"` to make sure KV Cache transfer goes through RDMA.
- **MNNVL environments** (GB200/GB300 NVL): recommend adding `NCCL_MNNVL_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, and `NCCL_NVLS_ENABLE=1`.
Expand Down