feat: add MiniMax-M3 RTX 6000 Pro vLLM recipe / 新增 MiniMax-M3 RTX 6000 Pro vLLM 基准配置 #2306
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- a/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py | ||
| +++ b/vllm/model_executor/layers/fused_moe/oracle/nvfp4.py | ||
| @@ -176,2 +176,3 @@ | ||
| NvFp4MoeBackend.FLASHINFER_TRTLLM, | ||
| + NvFp4MoeBackend.MARLIN, | ||
| } | ||
| --- a/vllm/model_executor/layers/fused_moe/experts/marlin_moe.py | ||
| +++ b/vllm/model_executor/layers/fused_moe/experts/marlin_moe.py | ||
| @@ -583,9 +583,12 @@ | ||
| - # Gated-activation params (used by SWIGLUOAI_UNINTERLEAVE on packed w13). | ||
| - # silu == swigluoai with alpha=1, beta=0; configs that don't set these | ||
| - # (plain silu) fall back to the silu identity. | ||
| - self.gemm1_alpha = ( | ||
| - quant_config.gemm1_alpha if quant_config.gemm1_alpha is not None else 1.0 | ||
| - ) | ||
| - self.gemm1_beta = ( | ||
| - quant_config.gemm1_beta if quant_config.gemm1_beta is not None else 0.0 | ||
| - ) | ||
| + if self.gemm1_clamp_limit is None: | ||
| + self.gemm1_clamp_limit = moe_config.swiglu_limit | ||
| + | ||
| + gemm1_alpha = quant_config.gemm1_alpha | ||
| + if gemm1_alpha is None: | ||
| + gemm1_alpha = moe_config.swiglu_alpha | ||
| + self.gemm1_alpha = 1.0 if gemm1_alpha is None else gemm1_alpha | ||
| + | ||
| + gemm1_beta = quant_config.gemm1_beta | ||
| + if gemm1_beta is None: | ||
| + gemm1_beta = moe_config.swiglu_beta | ||
| + self.gemm1_beta = 0.0 if gemm1_beta is None else gemm1_beta | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # MiniMax-M3 NVFP4 RTX PRO 6000 Blackwell single-node vLLM recipe. | ||
| # This is the PCIe/SM120 counterpart to minimaxm3_fp4_b200.sh. It keeps | ||
| # the ModelOpt NVFP4, FP8 KV-cache, and MSA block-size settings while using | ||
| # NCCL collectives instead of the B200-tuned FlashInfer/TRT-LLM all-reduce. | ||
| # | ||
| # The pinned vLLM image predates the MiniMax-M3 Marlin fixes tracked by | ||
| # vLLM PRs #45836 and #48929. Apply the narrow compatibility patch covered | ||
| # by docs/waiver/2306.md before importing vLLM. | ||
|
|
||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars \ | ||
| MODEL \ | ||
| TP \ | ||
| EP_SIZE \ | ||
| DP_ATTENTION \ | ||
| CONC \ | ||
| ISL \ | ||
| OSL \ | ||
| MAX_MODEL_LEN \ | ||
| RANDOM_RANGE_RATIO \ | ||
| RESULT_FILENAME | ||
|
|
||
| if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi | ||
|
|
||
| if [[ -n "$SLURM_JOB_ID" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" | ||
| fi | ||
|
|
||
| nvidia-smi | ||
|
|
||
| SERVER_LOG=/workspace/server.log | ||
| GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}" | ||
|
|
||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
| export VLLM_FLOAT32_MATMUL_PRECISION=high | ||
|
|
||
| if [ "${DP_ATTENTION}" = "true" ]; then | ||
| PARALLEL_ARGS=( | ||
| --tensor-parallel-size 1 | ||
| --data-parallel-size "$TP" | ||
| --enable-expert-parallel | ||
| ) | ||
| elif [ "$EP_SIZE" -gt 1 ]; then | ||
| PARALLEL_ARGS=( | ||
| --tensor-parallel-size "$TP" | ||
| --enable-expert-parallel | ||
| ) | ||
| else | ||
| PARALLEL_ARGS=(--tensor-parallel-size "$TP") | ||
| fi | ||
|
|
||
| if [ "${EVAL_ONLY}" = "true" ]; then | ||
| setup_eval_context | ||
| MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN" | ||
| fi | ||
| start_gpu_monitor | ||
|
|
||
| VLLM_SITE_PACKAGES="$( | ||
| python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])' | ||
| )" | ||
| VLLM_MARLIN_PATCH=/workspace/benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch | ||
| if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" \ | ||
| < "$VLLM_MARLIN_PATCH"; then | ||
| echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2 | ||
| exit 1 | ||
| fi | ||
|
Comment on lines
+61
to
+69
Contributor
There was a problem hiding this comment. 🟡 The Marlin compatibility patch step (lines 61-69) hard-aborts if Extended reasoning...The bug: if ! patch --batch --forward --fuzz=0 -p1 -d "$VLLM_SITE_PACKAGES" < "$VLLM_MARLIN_PATCH"; then
echo "Failed to apply the pinned MiniMax-M3 Marlin compatibility patch" >&2
exit 1
fiThe intent of Reachability: normal CI is unaffected, since Step-by-step proof:
Why nothing else prevents this: Suggested fix: mirror the b200 recipe's approach (an idempotent Python-based patch application, or a pre-check such as Severity: all three independent verifiers empirically confirmed the behavior with GNU patch and agreed on |
||
|
|
||
| set -x | ||
| vllm serve "$MODEL" --port "$PORT" \ | ||
| "${PARALLEL_ARGS[@]}" \ | ||
| --disable-custom-all-reduce \ | ||
| --gpu-memory-utilization "$GPU_MEM_UTIL" \ | ||
| --max-model-len "$MAX_MODEL_LEN" \ | ||
| --kv-cache-dtype fp8 \ | ||
| --block-size 128 \ | ||
| --language-model-only \ | ||
| --attention-backend TRITON_ATTN \ | ||
| --moe-backend marlin \ | ||
| --max-cudagraph-capture-size 2048 \ | ||
| --max-num-batched-tokens "$((ISL * 2))" \ | ||
| --stream-interval 20 \ | ||
| --no-enable-prefix-caching \ | ||
| --trust-remote-code > "$SERVER_LOG" 2>&1 & | ||
|
|
||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| run_benchmark_serving \ | ||
| --model "$MODEL" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts "$((CONC * 10))" \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir /workspace/ \ | ||
| --trust-remote-code | ||
|
|
||
| if [ "${RUN_EVAL}" = "true" ]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| append_lm_eval_summary | ||
| fi | ||
|
|
||
| stop_gpu_monitor | ||
| set +x | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Runtime patch waiver for PR 2306 | ||
|
|
||
| <div align="center"> | ||
|
|
||
| **English** | [中文](./2306_zh.md) | ||
|
|
||
| </div> | ||
|
|
||
| ## Scope | ||
|
|
||
| The MiniMax-M3 NVFP4 RTX PRO 6000 recipe applies | ||
| `benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch` to the vLLM Python | ||
| package in the pinned container before starting the server. The patch: | ||
|
|
||
| - admits Marlin to the clamped NVFP4 MoE backend allowlist; and | ||
| - makes Marlin fall back to the model's SwiGLU clamp, alpha, and beta when | ||
| those values are absent from the quantization config. | ||
|
|
||
| It does not modify a CUDA kernel, checkpoint weight, or benchmark result. | ||
|
|
||
| ## Why the unmodified image cannot run this benchmark | ||
|
|
||
| MiniMax-M3 sets a SwiGLU clamp. The pinned vLLM image consequently filters its | ||
| NVFP4 MoE candidates to FlashInfer TRT-LLM, whose device guard accepts SM100 | ||
| but rejects the RTX PRO 6000's SM120 compute capability. Explicitly selecting | ||
| Marlin in the unmodified image is also rejected by that allowlist. If only the | ||
| allowlist is widened, Marlin substitutes plain-SiLU defaults for the model's | ||
| alpha and beta and produces incorrect output. | ||
|
|
||
| The recipe therefore selects `--moe-backend marlin` and carries the minimum | ||
| Python adapter fix required to preserve the model's activation parameters. | ||
|
|
||
| ## Upstream tracking | ||
|
|
||
| The Marlin clamp allowlist is upstream in | ||
| https://github.com/vllm-project/vllm/pull/45836. The model-config parameter | ||
| fallback is the Marlin portion of | ||
| https://github.com/vllm-project/vllm/pull/48929. The latter pull request | ||
| includes focused tests and reports coherent deterministic MiniMax-M3 NVFP4 | ||
| output after the fix, where the unpatched Marlin path produced garbled output. | ||
|
|
||
| ## Removal plan | ||
|
|
||
| Replace the pinned image with the first suitable upstream vLLM image that | ||
| contains the Marlin fix, then remove the runtime patch application and patch | ||
| file. Revalidate TP8 8k/1k generation on SM120 before removing this waiver. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # PR 2306 运行时补丁豁免 | ||
|
|
||
| <div align="center"> | ||
|
|
||
| [English](./2306.md) | **中文** | ||
|
|
||
| </div> | ||
|
|
||
| ## 范围 | ||
|
|
||
| MiniMax-M3 NVFP4 RTX PRO 6000 配方会在启动服务前，将 | ||
| `benchmarks/patches/vllm/minimax_m3_nvfp4_marlin.patch` 应用到锁定镜像中的 | ||
| vLLM Python 包。该补丁： | ||
|
|
||
| - 将 Marlin 加入支持 clamp 的 NVFP4 混合专家(MoE)后端允许列表；以及 | ||
| - 当量化配置未提供 SwiGLU clamp、alpha 和 beta 时，让 Marlin 回退到模型配置中的对应值。 | ||
|
|
||
| 该补丁不修改 CUDA 内核、模型权重或基准测试结果。 | ||
|
|
||
| ## 为何无法直接使用未修改的镜像 | ||
|
|
||
| MiniMax-M3 设置了 SwiGLU clamp，因此锁定版本的 vLLM 会把 NVFP4 MoE 候选后端 | ||
| 限制为 FlashInfer TRT-LLM。该后端的设备检查只接受 SM100，不接受 RTX PRO 6000 | ||
| 的 SM120 计算能力。未修改的镜像还会通过允许列表拒绝显式选择 Marlin。如果仅 | ||
| 扩大允许列表，Marlin 会使用普通 SiLU 的 alpha、beta 默认值，并产生错误输出。 | ||
|
|
||
| 因此，该配方显式选择 `--moe-backend marlin`，并携带保留模型激活参数所需的最小 | ||
| Python 适配层修复。 | ||
|
|
||
| ## 上游跟踪 | ||
|
|
||
| Marlin 的 clamp 允许列表修复已由 | ||
| https://github.com/vllm-project/vllm/pull/45836 合入上游。模型配置参数回退逻辑来自 | ||
| https://github.com/vllm-project/vllm/pull/48929 中的 Marlin 部分。后者包含针对性 | ||
| 测试，并报告修复后 MiniMax-M3 NVFP4 的确定性输出从乱码恢复为连贯文本。 | ||
|
|
||
| ## 移除计划 | ||
|
|
||
| 当包含 Marlin 修复的合适 vLLM 上游镜像发布后，更新锁定镜像，并移除运行时补丁 | ||
| 应用逻辑及补丁文件。移除此豁免前，需要在 SM120 上重新验证 TP8 8k/1k 生成。 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe upstream if sm120 then marlin else flashinfer?