Add NVIDIA Nemotron 3.5 Lightning - #736
Conversation
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new configuration file for the NVIDIA Nemotron 3.5 Lightning model, defining its metadata, hardware requirements, features, and deployment guides. The review feedback highlights several critical configuration errors that would cause vLLM to fail at startup. Specifically, the tool_calling configuration references a non-existent qwen3_coder parser and should be removed. Additionally, the Mamba cache argument --mamba-ssm-cache-dtype needs to be corrected to --mamba-cache-dtype across the features and guide sections, and the speculative decoding configuration for the external draft model must use the correct vLLM parameters (draft and draft_model instead of dflash and model).
| tool_calling: | ||
| description: "Qwen3 Coder tool-call parser with automatic tool choice" | ||
| args: | ||
| - "--enable-auto-tool-choice" | ||
| - "--tool-call-parser" | ||
| - "qwen3_coder" |
There was a problem hiding this comment.
The tool_calling configuration appears to be a copy-paste error from a Qwen-based recipe. There is no qwen3_coder tool-call parser in vLLM (and Qwen 3 is not a released model generation). Since Nemotron 3.5 Lightning does not use this parser, this block should be removed to prevent vLLM from failing to start due to an invalid --tool-call-parser argument.
| - "--mamba-ssm-cache-dtype" | ||
| - "float16" |
There was a problem hiding this comment.
In vLLM, the argument to set the Mamba cache data type is --mamba-cache-dtype, not --mamba-ssm-cache-dtype. Using --mamba-ssm-cache-dtype will result in an unrecognized argument error.
- "--mamba-cache-dtype"
- "float16"| - "--speculative-config" | ||
| - '{"method":"dflash","model":"nvidia/DFlash-Nemotron-3.5-Nano-MOPD-Intermediate","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}' |
There was a problem hiding this comment.
In vLLM, the speculative decoding method for an external draft model is "draft" (not "dflash"), and the parameter to specify the draft model is "draft_model" (not "model"). Please update the speculative config JSON to use the correct method and parameter names.
- "--speculative-config"
- '{"method":"draft","draft_model":"nvidia/DFlash-Nemotron-3.5-Nano-MOPD-Intermediate","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'| --kv-cache-dtype fp8 \ | ||
| --mamba-backend flashinfer \ | ||
| --mamba-cache-mode align \ | ||
| --mamba-ssm-cache-dtype float16 \ |
There was a problem hiding this comment.
Update --mamba-ssm-cache-dtype to --mamba-cache-dtype to match the correct vLLM argument name.
--mamba-cache-dtype float16 \| --kv-cache-dtype fp8 \ | ||
| --mamba-backend flashinfer \ | ||
| --mamba-cache-mode align \ | ||
| --mamba-ssm-cache-dtype float16 \ |
There was a problem hiding this comment.
Update --mamba-ssm-cache-dtype to --mamba-cache-dtype to match the correct vLLM argument name.
--mamba-cache-dtype float16 \| Key flags: | ||
| - `--mamba-backend flashinfer` — required; the default backend has no kernel for this stack | ||
| - `--mamba-cache-mode align` — aligned Mamba state cache, needed for prefix caching | ||
| - `--mamba-ssm-cache-dtype float16` with stochastic rounding trades a little accuracy for decode speed; drop all three flags for the FP32 default |
There was a problem hiding this comment.
Update --mamba-ssm-cache-dtype to --mamba-cache-dtype in the explanation text.
1 participant
Adds a recipe for NVIDIA Nemotron 3.5 Lightning covering the NVFP4 (W4A4) and BF16 checkpoints on H100, GB200 and DGX Spark, with MTP and DFlash speculative decoding modes and the FlashInfer Mamba cache flags the model requires. Draft until the checkpoints are public on Hugging Face.