Add new recipes for Intel XPU(B60, B70) - #753
Conversation
Signed-off-by: yuwenzho <yuwen.zhou@intel.com>
Signed-off-by: Yintong Lu <yintong.lu@intel.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Code Review
This pull request introduces support and deployment documentation for Intel Arc Pro B60 and B70 (Intel XPU) hardware across various model recipes, including Gemma 4, Qwen 3.5, Qwen 3.6, and NVIDIA Nemotron 3. The review feedback points out critical configuration issues that would lead to out-of-memory (OOM) errors. Specifically, the NVIDIA Nemotron 3 Nano 30B recipe incorrectly configures a single-GPU setup with a tensor parallel size of 1 instead of a multi-GPU setup with a tensor parallel size of 4. Additionally, several Qwen model configurations are missing essential hardware_overrides for --tensor-parallel-size and --max-model-len to align with their validated Docker run commands.
| hardware_overrides: | ||
| xpu: | ||
| extra_args: | ||
| - "--enforce-eager" |
There was a problem hiding this comment.
The hardware_overrides.xpu configuration is missing the --tensor-parallel-size and --max-model-len parameters. Without these, running the recipe on XPU will default to TP=1 and full context length, which will cause an immediate OOM. Please add these overrides to match the validated Docker command.
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--tensor-parallel-size"
- "4"
- "--max-model-len"
- "8192"| hardware_overrides: | ||
| xpu: | ||
| extra_args: | ||
| - "--enforce-eager" |
There was a problem hiding this comment.
The hardware_overrides.xpu configuration is missing the --tensor-parallel-size and --max-model-len parameters. Without these, running the recipe on XPU will default to TP=1 and full context length, which will cause an immediate OOM. Please add these overrides to match the validated Docker command.
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--tensor-parallel-size"
- "4"
- "--max-model-len"
- "8192"| hardware_overrides: | ||
| xpu: | ||
| extra_args: | ||
| - "--enforce-eager" |
There was a problem hiding this comment.
The hardware_overrides.xpu configuration is missing the --tensor-parallel-size and --max-model-len parameters. Without these, running the recipe on XPU will default to TP=1 and full context length, which will cause an immediate OOM. Please add these overrides to match the validated Docker command.
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--tensor-parallel-size"
- "4"
- "--max-model-len"
- "8192"| xpu: | ||
| extra_args: | ||
| - "--enforce-eager" |
There was a problem hiding this comment.
The hardware_overrides.xpu configuration is missing the --tensor-parallel-size and --max-model-len parameters. Without these, running the recipe on XPU will default to TP=1 and full context length, which will cause an immediate OOM. Please add these overrides to match the validated Docker command.
xpu:
extra_args:
- "--enforce-eager"
- "--tensor-parallel-size"
- "4"
- "--max-model-len"
- "8192"| hardware_overrides: | ||
| xpu: | ||
| extra_args: | ||
| - "--enforce-eager" |
There was a problem hiding this comment.
The hardware_overrides.xpu configuration is missing the --max-model-len parameter. Without this, running the recipe on XPU will default to the full context length, which will cause an OOM. Please add this override to match the validated Docker command.
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--max-model-len"
- "8192"|
@jikunshang Could you help to review? |
| VLLM_WORKER_MULTIPROC_METHOD: "spawn" | ||
| extra_args: | ||
| - "--enforce-eager" | ||
| - "--attention-backend" |
There was a problem hiding this comment.
it's default option, we can remove.
There was a problem hiding this comment.
I kept --attention-backend flash_attn intentionally. For Gemma 4 on XPU, if the attention backend is not explicitly set, vLLM treats Gemma 4 as a multimodal prefix-LM model and the XPU backend selector falls back to Triton attention because Flash Attention does not support the multimodal prefix-LM mask path.
Removed other args as per suggested.
|
please fix dco issue. |
Signed-off-by: Yintong Lu <yintong.lu@intel.com>
Signed-off-by: Zhefeng, Qiao <zhefeng.qiao@intel.com>
for nemotron-3-nano, gemma-4 and qwen3.5/qwen3.6