Skip to content

Add new recipes for Intel XPU(B60, B70) #753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Dboyqiao wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add new recipes for Intel XPU(B60, B70) #753

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0d4e505
Add Intel Arc Pro B60/B70 recipes for nemotron-3-nano-4b and nemotron…
Dboyqiao Jul 31, 2026
5148f3e
Add Intel Arc Pro B60/B70 recipes for qwen3.5 / qwen3.6 models
yuwenzho Aug 3, 2026
5f72314
Add Intel Arc Pro B60/B70 recipes for gemma-4 models
yintong-lu Aug 3, 2026
8d39ffe
fix typo
Dboyqiao Aug 6, 2026
f064e20
modify Gemma 4 XPU recipe args
yintong-lu Aug 7, 2026
59f30b0
Merge branch 'main' into recipes_for_xpu
Dboyqiao Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion models/Google/gemma-4-12B-it.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ meta:
- google/gemma-4-31B-it
hardware:
h100: verified
arc_pro_b60: verified
arc_pro_b70: verified

model:
model_id: "google/gemma-4-12B-it"
Expand Down Expand Up @@ -74,7 +76,22 @@ compatible_strategies:
- single_node_tp
- multi_node_tp

hardware_overrides: {}
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"

@jikunshang jikunshang Aug 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's default option, we can remove.

@yintong-lu yintong-lu Aug 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept --attention-backend flash_attn intentionally. For Gemma 4 on XPU, if the attention backend is not explicitly set, vLLM treats Gemma 4 as a multimodal prefix-LM model and the XPU backend selector falls back to Triton attention because Flash Attention does not support the multimodal prefix-LM mask path.
Removed other args as per suggested.

@jikunshang jikunshang Aug 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it thanks.

- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.92"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "2"

strategy_overrides:
single_node_tp:
Expand Down Expand Up @@ -114,6 +131,7 @@ guide: |
### Docker (recommended)
```bash
docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts)
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.

Expand Down Expand Up @@ -166,6 +184,35 @@ guide: |
```
On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead.

### Docker (Intel XPU B60 / B70, TP=2, Full-Featured)

Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.

```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-xpu:latest \
-c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-12B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.92 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 2"
```

### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
Expand Down
47 changes: 47 additions & 0 deletions models/Google/gemma-4-26B-A4B-it.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ meta:
dgx_station_gb300: verified
rtx_pro_6000: verified
rtx_5090: verified
arc_pro_b60: verified
arc_pro_b70: verified

model:
model_id: "google/gemma-4-26B-A4B-it"
Expand Down Expand Up @@ -237,6 +239,21 @@ hardware_overrides:
extra_env:
VLLM_CPU_KVCACHE_SPACE: "40"
VLLM_CPU_ATTN_SPLIT_KV: "0"
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.92"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"

strategy_overrides:
single_node_tp:
Expand Down Expand Up @@ -287,6 +304,7 @@ guide: |
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.

Expand Down Expand Up @@ -375,6 +393,35 @@ guide: |

For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it).

### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)

Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.

```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-xpu:latest \
-c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-26B-A4B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling \
--gpu-memory-utilization 0.92 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4"
```

## Client Usage

### Text Generation
Expand Down
49 changes: 48 additions & 1 deletion models/Google/gemma-4-31B-it.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ meta:
mi355x: verified
trillium: verified
ironwood: verified
arc_pro_b60: verified
arc_pro_b70: verified

model:
model_id: "google/gemma-4-31B-it"
Expand Down Expand Up @@ -93,7 +95,22 @@ compatible_strategies:
- single_node_tp
- multi_node_tp

hardware_overrides: {}
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.92"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"

strategy_overrides:
single_node_tp:
Expand Down Expand Up @@ -150,6 +167,7 @@ guide: |
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.

Expand Down Expand Up @@ -220,6 +238,35 @@ guide: |
--host 0.0.0.0 --port 8000
```

### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)

Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.

```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-xpu:latest \
-c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-31B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.92 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4"
```

### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
Expand Down
31 changes: 29 additions & 2 deletions models/Qwen/Qwen3.5-0.8B.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ meta:
performance_headline: "Tiny Qwen3.5 dense for edge / draft-model use"
related_recipes:
- "Qwen/Qwen3.5-2B"
hardware:
arc_pro_b60: verified
arc_pro_b70: verified

model:
model_id: "Qwen/Qwen3.5-0.8B"
Expand Down Expand Up @@ -53,7 +56,10 @@ variants:
compatible_strategies:
- single_node_tp

hardware_overrides: {}
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"

strategy_overrides: {}

Expand All @@ -68,7 +74,7 @@ guide: |
## Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware:** any modern GPU (>=4 GB VRAM)
- **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70

### Install vLLM

Expand All @@ -78,6 +84,11 @@ guide: |
uv pip install -U vllm --torch-backend=auto
```

### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```

## Launching the Server

```bash
Expand All @@ -86,6 +97,22 @@ guide: |
--reasoning-parser qwen3
```

### Docker (Intel XPU B60 / B70)

Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-xpu:latest \
-c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-0.8B \
--reasoning-parser qwen3 \
--enforce-eager"
```

## Client Usage

```python
Expand Down
37 changes: 33 additions & 4 deletions models/Qwen/Qwen3.5-27B.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,14 @@ meta:
tasks:
- multimodal
- text
performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU BF16"
performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU or 2-4x Intel Arc Pro B60/B70 BF16"
related_recipes:
- "Qwen/Qwen3.5-397B-A17B"
- "Qwen/Qwen3.5-35B-A3B"
- "Qwen/Qwen3.5-9B"
hardware:
arc_pro_b60: verified
arc_pro_b70: verified

model:
model_id: "Qwen/Qwen3.5-27B"
Expand Down Expand Up @@ -61,7 +64,7 @@ variants:
default:
precision: bf16
vram_minimum_gb: 65
description: "Full precision BF16 — fits on 1x H200 or 2x H100"
description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70"
fp8:
model_id: "Qwen/Qwen3.5-27B-FP8"
precision: fp8
Expand All @@ -77,7 +80,10 @@ compatible_strategies:
- single_node_tp
- multi_node_tp

hardware_overrides: {}
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
Comment on lines +83 to +86

@gemini-code-assist gemini-code-assist Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The hardware_overrides.xpu configuration is missing the --tensor-parallel-size and --max-model-len parameters. Without these, running the recipe on XPU will default to TP=1 and full context length, which will cause an immediate OOM. Please add these overrides to match the validated Docker command.

hardware_overrides:
  xpu:
    extra_args:
      - "--enforce-eager"
      - "--tensor-parallel-size"
      - "4"
      - "--max-model-len"
      - "8192"


strategy_overrides: {}

Expand All @@ -93,7 +99,7 @@ guide: |
## Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 1x H200 or 2x H100
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70
- **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S)
- **Hardware (Int4):** single 24 GB GPU

Expand All @@ -105,6 +111,11 @@ guide: |
uv pip install -U vllm --torch-backend=auto
```

### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```

## Launching the Server

### Single-GPU FP8
Expand Down Expand Up @@ -141,6 +152,24 @@ guide: |
--enable-prefix-caching
```

### Docker (Intel XPU B60 / B70)

Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash vllm/vllm-openai-xpu:latest \
-c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-27B \
--reasoning-parser qwen3 \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--enforce-eager"
```

## Client Usage

```python
Expand Down
Loading