diff --git a/models/Google/gemma-4-12B-it.yaml b/models/Google/gemma-4-12B-it.yaml index 5d5f8d31..0e925b37 100644 --- a/models/Google/gemma-4-12B-it.yaml +++ b/models/Google/gemma-4-12B-it.yaml @@ -16,6 +16,8 @@ meta: - google/gemma-4-31B-it hardware: h100: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "google/gemma-4-12B-it" @@ -74,7 +76,22 @@ compatible_strategies: - single_node_tp - multi_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" + - "--attention-backend" + - "flash_attn" + - "--trust-remote-code" + - "--gpu-memory-utilization" + - "0.92" + - "--no-enable-prefix-caching" + - "--max-num-batched-tokens" + - "8192" + - "--max-model-len" + - "5120" + - "--tensor-parallel-size" + - "2" strategy_overrides: single_node_tp: @@ -114,6 +131,7 @@ guide: | ### Docker (recommended) ```bash docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts) + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. @@ -166,6 +184,35 @@ guide: | ``` On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead. + ### Docker (Intel XPU B60 / B70, TP=2, Full-Featured) + + Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-12B-it \ + --enforce-eager \ + --attention-backend flash_attn \ + --host 0.0.0.0 \ + --port 8000 \ + --trust-remote-code \ + --enable-auto-tool-choice \ + --reasoning-parser gemma4 \ + --tool-call-parser gemma4 \ + --chat-template examples/tool_chat_template_gemma4.jinja \ + --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ + --async-scheduling \ + --gpu-memory-utilization 0.92 \ + --no-enable-prefix-caching \ + --max-num-batched-tokens 8192 \ + --max-model-len 5120 \ + --tensor-parallel-size 2" + ``` + ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash diff --git a/models/Google/gemma-4-26B-A4B-it.yaml b/models/Google/gemma-4-26B-A4B-it.yaml index 73107c98..07e44745 100644 --- a/models/Google/gemma-4-26B-A4B-it.yaml +++ b/models/Google/gemma-4-26B-A4B-it.yaml @@ -37,6 +37,8 @@ meta: dgx_station_gb300: verified rtx_pro_6000: verified rtx_5090: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "google/gemma-4-26B-A4B-it" @@ -237,6 +239,21 @@ hardware_overrides: extra_env: VLLM_CPU_KVCACHE_SPACE: "40" VLLM_CPU_ATTN_SPLIT_KV: "0" + xpu: + extra_args: + - "--enforce-eager" + - "--attention-backend" + - "flash_attn" + - "--trust-remote-code" + - "--gpu-memory-utilization" + - "0.92" + - "--no-enable-prefix-caching" + - "--max-num-batched-tokens" + - "8192" + - "--max-model-len" + - "5120" + - "--tensor-parallel-size" + - "4" strategy_overrides: single_node_tp: @@ -287,6 +304,7 @@ guide: | docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. @@ -375,6 +393,35 @@ guide: | For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it). + ### Docker (Intel XPU B60 / B70, TP=4, Full-Featured) + + Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-26B-A4B-it \ + --enforce-eager \ + --attention-backend flash_attn \ + --host 0.0.0.0 \ + --port 8000 \ + --trust-remote-code \ + --enable-auto-tool-choice \ + --reasoning-parser gemma4 \ + --tool-call-parser gemma4 \ + --chat-template examples/tool_chat_template_gemma4.jinja \ + --limit-mm-per-prompt.image 4 \ + --async-scheduling \ + --gpu-memory-utilization 0.92 \ + --no-enable-prefix-caching \ + --max-num-batched-tokens 8192 \ + --max-model-len 5120 \ + --tensor-parallel-size 4" + ``` + ## Client Usage ### Text Generation diff --git a/models/Google/gemma-4-31B-it.yaml b/models/Google/gemma-4-31B-it.yaml index cf1e660b..66dd55a5 100644 --- a/models/Google/gemma-4-31B-it.yaml +++ b/models/Google/gemma-4-31B-it.yaml @@ -18,6 +18,8 @@ meta: mi355x: verified trillium: verified ironwood: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "google/gemma-4-31B-it" @@ -93,7 +95,22 @@ compatible_strategies: - single_node_tp - multi_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" + - "--attention-backend" + - "flash_attn" + - "--trust-remote-code" + - "--gpu-memory-utilization" + - "0.92" + - "--no-enable-prefix-caching" + - "--max-num-batched-tokens" + - "8192" + - "--max-model-len" + - "5120" + - "--tensor-parallel-size" + - "4" strategy_overrides: single_node_tp: @@ -150,6 +167,7 @@ guide: | docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9) docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. @@ -220,6 +238,35 @@ guide: | --host 0.0.0.0 --port 8000 ``` + ### Docker (Intel XPU B60 / B70, TP=4, Full-Featured) + + Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve google/gemma-4-31B-it \ + --enforce-eager \ + --attention-backend flash_attn \ + --host 0.0.0.0 \ + --port 8000 \ + --trust-remote-code \ + --enable-auto-tool-choice \ + --reasoning-parser gemma4 \ + --tool-call-parser gemma4 \ + --chat-template examples/tool_chat_template_gemma4.jinja \ + --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ + --async-scheduling \ + --gpu-memory-utilization 0.92 \ + --no-enable-prefix-caching \ + --max-num-batched-tokens 8192 \ + --max-model-len 5120 \ + --tensor-parallel-size 4" + ``` + ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash diff --git a/models/Qwen/Qwen3.5-0.8B.yaml b/models/Qwen/Qwen3.5-0.8B.yaml index 6fead71d..98903fe1 100644 --- a/models/Qwen/Qwen3.5-0.8B.yaml +++ b/models/Qwen/Qwen3.5-0.8B.yaml @@ -12,6 +12,9 @@ meta: performance_headline: "Tiny Qwen3.5 dense for edge / draft-model use" related_recipes: - "Qwen/Qwen3.5-2B" + hardware: + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-0.8B" @@ -53,7 +56,10 @@ variants: compatible_strategies: - single_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -68,7 +74,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware:** any modern GPU (>=4 GB VRAM) + - **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70 ### Install vLLM @@ -78,6 +84,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ```bash @@ -86,6 +97,22 @@ guide: | --reasoning-parser qwen3 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-0.8B \ + --reasoning-parser qwen3 \ + --enforce-eager" + ``` + ## Client Usage ```python diff --git a/models/Qwen/Qwen3.5-27B.yaml b/models/Qwen/Qwen3.5-27B.yaml index e982dca0..904b6c14 100644 --- a/models/Qwen/Qwen3.5-27B.yaml +++ b/models/Qwen/Qwen3.5-27B.yaml @@ -9,11 +9,14 @@ meta: tasks: - multimodal - text - performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU BF16" + performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU or 2-4x Intel Arc Pro B60/B70 BF16" related_recipes: - "Qwen/Qwen3.5-397B-A17B" - "Qwen/Qwen3.5-35B-A3B" - "Qwen/Qwen3.5-9B" + hardware: + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-27B" @@ -61,7 +64,7 @@ variants: default: precision: bf16 vram_minimum_gb: 65 - description: "Full precision BF16 — fits on 1x H200 or 2x H100" + description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70" fp8: model_id: "Qwen/Qwen3.5-27B-FP8" precision: fp8 @@ -77,7 +80,10 @@ compatible_strategies: - single_node_tp - multi_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -93,7 +99,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware (BF16):** 1x H200 or 2x H100 + - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70 - **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S) - **Hardware (Int4):** single 24 GB GPU @@ -105,6 +111,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ### Single-GPU FP8 @@ -141,6 +152,24 @@ guide: | --enable-prefix-caching ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-27B \ + --reasoning-parser qwen3 \ + --tensor-parallel-size 4 \ + --max-model-len 8192 \ + --enforce-eager" + ``` + ## Client Usage ```python diff --git a/models/Qwen/Qwen3.5-2B.yaml b/models/Qwen/Qwen3.5-2B.yaml index 25a99bce..ec4ffee5 100644 --- a/models/Qwen/Qwen3.5-2B.yaml +++ b/models/Qwen/Qwen3.5-2B.yaml @@ -9,10 +9,13 @@ meta: tasks: - multimodal - text - performance_headline: "Edge-scale Qwen3.5 dense — fits on 8 GB GPUs" + performance_headline: "Edge-scale Qwen3.5 dense — fits on 8 GB GPUs or 1x Intel Arc Pro B60/B70" related_recipes: - "Qwen/Qwen3.5-4B" - "Qwen/Qwen3.5-0.8B" + hardware: + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-2B" @@ -55,12 +58,15 @@ variants: default: precision: bf16 vram_minimum_gb: 5 - description: "Full precision BF16 — fits on an 8 GB GPU" + description: "Full precision BF16 — fits on an 8 GB GPU or Intel Arc Pro B60/B70" compatible_strategies: - single_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -75,7 +81,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware:** single 8 GB GPU + - **Hardware:** single 8 GB GPU or Intel Arc Pro B60/B70 ### Install vLLM @@ -85,6 +91,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ```bash @@ -93,6 +104,22 @@ guide: | --reasoning-parser qwen3 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-2B \ + --reasoning-parser qwen3 \ + --enforce-eager" + ``` + ## Client Usage ```python diff --git a/models/Qwen/Qwen3.5-35B-A3B.yaml b/models/Qwen/Qwen3.5-35B-A3B.yaml index 849facd6..e6d8f026 100644 --- a/models/Qwen/Qwen3.5-35B-A3B.yaml +++ b/models/Qwen/Qwen3.5-35B-A3B.yaml @@ -9,12 +9,14 @@ meta: tasks: - multimodal - text - performance_headline: "Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes BF16 serving" + performance_headline: "Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving" related_recipes: - "Qwen/Qwen3.5-397B-A17B" - "Qwen/Qwen3.5-122B-A10B" hardware: xeon6: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-35B-A3B" @@ -62,7 +64,7 @@ variants: default: precision: bf16 vram_minimum_gb: 84 - description: "Full precision BF16 — fits on 1x H200, 2x H100 or 2x Xeon 6 NUMA nodes" + description: "Full precision BF16 — fits on 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, or 4x Intel Arc Pro B60 / B70" fp8: model_id: "Qwen/Qwen3.5-35B-A3B-FP8" precision: fp8 @@ -82,7 +84,10 @@ compatible_strategies: - multi_node_dep - multi_node_tep -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -97,7 +102,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware (BF16):** 1x H200, 2x H100 or 2x Xeon 6 NUMA nodes + - **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70 - **Hardware (FP8):** single H100/H200 - **Hardware (Int4):** single 24 GB GPU @@ -120,6 +125,11 @@ guide: | uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ### Single-GPU FP8 @@ -162,6 +172,24 @@ guide: | --reasoning-parser qwen3 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-35B-A3B \ + --reasoning-parser qwen3 \ + --tensor-parallel-size 4 \ + --max-model-len 8192 \ + --enforce-eager" + ``` + ## Client Usage ```python diff --git a/models/Qwen/Qwen3.5-4B.yaml b/models/Qwen/Qwen3.5-4B.yaml index c0e86e74..30e62328 100644 --- a/models/Qwen/Qwen3.5-4B.yaml +++ b/models/Qwen/Qwen3.5-4B.yaml @@ -2,7 +2,7 @@ meta: title: "Qwen3.5-4B" slug: "qwen3.5-4b" provider: "Qwen" - description: "Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node" + description: "Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70" date_added: 2026-04-22 date_updated: 2026-04-22 difficulty: beginner @@ -15,6 +15,8 @@ meta: - "Qwen/Qwen3.5-2B" hardware: xeon6: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-4B" @@ -57,12 +59,15 @@ variants: default: precision: bf16 vram_minimum_gb: 10 - description: "Full precision BF16 — fits on a single 16 GB GPU or one Xeon 6 NUMA node" + description: "Full precision BF16 — fits on a single 16 GB GPU or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70" compatible_strategies: - single_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -77,7 +82,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node + - **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70 ### Install vLLM @@ -90,6 +95,11 @@ guide: | ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ```bash @@ -113,6 +123,22 @@ guide: | --port 8000 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-4B \ + --reasoning-parser qwen3 \ + --enforce-eager" + ``` + ### MTP speculative decoding diff --git a/models/Qwen/Qwen3.5-9B.yaml b/models/Qwen/Qwen3.5-9B.yaml index bb456348..ea9e3acf 100644 --- a/models/Qwen/Qwen3.5-9B.yaml +++ b/models/Qwen/Qwen3.5-9B.yaml @@ -13,6 +13,9 @@ meta: related_recipes: - "Qwen/Qwen3.5-27B" - "Qwen/Qwen3.5-4B" + hardware: + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.5-9B" @@ -55,13 +58,16 @@ variants: default: precision: bf16 vram_minimum_gb: 22 - description: "Full precision BF16 — single 24 GB GPU" + description: "Full precision BF16 — single 24 GB GPU or Intel Arc Pro B60/B70" compatible_strategies: - single_node_tp - multi_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -76,7 +82,7 @@ guide: | ## Prerequisites - **vLLM version:** >= 0.17.0 - - **Hardware:** single 24 GB GPU (RTX 4090 / L4 / A10G / H100) + - **Hardware:** single 24 GB GPU (RTX 4090 / L4 / A10G / H100) or Intel Arc Pro B60/B70 ### Install vLLM @@ -86,6 +92,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ### Single-GPU BF16 @@ -104,6 +115,23 @@ guide: | --reasoning-parser qwen3 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.5-9B \ + --reasoning-parser qwen3 \ + --max-model-len 8192 \ + --enforce-eager" + ``` + ## Client Usage ```python diff --git a/models/Qwen/Qwen3.6-27B.yaml b/models/Qwen/Qwen3.6-27B.yaml index 6736b36f..dd3c8a4e 100644 --- a/models/Qwen/Qwen3.6-27B.yaml +++ b/models/Qwen/Qwen3.6-27B.yaml @@ -9,7 +9,7 @@ meta: tasks: - multimodal - text - performance_headline: "Qwen3.6 flagship dense — single-GPU FP8 or 2x GPU BF16" + performance_headline: "Qwen3.6 flagship dense — single-GPU FP8 or 2x GPU or 2-4x Intel Arc Pro B60/B70 BF16" related_recipes: - "Qwen/Qwen3.6-35B-A3B" hardware: @@ -17,6 +17,8 @@ meta: rtx_pro_6000: verified dgx_station_gb300: verified rtx_5090: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.6-27B" @@ -64,7 +66,7 @@ variants: default: precision: bf16 vram_minimum_gb: 65 - description: "Full precision BF16 — fits on 1x H200 or 2x H100" + description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70" hardware_overrides: rtx_pro_6000: extra_args: @@ -130,7 +132,10 @@ compatible_strategies: - single_node_tp - multi_node_tp -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -147,7 +152,7 @@ guide: | - **vLLM version:** >= 0.17.0 - **DGX Spark NVFP4 vLLM version:** >= 0.24.0 - - **Hardware (BF16):** 1x H200 or 2x H100 + - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70 - **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S) - **Hardware (Int4):** single 24 GB GPU - **Hardware (NVFP4):** single NVIDIA Blackwell GPU (B200/B300/GB10 DGX Spark) @@ -160,6 +165,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ### Single-GPU FP8 @@ -366,6 +376,24 @@ guide: | --reasoning-parser qwen3 ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.6-27B \ + --reasoning-parser qwen3 \ + --tensor-parallel-size 4 \ + --max-model-len 8192 \ + --enforce-eager" + ``` + ## Processing Ultra-Long Texts Qwen3.6-27B natively supports `262,144` tokens. For longer inputs, apply diff --git a/models/Qwen/Qwen3.6-35B-A3B.yaml b/models/Qwen/Qwen3.6-35B-A3B.yaml index 0308f865..c3cae464 100644 --- a/models/Qwen/Qwen3.6-35B-A3B.yaml +++ b/models/Qwen/Qwen3.6-35B-A3B.yaml @@ -9,7 +9,7 @@ meta: tasks: - multimodal - text - performance_headline: "Compact Qwen3.6 MoE with 3B active parameters — single-GPU FP8 or 2-4 GPU BF16 serving" + performance_headline: "Compact Qwen3.6 MoE with 3B active parameters — single-GPU FP8 or 2-4 GPU or 4x Intel Arc Pro B60/B70 BF16 serving" related_recipes: - "Qwen/Qwen3.5-397B-A17B" hardware: @@ -22,6 +22,8 @@ meta: rtx_pro_6000: verified dgx_station_gb300: verified rtx_5090: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "Qwen/Qwen3.6-35B-A3B" @@ -69,7 +71,7 @@ variants: default: precision: bf16 vram_minimum_gb: 84 - description: "Full precision BF16 — fits on 1x H200 or 2x H100" + description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70" hardware_overrides: rtx_pro_6000: extra_args: @@ -133,6 +135,9 @@ hardware_overrides: extra_args: [] extra_env: VLLM_ROCM_USE_AITER: "1" + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -149,7 +154,7 @@ guide: | - **vLLM version:** >= 0.17.0 - **DGX Spark NVFP4 vLLM version:** >= 0.24.0 - - **Hardware (BF16):** 1x H200 or 2x H100 + - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70 - **Hardware (FP8):** single H100/H200 or 1x MI300X/MI325X/MI355X - **Hardware (NVFP4):** NVIDIA Blackwell GPUs, including DGX Spark (GB10) @@ -161,6 +166,11 @@ guide: | uv pip install -U vllm --torch-backend=auto ``` + ### Docker + ```bash + docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) + ``` + ## Launching the Server ### Single-GPU FP8 @@ -254,6 +264,24 @@ guide: | --enable-auto-tool-choice ``` + ### Docker (Intel XPU B60 / B70) + + Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve Qwen/Qwen3.6-35B-A3B \ + --reasoning-parser qwen3 \ + --tensor-parallel-size 4 \ + --max-model-len 8192 \ + --enforce-eager" + ``` + ## Processing Ultra-Long Texts Qwen3.6-35B-A3B natively supports `262,144` tokens. For longer inputs, apply diff --git a/models/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml b/models/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml index a1e0152f..de867e94 100644 --- a/models/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml +++ b/models/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml @@ -13,6 +13,8 @@ meta: hardware: h100: verified h200: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" @@ -60,9 +62,11 @@ variants: extra_args: - "--kv-cache-dtype" - "fp8" - extra_env: - VLLM_USE_FLASHINFER_MOE_FP8: "1" - VLLM_FLASHINFER_MOE_BACKEND: "throughput" + hardware_overrides: + nvidia: + extra_env: + VLLM_USE_FLASHINFER_MOE_FP8: "1" + VLLM_FLASHINFER_MOE_BACKEND: "throughput" compatible_strategies: - single_node_tp @@ -75,7 +79,10 @@ kv_offload_support: offloading_cpu: verified offloading_fs: verified -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" strategy_overrides: {} @@ -88,7 +95,7 @@ guide: | ## Prerequisites - - Hardware: 1x H100/H200 or comparable; DGX Spark and Jetson Thor supported + - Hardware: 1x H100/H200 or comparable; DGX Spark, Jetson Thor, and Intel Arc Pro B60/B70 supported - vLLM >= 0.11.2 (0.12.0 recommended for full support) - Docker with NVIDIA Container Toolkit (recommended) @@ -111,6 +118,12 @@ guide: | docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor ``` + Intel XPU (B60 / B70): + + ```bash + docker pull vllm/vllm-openai-xpu:latest + ``` + ## Launch commands FP8 with FlashInfer MoE backend (Blackwell/Hopper): @@ -142,6 +155,47 @@ guide: | --reasoning-parser nano_v3 ``` + ### Intel XPU (B60, B70) + + Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + FP8: + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ + --trust-remote-code \ + --async-scheduling \ + --kv-cache-dtype fp8 \ + --tensor-parallel-size 4 \ + --enforce-eager \ + --max-model-len 8192" + ``` + + BF16: + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ + --trust-remote-code \ + --async-scheduling \ + --kv-cache-dtype auto \ + --tensor-parallel-size 4 \ + --enforce-eager \ + --enable-auto-tool-choice \ + --tool-call-parser qwen3_coder \ + --reasoning-parser-plugin nano_v3_reasoning_parser.py \ + --reasoning-parser nano_v3 \ + --max-model-len 8192" + ``` + Key flags: - `kv-cache-dtype fp8` for FP8 variant, `auto` for BF16 - `async-scheduling` reduces host overhead between decode steps diff --git a/models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.yaml b/models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.yaml index 24a535ae..7444cb21 100644 --- a/models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.yaml +++ b/models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.yaml @@ -15,6 +15,8 @@ meta: h100: verified h200: verified b200: verified + arc_pro_b60: verified + arc_pro_b70: verified model: model_id: "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16" @@ -72,7 +74,11 @@ kv_offload_support: offloading_cpu: verified offloading_fs: verified -hardware_overrides: {} +hardware_overrides: + xpu: + extra_args: + - "--enforce-eager" + strategy_overrides: {} guide: | @@ -84,7 +90,7 @@ guide: | ## Prerequisites - - Hardware: 1x H100/H200/B200, DGX Spark, or Jetson Thor + - Hardware: 1x H100/H200/B200, DGX Spark, Jetson Thor, or Intel Arc Pro B60/B70 - vLLM >= 0.11.2 (0.12.0 recommended) - Docker with NVIDIA Container Toolkit (recommended) @@ -100,6 +106,12 @@ guide: | docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor ``` + Intel XPU (B60 / B70): + + ```bash + docker pull vllm/vllm-openai-xpu:latest + ``` + ## Launch commands BF16: @@ -128,6 +140,47 @@ guide: | --tensor-parallel-size 1 ``` + ### Intel XPU (B60, B70) + + Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the + official vLLM XPU image `vllm/vllm-openai-xpu:latest`. + + BF16: + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \ + --trust-remote-code \ + --async-scheduling \ + --kv-cache-dtype auto \ + --tensor-parallel-size 1 \ + --enforce-eager \ + --enable-auto-tool-choice \ + --tool-call-parser qwen3_coder \ + --reasoning-parser-plugin nano_v3_reasoning_parser.py \ + --reasoning-parser nano_v3 \ + --max-model-len 8192" + ``` + + FP8: + ```bash + docker run --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ + --privileged --ipc=host -p 8000:8000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --entrypoint bash vllm/vllm-openai-xpu:latest \ + -c "source /opt/intel/oneapi/setvars.sh && exec vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \ + --trust-remote-code \ + --async-scheduling \ + --kv-cache-dtype fp8 \ + --tensor-parallel-size 1 \ + --enforce-eager \ + --max-model-len 8192" + ``` + ## Benchmarking ```bash