diff --git a/CLAUDE.md b/CLAUDE.md index dd70ed74..bbfc5831 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,6 +73,7 @@ Top-level keys, in this order: - `precision_hardware` (optional, variant-level) — overrides the default hardware constraint implied by `precision`, using `{ brand?, generation? }`. Use only for exceptional checkpoints whose runtime fallback expands compatibility beyond the stored format, such as NVFP4 weights that vLLM dequantizes to BF16 on Hopper. Pair it with `supported_hardware` when only specific GPU profiles are documented. - `compatible_strategies[]` — subset of the serving strategy ids. Dense models typically only get `single_node_tp` + `multi_node_tp`; MoE models can have every serving strategy. Do NOT list `kv_store_*_mooncake` ids here — KV-offload options (Simple + both Mooncake modes) are implicit on every non-omni recipe and feed the "KV Offload" row, not the Strategy row. - `strategy_min_gpus` (optional) — minimum GPUs a strategy needs **for this model**. Either a bare number (one bar for every strategy) or a map keyed by strategy id (`{ multi_node_dep: 16, multi_node_tp: 8 }`) so one parallelism can demand more than another — e.g. Kimi-K3 runs TP/TEP from 8 GPUs but needs 16 for DEP. The map takes a second, disjoint keyspace: an **exact GPU id** (`h100: 32`, or nested per-strategy `h100: { multi_node_dep: 32 }`) layering a hardware-conditional floor on top of the strategy bars, **highest wins**. Use it when one GPU needs a bigger cluster than the recipe baseline — K3's ~1.68 TB of weights fits 2 H200 nodes but needs 4 on H100, so `h100: 32`, which also drops `single_node_tp` from the H100 pill row for free (8 GPUs can't clear a 32-GPU bar). Only exact GPU ids are read, never generations: GPUs of one generation differ in VRAM, which is exactly what moves the floor. Opt-in and fail-open: absent = no floor, so existing recipes are unaffected. A multi-node strategy stays offered and *grows the node count* to meet its bar when picked (the Nodes row gains the count that first clears it, e.g. 4 on a 4-GPU NVL tray); a single-node strategy can't scale, so it drops out when the floor exceeds one node's GPUs. Applies to PD pools too, keyed by pool parallelism (`tp`/`tep`/`dep` → the matching `multi_node_*` entry) — each pool is sized to its own floor. Helpers: `minGpusForStrategy` / `nodesForStrategy` / `isStrategyReachable` in `command-synthesis.js`, shared by the builder and the JSON API. Distinct from the (currently unread) `hardware_match.min_gpus` in `strategies/*.yaml`, which is a per-strategy structural minimum rather than a per-recipe one. +- `default_strategy` / `default_strategy_hardware` (optional) — recommendation only; the Strategy row still offers every compatible strategy. `default_strategy` sets the recipe-wide recommended serving strategy. `default_strategy_hardware: { : }` overrides that recommendation for exact hardware ids (e.g. `dgx_station_gb300: single_node_tp`) and falls back to `default_strategy` when absent or invalid. - `kv_cache_strategy_hardware` (optional) — per-strategy/per-GPU opt-OUT for the Mooncake pills: `{ : { : unsupported } }`. Follows the repo's fail-open hardware convention: absent = assumed to work (pill enabled on any scalable GPU); `unsupported` disables the pill and makes the JSON API skip that strategy on that hardware. "Off" and "Simple" need no gating (Simple is defined once in `taxonomy.yaml → kv_offload`). - `strategy_hardware` (optional) — the serving-Strategy-row analog of `kv_cache_strategy_hardware`: per-serving-strategy/per-GPU opt-OUT, `{ : { : unsupported } }`. Same fail-open convention (absent = works). `unsupported` renders the Strategy pill disabled with a tooltip, keeps it out of the recommended/active fallback, and makes the JSON API skip that strategy on that GPU. Use it when a strategy in `compatible_strategies` is genuinely unusable on a specific GPU (e.g. multi-node pure-TP on Hopper, which has no cross-node NVLink) — not for "untested". Helper: `isStrategySupported` in `CommandBuilder.jsx`. - `hardware_overrides` — optional per-generation tweaks keyed by `hopper`, `blackwell`, `amd`, or the brand-wide `nvidia`. Generation-specific blocks take precedence over the NVIDIA fallback. Each supplies `{ extra_args[], extra_env{} }`. diff --git a/models/deepseek-ai/DeepSeek-V4-Flash.yaml b/models/deepseek-ai/DeepSeek-V4-Flash.yaml index ea44cd49..328eab9a 100644 --- a/models/deepseek-ai/DeepSeek-V4-Flash.yaml +++ b/models/deepseek-ai/DeepSeek-V4-Flash.yaml @@ -117,12 +117,16 @@ variants: vram_minimum_gb: 170 description: "Native FP4+FP8 mixed checkpoint (MoE experts FP4, remaining params FP8)" # deep_gemm_mega_moe is an FP8-only MoE kernel — the FP8 checkpoint adds it on - # Blackwell; the NVFP4 variant below adds nothing (its experts can't use it). + # Blackwell datacenter GPUs. DGX Station keeps vLLM's default MoE backend. hardware_overrides: blackwell: extra_args: - "--moe-backend" - "deep_gemm_mega_moe" + dgx_station_gb300: + extra_args: + - "--moe-backend" + - "auto" nvfp4: model_id: "nvidia/DeepSeek-V4-Flash-NVFP4" precision: nvfp4 @@ -159,6 +163,9 @@ compatible_strategies: # TP+EP is the default deployment; single_node_tp is offered only as a # latency-oriented alternative. default_strategy: single_node_tep +# DGX Station defaults to TP because it uses vLLM's auto MoE backend instead of the deep_gemm_mega_moe kernel. +default_strategy_hardware: + dgx_station_gb300: single_node_tp hardware_overrides: blackwell: @@ -376,16 +383,26 @@ guide: | ### DGX Station Single-GPU + Use VLLM 0.24.0+ for serving on DGX Station. + ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash \ - --tensor-parallel-size 1 --pipeline-parallel-size 1 \ - --kv-cache-dtype fp8 --trust-remote-code --block-size 256 \ + --trust-remote-code \ + --kv-cache-dtype fp8 \ + --block-size 256 \ --gpu-memory-utilization 0.92 \ + --max-num-seqs 16 \ + --attention_config.use_fp4_indexer_cache=True \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \ - --attention_config.use_fp4_indexer_cache True \ - --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \ - --enable-auto-tool-choice --reasoning-parser deepseek_v4 \ --max-cudagraph-capture-size 128 \ + --tokenizer-mode deepseek_v4 \ + --tool-call-parser deepseek_v4 \ + --enable-auto-tool-choice \ + --reasoning-parser deepseek_v4 \ + --enable-chunked-prefill \ + --enable-prefix-caching \ + --max-num-batched-tokens 8192 \ + --no-disable-hybrid-kv-cache-manager \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` diff --git a/scripts/build-recipes-api.mjs b/scripts/build-recipes-api.mjs index d06563a5..5b3a5c02 100644 --- a/scripts/build-recipes-api.mjs +++ b/scripts/build-recipes-api.mjs @@ -452,7 +452,7 @@ function buildVariantRendering(recipe, variantKey, hwId, strategies, taxonomy) { }); const supportsMultiNode = scalable && compatible.some((s) => s.startsWith("multi_node_")); const baseNodeCount = !fitsSingleNode(hwProfile, variant) && supportsMultiNode ? 2 : 1; - let recommendedStrategy = recommendStrategy(recipe, hwProfile, baseNodeCount); + let recommendedStrategy = recommendStrategy(recipe, hwProfile, baseNodeCount, hwId); // Never recommend a strategy this GPU can't actually run — opted out via // strategy_hardware, or unreachable under its `strategy_min_gpus` floor // (e.g. a single-node layout on a GPU whose floor spans several nodes). @@ -494,7 +494,7 @@ function buildVariantRendering(recipe, variantKey, hwId, strategies, taxonomy) { if (!strategies[id] || d === "pd_cluster" || d === "kv_store_lb") return false; return nc > 1 ? d === "multi_node" : d !== "multi_node"; }; - const rec = recommendStrategy(recipe, hwProfile, nc); + const rec = recommendStrategy(recipe, hwProfile, nc, hwId); if (ok(rec)) return rec; return (recipe.compatible_strategies || []).find(ok) || null; }; @@ -761,15 +761,18 @@ for (const file of findYamlFiles(modelsDir)) { } } - // strategy_overrides, compatible_strategies, default_strategy are synthesis + // strategy_overrides, compatible_strategies, default_strategy, and + // default_strategy_hardware are synthesis // inputs whose effects are already baked into recommended_command + the // per-strategy alternative files (whose keys are the compatible_strategies - // set; the recommended one comes from default_strategy). Drop AFTER all + // set; the recommended one comes from default_strategy/default_strategy_hardware). + // Drop AFTER all // renderings (parent + promoted) have run — buildVariantRendering reads // them. The YAML on GitHub is the source of truth for anyone re-synthesizing. delete r.strategy_overrides; delete r.compatible_strategies; delete r.default_strategy; + delete r.default_strategy_hardware; // JSON at //.json — mirrors HF URL scheme. const out = writeRecipeJson(parentHfId, r); recipes.push(out); diff --git a/src/components/recipes/CommandBuilder.jsx b/src/components/recipes/CommandBuilder.jsx index 323449a4..096685e5 100644 --- a/src/components/recipes/CommandBuilder.jsx +++ b/src/components/recipes/CommandBuilder.jsx @@ -832,7 +832,7 @@ export function CommandBuilder({ recipe, strategies, taxonomy }) { // nodes, so multi-node is off and variants that don't fit are disabled. const hwScalable = isHardwareScalable(hwProfile); - const recommended = useMemo(() => recommendStrategy(recipe, hwProfile, nodeCount), [recipe, hwProfile, nodeCount]); + const recommended = useMemo(() => recommendStrategy(recipe, hwProfile, nodeCount, hwId), [recipe, hwProfile, nodeCount, hwId]); const perNode = hwProfile?.gpu_count || 8; // Nodes a strategy (or PD pool mode) needs to clear its `strategy_min_gpus` @@ -1179,7 +1179,7 @@ export function CommandBuilder({ recipe, strategies, taxonomy }) { // deliberate Single-/Multi-node click afterwards still wins. Non-scalable // hardware never bumps — it's single-node by definition. const fitsNew = fitsSingleNode(newProfile, activeVariant); - const recipeDefault = recipe.default_strategy; + const recipeDefault = recipe.default_strategy_hardware?.[id] || recipe.default_strategy; const recipeDefaultsSingleNode = typeof recipeDefault === "string" && recipeDefault.startsWith("single_node_"); const shouldBumpNodes = nodeCount === 1 && supportsMultiNode && newScalable && !fitsNew; diff --git a/src/lib/command-synthesis.js b/src/lib/command-synthesis.js index d70a611e..9bf6d99c 100644 --- a/src/lib/command-synthesis.js +++ b/src/lib/command-synthesis.js @@ -94,13 +94,18 @@ export function resolveSingleNodeTp( * tested, works for both dense and MoE. TEP / DEP / PD-cluster are * advanced strategies that users can opt into explicitly. */ -export function recommendStrategy(recipe, _hwProfile, nodeCount = 1) { +export function recommendStrategy(recipe, _hwProfile, nodeCount = 1, hwProfileId = null) { const compatible = recipe.compatible_strategies || []; // Recipe-level override — useful when the global TP-first preference is wrong // for a model (e.g. MoE recipes where TEP/DEP is the intended default and TP - // is offered only as a latency-oriented alternative). - const explicit = recipe.default_strategy; - if (explicit && compatible.includes(explicit)) { + // is offered only as a latency-oriented alternative). Per-hardware overrides + // let a specific GPU default differently while keeping every strategy offered. + const explicitCandidates = [ + hwProfileId ? recipe.default_strategy_hardware?.[hwProfileId] : null, + recipe.default_strategy, + ].filter(Boolean); + for (const explicit of explicitCandidates) { + if (!compatible.includes(explicit)) continue; if (nodeCount > 1 && explicit.startsWith("single_node_")) { // Single-node default at >1 node: prefer the multi-node sibling so a // recipe whose single-node default is single_node_tep doesn't fall back