Skip to content

Add GLM-5.2 low-latency serving recipe - #652

Draft
simon-mo wants to merge 1 commit into
mainfrom
agent/glm-5-2-low-latency
Draft

Add GLM-5.2 low-latency serving recipe#652
simon-mo wants to merge 1 commit into
mainfrom
agent/glm-5-2-low-latency

Conversation

@simon-mo

Copy link
Copy Markdown
Contributor

What changed

  • Pin the GLM-5.2 NVFP4 low-latency path to vllm/vllm-openai:v0.25.1.
  • Configure the single-node Blackwell command with TP8, FP8 KV cache, MTP=5, the fused GLM model-class override, breakable piecewise CUDA graphs, tool/reasoning parsers, and the stable served model name.
  • Make Docker the primary install path and replace the stale :glm52 guide example with the release-tested copy-paste command.
  • Preserve variant-forced default_modes when building the static JSON API.

Why

The public GLM-5.2 guide still showed the older image and a generic NVFP4 command, while the current low-latency path requires vLLM 0.25.1 and explicit fused-model/CUDA-graph configuration.

The API generator also excluded opt-in features unconditionally. As a result, promoted variant endpoints dropped MTP even when a variant's default_modes explicitly enabled it. The generator now mirrors the interactive command builder.

Impact

Selecting the NVFP4 variant on B200/B300, or consuming /nvidia/GLM-5.2-NVFP4.json, produces the same low-latency command with MTP=5 and the pinned Docker image.

Validation

  • node scripts/build-recipes-api.mjs
  • Directly inspected the generated native and Docker commands in public/nvidia/GLM-5.2-NVFP4.json.
  • git diff --check

pnpm lint was not available as a non-interactive check because this repository has no ESLint configuration and next lint opens the first-run configuration prompt.

Signed-off-by: simon-mo <simon.mo@hey.com>
@vercel

vercel Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vllm-recipes Ready Ready Preview, Comment Jul 17, 2026 8:59pm

Request Review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the GLM-5.2 model recipe to support the low-latency NVFP4 variant on Blackwell GPUs. It introduces specific installation notes, prerequisites, and detailed Docker/CLI command configurations for NVFP4, including MTP speculative decoding, model class overrides, and breakable piecewise CUDA graphs. Additionally, the recipe build script (build-recipes-api.mjs) is updated to automatically enable features forced by a variant's default_modes. There are no review comments, and I have no feedback to provide.

@gaby

gaby commented Jul 22, 2026

Copy link
Copy Markdown

@simon-mo Fix the merge conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants