diff --git a/models/Google/gemma-4-31B-it.yaml b/models/Google/gemma-4-31B-it.yaml index b9325120..c8dca77b 100644 --- a/models/Google/gemma-4-31B-it.yaml +++ b/models/Google/gemma-4-31B-it.yaml @@ -3,7 +3,7 @@ meta: slug: "gemma-4-31b-it" provider: "Google" description: "Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol." - date_updated: 2026-05-11 + date_updated: 2026-07-24 difficulty: intermediate tasks: - multimodal @@ -288,6 +288,12 @@ guide: | ### Structured Outputs vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions. + Gemma 4 31B can enter a grammar-valid whitespace loop after producing a valid JSON prefix, eventually reaching `max_tokens` with `finish_reason="length"`. If you observe this failure mode, add the following option to your existing server launch command to enable compact xgrammar JSON: + ```bash + --structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}' + ``` + This disables optional whitespace between JSON tokens; it does not remove whitespace inside string values. See [vLLM #40080](https://github.com/vllm-project/vllm/issues/40080) for background. + ## Configuration Tips - Set `--max-model-len` to match your workload.