Skip to content

docs(google/gemma-4-31b-it): document compact JSON workaround #679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mujae wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(google/gemma-4-31b-it): document compact JSON workaround #679

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion models/Google/gemma-4-31B-it.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ meta:
slug: "gemma-4-31b-it"
provider: "Google"
description: "Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_updated: 2026-05-11
date_updated: 2026-07-24
difficulty: intermediate
tasks:
- multimodal
Expand Down Expand Up @@ -288,6 +288,12 @@ guide: |
### Structured Outputs
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.

Gemma 4 31B can enter a grammar-valid whitespace loop after producing a valid JSON prefix, eventually reaching `max_tokens` with `finish_reason="length"`. If you observe this failure mode, add the following option to your existing server launch command to enable compact xgrammar JSON:
```bash
--structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}'
```
This disables optional whitespace between JSON tokens; it does not remove whitespace inside string values. See [vLLM #40080](https://github.com/vllm-project/vllm/issues/40080) for background.

## Configuration Tips

- Set `--max-model-len` to match your workload.
Expand Down