Skip to content

fix(qualify): render native JSON schema instructions in frozen prompts - #164

Merged
original-el8 merged 1 commit into
NVIDIA:mainfrom
Maxrio9:fix-qualify-json-schema-prompts-v2
Nov 13, 2026
Merged

original-el8 merged 1 commit into
NVIDIA:mainfrom
Maxrio9:fix-qualify-json-schema-prompts-v2

Conversation

@Maxrio9

@Maxrio9 Maxrio9 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Replaces #19: rebased onto main 6fa01a49 and the commit now carries the DCO sign-off that CONTRIBUTING.md requires. The change itself is unchanged. Re-run on this tree: python -m pytest tests/reference/test_speculative_workload.py passes 25/25, and the two new schema-instruction cases fail on current main without this change (2 failed, 23 passed).

Problem (demonstrated defect)

  • For response_format: json_schema, Helios prepare_request (helios/src/server/request.rs ~136-152) appends a
    system instruction containing the schema before rendering the chat template.
  • qualify/speculative_workload.template_input freezes prompt references without it. The strict analyzer's
    exact_prompt_ids / prompt_ids_first_event checks therefore fail on valid responses: one case expects 28 prompt
    tokens and the server uses 115.
  • Upstream's own workload builder emits json_schema cases (speculative_workload.py ~174-183), so its frozen
    references are affected too.
  • Earlier frontend checks rendered normalized messages only, so they never exercised full request processing.

Change

Apply the same instruction when constructing CPU prompt references:

  • verbatim json_schema.schema;
  • compact JSON, keys in wire order (matching serde_json preserve_order), raw UTF-8;
  • appended with "\n\n" to a leading string system message, or inserted as a new first system message.

Request bytes, seeds, budgets, expected semantics and analyzer checks are unchanged. Previously frozen references need
regeneration.

Validation

  • 25 CPU workload tests pass, including insert/append and wire immutability.
  • An isolated oracle calls Helios prepare_request on complete frozen payloads. Old references fail at the
    JSON-schema case. Corrected references match exact rendered text and token IDs for all 56 requests across 3
    datasets.
  • Normalized-template tool: 65/65.

Retested on this exact branch, based on 5f37bd1, with the full CPU suite run side by side against unmodified 5f37bd1 on the same host:

  • base: 4 failed, 3220 passed, 1666 skipped;
  • this branch: 4 failed, 3222 passed, 1666 skipped. The +2 are the new tests.
  • tests/reference: 341 → 343 passed.
  • The same 4 failures occur on base too and are environmental: a missing local Kimi-K3 fixture and two test_tuning_replicas tests.

Scope (stated in the commit)

json_schema on the Chat protocol with string-valued system content. These aren't covered:

  • non-integer numbers in schemas (Rust ryu vs Python repr formatting);
  • array-content system messages;
  • json_object on the legacy prompt path;
  • Responses/Messages schema locations;
  • mid-conversation system messages.

Disclosure

Written by an AI coding agent (Codex, GPT-6) with a staged design-review acknowledgement, and validated on my hardware (RTX PRO 4500 Blackwell, SM120, driver 615.78.08 open module, CUDA 13.4). Claude Code reviewed the storage-contract, CPU harness and reporting aspects and prepared this PR; it did not author or review kernel code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BJmfhfkwQM1xjdQVaUKwns

The authored JSON-schema HTTP cases freeze prompts before native request
processing adds its schema system instruction. On the dense Qwen native path,
json-incident therefore compares 28 expected tokens with 115 actual tokens,
even though the response is valid and the wire request is unchanged.

Apply the same schema instruction while constructing CPU prompt references,
including preserving an existing leading system message. Keep request bytes,
seeds, budgets, expected output semantics and strict stream checks unchanged.
Previously frozen references need regeneration; failed runs remain failures.

Validation: 25 CPU workload tests pass, including insertion/append and wire
immutability. An isolated CPU tool calls Helios prepare_request on complete
frozen payloads: old dense references fail at json-incident; corrected all-NVFP4,
official and thinking datasets pass exact rendered text and token IDs for all
56 requests. Only 4/4/1 schema references change; all wire bytes are retained.
The separate normalized-template tool also passes 65 fixture cases.

Scope: authored json_schema text cases and string-valued system instructions.
No serving, sampling, model arithmetic or runtime contract changes. This does
not establish statistical quality equivalence or complete MoE qualification.

Design-Signed-Off-By: Codex (GPT-6); design-review=6aeb
Signed-off-by: Maxrio9 <70799635+Maxrio9@users.noreply.github.com>
@original-el8 original-el8 added area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. bug Something isn't working priority:P2 Normal bug priority: workflow failure or diagnosis with narrower impact or an available workaround. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 5, 2026
@original-el8
original-el8 merged commit c67cb2a into NVIDIA:main Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. bug Something isn't working priority:P2 Normal bug priority: workflow failure or diagnosis with narrower impact or an available workaround. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants