feat(orchestrator): use the renderer client for evals#3017
Draft
faresobeid wants to merge 1 commit into
Draft
Conversation
Evals previously went through plain chat-completions, trusting the server's tool-call parser. vLLM 0.24's glm45 parser intermittently returns tool calls as raw text; the harness then ends the episode, and ~40% of our GLM-4.5-Air SWE-Bench eval rollouts were silently truncated (measured 24% vs ~42% actual after the switch). Train rollouts already parse through the renderer and were unaffected — this aligns evals with the same path, which also means evals measure the policy through the exact stack it trains under. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evals previously went through plain chat-completions clients, trusting the inference server's tool-call parser. That parser is a failure point: vLLM 0.24's
glm45parser intermittently returns tool calls as raw<tool_call>text (a regression vs 0.23), the harness sees "no tool call" and ends the episode — ~40% of our GLM-4.5-Air SWE-Bench Verified eval rollouts were silently truncated mid-work, depressing measured scores to 24% vs ~42% actual (verified by switching: first renderer-path evals read 42.3%, matching the pre-0.24 era's 31–40% trajectory).Train rollouts already parse through the renderer client and were unaffected — the asymmetry is what made this hard to notice: training healthy, evals mysteriously low.
This switches eval clients to the renderer as well (wiring
renderer_config/renderer_model_name/pool_sizethrough to the evalsetup_clientscall, which previously dropped them). Besides the robustness, evals now measure the policy through the exact parsing stack it trains under.Trade-off worth a reviewer's eye: evals no longer exercise the standard OpenAI-compatible serving path, so absolute numbers are no longer "as an API customer would see." For RL training campaigns we think train/eval consistency wins; a config knob could restore the old behavior if both are wanted.
🤖 Generated with Claude Code