Add lightweight tool-use verifier evaluations / 添加轻量级工具调用验证评估

[adibarra]

Summary

• Integrate the stock Kimi Vendor Verifier and MiniMax Provider Verifier with the existing InferenceX eval pipeline.
• Add pinned BFCL V4 smoke and model-quality suites without introducing a parallel eval framework.
• Reuse the existing eval dispatch, launcher, artifact upload, result collection, threshold validation, and dashboard paths.
• Add deterministic artifact cleanup, bounded retries and timeouts, single-node and multi-node staging, and normalized inferencex-eval-v1 results.
• Preserve the default GSM8K behavior. Kimi, MiniMax, and BFCL suites remain opt-in.

Validation

Pass rates

• Selected Kimi and MiniMax hardware runs on ancestor revisions: 7/7 passed, 100%.
• Selected post-integration Kimi and MiniMax runs listed below, including the DeepSeek V4 cross-model diagnostic: 7/8 passed, 87.5%.
• Local focused test suite: 549/549 passed, 100%.
• Kimi K3 full schema suite: 400/408 cases passed, 98.0%. This full suite is diagnostic and currently has a 0.0 threshold.
• MiniMax M3 full suite: 102/102 requests completed, 100%, with a final compatibility score of 96%.

Ancestor-revision hardware results

Hardware	Backend and model	Suite	Result
H100	SGLang, Qwen3.5	MiniMax smoke	1.0, passed
H200	vLLM, MiniMax M3	MiniMax full	102/102 completed, score 0.96, passed
B200	Dynamo vLLM, Kimi K3	Kimi full	400/408 cases, score 0.980392, diagnostic workflow passed
B300	vLLM, Kimi K3	MiniMax smoke	1.0, passed
GB200	Dynamo vLLM, Kimi K3	MiniMax smoke	1.0, passed
MI325X	vLLM, MiniMax M3	MiniMax smoke	1.0, passed
MI355X	vLLM, Kimi K3	MiniMax smoke	1.0, passed

These runs are on ancestor commits 7b6160322 or f020a8e27. The current PR head, b57d8bb679c332e30e408d79490e4f38a2093c12, has no GPU end-to-end run. Historical BFCL runs are excluded because bfcl_adapter.py and its shell dispatch were substantially rewritten afterward; the current BFCL path has no hardware proof.

All changed Python files compiled, all changed shell files passed bash -n, and a mock OpenAI-compatible endpoint exercised the MiniMax adapter end to end with score 1.0.

Known outcomes and remaining coverage

• The GB200 DeepSeek V4 diagnostic returned prose with finish_reason: stop instead of a tool call. The adapter correctly produced artifacts and failed the threshold with score 0.0 in run 32057478178. This is a model-behavior result, not an integration failure.
• The redundant GB300 DeepSeek V4 run was cancelled after 70 minutes in its multi-node launch stage and is excluded from pass-rate calculations.
• Kimi K3 produced a 98.0% full-suite score, but the suite currently has a diagnostic 0.0 threshold and eight JSON Schema edge cases remain model compatibility failures.
• ATOM and TensorRT-LLM launch plumbing is implemented but does not yet have end-to-end hardware validation in this PR.
• The current BFCL adapter, including both smoke and model-quality suites, remains opt-in with diagnostic full-suite thresholds and has not yet passed a post-rewrite hardware run.

Type of change

• New feature
• Documentation update
• Configuration change
• Bug fix

Checklist

• I have tested my changes locally.
• I have updated documentation.
• The performance changelog requirement is not applicable. This PR adds opt-in eval behavior and preserves default throughput benchmark configurations.
• A final PR sweep and /reuse-sweep-run authorization are pending.

中文说明

• 将 Kimi Vendor Verifier 和 MiniMax Provider Verifier 原生实现接入现有 InferenceX 评估流水线。
• 新增固定版本的 BFCL V4 冒烟测试和模型质量评估套件，不引入独立的平行评估框架。
• 复用现有评估分发、启动器、产物上传、结果汇总、阈值校验和看板链路。
• 新增确定性的历史产物清理、有界重试与超时、单节点及多节点产物暂存，并统一输出 inferencex-eval-v1 结果格式。
• 保持 GSM8K 默认行为不变。Kimi、MiniMax 和 BFCL 套件均需显式启用。

验证结果

通过率

• 在祖先提交上选取的 Kimi 和 MiniMax 硬件运行：7/7 通过，通过率 100%。
• 下表所列的 Kimi 和 MiniMax 集成后运行（计入 DeepSeek V4 跨模型诊断）：7/8 通过，通过率 87.5%。
• 本地聚焦测试：549/549 通过，通过率 100%。
• Kimi K3 完整 JSON Schema 套件：400/408 用例通过，通过率 98.0%。该完整套件目前仅用于诊断，阈值为 0.0。
• MiniMax M3 完整套件：102/102 请求完成，完成率 100%，最终兼容性得分为 96%。

祖先提交的硬件结果

硬件	后端与模型	套件	结果
H100	SGLang、Qwen3.5	MiniMax 冒烟测试	1.0，通过
H200	vLLM、MiniMax M3	MiniMax 完整套件	102/102 请求完成，得分 0.96，通过
B200	Dynamo vLLM、Kimi K3	Kimi 完整套件	400/408 用例通过，得分 0.980392，诊断工作流通过
B300	vLLM、Kimi K3	MiniMax 冒烟测试	1.0，通过
GB200	Dynamo vLLM、Kimi K3	MiniMax 冒烟测试	1.0，通过
MI325X	vLLM、MiniMax M3	MiniMax 冒烟测试	1.0，通过
MI355X	vLLM、Kimi K3	MiniMax 冒烟测试	1.0，通过

以上运行基于祖先提交 7b6160322 或 f020a8e27。当前 PR 头提交 b57d8bb679c332e30e408d79490e4f38a2093c12 尚未运行 GPU 端到端测试。历史 BFCL 运行未计入，因为之后 bfcl_adapter.py 及其 shell 分发路径进行了大幅重写；当前 BFCL 路径尚无硬件验证证据。

所有变更的 Python 文件均已通过编译检查，所有变更的 shell 文件均已通过 bash -n。此外，使用模拟 OpenAI 兼容端点对 MiniMax 适配器进行了端到端验证，得分为 1.0。

已知结果与待补充覆盖

• GB200 上的 DeepSeek V4 诊断运行返回了普通文本，finish_reason: stop，未触发工具调用。适配器正确生成产物，并在 run 32057478178 中以 0.0 分触发阈值失败。这是模型行为结果，不是集成故障。
• GB300 上重复的 DeepSeek V4 运行在多节点启动阶段持续 70 分钟后被取消，不计入通过率。
• Kimi K3 完整套件得分为 98.0%，但该套件目前使用仅用于诊断的 0.0 阈值，并且仍有八个 JSON Schema 边界用例属于模型兼容性失败。
• ATOM 和 TensorRT-LLM 的启动链路已接入，但本 PR 尚未完成对应的端到端硬件验证。
• 当前 BFCL 适配器的冒烟测试和模型质量套件仍需显式启用，完整套件使用诊断阈值，并且重写后尚未通过硬件运行。

变更类型

• 新功能
• 文档更新
• 配置变更
• 缺陷修复

检查清单

• 已完成本地测试。
• 已更新相关文档。
• 本 PR 仅新增显式启用的评估行为，并保持默认吞吐量基准测试配置不变，因此无需更新 perf-changelog.yaml。
• 最终 PR 扫描和 /reuse-sweep-run 授权尚未完成。