[nachiketb-nvidia]

Stack and Review Order

1. feat(libsy)!: route decision choices through explicit policies #1013: core ordered policy, required API/constructor migrations, and existing test fixtures.
2. This PR: standalone N-way runner model pools, deployment and benchmark TOML migrations, and documentation.

This PR targets nachiketb/feat-decision-cutoff-policy. Merge #1013 first, then rebase/retarget this PR onto main.

What and Why

Expose the core policy from #1013 through standalone N-way TOML routes and migrate the shipped examples. Every candidate becomes an available completion destination; decision judges no longer need a strong/weak pair. Ordinary LLM judges still require their pair, and composite/stage routes retain their two-tier boundary.

All auxiliary deployment/configuration documentation from #1013 is moved here. The assembled stack keeps the same routing implementation.

What Changed

• Runner target enumeration and runtime model pools include every standalone decision candidate. strong_target and weak_target are optional for those routes.
• dev-server/config.toml migrates its composite decision judge to policy.rules and an explicit fallback.
• benchmark/routing-profiles/tb21-jev-opus-glm.toml migrates the Jev task route. Its instructions are retained. The example 0.05 cutoff carries a recalibration warning; no new benchmark result is claimed.
• Server configuration notes, the composite routing guide, the N-way routing guide, and Python examples use candidate-only rules.
• The TOML reference distinguishes LLM and decision judge requirements and fallbacks, including composite behavior.

The TypeScript fixture migration is in #1013 because its existing tests must pass with the core API change. Other composite TOMLs use ordinary/custom LLM judges and do not need migration.

N-way Configuration

Assume completion targets strong, medium, and weak, and decision target judge, are already configured:

[routes.task]
id = "task"
type = "llm_classifier"
mode = "capability"
classifier_target = "judge"
classify_trigger = "new_session"
fail_open = false

[routes.task.decision]
candidates = { premium = "strong", balanced = "medium", fast = "weak" }
evidence = { description = "Add task-relevant candidate descriptions." }

[routes.task.decision.policy]
rules = [
  { candidate = "premium", cutoff = 0.6 },
  { candidate = "balanced", cutoff = 0.35 },
]
fallback = "fast"

if score[premium] > 0.60: select strong
else if score[balanced] > 0.35: select medium
else: select weak

Premium	Balanced	Fast	Completion target	Reason
0.65	0.25	0.10	strong	First rule matches.
0.45	0.40	0.15	medium	Second rule matches, despite premium having the largest score.
0.60	0.36	0.04	medium	Equality falls through.
0.60	0.35	0.05	weak	Neither score exceeds its cutoff.
0.61	0.36	0.03	strong	Declaration order wins.

Each rule checks and selects the same candidate. The fallback is used when no rule matches or the verdict is unavailable. Provider errors propagate with fail_open = false, or select the fallback when true. Every candidate needs a valid score, and all scores come from one judge call.

The same configuration shape supports more candidates. Evaluate thresholds on the intended workload. Candidate-choice probabilities are not calibrated task-success probabilities and do not preserve the old relative-advantage cutoff semantics.

What to Review

• Runner model pools and completion target enumeration include all candidates.
• The dev-server composite and Jev benchmark TOMLs use the required policy.
• The reference correctly documents optional tier fields, LLM-only settings, and explicit decision fallbacks.
• No libsy algorithm implementation or core public API changes are added by this PR.

Validation

• Existing default-feature Rust workspace tests and workspace Clippy with all targets and warnings denied pass on the complete stack.
• Native TypeScript binding build, typecheck, and all three existing integration tests pass.
• Existing Python decision/cancellation tests and policy/server smoke checks use the same core implementation.
• Strict MkDocs build and TOML migration audit pass.

No new test functions/files or dependencies. No paid provider calls. CI is not being awaited.

Diff Size

Layer	Added	Removed
Runner configuration/runtime model pools	57	29
Deployment and benchmark TOMLs	12	5
Documentation	119	22
Total	188	56

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-1014/
Built to branch gh-pages at 2026-10-19 15:54 UTC. Preview will be ready when the GitHub Pages deployment is complete.