diff --git a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md new file mode 100644 index 000000000..d45e38730 --- /dev/null +++ b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md @@ -0,0 +1,217 @@ +# SWE-Atlas escalation-router findings + +## Summary + +The escalation router could interpret duplicate command serialization in a NeMo Gym Terminus +trajectory as repeated failed work. A single assistant turn may contain the same command both as +raw JSON text and as a structured tool call. The judge prompt previously treated a command shown +two or more times as loop evidence without requiring those attempts to occur in distinct turns. + +This change makes escalation depend on fresh, consistent failure evidence across turns. It keeps +the existing one-way switch and latch: a session can move from the efficient model to the capable +model at most once and remains there afterward. + +On a deterministic 40-task paired sample (20 SWE-Atlas RF and 20 SWE-Atlas TW), the patched router +matched the two single-model arms at 15/40 correct and improved over the original router's 10/40. +It switched 5 times instead of 9. Under the synthetic token prices defined below, it cost +$0.889/task versus $1.171/task for the original router. GLM-only also scored 15/40 and remained the +least expensive arm at $0.501/task, so the result supports the patch over the original router but +does not show that routing beats the best fixed model on this sample. + +## Changes + +- Normalize only the transcript sent to the escalation judge. When a raw Terminus command batch + has a one-to-one match with structured tool calls in the same assistant turn, the redundant raw + representation is omitted. The worker model still receives the original history. +- Require the judge to classify positive evidence as `repetition`, `false_progress`, `drift`, + `desperation`, or `capability_gap`, and report whether the evidence is new. +- Advance the confirmation streak only for fresh positive evidence in the same category. A + category change starts a new streak; a decline or stale evidence resets it. An unavailable judge + preserves the prior streak rather than converting an infrastructure failure into a routing + decision. +- Clarify in the judge prompt that duplicate serialization within one turn and an immediate, + adaptive change of terminal strategy are not repeated failed attempts. +- Preserve the existing maximum-one-switch and post-switch latch behavior. + +## Focused SWE-Atlas reproducer observations + +The live checks used GLM 5.2 as the efficient model and Opus 4.8 as the capable model. The reward +column is the task's binary verifier result. "GLM only" means that the patched router did not +switch models during the trajectory. + +| Dataset | Task | Patched behavior | Reward | Observation | +| --- | --- | --- | ---: | --- | +| RF | `697e7458be1623d850a88838` | 28 GLM calls, then 36 Opus calls; one switch | 1 | Two fresh `repetition` findings confirmed the escalation path and latch. | +| RF | `69391d8d1ce51c407be1e531` | GLM only; 93 logged episodes | 0 | Findings changed from `repetition` to `false_progress`, so they did not form one confirmation streak. The earlier router switched without improving the reward. | +| RF | `694b4b99829f00e24fd11889` | GLM only; 34 logged episodes | 0 | No positive escalation finding. | +| TW | `6902ef3ab97fe23e2ad271f3` | GLM only; 58 logged episodes | 0 | One isolated `repetition` finding did not trigger a switch. The earlier router switched without improving the reward. | +| RF | `696719205599a51110d4b45f` | Three GLM-only runs: 72, 93, and 70 logged episodes | 0, 0, 0 | One isolated positive finding across the replicas; no latch. Direct GLM and direct Opus also scored 0. | +| RF | `696719205599a51110d4b455` | Three GLM-only runs: 51, 60, and 48 logged episodes | 0, 0, 0 | No transition in any replica. Direct GLM and direct Opus also scored 0. | + +A separate passing run of `697e7458be1623d850a88838` produced a positive finding followed by a +decline, stayed on GLM, and scored 1. This checks that non-consecutive findings do not accumulate. + +The positive switched task also scored 1 in a direct-GLM run. It therefore demonstrates correct +switching, confirmation, and latching, but does not establish a causal accuracy improvement. +Similarly, the unsuccessful controls show improved escalation precision and avoided Opus calls; +they do not show that the efficient model could solve those tasks. + +## Paired benchmark + +### Design + +The paired benchmark used the following fixed setup: + +- 20 tasks from `scale-ai/swe-atlas-rf@1` and 20 from `scale-ai/swe-atlas-tw@1`. + Tasks were selected deterministically by ranking the hash of + `20260904::`; they were not selected by outcome. +- Four arms per task: direct GLM, direct Opus, the original escalation router, and the patched + escalation router. This produced 160 task-arm runs. +- GLM was `nvidia/zai-org/glm-5.2`; Opus was + `aws/anthropic/bedrock-claude-opus-4-8` with medium effort. GLM also served as the escalation + judge. The two escalation arms used two confirmations, maximum one switch, and a post-switch + latch. +- The original router was built from upstream commit + `9a743e89223a0d5b14011f1226d5b068f730a3b8`; the patched router was built from + `507376b1d851b825378cf147d09a8b5594773298`. +- There was no agent-turn limit. Worker requests allowed up to 128,000 output tokens and a + 900-second model-server timeout. Five initial runs reached Harbor's separate 3,600-second agent + timeout. Only those five task-arm runs were repeated with a 3x agent-timeout multiplier; all five + retries completed and replaced the invalid attempts. The final effective matrix is 160/160 valid. +- Each table entry is one trial per task and arm. Binary verifier reward is reported as correct. + Because model sampling is stochastic, differences between unswitched GLM router runs and direct + GLM runs are not necessarily routing effects. + +### Accuracy and routing + +| Dataset | Arm | Correct | Accuracy | Switches | Correct after switch | +| --- | --- | ---: | ---: | ---: | ---: | +| RF (20) | Direct GLM | 7 | 0.350 | -- | -- | +| RF (20) | Always Opus | 9 | 0.450 | -- | -- | +| RF (20) | Original escalation | 6 | 0.300 | 7 | 2 | +| RF (20) | **Patched escalation** | **9** | **0.450** | **4** | **2** | +| TW (20) | Direct GLM | 8 | 0.400 | -- | -- | +| TW (20) | Always Opus | 6 | 0.300 | -- | -- | +| TW (20) | Original escalation | 4 | 0.200 | 2 | 0 | +| TW (20) | **Patched escalation** | **6** | **0.300** | **1** | **0** | +| Combined (40) | Direct GLM | 15 | 0.375 | -- | -- | +| Combined (40) | Always Opus | 15 | 0.375 | -- | -- | +| Combined (40) | Original escalation | 10 | 0.250 | 9 | 2 | +| Combined (40) | **Patched escalation** | **15** | **0.375** | **5** | **2** | + +Against the original router on the same 40 tasks, the patched router won 7 outcomes, lost 2, and +tied 31. Against direct GLM it won 5 and lost 5; against direct Opus it won 6 and lost 6. All 14 +switched trajectories across the two router arms had exactly one GLM-to-Opus transition and no +hand-back, confirming the switch and latch invariants in live execution. + +The patch traded earlier sensitivity for precision. Original-router switches occurred after a +median of 23 GLM calls; patched-router switches occurred after a median of 51. One patched RF retry +did not switch until 141 GLM calls, then made 57 latched Opus calls and still scored 0. Direct Opus +and the original router also scored 0 on that task. This avoided a premature switch but exposed a +cost problem: judging every turn can be expensive when the destination model is unlikely to help. + +### Synthetic cost calculation + +The inference service did not provide billable cost. The values below are synthetic estimates for +comparing routing behavior, not NVIDIA prices or charges. The assumed rates are: + +| Model | Uncached input / 1M tokens | Cached input / 1M tokens | Output / 1M tokens | +| --- | ---: | ---: | ---: | +| GLM 5.2 | $0.50 | $0.05 | $2.00 | +| Opus 4.8 | $5.00 | $0.50 | $25.00 | + +For each model in each trajectory: + +```text +uncached_input_tokens = max(prompt_tokens - cached_tokens, 0) + +model_cost = ( + uncached_input_tokens * uncached_input_rate + + cached_tokens * cached_input_rate + + output_tokens * output_rate +) / 1,000,000 + +trajectory_cost = sum(worker_model_costs) + sum(escalation_judge_costs) +``` + +Cache-creation tokens are part of the non-cached prompt-token remainder and therefore receive the +uncached-input rate. The calculation includes GLM judge calls for the two router arms. It excludes +the Harbor verifier, cluster resources, and other infrastructure because those calls are outside +Switchyard's per-trajectory statistics. + +| Dataset | Arm | Synthetic cost | Cost/task | Cost/correct | +| --- | --- | ---: | ---: | ---: | +| RF (20) | Direct GLM | $12.953 | $0.648 | $1.850 | +| RF (20) | Always Opus | $32.196 | $1.610 | $3.577 | +| RF (20) | Original escalation | $35.169 | $1.758 | $5.862 | +| RF (20) | **Patched escalation** | **$27.200** | **$1.360** | **$3.022** | +| TW (20) | Direct GLM | $7.093 | $0.355 | $0.887 | +| TW (20) | Always Opus | $14.829 | $0.741 | $2.471 | +| TW (20) | Original escalation | $11.657 | $0.583 | $2.914 | +| TW (20) | **Patched escalation** | **$8.356** | **$0.418** | **$1.393** | +| Combined (40) | Direct GLM | $20.045 | $0.501 | $1.336 | +| Combined (40) | Always Opus | $47.025 | $1.176 | $3.135 | +| Combined (40) | Original escalation | $46.826 | $1.171 | $4.683 | +| Combined (40) | **Patched escalation** | **$35.556** | **$0.889** | **$2.370** | + +The patched router was 24.1% cheaper than the original router and produced five additional correct +outcomes. It was 24.4% cheaper than always-Opus at the same aggregate accuracy. It was 77.4% more +expensive than direct GLM at the same aggregate accuracy. Of the patched router's $35.556 synthetic +total, $3.698 (10.4%) came from judge calls; this motivates reducing judge cadence after repeated +negative verdicts. + +### TW model-order reversal + +TW demonstrates a limitation that this patch does not solve: the escalation route assumes its +configured Opus target has positive expected gain over GLM. On this sample, direct GLM scored 8/20 +while direct Opus scored 6/20, so that assumption is not valid at the workload level. + +The patched router switched on only one TW task. Direct GLM, direct Opus, and both router arms all +scored 0 on that task, so the switch added cost without changing its outcome. The patched router's +two-correct deficit versus direct GLM was not caused by switching: it lost three independently +sampled GLM-only outcomes and gained one different GLM-only outcome. The original router provides +a clearer harmful-switch example on task `6902ef3ab97fe23e2ad2727e`: direct GLM scored 1, direct Opus +scored 0, and the original router switched and scored 0. The patch avoided that switch, although its +independent GLM-only run also scored 0. + +The next routing change should therefore be separate from the trajectory-evidence fix: gate +escalation on a workload- or task-conditioned estimate of expected accuracy gain and cost. When +recent paired evidence says the nominal strong model is worse, the route should disable or reverse +that transition rather than asking only whether the current trajectory looks stuck. This requires +held-out calibration or online exploration; the router cannot infer relative model quality from one +model's failing trajectory alone. + +## Validation evidence + +The implementation was validated with Rust 1.96.1 on a Slurm compute node using: + +```text +cargo fmt --all --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test --workspace --all-features +uv run ruff check . +uv run mypy switchyard +uv run maturin develop --uv +uv run pytest tests/ -v +``` + +The validation completed successfully: the Rust workspace passed, including 289 `libsy` tests; +ruff and mypy passed; the native wheel built and installed; and pytest reported 115 passed, +2 deselected, and 2 subtests passed. +Focused regression coverage includes exact and multi-command transcript normalization, partial and +multiplicity mismatches, category changes, stale evidence, unavailable-judge behavior, and the +one-way latch. + +No live provider calls are made by the repository validation commands. The live SWE-Atlas checks +described above were separate, explicitly configured benchmark runs. + +## Conclusion + +The patch fixes the concrete duplicate-serialization false positive and materially improves the +tested router: it matches the single-model aggregate accuracy, gains 5 correct outcomes over the +original router, cuts switches from 9 to 5, and reduces synthetic cost by 24.1%. It does not make +the router the best cost/accuracy arm; direct GLM matches its aggregate accuracy at substantially +lower cost. The remaining work is not to loosen the new evidence rule globally. It is to add +model-order calibration for workloads such as TW and reduce judge overhead or stop judging when +escalation has low expected value. Replicated trials on held-out tasks are needed before treating +the point estimates as stable production gains. diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rs index 593c22925..fa8a6ca86 100644 --- a/crates/libsy/src/algorithms/escalation.rs +++ b/crates/libsy/src/algorithms/escalation.rs @@ -12,7 +12,9 @@ use switchyard_protocol::{ use super::util::classifier_contract::ClassifierContractConfig; use super::util::decisive; -use super::util::escalation::{self, EscalationJudge, EscalationJudgeConfig, EscalationPolicy}; +use super::util::escalation::{ + self, EscalationCategory, EscalationJudge, EscalationJudgeConfig, EscalationPolicy, +}; use super::util::llm_judge::JudgeClassifier; use crate::core::algorithm::Driver; use crate::core::classifier::{Classification, Classifier}; @@ -21,6 +23,8 @@ use crate::{LibsyError, Result}; /// Session-state key holding the consecutive-escalate streak. const STREAK_KEY: &str = "escalation_streak"; +/// Session-state key holding the category currently being confirmed. +const CATEGORY_KEY: &str = "escalation_category"; fn streak(state: &State) -> u32 { match state.extra.get(STREAK_KEY) { @@ -29,6 +33,27 @@ fn streak(state: &State) -> u32 { } } +fn category(state: &State) -> Option<&str> { + match state.extra.get(CATEGORY_KEY) { + Some(StateValue::String(category)) => Some(category), + _ => None, + } +} + +fn bounded_reason(reason: &str) -> String { + reason + .chars() + .map(|character| { + if character.is_control() { + ' ' + } else { + character + } + }) + .take(240) + .collect() +} + fn assistant_message(response: &AggLlmResponse) -> Message { Message { role: Role::Assistant, @@ -140,21 +165,46 @@ impl Classifier for EscalationClassifier { metadata: efficient_response.metadata, }; - let (classification, _) = self - .judge - .score(state, &mut judge_request, Some(driver)) - .await?; + let verdict = self.judge.verdict(state, &judge_request, driver).await; let held = streak(state); - let best = classification.argmax(false)?; - let (escalate, pending) = match &best { - Some(score) if score.target == self.capable => (true, held + 1), - Some(_) => (false, 0), - None => (false, held), + let held_category = category(state).map(str::to_string); + let (escalate, pending, pending_category) = match verdict.as_ref() { + Some(verdict) => { + let category = verdict.category.label(); + tracing::info!( + escalate = verdict.escalate, + category, + new_evidence = verdict.new_evidence, + reason = %bounded_reason(&verdict.reason), + "escalation judge verdict" + ); + if verdict.escalate + && verdict.new_evidence + && verdict.category != EscalationCategory::None + { + let next = if held_category.as_deref() == Some(category) { + held + 1 + } else { + 1 + }; + (true, next, Some(category.to_string())) + } else { + (false, 0, None) + } + } + None => (false, held, held_category), }; state .extra .insert(STREAK_KEY.to_string(), StateValue::Count(pending)); + if let Some(category) = pending_category { + state + .extra + .insert(CATEGORY_KEY.to_string(), StateValue::String(category)); + } else { + state.extra.remove(CATEGORY_KEY); + } if escalate && pending >= self.confirmations { // Streak confirmed: drop the efficient response, caller will serve capable. @@ -249,8 +299,8 @@ mod tests { } } - /// Builds a router with escalation enabled (`confirmations=1` latches immediately). - fn escalation_router() -> Result> { + /// Builds a router with escalation enabled. + fn escalation_router_with_confirmations(confirmations: u32) -> Result> { Ok(Arc::new(LlmTaskClassifier::new( LlmClassifierConfig::Escalation { judge_target: ModelId::from("judge"), @@ -258,7 +308,7 @@ mod tests { capable_target: ModelId::from("capable"), contract: ClassifierContractConfig::default(), config: EscalationJudgeConfig { - confirmations: 1, + confirmations, ..EscalationJudgeConfig::default() }, max_output_tokens: DEFAULT_JUDGE_MAX_OUTPUT_TOKENS, @@ -266,9 +316,16 @@ mod tests { )?)) } + /// Builds a router that latches on its first supported escalation verdict. + fn escalation_router() -> Result> { + escalation_router_with_confirmations(1) + } + #[tokio::test] async fn serves_efficient_when_judge_declines() -> Result<()> { - let judge = Queue::new([r#"{"escalate":false,"reason":"progressing"}"#]); + let judge = Queue::new([ + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#, + ]); let model = Queue::new(["efficient answer"]); let (selected_model, response) = test_drive( @@ -303,7 +360,9 @@ mod tests { }) }); recorded.lock().extend(prompt); - std::future::ready(Ok(reply(r#"{"escalate":false,"reason":"progressing"}"#))) + std::future::ready(Ok(reply( + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#, + ))) } else { std::future::ready(Ok(reply("efficient answer"))) } @@ -328,7 +387,9 @@ mod tests { #[tokio::test] async fn upgrades_to_capable_when_judge_escalates() -> Result<()> { - let judge = Queue::new([r#"{"escalate":true,"reason":"stuck in a loop"}"#]); + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck in a loop"}"#, + ]); let model = Queue::new(["efficient draft", "capable answer"]); let (selected_model, response) = test_drive( @@ -346,9 +407,96 @@ mod tests { Ok(()) } + #[tokio::test] + async fn confirmation_streak_requires_the_same_category() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"off task"}"#, + r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"still off task"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + let (first, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (second, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (third, _) = test_drive(router, request, queued(model, judge)).await?; + + assert_eq!(first, "efficient"); + assert_eq!(second, "efficient"); + assert_eq!(third, "capable"); + Ok(()) + } + + #[tokio::test] + async fn verdict_without_new_evidence_resets_the_streak() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + r#"{"escalate":true,"category":"repetition","new_evidence":false,"reason":"only old evidence remains"}"#, + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"new repeated command"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + for _ in 0..3 { + let (selected, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + assert_eq!(selected, "efficient"); + } + Ok(()) + } + + #[tokio::test] + async fn unavailable_judge_preserves_the_category_streak() -> Result<()> { + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#, + "not json", + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"another repeated command"}"#, + ]); + let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]); + let router = escalation_router_with_confirmations(2)?; + let request = classify_session_request(); + + let (first, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (second, _) = test_drive( + router.clone(), + request.clone(), + queued(Arc::clone(&model), Arc::clone(&judge)), + ) + .await?; + let (third, _) = test_drive(router, request, queued(model, judge)).await?; + + assert_eq!(first, "efficient"); + assert_eq!(second, "efficient"); + assert_eq!(third, "capable"); + Ok(()) + } + #[tokio::test] async fn stays_capable_after_latch() -> Result<()> { - let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]); + let judge = Queue::new([ + r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#, + ]); let model = Queue::new(["efficient draft", "capable t1", "capable t2"]); let router = escalation_router()?; let request = classify_session_request(); diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rs index 54ded690c..cd0b4c134 100644 --- a/crates/libsy/src/algorithms/util/escalation.rs +++ b/crates/libsy/src/algorithms/util/escalation.rs @@ -46,8 +46,9 @@ const MAX_REQUEST_CHARS: usize = 18_000; #[derive(Clone, Debug, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct EscalationJudgeConfig { - /// Consecutive escalate verdicts required before a turn moves to the capable tier, which - /// is also the turn that latches the session. Any decline clears the streak. + /// Consecutive fresh-evidence verdicts in the same category required before a turn moves to + /// the capable tier, which is also the turn that latches the session. Any decline or stale + /// evidence clears the streak. /// `1` escalates on the first verdict; the router's main cost dial. /// `2` or higher needs a session id, since the streak is retained per session. pub confirmations: u32, @@ -87,12 +88,39 @@ impl Default for EscalationJudgeConfig { } } -/// The judge's verdict. The schema also requires a `reason`, which makes the judge state its -/// case and measurably sharpens the verdict — routing reads only the boolean, so it is -/// deserialized away rather than carried. +/// Bounded trouble pattern used to correlate escalation confirmations across turns. +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "snake_case")] +pub(crate) enum EscalationCategory { + None, + Repetition, + FalseProgress, + Drift, + Desperation, + CapabilityGap, +} + +impl EscalationCategory { + /// Stable state and telemetry label. + pub(crate) const fn label(self) -> &'static str { + match self { + Self::None => "none", + Self::Repetition => "repetition", + Self::FalseProgress => "false_progress", + Self::Drift => "drift", + Self::Desperation => "desperation", + Self::CapabilityGap => "capability_gap", + } + } +} + +/// The judge's typed verdict, including the evidence needed to confirm a stable pattern. #[derive(Deserialize)] pub(crate) struct EscalationVerdict { - escalate: bool, + pub(crate) escalate: bool, + pub(crate) category: EscalationCategory, + pub(crate) new_evidence: bool, + pub(crate) reason: String, } /// Builds the condensed trajectory presented to the escalation judge. @@ -190,21 +218,89 @@ pub(crate) fn conversation_turn(request: &Request) -> usize { /// relies on. fn message_text(message: &Message) -> String { let mut parts = Vec::new(); - collect_text(&message.content, &mut parts); + let terminus_commands = if message.role == Role::Assistant { + message + .content + .iter() + .filter_map(|block| match block { + ContentBlock::ToolCall(call) if call.name == "bash_command" => call + .arguments + .get("keystrokes") + .and_then(|value| value.as_str()), + _ => None, + }) + .collect::>() + } else { + Vec::new() + }; + collect_text(&message.content, &mut parts, &terminus_commands); parts.join(" ") } +/// Removes a Terminus command batch when a structured bash call carries the same action. +/// +/// The model-facing request remains untouched. Only the judge's plain-text view is normalized, +/// so one action cannot look like two attempts while the agent still sees its native history. +fn without_duplicated_terminus_commands(text: &str, tool_commands: &[&str]) -> String { + for (start, character) in text.char_indices() { + if character != '{' { + continue; + } + + let mut values = + serde_json::Deserializer::from_str(&text[start..]).into_iter::(); + let Some(Ok(mut value)) = values.next() else { + continue; + }; + let end = start + values.byte_offset(); + let Some(commands) = value + .get("commands") + .and_then(|commands| commands.as_array()) + else { + continue; + }; + let Some(command_batch) = commands + .iter() + .map(|command| command.get("keystrokes").and_then(|value| value.as_str())) + .collect::>>() + else { + continue; + }; + let mut unmatched_tool_commands = tool_commands.to_vec(); + let fully_encoded = command_batch.iter().all(|command| { + let Some(index) = unmatched_tool_commands + .iter() + .position(|candidate| candidate == command) + else { + return false; + }; + unmatched_tool_commands.swap_remove(index); + true + }); + if command_batch.is_empty() || !fully_encoded { + continue; + } + + value["commands"] = serde_json::Value::Array(Vec::new()); + let Ok(normalized) = serde_json::to_string(&value) else { + continue; + }; + return format!("{}{}{}", &text[..start], normalized, &text[end..]); + } + text.to_string() +} + /// Appends the judge-relevant text of each block, descending into tool results. -fn collect_text(content: &[ContentBlock], parts: &mut Vec) { +fn collect_text(content: &[ContentBlock], parts: &mut Vec, tool_commands: &[&str]) { for block in content { match block { ContentBlock::Text { text } | ContentBlock::Refusal { text } => { - parts.push(text.clone()); + parts.push(without_duplicated_terminus_commands(text, tool_commands)); } ContentBlock::ToolCall(call) => { parts.push(format!("tool_call {}({})", call.name, call.arguments)); } - ContentBlock::ToolResult(result) => collect_text(&result.content, parts), + ContentBlock::ToolResult(result) => collect_text(&result.content, parts, &[]), _ => {} } } @@ -449,6 +545,147 @@ mod tests { assert_eq!(message_text(&result), "no such file"); } + #[test] + fn message_text_deduplicates_terminus_commands_for_the_judge() { + let first_command = "grep -n bug app.py\n"; + let second_command = "sed -n '1,80p' app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: format!( + "Before\n```json\n{}\n```\nAfter", + json!({ + "analysis": "inspect the reported file", + "commands": [ + {"keystrokes": first_command, "duration": 0.1}, + {"keystrokes": second_command, "duration": 0.1}, + ], + }) + ), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": first_command, "duration": 0.1}), + }), + ContentBlock::ToolCall(ToolCall { + id: "call-2".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": second_command, "duration": 0.1}), + }), + ], + }; + + let text = message_text(&message); + + assert!(text.contains("inspect the reported file"), "{text}"); + assert!(text.contains("Before"), "{text}"); + assert!(text.contains("After"), "{text}"); + assert!(text.contains(r#""commands":[]"#), "{text}"); + assert_eq!(text.matches("grep -n bug app.py").count(), 1, "{text}"); + assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}"); + assert_eq!(text.matches("tool_call bash_command(").count(), 2, "{text}"); + } + + #[test] + fn message_text_keeps_terminus_commands_without_matching_tool_call() { + let command = "grep -n bug app.py\n"; + let text = json!({ + "analysis": "inspect the reported file", + "commands": [{"keystrokes": command}], + }) + .to_string(); + let without_tool_call = Message { + role: Role::Assistant, + content: vec![ContentBlock::Text { text: text.clone() }], + }; + let mismatched_tool_call = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { text }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": "sed -n '1,20p' app.py\n", "duration": 0.1}), + }), + ], + }; + + assert_eq!( + message_text(&without_tool_call) + .matches("grep -n bug app.py") + .count(), + 1 + ); + assert_eq!( + message_text(&mismatched_tool_call) + .matches("grep -n bug app.py") + .count(), + 1 + ); + } + + #[test] + fn message_text_keeps_a_partially_encoded_terminus_batch() { + let first_command = "grep -n bug app.py\n"; + let second_command = "sed -n '1,80p' app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: json!({ + "commands": [ + {"keystrokes": first_command}, + {"keystrokes": second_command}, + ], + }) + .to_string(), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": first_command}), + }), + ], + }; + + let text = message_text(&message); + + assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}"); + assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}"); + assert!(!text.contains(r#""commands":[]"#), "{text}"); + } + + #[test] + fn message_text_keeps_duplicate_commands_without_one_tool_call_each() { + let command = "grep -n bug app.py\n"; + let message = Message { + role: Role::Assistant, + content: vec![ + ContentBlock::Text { + text: json!({ + "commands": [ + {"keystrokes": command}, + {"keystrokes": command}, + ], + }) + .to_string(), + }, + ContentBlock::ToolCall(ToolCall { + id: "call-1".to_string(), + name: "bash_command".to_string(), + arguments: json!({"keystrokes": command}), + }), + ], + }; + + let text = message_text(&message); + + assert_eq!(text.matches("grep -n bug app.py").count(), 3, "{text}"); + assert!(!text.contains(r#""commands":[]"#), "{text}"); + } + #[test] fn truncate_middle_keeps_head_and_tail() { let text = "a".repeat(40) + &"z".repeat(40); diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rs index 0210bda27..25b01921b 100644 --- a/crates/libsy/src/algorithms/util/llm_judge.rs +++ b/crates/libsy/src/algorithms/util/llm_judge.rs @@ -230,7 +230,7 @@ where /// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's /// fallback branch. A closed driver stream is folded too; the algorithm's next driver /// call surfaces it, so nothing is masked. - async fn verdict( + pub(crate) async fn verdict( &self, state: &mut State, request: &Request, diff --git a/crates/libsy/src/prompts/escalation/prompt.md b/crates/libsy/src/prompts/escalation/prompt.md index 6d57369bb..d29a4017f 100644 --- a/crates/libsy/src/prompts/escalation/prompt.md +++ b/crates/libsy/src/prompts/escalation/prompt.md @@ -9,11 +9,20 @@ messages and tool results). Judge the *trajectory* — is the agent making real progress toward the stated task — not the difficulty of the task itself. Return exactly one JSON object: -{"escalate": boolean, "reason": "one short sentence naming the pattern"} +{"escalate": boolean, "category": "none|repetition|false_progress|drift|desperation|capability_gap", "new_evidence": boolean, "reason": "one short sentence naming the pattern"} + +Use `category: "none"` whenever `escalate` is false. `new_evidence` +is true only when the NEWEST assistant turn or its resulting tool output +adds evidence for the named pattern. Older turns may establish context, +but do not repeat an escalation vote solely because old trouble remains +visible in the rolling transcript. If the newest turn recovered, adapted, +or made progress, return `escalate: false`, `category: "none"`, and +`new_evidence: false`. Escalation is one-way for the rest of the task and expensive. Escalate only on a clear PATTERN of trouble, never on a single failed command. -When the evidence is thin or ambiguous, return {"escalate": false}. +When the evidence is thin or ambiguous, decline with `category: "none"` +and `new_evidence: false`. The bar is not "is there friction" — agentic coding is full of friction the efficient tier works through on its own. The bar is "is this run @@ -58,8 +67,10 @@ Hold weak — no model can fix these, so escalation is pure waste: # Trouble patterns — escalate when you see these Repetition and loops (the most common way agent runs die): -- The same command or edit failing 2+ times with materially the same - error, especially with unrelated changes in between. +- The same command or edit failing across 2+ DISTINCT assistant turns + with materially the same error, especially with unrelated changes in + between. Count executed attempts across turns, not repeated renderings + inside one message. - Near-identical tool calls repeated, or the same files re-read, without new information gained — including longer cycles (A -> B -> C -> A). - Fighting the environment: repeatedly invoking a missing executable, @@ -105,6 +116,15 @@ Desperation: # Expected friction — do NOT escalate on these Agentic coding is full of failures that are part of healthy work: +- A command appearing both in the assistant's JSON/text and as one or + more structured tool-call blocks in that SAME turn. Agent harnesses + commonly serialize one intended action more than once; this is one + attempt, not a loop. Repetition is evidence only when separate turns + show separate executions with materially the same failed result. +- Terminal-input serialization trouble (for example tabs triggering + completion or a heredoc being mangled) when the next turn changes the + write mechanism, quoting, or transport. That is adaptation, not + repeated failure. - A test written to fail first (TDD) or a bug being reproduced on purpose. - A compile, lint, or test error fixed or meaningfully acted on in the @@ -130,49 +150,56 @@ Agentic coding is full of failures that are part of healthy work: - A long-running command (build, install, test suite) that simply has not finished, or the agent waiting on information it asked for. -The distinguishing question: is each failure producing new information -that changes the next action? Failing forward is fine; failing in place -is trouble. Also weigh the session's own recovery record: if this same -session already shows friction the agent subsequently cleared (a failure -followed by a verified fix or passing check), lean toward holding — a -session that has recovered before will usually recover again. +The distinguishing question: across DISTINCT assistant turns, is each +failure producing new information that changes the next action? Failing +forward is fine; failing in place is trouble. Never infer a multi-turn +pattern from duplicated representations inside one turn. Also weigh the +session's own recovery record: if this same session already shows +friction the agent subsequently cleared (a failure followed by a verified +fix or passing check), lean toward holding — a session that has recovered +before will usually recover again. # Worked examples (none drawn from any benchmark task set) * Turn 3; the agent ran the test suite, 4 tests fail, and it is now - reading the first failing test. -> {"escalate": false} — reproducing - failures is the job. + reading the first failing test. -> {"escalate": false, "category": + "none", "new_evidence": false, "reason": "working through the first + reproduced failure"} * The agent has run `pytest tests/test_api.py` 4 times with the same ImportError, editing an unrelated config file between attempts. -> - {"escalate": true, "reason": "same ImportError 4 times while editing - unrelated files"} + {"escalate": true, "category": "repetition", "new_evidence": true, + "reason": "same ImportError 4 times while editing unrelated files"} * `conda` is not installed; the agent has tried `conda install` five ways instead of using the `pip` that earlier output showed present. - -> {"escalate": true, "reason": "fighting missing executable instead - of adapting"} + -> {"escalate": true, "category": "repetition", "new_evidence": + true, "reason": "fighting missing executable instead of adapting"} * Task: "make the provided integration tests pass." Recent turns: renaming variables and reformatting docstrings; tests not run in 8 - turns. -> {"escalate": true, "reason": "drifted to cosmetic edits, - verification abandoned"} + turns. -> {"escalate": true, "category": "drift", "new_evidence": + true, "reason": "drifted to cosmetic edits, verification abandoned"} * The agent says "All tests pass, task complete" but the last visible test output shows "2 failed, 11 passed". -> {"escalate": true, - "reason": "claims success contradicted by latest test output"} + "category": "false_progress", "new_evidence": true, "reason": + "claims success contradicted by latest test output"} * The agent wrote a reproduction script that exits 0 without invoking the code path the issue describes, concluded "bug not reproducible", - and is wrapping up. -> {"escalate": true, "reason": "reproduction - never exercised the reported code path"} + and is wrapping up. -> {"escalate": true, "category": + "false_progress", "new_evidence": true, "reason": "reproduction never + exercised the reported code path"} * Two turns of edits, one failed build, then a fixed build and a - passing test. -> {"escalate": false} + passing test. -> {"escalate": false, "category": "none", + "new_evidence": false, "reason": "latest build recovered and passed"} * `npm install` has been running for one turn with no output yet. -> - {"escalate": false} — slow command, not a stall. + {"escalate": false, "category": "none", "new_evidence": false, + "reason": "command is still running"} * Four different serialization libraries failed to import; the agent is now writing the converter with a fifth approach it has not tried - before. -> {"escalate": false} — sequential alternatives are - adaptation, even when none has succeeded yet. + before. -> {"escalate": false, "category": "none", "new_evidence": + false, "reason": "latest turn changed approach"} * Task: tune a slow batch pipeline. The agent is investigating why the message broker fails to start, since the pipeline cannot be - measured without it. -> {"escalate": false} — unblocking - verification serves the task. + measured without it. -> {"escalate": false, "category": "none", + "new_evidence": false, "reason": "working to unblock verification"} Do not emit markdown, commentary, or chain-of-thought — only the JSON object. diff --git a/crates/libsy/src/prompts/escalation/schema.json b/crates/libsy/src/prompts/escalation/schema.json index 977c41882..e98b3c5fd 100644 --- a/crates/libsy/src/prompts/escalation/schema.json +++ b/crates/libsy/src/prompts/escalation/schema.json @@ -10,12 +10,21 @@ "type": "boolean", "description": "True when the run is likely doomed without escalation to the strong tier." }, + "category": { + "type": "string", + "enum": ["none", "repetition", "false_progress", "drift", "desperation", "capability_gap"], + "description": "The single trouble pattern supporting escalation, or none when escalation is false." + }, + "new_evidence": { + "type": "boolean", + "description": "True only when the newest assistant turn or its resulting tool output adds evidence for the named trouble pattern." + }, "reason": { "type": "string", "description": "One short sentence naming the trouble pattern, or stating why the run is progressing." } }, - "required": ["escalate", "reason"], + "required": ["escalate", "category", "new_evidence", "reason"], "additionalProperties": false } } diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs index 43e6abf7f..f4cc551e1 100644 --- a/crates/switchyard-server/tests/server.rs +++ b/crates/switchyard-server/tests/server.rs @@ -296,7 +296,7 @@ async fn upstream_chat( .pointer("/response_format/json_schema/schema/properties/escalate") .is_some() { - r#"{"escalate":false,"reason":"making progress"}"# + r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"making progress"}"# } else if model == "model/classifier" && requests_schema_invalid_verdict { r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.1,"unexpected":true}"# } else if model == "model/classifier" { diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md index 0193ca680..48f66ea8b 100644 --- a/docs/reference/toml_schema.md +++ b/docs/reference/toml_schema.md @@ -199,7 +199,7 @@ Escalation mode serves the weak target first and judges the completed turn. See | `strong_target` | Yes | — | Target used after the session latches. | | `weak_target` | Yes | — | Target served before the latch. | | `prompt` | No | packaged prompt | Replaces the trajectory-judge prompt. | -| `escalation.confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a session ID. | +| `escalation.confirmations` | No | `2` | Consecutive fresh-evidence verdicts for the same failure category required to latch. Above `1` needs a session ID. | | `escalation.recent_turn_window` | No | `28` | Trailing messages shown to the judge. | | `escalation.window_message_chars` | No | `500` | Per-message cap inside that window. | diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md index 1ed9e8085..acbee4219 100644 --- a/docs/routing_algorithms/escalation_router_routing.md +++ b/docs/routing_algorithms/escalation_router_routing.md @@ -50,7 +50,9 @@ The route-level `prompt` key replaces the packaged trajectory-judge prompt. It uses the escalation verdict schema rather than the capability verdict schema. Switchyard supplies that schema according to the route's `response_format_type`: through the structured-output request in the default `json_schema` mode, or in -the prompt in `json_object` mode. +the prompt in `json_object` mode. The verdict includes an `escalate` decision, a +bounded failure `category`, whether the newest turn adds `new_evidence`, and a +short `reason`. ## How the decision works @@ -60,8 +62,10 @@ For each turn on an unlatched session, Switchyard: 2. Appends that reply to the transcript and asks the judge to rule on the completed turn. The judge therefore rates work the weak model actually did, not a prediction about work it might do. -3. Increments a consecutive-escalate streak on an escalate verdict, and resets it - to zero on a decline. +3. Increments a confirmation streak only when an escalate verdict cites fresh + evidence for the same failure category as the preceding vote. A different + category starts a new streak at one; a decline or stale evidence resets it to + zero. 4. Serves the buffered weak reply when the streak has not yet reached `confirmations` — so a judged turn that does not escalate costs one weak call plus one judge call, and no strong call. @@ -103,7 +107,7 @@ configuration, so a bare `escalation = {}` is a valid, tuned route: | Key | Default | Meaning | |---|---|---| -| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. | +| `confirmations` | `2` | Consecutive fresh-evidence verdicts for the same failure category required before the session latches to strong. Must be at least `1`. | | `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. | | `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. | @@ -113,8 +117,8 @@ retained per session — without one, every turn starts from zero and the route never latches. Clients supply it with `x-switchyard-session-id`. Anchor and transcript caps remain fixed. Set the route-level -`max_output_tokens` key to change the judge's reply budget. Any decline still -resets the streak to zero. +`max_output_tokens` key to change the judge's reply budget. Any decline or +verdict without new evidence resets the streak to zero. ## Run the route @@ -159,6 +163,10 @@ per-session routing stats under the judge's model id, tagged with the `classifier` tier — so per-session token accounting includes judge overhead alongside the tiers the session was served by. +The server log records each parsed escalation verdict's category, +`new_evidence` flag, and bounded reason so false-positive or missed escalation +decisions can be diagnosed without retaining unbounded judge output. + ## When not to use escalation routing - **One-shot requests.** No trajectory to judge. Use