Issue 069: Switchyard loses Responses refusal semantics

README.md

@@ -26,12 +26,12 @@ offline with no provider keys.
 <!-- kairo-counts:start -->
 | Metric | Value |
 |---|---|
-| Reproduced issue folders | 46 |
+| Reproduced issue folders | 47 |
 | Gateways under test | LiteLLM, NVIDIA Switchyard, Bifrost, GoModel, AxonHub, and any-llm |
-| Harness tests | 124 (110 conformance checks against recorded transcripts, 14 unit) |
+| Harness tests | 130 (114 conformance checks against recorded transcripts, 16 unit) |
 <!-- kairo-counts:end -->
 
-The 46 folders cover 50 distinct defects confirmed on the wire, 49 of them on
+The 47 folders cover 51 distinct defects confirmed on the wire, 50 of them on
 current releases at the time of capture. The full ledger, including the cited
 bugs that did not reproduce, is [`issues/SCOREBOARD.md`](issues/SCOREBOARD.md).
 One finding is filed upstream as
@@ -49,7 +49,7 @@ reproduction commands.
 |---|---|---|---|---|---|---|
 | Terminal reason survives translation (`tool_use`, `content_filter`, `max_tokens`, refusal) | [001](issues/001-anthropic-stream-toolcall-translation), [002](issues/002-litellm-ollama-toolcall-loss) | [010](issues/010-switchyard-content-filter-and-reorder) | [030](issues/030-bifrost-anthropic-stream-stop-reason), [034](issues/034-bifrost-erases-content-filter), [035](issues/035-bifrost-erases-truncation), [036](issues/036-bifrost-drops-refusal-content) | | | |
 | Request constraints survive (`disable_parallel_tool_use`, `stop_sequences`, `tools[].strict`, `output_format`) | [017](issues/017-parallel-tool-flag-dropped), [041](issues/041-litellm-drops-stop-sequences), [064](issues/064-litellm-drops-tool-strict) | [006](issues/006-switchyard-crossformat-losses), [017](issues/017-parallel-tool-flag-dropped), [040](issues/040-switchyard-drops-output-format), [065](issues/065-switchyard-responses-instruction-loss), [066](issues/066-switchyard-drops-tool-strict) | [031](issues/031-bifrost-drops-parallel-tool-flag), [032](issues/032-bifrost-drops-stop-sequences) | [042](issues/042-gomodel-drops-output-format), [043](issues/043-gomodel-drops-parallel-tool-flag) | [051](issues/051-axonhub-drops-output-format) | [058](issues/058-any-llm-drops-parallel-tool-flag), [062](issues/062-any-llm-empty-schema-shell) |
-| Content blocks survive (refusal, `is_error`, image and document blocks in tool results and user turns) | [006](issues/006-switchyard-crossformat-losses), [007](issues/007-switchyard-toolresult-multimodal-stringified), [018](issues/018-user-document-dropped), [067](issues/067-litellm-drops-refusal-content) | [006](issues/006-switchyard-crossformat-losses), [007](issues/007-switchyard-toolresult-multimodal-stringified), [018](issues/018-user-document-dropped), [068](issues/068-switchyard-drops-refusal-content) | | | | [059](issues/059-any-llm-drops-is-error), [060](issues/060-any-llm-drops-toolresult-image), [061](issues/061-any-llm-drops-toolresult-document) |
+| Content blocks survive (refusal, `is_error`, image and document blocks in tool results and user turns) | [006](issues/006-switchyard-crossformat-losses), [007](issues/007-switchyard-toolresult-multimodal-stringified), [018](issues/018-user-document-dropped), [067](issues/067-litellm-drops-refusal-content) | [006](issues/006-switchyard-crossformat-losses), [007](issues/007-switchyard-toolresult-multimodal-stringified), [018](issues/018-user-document-dropped), [068](issues/068-switchyard-drops-refusal-content), [069](issues/069-switchyard-responses-refusal) | | | | [059](issues/059-any-llm-drops-is-error), [060](issues/060-any-llm-drops-toolresult-image), [061](issues/061-any-llm-drops-toolresult-document) |
 | Assistant history survives replay (`thinking` blocks and signatures) | [016](issues/016-thinking-history-lost) (leaked as visible text) | [016](issues/016-thinking-history-lost) (dropped) | [033](issues/033-bifrost-drops-thinking-history) | | | [057](issues/057-any-llm-drops-thinking-history) |
 | Tool-call ids round-trip | [004](issues/004-gemini-thought-signature) | [005](issues/005-switchyard-toolid-sanitizer) | [037](issues/037-bifrost-toolid-not-restored) | | | |
 | Nothing is invented (empty text blocks, phantom message items, `cache_control`) | [001](issues/001-anthropic-stream-toolcall-translation), [009](issues/009-litellm-responses-phantom-message) | [019](issues/019-switchyard-invents-prompt-cache), [045](issues/045-switchyard-empty-text-before-tooluse), [068](issues/068-switchyard-drops-refusal-content) | | | | |

crates/harness/src/checks.rs

@@ -838,6 +838,175 @@ pub fn refusal_text_preserved(upstream_response_json: &str, client_response_json
     Verdict::Conformant
 }
 
+fn refusal_strings(response: &str) -> Result<Vec<String>, String> {
+    if response.lines().any(|line| line.starts_with("data:")) {
+        let events = sse_data_json(response);
+        if events.is_empty() {
+            return Err("upstream response has no parseable SSE data events".to_string());
+        }
+        let chat_refusal = events
+            .iter()
+            .flat_map(|event| {
+                event
+                    .get("choices")
+                    .and_then(Value::as_array)
+                    .into_iter()
+                    .flatten()
+            })
+            .filter_map(|choice| choice.pointer("/delta/refusal").and_then(Value::as_str))
+            .collect::<String>();
+        if !chat_refusal.is_empty() {
+            return Ok(vec![chat_refusal]);
+        }
+        let response_done_refusals = events
+            .iter()
+            .filter(|event| {
+                event.get("type").and_then(Value::as_str) == Some("response.refusal.done")
+            })
+            .filter_map(|event| event.get("refusal").and_then(Value::as_str))
+            .map(str::to_string)
+            .collect::<Vec<_>>();
+        if !response_done_refusals.is_empty() {
+            return Ok(response_done_refusals);
+        }
+        let mut refusals = Vec::new();
+        for event in &events {
+            collect_keyed_strings(event, "refusal", &mut refusals);
+        }
+        return Ok(refusals);
+    }
+
+    let body = serde_json::from_str::<Value>(response)
+        .map_err(|_| "upstream response body is not valid JSON".to_string())?;
+    let mut refusals = Vec::new();
+    collect_keyed_strings(&body, "refusal", &mut refusals);
+    Ok(refusals)
+}
+
+fn responses_event_key(event: &Value) -> Option<(&str, u64, u64)> {
+    Some((
+        event.get("item_id")?.as_str()?,
+        event.get("output_index")?.as_u64()?,
+        event.get("content_index")?.as_u64()?,
+    ))
+}
+
+fn refusal_part_event_matches(
+    event: &Value,
+    event_type: &str,
+    key: (&str, u64, u64),
+    expected_refusal: Option<&str>,
+) -> bool {
+    if event.get("type").and_then(Value::as_str) != Some(event_type)
+        || responses_event_key(event) != Some(key)
+        || event.pointer("/part/type").and_then(Value::as_str) != Some("refusal")
+    {
+        return false;
+    }
+    expected_refusal.is_none_or(|expected| {
+        event.pointer("/part/refusal").and_then(Value::as_str) == Some(expected)
+    })
+}
+
+/// Invariant (bug 069): a structured refusal returned by an upstream dialect
+/// MUST remain machine-identifiable when encoded as an OpenAI Responses result.
+/// Buffered Responses use a `refusal` content part. Streams use refusal delta
+/// and done events. Merely copying the explanation into `output_text` preserves
+/// bytes but destroys the semantic signal used by refusal-aware consumers.
+pub fn responses_refusal_semantics_preserved(
+    upstream_response: &str,
+    client_response: &str,
+) -> Verdict {
+    let upstream_refusals = match refusal_strings(upstream_response) {
+        Ok(refusals) if !refusals.is_empty() => refusals,
+        Ok(_) => {
+            return Verdict::Violation(
+                "upstream response contains no structured refusal signal".to_string(),
+            );
+        }
+        Err(message) => return Verdict::Violation(message),
+    };
+
+    if client_response
+        .lines()
+        .any(|line| line.starts_with("data:"))
+    {
+        let events = sse_data_json(client_response);
+        if events.is_empty() {
+            return Verdict::Violation(
+                "client response has no parseable Responses SSE data events".to_string(),
+            );
+        }
+        for refusal in upstream_refusals {
+            let represented = events
+                .iter()
+                .filter(|event| {
+                    event.get("type").and_then(Value::as_str) == Some("response.refusal.done")
+                        && event.get("refusal").and_then(Value::as_str) == Some(refusal.as_str())
+                })
+                .filter_map(responses_event_key)
+                .any(|key| {
+                    let delta_text = events
+                        .iter()
+                        .filter(|event| {
+                            event.get("type").and_then(Value::as_str)
+                                == Some("response.refusal.delta")
+                                && responses_event_key(event) == Some(key)
+                        })
+                        .filter_map(|event| event.get("delta").and_then(Value::as_str))
+                        .collect::<String>();
+                    delta_text == refusal
+                        && events.iter().any(|event| {
+                            refusal_part_event_matches(
+                                event,
+                                "response.content_part.added",
+                                key,
+                                None,
+                            )
+                        })
+                        && events.iter().any(|event| {
+                            refusal_part_event_matches(
+                                event,
+                                "response.content_part.done",
+                                key,
+                                Some(refusal.as_str()),
+                            )
+                        })
+                });
+            if !represented {
+                return Verdict::Violation(format!(
+                    "upstream refusal {refusal:?} is not represented by one correlated Responses refusal content-part lifecycle"
+                ));
+            }
+        }
+        return Verdict::Conformant;
+    }
+
+    let Ok(client) = serde_json::from_str::<Value>(client_response) else {
+        return Verdict::Violation("client response body is not valid JSON".to_string());
+    };
+    let typed_refusals = client
+        .get("output")
+        .and_then(Value::as_array)
+        .into_iter()
+        .flatten()
+        .filter(|item| item.get("type").and_then(Value::as_str) == Some("message"))
+        .filter(|item| item.get("role").and_then(Value::as_str) == Some("assistant"))
+        .filter_map(|item| item.get("content").and_then(Value::as_array))
+        .flatten()
+        .filter(|part| part.get("type").and_then(Value::as_str) == Some("refusal"))
+        .filter_map(|part| part.get("refusal").and_then(Value::as_str))
+        .collect::<Vec<_>>();
+    for refusal in upstream_refusals {
+        if !typed_refusals.contains(&refusal.as_str()) {
+            return Verdict::Violation(format!(
+                "upstream refusal {refusal:?} is not a typed Responses refusal content part"
+            ));
+        }
+    }
+    Verdict::Conformant
+}
+
 /// Invariant (bug 037): when a gateway rewrites an upstream tool-call id to satisfy
 /// a client-side charset contract, it MUST reverse the rewrite before sending the
 /// id back upstream. The upstream never issued the sanitized id; echoing it breaks
@@ -937,6 +1106,120 @@ mod tests {
         );
     }
 
+    #[test]
+    fn responses_refusal_checker_requires_typed_buffered_content() {
+        let upstream = r#"{"choices":[{"message":{"content":null,"refusal":"cannot help"}}]}"#;
+        let conformant = r#"{"output":[{"type":"message","role":"assistant","content":[{"type":"refusal","refusal":"cannot help"}]}]}"#;
+        assert_eq!(
+            responses_refusal_semantics_preserved(upstream, conformant),
+            Verdict::Conformant
+        );
+
+        let flattened = r#"{"output":[{"type":"message","content":[{"type":"output_text","text":"cannot help"}]}]}"#;
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, flattened),
+            Verdict::Violation(_)
+        ));
+        assert!(matches!(
+            responses_refusal_semantics_preserved(r#"{"choices":[]}"#, conformant),
+            Verdict::Violation(_)
+        ));
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, "not-json"),
+            Verdict::Violation(_)
+        ));
+        let wrong_output_item = r#"{"output":[{"type":"function_call","role":"assistant","content":[{"type":"refusal","refusal":"cannot help"}]}]}"#;
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, wrong_output_item),
+            Verdict::Violation(_)
+        ));
+        assert!(matches!(
+            responses_refusal_semantics_preserved("not-json", conformant),
+            Verdict::Violation(_)
+        ));
+        let two_upstream =
+            r#"{"choices":[{"message":{"refusal":"first"}},{"message":{"refusal":"second"}}]}"#;
+        let two_client = r#"{"output":[{"type":"message","role":"assistant","content":[{"type":"refusal","refusal":"first"},{"type":"refusal","refusal":"second"}]}]}"#;
+        assert_eq!(
+            responses_refusal_semantics_preserved(two_upstream, two_client),
+            Verdict::Conformant
+        );
+        assert!(matches!(
+            responses_refusal_semantics_preserved(two_upstream, conformant),
+            Verdict::Violation(_)
+        ));
+    }
+
+    #[test]
+    fn responses_refusal_checker_requires_stream_delta_and_done() {
+        let upstream = "data: {\"choices\":[{\"delta\":{\"refusal\":\"cannot \"}}]}\n\n\
+            data: {\"choices\":[{\"delta\":{\"refusal\":\"help\"}}]}\n\n";
+        let conformant = "event: response.content_part.added\n\
+            data: {\"type\":\"response.content_part.added\",\"item_id\":\"msg_1\",\"output_index\":0,\"content_index\":0,\"part\":{\"type\":\"refusal\",\"refusal\":\"\"}}\n\n\
+            event: response.refusal.delta\n\
+            data: {\"type\":\"response.refusal.delta\",\"item_id\":\"msg_1\",\"output_index\":0,\"content_index\":0,\"delta\":\"cannot help\"}\n\n\
+            event: response.refusal.done\n\
+            data: {\"type\":\"response.refusal.done\",\"item_id\":\"msg_1\",\"output_index\":0,\"content_index\":0,\"refusal\":\"cannot help\"}\n\n\
+            event: response.content_part.done\n\
+            data: {\"type\":\"response.content_part.done\",\"item_id\":\"msg_1\",\"output_index\":0,\"content_index\":0,\"part\":{\"type\":\"refusal\",\"refusal\":\"cannot help\"}}\n\n";
+        assert_eq!(
+            responses_refusal_semantics_preserved(upstream, conformant),
+            Verdict::Conformant
+        );
+
+        let flattened = "event: response.output_text.delta\n\
+            data: {\"type\":\"response.output_text.delta\",\"delta\":\"cannot help\"}\n\n";
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, flattened),
+            Verdict::Violation(_)
+        ));
+        let missing_done = "event: response.refusal.delta\n\
+            data: {\"type\":\"response.refusal.delta\",\"delta\":\"cannot help\"}\n\n";
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, missing_done),
+            Verdict::Violation(_)
+        ));
+        let crossed_items = conformant.replace(
+            "\"type\":\"response.refusal.delta\",\"item_id\":\"msg_1\"",
+            "\"type\":\"response.refusal.delta\",\"item_id\":\"msg_2\"",
+        );
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, &crossed_items),
+            Verdict::Violation(_)
+        ));
+        assert!(matches!(
+            responses_refusal_semantics_preserved("data: not-json\n\n", conformant),
+            Verdict::Violation(_)
+        ));
+        assert!(matches!(
+            responses_refusal_semantics_preserved(upstream, "data: not-json\n\n"),
+            Verdict::Violation(_)
+        ));
+        let responses_upstream =
+            "data: {\"type\":\"response.refusal.delta\",\"delta\":\"cannot help\"}\n\n\
+            data: {\"type\":\"response.refusal.done\",\"refusal\":\"cannot help\"}\n\n";
+        assert_eq!(
+            responses_refusal_semantics_preserved(responses_upstream, conformant),
+            Verdict::Conformant
+        );
+        let two_upstream =
+            "data: {\"type\":\"response.refusal.done\",\"refusal\":\"cannot help\"}\n\n\
+            data: {\"type\":\"response.refusal.done\",\"refusal\":\"second refusal\"}\n\n";
+        let second_lifecycle = conformant
+            .replace("msg_1", "msg_2")
+            .replace("cannot help", "second refusal")
+            .replace("\"output_index\":0", "\"output_index\":1");
+        let two_client = format!("{conformant}{second_lifecycle}");
+        assert_eq!(
+            responses_refusal_semantics_preserved(two_upstream, &two_client),
+            Verdict::Conformant
+        );
+        assert!(matches!(
+            responses_refusal_semantics_preserved(two_upstream, conformant),
+            Verdict::Violation(_)
+        ));
+    }
+
     #[test]
     fn json_schema_forwarded_needs_the_wire_field() {
         let present = r#"{"body":{"text":{"format":{"type":"json_schema"}}}}"#;