Issue 069: Switchyard loses Responses refusal semantics

[Atharva-Kanherkar]

Finding

Claim

Switchyard loses the machine-readable OpenAI refusal type when translating an OpenAI Chat refusal to OpenAI Responses output, and pull request #623 preserves only the words as ordinary output text.

• Upstream project: NVIDIA-NeMo/Switchyard
• Cited upstream issue or report: [bug] OpenAI-to-Anthropic translation erases structured refusal text and emits empty text block NVIDIA-NeMo/Switchyard#622 and pull request #623
• Tested release or commit: main 7a23989cbe18f1c6c67ee03684ce76bd5901a27d and pull request #623 head 2765f46972bf89a96beb5b2158b0fc56a3a72288, both reporting 0.2.0
• Client dialect and endpoint: OpenAI Responses, /v1/responses, buffered and streaming
• Backend dialect and provider or capture upstream: OpenAI Chat, deterministic local capture upstream
• Model, if relevant: captured-model
• Relevant configuration: openai_chat backend, passthrough route, max_retries=0

Gate 1: correctness

Exact reproduction

git clone --filter=blob:none https://github.com/NVIDIA-NeMo/Switchyard.git /tmp/switchyard-069
rustup toolchain install 1.96.1
git -C /tmp/switchyard-069 checkout --detach 7a23989cbe18f1c6c67ee03684ce76bd5901a27d
python3 transcripts/069/reproduce.py --switchyard-source /tmp/switchyard-069 --label main --expected-commit 7a23989cbe18f1c6c67ee03684ce76bd5901a27d
git -C /tmp/switchyard-069 checkout --detach 2765f46972bf89a96beb5b2158b0fc56a3a72288
python3 transcripts/069/reproduce.py --switchyard-source /tmp/switchyard-069 --label pr623 --expected-commit 2765f46972bf89a96beb5b2158b0fc56a3a72288

• Expected behavior: Responses output carries a refusal content part, or correlated response.refusal.delta and response.refusal.done events.
• Observed behavior: current main erases the refusal; pull request #623 flattens it to output_text and response.output_text.delta.
• Raw request evidence: transcripts/069/switchyard-{main,pr623}-{responses,chat}-*.jsonl
• Raw response evidence: the same JSONL files record exact upstream and client bytes.
• Forwarded-request evidence, if applicable: every line records the exact /v1/chat/completions request Switchyard sent upstream.
• Reproduction rate: Responses typed-refusal misses 20/20; Chat controls preserve 20/20.
• Smallest isolated trigger: change only the client route from /v1/chat/completions to /v1/responses while keeping the same process, prompt, model, upstream path, and upstream refusal bytes.

Control

cargo test -p kairo --test conformance switchyard_chat_route_preserves_the_same_refusal_control

• Control result: the same-dialect Chat route preserves message.refusal and delta.refusal byte-for-byte in 20/20 trials.
• Why this attributes the failure to the claimed layer: the control changes only the client output dialect. The upstream request and canned response remain the same.

False-positive checks

• Tested the exact cited behavior, not a similar symptom.
• Pinned and reported the target version or commit.
• Ruled out bad configuration and malformed input.
• Ruled out model nondeterminism or reported why it is irrelevant.
• Confirmed the failure is not created only by the mock or harness.
• Sanitized all recorded evidence.

The reproducer checks a clean source tree, exact commit, reported version, compiler version, built binary SHA-256, exact translated request, exact canned upstream response, and monotonic exchange count before writing evidence. The upstream refusal shape is part of OpenAI's documented Chat schema and is independently present in pull request #623's live-provider evidence.

Gate 2: usefulness

Who gets bitten

• Affected user or customer: an application, evaluator, or guardrail using the OpenAI Responses API through Switchyard with an OpenAI Chat backend.
• Real workflow: send a Responses request, receive an upstream policy refusal, then branch on the documented refusal content type or stream events.
• Preconditions and likely frequency: the backend returns message.refusal or delta.refusal; real-world refusal frequency was not measured.

Observable consequence

• User action: send a refusal-triggering Responses request through Switchyard.
• Wire-level defect: Switchyard emits empty success output on main or ordinary output text on pull request #623 instead of typed refusal output.
• End-user or agent-level failure: a refusal-aware consumer follows the ordinary-answer path or records a false negative.
• Consumer-boundary demonstration or transcript: every JSONL line contains the refusal-aware consumer classification.
• Measured impact: 20/20 Responses trials missed the refusal type; all 20 Chat controls detected it.
• Inferred impact, clearly labeled: refusal analytics, evaluation scoring, retry policy, and safety handling can be wrong whenever this backend shape occurs.

Gate 3: upstream status

• Date checked: 2026-09-05
• Upstream version checked: Switchyard 0.2.0, main 7a23989, pull request #623 2765f46
• Search terms used: message.refusal, delta.refusal, response.refusal.delta, response.refusal.done, output_text, Responses refusal, refusal
• Issues searched: open and closed Switchyard issues, including #622
• Pull requests searched: open and closed Switchyard pull requests, including #623 and #370
• Releases, changelog, and documentation searched: release 0.2.0, CHANGELOG.md, repository documentation, official OpenAI Chat and Responses references
• Relevant commits searched: current main, pull request #623 head, and related refusal commits
• Matching links: [bug] OpenAI-to-Anthropic translation erases structured refusal text and emits empty text block NVIDIA-NeMo/Switchyard#622, fix(translation): preserve OpenAI Chat refusal text NVIDIA-NeMo/Switchyard#623, fix(translation): report content filter stops as Anthropic refusal NVIDIA-NeMo/Switchyard#370

Classification:

• Novel
• Duplicate, open and still reproducible
• Fixed on current release
• Regression
• Documented behavior
• Discussed upstream without a dedicated ticket
• Incomplete, current upstream state could not be verified

Issue #622 and pull request #623 cover adjacent refusal-text loss on Chat to Anthropic output. No dedicated ticket or fix was found for Chat to Responses refusal typing. This PR freezes the separate type-loss defect, including the state after pull request #623.

Frozen invariant

• Issue writeup: issues/069-switchyard-responses-refusal/README.md
• Checker added or updated: responses_refusal_semantics_preserved
• Conformance test added or updated: main loss, pull request #623 flattening, same-dialect controls, and documented positive Responses fixtures
• Why the checker tests the invariant rather than one implementation detail: it accepts documented typed refusal representations, correlates stream events by item and indexes, rejects ordinary text flattening, and requires non-vacuous structured upstream evidence.

Validation

Check	Command	Result
Reproduction	python3 transcripts/069/reproduce.py ... on both pinned revisions	PASS, 5/5 per scenario
Control	same-process Chat buffered and streaming	PASS, 20/20
Harness	cargo test --workspace	PASS, 130 tests
Formatting	cargo fmt --all -- --check	PASS
Lint	cargo clippy --workspace --all-targets -- -D warnings	PASS
README counts	python3 tools/update-readme-counts.py --check	PASS

Security and scope

• No API key, credential, private prompt, or unsanitized response is committed.
• Only environment variable names appear in commands and documentation.
• The pull request contains one finding.
• Unrelated generated files and local state are excluded.

Author verdict

• Correctness: PASS
• Usefulness: PASS
• Upstream status: PASS
• Overall: ACCEPT

Independent review

Run .github/agents/kairo-reproduction-reviewer.agent.md against this pull request. Approval is blocked until the reviewer independently reruns the critical path and all three gates pass.

Independent read-only rerun at stable commit 659d3ff: ACCEPT.

• Correctness: PASS. Fresh outputs for both pinned revisions were byte-identical to all committed captures, including binary SHA-256 provenance.
• Usefulness: PASS. The refusal-aware consumer missed 20/20 Responses refusals while the Chat control detected 20/20. A trigger-removed ordinary-text Responses control also passed 6/6 across both modes.
• Upstream status: PASS. Classified discussed-no-ticket on 2026-09-05.
• Repository checks: 130 tests, formatting, clippy, README counts, diff checks, fixture replay, credential scan, and style scan passed.
• Blocking findings: none.