Skip to content

fix(beta): persist OpenAI server-side builtin tool calls in history#2712

Closed
vvlrff wants to merge 2 commits intoag2ai:mainfrom
vvlrff:fix/openai-builtin-history
Closed

fix(beta): persist OpenAI server-side builtin tool calls in history#2712
vvlrff wants to merge 2 commits intoag2ai:mainfrom
vvlrff:fix/openai-builtin-history

Conversation

@vvlrff
Copy link
Copy Markdown
Collaborator

@vvlrff vvlrff commented Apr 21, 2026

Problem

The OpenAI Responses API requires that reasoning items be paired with their server-side tool calls (web_search, code_interpreter, image_generation) when history is replayed back to the model. Otherwise the API rejects the request.

Current behaviour was broken for replay:

  • Server-side calls were emitted as generic BuiltinToolCallEvent / BuiltinToolResultEvent — with no reference to the original SDK object. When rebuilding history for a follow-up request there was nothing to re-emit.
  • Reasoning items were emitted as ModelReasoning(text) — also with no SDK object attached.
  • In the streaming path, some builtin items were processed on ResponseOutputItemAddedEvent, when the SDK object was not yet fully populated (e.g. outputs missing on code_interpreter).

Net effect: a conversation that used any builtin server-side tool could not be fed back to the Responses API in a subsequent turn.

Solution

New provider-specific events

File: autogen/beta/config/openai/events.py

Event Base class Payload
OpenAIServerToolCallEvent BuiltinToolCallEvent item: ResponseFunctionWebSearch | ResponseCodeInterpreterToolCall | ImageGenerationCall
OpenAIServerToolResultEvent BuiltinToolResultEvent observability-only, no payload
OpenAIReasoningEvent ModelReasoning item: ResponseReasoningItem

Each event carries the original SDK object (item) so the mapper can replay it verbatim.

History replay

File: autogen/beta/config/openai/mappers.py

New branch in events_to_responses_input:

elif isinstance(message, (OpenAIReasoningEvent, OpenAIServerToolCallEvent)):
    result.append(message.item.model_dump(exclude_none=True, mode="json"))

OpenAIServerToolResultEvent is intentionally not re-emitted — the Responses API models a server-side tool as a single combined item, already covered by the paired call event.

Client

File: autogen/beta/config/openai/openai_responses_client.py

_process_response (non-streaming):

  • web_search / code_interpreter / image_generation now emit OpenAIServerToolCallEvent(item=...) followed by an empty OpenAIServerToolResultEvent.
  • Reasoning is emitted as OpenAIReasoningEvent(text, item=...), where text joins all summary[].text with \n\n.
  • Duplicate data (status / container_id / outputs) removed from ToolResult payloads — it all lives on item now.

_process_stream (streaming):

  • All builtin / reasoning handling moved from ResponseOutputItemAddedEvent to ResponseOutputItemDoneEvent, so the SDK object is fully populated (on Added, code_interpreter.outputs is empty).
  • Dropped provider_data on BuiltinToolCallEvent — this info is on item now.
  • Removed the now-unused ResponseOutputItemAddedEvent import.

@vvlrff
Copy link
Copy Markdown
Collaborator Author

vvlrff commented Apr 21, 2026

This pr can be considered after #2695

@vvlrff vvlrff marked this pull request as draft April 23, 2026 16:55
@vvlrff vvlrff closed this Apr 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant