Skip to content

Support evaluation of models with interleaved thinking #14

Description

@lucenzhong

To support evaluation of models with interleaved thinking, the input message should preserve reasoning_content from previous turns to maintain reasoning consistency.

The relevant code changes are as follows:

  1. Enable thinking when calling the model
  2. Support returning reasoning_content in llm.py and schema.py

assistant_message = AssistantMessage(
role="assistant",
content=response.choices[0].message.content,
tool_calls=tool_calls,
)

class AssistantMessage(BaseModel):
"""Assistant message."""
role: Literal["assistant"]
content: Optional[str] = None
tool_calls: Optional[List[ToolCall]] = None

Modified to:

 assistant_message = AssistantMessage( 
     role="assistant", 
     content=response.choices[0].message.content, 
     tool_calls=tool_calls, 
     reasoning_content=response.choices[0].message.reasoning_content,
 ) 
class AssistantMessage(BaseModel):
    """Assistant message."""

    role: Literal["assistant"]
    content: Optional[str] = None
    tool_calls: Optional[List[ToolCall]] = None
    reasoning_content: Optional[str] = None

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions