Skip to content

common : preserve original Gemma 4 tool responses even when JSON-like#21522

Open
kiwixz wants to merge 1 commit intoggml-org:masterfrom
kiwixz:gemma_tool_json
Open

common : preserve original Gemma 4 tool responses even when JSON-like#21522
kiwixz wants to merge 1 commit intoggml-org:masterfrom
kiwixz:gemma_tool_json

Conversation

@kiwixz
Copy link
Copy Markdown

@kiwixz kiwixz commented Apr 6, 2026

Overview

Before this PR, llama.cpp Gemma 4 specialized parser (#21418) would attempt to parse the string-typed tool responses as JSON; and convert it to an object if successful. It would then be minified by the template:

{%- if tool_response['response'] is mapping -%}
    {{- 'response:' + tool_response['name'] | default('unknown') + '{' -}}
    {%- for key, value in tool_response['response'] | dictsort -%}
        {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
        {%- if not loop.last %},{% endif -%}
    {%- endfor -%}
    {{- '}' -}}
{%- else -%}
    {{- 'response:' + tool_response['name'] | default('unknown') + '{value:' + format_argument(tool_response['response'], escape_keys=False) + '}' -}}
{%- endif -%}

This means that anything that looks like JSON would get unexpectedly altered.
I encountered this with Gemma 4 failing to edit a JSON file; it was getting confused as the inconsistent line numbers and indentations between what it read and what it attempted to write/edit.

Additional information

Requirements

@aldehir
Copy link
Copy Markdown
Contributor

aldehir commented Apr 6, 2026

Which coding agent are you using? This might be the best thing to do, but it does have negative implications for clients that send structured tool output.

@kiwixz
Copy link
Copy Markdown
Author

kiwixz commented Apr 6, 2026

I'm developing my own thing. I found that text works better than structured things at least for simple tools.
The only difference for structured output would be keeping the quotes around JSON properties and wrapping the response in a {value: } object.

It's too bad that OpenAI API doesn't support just passing a JSON object directly in there; would be much more elegant than forcing it to a string and then guessing what it was...

@aldehir
Copy link
Copy Markdown
Contributor

aldehir commented Apr 6, 2026

Many read tools prepend the line number to each line, which would make this get handled as a string.

Here is my concern: the model may in-context learn to start producing JSON instead of its native structure. We see this happen in other models where we manipulate the input in a way it wasn't trained.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants