Skip to content

Fix Gemini ACP model IDs with Google provider prefixes - #1030

Open
kywch wants to merge 6 commits into
mainfrom
fix/gemini-acp-model-id
Open

Fix Gemini ACP model IDs with Google provider prefixes#1030
kywch wants to merge 6 commits into
mainfrom
fix/gemini-acp-model-id

Conversation

@kywch

@kywch kywch commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Fixes #975

Summary

  • Normalize models.dev google/gemini-* and google/gemma-* IDs to bare model IDs on the Gemini ACP path.
  • Normalize the same Google model families to a single gemini/ provider prefix when routing through the generic LiteLLM gateway.
  • Preserve unrelated google/* model IDs.
  • Port fix(gemini): strip Google model prefix #979 stronger connect_acp -> session/set_model regression coverage for Gemini and Gemma, plus focused gateway boundary coverage.

Validation

  • uv run pytest -q tests/test_acp.py tests/test_litellm_config.py tests/test_litellm_hardening.py -> 175 passed after merging current main.
  • Ruff lint and format checks passed for all changed files.
  • Live A/B evidence already documented in this review comment: real @google/gemini-cli@0.42.0 and Google GenerateContent returned 200 for normalized Gemini and Gemma IDs; prefixed IDs reproduced the provider 404.

@kywch
kywch deployed to pypi-internal-preview August 16, 2026 20:51 — with GitHub Actions Active
@tulerfeng

tulerfeng commented Aug 20, 2026

Copy link
Copy Markdown

I have Gemini API credit available, so I was able to run the live checks that
this PR and #979 were both missing — a real @google/gemini-cli@0.42.0 talking
to the real Google GenerateContent endpoint, rather than only the offline
string-level reproduction.

Short version: this PR fixes the reported bug, and it fixes strictly more than
#979 does.

Setup

Live A/B

requested model tree ACP modelId sent live gemini-cli 0.42.0
google/gemini-3.1-flash-lite-preview 29c64e7 (before fix) google/gemini-3.1-flash-lite-preview 404 ModelNotFoundError
google/gemini-3.1-flash-lite-preview bc9a1c3 (#979) gemini-3.1-flash-lite-preview 200 ok
google/gemini-3.1-flash-lite-preview 0845235 (this PR) gemini-3.1-flash-lite-preview 200 ok
google/gemma-4-31b-it 29c64e7 (before fix) google/gemma-4-31b-it 404 ModelNotFoundError
google/gemma-4-31b-it bc9a1c3 (#979) google/gemma-4-31b-it 404 ModelNotFoundError
google/gemma-4-31b-it 0845235 (this PR) gemma-4-31b-it 200 ok
gemini-3.5-flash-lite (already bare) all three trees gemini-3.5-flash-lite 200 ok

The gemma rows are the reason I'd land this one rather than #979. #979 matches
google/gemini-, so google/gemma-* still reaches the CLI prefixed and still
404s. This PR matches google/ and normalizes both. That shape is not
hypothetical for BenchFlow: _MODELSDEV_PROVIDER_HEURISTICS already carries
("gemma", "google"), so the codebase produces google/gemma-* in the reverse
direction.

The 404 in #975, confirmed end to end

The empty error message in the original report ("message":"b''") is not a
truncation. Google returns a routing 404 with no body at all when the / in the
model name breaks the URL path:

POST /v1beta/models/gemini-3.5-flash-lite:generateContent         -> 200
POST /v1beta/models/google/gemini-3.5-flash-lite:generateContent  -> HTTP/2 404
                                                content-type: text/html
                                                content-length: 0

Which is exactly what the CLI then surfaces:

ModelNotFoundError: {"error":{"message":"","code":404,"status":"Not Found"}}

Empty body -> empty message -> b'' once it reaches BenchFlow. The causal chain
in #975 holds all the way to the provider.

Tests, lint, and mutation

  • uv run python -m pytest tests/test_acp.py -q -> 96 passed
  • uv run python -m pytest tests/test_litellm_hardening.py -q -> 52 passed
  • uv run ruff check / ruff format --check on both touched files -> clean
  • Mutation: deleting the two added lines makes
    test_gemini_acp_model_strips_google_prefix fail and restoring them makes it
    pass again, so the new test genuinely guards the fix.
  • Cross-check: applying fix(gemini): strip Google model prefix #979's tests/test_acp.py diff on top of this PR gives
    14 passed in TestConnectAcpModelSelection. This implementation satisfies
    fix(gemini): strip Google model prefix #979's assertions as well; the reverse is not true because of the gemma gap.

Non-regression, identical on all three trees: opencode + gemini-3.5-flash-lite
still resolves to google/gemini-3.5-flash-lite, and pi-acp + benchflow-x
still resolves to litellm/benchflow-x.

Suggestion

Land this implementation, and consider folding in #979's parametrized case in
tests/test_acp.py — it exercises connect_acp and asserts on the actual
session/set_model call, which is a stronger guard than the direct
_format_acp_model unit test here. Adding a google/gemma-* case would pin the
behaviour that distinguishes the two approaches.

Minor: this PR describes #975 but does not use a closing keyword, so it is not
linked to the issue. #979 has Fixes #975.

Everything above was validated end to end against the live Gemini API — real
CLI, real provider, real 404 — rather than at the string level only.

@kywch
kywch deployed to pypi-internal-preview August 30, 2026 20:19 — with GitHub Actions Active
@kywch
kywch deployed to pypi-internal-preview August 30, 2026 21:33 — with GitHub Actions Active
@kywch

kywch commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator Author

#979 will close unmerged after this replacement lands. Its stronger connect_acp -> session/set_model coverage is now ported here for both Gemini and Gemma. Its unrelated judge change (bc9a1c3) will move to a separate PR.

Validation after merging current main: 175 affected tests passed; Ruff lint/format and git diff --check passed. Branch pushed normally and remains draft pending fresh CI/review.

@kywch
kywch marked this pull request as ready for review August 30, 2026 22:05
@kywch
kywch deployed to pypi-internal-preview August 30, 2026 22:06 — with GitHub Actions Active

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@kywch
kywch deployed to pypi-internal-preview August 31, 2026 02:18 — with GitHub Actions Active
@kywch

kywch commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed Devin’s acpx/gemini finding using existing ACPX normalization, added integration coverage, and removed duplicate unit coverage. Merged current main for RestrictedPython 8.3. All CI checks pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemini ACP passes google/ provider prefixes into Gemini CLI model IDs

2 participants