Skip to content

fix: duplicate litellm stream call doubles provider requests - #476

Closed
quevedoSteven wants to merge 1 commit into
aliasrobotics:mainfrom
quevedoSteven:fix/duplicate-litellm-stream-request
Closed

fix: duplicate litellm stream call doubles provider requests#476
quevedoSteven wants to merge 1 commit into
aliasrobotics:mainfrom
quevedoSteven:fix/duplicate-litellm-stream-request

Conversation

@quevedoSteven

Copy link
Copy Markdown

Summary

The streaming path in fetch_response_litellm_openai called litellm.acompletion twice with the same kwargs. The first result was assigned to a variable that is never read — only the second call's stream was returned.

Impact

Every streamed request consumed two provider requests (e.g. two NVIDIA NIM requests) with the same API key:

  • Halves the effective per-key rate limit (a 40 req/min NIM key only supports 20 streamed turns/min).
  • With multi-key round-robin rotation, both calls share the same rotated key, so rotation spreads turns — not requests — and a busy session (several concurrent agents) quickly exhausts both keys → 429 → LLMRateLimited: Rate limit after 3 attempts.
  • The first call's stream is abandoned, leaking an unconsumed stream on every request.

Change

Keep a single litellm.acompletion call for the streamed path, and the same fix in the tool_call_id truncation retry branch (which had the identical double-call pattern).

Verification

  • stream=True: one acompletion call; the returned stream is the only request sent.
  • stream=False: unchanged.
  • Syntax checked (ast.parse); no references to the removed ret variable remain.

…penai

The streaming path called litellm.acompletion twice with the same kwargs.
The first result was assigned to a variable that is never read, and only
the second call's stream was returned. Every streamed request therefore
consumed two provider requests (e.g. two NVIDIA NIM requests) with the
same API key, halving effective per-key rate limits and leaking the
first stream. Keep a single acompletion call for the streamed path; same
fix in the tool_call_id truncation retry branch.
@quevedoSteven

Copy link
Copy Markdown
Author

Superseded by #477 which includes the required NIM rotation code from #475.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants