Skip to content

[Bug] OpenAI-compatible providers never request streaming usage, so Ollama and similar servers report no tokens #1658

Closed

Description

@addyCooks

Description

createOpenAICompatible(...) in source/ai-sdk-client/providers/provider-factory.ts (lines 532-538) is created without includeUsage: true, so the SDK never sends stream_options: {include_usage: true}. Ollama and some other servers only send token usage in a streamed reply when asked, so Nanocoder gets none. The per-response usage indicator falls back to estimates, and real token counts, cost and cache-hit data are missing. The --json report from run has no usage, and the agent eval harness records 0 tokens for local models (benchmarks/agent/harness.ts:441-443 uses usage.totalTokens ?? 0).

Environment

  • OS: Windows 11 (10.0.26300)
  • Node version: 22.22.3
  • Nanocoder version: 1.31.0 (main at 5c2e693)
  • Provider: Ollama 0.35.1 (OpenAI-compatible endpoint)
  • Model: qwen2.5:1.5b

Steps to Reproduce

  1. Configure an Ollama provider.
  2. Put a small logging proxy between Nanocoder and Ollama (or inspect the request another way).
  3. Run nanocoder --trust-directory --json run "Reply with the single word OK".
  4. Look at the request body and at the JSON report.
  5. As a control, send the same request to Ollama by hand, once with "stream_options":{"include_usage":true} and once without.

Expected Behavior

The request contains "stream_options": {"include_usage": true}, and the report includes token usage.

Actual Behavior

The request keys are only model, messages, tools, tool_choice, stream, and the streamed reply has no usage. The report has no usage field:

keys: kind, exitCode, finalText, reasoning, toolCalls, steps, filesChanged
usage: undefined

The control request shows the server does send usage when asked:

with stream_options:    "usage":{"prompt_tokens":31,"prompt_tokens_details":{"cached_tokens":5}, ...
without stream_options: 0 usage chunks

Suggested Fix

  • Pass includeUsage: true to createOpenAICompatible in provider-factory.ts (the SDK option exists in @ai-sdk/openai-compatible).
  • Consider a per-provider setting to turn it off, for any server that rejects stream_options.
  • Add a test in provider-factory.spec.ts: start a small local HTTP server, stream one reply through the provider, and check that the request body has stream_options.include_usage === true and that the usage comes back.
  • Related: the harness could store missing usage as null instead of 0.

Additional Context

Activity

  1. added theissue type on Oct 8, 2026
  2. dhruvil-codes commented on Oct 9, 2026

    @dhruvil-codes
    Contributor

    Hi @akramcodez @addyCooks, I would like to take this on!

    Proposed Approach:

    1. Enable Streaming Usage in Factory: In source/ai-sdk-client/providers/provider-factory.ts (inside createOpenAICompatible(...)), pass includeUsage: true so the underlying @ai-sdk/openai-compatible client sends "stream_options": { "include_usage": true } on streaming requests.
    2. Provider Config Compatibility: Ensure this plays nicely with existing OpenAI-compatible endpoints (like Ollama, LM Studio, and vLLM) while preserving any custom options.
    3. Unit Tests: Add a test case in source/ai-sdk-client/providers/provider-factory.spec.ts asserting that streamed requests to OpenAI-compatible providers include stream_options.include_usage === true and return token metrics.
    4. Changeset: Include a patch changeset as required by the contribution guidelines.

    Could you please assign this issue to me? I will have a PR ready shortly.

  3. addyCooks commented on Oct 9, 2026

    @addyCooks
    CollaboratorAuthor

    Thanks @dhruvil-codes,
    assigning this to you for now..
    I can see PR #1671 is already up

    For future issues, please wait until you've been assigned before opening a PR.
    Assignment avoids duplicate work, and it lets us flag design concerns before you've written the code.
    Comment on the issue as you did here, then start once you're assigned.

  4. dhruvil-codes commented on Oct 9, 2026

    @dhruvil-codes
    Contributor

    Thanks @addyCooks for assigning this to me! Understood completely regarding the workflow. I'll be sure to wait for the official assignment before opening PRs on future issues.

    Appreciate the guidance!

  5. added a commit that references this issue on Oct 12, 2026
    37d751e
  6. added a commit that references this issue on Oct 12, 2026
    d50fb18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:providersLLM providers, models, tokenisation, context limitsbugSomething isn't workinggood first issueGood for newcomers

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions