Add provider-level tool_use_kcount for prices shared across all models#291
Add provider-level tool_use_kcount for prices shared across all models#291Kludex wants to merge 1 commit intoadd-web-search-request-pricingfrom
Conversation
Adds `tool_use_kcount` to the Provider type so tool use prices that apply uniformly can be defined once. Model-level tool_use_kcount overrides provider-level for the same unit. Moves Anthropic web_search ($10/1k) and OpenAI file_search ($2.50/1k) to provider level, removing duplication across 14 and 11 models respectively.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const timestamp = options?.timestamp ?? new Date() | ||
| const modelPrice = getActiveModelPrice(model, timestamp) | ||
| const priceResult = calcPriceInternal(usage, modelPrice) | ||
| const priceResult = calcPriceInternal(usage, modelPrice, provider.tool_use_kcount) |
There was a problem hiding this comment.
🔴 Provider-level tool_use_kcount lost when model is resolved via fallback provider
When a model is found through the fallback_model_providers mechanism (e.g., Azure falling back to OpenAI for gpt-4o), the code passes the requesting provider's tool_use_kcount — not the fallback provider's. Since Azure and Google don't define tool_use_kcount, all provider-level tool pricing moved by this PR is silently dropped for fallback lookups.
Detailed explanation of the fallback pricing loss
Azure has fallback_model_providers: ['openai', 'anthropic'] (packages/js/src/data.ts:1517) and does not define its own gpt-4o model. When a user queries Azure for gpt-4o, the matchModelWithFallback function finds OpenAI's gpt-4o model definition. However, the provider variable at packages/js/src/api.ts:73-74 remains the Azure provider (which has no tool_use_kcount).
Before this PR, OpenAI's gpt-4o model had tool_use_kcount: {web_search: 25, file_search: 2.5} at the model level, so fallback worked correctly. After this PR, file_search: 2.5 was moved to OpenAI's provider-level tool_use_kcount, leaving the model with only {web_search: 25}. Since provider.tool_use_kcount is Azure's (undefined), file_search pricing is completely lost.
The same issue applies to:
- Azure → Anthropic fallback: Claude models lose
web_search: 10pricing - Google → Anthropic fallback: Claude models lose
web_search: 10pricing
Impact: Users calculating prices through Azure or Google providers for fallback models will get incorrect (lower) total_price values — file_search and web_search tool use costs will be $0 instead of the correct amount.
Prompt for agents
The fix needs to handle the case where a model is resolved via fallback_model_providers. When the model comes from a fallback provider, the provider-level tool_use_kcount should come from the fallback provider, not the original requesting provider.
In packages/js/src/api.ts line 80, instead of always using provider.tool_use_kcount, the code needs to determine which provider actually supplied the model. One approach:
1. Modify matchModelWithFallback in packages/js/src/engine.ts (around line 219) to also return which provider the model was found in.
2. Use that provider's tool_use_kcount in the calcPriceInternal call at packages/js/src/api.ts line 80.
The same fix is needed in the Python side:
- In packages/python/genai_prices/data_snapshot.py around line 59-65, the calc method passes the original provider to model.calc_price. When the model was found via fallback, the fallback provider's tool_use_kcount should be used instead.
- One approach: modify find_provider_model to also track which provider the model was actually found in, and pass that provider's tool_use_kcount.
Alternatively, a simpler fix would be to keep file_search and web_search at the model level (reverting the data change) while still supporting provider-level tool_use_kcount for future use cases where fallback is not involved.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def calc_price(self, usage: types.AbstractUsage, **kwargs: Any) -> types.CalcPrice: | ||
| price = super().calc_price(usage, **kwargs) |
There was a problem hiding this comment.
🚩 **Custom ModelPrice subclass compatibility preserved via kwargs
The test file tests/test_custom_prices.py:20-21 was updated to pass **kwargs through to super().calc_price(). This correctly forwards the new provider_tool_use_kcount keyword argument. The signature change from def calc_price(self, usage) to def calc_price(self, usage, **kwargs) is backward compatible for existing subclasses that don't override calc_price, but any external subclass that overrides calc_price with the old signature (positional usage only, no **kwargs) will break at runtime when called from ModelInfo.calc_price at packages/python/genai_prices/types.py:591 because provider_tool_use_kcount is passed as a keyword argument. This is a minor breaking change for external consumers who subclass ModelPrice.
Was this helpful? React with 👍 or 👎 to provide feedback.
| Provider: OpenAI | ||
| Model: gpt 4o | ||
| Model Prices: $2.5/input MTok, $1.25/cache read MTok, $10/output MTok, $25 / K web search, $2.5 / K file search | ||
| Model Prices: $2.5/input MTok, $1.25/cache read MTok, $10/output MTok, $25 / K web search |
There was a problem hiding this comment.
🚩 CLI snapshot change reflects the data migration correctly
The CLI test snapshot at tests/test_cli.py:23 changed from showing $25 / K web search, $2.5 / K file search to just $25 / K web search for gpt-4o. This is correct because the ModelPrice.__str__ method at packages/python/genai_prices/types.py:707-723 only displays model-level tool_use_kcount, not provider-level. The file_search price still applies during calculation (via provider-level merge), it just doesn't show in the model's string representation. This is a minor UX change - users inspecting model prices via the CLI won't see provider-level tool prices, which could cause confusion.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
tool_use_kcountto theProvidertype so tool use prices that apply uniformly across all models can be defined once at the provider leveltool_use_kcountoverrides provider-level for the same unit keyweb_search: 10to provider level (was duplicated across 14 models)file_search: 2.5to provider level (was duplicated across 11 models)web_searchstays at model level since it varies ($25-30/1k depending on model)Test plan