Add provider model and token limit overrides to ProviderConfig by MackinnonBuck · Pull Request #966 · github/copilot-sdk

MackinnonBuck · 2026-03-31T17:48:08Z

Summary

Extends ProviderConfig across all four language SDKs with optional fields for model overrides and token limits. These fields match the runtime's wire format and let BYOK users decouple the model name sent to their provider from the well-known model ID used for agent configuration and capability lookup.

New fields

Wire name	Node.js	Python	.NET	Go
`modelId`	`modelId?: string`	`model_id: str`	`string? ModelId`	`ModelID string`
`wireModel`	`wireModel?: string`	`wire_model: str`	`string? WireModel`	`WireModel string`
`maxInputTokens`	`maxInputTokens?: number`	`max_input_tokens: int`	`int? MaxInputTokens`	`MaxInputTokens int`
`maxOutputTokens`	`maxOutputTokens?: number`	`max_output_tokens: int`	`int? MaxOutputTokens`	`MaxOutputTokens int`

All fields are optional.

modelId — Well-known model ID used to look up agent configuration (tools, prompts, reasoning behavior) and default token limits from the capability catalog. Useful for fine-tunes that should inherit a base model''s configuration. Defaults to the session''s configured model (SessionConfig.model) when unset.
wireModel — Model identifier sent to the provider API for inference. Use this when the name your provider knows (e.g. an Azure deployment name or custom fine-tune name) differs from the well-known model ID. Defaults to the session''s configured model when unset.
maxPromptTokens — Maximum tokens allowed in the prompt for a single request. The runtime triggers conversation compaction before sending a request when the prompt exceeds this limit.
maxOutputTokens — Maximum tokens the model can generate in a single response.

modelId and wireModel are independent knobs — set just one or both depending on whether you need to override config lookup, wire transmission, or both.

Changes

Node.js (nodejs/src/types.ts) — Added fields to ProviderConfig interface
Python (python/copilot/session.py) — Added fields to ProviderConfig TypedDict
Python (python/copilot/client.py) — Updated _convert_provider_to_wire_format to map new snake_case fields to camelCase
.NET (dotnet/src/Types.cs) — Added nullable properties with [JsonPropertyName] attributes
Go (go/types.go) — Added fields with json:"...,omitempty" tags

Testing

Extended existing provider-forwarding/serialization unit tests across all four SDKs to cover the new fields:

Node.js (nodejs/test/client.test.ts) — Existing forwards provider headers tests for session.create / session.resume now also assert all four new fields on the wire payload
Python (python/test_client.py) — Same — extended test_create_session_forwards_provider_headers / _resume_ to assert camelCase mapping for all four new fields
.NET (dotnet/test/Unit/SerializationTests.cs) — Extended ProviderConfig_CanSerializeHeaders_WithSdkOptions to round-trip all four new fields with correct JSON property names
Go (go/types_test.go) — Added TestProviderConfig_JSONIncludesAllFields (correct JSON tags) and TestProviderConfig_JSONOmitsUnsetTokenFields (omitempty works)

All four SDKs build clean (tsc --noEmit, go build ./..., dotnet build, python -c "import copilot") and all unit tests pass.

Copilot

Pull request overview

This PR adds four optional token-limit configuration fields to ProviderConfig across the Node.js, Python, Go, and .NET SDKs so BYOK/custom-provider users can override model token limits and/or specify an alternate model ID for capability-catalog limit lookup.

Changes:

Added maxOutputTokens, maxPromptTokens, maxContextWindowTokens, and modelLimitsId to ProviderConfig in Node.js, Go, and .NET.
Added corresponding snake_case fields to Python ProviderConfig and mapped them to camelCase wire keys in the Python client.
Documented the behavior/precedence of these fields via inline comments/XML docs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`nodejs/src/types.ts`	Extends TS `ProviderConfig` interface with optional token limit fields and `modelLimitsId`.
`python/copilot/session.py`	Extends Python `ProviderConfig` `TypedDict` with optional token-limit and model-limit lookup fields.
`python/copilot/client.py`	Maps new snake_case Python fields onto the expected camelCase wire names.
`go/types.go`	Extends Go `ProviderConfig` struct with new JSON fields (omitempty).
`dotnet/src/Types.cs`	Extends .NET `ProviderConfig` with nullable properties and JSON property names.

stephentoub · 2026-03-31T18:48:45Z

+    /// When set, takes precedence over the default limit resolved from the model's capability catalog entry.
+    /// </summary>
+    [JsonPropertyName("maxPromptTokens")]
+    public int? MaxPromptTokens { get; set; }


Should this be called MaxInputTokens instead of MaxPromptTokens?

Does this include cached tokens?

Same question as above... is this about one request or across a sequence of calls?

Per-request, but not sent to the API. The runtime uses this internally to decide when to truncate or compact conversation history before each LLM call. "Prompt" here means everything sent to the model in one request: system message, full conversation history up to that point, tool definitions, and the new user message. Cached tokens are counted toward the limit.

The name matches the upstream CAPI /models field (max_prompt_tokens), though MaxInputTokens would also be reasonable.

I'd prefer MaxInputTokens. That's the more modern terminology, right? And it maps to what we show in the CLI UI?

We could change this; it would just require changing the runtime representation as well (including the COPILOT_PROVIDER_MAX_PROMPT_TOKENS environment variable, which would probably need to be renamed).

Changing the sdk name wouldn't, right? Only if we also wanted to change the wire name?

Oh, if we just changed the public API but configured it to serialize with the maxInputTokens naming? Yeah that would work.

rramos-seidor · 2026-04-13T14:04:33Z

Hi @MackinnonBuck! Just wanted to follow up on this PR — is it still pending review? Adding token limit fields to ProviderConfig across all SDKs seems like a key capability for BYOK users who need to configure custom providers with specific limits. Would be great to see this move forward. Is there anything blocking it from being marked as ready for review? Thanks.

stephentoub · 2026-05-04T01:04:46Z

@MackinnonBuck, are you still working on this?

MackinnonBuck · 2026-05-04T01:17:29Z

@stephentoub Thanks for the ping. I was holding off on this because we've had requests on the CLI side to allow for multiple provider configs / models per provider config, and I wanted to make sure we were properly accounting for that on the SDK APIs. However, adding new fields to the existing provider config SDK type probably doesn't make it any harder for us to adjust the API later, so it's probably fine to take this now. I'll work on getting this in a ready-to-review state tomorrow.

Adds the following optional fields to ProviderConfig across all SDKs: - modelId: well-known model ID for agent config + token limit lookup - wireModel: model name sent to the provider API for inference - maxPromptTokens: prompt token cap (triggers compaction) - maxOutputTokens: response token cap Both modelId and wireModel default to the session's configured model when unset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extends existing provider-forwarding/serialization tests across all 4 SDKs to cover modelId, wireModel, maxPromptTokens, and maxOutputTokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

stephentoub · 2026-05-04T18:41:03Z

+    /// when not explicitly set.
+    /// </summary>
+    [JsonPropertyName("modelId")]
+    public string? ModelId { get; set; }


So there's:
SessionConfig.Model
ProviderConfig.ModelId
ProviderConfig.WireModel

Help me understand the relationship? If I specify SessionConfig.Model, it's used as the default for both options on ProviderConfig, and those options on provider config then represent the two different groupings in which a model would be used, such that I can override one of them? Is there any situation where I would specify the same model ID for both ModelId and WireModel?

If only ProviderConfig.ModelId is specified, then that controls multiple things:

The model name used by the runtime to determine model limits and model-specific agent configuration

The model name sent to the custom provider for inference

If the model provider recognizes a model name that doesn't match the model ID known by the runtime, then ProviderConfig.WireModel can specify that.

SessionConfig.Model acts a default in case neither option is specified.

Is there any situation where I would specify the same model ID for both ModelId and WireModel?

It has the same effect as just specifying ModelId, so it's not really necessary.

stephentoub · 2026-05-04T18:41:42Z

+    /// when not explicitly set.
+    /// </summary>
+    [JsonPropertyName("wireModel")]
+    public string? WireModel { get; set; }


Out of the three:
SessionConfig.Model
ProviderConfig.ModelId
ProviderConfig.WireModel

what's the meaning behind ProviderConfig.ModelId having an "Id" suffix and the other two not?

I figured the "ID" more strongly implied that the value was identifying a well-known model kind. We could also consider ModelFamily? It would just require changing the runtime as well.

Adds two end-to-end tests per SDK (Node, Python, Go, .NET) that exercise the new ProviderConfig fields against the replaying CAPI proxy: - should_forward_provider_wire_model_and_max_output_tokens: verifies wireModel overrides the wire request model and maxOutputTokens is forwarded as max_tokens. - should_use_provider_model_id_as_wire_model: verifies modelId acts as the wire model when wireModel is unspecified and SessionConfig.Model is omitted. Also adds MaxTokens to the Go and .NET ChatCompletionRequest harness types so the assertion is observable, and ships two shared snapshot YAMLs under test/snapshots/session_config/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The OpenAI BYOK provider code path in the CLI does not echo the configured maxOutputTokens as max_tokens on the wire request body (it's used internally for token budgeting and only appears on Anthropic-style requests). The new wire model E2E test asserted on max_tokens in the captured chat completion request, which always returned undefined/nil and failed across all four SDKs. Rename the test to `should forward provider wire model'' and drop the wire-side max_tokens assertion. The test still sets maxOutputTokens to confirm the SDK serializes the field without errors; per-SDK unit tests already cover ProviderConfig serialization in detail. Also drop the now-unused MaxTokens field from the Go and .NET harness ChatCompletionRequest types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Renames the SDK-facing ProviderConfig field across all four languages while preserving the wire JSON key as maxPromptTokens: - .NET: MaxPromptTokens -> MaxInputTokens (JsonPropertyName unchanged) - Go: MaxPromptTokens -> MaxInputTokens (json tag unchanged) - Python: max_prompt_tokens -> max_input_tokens (wire conversion in _convert_provider_to_wire_format unchanged) - Node: maxPromptTokens -> maxInputTokens; adds a small toWireProviderConfig helper in client.ts that remaps the field before sending session.create / session.resume. Also rewrites the doc comments for modelId, wireModel, maxInputTokens, and maxOutputTokens to make the priority order clear: WireModel falls back to ModelId falls back to SessionConfig.Model, and ModelId drives both runtime configuration lookup and the wire model when WireModel is unset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove 'using var' from three ClientE2ETests that also call ForceStopAsync in their finally block. The double-disposal (using Dispose → DisposeAsync → ForceStopAsync, plus the explicit ForceStopAsync) races on Windows when the CLI process/pipes are mid-teardown, causing OperationCanceledException to bubble up and fail an otherwise-passing test. Matches the existing pattern in SessionFsE2ETests where ForceStopAsync is wrapped in try-catch to swallow teardown-only exceptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This reverts commit 6c794c4.

…byok-provider-token-limits

github-actions · 2026-05-05T21:15:29Z

Cross-SDK Consistency Review ✅

All four SDK implementations (Node.js, Python, Go, .NET) are consistent in this PR. Specifically:

Field names: All four SDKs use the same semantic naming (modelId/model_id/ModelId/ModelID, wireModel/wire_model/WireModel/WireModel, maxInputTokens/max_input_tokens/MaxInputTokens/MaxInputTokens, maxOutputTokens/max_output_tokens/MaxOutputTokens/MaxOutputTokens) following each language's idioms.
Wire format: All four SDKs serialize to the same JSON wire keys (modelId, wireModel, maxPromptTokens, maxOutputTokens).
Optionality: All fields are optional/nullable in all SDKs.
Test coverage: Unit tests and E2E tests added for all four SDKs.

Minor PR description note

The table in the PR summary has a small labeling inconsistency in the "Wire name" column — the row for the prompt-token limit lists maxInputTokens as the wire name, but the actual JSON wire key (used by all four SDKs) is maxPromptTokens. The description text below the table correctly documents maxPromptTokens, so this is just a table header mix-up and doesn't affect the code.

Generated by SDK Consistency Review Agent for issue #966 · ● 500.2K · ◷

Brought in 12 commits from origin/main, including CLI bumps to 1.0.41-0 and 1.0.41-1, plus upstream PR #966 ("Add provider model and token limit overrides to ProviderConfig"). One trivial codegen diff (single doc-comment update on `CustomAgentsUpdatedAgent.tools` for the new "or null when all tools are available" semantics). PR #966 added four new fields to ProviderConfig across all SDKs: - `model_id: Option<String>` (well-known model ID for agent config + token limit lookup; falls back to SessionConfig::model) - `wire_model: Option<String>` (model name sent to provider API for inference; falls back to model_id, then to SessionConfig::model) - `max_prompt_tokens: Option<i64>` (overrides resolved model's default max prompt tokens; triggers compaction) - `max_output_tokens: Option<i64>` (overrides resolved model's default max output tokens; truncates response) Plus matching `with_*` builders. Wire-shape: camelCase (`modelId`/`wireModel`/`maxPromptTokens`/`maxOutputTokens`), skip_serializing_if when unset. Extended `provider_config_builder_composes` test to exercise all four fields and assert their wire shape (camelCase + omission). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 31, 2026 17:48

MackinnonBuck requested a review from a team as a code owner March 31, 2026 17:48

Copilot started reviewing on behalf of MackinnonBuck March 31, 2026 17:48 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Comment thread go/types.go Outdated

Comment thread python/copilot/client.py Outdated

This comment has been minimized.

Sign in to view

MackinnonBuck commented Mar 31, 2026

View reviewed changes

Comment thread dotnet/src/Types.cs Outdated

stephentoub reviewed Mar 31, 2026

View reviewed changes

Comment thread dotnet/src/Types.cs Outdated

stephentoub reviewed Mar 31, 2026

View reviewed changes

Comment thread dotnet/src/Types.cs Outdated

This comment has been minimized.

Sign in to view

MackinnonBuck marked this pull request as draft April 2, 2026 16:42

MackinnonBuck force-pushed the mackinnonbuck/byok-provider-token-limits branch from 47d0781 to f993631 Compare May 4, 2026 17:22

Add tests for new ProviderConfig fields

09c17f7

Extends existing provider-forwarding/serialization tests across all 4 SDKs to cover modelId, wireModel, maxPromptTokens, and maxOutputTokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

MackinnonBuck force-pushed the mackinnonbuck/byok-provider-token-limits branch from 4778c24 to 09c17f7 Compare May 4, 2026 17:34

MackinnonBuck changed the title ~~Add token limit fields to ProviderConfig across all SDKs~~ Add provider model and token limit overrides to ProviderConfig May 4, 2026

MackinnonBuck marked this pull request as ready for review May 4, 2026 17:38

This comment has been minimized.

Sign in to view

stephentoub reviewed May 4, 2026

View reviewed changes

Comment thread dotnet/src/Types.cs

stephentoub approved these changes May 4, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

patniko approved these changes May 5, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

github-code-quality Bot found potential problems May 5, 2026

View reviewed changes

MackinnonBuck added 2 commits May 5, 2026 14:09

Revert "Fix Windows teardown flake in ClientE2ETests"

fcbec67

This reverts commit 6c794c4.

Merge remote-tracking branch 'origin/main' into pr/966/mackinnonbuck/…

6c2212b

…byok-provider-token-limits

MackinnonBuck added this pull request to the merge queue May 5, 2026

Merged via the queue into main with commit 58cf64d May 5, 2026
35 checks passed

MackinnonBuck deleted the mackinnonbuck/byok-provider-token-limits branch May 5, 2026 21:25

Conversation

MackinnonBuck commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New fields

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

rramos-seidor commented Apr 13, 2026

Uh oh!

stephentoub commented May 4, 2026

Uh oh!

MackinnonBuck commented May 4, 2026

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 5, 2026

Cross-SDK Consistency Review ✅

Minor PR description note

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

MackinnonBuck commented Mar 31, 2026 •

edited

Loading