fix(bedrock): inline mid-conversation system messages to preserve prompt cache#4534
Open
mickgvirtu wants to merge 1 commit into
Open
fix(bedrock): inline mid-conversation system messages to preserve prompt cache#4534mickgvirtu wants to merge 1 commit into
mickgvirtu wants to merge 1 commit into
Conversation
|
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughSummary by CodeRabbit
Walkthrough
ChangesBedrock inline system reminders and tool CachePoint
Sequence DiagramssequenceDiagram
participant Client
participant BedrockConverter
participant SystemMessage
participant ToolCall
participant ToolResult
Client->>BedrockConverter: ConvertBifrostMessagesToBedrockMessages(inlineSystemReminders=true)
BedrockConverter->>SystemMessage: Leading system messages → hoist to system block
BedrockConverter->>SystemMessage: Mid-conversation system messages → wrap as <system-reminder> user turn
BedrockConverter->>ToolCall: Emit tool call, append CachePoint if CacheControl present
BedrockConverter->>ToolResult: Emit tool result, append CachePoint if CacheControl present
BedrockConverter-->>Client: messages[], systemBlocks[], error
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
core/providers/bedrock/bedrock_test.go (1)
6471-6477: ⚡ Quick winCover the lone
developerbranch too.The test comment and converter predicate both cover
systemanddeveloper, but this test only exercisessystemReminderTextMsg; add a developer case so the single-message early return cannot regress forrole=developer.Suggested test expansion
func TestLoneSystemMessageReturnsUserMessage(t *testing.T) { - for _, inline := range []bool{true, false} { - input := []schemas.ResponsesMessage{systemReminderTextMsg("You are Claude Code.")} - messages, systemMessages, err := bedrock.ConvertBifrostMessagesToBedrockMessages(context.Background(), input, inline) - require.NoError(t, err) - assert.Empty(t, systemMessages, "lone system message must not populate the system block (inline=%v)", inline) - require.Len(t, messages, 1, "lone system message must yield exactly one message (inline=%v)", inline) - assert.Equal(t, bedrock.BedrockMessageRoleUser, messages[0].Role) - } + cases := []struct { + name string + msg schemas.ResponsesMessage + }{ + {name: "system", msg: systemReminderTextMsg("You are Claude Code.")}, + {name: "developer", msg: developerReminderTextMsg("Developer instructions.")}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + for _, inline := range []bool{true, false} { + input := []schemas.ResponsesMessage{tc.msg} + messages, systemMessages, err := bedrock.ConvertBifrostMessagesToBedrockMessages(context.Background(), input, inline) + require.NoError(t, err) + assert.Empty(t, systemMessages, "lone %s message must not populate the system block (inline=%v)", tc.name, inline) + require.Len(t, messages, 1, "lone %s message must yield exactly one message (inline=%v)", tc.name, inline) + assert.Equal(t, bedrock.BedrockMessageRoleUser, messages[0].Role) + } + }) + } }As per coding guidelines, Go changes should include deterministic tests and table-driven coverage for behavior changes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/providers/bedrock/bedrock_test.go` around lines 6471 - 6477, The test TestLoneSystemMessageReturnsUserMessage only covers the system message role by using systemReminderTextMsg, but the converter predicate and test comment indicate both system and developer roles should be handled. Expand the test to also cover the developer message role by adding a developer message case alongside the existing system message case. Use a table-driven approach or add a separate developer message input to ensure the single-message early return path in ConvertBifrostMessagesToBedrockMessages is exercised for both role types without regression.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@core/providers/bedrock/bedrock_test.go`:
- Around line 6471-6477: The test TestLoneSystemMessageReturnsUserMessage only
covers the system message role by using systemReminderTextMsg, but the converter
predicate and test comment indicate both system and developer roles should be
handled. Expand the test to also cover the developer message role by adding a
developer message case alongside the existing system message case. Use a
table-driven approach or add a separate developer message input to ensure the
single-message early return path in ConvertBifrostMessagesToBedrockMessages is
exercised for both role types without regression.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 16db7350-4432-40ec-b7cf-7f4942b4ecba
📒 Files selected for processing (2)
core/providers/bedrock/bedrock_test.gocore/providers/bedrock/responses.go
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…cache Bedrock's prompt cache is prefix-based: a mid-conversation role=system message (e.g. the reminders Claude Code injects) hoisted into the top-level system block grows that prefix every turn and collapses the cached conversation to the tools/system floor. This is the Bedrock counterpart of the native Anthropic provider's mid-conversation system support (SupportsMidConversationSystem) — Bedrock has no message-level system role, so the inlined message is rendered as a user turn. Gated by an inlineSystemReminders bool the caller computes via IsAnthropicModelFamily(ctx, model) (alias-aware), so non-Anthropic families keep the historical hoist-everything behavior. Tool-call/result cache_control breakpoints are preserved as CachePoint blocks carrying the requested TTL. Adds regression tests including a positive cache_control->CachePoint+TTL assertion, the lone-system early return, and the no-leading-system gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Author
|
Thanks — both addressed:
|
mickgvirtu
force-pushed
the
pr-bedrock-midconv-system-cache
branch
from
d0aa598 to
c8cc345
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #4068. On the Bedrock provider,
ConvertBifrostMessagesToBedrockMessageshoists everyrole:system/role:developermessage into Bedrock's top-levelsystemblock, regardless of position. Because Bedrock's Converse prompt cache is prefix-based, arole:systemmessage injected mid-conversation (e.g. the reminders Claude Code emits) grows thesystemprefix in front of the cached conversation and collapses cache reads to the system+tools floor — recurring on every such turn.Changes
system; messages appearing after the conversation starts are inlined in place. Non-Anthropic models keep the historical hoist-everything behavior.SupportsMidConversationSystemhandling. Bedrock has no message-levelsystemrole, so an inlined message is rendered as auserturn (wrapped in<system-reminder>…</system-reminder>, matching the convention clients already use for pre-wrapped reminders).inlineSystemReminders boolcomputed by the caller viaIsAnthropicModelFamily(ctx, model)(alias-aware, consistent with the other Anthropic gates in the file).cache_controlon tool calls/results is preserved as aCachePointcarrying the requested TTL.Type of change
Affected areas
How to test
go test ./core/providers/bedrock/Adds
TestMidConversationSystemReminderStaysInline,…HoistedForNonAnthropic,TestToolCacheControlBecomesCachePointWithTTL(positive TTL assertion), a lone-system early-return test, and a no-leading-system-block gate test.Issue #4068 has the full root-cause plus a real cache-read trace (cached tokens dropping to the system/tools floor and recovering after the prefix re-warms). Related native-side work: #4276, #3879.
Notes for reviewers: this re-adds a parameter the converter previously dropped — happy to thread it differently (e.g. derive from a typed context) if you prefer. The
<system-reminder>wrapping follows the client convention; open to gating it if you'd rather it not be implicit.