Add prompt cache support and implement for Anthropic

[arunkumarry]

What this does

Adds prompt caching support for Anthropic Claude models via a cache_point: keyword on Chat#with_instructions and Chat#ask. When a message is marked as a cache point, the gem injects Anthropic's cache_control: { type: 'ephemeral' } header on the last content block of that message and automatically adds the required anthropic-beta: prompt-caching-2024-07-31 request header. The static portion of the prompt is cached server-side by Anthropic for 5 minutes, reducing input token costs on repeated calls.

Fixes #706

Usage

chat = RubyLLM.chat(model: 'claude-haiku-4-5')
chat.with_instructions(large_static_system_prompt, cache_point: true)

response = chat.ask(user_message)

puts response.input_tokens        # total tokens processed
puts response.cached_tokens       # tokens served from cache (billed at 10x lower rate)
puts response.cache_creation_tokens # tokens written to cache on first call

Multiple cache points are supported (up to Anthropic's limit of 4 per request):

chat = RubyLLM.chat(model: 'claude-sonnet-4-5')
  .with_instructions(static_prefix, cache_point: true)
  .with_instructions(session_config, append: true, cache_point: true)

chat.ask(dynamic_user_query)

Cache points on ask are also supported for caching user messages:

chat.ask(large_static_user_context, cache_point: true)

Future extensibility

The cache_point attribute on Message is provider agnostic. Adding support for other providers requires only provider-specific formatting logic:

For providers with inline cache markers (like Anthropic):

Override complete method to add any required headers/beta flags when messages.any?(&:cache_point?)
Add an inject_cache_* helper in the provider's Chat module that modifies content blocks
Call the helper in the message formatting methods when msg.cache_point?
For providers with separate cache APIs (like Gemini's Context Caching):

Override complete to manage the cache lifecycle (create → reuse → retry on expiry)
Modify render_payload to accept a cached_content_name: parameter
When the name is present, split messages at the last cache point and only send the dynamic suffix inline
For providers without caching support:

No changes needed cache_point flags are silently ignored, preserving existing behavior
The core Message and Chat changes are already in place. Future PRs for Gemini, OpenAI (when they add caching), or other providers only need to touch their respective provider modules.

Type of change

• Bug fix
• New feature
• Breaking change
• Documentation
• Performance improvement

Scope check

• I read the Contributing Guide
• This aligns with RubyLLM's focus on LLM communication
• This isn't application-specific logic that belongs in user code
• This benefits most users, not just my specific use case

Required for new features

• I opened an issue before writing code and received maintainer approval
• Linked issue: [FEATURE] Add prompt caching support for providers(Currently for Anthropic) #706

Quality check

• I ran overcommit --install and all hooks pass
  There are existing rubocop offenses in - spec/ruby_llm/generators/chat_ui_generator_spec.rb
  There are existing Flay offenses in following files - spec/ruby_llm/generators/chat_ui_generator_spec.rb and lib/ruby_llm/error.rb
• I tested my changes thoroughly
  • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
  • All tests pass: bundle exec rspec
• I updated documentation if needed
• I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

• I used AI tools to help write this code
• I have reviewed and understand all generated code (required if above is checked)

API changes

• Breaking change
• New public methods/classes
• Changed method signatures
• No API changes

[arunkumarry]

After the initial Anthropic implementation, I extended the same cache_point: API to AWS Bedrock as well. The usage is identical:

chat = RubyLLM.chat(
  model: "anthropic.claude-haiku-4-5-20251001-v1:0",
  provider: :bedrock,
  assume_model_exists: true
)


chat.with_instructions(large_static_context, cache_point: true)
response = chat.ask(user_message)

puts response.cached_tokens         # cacheReadInputTokens
puts response.cache_creation_tokens # cacheWriteInputTokens

Bedrock uses a different wire format than direct Anthropic, instead of merging cache_control into the last content block, it appends a separate cachePoint block:

{ "cachePoint": { "type": "default" } }

Changes:
Bedrock::Chat#render_system - appends cachePoint block after system message content when cache_point?
Bedrock::Chat#render_message_content - appends cachePoint block after message content when cache_point?, extracted tool use block rendering into append_tool_use_blocks helper to keep complexity in check
Token reporting (cacheReadInputTokens → cached_tokens, cacheWriteInputTokens → cache_creation_tokens) was already implemented in parse_completion_response - no changes needed there
Added
chat_cache_point_spec.rb with unit tests covering system messages, user messages, and multiple cache points