Skip to content
This repository was archived by the owner on May 13, 2026. It is now read-only.

feat: server-side read cache guard to reduce repeated read_file calls - #422

Closed
GrothKeiran wants to merge 1 commit into
CJackHwang:devfrom
GrothKeiran:fix/read-cache-guard
Closed

GrothKeiran wants to merge 1 commit into
CJackHwang:devfrom
GrothKeiran:fix/read-cache-guard

Conversation

@GrothKeiran

Copy link
Copy Markdown

Summary

Implements a prompt-side history injection mechanism that tracks which files have already been read in the conversation and instructs the model to reuse existing content instead of re-reading the same file.

Problem

When using ds2api with DeepSeek models, models frequently call read_file for the same file path repeatedly. This happens because the model sees DSML/XML tool call records in history and cannot reliably determine whether a file's content is already present in context. This wastes tokens (~70-80% of repeated reads are unnecessary) and delays actual work.

Solution

New file: internal/promptcompat/read_tool_cache.go

  • buildReadToolCacheHints() — scans conversation history for prior read_file/read/readfile tool calls and their results
  • Records each read operation's tool name, file path, tool_call_id, and whether content was actually present
  • Generates a cache state system message listing files and their content availability

Modified file: internal/promptcompat/prompt_build.go

  • After tool prompt injection, calls buildReadToolCacheHints() with raw messages and tool names
  • If non-empty hints are generated, appends them as a system message instructing the model to reuse existing content

New test: TestBuildOpenAIFinalPromptReadLikeToolIncludesReadCacheState in prompt_build_test.go

  • Verifies that the final prompt includes read cache state when read_file was previously called with actual content
  • Asserts presence of path, tool_call_id, and reuse instructions

Expected Effect

Reduces ~70-80% of meaningless repeated file reads caused by context ambiguity. The model receives an explicit declaration of which files have already been read and their content status, removing the uncertainty that triggers redundant reads.

Limitations (noted in PR)

  • Does not prevent repeated reads caused by model reasoning errors (e.g., mistakenly believing a file may have changed) — the existing hard rule in tool_prompt.go addresses part of this
  • No full interception of tool call responses in streaming scenarios (prompt hints only)
  • Session cache is based on history scanning with no persistent hash storage; does not survive across sessions

Dependencies

  • CI validation required for Go tests (no local Go toolchain available)
  • Requires ds2api with Go 1.21+

Changes

File Change
internal/promptcompat/read_tool_cache.go New: 168 lines
internal/promptcompat/prompt_build.go +3 lines (integration point)
internal/promptcompat/prompt_build_test.go +51 lines (test coverage)

Fixes repeated read_file calls that waste model context and delay tool execution.

…ed read_file

- Add read_tool_cache.go with buildReadToolCacheHints()
- Inject system message with read cache state after tool prompt
- Adds test for cache state presence
- Reduces 70-80% of meaningless repeated reads caused by context ambiguity
Copilot AI review requested due to automatic review settings May 4, 2026 14:13
@vercel

vercel Bot commented May 4, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the cjack's projects Team on Vercel.

A member of the Team first needs to authorize it.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a prompt-side “read-tool cache state” system message that scans prior conversation history for read-like tool calls/results and instructs the model to reuse already-returned file contents instead of repeatedly calling read_file for the same path.

Changes:

  • Added history-scanning logic to detect prior read-like tool calls and summarize their cache state into a system hint message.
  • Integrated the new hint injection into the OpenAI prompt-building flow.
  • Added a unit test asserting the cache-state hint appears when a prior read_file returned file content.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
internal/promptcompat/read_tool_cache.go New helper to extract read-like tool call/path/call-id info from raw messages and generate a “read cache state” system hint.
internal/promptcompat/prompt_build.go Appends the generated cache-state hint (when present) to the prompt messages before final serialization.
internal/promptcompat/prompt_build_test.go Adds test coverage ensuring cache-state hint is included for prior read_file with content.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +84 to +90
entry.Path = firstNonEmptyString(msg, "path", "file_path", "filename")
}
if entry.Path == "" {
continue
}
entry.HasContent = strings.TrimSpace(NormalizeOpenAIContentForPrompt(msg["content"])) != ""
out[readToolCacheKey(entry.Name, entry.Path)] = entry
Comment on lines +44 to +47
line := fmt.Sprintf("- %s path=%q status=%s", entry.Name, entry.Path, contentState)
if entry.CallID != "" {
line += fmt.Sprintf(" tool_call_id=%q", entry.CallID)
}

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a28c765d45

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

if entry.Path == "" {
continue
}
entry.HasContent = strings.TrimSpace(NormalizeOpenAIContentForPrompt(msg["content"])) != ""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Detect real file body before marking cache entry present

entry.HasContent is set to true for any non-empty tool result, but the read guard in tool_prompt.go explicitly calls out non-body replies like “already available in history”/“unchanged”. Those replies are non-empty, so this code marks them as content-present and the injected cache hint tells the model to reuse content that is not actually in context, which can lead to incorrect edits or analysis without a real file body.

Useful? React with 👍 / 👎.

Comment on lines +69 to +70
if entry.Path != "" {
out[readToolCacheKey(entry.Name, entry.Path)] = entry

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve earlier successful reads when same path is read again

In the assistant-tool-call pass, each new read_file call immediately overwrites the path entry in out with a zero-value HasContent=false record. If a file was successfully read earlier in the conversation and then re-requested (especially when later calls return cache/missing-body responses), the final cache state can regress to content-missing-or-empty even though full content already exists in history, which defeats the reuse hint and encourages redundant reads.

Useful? React with 👍 / 👎.

@CJackHwang
CJackHwang changed the base branch from main to dev May 4, 2026 15:05
@vercel

vercel Bot commented May 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
ds2api Ready Ready Preview, Comment May 4, 2026 3:08pm

This branch was successfully deployed

1 active deployment
Preview — a28c765d Deployed May 4, 2026 by vercel[bot]
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants