Skip to content

docs(token-budget): make the clip_tokens docstring examples match its output - #5111

Open
thejesh23 wants to merge 1 commit into
The-PR-Agent:mainfrom
thejesh23:fix/clip-tokens-docstring-examples
Open

thejesh23 wants to merge 1 commit into
The-PR-Agent:mainfrom
thejesh23:fix/clip-tokens-docstring-examples

Conversation

@thejesh23

@thejesh23 thejesh23 commented Nov 22, 2026 •

Copy link
Copy Markdown

User description

Summary

The Examples section of the clip_tokens docstring (pr_agent/algo/token_budget.py) shows outputs the function does not produce:

  • Examples 1 and 2: "This is a sample text that might be too long" is exactly 10 tokens, so clip_tokens(text, max_tokens=10) returns it unchanged rather than This is a sample....
  • Truncation marker: the function appends "\n...(truncated)", not ... at the end of the line followed by a (truncated) line. The Args section already states the right marker; the examples contradict it.
  • delete_last_line example: "Line 1\nLine 2\nLine 3" cannot keep two lines within 3 tokens; the real output is 'Line\n...(truncated)'.

Pytest doesn't collect doctests here (no --doctest-modules), so nothing caught the drift.

Actual output on main @ db4c4ad (Python 3.12, o200k_base)

clip_tokens("This is a sample text that might be too long", max_tokens=10)                       -> 'This is a sample text that might be too long'
clip_tokens(..., max_tokens=10, add_three_dots=False)                                            -> 'This is a sample text that might be too long'
clip_tokens(..., max_tokens=5, num_input_tokens=15)                                              -> 'This is a sam\n...(truncated)'
clip_tokens("Line 1\nLine 2\nLine 3", max_tokens=3, delete_last_line=True)                       -> 'Line\n...(truncated)'

Change

  • Docstring only in pr_agent/algo/token_budget.py; no code change. The examples now use a 15-token sample that does get clipped, and show the returned value (repr) so the \n...(truncated) marker and the trailing space are visible. The delete_last_line example uses max_tokens=9, which keeps Line 1 and Line 2 and drops the partial third line, as the old example intended.
  • New test TestClipTokens::test_docstring_examples_match_actual_output runs the docstring examples with doctest, with the tokenizer pinned to o200k_base (what the default model resolves to) so the result does not depend on the configured model. On main it fails with 4 of 10 examples wrong; with this change it passes.

Verification

  • tests/unittest/test_clip_tokens.py and test_clip_tokens_invalid_budget.py: 30 passed.
  • Full unit suite on Python 3.12.13 (macOS): PYTHONPATH=. pytest -q tests/unittest -> 18718 passed, 66 skipped, 0 failed (baseline on main @ db4c4ad: 18717 passed, 66 skipped; the 1 extra is the new test). tests/packaging/test_help_docs_package.py: 4 passed; uv lock --check: clean. Not run locally: Python 3.13/3.14, the Docker build, and scripts/review_context.
  • uv run ruff check and uvx pre-commit run --files on both changed files: clean.

AI disclosure: this change was prepared by an AI coding agent (Claude Code) and submitted with the authorization of the PR author.

🤖 Generated with Claude Code


PR Type

Documentation, Tests


Description

  • Correct clip_tokens examples to match actual outputs.

  • Add doctest regression coverage with pinned tokenizer.


File Walkthrough

Relevant files
Documentation
token_budget.py
Correct clipping examples and truncation marker output     

pr_agent/algo/token_budget.py

  • Replace the sample text with a 15-token input that actually requires
    clipping.
  • Show returned string representations, including trailing spaces and
    the \n...(truncated) marker.
  • Correct precomputed-token and line-deletion examples without changing
    runtime behavior.
+11/-20 
Tests
test_clip_tokens.py
Add deterministic doctest coverage for clipping examples 

tests/unittest/test_clip_tokens.py

  • Execute clip_tokens docstring examples using Python's doctest module.
  • Pin TokenEncoder.get_token_encoder to o200k_base for model-independent
    results.
  • Assert that examples execute and all expected outputs match.
+14/-0   


⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 5,752 in / 314 out / 6,066 total
  • Time cost: 6.2s
  • AI calls: 1

… output

The examples clipped a 10-token sample at max_tokens=10 (returned unchanged), showed the
truncation marker as '...' plus a '(truncated)' line instead of '\n...(truncated)', and kept two
lines of a multi-line text within 3 tokens. Replace them with outputs the function actually
returns, and run them as a doctest in the unit suite so they stay accurate.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 22, 2026
@the-pr-agent-bot

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
⚠️ Risk level: Low
✅ Merge recommendation: No concerns found
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 11,532 in / 135 out / 11,667 total
  • Time cost: 3.5s
  • AI calls: 1

@the-pr-agent-bot

Copy link
Copy Markdown

PR Code Suggestions ✨

No code suggestions found for the PR.


⚙️ Agent run details
  • Model: openai/gpt-6.1-sol
  • Tokens: 5,573 in / 134 out / 5,707 total
  • Time cost: 4.5s
  • AI calls: 1

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation Review effort 1/5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant