Repository navigation
Conversation
… output The examples clipped a 10-token sample at max_tokens=10 (returned unchanged), showed the truncation marker as '...' plus a '(truncated)' line instead of '\n...(truncated)', and kept two lines of a multi-line text within 3 tokens. Replace them with outputs the function actually returns, and run them as a doctest in the unit suite so they stay accurate.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
⚙️ Agent run details
|
PR Code Suggestions ✨No code suggestions found for the PR. ⚙️ Agent run details
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Summary
The
Examplessection of theclip_tokensdocstring (pr_agent/algo/token_budget.py) shows outputs the function does not produce:"This is a sample text that might be too long"is exactly 10 tokens, soclip_tokens(text, max_tokens=10)returns it unchanged rather thanThis is a sample...."\n...(truncated)", not...at the end of the line followed by a(truncated)line. TheArgssection already states the right marker; the examples contradict it.delete_last_lineexample:"Line 1\nLine 2\nLine 3"cannot keep two lines within 3 tokens; the real output is'Line\n...(truncated)'.Pytest doesn't collect doctests here (no
--doctest-modules), so nothing caught the drift.Actual output on
main@ db4c4ad (Python 3.12,o200k_base)Change
pr_agent/algo/token_budget.py; no code change. The examples now use a 15-token sample that does get clipped, and show the returned value (repr) so the\n...(truncated)marker and the trailing space are visible. Thedelete_last_lineexample usesmax_tokens=9, which keepsLine 1andLine 2and drops the partial third line, as the old example intended.TestClipTokens::test_docstring_examples_match_actual_outputruns the docstring examples withdoctest, with the tokenizer pinned too200k_base(what the default model resolves to) so the result does not depend on the configured model. Onmainit fails with 4 of 10 examples wrong; with this change it passes.Verification
tests/unittest/test_clip_tokens.pyandtest_clip_tokens_invalid_budget.py: 30 passed.PYTHONPATH=. pytest -q tests/unittest-> 18718 passed, 66 skipped, 0 failed (baseline onmain@ db4c4ad: 18717 passed, 66 skipped; the 1 extra is the new test).tests/packaging/test_help_docs_package.py: 4 passed;uv lock --check: clean. Not run locally: Python 3.13/3.14, the Docker build, andscripts/review_context.uv run ruff checkanduvx pre-commit run --fileson both changed files: clean.AI disclosure: this change was prepared by an AI coding agent (Claude Code) and submitted with the authorization of the PR author.
🤖 Generated with Claude Code
PR Type
Documentation, Tests
Description
Correct
clip_tokensexamples to match actual outputs.Add doctest regression coverage with pinned tokenizer.
File Walkthrough
token_budget.py
Correct clipping examples and truncation marker outputpr_agent/algo/token_budget.py
clipping.
the
\n...(truncated)marker.runtime behavior.
test_clip_tokens.py
Add deterministic doctest coverage for clipping examplestests/unittest/test_clip_tokens.py
clip_tokensdocstring examples using Python'sdoctestmodule.TokenEncoder.get_token_encodertoo200k_basefor model-independentresults.
⚙️ Agent run details