tracking: estimate_tokens docstring says chars/4 but code divides byte length

Summary

estimate_tokens in src/core/tracking.rs divides text.len() by 4 to approximate a token count. str::len() returns the UTF-8 byte length, not the character count. But the docstring right above the function says ~4 chars = 1 token and tokens = ceil(chars / 4). The documented unit (chars) and the implemented unit (bytes) don't match.

For ASCII text the two are identical, so nothing looks wrong. For any non-ASCII text (CJK, emoji, accented Latin) they diverge — one CJK char is 3 bytes, one emoji is 4.

The existing unit test test_estimate_tokens only feeds ASCII strings ("abcd", "hello world", "12345678"), where bytes == chars, so the mismatch is invisible to the test suite.

Environment

• rtk: 0.42.4 (built from master)
• OS: macOS (platform-independent — it's Rust str::len semantics, not OS behavior)
• rustc: 1.91 (MSRV per Cargo.toml)

Reproduction

The mismatch is at the source level. The function and its docstring (src/core/tracking.rs, around line 1272-1285):

/// Estimate token count from text using ~4 chars = 1 token heuristic.
///
/// # Formula
///
/// `tokens = ceil(chars / 4)`
pub fn estimate_tokens(text: &str) -> usize {
    // ~4 chars per token on average
    (text.len() as f64 / 4.0).ceil() as usize
}

text.len() is byte length. To see the divergence, run this standalone snippet (no rtk build needed):

cat <<'EOF' > /tmp/repro.rs
fn main() {
    let s = "你好世界测试"; // constructed test case, 6 chars
    println!("chars            : {}", s.chars().count());                          // 6
    println!("bytes            : {}", s.len());                                    // 18
    println!("docstring chars/4: {}", (s.chars().count() as f64 / 4.0).ceil() as usize); // 2
    println!("code     bytes/4 : {}", (s.len() as f64 / 4.0).ceil() as usize);          // 5
}
EOF
rustc /tmp/repro.rs -o /tmp/repro && /tmp/repro

Expected per the docstring (chars / 4): 2.
Actual from the code (bytes / 4): 5.

Then confirm the unit test only covers ASCII, so the bug can't surface there:

git clone https://github.com/rtk-ai/rtk && cd rtk
grep -n 'estimate_tokens' src/core/tracking.rs   # impl ~line 1284, test ~line 1431
cargo test --lib test_estimate_tokens             # passes — but every input is ASCII

test_estimate_tokens asserts on "abcd", "abcde", "a", "12345678" — all ASCII, so byte length equals char length on every case.

Impact

estimate_tokens is the path rtk uses to turn command output into a token count. TimedExecution::track (src/core/tracking.rs, ~line 1361) calls it for both the input and the output, and those numbers flow into the saved-tokens figures users see. For non-ASCII output (error messages with CJK, emoji in commit messages, accented names) the estimate no longer matches the chars / 4 formula the docstring promises, and the gap widens the more non-ASCII the text is.

I'm not claiming a specific real-world savings number here — the concrete, checkable problem is that the docstring and the implementation disagree on the unit.

Suggested fix

Pick one so the doc and the code agree:

• Match the docstring: use text.chars().count() instead of text.len(). One-line change, keeps the heuristic.
• Or update the docstring to say bytes / 4 if the byte heuristic is intentional. (For CJK this is arguably a closer multiplier, though neither is precise.)

Either way, add a non-ASCII case to test_estimate_tokens so the chosen unit is actually enforced — right now the test cannot tell the two apart:

// "你好" = 2 chars, 6 bytes
assert_eq!(estimate_tokens("你好"), 1); // chars/4 -> ceil(0.5)=1; bytes/4 -> ceil(1.5)=2

The docstring already notes "For precise counts, integrate with your LLM's tokenizer API" — a real tokenizer is the only way to get accurate counts, but that's a larger change and out of scope for this report.