fix(claude-code): retain the whole unprocessed suffix, not just its last turn

[merlinr68]

The invariant that breaks

Full-session retention selects the unprocessed transcript suffix using the persisted offset, then narrows it a second time:

messages_to_retain = all_messages[retention_progress.start_index :]
retain_full_window = retention_progress.start_index == 0        # ← narrows it again

retain_full_window=False makes prepare_retention_transcript() keep only the last turn. But commit_retention() advances the cursor to the end of the whole suffix either way:

if not transcript:
    commit_retention(session_id, len(all_messages), retention_progress.chunk_index)
    return

So the rule that should hold —

once plan_retention() establishes [start_index, len(all_messages)) as the unprocessed range, every message in it must be formatted, filtered by retention policy, or left uncommitted

— is violated: messages are dropped by a second positional window while the cursor records them as processed. That loss is permanent.

This holds on its own merits. Claude Code's transcript shape then makes it dramatically worse.

Why it's severe in Claude Code specifically

Tool results are stored as role: "user" messages. So the last_user_idx scan usually anchors on a tool result rather than the human prompt. With the resolved default retainToolCalls: false (both config.py DEFAULTS and settings.json set False; the True fallback at retain.py:149 is unreachable because load_config() always supplies the key), that message extracts to empty — prepare_retention_transcript returns (None, 0), and the entire retain is skipped while the cursor still advances.

In an agentic session the trailing message is a tool result most of the time, so this is the common case, not an edge case.

Measurements

Formatting the same 30-message unprocessed suffix from a real 1,500-message transcript:

config	captured
last-turn, tools off (current default)	0 chars — retain skipped
last-turn, tools on	523 chars, 1 msg
full-window, tools off	3,697 chars, 6 msgs
full-window, tools on	17,744 chars, 24 msgs

One session's actual retain history:

chunk	captured
first	54,317 chars
c1	2,319
c4	1,180
c5	287
c7	300

The first retain (start_index == 0) captures everything; every later one captures a fragment, regardless of how much work happened in between.

Fix

retain_full_window = True

messages_to_retain is already the not-yet-retained slice, so windowing it again is redundant at best and lossy at worst. There's no duplicate-send risk — the cursor, not the window, is what prevents that. Role, memory-tag and tool-content filtering are unchanged.

This also restores the behaviour the surrounding comments already describe (retain.py:108-115): "each retain only sends messages not retained by a previous Stop hook."

The commit_retention() on an empty transcript is deliberately left alone. With this fix, "empty" can only mean the entire unprocessed suffix was filtered out by policy, so advancing is correct — and not committing would make every later Stop reprocess the same suffix forever.

Tests

5 new tests in tests/test_retain_incremental_window.py. Every one drives a second retain, because the defect only appears once a cursor exists — on a fresh session start_index == 0 and the old expression evaluates True by accident. My first draft omitted this and passed against the unfixed code; mutation testing caught it.

• multiple human turns in one window are all sent
• a trailing tool result doesn't swallow the window (the core case)
• interleaved tool results keep all assistant reasoning
• already-retained messages are not re-sent (guards against over-correcting)
• a tool-only window sends nothing

Verification:

• 206 pass with the fix
• reverting the one-line change fails 3 of 5
• ruff check clean

Docstring

retain_full_window was documented as "retain all messages (chunked mode)", which invited exactly this misuse. Reworded to say it means the caller has already selected the complete window, and to warn that callers advancing a retention cursor must not pass False.

Related, not included

slice_last_turns_by_user_boundary() counts every role == "user" message as a turn boundary, so it has the same tool-result confusion. It affects chunked mode (retain.py:120) and recall context composition (content.py:81), so recallContextTurns: N can compose N tool results instead of N turns. That's a wider semantic change with a shared call site, so it deserves its own PR — happy to open one.

Same family as #2869 (SessionEnd retain cancelled at shutdown) and #2974 (retain never sent MemoryItem.timestamp).