Skip to content

fix: reduce repeated history processing and token estimation - #250

Merged
hetaoBackend merged 3 commits into
mainfrom
fix/history-processing-overhead
Sep 20, 2026
Merged

hetaoBackend merged 3 commits into
mainfrom
fix/history-processing-overhead

Conversation

@hetaoBackend

@hetaoBackend hetaoBackend commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Change

Long histories repeatedly paid for JavaScript SHA-256, eager fingerprinting of value-only snapshots, BPE tokenization of unchanged text, and full post-commit history rereads.

  • Use Node's native SHA-256 with the existing bounded UTF-8 chunk and update semantics.
  • Validate and detach external snapshots eagerly with a single native structuredClone of the entire input graph. Compute fingerprints on demand and reuse only an already-owned, deeply frozen root. Native graph cloning preserves shared references even across side-effecting getters and normalized class instances.
  • Reuse text token counts in an estimator-local LRU capped at 2,048 entries and 1,048,576 UTF-16 code units. Content edits use new keys; model projection and request budget calculations still run against current inputs. Tokenizer failures are not cached.
  • Consume the Session provider's verified post-write snapshot in the same operation lane and pass verified records into index synchronization. Legacy adapters returning void still trigger a strict reread. Explicit reads continue to observe external file edits and reject corruption.

This reduces repeated work; it does not eliminate every history scan or file read. No user-facing configuration or documentation changes are required.

Tradeoffs

  • The token cache retains text until eviction or estimator collection; the default estimator lives for the process lifetime. The limits bound entries and logical UTF-16 text length, not total RSS or container/backing-storage overhead. Workloads with frequently changing text may see lower hit rates; lower RSS is not guaranteed.
  • A provider returning a snapshot must return the current durable post-write snapshot from the same serialized operation. Structural validation cannot detect a stale but well-formed result from an incorrect adapter. The production Session provider performs its own reread and validation; adapters without that guarantee should return void to request the strict fallback read.

Validation

  • PASS: full pnpm verify on clean commit 761b4c683ab1e8dfac4edaa490cfe27f8644d592, macOS arm64 / Node v26.4.0. All 14 gates passed, including 3,886 capability tests across 144 files.
  • PASS: the upstream harness-perf-benchmark at 8ad1e09f7653d7e97b83b4f5b4d347ca2fa129c7, using its unchanged 100-turn, approximately 4 KiB/turn, 64-character chunk, zero-delay scenario and 100 ms process-tree sampler. Base a55d629 and this commit each received one excluded warmup and three serial, interleaved measured runs on Apple M5 / 24 GB / macOS 26.6.2 / Node v26.4.0 / Bun 1.4.2. Every run completed with 101 mock requests, 100 successful tool calls, exit code 0 and the same final response.
Median of three measured runs Base This commit Change
Duration (s) 18.62 9.32 -49.9%
CPU (core-s) 17.86 7.61 -57.4%
Sampled peak RSS (MiB) 884.47 736.52 -16.7%
Mean RSS (MiB) 619.71 550.19 -11.2%

These measurements are for this scripted workload and final commit; RSS is sampled and workload-dependent. They supersede the earlier snapshot-optimization measurements.

  • Regression tests cover UTF-8 boundaries, eager validation/detachment, lazy fingerprints, owned-root reuse, getter/shared-reference ordering, class-instance aliases, native proxy rejection, token cache edits/eviction/failures, commit validation/fallback, and real-file edits/corruption. The three aliasing cases reproduced failures before the snapshot-boundary repair.
  • NOT RUN: live-model acceptance, desktop acceptance, Linux and Windows locally. Remote CI status is separate from local verification. Local benchmark results do not establish absolute performance on another machine.

Publication and contribution checks

  • I have permission to contribute these changes under the existing licenses applicable to the changed files/packages; imported material and its provenance are identified and existing notices are preserved.
  • No credentials, account data, real user content, internal source history or private review material is included.
  • Added/removed source files were reviewed before regenerating release/public-source.json; new tests are declared in test/vitest-suites.json where applicable.
  • Shared English/Chinese documentation and capability/verification records are updated where applicable. Mock/offline results are not described as live-service acceptance.

Maintainer handoff

Publication scope or license changes: none beyond the registered regression test.

Shared-source port: pending.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@hetaoBackend
hetaoBackend merged commit 3b3310f into main Sep 20, 2026
12 checks passed
@hetaoBackend
hetaoBackend deleted the fix/history-processing-overhead branch September 20, 2026 12:01
@1anZhang

Copy link
Copy Markdown
Collaborator

Overall: this is a well-scoped performance optimization PR. I would approve with follow-up.

Main benefits:

  • Replaces the pure-JS SHA-256 implementation with Node’s native implementation.
  • Defers semantic fingerprinting while preserving eager detach/freeze and immutable subtree reuse.
  • Adds a bounded LRU cache for repeated token estimation.
  • Reuses the provider’s verified post-write snapshot to avoid redundant history reads.

Key concerns:

  1. The provider return contract changes from Promise<unknown> to CanonicalHistorySnapshot | void. Any legacy adapter returning a non-void, non-snapshot value will now fail validation after the write. Please confirm all adapters follow the new contract and add a compatibility regression test if needed.
  2. Reusing the same-lane snapshot assumes this lane is the only writer. Concurrent external edits could leave a larger stale snapshot/index window, although explicit reads still re-read and validate the file.
  3. The token cache is per estimator and assumes the encoder is pure. If tokenizers/configuration become model-specific later, the cache key must include that configuration.

Validation: remote CodeQL, source audit, and macOS/Ubuntu verify checks pass. The reported synthetic benchmark shows roughly 11.7–14.2x speedup for value-only snapshots and 2.9–3.2x for fingerprinting, but it is not an end-to-end or RSS benchmark.

Recommendation: merge after confirming provider-contract coverage; a production-adapter/compaction regression test would be a useful follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants