Skip to content

perf(javascript): reduce retained semantic analysis allocations - #1601

Merged
N0zoM1z0 merged 12 commits into
mainfrom
perf/javascript-semantic-ids-1563
Oct 10, 2026
Merged

N0zoM1z0 merged 12 commits into
mainfrom
perf/javascript-semantic-ids-1563

Conversation

@N0zoM1z0

@N0zoM1z0 N0zoM1z0 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Large JavaScript bundles retain concatenated semantic IDs, empty collections for every mutation observer, and an eager scope index for every AST node. Construct high-volume IDs with one join, materialize observer collections only when they acquire facts, and resolve lexical scopes through explicit boundaries and the existing AST parent index. Cache traversed ancestry on demand so repeated scope lookup remains linear in the controlled depth case.

Refs #1065 and #1563. This is an allocation reduction: no new source cutoff, skipped facts, result cap, schema, tool or authority change. The full-directory OOM remains; #1563 must stay open. The PR diff remains seven files and preserves the original implementation and independent cache follow-up history.

Validation:

  • Current head a2255f49 integrates main 16d9c5ad, including the existing fix(ci): preserve cleanup boundaries and bounded failure assertions #1618 cleanup/CI repair. Fresh runtime preparation, 85 affected browser/process-output/Qwen/EVM/firmware cases across seven files (2 host-only skips), and npm run check:fast pass. The preceding integration 30a33f92 passed 495 semantic cases across 20 files plus build/static checks; the seven changed files are identical to measured head 03994825.
  • Independent follow-up at fa85af8e: the depth regression fails before the ancestry cache and passes after it. At depths 80/160/320, AST-parent reads change from 7,161/27,081/105,321 to 332/652/1,292. Its verification reports 494 semantic tests across 21 files, all static/documentation/architecture/discovery checks, four unchanged complete-IR controls, and a real 674-byte public CLI/pinned-SDK MCP control preserving Evidence IDs, text/structured parity, source bytes and subsequent ping. These structural counts do not establish general throughput or billed-token improvements.
  • Real official Claude Code 2.1.296 member at 03994825: 4,762,637 bytes, SHA256 64516393b8f06dce5b74140781d43b1df4e743de3e283526a533508ca1c58147. Production domain analysis completes with 768 MiB old-space, retaining the complete 256,300,420-byte IR, SHA256 c109f190e1fb1719b73705096730bf4876338d191a9a07d88e3f6eaed019df04. Peak family RSS 1,006,104,576 bytes. Thirty additional complete-IR corpus digests matched the original source owners before the later integrations. Domain equality does not establish full public CLI success at that heap size.
  • Real pinned-SDK MCP at 03994825, parent 512/server 1024 MiB: complete 170-tool catalog; the completed member returns a typed oversized-transport constraint with retained Evidence. Summary, module page, exact node selection, one-node incoming trace (one match), ping, close/stale-reference and subsequent small analysis pass. Source SHA unchanged; Evidence ev_c11a23f39bc7f3f371eeea1c5754dbc2d3923b124b2ac83562c73112f27c87a9; peak family RSS 1,656,877,056 bytes. The application graph explicitly retains its existing ceiling/partial semantic coverage. These public proofs remain tied to the stated frozen revision.
  • Full original 2351-relevant-file directory at 30a33f92, including main dce81ca4 and all seven allocation/cache changes: still fatal heap exhaustion at chunk-kasbfbhj.js (1432/2351), exit 134 at 1024 MiB, zero CLI stdout, 177.39 seconds, peak family RSS 1,362,915,328 bytes. The external guard did not stop it. All 2589 extracted file digests were independently rechecked: zero mismatches. No causal speedup follows from single runs at different revisions.

Linux x64, Node 24.18.0/npm 11.16.0, CPUs 13/15, nice 10, one heavy lane, 2304 MiB process-family RSS guard and 2 GiB host reserve. Real Hopper/Ghidra and Windows/macOS verification were not performed for this JavaScript-only change. Final-head CI is pending; merge requires all checks on the exact final head to pass.

Separate follow-up: #1610's fixed source cutoff was reviewed through real full-directory CLI/MCP and a below-cutoff synthetic module and still permits host termination: #1610 (review) . No cutoff or worker protocol is included here. My overlapping test-only #1617 was closed as superseded by #1618; it adds no files to this PR.

@N0zoM1z0 N0zoM1z0 added area: cli CLI adapter and one-shot commands area: mcp MCP server, transport, and tool registration area: domain Domain logic, errors, and pure functions performance Performance regressions, scaling limits, and bounded-resource work labels Oct 10, 2026
@N0zoM1z0

Copy link
Copy Markdown
Collaborator Author

Independent review of fc8bfda6: the compiled analyzer preserves complete IR digests for four small nested-expression controls, and a 674-byte target preserves full public CLI/pinned-SDK MCP results and Evidence IDs relative to current main. However, nearest-scope lookup now repeatedly walks the same ancestry: depths 80/160/320 require 7,161/27,081/105,321 AST-parent reads, showing quadratic growth. The new regression test fails at the expected growth assertion. I am preparing an on-demand path-cache follow-up on a separate review branch, retaining lazy allocation and validating complete outputs and lexical-scope semantics. These are structural operation counts, not a general throughput or token-cost claim. Refs #1065 and #1563.

@N0zoM1z0
N0zoM1z0 marked this pull request as ready for review October 10, 2026 17:25
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@N0zoM1z0

Copy link
Copy Markdown
Collaborator Author

Final-head CI at 03994825c118ced5eef90d62d5757c49d76147bd is currently red in shard 1. I independently reproduced all six failures with npm run test:focused -- on the same four exact files: 58 pass, 6 fail, 2 platform skips. Node 24.18.0; bounded family RSS 1,040,240,640 bytes; no external guard stop.

  • cdpCleanupCancellation.test.ts: cancellation after capture and Electron cleanup assertions expect commands that were not observed.
  • firmwareIntegration.test.ts: the uncertain-workspace case expects a second launch but records one; relevant recovery work is already in fix(firmware): recover after workspace removal failures #1616.
  • qwenClientConfiguration.test.ts: both custom-home variants expect removal of a destination that remains accessible; relevant maintenance boundary work is already in fix(ci): preserve Git reads and simplify boundary fixtures #1603.
  • evmoleInterfaceProvider.test.ts: compounded overflow/cleanup failure no longer matches the asserted diagnostic reason.

These failures concern files outside this seven-file allocation change. The independent final-head semantic and real large-module CLI/MCP results in the PR description remain tied to their stated revisions. I am tracking the maintenance failures rather than bypassing the gate; this PR must remain unmerged until all checks on its eventual final head pass.

CI: https://github.com/morluto/rea/actions/runs/38073016162/job/114275323749

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli CLI adapter and one-shot commands area: domain Domain logic, errors, and pure functions area: mcp MCP server, transport, and tool registration performance Performance regressions, scaling limits, and bounded-resource work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant