Skip to content

feat(sqlite): inspect offline snapshots with selective evidence - #2271

Merged
N0zoM1z0 merged 5 commits into
mainfrom
feat/sqlite-analysis-1356-20261012
Oct 12, 2026
Merged

N0zoM1z0 merged 5 commits into
mainfrom
feat/sqlite-analysis-1356-20261012

Conversation

@N0zoM1z0

@N0zoM1z0 N0zoM1z0 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

REA currently cannot inspect application SQLite storage through its public CLI/MCP. This adds inspect_sqlite_database / inspect-sqlite-database for explicit offline database/WAL snapshots, developed from @Y0UW1SH's implementation in #1356. This PR supersedes that proposal and includes its contributor credit.

  • Return compact inventory by default; selecting a table expands only its schema and samples 20 rows. Exact ordered columns / repeated --column avoids evaluating unrelated cells; schema_scope: database explicitly requests complete stored schema.
  • Preserve database/WAL identities, integer precision, raw text bytes, blobs, unknown virtual/shadow classification and sample truncation. Source files remain unchanged; this does not claim atomic acquisition from a live writer or forensic recovery.
  • Reuse canonical grouped discovery and admitted session handling. Preserve completed observations after late cancellation; private-root cleanup failures retain Evidence and resource owners. MCP records completed Evidence before returning a compact retained-evidence reference.
  • Address the original review's aggregate-resource finding: at most two inspections per provider through acquisition, worker and cleanup. Occupied capacity is retryable before new filesystem/process effects. Recheck rollback-journal validity after copying even when timestamps cannot distinguish an activation. Account for JSON escaping in bounded serializer parts before allocating full output.

Validation on Linux, with two CPU cores and an 8 GiB process-family guard:

  • npm run check, npm run docs:check, and npm run verify:test-discovery passed on the updated main base.
  • Final pinned Node 24.18.0 SQLite suite: 105 passed, 2 conditional skips. Node 22.23.3: 68 passed, 39 expected unavailable-runtime skips; public CLI/MCP discovery and diagnostics passed.
  • Minimum Node 24.15.0 full suite after journal fix: 104 passed, 2 skips; the subsequent bounded-JSON inspection cases also checked separately.
  • Actual packaged CLI and persistent MCP on official Chinook v1.4.5 and an owned real Firefox persistent profile (eight local history entries), with independent read-only SQLite row/digest oracles. Also inspected a 64,163,840-byte, 200,000-row SQLite fixture. No binaries, database files or profiles are committed.
  • Native regressions cover committed WAL, DELETE/PERSIST/TRUNCATE, exact names and column order, empty identifiers, malformed data/encoding, generated-value expansion, capacity, cancellation, descriptor cleanup, retained roots, MCP error retention and session draining. Journal activation with unchanged metadata failed before and passed after the producing-boundary fix.
  • SDK-converted advertised JSON Schemas validated against Draft 2020-12 and the supported client profile; related CI scope/required-lane and discovery tests passed.

Representative context measurements (MCP response text, same frozen inputs):

Request Original Scoped output
Chinook inventory 19,538 B 6,600 B
Chinook Track, 20 full rows 29,431 B 16,401 B
Firefox moz_places 43,868 B 20,201 B
Firefox selected id/url/title 43,868 B 12,918 B

The completed Chinook discovery + twenty full-row task, counting one complete SQLite tool definition plus both request JSON payloads and response text, estimates 17,687 → 10,300 o200k_base tokens (about 42% less). Other model tokenizers, catalog/prompt conversion, agent reasoning and billed model usage are unmeasured. Both workflows use two calls without retries and retain identical requested rows.

Isolated final 24-call MCP probe: 10.76 s wall time, 12.72 s process-family CPU time, peak 843.22 MiB RSS. Three repetitions per input; the 64 MB projected sample took 0.67–0.93 s. Each call starts a fresh worker; the MCP server stays warm and filesystem caches were not cleared. Earlier baseline CPU time was not measured. Each call checks source bytes; persistent MCP amortizes server startup rather than reusing potentially stale database contents.

Latest main lifecycle integration passed 35 affected SQLite/MCP/EVM/crash cases. The first CI run exposed two PR integration mistakes: an obsolete runtime artifact name and a missing trusted-worker wording-audit entry. Both are fixed, with a producer/consumer artifact-name-and-layout regression; 121 focused cases and the full local gates passed.

Node 26 and non-Linux behavior are not claimed from local verification; the new compatibility matrix and existing CI provide additional coverage. Native SQLite allocations remain distinct from the 256 MiB worker JavaScript heap; no overall native heap or OS sandbox claim is made.

N0zoM1z0 and others added 3 commits October 12, 2026 17:15
Import the SQLite inspection proposed in #1356 and integrate its CLI, MCP,
Evidence, cleanup, and runtime compatibility boundaries with current main.

Co-authored-by: Y0UW1SH <29498139+Y0UW1SH@users.noreply.github.com>
Use compact schema inventory and selected-table details by default, with
explicit complete schema and exact ordered column selection. Sample 20 rows
by default while keeping larger caller-selected limits and lossless cells.

Reuse grouped capability projection and session admission. Preserve completed
observations through late cancellation and cleanup failures, recording their
Evidence before delivery. Bound provider concurrency to two inspections and
report occupied capacity as a retryable typed resource constraint.

Verified real SQLite CLI/MCP workflows, Chinook and Firefox snapshots,
malformed inputs, WAL, encoding, cancellation, cleanup retry and resource limits.

Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
Re-read inactive rollback journal headers after copying the database/WAL,
before final file-set identity checks. A previously inactive journal that
becomes active is changed evidence even when its metadata cannot distinguish
the write. Keep descriptor cleanup ownership and refuse recovery.

Count bounded JSON parts through the shared serializer so escaping is
accounted for without allocating an oversized document to measure it.

Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
@N0zoM1z0 N0zoM1z0 self-assigned this Oct 12, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

Retain main's static EVM descriptor and shared binary-layout/crash cleanup
instances alongside the SQLite service registered and closed by this server.

Validated check, docs, test discovery and 35 SQLite/MCP/EVM/crash integration cases.
Download the build-owned rea-test-runtime artifact at the repository root,
reusing its compiled runtime and prepared test metadata across Node versions.
Add a producer/consumer contract regression for the artifact name and layout.

Document that execution wording denies application/caller SQL execution
while a trusted inspector worker is launched. Reuse the invariant cell-byte
length parser rather than constructing it for every sampled row.

Validated 121 focused CI, effects, native inspection and public workflow
cases plus the artifact contract; check, docs and test discovery pass.

Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
@N0zoM1z0
N0zoM1z0 merged commit aeee507 into main Oct 12, 2026
37 checks passed
@N0zoM1z0
N0zoM1z0 deleted the feat/sqlite-analysis-1356-20261012 branch October 12, 2026 18:33
@github-actions github-actions Bot mentioned this pull request Oct 12, 2026
3 tasks done
@morluto morluto added enhancement New feature or request area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior labels Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: native Native analysis and decompilation, including Ghidra, IDA, and Hopper analysis behavior enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants