Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
beb2d8b
feat(sqlite): inspect database snapshots and selected table rows
Y0UW1SH Oct 9, 2026
443ecd2
fix: avoid SQLite schema collisions and stabilize cleanup fixture
Y0UW1SH Oct 9, 2026
76d7036
fix(sqlite): preserve schema bytes and refuse ambiguous shadow rows
Y0UW1SH Oct 9, 2026
f3212fb
fix(ci): complete the packaged managed-native fixture
Y0UW1SH Oct 9, 2026
00243e4
chore: merge main and retain the shared package fixture
Y0UW1SH Oct 9, 2026
a62af04
fix(sqlite): accept inactive journals and preserve acquisition failures
Y0UW1SH Oct 9, 2026
878b873
fix(sqlite): preserve negative zero in JSON evidence
Y0UW1SH Oct 9, 2026
75930f0
fix(sqlite): retain cleanup owners and honor native runtime capabilities
Y0UW1SH Oct 10, 2026
a9ce588
fix(sqlite): retain worker and transport cleanup owners
Y0UW1SH Oct 10, 2026
544c7a1
fix(sqlite): retain descriptor cleanup owners and verify runtime comp…
Y0UW1SH Oct 10, 2026
e59713d
fix(sqlite): preserve inspection and cancellation failures during cle…
Y0UW1SH Oct 10, 2026
f4791a8
fix(ci): correct Windows path assertions and synchronize upstream
Y0UW1SH Oct 10, 2026
eed34a6
fix(ci): consolidate SQLite regression fixtures and synchronize main
Y0UW1SH Oct 10, 2026
7caf4db
fix(sqlite): expose runtime readiness in CLI capabilities
Y0UW1SH Oct 11, 2026
d63d1b1
test(applications): repair Electron fixtures and catalog budget
Y0UW1SH Oct 11, 2026
ceb17b5
chore(sqlite): merge main and resolve fixture conflicts
Y0UW1SH Oct 11, 2026
2d96bf7
ci(sqlite): harden runtime coverage and update integration
Y0UW1SH Oct 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .agents/skills/reverse-engineer-anything/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,13 @@ explicit path or endpoint and do not need it.
argument strings and mutability as inferred candidates; this performs no chain
lookup or target execution. See the
[EVM bytecode guide](https://github.com/morluto/rea/blob/main/docs/evm-bytecode.md).
- Explicit local SQLite database snapshot: `inspect_sqlite_database`. Inspect
with Node.js 24.x >=24.15 or 26+ providing native SQLite limits. Read schema
first or select an exact ordinary table and `row_limit` when rows are
needed. Preserve both database and existing sibling WAL identity; row samples
report truncation and have unspecified order. Use a quiescent snapshot rather
than assuming a live database/WAL pair was copied atomically. See the
[SQLite guide](https://github.com/morluto/rea/blob/main/docs/sqlite-databases.md).
- Explicit Linux ELF file for offline layout, symbols, relocations or static
mitigation evidence: `inspect_binary_layout`. This target-free operation uses
caller-supplied pwntools without opening a disassembler database. Preserve its
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Native, managed, and packaged artifacts

## SQLite snapshots

Use `inspect_sqlite_database` with an explicit local database snapshot for its
schema and optional rows from one exact ordinary table. It requires Node.js
24.x >=24.15 or 26+ with the native `DatabaseSync.limits` API. It also consumes the
existing sibling `-wal`; retain both original digests when citing its result.
Choose `row_limit` when a finite sample is useful and keep truncation and
unspecified row order explicit. Virtual tables, shadow tables and views are
schema facts, not row-read targets. REA inspects private copies and leaves
the source files untouched; obtain a coherent quiescent database/WAL snapshot
before inspection. See [SQLite snapshots](https://github.com/morluto/rea/blob/main/docs/sqlite-databases.md).

## Recorded Linux crashes

Use `inspect_recorded_crash` with an explicit Linux x86-64 ELF core path to
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,11 @@ jobs:
if-no-files-found: error
retention-days: 7

sqlite-runtime:
needs: [changes, build]
if: needs.changes.outputs.code == 'true'
uses: ./.github/workflows/sqlite-runtime.yml

test:
name: Test (coverage + timing)
needs: [changes, test-shard]
Expand Down Expand Up @@ -561,6 +566,7 @@ jobs:
static,
build,
test-shard,
sqlite-runtime,
test,
conformance-fixtures,
windows-native-build,
Expand Down
43 changes: 43 additions & 0 deletions .github/workflows/sqlite-runtime.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: SQLite runtime compatibility

on:
workflow_call:

permissions:
contents: read

env:
HUSKY: "0"

jobs:
sqlite:
name: SQLite runtime compatibility (Node ${{ matrix.node }})
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
# Exercise unavailable-runtime diagnostics, the first supported release,
# and new patches on both supported lines without hand-editing versions.
# The repository-pinned Node 24 version also runs the complete test suite.
node: ["22.x", "24.15.0", "24.x", "26.x"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
check-latest: true
cache: npm
- run: npm ci
- uses: actions/download-artifact@v4
with:
name: rea-dist
path: dist
- name: Generate test artifacts from the downloaded runtime
run: |
node scripts/generate-mcp-tool-catalog.mjs
node scripts/generate-skill-metadata.mjs
node scripts/generate-product-catalog.mjs
node scripts/generate-completion-ledger.mjs
- name: Verify SQLite discovery, inspection and cleanup
run: npm run test:sqlite:run
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ Additional tools and host support depend on the target:
| Native binaries | Pseudocode, assembly, strings, symbols, calls and references | Hopper, Ghidra or IDA; [native analysis](https://rea.tools/guides/native/) |
| Offline ELF layout | Sections, segments, original symbols/relocations and static mitigation candidates | Caller-supplied pwntools on Linux x64; [binary diagnostics](docs/binary-diagnostics.md) |
| EVM bytecode | Dispatch selectors, byte offsets, inferred arguments and mutability | Local raw/hex carrier on Linux x64; [offline EVM guide](docs/evm-bytecode.md) |
| SQLite databases | Tables, columns, indexes, views, triggers and selected ordinary-table rows | Local database/WAL snapshot; Node.js 24.x >=24.15 or 26+ with native SQLite limits |
| Recorded Linux crashes | Raw notes, every recorded thread's registers/signals and optional mapping candidates | Caller-supplied pwntools on Linux x64; optional GDB/pwndbg; [recorded crashes](docs/recorded-crashes.md) |
| JavaScript / Electron | Modules, imports, source maps, routes, IPC and native add-on relationships | Node.js and npm; [application analysis](https://rea.tools/guides/javascript/) |
| Websites | Page structure, scripts, network observations and requested screenshots | A Chrome-family browser; [browser analysis](https://rea.tools/guides/browser/) |
Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ export default defineConfig({
{ text: "Android applications", link: "/android-analysis" },
{ text: "Apple applications", link: "/apple-application-analysis" },
{ text: "Firmware", link: "/firmware-analysis" },
{ text: "SQLite databases", link: "/sqlite-databases" },
{
text: "Reconstruction readiness",
link: "/reconstruction-readiness",
Expand Down
8 changes: 8 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ module page. See
[JavaScript application workflows](javascript-application-workflows.md) for
views, feature traces and comparisons that take the same saved Evidence.

## SQLite snapshots

For a local database snapshot, use `rea inspect-sqlite-database PATH --json`.
Add `--table NAME --row-limit 20` to read a bounded sample from an ordinary
table. This command does not require a native analysis provider. See
[SQLite database snapshots](sqlite-databases.md) for database/WAL identity,
source-file handling and row-value semantics.

## Native analysis

Configure [Hopper or Ghidra](installation.md#hopper), or the
Expand Down
121 changes: 121 additions & 0 deletions docs/sqlite-databases.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# SQLite database snapshots

`inspect_sqlite_database` / `inspect-sqlite-database` inspects one explicitly
selected local SQLite database snapshot without an active native analysis
target. Use it to understand application storage, configuration or retained
database evidence. No separate SQLite installation is required.

This operation requires Node.js 24.x >=24.15 or 26+ with the native
`DatabaseSync.limits` API. The worker verifies that capability before querying
the copied database. Other REA features retain their published Node.js 22.x,
24.x or 26+ requirements.

`rea capabilities` reports `inspect_sqlite_database` availability and the
runtime requirement without opening a database. Unsupported runtimes show
`available: false`, `availability_code: "unsupported_version"`, and an upgrade
note while the inventory command still succeeds.

```sh
rea capabilities --json
rea inspect-sqlite-database ./application.sqlite --json
rea inspect-sqlite-database ./application.sqlite --table settings --row-limit 20 --json
```

```json
{
"name": "inspect_sqlite_database",
"arguments": {
"path": "/artifacts/application.sqlite",
"table": "settings",
"row_limit": 20
}
}
```

MCP paths must be absolute filesystem paths on the current host. CLI paths
resolve against the operator's working directory. Select a quiescent snapshot:
stop the writer or obtain a consistent database/WAL pair using the application's
backup workflow before inspection. A file copy of a live pair does not establish
an atomic SQLite snapshot, even if no change is detected during inspection.

## Returned evidence

The complete schema inventory includes tables and their columns, indexes, views
and triggers. SQL declarations retain the original schema text. Virtual and
shadow tables are identified as such; their supported metadata coverage is
reported. If a virtual table cannot be resolved, possible backing tables have
`kind: "unknown"` and an explicit limitation; their rows are refused. Unrelated
ordinary tables remain selectable. Schema inventory does not execute views,
triggers or arbitrary SQL.

Rows are optional. `table` selects one exact ordinary-table name. `row_limit`
requires `table`, defaults to 100 and accepts positive safe integers. Samples
remain subject to the byte and worker resource limits below, regardless of row count. The result reports
the selected limit, returned count and whether more rows were observed. Row
order is unspecified; a truncated sample does not prove anything about omitted
records. View, virtual-table and shadow-table row reads are unsupported.

SQLite integers are decimal strings so every signed 64-bit value survives JSON
serialization. Real values retain their numbers; `"Infinity"`, `"-Infinity"`
and `"-0"` explicitly preserve infinities and negative zero through JSON.
Text returns the decoded string together with its bytes as hex and the database
encoding, preserving malformed text that the SQLite binding may replace for
display. Blobs are hex, and SQL NULL is a distinct tagged value.

Database and existing sibling `-wal` files have separate original SHA-256 and
byte counts. The Evidence subject identifies the original database file;
the WAL identity is also necessary to interpret pending committed data. The
SQLite engine version is reported with the observed result. Neither digest is
a claim that the database is healthy, complete or trusted.

## Source files and failures

REA copies the selected database and existing sibling WAL to an owned private
temporary directory before SQLite opens them. Source files are not opened by
SQLite, so its locks or shared-memory sidecars cannot modify the original
snapshot. Temporary copies and the owned worker are cleaned up on success,
failure and cancellation. Extension loading and schema trust are disabled.
Empty rollback journals and fully zeroed journal headers left by successful
`TRUNCATE` and `PERSIST` commits are accepted after stable identity checks.
Other retained rollback journals are refused because recovery is outside this
profile. Journal files are never modified or copied into the worker snapshot.

The database and WAL together are limited to 256 MiB. SQLite values/records,
SQL statements and the complete worker JSON reply each have a 16 MiB limit.
The owned worker has a 256 MiB JavaScript heap limit and a 30-second inspection
deadline. SQLite native allocations are separate from the JavaScript heap;
overall native heap and total resident memory are not capped. Row limits do not
override these byte and resource limits. Resource-limit failures do not return
an incomplete schema as successful complete evidence.

MCP also applies its configured response budget to the complete Evidence
envelope. If delivery exceeds that budget, it reports a transport constraint
with guidance for exporting retained Evidence, requesting a matching larger
client/server response budget or using complete CLI JSON output.

Changed input identities, special files, malformed databases, unsupported
formats and resource failures are reported as errors with the selected path and
meaningful reason. Encrypted databases, database repair, arbitrary SQL,
cross-database queries and live capture are outside this inspection's coverage.

## Compatibility verification

Run `npm run test:sqlite` after changing the SQLite adapter or its shared
process, filesystem, CLI or MCP boundaries. It prepares the compiled runtime
and runs SQLite service, snapshot, inspection, cleanup and public CLI/MCP
tests. Directory-based selection includes new SQLite service and boundary
tests automatically. `npm run test:sqlite:run` runs the same cases when the
compiled runtime and generated test artifacts are already current.

The reusable SQLite CI workflow consumes the shared build artifact and tests
the latest Node 22.x unavailable-runtime diagnostics, the minimum supported
24.15.0 release, and the latest 24.x and 26.x releases. The repository-pinned
runtime also runs the full suite. CI's required gate waits for every selected
SQLite matrix job; a failed, cancelled, skipped or missing selected lane blocks
the gate. Workflow changes themselves select the build and compatibility lane.

PR checks run against GitHub's merge with the current base branch. A later
change to `main` can introduce failures outside SQLite; inspect the failing
job and recheck the updated merge before treating an earlier green run as
current verification. Compatibility checks detect regressions, but cannot
guarantee that an arbitrary future upstream change remains compatible.
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,8 @@
"verify:native-calls": "npm run build:cached && node scripts/verify-native-calls.mjs",
"verify:native-ui:permissions": "npm run build:cached && node scripts/verify-native-ui.mjs --allow-permission-denial",
"test:focused": "node scripts/run-development-tests.mjs focused",
"test:sqlite": "npm run test:prepare && npm run test:sqlite:run",
"test:sqlite:run": "vitest run src/application/sqlite/ tests/boundary/sqlite/ tests/acceptance/sqliteDatabaseCli.test.ts tests/boundary/mcp/serverLifecycleMcp.test.ts",
"build:windows-native": "node scripts/build-windows-native.mjs",
"verify:windows-native:artifact": "node scripts/verify-windows-native-artifact.mjs",
"prepublishOnly": "npm run verify:windows-native:artifact",
Expand Down
2 changes: 2 additions & 0 deletions scripts/ci/plan.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,8 @@ if (
file === ".github/actionlint.yaml"
) {
selected.workflows = true;
// SQLite reuses this run's compiled artifact, so workflow edits need build.
if (file === ".github/workflows/sqlite-runtime.yml") selected.code = true;
if (file !== ".github/workflows/ci.yml") continue;
}
if (owners.length === 1 && owners[0] === "website") continue;
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/required.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ const jobs = JSON.parse(process.env.CI_JOBS);
const outputs = jobs.changes.outputs;
const required = ["changes", "static", "test"];
const lanes = {
code: ["build", "test-shard"],
code: ["build", "test-shard", "sqlite-runtime"],
runtime: ["windows-native-build", "package-e2e"],
package: ["windows-curated"],
apple: ["apple-artifacts"],
Expand Down
5 changes: 5 additions & 0 deletions scripts/lib/catalog-builders.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ export const toolFamilyCatalog = (sources) => {
contracts: [
...sources.artifactContracts.ARTIFACT_TOOL_CONTRACTS,
...sources.evmContracts.EVM_TOOL_CONTRACTS,
...sources.sqliteContracts.SQLITE_TOOL_CONTRACTS,
],
},
{
Expand Down Expand Up @@ -183,6 +184,10 @@ export const providerCatalog = (sources) => {
identity: sources.evmProvider.EVMOLE_PROVIDER_IDENTITY,
contracts: sources.evmContracts.EVM_TOOL_CONTRACTS,
},
{
identity: sources.sqliteProvider.SQLITE_PROVIDER_IDENTITY,
contracts: sources.sqliteContracts.SQLITE_TOOL_CONTRACTS,
},
{
identity: sources.artifactProviders.ARTIFACT_GRAPH_PROVIDER,
contracts: sources.artifactContracts.ARTIFACT_TOOL_CONTRACTS,
Expand Down
2 changes: 2 additions & 0 deletions scripts/lib/catalog-core.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ const SOURCE_PATHS = {
pwntoolsProvider: "dist/native/pwntools/PwntoolsRelease.js",
nativeContracts: "dist/contracts/native/nativeToolContracts.js",
evmContracts: "dist/contracts/evm/evmToolContracts.js",
sqliteContracts: "dist/contracts/sqlite/sqliteToolContracts.js",
sqliteProvider: "dist/sqlite/SqliteDatabaseLimits.js",
evmProvider: "dist/evm/EvmoleRelease.js",
artifactContracts: "dist/contracts/artifactToolContracts.js",
managedContracts: "dist/contracts/managed/managedToolContracts.js",
Expand Down
1 change: 1 addition & 0 deletions scripts/lib/module-boundaries.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { fileURLToPath, pathToFileURL } from "node:url";

const PURE_LAYERS = new Set(["domain", "contracts"]);
const MIGRATED_PROVIDER_ROOTS = new Set([
"sqlite",
"evm",
"android",
"firmware",
Expand Down
10 changes: 10 additions & 0 deletions src/application/CapabilityInventory.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { z } from "zod";
import { sqliteDatabaseCapability } from "./sqlite/SqliteDatabaseCapability.js";
import type {
OptionalProviderLoadFailures,
OptionalProviderLoadFailure,
Expand Down Expand Up @@ -270,6 +271,15 @@ const workflowAvailabilityFor = ({
kind,
policy,
}: AvailabilityContext): Availability | null => {
if (name === "inspect_sqlite_database") {
const capability = sqliteDatabaseCapability(process.versions.node);
return capability.available
? { reason: "available", remediation: null }
: {
reason: "provider_unavailable",
remediation: capability.reason,
};
}
if (name === "trace_web_module_imports")
return {
reason: "available",
Expand Down
9 changes: 8 additions & 1 deletion src/application/DirectAnalysisStatus.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import { createServerIdentity } from "../serverIdentity.js";
import { silentLogger } from "../logger.js";
import type { Logger } from "pino";
import type { DirectAnalysisDependencies } from "./DirectAnalysisDependencies.js";
import { providerCapability } from "../contracts/toolOutputSchemaPrimitives.js";
import { sqliteDatabaseCapability } from "./sqlite/SqliteDatabaseCapability.js";

/** Read target-free provider status and operation availability. */
export const runSessionStatus = async (
Expand All @@ -17,8 +19,13 @@ export const runSessionStatus = async (
if (!config.ok) return { error: projectAnalysisError(config.error) };
const session = dependencies.createBinarySession(config.value, logger);
try {
const status = jsonObjectSchema.parse(session.status());
return {
...jsonObjectSchema.parse(session.status()),
...status,
capabilities: [
...providerCapability.array().parse(status.capabilities),
sqliteDatabaseCapability(process.versions.node),
].sort((left, right) => left.operation.localeCompare(right.operation)),
server_identity: createServerIdentity({
startedAt: new Date().toISOString(),
}),
Expand Down
30 changes: 30 additions & 0 deletions src/application/sqlite/SqliteDatabaseCapability.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { expect, it } from "vitest";
import { providerCapability } from "../../contracts/toolOutputSchemaPrimitives.js";
import { sqliteDatabaseCapability } from "./SqliteDatabaseCapability.js";

it.each([
["22.23.1", false],
["24.11.0", false],
["24.14.9", false],
["24.15.0", true],
["25.1.0", false],
["26.0.0", true],
["24.15.0-rc.1", false],
["invalid", false],
])(
"reports SQLite inspection availability on Node %s as %s",
(version, available) => {
const capability = providerCapability.parse(
sqliteDatabaseCapability(version),
);
expect(capability).toMatchObject({
operation: "inspect_sqlite_database",
available,
reason: available
? null
: expect.stringContaining("Node.js 24.x >=24.15 or 26+"),
availability_code: available ? null : "unsupported_version",
limitations: [expect.stringContaining("DatabaseSync.limits")],
});
},
);
Loading
Loading