Skip to content

feat: wal durability - #26

Closed
notxorand wants to merge 13 commits into
devfrom
wal-durability
Closed

notxorand wants to merge 13 commits into
devfrom
wal-durability

Conversation

@notxorand

@notxorand notxorand commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

closes: #21

Greptile Summary

The PR adds configurable eventual or strict WAL durability and integrates durable source and cascade checkpoints into runtime execution.

  • Adds WAL-backed fact storage, recovery, batching, and durability modes.
  • Routes inference cascades through durable pending work and atomic cascade checkpoints.
  • Adds rollback handling for failed cascade mutations and queue scheduling.
  • Integrates shared storage into CLI and supervisor-managed sessions.
  • Adds storage and WAL tests and benchmarks.

Confidence Score: 3/5

The PR is not safe to merge until cascade rollback preserves unrelated queued work and strict WAL publication resolves terminal failures without hanging.

Cascade rollback currently deletes all shared dirty signals, while a persistent post-fsync publication failure traps the WAL worker and strict caller in an unbounded retry loop.

Files Needing Attention: src/engine/context.zig and src/wal.zig

Important Files Changed

Filename Overview
src/wal.zig Implements WAL admission, recovery, cascade checkpoints, and strict publication, but strict publication can retry forever after a synced commit.
src/engine/context.zig Adds cascade accumulation and rollback, but rollback clears unrelated work from the shared dirty queue.
src/engine/loop.zig Drives durable source work and transitive dirty entries through bounded inference cycles with convergence checks.
src/wasm/host_abi/generic.zig Integrates guest writes with cascade accumulation, queue signaling, and rollback ownership.
src/storage.zig Exposes WAL-backed persistence and configurable durability to engine callers.
bin/supervisor.zig Opens shared durable storage and supplies it to managed module sessions.

Reviews (20): Last reviewed commit: "fix: keep cascade and WAL state recovera..." | Re-trigger Greptile

Greptile also left 2 inline comments on this PR.

+ integrate storage durability into runtime and CLI
+ add custom WAL recovery, batching, acknowledgements, and fsync modes
+ add SQLite and WAL performance benchmarks
+ update HLC initialization and ignore database artifacts
+ batch queued requests into one fsync
+ enforce record, field, queue, and mutation limits
+ mark terminal writer failures and reject pending work
+ distinguish torn tails from corrupt WAL records
@notxorand
notxorand marked this pull request as ready for review September 1, 2026 16:11
Comment thread src/engine/loop.zig
Comment thread src/engine/loop.zig Outdated
Comment thread src/wal.zig Outdated
@codspeed

codspeed Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.97%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 3 untouched benchmarks
🆕 1 new benchmark

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ runtime.execution 7.4 s 8.4 s -11.97%
🆕 runtime.storage N/A 2.1 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing wal-durability (52ab58f) with dev (89658db)

Open in CodSpeed

expose durability as a build option and use it when opening storage,
with benchmarks reduced to a single inference iteration.
Comment thread src/engine/loop.zig
Comment thread src/wal.zig Outdated
Comment thread src/engine/loop.zig Outdated
Comment thread src/wal.zig
Comment thread src/wasm/host_abi/generic.zig Outdated
Comment thread src/wal.zig Outdated
Comment thread src/wasm/host_abi/generic.zig Outdated
Comment thread src/wasm/host_abi/generic.zig
Comment thread src/wal.zig Outdated
+ add queue capacity checks before publishing facts
+ defer eventual WAL cleanup until flush completion
+ expose queue fullness through DirtyQueue
Comment thread src/wal.zig Outdated
Comment thread src/wal.zig Outdated
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
Comment thread src/wal.zig Outdated
Comment thread src/memory/lww.zig Outdated
Comment thread src/wal.zig
Comment thread src/wasm/host_abi/generic.zig
Comment thread src/wal.zig Outdated
@notxorand
notxorand force-pushed the wal-durability branch 4 times, most recently from cd138f8 to 8eeee04 Compare September 4, 2026 03:20
Comment thread src/engine/loop.zig
@notxorand
notxorand force-pushed the wal-durability branch 2 times, most recently from a33c610 to 52ab58f Compare September 4, 2026 15:07
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
@slunghq slunghq deleted a comment from greptile-apps Bot Sep 4, 2026
Comment thread src/memory/lww.zig
Comment thread src/engine/context.zig Outdated
Comment thread src/wal.zig
Comment thread src/engine/context.zig Outdated
Comment thread src/wal.zig Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: persistence for durability

1 participant