Use `posix_fadvise` on ImmutableDB chunk files to bound page cache footprint during sync and replay

[jasagredo]

Summary

During both sync and replay, ImmutableDB chunk files are read and written sequentially without giving the kernel any hint about the access pattern. On Linux this leaves everything in the page cache indefinitely, even though chunks have effectively single-use access semantics from the node's perspective (no rollbacks cross into the immutable region; block-serving to peers only matters near tip). On memory-constrained hosts this consumes several GB of RAM that is structurally useless to the node.

This issue proposes adding a posix_fadvise/madvise hook to the filesystem abstraction and using it on chunk open/close and on chunk completion during append.

Observed behaviour

cardano-node syncing mainnet from genesis on a Raspberry Pi 5 (8 GB RAM, NVMe SSD, no other meaningful load):

• During Byron replay, /proc/meminfo: Cached grows almost perfectly linearly with ChainDB.tip.slot, from ~470 MB at startup to ~2.7 GB after ~2 h 30 m and ~2.3 M slots.
• Slope ≈ 1.3 KB / slot, matching the expected Byron per-block ImmutableDB footprint.
• vmtouch -v db/immutable/ confirms the cache is dominated by immutable chunk files. The LSM ledger state on disk contributes negligibly — as expected, since Byron replay has not yet materialised significant UTxO-HD state.

This was observed during a cold sync (writing chunks as blocks arrive from upstream), not a pure replay pass, so the pattern applies to the write path as well as the read path.

Why it matters:

• Cached is reclaimable, so this is not a correctness issue and MemAvailable stays healthy during Byron.
• But the trajectory extrapolates. Post-Byron blocks are larger, and the LSM backend will start carrying non-trivial on-disk state that does benefit from being cached. Giving ~hundreds of MB to use-once chunk data competes with that legitimately-hot LSM state for RAM, and with the RTS heap, on 8 GB targets.
• The optimisation is architecturally clean on any host: we know these pages are single-use; their retention has no functional benefit to the node.

Background: what posix_fadvise offers

posix_fadvise(fd, offset, len, advice) lets the process tell the kernel how a file range will be accessed. The relevant advices:

• POSIX_FADV_SEQUENTIAL — increases readahead for streaming reads.
• POSIX_FADV_DONTNEED — drops the specified range from the page cache.
• POSIX_FADV_WILLNEED — prefetches a range.
• POSIX_FADV_NOREUSE — "accessed once"; historically a no-op on Linux, honoured on 6.3+.

A codebase search for fadvise, madvise, mmap, O_DIRECT finds no existing usage anywhere in ouroboros-consensus.

Proposal

1. fs-api (upstream) — extend HasFS

Add an hAdvise operation with a no-op default for non-POSIX backends (IOSim, Windows for the first cut):

data Advice = AdviceNormal | AdviceSequential | AdviceRandom
            | AdviceWillNeed | AdviceDontNeed | AdviceNoReuse

hAdvise :: HasFS m h -> Handle h -> Word64 {- offset -} -> Word64 {- length, 0 = whole file -} -> Advice -> m ()

Linux/POSIX backend wraps posix_fadvise(2); Windows backend is a no-op for now (can later be backed by SetFileInformationByHandle / FILE_FLAG_SEQUENTIAL_SCAN).

2. ImmutableDB read path

In ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/Iterator.hs:

• In iteratorStateForChunk (currently around L537–545), immediately after opening a chunk file with hOpen … ReadMode, call hAdvise handle 0 0 AdviceSequential.
• When the iterator transitions past a chunk (closing the handle), call hAdvise handle 0 0 AdviceDontNeed before hClose so those pages are evicted.

3. ImmutableDB write path

When appending, each completed chunk is written sequentially, fsynced, and then (once the tip advances into the next chunk) never written again. Apply AdviceDontNeed to the completed chunk after its final hSync, in the appender module (exact call site needs a short investigation — likely the code path that rolls over to a new chunk in Impl/State.hs / its appender helper). During active append of the current chunk the hint should NOT be used, since we do read back freshly-written blocks during ledger application.

Open questions

• Block serving at tip. Once the node is caught up, BlockFetch requests from peers benefit from cached chunks near tip. DontNeed is always safe for chunks older than k blocks behind the immutable tip; the simplest policy is to apply it only to chunks that have been fully consumed and are not near tip. During sync-from-genesis this distinction is moot — no meaningful peer traffic is being served.
• Granularity. Per-chunk at close time is simpler and almost certainly sufficient; per-read-chunk streaming hints are overkill.
• Ordering vs. fSync. On the write path, AdviceDontNeed only drops clean pages; we must ensure writeback has completed (explicit hSync or sync_file_range with WAIT_AFTER) before advising, otherwise the hint is silently ineffective for dirty pages.
• Upstream coordination. The HasFS change lives in fs-api (separate repo). We may want to land the consensus-side call sites behind a CPP guard or a conditional dependency bump.
• IOSim semantics. No-op is the right default. We don't need to model cache behaviour in simulation.

Acceptance

• vmtouch -v db/immutable/ during a cold sync shows page cache residency bounded by the number of chunks held open by active iterators + the currently-appended chunk, rather than growing linearly with tip slot.
• No regression in sync throughput. On NVMe the re-read penalty if a hint turned out to be wrong is negligible; on slower storage, this matters only if we incorrectly DontNeed a range we will re-read, which the policy above avoids.
• Unit/integration tests for the new HasFS operation in fs-api (no-op semantics) and a consensus-level test that exercises the iterator close path.

Motivating graph from the Raspberry Pi

[jorisdral]

Note that the HasBlockIO API from the blockio library already has an implementation for dealing with hints for the page cache. You could choose to move over this code to fs-api/fs-sim. It's probably not useful to use HasBlockIO directly because it opens a io_uring context, and you only need the cache hints

The API:
https://github.com/IntersectMBO/lsm-tree/blob/b70108482036805afdcb4a9ab847ef3c70d40148/blockio/src/System/FS/BlockIO/API.hs#L103-L113

The IO implementation for Linux:
https://github.com/IntersectMBO/lsm-tree/blob/main/blockio/src-linux/System/FS/BlockIO/Internal.hs#L53-L71

The simulator implementation:
https://github.com/IntersectMBO/lsm-tree/blob/b70108482036805afdcb4a9ab847ef3c70d40148/blockio/src-sim/System/FS/BlockIO/Sim.hs#L94-L100