[jayinsine]

Problem

With disk-backed Engram tables, cold long prefill waits on table-row gathers. On a fresh 106k-token prompt the GPUs spent 5.0–5.9 s of 10.1–10.7 s waiting on Engram (replay_readiness_wait_ns). This is the slow-prefill-after-compaction issue reported in the LiL "Disk engram prefill problem" thread.

On Linux, every gather goes through one io_uring on the delivery thread, and --embedding-table-workers above 1 is rejected. fio with the same pattern (4 KiB O_DIRECT random reads, 128 per batch, drained per batch) on the package's Engram files shows that one ring is the ceiling:

fio setup (same files)	reads/s
1 ring, QD128, drain per batch (current reader)	264k
1 ring, QD128, continuous refill	265k
1 ring, QD512, continuous refill	260k
2 rings × QD64 (128-slot budget)	386k
4 rings × QD32	499k
8 rings × QD16	590k

In production Flare measured 270k reads/s during that prefill, at the one-ring ceiling.

Change

Linux uses the existing Windows worker design, with io_uring lanes:

• Lanes: 1, 2, 4 or 8 lanes, each owning a ring and registered staging. They share the same 128 slots and 8 MiB of staging per table, so memory is unchanged.
• Splitting: large gathers split their planned reads across scoped helper threads; small gathers stay on the delivery thread (same threshold as Windows).
• Refill and recovery: each lane refills its window as reads complete. After an error or a panic, it drains reads the kernel still owns, so the next gather never sees stale completions.
• Unchanged: row planning, coalescing, output ownership and the counters. The Windows reader code is untouched.
• Default: the Linux default stays at 1 worker; --embedding-table-workers now works on Linux.
• Docs: docs/cli-design.md is updated.

Tests

• Shared worker tests: the two portable worker tests move from reader/windows.rs into the shared reader tests. Linux now checks that 1, 2, 4 and 8 workers return byte-identical rows within guard bands, issue identical read counts, and recover after a truncated file.
• New: a Linux budget test for the shared slot and staging budget.
• Helios results (runtime-package,server, memlock unlimited):
  • this branch: 1279 passed, 7 failed
  • cc696a6b: 1276 passed, the same 7 failed (chat-template and Gemma template tests that need checkpoint templates not present on this host)
  • the 3 extra passes are the moved tests and the new budget test; all 29 Engram tests pass.

GPU measurement

Setup:

• TP4, 4× RTX PRO 6000 Blackwell, running a DeepSeek-V4.1-Flash-1017 derivative package.
• Disk Engram on Btrfs RAID0 over two PCIe 5.0 NVMe drives, with a 64 GiB row cache and resident scales.
• A fresh server per arm (empty row cache), all serving the same fresh prompts.
• Two rounds; the second round gave each lane a full 128-slot window instead, with no material difference, so the PR keeps the shared budget.

Results:

Workers	Random 106k tokens	Python stdlib 77k tokens	GPU wait on Engram (106k)	Decode C1 tok/s
1	10.1–10.7 s	6.9–7.0 s	5.0–5.9 s	400–462
2	9.1–10.0 s	6.2 s	4.1–4.8 s	427–450
4	9.2–9.7 s	6.2–6.4 s	4.1–4.3 s	399–432
8	9.1–9.9 s	6.3 s	4.1–4.6 s	419–461

• Drift: a repeated 1-worker arm reproduced the first one.
• Correctness: a fixed arithmetic check answered correctly on every arm.

Limitations

• Modest gain: about 10–12 % faster cold prefill, and it doesn't grow beyond 2 workers on this host. Gathers average ~40k reads, and the per-lane wait stays near 7 s at 2, 4 and 8 workers, so the remaining disk time isn't the single ring.
• Default unchanged on Linux: measurements cover one host, one filesystem and one drive pair.
• Windows: not built or run here. Its reader code is unchanged; only its two portable tests moved.
• Your design call: --embedding-table-workers was described as a Windows-only setting in Discord. This opens it on Linux, but whether Linux should have it is your decision.

Prepared with an AI agent (Claude Opus 5.5); see the Design-Signed-Off-By trailer. This is host I/O in Helios; it touches no CUDA, Triton or GPU-optimization code (design-review principle 14).

[jayinsine]

Closing this. A follow-up test showed the filesystem was the real bottleneck, not the single reader thread.

Same package on the same box. The only change was moving the Engram files from Btrfs RAID0 (2× Gen5) to XFS on a single Gen4 drive:

Fresh prompt	Btrfs, 1 worker	XFS, 1 worker	XFS, 2 workers	XFS, 4 workers
random 106k	10.2–11.2 s	7.7–7.9 s	7.5–7.7 s	7.7–8.4 s
stdlib 77k	7.0 s	5.3 s	5.3 s	5.4 s
GPU wait on Engram (106k)	5.1–5.9 s	2.4–2.6 s	2.4–2.5 s	2.5–3.0 s

• Btrfs: with O_DIRECT + io_uring, Btrfs pushes the reads through io-wq helper threads. fio on one Engram file reached 0.41M reads/s on Btrfs, against 1.17M on each raw drive.
• XFS: completes them inline. fio reached 1.44M reads/s on XFS, and Flare's existing single ring reached 549k, with no helper threads.

On XFS, extra workers add nothing, so this change only helps Btrfs users. The practical advice for disk Engram on Linux is XFS or ext4, not more workers.