TBL-RAW pre-processed pipeline: decode once, mmap-serve every epoch — new efficiency frontier - #18
Merged
Merged
Conversation
added 4 commits
August 3, 2026 22:52
…epoch
New efficiency frontier for many-epoch training, portably (FFCV's idea
without the .beton lock-in): preprocess_to_tbl runs the C++ fast path over a
TAR and stores RGB uint8 samples (SampleFormat.RAW_U8) in TBL v2;
TblRawImageLoader (and DataLoader('.tbl')) then serves batches from a
zero-copy mmap view through ONE fused parallel SIMD pass
(normalize_u8_gather: index gather + u8 HWC -> normalized f32 CHW, GIL
released) — bit-identical output to the TAR pipeline (tested array_equal).
M4 Max, Imagenette 160px: 525k img/s produce / 88k np.sum-consumed vs
on-the-fly 33k/31k and float32 cache_decoded 65k/59k — faster than the RAM
cache at BOTH consumption levels while owning file-backed pages instead of
its ~2.9 GB anonymous RAM (peak-RSS growth +448 MB vs +7.8 GB). LZ4 on
decoded photos measured 1.06x -> RAW defaults compression=False (documented).
MetalResidentLoader ingests .tbl directly (upload = one memcpy) and
CudaResidentLoader.from_tbl uploads through the mmap in chunks — the
decode-all pass disappears. Serve-time hflip_prob is the one aug this path
supports (crop/color must be baked; TAR pipeline keeps train_aug). e2e
benchmark gains --tbl; new benchmark_tbl_raw.py reports both consumption
levels + RSS + honest LZ4 ratio.
First 3090 e2e run exposed the design gap honestly: synchronous serve put ~10ms/step of gather+flip+normalize ON the training thread, making TBL-RAW SLOWER e2e (4.51s) than the TAR pipeline's threaded prefetch (3.73s) despite being 2x faster loader-only. A stop-aware producer thread now builds batches ahead (the SIMD ops release the GIL, so production overlaps the training step); pinned-ring depth is clamped to ring-2 so no buffer is overwritten while held. Prefetch output is asserted identical to sync (incl. hflip path), early-exit winds the thread down, resume works through the queue. Docs + CHANGELOG carry the full honest numbers.
…docs numbers In-process stage order contaminated the sync serve path by up to 6x (thread pool/allocator state) — every configuration now runs in its own subprocess, which also makes per-stage peak RSS exact. Docs carry the isolated numbers: M4 raw serve 531k produce / 89k sum, prefetch 98.6k sum, cache 62k at 8.8 GB RSS; 3090 e2e TBL-RAW 3.64s vs TAR 3.76 / PT 3.92 / floor 3.39 — fastest input pipeline measured on this benchmark, incl. the honest 4.51s-before- prefetch story.
…l. the GIL nuance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The efficiency-frontier sprint: memory, compression, and a real use for the TBL v2 format, which was previously disconnected from the training path entirely.
The pipeline
SampleFormat.RAW_U8+preprocess_to_tbl(tar, tbl, image_size=N)— one-time parallel decode+resize (9,469 images in 6s on M4), exact uint8 recovery.TblRawImageLoader/DataLoader('data.tbl')— batches served from a zero-copy mmap view through ONE fused parallel SIMD pass (normalize_u8_gather), bit-identical to the TAR pipeline (testedarray_equal). Full contract: (seed, epoch) determinism, resume, drop_last, pinned ring, serve-time hflip, background prefetch.MetalResidentLoader('data.tbl')andCudaResidentLoader.from_tbl(...)ingest the same file — their decode-all pass disappears.Measured (per-stage subprocesses, dual consumption levels)
Tests
26 new tests (bit-parity, prefetch==sync equivalence incl. hflip, ring lifetime, resume, early-exit wind-down, format/compression/dims rejection, resident ingestion) — 25 pass on M4, 21 pass on the 3090 incl. the CUDA resident test.