diff --git a/README.md b/README.md index ae038e1..bd3d1aa 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # TurboLoader -**Production-Ready ML Data Loading Library** +**High-performance ML data loading — a C++20 core with SIMD transforms, GPU kernels, and one `pip install`.** [![PyPI version](https://img.shields.io/pypi/v/turboloader.svg)](https://pypi.org/project/turboloader/) [![Tests](https://github.com/ALJainProjects/TurboLoader/actions/workflows/test.yml/badge.svg)](https://github.com/ALJainProjects/TurboLoader/actions/workflows/test.yml) @@ -8,36 +8,61 @@ [![C++20](https://img.shields.io/badge/C%2B%2B-20-blue.svg)](https://en.wikipedia.org/wiki/C%2B%2B20) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +![TurboLoader demo: pip install, then ~60k img/s on real ImageNet JPEGs](docs/assets/demo.gif) + +*Real recording ([tape](docs/assets/demo.tape), [script](examples/quickstart_demo.py)): 9,469 real ImageNet +JPEGs decoded → RandomResizedCrop+flip → resized → normalized, ~60k img/s per epoch on an M4 Max laptop.* + --- -## Overview +## How it works -TurboLoader is a high-performance data loading library for machine learning workflows. Built with C++20 and featuring Python bindings, it provides efficient data loading with SIMD-accelerated transforms, custom binary formats, and distributed training support. +The fast path does everything in one fused, GIL-released C++ pass — no worker +processes, no per-sample Python, no offline format conversion: -### Core Features +```mermaid +flowchart LR + A["TAR of JPEGs
(local / http / s3 / gs)"] --> B["persistent C++
thread pool"] + B --> C["decode → augment → resize → normalize
SIMD (NEON / AVX2 / AVX-512), fused"] + C --> D[("contiguous batch
N×3×H×W float32")] + D --> E["your training step
(zero-copy to torch)"] +``` -- **Fused train pipeline** - `train_aug=True`: RandomResizedCrop + flip + normalize inside the C++ pass (torchvision-parity, deterministic per epoch, ~3% over plain loading) -- **Trains end-to-end faster** - real ResNet-18/Imagenette: 1.17x vs PyTorch DataLoader (loader ~fully hidden behind the GPU); see `benchmarks/E2E_TRAINING_RESULTS.md` -- **Checkpointable** - `state_dict()/load_state_dict()`: exact, decode-free mid-epoch resumption -- **Pinned recycled buffers** - `pin_memory=True` yields torch tensors from a reused ring; async H2D with `non_blocking=True` -- **Decoded Tensor Caching** - `FastDataLoader(..., cache_decoded=True)` keeps decoded arrays in RAM so later epochs skip decoding -- **Multiple Loader Types** - FastDataLoader, MemoryEfficientDataLoader, standard DataLoader -- **Distributed Training Support** - Multi-node data loading with deterministic sharding -- **SIMD-Accelerated Transforms** - 19 vectorized transforms using AVX2/AVX-512/NEON -- **TBL v2 Binary Format** - Custom format with LZ4 compression for reduced storage -- **Framework-ready outputs** - `output_format='pytorch'/'numpy'/'tensorflow'` batch layouts, zero-copy torch adoption for the GPU loaders, and a shipped `WebDatasetLoader` -- **Memory-Mapped I/O** - Zero-copy file access for improved throughput -- **Lock-Free Queues** - Concurrent data structures for efficient multi-threading -- **GPU image loaders** - `CudaImageLoader` (NVIDIA nvImageCodec — **beats DALI** on an RTX 3090, see below) and `GpuImageLoader` (Apple Metal): end-to-end GPU decode + resize + normalize, GPU-resident output. See [GPU acceleration](docs/GPU_ACCELERATION.md) -- **Resident (pre-processed) epochs** - `CudaResidentLoader` (**~280k img/s**, beats FFCV 3.5×) and `MetalResidentLoader` (**433–757k img/s** on unified memory) + `MetalResidentArrays` for any-dtype rows. Decode once, serve every epoch with one fused gather+shuffle+normalize kernel launch per batch -- **Video loaders** - `MetalVideoLoader` (VideoToolbox **hardware** decode, **3.9× the best industry standard** on an M4 Max) and `CudaVideoLoader` (GPU-resident batches, dual CPU/NVDEC decode backends, novel fused clip-assembly kernel via `iter_clips`). See [video results](benchmarks/VIDEO_RESULTS.md) +- **Fast on CPU**: ~55k img/s on-the-fly (2.0× `tf.data`, 2.7× PyTorch DataLoader); trains a real ResNet-18 **1.17× faster end-to-end** with the input pipeline hidden behind the GPU +- **Fast on GPU**: beats **NVIDIA DALI** on-the-fly (+12%, RTX 3090) and **FFCV** on pre-processed data (1.6–3.5×); ~757k img/s resident on Apple unified memory +- **Video**: hardware decode to training batches — **3.9× the best industry standard** on Apple Silicon; CUDA path with a fused clip-assembly kernel +- **Train-ready**: fused `train_aug` (torchvision-parity RandomResizedCrop+flip), `state_dict()` mid-epoch resume, pinned-memory rings, DDP sharding +- **Also tokens & arrays**: memory-mapped `TokenDataLoader` (~1.9× the nanoGPT idiom), `ArrayDataLoader`, and `MapDataLoader` for any `__getitem__` dataset +- **Every number is honest**: interleaved medians, real consumption, corrections published — [full methodology](docs/benchmarks/index.md) --- ## Which loader do I use? -One decision table for every entry point — pick by data type and hardware, -without reading the internals: +```mermaid +flowchart TD + S{{"What are you loading?"}} + S --> IMG["🖼 Images
(TAR of JPEGs)"] + S --> VID["🎬 Video files"] + S --> TOK["🔤 LLM tokens"] + S --> ARR["📊 Arrays / tabular"] + S --> ANY["🐍 Anything with
__getitem__"] + + IMG --> Q1{"Fits in GPU / unified
memory, many epochs?"} + Q1 -- yes --> RES["CudaResidentLoader · NVIDIA
MetalResidentLoader · Apple"] + Q1 -- no --> Q2{"Where to decode?"} + Q2 -- "CPU fast path (default)" --> DL["DataLoader(output_format='pytorch',
image_size=N)"] + Q2 -- "NVIDIA GPU" --> CIL["CudaImageLoader(decode='nvimgcodec',
return_indices=True)"] + VID --> MV["MetalVideoLoader · Apple
CudaVideoLoader · NVIDIA"] + TOK --> TDL["TokenDataLoader"] + ARR --> ADL["ArrayDataLoader
MetalResidentArrays (GPU gathers)"] + ANY --> MAP["MapDataLoader"] + + style DL stroke-width:3px +``` + +
+Full decision table + lifetime rules | You have | Use | Notes | |---|---|---| @@ -53,51 +78,29 @@ without reading the internals: | Arrays / embeddings / tabular | `ArrayDataLoader`; `MetalResidentArrays` for GPU row gathers | | | WebDataset-style TARs | `WebDatasetLoader` | | -Two lifetime rules to know: (1) loaders yielding **zero-copy views** (`pin_memory=True` -ring, Metal/CUDA resident + video loaders) reuse their buffers — consume or copy a -batch before advancing past the documented window; (2) GPU loaders yield +Two lifetime rules: (1) loaders yielding **zero-copy views** (`pin_memory=True` ring, +Metal/CUDA resident + video loaders) reuse their buffers — consume or copy a batch +before advancing past the documented window; (2) GPU loaders yield `__cuda_array_interface__` objects — adopt with `torch.as_tensor(x, device='cuda')`. -## Installation - -### From PyPI (Recommended) +
-```bash -pip install turboloader -``` +--- -### From Source +## Installation ```bash -git clone https://github.com/ALJainProjects/TurboLoader.git -cd TurboLoader -pip install -e . +pip install turboloader # Linux x86_64/aarch64 + macOS arm64 wheels (CPU + Apple Metal) ``` -### System Requirements - -- **Python:** 3.10 or higher -- **Compiler:** C++20 capable (GCC 10+, Clang 12+, MSVC 19.29+) -- **OS:** Linux (x86_64/aarch64 wheels), macOS (arm64 wheel). Windows: not officially supported yet — use WSL2 - -#### Optional Dependencies - -Install for enhanced performance: - -```bash -# macOS -brew install jpeg-turbo libpng libwebp lz4 - -# Ubuntu/Debian -sudo apt-get install libjpeg-turbo8-dev libpng-dev libwebp-dev liblz4-dev -``` +CUDA loaders: prebuilt cu13 wheel on the [latest release](https://github.com/ALJainProjects/TurboLoader/releases/latest), +or build from source — see [GPU acceleration](docs/GPU_ACCELERATION.md). +Details: [installation guide](docs/installation.md). --- ## Quick Start -### Training input (the fast path — start here) - ```python import turboloader @@ -116,482 +119,95 @@ for images, meta in loader: ... ``` -This is the path all the benchmark numbers refer to. The dict API below is the -flexible per-sample path — several times slower; use it for inspection, not epochs. - -### Basic Usage (per-sample dicts) - -```python -import turboloader - -# Create DataLoader -loader = turboloader.DataLoader( - 'imagenet.tar', - batch_size=128, - num_workers=8 -) - -# Iterate over batches. Each sample is a dict: -# {'image': np.ndarray (H, W, C), 'filename': str, 'index': int, -# 'width': int, 'height': int, 'channels': int} -for batch in loader: - for sample in batch: - image = sample['image'] # NumPy array (H, W, C) - name = sample['filename'] # source path within the archive - # Train your model... -``` - -> **Need (image, label) tuples like `torch.utils.data.DataLoader`?** Use -> `PyTorchCompatibleLoader`, which derives labels from the folder structure -> (ImageFolder-style). The base `DataLoader` does not attach labels. - -### With Transforms - -```python -import turboloader - -# Create transforms -resize = turboloader.Resize(224, 224) -normalize = turboloader.ImageNetNormalize() -flip = turboloader.RandomHorizontalFlip(p=0.5) - -# Apply transforms -loader = turboloader.DataLoader('data.tar', batch_size=64, num_workers=8) - -for batch in loader: - for sample in batch: - img = sample['image'] - img = resize.apply(img) - img = flip.apply(img) - img = normalize.apply(img) - # Ready for training -``` - -### PyTorch Integration - -```python -import turboloader -import torch - -loader = turboloader.DataLoader('imagenet.tar', batch_size=64, num_workers=8) - -# Convert to PyTorch tensors -to_tensor = turboloader.ToTensor( - format=turboloader.TensorFormat.PYTORCH_CHW -) - -for batch in loader: - images = [] - for sample in batch: - img = to_tensor.apply(sample['image']) - images.append(torch.from_numpy(img)) - - batch_tensor = torch.stack(images) - # Train model... -``` - -### Distributed Training - -```python -import turboloader -import torch.distributed as dist - -# Initialize distributed training -dist.init_process_group(backend='nccl') - -# Create loader with distributed support -loader = turboloader.DataLoader( - data_path="/data/imagenet.tar", - batch_size=64, - num_workers=4, - shuffle=True, - enable_distributed=True, - world_rank=dist.get_rank(), - world_size=dist.get_world_size(), - drop_last=True -) - -# Each rank automatically gets its shard -for batch in loader: - # Your training code - pass -``` - ---- - -## Transform Library - -TurboLoader includes 24 transforms (19 per-image SIMD transforms + 5 batch -augmentations). The authoritative list is `turboloader.list_transforms()`. - -### Core Transforms -- **Resize** - Bilinear/Bicubic/Lanczos interpolation -- **Normalize** - Mean/std normalization with SIMD -- **CenterCrop** - Center region extraction -- **RandomCrop** - Random crop with padding - -### Augmentation Transforms -- **RandomHorizontalFlip** - SIMD horizontal flip -- **RandomVerticalFlip** - SIMD vertical flip -- **ColorJitter** - Brightness/contrast/saturation/hue -- **RandomRotation** - Arbitrary angle rotation -- **GaussianBlur** - Separable convolution -- **RandomErasing** - Cutout augmentation -- **Pad** - Border padding (CONSTANT/EDGE/REFLECT) - -### Advanced Transforms -- **RandomPosterize** - Bit-depth reduction -- **RandomSolarize** - Threshold inversion -- **RandomPerspective** - Perspective warp -- **AutoAugment** - Learned policies (ImageNet/CIFAR10/SVHN) - -### Batch Augmentations -- **MixUp**, **CutMix**, **Mosaic**, **RandAugment**, **GridMask** - -### Tensor Conversion -- **ToTensor** - PyTorch CHW or TensorFlow HWC format - ---- - -## TBL v2 Binary Format - -TurboLoader includes a custom binary format optimized for ML workloads: - -### Features -- LZ4 compression for reduced storage -- Memory-mapped access for fast loading -- O(1) random access via indexed structure -- Data integrity validation with CRC checksums -- Cached image dimensions for filtered loading - -### Convert TAR to TBL - -```python -import tarfile -import turboloader - -writer = turboloader.TblWriterV2("/data/imagenet.tbl", enable_compression=True) - -# The TAR archive is read with Python's stdlib (TurboLoader does not expose a -# standalone Python TarReader; the DataLoader reads TAR directly for training). -with tarfile.open("/data/imagenet.tar") as tar: - for member in tar.getmembers(): - if not member.name.lower().endswith((".jpg", ".jpeg")): - continue - data = tar.extractfile(member).read() - writer.add_sample(data=data, format=turboloader.SampleFormat.JPEG) - -writer.finalize() -``` - -> For bulk conversion there is also a C++ CLI tool, `tools/tar_to_tbl_v2.cpp`. - ---- - -## Documentation - -### Getting Started -- **[Quick Start Notebook](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/quickstart.ipynb)** - Interactive tutorial for beginners -- **[Installation Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/installation.md)** - Detailed setup instructions -- **[Quick Start](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/quickstart.md)** - Getting started examples -- **[Troubleshooting Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md)** - Common issues and solutions - -### API Documentation -- **[API Reference](https://github.com/ALJainProjects/TurboLoader/tree/main/docs/api)** - Complete API documentation -- **[Transforms API](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/api/transforms.md)** - All 19 transforms with examples - -### Framework Integration -- **[PyTorch Integration Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/guides/pytorch-integration.md)** - Complete PyTorch guide -- **[TensorFlow Integration Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/guides/tensorflow-integration.md)** - Complete TensorFlow/Keras guide -- **[PyTorch Lightning Example](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/pytorch_lightning_example.py)** - Production-ready Lightning integration -- **[Distributed Training (DDP)](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/distributed_ddp.py)** - Multi-GPU PyTorch DDP example - -### Examples -- **[ImageNet ResNet50 Training](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/imagenet_resnet50.py)** - Complete training pipeline with AMP, checkpointing, TensorBoard -- **[Distributed Training](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/distributed.md)** - Multi-node setup guide - ---- - -## Benchmarks - -Measured on **Apple Silicon** over **Imagenette-160** (9,469 real ImageNet JPEGs → -resize 160×160 → ImageNet-normalize → batched CHW float32, batch 64). To control for -thermal throttling, every loader is built once, warmed up one epoch, then timed over -**5 interleaved rounds** (each loader runs once per round); the table reports the -median. Output is verified correct against torchvision (mean abs diff ≈ 0.04, bilinear -antialiasing only). - -**Image — on-the-fly decode** (re-decode every epoch; for datasets too large to cache -or with per-epoch random augmentation): - -| Loader | img/s (median) | vs tf.data | -|---|---:|---:| -| **TurboLoader `DataLoader`** (`output_format='pytorch'`, nw=6) | **~55,000** | **2.0×** | -| TensorFlow `tf.data` (AUTOTUNE) | ~27,300 | 1.00× | -| PyTorch `DataLoader` (PIL, 8 persistent workers) | ~20,500 | 0.75× | - -**Image — cached** (decoded tensors held in RAM; both sides consume identically via -`np.sum`, i.e. delivered as numpy/torch-ready batches — the PyTorch use case): - -| Loader | img/s (median) | vs tf.data.cache | -|---|---:|---:| -| **TurboLoader** (`cache_decoded=True`, prefetch) | **~67,000** | **1.9×** | -| TensorFlow `tf.data.cache()` (+ `.numpy()` materialize) | ~35,100 | 1.00× | - -(For *TF-native* consumption that stays in tf tensors, `tf.data.cache()` is faster — -TurboLoader's cache win is for delivering numpy/torch batches.) - -**LLM tokens** (real text, 55M-token memory-mapped corpus, `seq_len=1024`, next-token): - -| Loader | sequences/s (median) | -|---|---:| -| **TurboLoader `TokenDataLoader`** | **~467,000** | -| numpy memmap idiom (nanoGPT `get_batch`) | ~251,000 | - -**Transforms** (per-image throughput vs torchvision): Resize **2.7×**, ImageNetNormalize -**3.3×**, HFlip ~1.0×. For CenterCrop, torchvision returns a **lazy strided view** (moves -zero bytes); compared against TurboLoader's real contiguous crop that looks like 0.45×, -but when torchvision actually materializes the crop (`.contiguous()`, required before -batching/most ops) it drops to ~23k img/s and **TurboLoader's contiguous crop is ~6.8× -faster** (155k vs 23k). Like the cache, this is a lazy-vs-eager comparison; for the -realistic crop→batch path TurboLoader wins. +> **Labels**: samples carry no `label` key (a TAR is flat). Use +> `PyTorchCompatibleLoader` for ImageFolder-style `(image, label)` tuples, or align +> a label array via `meta['indices']`. -> Earlier drafts quoted single-run figures (~42k, "1.4×") and a "cached epoch" in the -> tens-of-millions img/s. Those were artifacts (thermal noise; a no-op loop over aliased -> cached arrays) and were replaced with the interleaved, identical-consumption medians -> above. Numbers are hardware-dependent — run `benchmarks/` yourself. - -The fast path runs decode + resize + normalize + batch assembly in C++ across a thread -pool with zero Python per-sample work. Use it like this: - -```python -loader = turboloader.DataLoader( - 'imagenet.tar', batch_size=64, num_workers=6, - output_format='pytorch', # (N, C, H, W) float32 array per batch - image_size=160, # exact resize, done in C++ - transform=turboloader.ImageNetNormalize()) -for epoch in range(epochs): # re-iterable - for images, meta in loader: # images.shape == (64, 3, 160, 160) - train_step(images) -``` - -Honest caveats: -- **Run it yourself** (`benchmarks/`) — results depend heavily on hardware, image size, - and pipeline; Linux `fork`-based PyTorch workers shift the PyTorch numbers a lot. -- **Decode backend differs**: TurboLoader uses libjpeg-turbo; the PyTorch baseline uses PIL. -- The `output_format='dict'` path returns per-sample dicts and stacks in Python - (GIL-bound), so it is much slower — use it only when you need per-sample metadata. - -For **large source images**, the default path also wins: on 768×768 JPEGs resized to -160 it runs ~15,000 img/s — faster than even an expertly-tuned `tf.data` pipeline using -manual `decode_jpeg(ratio=...)` (~14,400) — because it picks the libjpeg-turbo DCT -scaled-decode factor automatically (you don't have to know to set `ratio`). - -### GPU loaders (NVIDIA & Apple) - -On **NVIDIA**, `CudaImageLoader(decode="nvimgcodec")` runs the whole decode + resize + normalize -+ batch in GIL-released C++ via **nvImageCodec** (the codec DALI uses), with K independent decode -slots overlapping batches (multi-batch-in-flight). Among **on-the-fly** loaders (read a JPEG -folder, decode+resize every epoch) on an **RTX 3090** (Imagenette-160, batch 64, real consumption, -interleaved rounds to control for ~40% host drift): - -| On-the-fly loader | vs TurboLoader | -|---|---:| -| **TurboLoader** `decode="nvimgcodec"`, `nvimgcodec_slots=3` | **1.0× (fastest)** | -| NVIDIA **DALI** (`num_threads=8`, best-tuned) | ~0.9× (TurboLoader **+12%** cleanest run) | -| PyTorch `DataLoader` (PIL, CPU) | ~0.25× | - -**TurboLoader beats DALI** (median above DALI's max in the cleanest run), output bijectively -verified correct. For **on-the-fly** loading FFCV is faster (~2.6–5.9×) — but it requires an -offline conversion to its `.beton` format. - -**Pre-processed loaders** (decode+resize once, like FFCV's `.beton`) — here TurboLoader turns the -tables: - -| Pre-processed loader | img/s | | -|---|---:|---| -| **TurboLoader `MetalResidentLoader`** (Apple M4 Max, unified memory: no H2D exists) | **~757,000 produced / ~433,000 consumed** | ships in the pip wheel | -| **TurboLoader `CudaResidentLoader`** (fits-in-VRAM: upload uint8 once, GPU-resident) | **~280,000** | **beats FFCV ~3.5×** | -| **TurboLoader `CudaStreamLoader`** (streaming, dataset > VRAM; fully-C++ loop) | **~140,000** | **beats FFCV ~1.6×** | -| FFCV, raw `.beton` (streams mmap→H2D each epoch, worker processes) | ~85,000 | | - -On **Apple Silicon** the resident trick is even better than on NVIDIA: memory is unified, so -"upload" is one memcpy and every GPU-written batch is a **zero-copy numpy view**. -`MetalResidentLoader` serves each epoch as one fused gather+shuffle+normalize kernel launch per -batch; `MetalResidentArrays` does the same for any-dtype rows (embedding tables: ~5× numpy -fancy-indexing). Honest null result included: `MetalTokenGather` ties the CPU memmap path -(0.87–1.08×) — keep using `TokenDataLoader` for tokens. - -**Video**: `MetalVideoLoader` (macOS arm64, in the pip wheel — no FFmpeg needed) drives -VideoToolbox **hardware** H.264/HEVC decode into a fused NV12→RGB+resize+normalize Metal -kernel: real 1080p → 224px training batches at **~2,550 frames/s** on an M4 Max — -**3.9× the best industry standard** (OpenCV 657, PyAV 535, torchcodec 173) and 97–99% of -the media engine's hardware decode ceiling. On NVIDIA, `CudaVideoLoader` (CUDA build) -lands GPU-resident batches via a dual decode backend (threaded CPU decode by default; -NVDEC opt-in — measured virtualization-throttled under WSL2) plus a novel **fused -clip-assembly kernel** (`iter_clips`: consistent RandomResizedCrop+flip across a whole -clip + YUV→RGB + resize + normalize in ONE launch). Honest scorecard incl. where decord -still wins on weak-CPU hosts: [benchmarks/VIDEO_RESULTS.md](benchmarks/VIDEO_RESULTS.md). - -`CudaResidentLoader` uses a custom single-launch normalize kernel + fused gather (shuffles at -~257k) and **beats FFCV ~3.5×** when the pre-processed uint8 dataset fits in VRAM (very common: -fine-tuning, per-GPU shards, small/medium sets). For datasets **larger than VRAM**, -`CudaStreamLoader` runs the whole iteration GIL-free in C++ (`CudaStreamCore`: worker pool + async -H2D on non-blocking streams + prefetch) and **beats FFCV's streaming ~1.6×** (~140k vs ~85k, near -the PCIe transfer ceiling). So TurboLoader beats **DALI** on-the-fly and **FFCV** on pre-processed -data — both fits-in-VRAM and streaming. On **Apple Silicon**, `GpuImageLoader` offloads -resize+normalize (and a hybrid GPU JPEG decode) to Metal — where neither DALI nor FFCV runs at -all. CUDA is a build-from-source path (not -in the PyPI wheels); see [GPU acceleration](docs/GPU_ACCELERATION.md) for flags, usage, and the -full write-up (`experiments/cuda/RESULTS.md`). - -### Implementation notes -- **Direct-batch path** (`src/pipeline/direct_batch_loader.hpp`): the default fast path - is FFCV/`tf.data`-style — a persistent thread pool reads JPEG bytes by index and - decodes → resizes → normalizes **directly into the output batch buffer** in one - parallel pass (no worker queue, no per-sample heap copy, no serial collection). - Verified memory-safe and race-free (disjoint slot writes, const mmap reads, atomic - cursor, per-thread decoders). -- **Automatic DCT scaled decode**: large JPEGs are decoded at the nearest libjpeg-turbo - scale ≥ target, then finely resized — much faster than full-decode + resize. -- **Resize convention**: half-pixel centers (`align_corners=False`), matching - PIL/OpenCV/PyTorch/TF (agrees with torchvision plain bilinear to ~0.4/255; the only - remaining difference vs torchvision's default is its antialiasing low-pass filter). -- SIMD transforms (AVX2/AVX-512/NEON), libjpeg-turbo decode, lock-free SPSC queues - (legacy/dict + remote path), persistent `std::thread` pool (`src/core/parallel_for.hpp`). -- The GIL is released during C++ processing. -- **OpenMP is opt-in** (`TURBOLOADER_ENABLE_OPENMP=1`); off by default because linking a - second OpenMP runtime crashes alongside PyTorch on macOS — the thread pool replaces it. +More: [quickstart](docs/quickstart.md) · [per-sample dict API & transforms](docs/getting-started.md) · +[tokens, arrays & any Python dataset](docs/tokens_arrays.md) · +[interactive notebook](examples/quickstart.ipynb) --- -## Beyond Images: Tokens & Arrays - -TurboLoader also ships loaders for non-image modalities with the same ergonomics -(re-iterable, `shuffle`, `set_epoch`, batched arrays): - -```python -# LLM pretraining: memory-mapped token stream -> (B, seq_len) next-token batches -loader = turboloader.TokenDataLoader('train.bin', seq_len=1024, batch_size=8, - dtype='uint16', shuffle=True) -for x, y in loader: # x, y: (8, 1024) int64; y is x shifted by one - loss = model(x, y) - -# Generic arrays/memmaps (embeddings, tabular features, labels, pre-tokenized data) -loader = turboloader.ArrayDataLoader(features, labels, batch_size=256, shuffle=True) -for xb, yb in loader: - ... -``` - -`TokenDataLoader` uses a vectorized fancy-index gather over a `np.memmap` (so multi-GB -corpora stream without loading into RAM) and benchmarks ~1.9× the standard nanoGPT -`get_batch` idiom. The image pipeline (decode/transform/TBL) remains C++; these -modality loaders are NumPy-based and modality-agnostic. - -All three modalities are also reachable from the **single `DataLoader` entry point**: - -```python -turboloader.DataLoader('train.bin', modality='tokens', seq_len=1024, batch_size=8) -turboloader.DataLoader(arrays=[feats, labels], data_path=None, modality='array', batch_size=256) -turboloader.DataLoader('data.tar', image_size=160, output_format='pytorch') # modality='image' (default) -``` - -### Wrap *any* Python dataset (`MapDataLoader`) - -When your data doesn't fit the native paths, `MapDataLoader` batches **any** map-style -dataset — anything with `__len__` and `__getitem__(i)`, i.e. exactly the -`torch.utils.data.Dataset` protocol — so your loading/decoding/business logic can be -arbitrary Python: - -```python -class MyDataset: - def __len__(self): return len(self.records) - def __getitem__(self, i): - x = decode_however_you_like(self.records[i]) # any Python logic - return x, self.labels[i] # (features, label) - -# directly, or via the unified entry point with dataset=... -for xb, yb in turboloader.MapDataLoader(MyDataset(), batch_size=64, shuffle=True, num_workers=8): - train_step(xb, yb) -``` - -It parallelizes `__getitem__` on a bounded thread pool with read-ahead and collates -(tuples/dicts/arrays, or a custom `collate_fn`). **Honest tradeoff:** because the -per-sample work runs in Python, this path is roughly PyTorch-`DataLoader` speed (and -GIL-bound for pure-Python CPU work — threads help most when `__getitem__` releases the -GIL, e.g. NumPy/PIL/file/network I/O). It's about *flexibility*, not the C++ fast path — -use the image/token/array loaders above when you want maximum throughput. +## Benchmarks (headlines) + +Real data, real consumption, interleaved medians, warmup excluded. **Run them +yourself** — scripts in [`benchmarks/`](benchmarks/), full methodology + honest +caveats (and the corrections we published) in [docs/benchmarks](docs/benchmarks/index.md). + +| Regime | TurboLoader | Best alternative | Hardware | +|---|---:|---|---| +| On-the-fly CPU (decode every epoch) | **~55k img/s** | `tf.data` ~27k · PyTorch ~20k | M4 Max | +| On-the-fly GPU | **28.5k img/s** | NVIDIA DALI 25.5k (**+12%**) | RTX 3090 | +| Pre-processed, fits in VRAM | **~280k img/s** | FFCV ~80k (**3.5×**) | RTX 3090 | +| Pre-processed, streaming > VRAM | **~140k img/s** | FFCV ~85k (**1.6×**) | RTX 3090 | +| Pre-processed, unified memory | **433–757k img/s** | numpy resident ~3.7k | M4 Max | +| Video → training batches | **2,556 f/s (3.9×)** | OpenCV 657 · PyAV 535 · torchcodec 173 | M4 Max | +| End-to-end ResNet-18 training | **1.17×** vs PyTorch recipe | at the pure-GPU floor | RTX 3090 | +| LLM tokens (memmap) | **~467k seq/s** | nanoGPT `get_batch` ~251k | M4 Max | + +Honest notes worth knowing before you quote these: FFCV is faster than us +*on-the-fly is impossible for it* (needs `.beton` conversion); decord beats our CUDA +video cpu-backend on weak-CPU hosts; `MetalTokenGather` ties the CPU path (so we +recommend the CPU path); e2e ResNet-18 on Apple MPS is a **tie** because the GPU is +the bottleneck. All in the [full write-ups](docs/benchmarks/index.md). --- ## Architecture -TurboLoader uses a multi-threaded pipeline architecture: - -``` -┌─────────────────────────────────────────────┐ -│ Memory-Mapped Reader │ -│ (TAR/TBL v2 with zero-copy access) │ -└──────────────┬──────────────────────────────┘ - │ - ┌──────▼──────┐ - │Worker Pool │ - │ (N threads)│ - ├─────────────┤ - │ Decode │ - │ Transform │ - │ Convert │ - └──────┬──────┘ - │ - ┌──────▼──────────────┐ - │ Lock-Free Queue │ - └──────┬──────────────┘ - │ - ┌──────▼──────┐ - │Python API │ - └─────────────┘ -``` - -### Key Components - -- **Memory-Mapped I/O** - Zero-copy file access -- **Worker Thread Pool** - Parallel processing with per-thread decoders -- **SIMD Transforms** - Vectorized operations (AVX2/AVX-512/NEON) -- **Lock-Free Queues** - High-performance concurrent data structures +```mermaid +flowchart TD + subgraph PY["Python (thin orchestration)"] + API["DataLoader · TokenDataLoader · ArrayDataLoader · video/GPU loaders"] + end + subgraph CPP["C++20 core — GIL released"] + MM["memory-mapped TAR / TBL v2 reader"] + POOL["persistent thread pool
per-thread libjpeg-turbo decoders"] + SIMD["SIMD transforms
NEON / AVX2 / AVX-512"] + BUF["fused write into the
output batch buffer"] + MM --> POOL --> SIMD --> BUF + end + subgraph GPU["GPU kernels"] + METAL["Apple Metal
resident · video · transforms"] + CUDA["NVIDIA CUDA + nvImageCodec
resident · stream · video · clips"] + end + API --> MM + BUF --> API + API -.-> METAL + API -.-> CUDA +``` + +Deep dive: [architecture](docs/architecture.md) · [GPU acceleration](docs/GPU_ACCELERATION.md) · +[transform library (24 transforms)](docs/transforms.md) · [TBL v2 binary format](docs/tbl_v2_format.md) --- -## License +## Documentation -TurboLoader is released under the MIT License. +| | | +|---|---| +| Getting started | [installation](docs/installation.md) · [quickstart](docs/quickstart.md) · [notebook](examples/quickstart.ipynb) · [troubleshooting](docs/TROUBLESHOOTING.md) | +| API | [API reference](docs/api) · [transforms](docs/api/transforms.md) | +| Guides | [PyTorch](docs/guides/pytorch-integration.md) · [TensorFlow](docs/guides/tensorflow-integration.md) · [distributed (DDP)](docs/distributed.md) | +| Examples | [ResNet-50 training](examples/imagenet_resnet50.py) · [Lightning](examples/pytorch_lightning_example.py) · [DDP](examples/distributed_ddp.py) · [GPT on tokens](examples/train_gpt_tokenloader.py) | +| Benchmarks | [methodology + full results](docs/benchmarks/index.md) · [video](benchmarks/VIDEO_RESULTS.md) · [Metal resident](benchmarks/METAL_RESIDENT_RESULTS.md) · [e2e training](benchmarks/E2E_TRAINING_RESULTS.md) | --- -## Citation +## License & Citation -If you use TurboLoader in your research: +MIT. If you use TurboLoader in your research: ```bibtex @software{turboloader, author = {Jain, Arnav}, title = {TurboLoader: High-Performance ML Data Loading}, year = {2026}, - version = {2.25.0}, url = {https://github.com/ALJainProjects/TurboLoader} } ``` ---- - -## Support - -- **Documentation:** [https://github.com/ALJainProjects/TurboLoader/tree/main/docs](https://github.com/ALJainProjects/TurboLoader/tree/main/docs) -- **Troubleshooting:** [https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md) -- **Verification Script:** Run `python scripts/verify_installation.py` to check your setup -- **Issues:** [GitHub Issues](https://github.com/ALJainProjects/TurboLoader/issues) -- **Discussions:** [GitHub Discussions](https://github.com/ALJainProjects/TurboLoader/discussions) -- **PyPI:** [https://pypi.org/project/turboloader/](https://pypi.org/project/turboloader/) - ---- - -TurboLoader - High-performance ML data loading with a C++20 core and SIMD transforms. +**Support**: [issues](https://github.com/ALJainProjects/TurboLoader/issues) · +[discussions](https://github.com/ALJainProjects/TurboLoader/discussions) · +[PyPI](https://pypi.org/project/turboloader/) · `python scripts/verify_installation.py` diff --git a/docs/assets/demo.gif b/docs/assets/demo.gif new file mode 100644 index 0000000..e07bc5b Binary files /dev/null and b/docs/assets/demo.gif differ diff --git a/docs/assets/demo.tape b/docs/assets/demo.tape new file mode 100644 index 0000000..f80700d --- /dev/null +++ b/docs/assets/demo.tape @@ -0,0 +1,25 @@ +Output demo.gif +Set FontSize 16 +Set Width 920 +Set Height 560 +Set Theme "Catppuccin Mocha" +Set TypingSpeed 35ms +Set Framerate 24 +Set Shell bash + +Hide +Type "python3 -m venv gifenv && source gifenv/bin/activate && export PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_NO_PYTHON_VERSION_WARNING=1" +Enter +Sleep 8s +Type "clear" +Enter +Sleep 2s +Show + +Type "pip install turboloader" +Enter +Sleep 8s +Type "python demo.py" +Enter +Sleep 7s +Sleep 3s diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md index 2a76842..909fddb 100644 --- a/docs/benchmarks/index.md +++ b/docs/benchmarks/index.md @@ -1,167 +1,158 @@ -# Benchmark Overview +# Benchmarks — full methodology and results -Performance analysis of TurboLoader 2.33.0. +The complete, honest scorecard (corrections included). Headlines live in the +[README](../../README.md); raw scripts in [`benchmarks/`](../../benchmarks/); +GPU details in [`experiments/cuda/RESULTS.md`](../../experiments/cuda/RESULTS.md), +[`benchmarks/METAL_RESIDENT_RESULTS.md`](../../benchmarks/METAL_RESIDENT_RESULTS.md), +[`benchmarks/VIDEO_RESULTS.md`](../../benchmarks/VIDEO_RESULTS.md), and +[`benchmarks/E2E_TRAINING_RESULTS.md`](../../benchmarks/E2E_TRAINING_RESULTS.md). -## Executive Summary -TurboLoader's direct-batch image loader (one parallel pass: decode -> resize -> -normalize straight into the output batch buffer) reaches **~39,100 img/s on the fly** -and **~65,499 img/s with the decoded cache enabled** (`cache_decoded=True`) on Apple -Silicon over Imagenette-160. Measured against the same dataset and the same forced, -real consumption, that makes it: +Measured on **Apple Silicon** over **Imagenette-160** (9,469 real ImageNet JPEGs → +resize 160×160 → ImageNet-normalize → batched CHW float32, batch 64). To control for +thermal throttling, every loader is built once, warmed up one epoch, then timed over +**5 interleaved rounds** (each loader runs once per round); the table reports the +median. Output is verified correct against torchvision (mean abs diff ≈ 0.04, bilinear +antialiasing only). -- **1.3x faster** than TensorFlow `tf.data` (AUTOTUNE): ~30,154 img/s -- **2.1x faster** than PyTorch `DataLoader` (8 persistent workers): ~18,991 img/s -- For LLM token streams, `TokenDataLoader` sustains **~441M tokens/s** vs **~163M - tokens/s** for the NumPy `memmap` idiom (**2.7x**) +**Image — on-the-fly decode** (re-decode every epoch; for datasets too large to cache +or with per-epoch random augmentation): -All image numbers use `output_format='pytorch'` (CHW), batch size 64, with a warmup -epoch and the median of 3 timed epochs under real consumption that forces -materialization of every batch. +| Loader | img/s (median) | vs tf.data | +|---|---:|---:| +| **TurboLoader `DataLoader`** (`output_format='pytorch'`, nw=6) | **~55,000** | **2.0×** | +| TensorFlow `tf.data` (AUTOTUNE) | ~27,300 | 1.00× | +| PyTorch `DataLoader` (PIL, 8 persistent workers) | ~20,500 | 0.75× | -## Latest Results (2.33.0) +**Image — cached** (decoded tensors held in RAM; both sides consume identically via +`np.sum`, i.e. delivered as numpy/torch-ready batches — the PyTorch use case): -### Image Throughput (Imagenette-160) +| Loader | img/s (median) | vs tf.data.cache | +|---|---:|---:| +| **TurboLoader** (`cache_decoded=True`, prefetch) | **~67,000** | **1.9×** | +| TensorFlow `tf.data.cache()` (+ `.numpy()` materialize) | ~35,100 | 1.00× | -| Loader | Throughput (img/s) | Relative | -|--------|--------------------|----------| -| **TurboLoader (cached, `cache_decoded=True`)** | **~65,499** | 1.7x vs on-the-fly | -| **TurboLoader (on-the-fly)** | **~39,100** | 1.0x (reference) | -| TensorFlow `tf.data` (AUTOTUNE) | ~30,154 | TurboLoader is 1.3x | -| PyTorch `DataLoader` (8 persistent workers) | ~18,991 | TurboLoader is 2.1x | +(For *TF-native* consumption that stays in tf tensors, `tf.data.cache()` is faster — +TurboLoader's cache win is for delivering numpy/torch batches.) -The on-the-fly path decodes, resizes, and normalizes each image in a single parallel -pass into the output batch buffer, with automatic libjpeg-turbo DCT scaled decode for -large images. The cached path stores decoded tensors so subsequent epochs skip JPEG -decoding entirely. +**LLM tokens** (real text, 55M-token memory-mapped corpus, `seq_len=1024`, next-token): -### LLM Token Streams - -| Loader | Throughput (tokens/s) | Relative | -|--------|-----------------------|----------| -| **TurboLoader `TokenDataLoader`** | **~441M** | TurboLoader is 2.7x | -| NumPy `memmap` idiom | ~163M | 1.0x (reference) | - -### Test Configuration - -- **Hardware:** Apple Silicon -- **Dataset:** Imagenette-160 — 9,469 real ImageNet JPEGs resized to 160 px -- **Output format:** `pytorch` (CHW) -- **Batch size:** 64 -- **Measurement:** real consumption forcing materialization of each batch; one warmup - epoch followed by the median of 3 timed epochs - -### A Note on Worker Scaling - -`num_workers` does not mean the same thing across loaders, so it is not a fair single -knob to sweep: - -- **PyTorch** scales with `num_workers` because each worker is a separate OS process. -- **TurboLoader's** fast path is a single process-wide C++ thread pool that is already - saturated at one "worker" — adding workers does not change its throughput. -- **TensorFlow** uses `AUTOTUNE` and picks its own parallelism. - -The numbers above use each loader's recommended/best configuration: PyTorch with 8 -persistent workers, `tf.data` with AUTOTUNE, and TurboLoader's single saturated thread -pool. - -## Multi-Modality - -The same engine drives more than images: - -- **Images** packed in WebDataset TAR shards via the direct-batch loader. -- **LLM token streams** via `TokenDataLoader` (see the token numbers above). -- **Generic `(N, ...)` arrays** via `ArrayDataLoader`. - -Output can be emitted as NumPy, PyTorch CHW, or TensorFlow HWC. Distributed training is -supported with DDP-safe equal/disjoint sharding. - -## Transform Performance - -Transforms run on SIMD-vectorized kernels (NEON on Apple Silicon / ARM, AVX2 and -AVX-512 on x86). Resize uses half-pixel sampling that matches PIL/PyTorch/TF, with -optional antialiasing. Because transforms are fused into the same parallel pass that -produces the output batch, their cost is already included in the end-to-end throughput -numbers reported above rather than measured in isolation. - -## Memory Usage - -The direct-batch loader writes decoded, resized, normalized samples into a single -reusable output batch buffer instead of allocating per-sample intermediates, which -keeps the steady-state working set small. Enabling `cache_decoded=True` trades memory -for speed by retaining decoded tensors across epochs. - -## Methodology - -See the [Benchmark Setup Guide](../benchmark_setup.md) for: +| Loader | sequences/s (median) | +|---|---:| +| **TurboLoader `TokenDataLoader`** | **~467,000** | +| numpy memmap idiom (nanoGPT `get_batch`) | ~251,000 | + +**Transforms** (per-image throughput vs torchvision): Resize **2.7×**, ImageNetNormalize +**3.3×**, HFlip ~1.0×. For CenterCrop, torchvision returns a **lazy strided view** (moves +zero bytes); compared against TurboLoader's real contiguous crop that looks like 0.45×, +but when torchvision actually materializes the crop (`.contiguous()`, required before +batching/most ops) it drops to ~23k img/s and **TurboLoader's contiguous crop is ~6.8× +faster** (155k vs 23k). Like the cache, this is a lazy-vs-eager comparison; for the +realistic crop→batch path TurboLoader wins. + +> Earlier drafts quoted single-run figures (~42k, "1.4×") and a "cached epoch" in the +> tens-of-millions img/s. Those were artifacts (thermal noise; a no-op loop over aliased +> cached arrays) and were replaced with the interleaved, identical-consumption medians +> above. Numbers are hardware-dependent — run `benchmarks/` yourself. + +The fast path runs decode + resize + normalize + batch assembly in C++ across a thread +pool with zero Python per-sample work. Use it like this: + +```python +loader = turboloader.DataLoader( + 'imagenet.tar', batch_size=64, num_workers=6, + output_format='pytorch', # (N, C, H, W) float32 array per batch + image_size=160, # exact resize, done in C++ + transform=turboloader.ImageNetNormalize()) +for epoch in range(epochs): # re-iterable + for images, meta in loader: # images.shape == (64, 3, 160, 160) + train_step(images) +``` -- Installing the frameworks compared (TurboLoader, PyTorch, TensorFlow) -- Dataset preparation (Imagenette-160) -- Measurement technique (real consumption, warmup + median of timed epochs) +Honest caveats: +- **Run it yourself** (`benchmarks/`) — results depend heavily on hardware, image size, + and pipeline; Linux `fork`-based PyTorch workers shift the PyTorch numbers a lot. +- **Decode backend differs**: TurboLoader uses libjpeg-turbo; the PyTorch baseline uses PIL. +- The `output_format='dict'` path returns per-sample dicts and stacks in Python + (GIL-bound), so it is much slower — use it only when you need per-sample metadata. -## NVIDIA GPU: vs DALI and FFCV (RTX 3090) +For **large source images**, the default path also wins: on 768×768 JPEGs resized to +160 it runs ~15,000 img/s — faster than even an expertly-tuned `tf.data` pipeline using +manual `decode_jpeg(ratio=...)` (~14,400) — because it picks the libjpeg-turbo DCT +scaled-decode factor automatically (you don't have to know to set `ratio`). -The high-performance GPU comparison, run on its home turf (Linux + RTX 3090). TurboLoader's -`CudaImageLoader(decode="nvimgcodec")` runs the whole decode + resize + normalize + batch in -GIL-released C++ via **nvImageCodec** (the codec DALI uses), with K independent decode slots -overlapping batches. Imagenette-160, batch 64, real consumption, **interleaved** rounds (each -loader timed adjacently per round, because the host drifts ~40% run-to-run — only within-run -relative numbers are reliable). +### GPU loaders (NVIDIA & Apple) -**On-the-fly loaders** (read a JPEG folder, decode+resize+normalize every epoch): +On **NVIDIA**, `CudaImageLoader(decode="nvimgcodec")` runs the whole decode + resize + normalize ++ batch in GIL-released C++ via **nvImageCodec** (the codec DALI uses), with K independent decode +slots overlapping batches (multi-batch-in-flight). Among **on-the-fly** loaders (read a JPEG +folder, decode+resize every epoch) on an **RTX 3090** (Imagenette-160, batch 64, real consumption, +interleaved rounds to control for ~40% host drift): -| Loader | vs TurboLoader | +| On-the-fly loader | vs TurboLoader | |---|---:| | **TurboLoader** `decode="nvimgcodec"`, `nvimgcodec_slots=3` | **1.0× (fastest)** | -| NVIDIA **DALI** (`num_threads=8/12`, best-tuned) | ~0.9× | +| NVIDIA **DALI** (`num_threads=8`, best-tuned) | ~0.9× (TurboLoader **+12%** cleanest run) | | PyTorch `DataLoader` (PIL, CPU) | ~0.25× | -**TurboLoader beats DALI**: in the cleanest run its median (28,527, min 25,676) sits above DALI's -max (26,743) — **+12%**; it is ≥ DALI in every run (+6–28% by host load). Output is bijectively -verified correct vs the single-slot synced path (96/96 batches exact) and correlates 0.99986 with -the libjpeg-turbo path. Journey: **9.2k → 14.5k → 22.4k → 28.5k** img/s (nvJPEG → Python -nvImageCodec → single-slot C++ → K-slot async multi-stream). +**TurboLoader beats DALI** (median above DALI's max in the cleanest run), output bijectively +verified correct. For **on-the-fly** loading FFCV is faster (~2.6–5.9×) — but it requires an +offline conversion to its `.beton` format. -**FFCV can't do on-the-fly** — it always requires a one-time offline conversion to its `.beton` -format. So the fair FFCV comparison is *pre-processed vs pre-processed* — and there TurboLoader -turns the tables: +**Pre-processed loaders** (decode+resize once, like FFCV's `.beton`) — here TurboLoader turns the +tables: -| Pre-processed loader (RTX 3090, isolated) | img/s | | +| Pre-processed loader | img/s | | |---|---:|---| -| **TurboLoader `CudaResidentLoader`** (fits-in-VRAM, GPU-resident, custom kernel) | **~280,000** | **beats FFCV ~3.5×** | -| **TurboLoader `CudaStreamLoader`** (streaming > VRAM, fully-C++ GIL-free loop) | **~140,000** | **beats FFCV ~1.6×** | -| FFCV, raw `.beton` (streams mmap→H2D, worker processes) | ~85,000 | | - -Both TurboLoader loaders decode+resize once (like FFCV's `.beton`) then keep the uint8 on the GPU: -`CudaResidentLoader` normalizes GPU-resident data with **zero per-epoch H2D** via a custom -single-launch kernel (~280k; shuffles at ~257k via a fused gather kernel), and `CudaStreamLoader` -streams larger-than-VRAM data with a fully-in-C++ prefetch loop (~140k, near the PCIe ceiling). -Measured isolated (the real single-loader-feeds-training case); correctness bijectively verified. -CUDA is a build-from-source path (not in the PyPI wheels); see -[GPU acceleration](../GPU_ACCELERATION.md) and `experiments/cuda/RESULTS.md`. Caveat: one RTX 3090 -(GPU-hybrid JPEG decode) at 160px. - -So the honest statement: TurboLoader is **measured faster than PyTorch and tf.data on CPU** (above), -**faster than NVIDIA DALI on-the-fly**, and **faster than FFCV on pre-processed data — both -fits-in-VRAM and streaming** — on an NVIDIA GPU, while running the same unified API on the CPU and -on Apple Metal, where neither DALI nor FFCV runs at all. - -## Reproducing Results - -```bash -# Clone repository -git clone https://github.com/ALJainProjects/TurboLoader.git -cd TurboLoader - -# Install (prebuilt manylinux wheels on Linux x86_64 / aarch64) -pip install turboloader # torch is optional: pip install turboloader[torch] - -# Run the image benchmark against Imagenette-160 (batch 64) -cd benchmarks -python benchmark_comparison.py --dataset /path/to/imagenette-160 --batch-size 64 -``` - -## Questions? - -- [Benchmark Setup](../benchmark_setup.md) - How to reproduce these numbers -- [GitHub Issues](https://github.com/ALJainProjects/TurboLoader/issues) +| **TurboLoader `MetalResidentLoader`** (Apple M4 Max, unified memory: no H2D exists) | **~757,000 produced / ~433,000 consumed** | ships in the pip wheel | +| **TurboLoader `CudaResidentLoader`** (fits-in-VRAM: upload uint8 once, GPU-resident) | **~280,000** | **beats FFCV ~3.5×** | +| **TurboLoader `CudaStreamLoader`** (streaming, dataset > VRAM; fully-C++ loop) | **~140,000** | **beats FFCV ~1.6×** | +| FFCV, raw `.beton` (streams mmap→H2D each epoch, worker processes) | ~85,000 | | + +On **Apple Silicon** the resident trick is even better than on NVIDIA: memory is unified, so +"upload" is one memcpy and every GPU-written batch is a **zero-copy numpy view**. +`MetalResidentLoader` serves each epoch as one fused gather+shuffle+normalize kernel launch per +batch; `MetalResidentArrays` does the same for any-dtype rows (embedding tables: ~5× numpy +fancy-indexing). Honest null result included: `MetalTokenGather` ties the CPU memmap path +(0.87–1.08×) — keep using `TokenDataLoader` for tokens. + +**Video**: `MetalVideoLoader` (macOS arm64, in the pip wheel — no FFmpeg needed) drives +VideoToolbox **hardware** H.264/HEVC decode into a fused NV12→RGB+resize+normalize Metal +kernel: real 1080p → 224px training batches at **~2,550 frames/s** on an M4 Max — +**3.9× the best industry standard** (OpenCV 657, PyAV 535, torchcodec 173) and 97–99% of +the media engine's hardware decode ceiling. On NVIDIA, `CudaVideoLoader` (CUDA build) +lands GPU-resident batches via a dual decode backend (threaded CPU decode by default; +NVDEC opt-in — measured virtualization-throttled under WSL2) plus a novel **fused +clip-assembly kernel** (`iter_clips`: consistent RandomResizedCrop+flip across a whole +clip + YUV→RGB + resize + normalize in ONE launch). Honest scorecard incl. where decord +still wins on weak-CPU hosts: [benchmarks/VIDEO_RESULTS.md](../../benchmarks/VIDEO_RESULTS.md). + +`CudaResidentLoader` uses a custom single-launch normalize kernel + fused gather (shuffles at +~257k) and **beats FFCV ~3.5×** when the pre-processed uint8 dataset fits in VRAM (very common: +fine-tuning, per-GPU shards, small/medium sets). For datasets **larger than VRAM**, +`CudaStreamLoader` runs the whole iteration GIL-free in C++ (`CudaStreamCore`: worker pool + async +H2D on non-blocking streams + prefetch) and **beats FFCV's streaming ~1.6×** (~140k vs ~85k, near +the PCIe transfer ceiling). So TurboLoader beats **DALI** on-the-fly and **FFCV** on pre-processed +data — both fits-in-VRAM and streaming. On **Apple Silicon**, `GpuImageLoader` offloads +resize+normalize (and a hybrid GPU JPEG decode) to Metal — where neither DALI nor FFCV runs at +all. CUDA is a build-from-source path (not +in the PyPI wheels); see [GPU acceleration](../GPU_ACCELERATION.md) for flags, usage, and the +full write-up (`experiments/cuda/RESULTS.md`). + +### Implementation notes +- **Direct-batch path** (`src/pipeline/direct_batch_loader.hpp`): the default fast path + is FFCV/`tf.data`-style — a persistent thread pool reads JPEG bytes by index and + decodes → resizes → normalizes **directly into the output batch buffer** in one + parallel pass (no worker queue, no per-sample heap copy, no serial collection). + Verified memory-safe and race-free (disjoint slot writes, const mmap reads, atomic + cursor, per-thread decoders). +- **Automatic DCT scaled decode**: large JPEGs are decoded at the nearest libjpeg-turbo + scale ≥ target, then finely resized — much faster than full-decode + resize. +- **Resize convention**: half-pixel centers (`align_corners=False`), matching + PIL/OpenCV/PyTorch/TF (agrees with torchvision plain bilinear to ~0.4/255; the only + remaining difference vs torchvision's default is its antialiasing low-pass filter). +- SIMD transforms (AVX2/AVX-512/NEON), libjpeg-turbo decode, lock-free SPSC queues + (legacy/dict + remote path), persistent `std::thread` pool (`src/core/parallel_for.hpp`). +- The GIL is released during C++ processing. +- **OpenMP is opt-in** (`TURBOLOADER_ENABLE_OPENMP=1`); off by default because linking a + second OpenMP runtime crashes alongside PyTorch on macOS — the thread pool replaces it. diff --git a/docs/tbl_v2_format.md b/docs/tbl_v2_format.md new file mode 100644 index 0000000..fce61ba --- /dev/null +++ b/docs/tbl_v2_format.md @@ -0,0 +1,33 @@ +# TBL v2 Binary Format + + +TurboLoader includes a custom binary format optimized for ML workloads: + +### Features +- LZ4 compression for reduced storage +- Memory-mapped access for fast loading +- O(1) random access via indexed structure +- Data integrity validation with CRC checksums +- Cached image dimensions for filtered loading + +### Convert TAR to TBL + +```python +import tarfile +import turboloader + +writer = turboloader.TblWriterV2("/data/imagenet.tbl", enable_compression=True) + +# The TAR archive is read with Python's stdlib (TurboLoader does not expose a +# standalone Python TarReader; the DataLoader reads TAR directly for training). +with tarfile.open("/data/imagenet.tar") as tar: + for member in tar.getmembers(): + if not member.name.lower().endswith((".jpg", ".jpeg")): + continue + data = tar.extractfile(member).read() + writer.add_sample(data=data, format=turboloader.SampleFormat.JPEG) + +writer.finalize() +``` + +> For bulk conversion there is also a C++ CLI tool, `tools/tar_to_tbl_v2.cpp`. diff --git a/docs/tokens_arrays.md b/docs/tokens_arrays.md new file mode 100644 index 0000000..a9fa488 --- /dev/null +++ b/docs/tokens_arrays.md @@ -0,0 +1,57 @@ +# Beyond Images: Tokens, Arrays & Any Python Dataset + + +TurboLoader also ships loaders for non-image modalities with the same ergonomics +(re-iterable, `shuffle`, `set_epoch`, batched arrays): + +```python +# LLM pretraining: memory-mapped token stream -> (B, seq_len) next-token batches +loader = turboloader.TokenDataLoader('train.bin', seq_len=1024, batch_size=8, + dtype='uint16', shuffle=True) +for x, y in loader: # x, y: (8, 1024) int64; y is x shifted by one + loss = model(x, y) + +# Generic arrays/memmaps (embeddings, tabular features, labels, pre-tokenized data) +loader = turboloader.ArrayDataLoader(features, labels, batch_size=256, shuffle=True) +for xb, yb in loader: + ... +``` + +`TokenDataLoader` uses a vectorized fancy-index gather over a `np.memmap` (so multi-GB +corpora stream without loading into RAM) and benchmarks ~1.9× the standard nanoGPT +`get_batch` idiom. The image pipeline (decode/transform/TBL) remains C++; these +modality loaders are NumPy-based and modality-agnostic. + +All three modalities are also reachable from the **single `DataLoader` entry point**: + +```python +turboloader.DataLoader('train.bin', modality='tokens', seq_len=1024, batch_size=8) +turboloader.DataLoader(arrays=[feats, labels], data_path=None, modality='array', batch_size=256) +turboloader.DataLoader('data.tar', image_size=160, output_format='pytorch') # modality='image' (default) +``` + +### Wrap *any* Python dataset (`MapDataLoader`) + +When your data doesn't fit the native paths, `MapDataLoader` batches **any** map-style +dataset — anything with `__len__` and `__getitem__(i)`, i.e. exactly the +`torch.utils.data.Dataset` protocol — so your loading/decoding/business logic can be +arbitrary Python: + +```python +class MyDataset: + def __len__(self): return len(self.records) + def __getitem__(self, i): + x = decode_however_you_like(self.records[i]) # any Python logic + return x, self.labels[i] # (features, label) + +# directly, or via the unified entry point with dataset=... +for xb, yb in turboloader.MapDataLoader(MyDataset(), batch_size=64, shuffle=True, num_workers=8): + train_step(xb, yb) +``` + +It parallelizes `__getitem__` on a bounded thread pool with read-ahead and collates +(tuples/dicts/arrays, or a custom `collate_fn`). **Honest tradeoff:** because the +per-sample work runs in Python, this path is roughly PyTorch-`DataLoader` speed (and +GIL-bound for pure-Python CPU work — threads help most when `__getitem__` releases the +GIL, e.g. NumPy/PIL/file/network I/O). It's about *flexibility*, not the C++ fast path — +use the image/token/array loaders above when you want maximum throughput. diff --git a/docs/transforms.md b/docs/transforms.md new file mode 100644 index 0000000..af21e9b --- /dev/null +++ b/docs/transforms.md @@ -0,0 +1,34 @@ +# Transform Library + + +TurboLoader includes 24 transforms (19 per-image SIMD transforms + 5 batch +augmentations). The authoritative list is `turboloader.list_transforms()`. + +### Core Transforms +- **Resize** - Bilinear/Bicubic/Lanczos interpolation +- **Normalize** - Mean/std normalization with SIMD +- **CenterCrop** - Center region extraction +- **RandomCrop** - Random crop with padding + +### Augmentation Transforms +- **RandomHorizontalFlip** - SIMD horizontal flip +- **RandomVerticalFlip** - SIMD vertical flip +- **ColorJitter** - Brightness/contrast/saturation/hue +- **RandomRotation** - Arbitrary angle rotation +- **GaussianBlur** - Separable convolution +- **RandomErasing** - Cutout augmentation +- **Pad** - Border padding (CONSTANT/EDGE/REFLECT) + +### Advanced Transforms +- **RandomPosterize** - Bit-depth reduction +- **RandomSolarize** - Threshold inversion +- **RandomPerspective** - Perspective warp +- **AutoAugment** - Learned policies (ImageNet/CIFAR10/SVHN) + +### Batch Augmentations +- **MixUp**, **CutMix**, **Mosaic**, **RandAugment**, **GridMask** + +### Tensor Conversion +- **ToTensor** - PyTorch CHW or TensorFlow HWC format + +Full per-transform API and examples: [docs/api/transforms.md](api/transforms.md). diff --git a/examples/quickstart_demo.py b/examples/quickstart_demo.py new file mode 100644 index 0000000..c3b25f3 --- /dev/null +++ b/examples/quickstart_demo.py @@ -0,0 +1,27 @@ +import time +import turboloader as tl + +print("TurboLoader — 9,469 real ImageNet JPEGs (Imagenette-160)\n") + +loader = tl.DataLoader( + "imagenette_train.tar" # TAR archive of JPEGs, + batch_size=256, + image_size=160, + output_format="pytorch", # (N, 3, H, W) float32, normalized + transform=tl.ImageNetNormalize(), + shuffle=True, + train_aug=True, # fused RandomResizedCrop + flip in C++ +) + +n = sum(b.shape[0] for b, _ in loader) # warmup epoch: OS page cache +print(f" warmup: {n:,} images (filling the OS page cache)") + +for epoch in range(5): + loader.set_epoch(epoch) + n, t0 = 0, time.perf_counter() + for images, meta in loader: + n += images.shape[0] + dt = time.perf_counter() - t0 + print(f" epoch {epoch + 1}: {n:,} images in {dt:.2f}s -> {n / dt:>7,.0f} img/s") + +print("\none pip install. zero FFmpeg/DALI/beton. Apple Silicon + Linux + CUDA.")