docs: README overhaul — GIF, mermaid diagrams, focused split

docs/assets/demo.tape

@@ -0,0 +1,25 @@
+Output demo.gif
+Set FontSize 16
+Set Width 920
+Set Height 560
+Set Theme "Catppuccin Mocha"
+Set TypingSpeed 35ms
+Set Framerate 24
+Set Shell bash
+
+Hide
+Type "python3 -m venv gifenv && source gifenv/bin/activate && export PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_NO_PYTHON_VERSION_WARNING=1"
+Enter
+Sleep 8s
+Type "clear"
+Enter
+Sleep 2s
+Show
+
+Type "pip install turboloader"
+Enter
+Sleep 8s
+Type "python demo.py"
+Enter
+Sleep 7s
+Sleep 3s

docs/tbl_v2_format.md

@@ -0,0 +1,33 @@
+# TBL v2 Binary Format
+
+
+TurboLoader includes a custom binary format optimized for ML workloads:
+
+### Features
+- LZ4 compression for reduced storage
+- Memory-mapped access for fast loading
+- O(1) random access via indexed structure
+- Data integrity validation with CRC checksums
+- Cached image dimensions for filtered loading
+
+### Convert TAR to TBL
+
+```python
+import tarfile
+import turboloader
+
+writer = turboloader.TblWriterV2("/data/imagenet.tbl", enable_compression=True)
+
+# The TAR archive is read with Python's stdlib (TurboLoader does not expose a
+# standalone Python TarReader; the DataLoader reads TAR directly for training).
+with tarfile.open("/data/imagenet.tar") as tar:
+    for member in tar.getmembers():
+        if not member.name.lower().endswith((".jpg", ".jpeg")):
+            continue
+        data = tar.extractfile(member).read()
+        writer.add_sample(data=data, format=turboloader.SampleFormat.JPEG)
+
+writer.finalize()
+```
+
+> For bulk conversion there is also a C++ CLI tool, `tools/tar_to_tbl_v2.cpp`.

docs/tokens_arrays.md

@@ -0,0 +1,57 @@
+# Beyond Images: Tokens, Arrays & Any Python Dataset
+
+
+TurboLoader also ships loaders for non-image modalities with the same ergonomics
+(re-iterable, `shuffle`, `set_epoch`, batched arrays):
+
+```python
+# LLM pretraining: memory-mapped token stream -> (B, seq_len) next-token batches
+loader = turboloader.TokenDataLoader('train.bin', seq_len=1024, batch_size=8,
+                                     dtype='uint16', shuffle=True)
+for x, y in loader:          # x, y: (8, 1024) int64; y is x shifted by one
+    loss = model(x, y)
+
+# Generic arrays/memmaps (embeddings, tabular features, labels, pre-tokenized data)
+loader = turboloader.ArrayDataLoader(features, labels, batch_size=256, shuffle=True)
+for xb, yb in loader:
+    ...
+```
+
+`TokenDataLoader` uses a vectorized fancy-index gather over a `np.memmap` (so multi-GB
+corpora stream without loading into RAM) and benchmarks ~1.9× the standard nanoGPT
+`get_batch` idiom. The image pipeline (decode/transform/TBL) remains C++; these
+modality loaders are NumPy-based and modality-agnostic.
+
+All three modalities are also reachable from the **single `DataLoader` entry point**:
+
+```python
+turboloader.DataLoader('train.bin', modality='tokens', seq_len=1024, batch_size=8)
+turboloader.DataLoader(arrays=[feats, labels], data_path=None, modality='array', batch_size=256)
+turboloader.DataLoader('data.tar', image_size=160, output_format='pytorch')   # modality='image' (default)
+```
+
+### Wrap *any* Python dataset (`MapDataLoader`)
+
+When your data doesn't fit the native paths, `MapDataLoader` batches **any** map-style
+dataset — anything with `__len__` and `__getitem__(i)`, i.e. exactly the
+`torch.utils.data.Dataset` protocol — so your loading/decoding/business logic can be
+arbitrary Python:
+
+```python
+class MyDataset:
+    def __len__(self): return len(self.records)
+    def __getitem__(self, i):
+        x = decode_however_you_like(self.records[i])   # any Python logic
+        return x, self.labels[i]                       # (features, label)
+
+# directly, or via the unified entry point with dataset=...
+for xb, yb in turboloader.MapDataLoader(MyDataset(), batch_size=64, shuffle=True, num_workers=8):
+    train_step(xb, yb)
+```
+
+It parallelizes `__getitem__` on a bounded thread pool with read-ahead and collates
+(tuples/dicts/arrays, or a custom `collate_fn`). **Honest tradeoff:** because the
+per-sample work runs in Python, this path is roughly PyTorch-`DataLoader` speed (and
+GIL-bound for pure-Python CPU work — threads help most when `__getitem__` releases the
+GIL, e.g. NumPy/PIL/file/network I/O). It's about *flexibility*, not the C++ fast path —
+use the image/token/array loaders above when you want maximum throughput.

docs/transforms.md

@@ -0,0 +1,34 @@
+# Transform Library
+
+
+TurboLoader includes 24 transforms (19 per-image SIMD transforms + 5 batch
+augmentations). The authoritative list is `turboloader.list_transforms()`.
+
+### Core Transforms
+- **Resize** - Bilinear/Bicubic/Lanczos interpolation
+- **Normalize** - Mean/std normalization with SIMD
+- **CenterCrop** - Center region extraction
+- **RandomCrop** - Random crop with padding
+
+### Augmentation Transforms
+- **RandomHorizontalFlip** - SIMD horizontal flip
+- **RandomVerticalFlip** - SIMD vertical flip
+- **ColorJitter** - Brightness/contrast/saturation/hue
+- **RandomRotation** - Arbitrary angle rotation
+- **GaussianBlur** - Separable convolution
+- **RandomErasing** - Cutout augmentation
+- **Pad** - Border padding (CONSTANT/EDGE/REFLECT)
+
+### Advanced Transforms
+- **RandomPosterize** - Bit-depth reduction
+- **RandomSolarize** - Threshold inversion
+- **RandomPerspective** - Perspective warp
+- **AutoAugment** - Learned policies (ImageNet/CIFAR10/SVHN)
+
+### Batch Augmentations
+- **MixUp**, **CutMix**, **Mosaic**, **RandAugment**, **GridMask**
+
+### Tensor Conversion
+- **ToTensor** - PyTorch CHW or TensorFlow HWC format
+
+Full per-transform API and examples: [docs/api/transforms.md](api/transforms.md).

examples/quickstart_demo.py

@@ -0,0 +1,27 @@
+import time
+import turboloader as tl
+
+print("TurboLoader — 9,469 real ImageNet JPEGs (Imagenette-160)\n")
+
+loader = tl.DataLoader(
+    "imagenette_train.tar"  # TAR archive of JPEGs,
+    batch_size=256,
+    image_size=160,
+    output_format="pytorch",        # (N, 3, H, W) float32, normalized
+    transform=tl.ImageNetNormalize(),
+    shuffle=True,
+    train_aug=True,                 # fused RandomResizedCrop + flip in C++
+)
+
+n = sum(b.shape[0] for b, _ in loader)   # warmup epoch: OS page cache
+print(f"  warmup:   {n:,} images (filling the OS page cache)")
+
+for epoch in range(5):
+    loader.set_epoch(epoch)
+    n, t0 = 0, time.perf_counter()
+    for images, meta in loader:
+        n += images.shape[0]
+    dt = time.perf_counter() - t0
+    print(f"  epoch {epoch + 1}:  {n:,} images in {dt:.2f}s   ->  {n / dt:>7,.0f} img/s")
+
+print("\none pip install. zero FFmpeg/DALI/beton. Apple Silicon + Linux + CUDA.")