E2E speed sprint: video training loader, token fast path, honest e2e reruns, CUDA self-hosted CI - #16
Merged
Merged
Conversation
added 8 commits
August 1, 2026 11:37
…oops Batch k+1's host->device copy runs on a side CUDA stream while the model computes on batch k (apex-style; wait_stream BEFORE staging the next copy — staging first would make compute wait on the new copy and re-serialize the transfer). Pairs with pin_memory=True; one-batch-ahead staging stays within the pinned ring's reuse window. e2e benchmark gains a tl+prefetcher row.
One np.take of the seq_len+1 window feeds both x and the shifted target (the numpy path gathers each separately — twice the memory traffic), cast straight into preallocated pinned int64 tensors. device='cuda' additionally stages H2D on a side stream with per-slot CUDA events guarding buffer reuse, yielding device tensors with no lifetime rules for the caller. GPT example gains a turboloader-dev pipeline; gather equivalence tested on CPU, ring reuse/device parity/resume tested under CUDA.
…tring 3090 e2e measurement (160px AND 224px) showed CudaPrefetcher NEUTRAL — decode delivery, not H2D, binds those epochs; docstring now says so instead of the projected claim. New loader-only benchmark isolates the token fast path at GPT-2 shape against the exact nanoGPT get_batch idiom (incl. pin+non_blocking).
…y (CUDA) ImageFolder-style discovery, N PyAV decoder threads (per-thread container cache, pts-derived seek with a counting fallback for untagged streams), frame-weighted uniform (video, start) sampling reproducible per (seed, epoch), aug decisions drawn in task order + results resequenced so batches are deterministic under any worker scheduling, stop-aware queue lifecycle (no stranded threads on early exit), flat pinned staging (a 2-D column slice would silently unpin via .contiguous()), one fused cuda_video_clip_yuv420 launch per clip. Plus the first e2e VIDEO training benchmark: r3d_18 on real footage (BBB vs Jellyfish segments), TL vs PyTorch DataLoader + PyAV recipe.
…ed workflow cuda-tests.yml runs the CUDA suite on the registered 3090-wsl runner, workflow_dispatch ONLY (self-hosted must never execute fork-PR code).
…en by comment) E2E_TRAINING_RESULTS gains 160/224px reruns (gap-to-floor framing), the CudaPrefetcher neutral finding, the first video-training e2e section, and the GPT/token numbers. README headline + which-loader updated (VideoDatasetLoader, TokenDataLoader device=), tokens_arrays documents the fast path, CHANGELOG [Unreleased] written. quickstart_demo.py had the DataLoader comma inside an inline comment — a syntax error examples/ lint never covered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Addresses the gaps/weaknesses from the system-strength assessment and pushes end-to-end training speed on all three modalities, with every number measured on real hardware (RTX 3090 + M4 Max) and reported honestly.
Image (ResNet-18 e2e)
CudaPrefetcher(apex-style overlapped H2D). Honest measured result: neutral in these e2e runs — decode delivery, not H2D, binds them; docstring + docs say exactly that.LLM tokens
TokenDataLoaderfast path:pin_memory=Truepinned ring +device='cuda'(side-stream H2D, CUDA-event-guarded reuse, ready GPU tensors). ONE seq_len+1 gather feeds x AND y.Video
VideoDatasetLoader: labeled clip batches fromroot/class_x/*.mp4— threaded PyAV decode, pts-derived seek, frame-weighted (seed, epoch)-reproducible sampling, ONE fused clip kernel launch per clip, stop-aware lifecycle.Platform/CI
cuda-tests.yml(workflow_dispatch ONLY — no fork-PR exposure) builds in-place and runs the CUDA suite.examples/quickstart_demo.pysyntax error (comma swallowed by inline comment).Tests