Move VLM training configs into examples/vlm/

CHANGELOG.md

@@ -106,6 +106,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 
 - **BREAKING: the COCO-Karpathy prep helper moved to `examples/vlm/data/prep_vlm_coco.py`** (was `scripts/prep_vlm_coco.py`). It is VLM-experiment data prep, not general tooling, so it belongs with the example rather than in `scripts/`. No compat shim — invoke it at the new path. `--caption-json` and `--image-root` are now required: their old defaults were cluster-specific absolute paths, which a shipped example must not carry.
+- **BREAKING: VLM training configs moved to `examples/vlm/`.** The 13 `configs/train/vlm_*.toml` presets now live in `examples/vlm/configs/` (same filenames), alongside the example's own `train.py` entry point, `README.md`, and tests. No shim or symlink: update any path that named `configs/train/vlm_*.toml`. `[video].data_root` in `vlm_video_webvid.toml` is now a placeholder rather than a site-specific path.
 - **Training entry point is now a library.** `scripts/train.py:main()` (~1000 lines) is decomposed into `kempnerforge/training/` modules: `runtime.py` (`RuntimeContext`, `PipelineBundle`, `setup_distributed`), `data_pipeline.py` (`DataPipeline`, `PhaseState`, data/eval/phase builders), `loop.py` (`BatchStream`, `StepResult`, `TrainingSession`, `run_training_loop`, and the `text_step` / `vlm_step` / `pipeline_step` bodies picked by `select_step_fn`), and `entry.py` (`run_training` plus the model/checkpoint/resume builders). `scripts/train.py` is now a thin CLI wrapper — same CLI, same log lines, same metric keys, same checkpoint format. Behavior-preserving; the loop is unit-testable with a fake `CheckpointManager` (`tests/unit/test_train_entry.py`). `run_training(config, *, step_fn=None, hooks=None)` lets an experiment own the step body or register hooks without copying the build phases, and both `run_training` and `run_training_loop` tear down in `finally` now that they are library calls rather than a script about to exit.
 - **BREAKING — VLM evaluation moved out of core into `examples/vlm/eval/`.** `kempnerforge/eval/` and `scripts/vlm_eval_harness.py` are gone; the lmms-eval adapter, CLI, tests and how-to now live in the example, so core carries no lmms-eval-facing code. The `[project.entry-points."lmms_eval.models"]` declaration is removed with no shim — `lmms_eval --model kempnerforge_vlm` no longer resolves, and the harness builds the adapter itself. See `examples/vlm/eval/README.md` for usage.
 - New `ci-example.yml` workflow runs the hermetic example test tiers, which sit outside the root `testpaths` and so were collected by no CI step. One job per example behind a per-example path filter, so touching one example does not run another's tests; one pytest session per tier. The VLM-eval integration tier stays manual (it needs the undeclared `lmms-eval`).

README.md

@@ -137,27 +137,29 @@ KempnerForge supports VLM training — images **or video** (a clip is an ordered
 - **Mixture-of-Transformers** (`arch = "mot"`, Liang et al. 2024 Algorithm 1): every layer carries per-modality Q/K/V/O projections plus a per-modality FFN; a single global self-attention mixes all modality streams. Image tokens prepend the text sequence (same residual layout as Joint-Decoder); per-modality residual projections are zero-initialized so a fresh MoT block is identity at construction. A warm-start helper (`mot_warm_start_from_text_stack`) translates a JD or text-only checkpoint into per-modality copies — toggle via `[model.vlm].mot_warm_start_from_text` + `mot_warm_start_path`.
 - **Mixture of Modality-Aware Experts** (`arch = "moma"`, Lin et al. 2024 arXiv:2407.21770): one shared set of Q/K/V/O projections feeding a global self-attention, plus per-modality MoE FFN groups (paper's optimal default 4 image + 4 text experts per layer). Tokens route deterministically to their modality group (level-1, reusing the same `modality_ids` mechanism MoT uses) and then through a learned expert-choice + Sigmoid router within the group (level-2, with Gumbel-Sigmoid noise during training, paper Eq. 5). Image tokens prepend the text sequence (same residual layout as JD/MoT). v1 supports training only — expert-choice routing is non-causal, so autoregressive generation requires auxiliary routers (paper §2.4) which are deferred to a follow-up.
 
-**Video** works across all four archs with no arch-specific changes: a clip is decoded into frames (by a registered `sampling_policy` — default `uniform`: by timestamp at a target fps, first and last frame kept), each frame is encoded and pooled by the connector, and the `F × tokens_per_frame` visual tokens enter the backbone exactly like image tokens. The data side is **pluggable** — `[video].dataset_type` selects a registered dataset builder (`webvid` ships; `dataset_name` picks the corpus within a WebVid-style layout) and `sampling_policy` selects the frame-sampling policy, so new dataset styles / policies are additive registrations. Configure the `[video]` section (`data_root`, `dataset_type`, `dataset_name`, `sampling_policy`, `fps`, `max_frames`, `frame_size`); see `configs/train/vlm_video_webvid.toml`. Video decoding uses PyAV, an optional dependency — install it with `uv sync --group video`.
+**Video** works across all four archs with no arch-specific changes: a clip is decoded into frames (by a registered `sampling_policy` — default `uniform`: by timestamp at a target fps, first and last frame kept), each frame is encoded and pooled by the connector, and the `F × tokens_per_frame` visual tokens enter the backbone exactly like image tokens. The data side is **pluggable** — `[video].dataset_type` selects a registered dataset builder (`webvid` ships; `dataset_name` picks the corpus within a WebVid-style layout) and `sampling_policy` selects the frame-sampling policy, so new dataset styles / policies are additive registrations. Configure the `[video]` section (`data_root`, `dataset_type`, `dataset_name`, `sampling_policy`, `fps`, `max_frames`, `frame_size`); see `examples/vlm/configs/vlm_video_webvid.toml`. Video decoding uses PyAV, an optional dependency — install it with `uv sync --group video`.
+
+The example ships in [`examples/vlm/`](examples/vlm/) — its configs and its own training entry point:
 
 ```bash
 # 1-GPU smoke (random encoder, Joint-Decoder)
-uv run python scripts/train.py configs/train/vlm_debug.toml \
+uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug.toml \
     --data.hf_dataset_name=<vlm-dataset> --data.tokenizer_path=gpt2
 
 # 4-GPU SigLIP2 + 7B Joint-Decoder
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_siglip2.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_siglip2.toml
 
 # 4-GPU 7B Cross-Attention (8 CA blocks at cadence 4)
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_cross_attn.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_cross_attn.toml
 
 # 4-GPU 7B Mixture-of-Transformers
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_mot.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_mot.toml
 
 # 4-GPU 7B Mixture of Modality-Aware Experts (4 text + 4 image experts per layer)
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_moma.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_moma.toml
 
 # 4-GPU video training on WebVid (Joint-Decoder; flip [vlm].arch for cross_attention / mot / moma)
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_video_webvid.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_video_webvid.toml
 ```
 
 Configs set `[model.vlm]` with `arch`, the encoder registry key, the number of image tokens, and a freeze list (`FreezeSpec`). For Cross-Attention, set `cross_attention_every_n_layers` and optionally `cross_attention_n_kv_heads` (0 → MHA; positive → GQA on the cross path). For MoT, set `mot_modalities` (must include both `"image"` and `"text"`); `mot_image_n_heads` / `mot_image_n_kv_heads` are forward-looking per-modality head fields (v1 enforces equality with the text-side counts since the operator runs a single global SDPA). For MoMa, set `moma_experts_per_modality = {image = N, text = M}` as a nested TOML table (the paper's optimal balanced default is `4t4i`; unbalanced allocations like `{image = 1, text = 7}` match the paper's `moe_7t1i` ablation), and optionally `moma_capacity_factor` (defaults to `1/|E^M|` per modality — the paper's perfect-balance setting) and `moma_gumbel_noise` (`true` by default for paper-faithful EC routing). `model.num_experts` must be `0` when `arch = "moma"`; the per-modality counts supersede it, and JobConfig.validate rejects the combination. The vision encoder stays in its HF-loaded dtype; the transformer, adapter, and CA / MoT / MoMa blocks are cast to `param_dtype`. Pipeline Parallel + VLM is not supported on this branch (raises at startup); MoMa + Expert Parallelism is also rejected in v1. Video is supported across all four archs via the `[video]` section (a clip is decoded into frames, pooled by the connector, and fed like image tokens); multi-image inputs and video *grounding* (point/track outputs with per-frame timestamps) are reserved for follow-up work.

docs/how-to/train-on-video.md

@@ -44,7 +44,7 @@ build- and config-time checks enforce this and fail before any GPU work.
 
 A video run adds a `[video]` section (sibling of `[vision_encoder]` /
 `[adapter]` / `[vlm]`) and a token-reducing connector. See
-`configs/train/vlm_video_webvid.toml` for a complete example; the key parts:
+`examples/vlm/configs/vlm_video_webvid.toml` for a complete example; the key parts:
 
 ```toml
 [adapter]
@@ -55,7 +55,7 @@ pool_window = 2           # 14×14 grid -> 7×7 = 49 tokens/frame
 arch = "joint_decoder"    # also: cross_attention | mot | moma
 
 [video]
-data_root = "/path/to/webvid-10m"
+data_root = "path-to-webvid-10m"
 dataset_type = "webvid"      # registry key; add styles via @registry.register_video_dataset
 dataset_name = "webvid-10M"  # corpus dir under raw/<dataset_name>/data (WebVid style)
 sampling_policy = "uniform"  # registry key; the frame-sampling policy
@@ -85,10 +85,10 @@ requires it.
 
 ```bash
 # 4-GPU video training (Joint-Decoder)
-uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_video_webvid.toml
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_video_webvid.toml
 
 # Quick smoke: no SigLIP download, a few clips, few steps
-uv run torchrun --nproc_per_node=2 scripts/train.py configs/train/vlm_video_webvid.toml \
+uv run torchrun --nproc_per_node=2 examples/vlm/train.py examples/vlm/configs/vlm_video_webvid.toml \
     --vision_encoder.type=random --vision_encoder.num_tokens=196 \
     --vision_encoder.feature_dim=768 --video.max_samples=256 --train.max_steps=20
 ```

examples/vlm/README.md

@@ -0,0 +1,61 @@
+# VLM example
+
+Vision-language training — images **or** video — on the core `Transformer`. A
+frozen HF vision encoder produces visual tokens, a connector projects (and
+optionally pools) them, and an arch-specific path feeds the backbone. Everything
+here is configuration plus an entry point; `kempnerforge/` never imports it, so
+this directory can be deleted without touching the core.
+
+## Configs
+
+`vlm_debug*` are 1-GPU smoke presets — tiny backbone, `random` encoder, so they
+run on a fresh clone with no download. The `vlm_7b*` presets are 4-8 GPU
+starting points.
+
+| Config | Arch | Encoder | For |
+| --- | --- | --- | --- |
+| `vlm_debug.toml` | joint_decoder | random | 1-GPU smoke |
+| `vlm_debug_mot.toml` | mot | random | 1-GPU smoke |
+| `vlm_debug_moma.toml` | moma | random | 1-GPU smoke |
+| `vlm_debug_moe.toml` | cross_attention | random | 1-GPU smoke, MoE FFN |
+| `vlm_7b.toml` | joint_decoder | random | 7B, AC off (VRAM stress) |
+| `vlm_7b_ac.toml` | joint_decoder | random | 7B, AC full + longer seq |
+| `vlm_7b_mot.toml` | mot | random | 7B |
+| `vlm_7b_moma.toml` | moma | random | 7B |
+| `vlm_7b_cross_attn.toml` | cross_attention | random | 7B |
+| `vlm_7b_freeze_schedule.toml` | cross_attention | random | multi-stage `FreezeStage` schedule |
+| `vlm_7b_siglip2.toml` | joint_decoder | siglip2 | real-run starting point |
+| `vlm_7b_siglip2_cross_attn.toml` | cross_attention | siglip2 | real-run starting point |
+| `vlm_video_webvid.toml` | joint_decoder | siglip2 | video (WebVid-10M) |
+
+Paths in these configs are placeholders (`data_root = "path-to-webvid-10m"`) —
+point them at your own data and output directories, or override on the CLI.
+
+## Run it
+
+```bash
+# 1-GPU smoke
+uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug.toml
+
+# 4 GPUs, single node
+uv run torchrun --nproc_per_node=4 examples/vlm/train.py \
+    examples/vlm/configs/vlm_7b_siglip2.toml
+
+# Override anything on the CLI
+uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug.toml \
+    --train.max_steps=20 --checkpoint.dir=/your/run/dir
+```
+
+Video needs PyAV: `uv sync --group video`.
+
+Tests: `uv run pytest examples/vlm/tests/ -v` (they are outside the core
+`testpaths`, so run them by path).
+
+## Data prep
+
+`data/prep_vlm_coco.py` writes a COCO-Karpathy `save_to_disk` directory for
+`data.hf_dataset_name` to point at.
+
+## Evaluation
+
+Benchmark evaluation of the resulting checkpoints lives in `eval/`.

configs/train/vlm_7b.toml → examples/vlm/configs/vlm_7b.toml

@@ -16,7 +16,7 @@
 #   throughput: ~41k tok/s, ~46% MFU
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_7b_ac.toml → examples/vlm/configs/vlm_7b_ac.toml

@@ -16,8 +16,8 @@
 #   throughput: ~31k tok/s, ~35% MFU
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py \
-#       configs/train/vlm_7b_ac.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py \
+#       examples/vlm/configs/vlm_7b_ac.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_7b_cross_attn.toml → examples/vlm/configs/vlm_7b_cross_attn.toml

@@ -26,8 +26,8 @@
 # of compute per step.
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py \
-#       configs/train/vlm_7b_cross_attn.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py \
+#       examples/vlm/configs/vlm_7b_cross_attn.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_7b_freeze_schedule.toml → examples/vlm/configs/vlm_7b_freeze_schedule.toml

@@ -19,8 +19,8 @@
 # transition spec (adapter frozen).
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py \
-#       configs/train/vlm_7b_freeze_schedule.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py \
+#       examples/vlm/configs/vlm_7b_freeze_schedule.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_7b_moma.toml → examples/vlm/configs/vlm_7b_moma.toml

@@ -30,7 +30,7 @@
 # must cover both modalities.
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_moma.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_moma.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training override:

configs/train/vlm_7b_mot.toml → examples/vlm/configs/vlm_7b_mot.toml

@@ -29,7 +29,7 @@
 #   loss: 11.6 -> 6.4 over 50 steps (clear convergence on real data)
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_mot.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_mot.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_7b_siglip2.toml → examples/vlm/configs/vlm_7b_siglip2.toml

@@ -23,7 +23,7 @@
 #   Leaves ~90 GB / H100 for activations and grad accumulation headroom.
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_7b_siglip2.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_7b_siglip2.toml
 #
 # Data:
 #   Default hf_dataset_name is a 30-sample COCO val substitute

configs/train/vlm_7b_siglip2_cross_attn.toml → examples/vlm/configs/vlm_7b_siglip2_cross_attn.toml

@@ -24,8 +24,8 @@
 # carries text only.
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=4 scripts/train.py \
-#       configs/train/vlm_7b_siglip2_cross_attn.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py \
+#       examples/vlm/configs/vlm_7b_siglip2_cross_attn.toml
 #
 # Default points at a 30-sample COCO val substitute (sayakpaul/coco-30-val-2014)
 # so a fresh clone runs without external setup. For real training, override

configs/train/vlm_debug.toml → examples/vlm/configs/vlm_debug.toml

@@ -5,14 +5,14 @@
 # Runs end-to-end in <1 minute on 1 GPU. Uses RandomVisionEncoder so no
 # HF download is needed; pair with any HF image-text dataset (the default
 # below is a placeholder). For a real vision encoder, see
-# configs/train/vlm_7b_siglip2.toml.
+# examples/vlm/configs/vlm_7b_siglip2.toml.
 #
 # max_seq_len allocation (JD/MoT): residual_image_tokens + max_text_len.
 # Image tokens prepend the text sequence in the residual stream, so the
 # budget must cover both modalities.
 #
 # Usage:
-#   uv run python scripts/train.py configs/train/vlm_debug.toml \
+#   uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug.toml \
 #       --data.hf_dataset_name=... --data.tokenizer_path=gpt2
 
 [model]

configs/train/vlm_debug_moe.toml → examples/vlm/configs/vlm_debug_moe.toml

@@ -19,8 +19,8 @@
 #     through the VLMWrapper without exposing them as wrapper attrs.
 #
 # Usage:
-#   uv run torchrun --nproc_per_node=2 scripts/train.py \
-#       configs/train/vlm_debug_moe.toml \
+#   uv run torchrun --nproc_per_node=2 examples/vlm/train.py \
+#       examples/vlm/configs/vlm_debug_moe.toml \
 #       --data.hf_dataset_name=<dataset> --data.tokenizer_path=gpt2
 
 [model]

configs/train/vlm_debug_moma.toml → examples/vlm/configs/vlm_debug_moma.toml

@@ -17,7 +17,7 @@
 # (paper §2.4), deferred to a follow-up.
 #
 # Usage:
-#   uv run python scripts/train.py configs/train/vlm_debug_moma.toml \
+#   uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug_moma.toml \
 #       --data.hf_dataset_name=... --data.tokenizer_path=gpt2
 
 [model]

configs/train/vlm_debug_mot.toml → examples/vlm/configs/vlm_debug_mot.toml

@@ -11,7 +11,7 @@
 # budget must cover both modalities.
 #
 # Usage:
-#   uv run python scripts/train.py configs/train/vlm_debug_mot.toml \
+#   uv run python examples/vlm/train.py examples/vlm/configs/vlm_debug_mot.toml \
 #       --data.hf_dataset_name=... --data.tokenizer_path=gpt2
 
 [model]

configs/train/vlm_video_webvid.toml → examples/vlm/configs/vlm_video_webvid.toml

@@ -12,7 +12,7 @@
 #   8 frames -> 8*49 = 392 visual + 64 text = 456 <= 576.
 #
 # Launch (single node, 4 GPUs):
-#   uv run torchrun --nproc_per_node=4 scripts/train.py configs/train/vlm_video_webvid.toml
+#   uv run torchrun --nproc_per_node=4 examples/vlm/train.py examples/vlm/configs/vlm_video_webvid.toml
 #
 # Quick smoke (no SigLIP download, a few clips; pair with a small step count):
 #   ... --vision_encoder.type=random --vision_encoder.num_tokens=196 \
@@ -44,7 +44,7 @@ max_text_len = 64
 freeze = [{module = "vision_encoder", frozen = true}]
 
 [video]
-data_root = "/n/holylfs06/LABS/kempner_shared/Everyone/testbed/video/webvid-10m"
+data_root = "path-to-webvid-10m"
 split = "train"
 fps = 2.0
 max_frames = 8