While extracting embedding matrices across the full Pythia checkpoint grid (HTTP range reads only — no full downloads), we found that the pythia-2.8b training history is largely not step-specific.
File-level evidence (LFS sha256 of model.safetensors, via the HF API)
main = ab496f1c3fd79e3c749a9d5414136a2c8e4224f94eecb261970315cdb0f813fe- 76 step branches are bit-identical to
main: steps 0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512; every 1000 from 1000 through 53000; plus 55000, 60000, 65000, 70000, 80000, 90000, 100000, 103000, 110000, 120000, 130000, 140000.
- A second blob (
462f2b960062159c…) is served by 10 branches: 54000, 56000, 57000, 58000, 59000, 61000, 62000, 63000, 64000, and 143000.
- The remaining 68 branches (non-round steps 66000–142000) each serve a unique file.
Full branch → sha256 table attached (upstream_2p8b_branch_hashes.csv).
Tensor-level evidence (embedding byte ranges read from every branch)
- All 86 branches in the two duplicated groups serve byte-identical
embed_in and embed_out tensors (re-fetching step0 and step143000 returns identical bytes; e.g. row-0 norm 0.889188, abs-sum 2012294.875 on both). The two blobs differ somewhere outside the embedding ranges.
- The 68 unique files carry genuinely distinct, evolving embeddings consistent with late-training checkpoints of the same run (cosine ≥ 0.86 to the shared snapshot).
The shared snapshot is end-of-training content, not an init
Its never-used vocabulary rows (padding ids ≥ 50277) have median L2 norm 0.19436, i.e. ~0.308 of the init norm (~0.632). Under decoupled AdamW with the published 2.8b schedule (peak LR 1.6e-4, cosine to 1.6e-5, weight decay 0.1), unused rows decay as exp(−λτ), predicting an end-of-training ratio of 0.284 — an ~8% match. A genuine step-0 branch would sit at ratio 1.0 and genuine early steps at ≥0.97. So the branches labelled step0…step53000 (and all round steps) serve final-era content; step143000's embeddings equal main's and may themselves be fine.
Impact
Any research using pythia-2.8b intermediate checkpoints (learning dynamics, interpretability-over-training) is silently analyzing one end-of-training snapshot for the init, the entire early grid, and every round-numbered step. Genuine history appears to survive only on the 68 non-round branches 66000–142000.
Reproduction (no downloads required)
GET https://huggingface.co/api/models/EleutherAI/pythia-2.8b/tree/<branch> and compare lfs.oid of model.safetensors across branches.- For tensor identity: HTTP range-read
gpt_neox.embed_in.weight (offsets from the safetensors header) from any two same-hash branches and compare bytes.
Happy to share the extraction code and the full per-branch statistics.
upstream_2p8b_branch_hashes.csv
While extracting embedding matrices across the full Pythia checkpoint grid (HTTP range reads only — no full downloads), we found that the pythia-2.8b training history is largely not step-specific.
File-level evidence (LFS sha256 of
model.safetensors, via the HF API)main=ab496f1c3fd79e3c749a9d5414136a2c8e4224f94eecb261970315cdb0f813femain: steps 0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512; every 1000 from 1000 through 53000; plus 55000, 60000, 65000, 70000, 80000, 90000, 100000, 103000, 110000, 120000, 130000, 140000.462f2b960062159c…) is served by 10 branches: 54000, 56000, 57000, 58000, 59000, 61000, 62000, 63000, 64000, and 143000.Full branch → sha256 table attached (
upstream_2p8b_branch_hashes.csv).Tensor-level evidence (embedding byte ranges read from every branch)
embed_inandembed_outtensors (re-fetchingstep0andstep143000returns identical bytes; e.g. row-0 norm 0.889188, abs-sum 2012294.875 on both). The two blobs differ somewhere outside the embedding ranges.The shared snapshot is end-of-training content, not an init
Its never-used vocabulary rows (padding ids ≥ 50277) have median L2 norm 0.19436, i.e. ~0.308 of the init norm (~0.632). Under decoupled AdamW with the published 2.8b schedule (peak LR 1.6e-4, cosine to 1.6e-5, weight decay 0.1), unused rows decay as exp(−λτ), predicting an end-of-training ratio of 0.284 — an ~8% match. A genuine step-0 branch would sit at ratio 1.0 and genuine early steps at ≥0.97. So the branches labelled step0…step53000 (and all round steps) serve final-era content;
step143000's embeddings equalmain's and may themselves be fine.Impact
Any research using pythia-2.8b intermediate checkpoints (learning dynamics, interpretability-over-training) is silently analyzing one end-of-training snapshot for the init, the entire early grid, and every round-numbered step. Genuine history appears to survive only on the 68 non-round branches 66000–142000.
Reproduction (no downloads required)
GET https://huggingface.co/api/models/EleutherAI/pythia-2.8b/tree/<branch>and comparelfs.oidofmodel.safetensorsacross branches.gpt_neox.embed_in.weight(offsets from the safetensors header) from any two same-hash branches and compare bytes.Happy to share the extraction code and the full per-branch statistics.
upstream_2p8b_branch_hashes.csv