While extracting embedding matrices across the full Pythia checkpoint grid (HTTP range reads only), we found two pythia-12b step branches whose first weight shard is the final model's.
File-level evidence
pytorch_model-00001-of-00003.bin — the shard holding gpt_neox.embed_in.weight per pytorch_model.bin.index.json on every branch checked — has sha256
9d21715462588073675eca9c65ba55a47d911da460a6083938d04acd4589f542
on exactly four refs: main, step143000, step50000, and step58000. All other 151 step branches have unique first shards, and shards 2–3 (including embed_out.weight, shard 3) are unique per branch everywhere. (main = step143000 is expected; the anomaly is the two mid-training branches.)
Hash table attached (upstream_12b_shard1_hashes.csv).
Tensor-level evidence
The input-embedding statistics at steps 50000, 58000, and 143000 are bit-identical, and discontinuous with their neighbours — while the true end of training is smooth (median input-embedding L2 | median cosine-to-initialization):
| step |
in-L2 |
cos-init |
| 49000 |
1.5169 |
0.5935 |
| 50000 |
1.3056 |
0.5110 |
| 51000 |
1.5132 |
0.5869 |
| 57000 |
1.5003 |
0.5695 |
| 58000 |
1.3056 |
0.5110 |
| 59000 |
1.4952 |
0.5646 |
| 141000 |
1.3082 |
0.5112 |
| 142000 |
1.3069 |
0.5111 |
| 143000 |
1.3056 |
0.5110 |
The duplicated values continue the 141000→143000 trend exactly, so the shard's true home is the final checkpoint: step50000 and step58000 carry the final model's shard 1 in place of their own.
Scope
We verified shard 1 by sha256 and by embedding bytes. Shards 2–3 have unique hashes on these branches, so the swap appears limited to shard 1 — but every tensor within shard 1 (embeddings and early layers) on those two branches would be final-model content.
Impact
Learning-dynamics analyses touching pythia-12b around steps 50000/58000 will silently mix final-model weights into mid-training trajectories. (In our own analyses we drop these two steps and keep the genuine final checkpoint.)
Reproduction (no downloads required)
GET https://huggingface.co/api/models/EleutherAI/pythia-12b/tree/<branch>; compare lfs.oid of pytorch_model-00001-of-00003.bin across branches.- Range-read
gpt_neox.embed_in.weight from step50000 and step143000 and compare bytes.
Happy to share the extraction code and full per-step statistics.
upstream_12b_shard1_hashes.csv
While extracting embedding matrices across the full Pythia checkpoint grid (HTTP range reads only), we found two pythia-12b step branches whose first weight shard is the final model's.
File-level evidence
pytorch_model-00001-of-00003.bin— the shard holdinggpt_neox.embed_in.weightperpytorch_model.bin.index.jsonon every branch checked — has sha2569d21715462588073675eca9c65ba55a47d911da460a6083938d04acd4589f542on exactly four refs:
main,step143000,step50000, andstep58000. All other 151 step branches have unique first shards, and shards 2–3 (includingembed_out.weight, shard 3) are unique per branch everywhere. (main=step143000is expected; the anomaly is the two mid-training branches.)Hash table attached (
upstream_12b_shard1_hashes.csv).Tensor-level evidence
The input-embedding statistics at steps 50000, 58000, and 143000 are bit-identical, and discontinuous with their neighbours — while the true end of training is smooth (median input-embedding L2 | median cosine-to-initialization):
The duplicated values continue the 141000→143000 trend exactly, so the shard's true home is the final checkpoint:
step50000andstep58000carry the final model's shard 1 in place of their own.Scope
We verified shard 1 by sha256 and by embedding bytes. Shards 2–3 have unique hashes on these branches, so the swap appears limited to shard 1 — but every tensor within shard 1 (embeddings and early layers) on those two branches would be final-model content.
Impact
Learning-dynamics analyses touching pythia-12b around steps 50000/58000 will silently mix final-model weights into mid-training trajectories. (In our own analyses we drop these two steps and keep the genuine final checkpoint.)
Reproduction (no downloads required)
GET https://huggingface.co/api/models/EleutherAI/pythia-12b/tree/<branch>; comparelfs.oidofpytorch_model-00001-of-00003.binacross branches.gpt_neox.embed_in.weightfromstep50000andstep143000and compare bytes.Happy to share the extraction code and full per-step statistics.
upstream_12b_shard1_hashes.csv