thoughtful work in pre-training. they test every recipe change on a ladder of small models (about 300M to 6B active params) before trusting it for the final run but the ladder is only useful if the small models face the same conditions as the big one. like this approach differs for data. if taken a small ladder model trained on (lets say 238B tokens which is 1% of the final run). if you sample its data from the full 13.7T pool, it never sees anything twice and the pool is about 58 times bigger than its budget. so the small model is always training on fresh data and the big one is not. they fixed it with data-limited subsampling. for each ladder size, shrink each source’s pool by the same factor as the training budget. At 1% scale, the code pool becomes about 36.5B unique tokens, and the small model trains on about 84B code tokens (2.3 passes, same as the final run).