Pretraining progress seems to be coming mostly from data improvements. @who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales. Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x). And the gains stack independently - a better dataset helps every architecture about equally, and vice versa. Here are full results, plus what we think this means for the future of AI progress:

Massive pretraining is really starting to feel different. We're committed to $3.5B of compute for our next version of Helix, and Index is growing every day. It's time to scale up. See the full Helix 2.5 report here:

PyTorch-native NeMo AutoModel handles transformer pretraining in @nvidia's end-to-end workflow for building a transaction foundation model. The workflow combines GPU-accelerated data processing and tokenization, decoder-only model pretraining, embedding extraction, and XGBoost fraud classification. On the synthetic @IBM TabFormer dataset, combining raw features with learned embeddings increased Average Precision by 41.76% over the raw-feature baseline. 🔗 Read the full post:

Liam Fedus says an AI model that memorized the answer in pretraining can "fake work" in training, and what it learns won't generalize. "If the pre-trained model has already memorized some of this data, and then you try to do reinforcement learning on that, if it already knows the answer and you construct reinforcement learning tasks that depend on that answer, it can fake work." "It already knows the answer, so it doesn't actually have to do the hard physics reasoning. It doesn't have to actually do the calculation simulations, and it gets to the right answer." "And then you reinforce that policy and you upweight those reasoning strategies. Those reasoning strategies will not generalize to novel systems." His fix is a timestamped record of lab evidence: "You could say at this date, this was our experimental evidence up until that point." - Liam Fedus, cofounder of Periodic Labs, on Latent Space with @swyx

.@nathanbenaich explains why AI labs are shifting compute from pretraining to RL: "The last 4-5 years have been trying to figure out what the recipe is for pretraining, what is the best data mixture, what are the best ingredients to this whole magical soup." "Then over the years we figured out how to do that better. So it strikes me as normal that one would end up spending less money on exploratory work because the solution is more in the plain eye." "The whole pitch with RL is this idea that what we have as solutions today are really local maxima that were discovered through human ingenuity and sharing, through reading and whatnot. But that doesn't necessarily mean it's the global maxima."

Discover how enhanced patch-text alignment advances vision-language pretraining at #ECCV2026#. Join Gabriele Berton and Ye Xia at the Google booth (#2#) today at 1:00pm CEST for an interactive showcase of TIPSv2. Check out the paper and demo: @GoogleDeepMind