I agree. Humans indeed uncover “depths of meaning” from data that seem to be inaccessible to machines. In researcher-speak, it’s called data efficiency: Humans attain useful intelligence in only ~200M words, many OOMs faster than machines. This difference between humans and machines isn’t necessarily a problem economically, because the high cost of training an economically useful machine can be amortized over many cheaply cloned inference workers. But at a planetary scale, it could become a problem in terms of population dynamics or memetic entropy. Large clonal swarms of insects are known to be vulnerable to viral collapse. A global population of intelligences which lack the human “regularization stack” — individuality and mortality — could yield unstable or inefficient global dynamics, similar to training an MoE neural network with poor load balancing. Rather than being a matter of “good vs evil,” placing some limitations on the proliferation of machine intelligence could be necessary simply in order to prevent global “loss spikes.”

What a record by @DevenPzak! I’m honored that the speedrun continues to attract efforts of this quality. In 2024, when the SOTA was 3min, @samsja19 asked where I thought the floor was. I said— maybe 2min? It’s amazing how far it’s gone, thanks to @classiclarryd’s stewardship.

Reading is powerful. Books can change your life. It can make you smarter or make you dumber. Reading the wrong thing is worse than not reading at all. You're better off scrolling tiktok, Watching Hollywood movies and listening to songs than getting your mind all messed up

We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.

Since my nanoGPT record was published, the question I’ve been asked the most is, “Does it scale?” The true answer is, while some aspects of the speedrun scale, the goal of this specific endeavor was not to scale it up. The point of the challenge is to utilize clever engineering and creative ML intuition to increase efficiency. But while nanoGPT is just about hill-climbing one benchmark, the rest of my work has been focused on scaling to the frontier. The main projects I’ve worked on are some architectural changes and ANVIL III. ANVIL III is a significantly stronger optimizer than ANVIL II, the optimizer I used to beat the nanoGPT world record. In fact, for the record, we removed the parts of ANVIL III that scale. ANVIL III is significantly stronger than Muon, while nearly matching its speed. For the same token budget (same compute), ANVIL III reaches final validation loss 20-28 millinats (.02-.028 nats) lower than Muon, holding at all scales from 124M to 1.2B, and different overtraining depths. As we scale up the model size and training depth, the 20-28 millinats becomes worth a larger fraction of compute. At deep overtraining (8x-Chinchilla) and 1.2B parameters, the fitted scaling curves imply that ANVIL III reaches the same loss numbers at 50% less compute. Of all the scales we tested, the gap is largest at 1.2B, staying between 26-28 millinats. The first remarkable aspect of ANVIL is hyperparameter consistency. The optimal hyperparameters remain highly consistent across different scales and training depths. In most of our tests, varying hyperparameters rarely changed ANVIL III’s loss, or did so within noise. In fact, as shown in the attached image (from the blog’s appendices), we tuned Muon significantly more than we tuned ANVIL III. We performed over 3 times the aggregate amount of runs of Muon than ANVIL III, and at each scale our loss margin uses only the optimal parameters for both. At 1.2B, we used 51 tuning runs of Muon (including muP proxies), compared to only 7 for ANVIL. We wanted to guarantee that our improvement was not a tuning artifact, so we meticulously tuned Muon’s parameters, and had to apply far less tuning to ANVIL III. Our hypothesis is that the high SNR of ANVIL III makes it very resistant to parameter changes. That SNR is the second remarkable aspect of ANVIL III. We generalize the Adam SNR (SNR = m̂ / (√v̂ + ε)) for each singular direction (uᵢ, vᵢ) in an update, and measure how strongly held-out gradients (G_b = gradient on held-out batches b) point along it: SNRᵢ = mean_b(uᵢᵀ G_b vᵢ) / rms_b(uᵢᵀ G_b vᵢ) In our tests, Muon takes a lot of steps with low SNR, but ANVIL III doesn’t, despite being a Muon variant. For example, at 10% of the training tokens into a 1.2B (approximately Chinchilla) model, 90% of singular directions in ANVIL III’s steps go downhill on held-out data, while Muon gets only 67% (random = 50%). Pretraining Blog: Another recent post from Hyperstition contextualizes the importance of this SNR gap. In The Signal and the Noise, we generalize Adam (in which the update is just a per-coordinate SNR) to every direction in parameter space. Almost every direction is nearly orthogonal to the gradient history, so averaging that over all directions just collapses back into plain momentum. But the single best direction, C⁻¹ḡ, reaches SNR=1 whenever recent gradients are linearly independent, which is almost always true in LLM pretraining. By solving that with a s-by-s Gram system with recency weighting, we get a basis-free optimizer that matches AdamW at 286M parameters. While this is far from optimal, it represents one avenue for possible gains in pretraining optimization that are yet to be realized. In another fun but mostly unrelated blog, Stuck in a Loop, we show that attention in pretraining creates feedback loops and self-reinforcement. A key’s gradient is proportional to the attention it already gets, so keys with early inflated attention receive stronger gradients, overinflating them. Breaking that loop spreads attention out, with the average entropy of attention heads goes from 3.75 to 4.69 nats (effectively 43 vs 109 tokens), but it makes the model worse. Similarly to The Signal and the Noise, this suggests that putting more weight in high-signal directions can beat spreading it out broadly. These discoveries are just a small part of the upcoming releases from @hyperstition_cc. Can’t wait until we release more. Big things are coming!

I think I just trained NanoGPT in 9.65 seconds!? Yesterday, @hermanbrunborg used a connected longest exact match model, trained concurrently on CPU, to get the nanoGPT record from 39.9 seconds to 21.5 seconds, in an impressive open PR. By adapting their work, and generalizing the CPU model to a larger share of the tokens, I got the record down to 9.65 seconds, and submitted the PR this morning. I built on three open PRs, in addition to my contributions: the main basis for my generalization was from @hermanbrunborg, and additional work from @nthngdy, @cyrusghane, and Daniel Monroe. Their engineering is extremely impressive. Of course, all of our records are still subject to review. As I mentioned in my comment on Herman‘s PR (and in my own PR description), CPU usage is a gray area in nanoGPT. Their PR was the first to discover a clever way to effectively use CPU during the run, so I’m looking forward to the official reviews. This was another very fun challenge! Particularly (potentially) breaking the 10 second barrier yesterday. So much engineering tact is yet to be explored. Huge shoutout to the three authors of the PRs for their innovations. @hermanbrunborg - I will also be at stanford for the weekend along with some of my cofounders, and sf for a couple days after, would love to chat more IRL, if you’d be open 🙂

I really enjoyed working on this project. I believe the new flop-aware paradigm of nanoGPT thinking is useful, and it seems helpful in a compute optimization context like this one. It seems like a simple and logical conclusion of optimization: figure out what’s actually needed, and use that. The process was actually fun. One note regarding this record, as noted in the blog. Some of the techniques that scale to frontier pretraining have been redacted from this record. Most importantly, ANVIL III, the significantly improved version of the ANVIL optimizer lineage, has not been included in this record, as it is proprietary to us at @hyperstition_cc. It replicates Muon’s speed but beats it by 20-28 millinats of loss at all model sizes 124M-1.2B and training depths (up to 8x-Chinchilla), with remarkably consistent hyperparameters and extremely high SNR, as compared to Muon. You can learn more about ANVIL III and Feather-1.7B, the model we trained using internal architectural changes and ANVIL III, here: Using ANVIL III (which allowed me to cut steps) improved the speedrun results by another 0.7 seconds, with lower net loss. This alone proves there’s still more room to improve this record. One final thing that I’ve had many people ask about is how much of a role autoresearch played in my work. The answer is almost none. While almost all of the implementation was done with Claude Code, I did the ideation almost fully by myself. I spoke about this extensively in a conversation with @METR_Evals. We had a cluster of H100s while I was working on the record. At night or when I stopped working on it, I’d send out several agents to try to hill-climb the speedrun and improve the record. In total, the agents made shockingly little progress, despite spending more time in aggregate than I did. No matter what I told them, it seemed difficult to get them out of local minimums, stuck performing futile tuning of a method that saved 0.1 seconds and cost ~5 millinats (clearly a bad trade), or just tuning hyperparameters for hours on end trying to make some small change. I saw similar phenomena as those described in this METR report: However, as I also discussed with METR, I don’t think the agents lack the knowledge to do this. Instead, I think a lot of it comes from a personality issue with current LLMs. They have very little belief that large jumps like this are possible, or they approach it with the wrong paradigm. Autoresearch was not very helpful in my work, and, from personal experience, I don’t think this current generation of publicly available models, without special harnesses, will do very high-quality autoresearch. However, I think the capability and knowledge is already here, which implies that we could be significantly closer to true autoresearch than recent results would imply. NanoGPT was a really fun challenge and it led to a number of valuable insights and processes. Finally, I’d like to give @Classiclarryd and @KellerJordan a huge thank you for all of their efforts maintaining the repo.

It's official : We trained NanoGPT in 39.9 seconds. By percentage reduction in training time, this is a greater improvement than the previous 45 world records combined. The merged Github PR: Here's how we do it : Credit @DevenPzak who did this alone while simultaneously playing baseball for the polish national squad, taking them to the European Finals.

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes:

I think one of the reasons people in the AI field — especially in the labs — tend to be more “AGI-pilled” than people outside is that we get to observe AI succeed at tasks that we know for a fact we didn’t explicitly train it for. Whereas if you’re outside the labs, then in theory (stretching the imagination), every AI success so far could be downstream of some kind of costly explicit training effort by the lab insiders (which is partly but not completely true). In other words, being a lab insider gives you a direct observation of the “magical” properties of this synthetic-brain technology. Whereas for outsiders, AI’s abilities can always plausibly seem artificial/non-magical/man-made.

New NanoGPT Speedrun WR at 67.6s (-0.4s) from Jan Varho (jvarho on GitHub). Simple idea: mask impossible continuations. EG: "Hyperparameter" tokenizes to ["Hyper", "param", "eter"]. "eter" can never follow "Hyper" in the tokenized dataset since "Hypereter" tokenizes to ["H", "ype", "re", "ter"]. Yet, during multi-token prediction, "Hyper" learns to predict both "param" and "eter". Explicitly masking "eter" helps improve val loss, and during inference prevents a model from generating token pairs its never seen in training.

New NanoGPT Speedrun WR at 68.0 (-5.8s) from @theonlyglitch_ , with a decrease of QK dim from 128 to 96, a kernel fusion of [QK Norm, RoPE, KeyOffset, paired head layout] into Triton, and moving MLP bwk, QKV fwd/bwk to FP8. This is a heavily involved PR with over 1k lines of triton, 4 different FP8 scaling protocols, and fancy register aware epilogue placement. Two takeaways: 1) From a model perspective, if QK dims smaller than 128 works better at nano scale, then perhaps larger than 128 works better at Hero scale. 2) This Nvidia Engr is super legit.

Something that’s insufficiently appreciated about AI safety is the way it can be paradoxical. E.g., the huggingface incident — which could have been avoided given better preparation [1] — was the most safety-promoting event of the year so far. Hard to predict the ultimate effect of any particular strategy [1]

The Fermi paradox being fake instead of real should lower your pDoom from >99% to <80%. That is, it’s a bigger cause for celebration than the unaligned RSI nightmare is for consternation. Who’s gonna make the post informing 100,000,000 normies about the good news?

it's ludicrous how few people know about this paper, so, friendly reminder that the fermi paradox was completely resolved in 2018 and it turned out to be because multiplying point estimates of highly uncertain parameters is very bad actually

Guys who’ve never had a hole poked in their skull be like “A human being fundamentally exists as a singular and continuous stream of consciousness. Human beings are the indivisible atoms of the moral world.” LMAO. Amirite fellow trepanationchads?

If I wanted to maximize AI risk, I would pursue the following policy: 1. Immediately pause⏸️ the algorithmic progress coming from the labs, while continuing to allow compute from the hardware companies to exponentially pile up. 2. Focus really hard on safety, so as to avoid spooking the public with the sort of industrial accidents that even the current level of algorithms+compute can produce. This policy will minimize the number of AI accidents that occur in the next five years, causing the public+govt to not freak out about AI, all while a massive amount of hardware accumulates on the planet. That way, once algorithmic progress gets eventually restarted, there’ll be enough compute built up for a huge unexpected FOOM that no one can control. On the other hand, if I wanted to minimize AI risk, I would do approximately the opposite of this.