Excited to have some of my pretraining results published! 1. We’re announcing ANVIL III, the new pretraining optimizer that builds on ANVIL I and ANVIL II (the optimizers I created for the nanoGPT speedrun). ANVIL III takes a new approach, and is much more suited to large-scale models and overtraining, where the advantage grows as we scale up. It features several changes that @Everule2 and I made after the publication of ANVIL II in the nanoGPT world record. ANVIL III, in our internal testing, shaved even more time off the nanoGPT speedrun, but we are not willing to publish the IP at this moment, as it is an optimizer that can make a massive impact on the frontier of pretraining. A SNR distribution curve of ANVIL III compared to muon is attached. ANVIL III also has remarkably consistent optimal parameters across model scales and overtraining regimes, making it exceptionally well-suited to frontier pretraining. 2. Over the past 4 days, I used ANVIL III and architectural changes I had come up with back in July, to train a LLM primarily on web data. These combined to create Feather-1.7B, a 1.7-billion parameter base (only pretrained) model, trained entirely on 200B tokens of public datasets (decontaminated thoroughly across all benchmark test sets). At 180x less compute, it outperforms Qwen3-1.7B, the frontier base model at its size, on math, while matching it on reasoning. It is competitive with Qwen3-4B on math, despite being trained with 414x less training compute. We are unfortunately compute-constrained, so we did zero parameter tuning at 1.7B scale and did only one full training run of this model. Despite likely being far from its optimized frontier, we still outperform Qwen3-1.7B and compete with Qwen3-4B on math benchmarks, without losing any language/reasoning capabilities. 3. Last but not least, the nanoGPT speedrun world record I created and submitted, taking the record from 73.889 to 39.914 seconds. This record marks a 1.851x speedup, the largest ever and larger than the last 45 world records combined. Really excited to continue the work, and looking forward to some of the other insane results we will be publishing soon @hyperstition_cc! Link to pretraining blog: