This is misleading from the lens of practical model architecture research. Some clear limitations that weaken the conclusion: - While SWA can inherit weights learned by full attention, you don't assume the same for linear attention and train them on top of the QKV proj matrices learned by full attn with zero ablations on whether this is close to optimal (in fact this training method is far from optimal). - Apart from how you train it, the linear attn variants tested here are also years behind the current sota. No frontier model uses pure linear attn. It has repeatedly been shown that the optimal config is a hybrid of linear and full, which beats both pure linear and pure full - e.g. Kimi Linear's architecture with 3 kimi delta attention to 1 full attn is, to date, still by far the best-performing arch among the ones I've tried. And yes, SWA hybrids were included in the comparison. - "Training linear attention models is expensive and most software/hardware is made for Softmax attention" This is untrue. There are many fast open-source kernels written for linear attn models whose speeds are even faster than training full attn models under fair context length. Speaking of which, I don't see a training sequence length specified in the paper. - I understand the limited compute. You don't have to do from-scratch or large scale training, but even under this switching setting a principled work still need to properly compare the variants. Overall, I think it is important to have a full understanding of the current state of method X before claiming its false inferiority. If one lacks the compute/infrastructure/data/etc., open research is still very appreciated, but overly strong and unprincipled claims could be misleading.

glm 5.3 flash uses the same kda gate lower bound, even the same numerical value, as kimi k3. k3's tech report was released exactly a month ago. if there's no coincidence here, getting such a model done within one month is quite insane (plus ox alpha was out a week ago..)

This is very much a work log showing what it is like to do principled model architecture research! I especially like ideas like this, ones inspired by a problem that was well understood from the beginning. I think the insights and analyses here are far more valuable than the solution itself. Dropping a few things I liked about the blog! - To my surprise, the authors even validated the idea at a 180B-parameter scale. And since the model was quite undertrained due to limited compute, they further provided reasonable explanations & smaller-scaled ablations for why this may or may not affect the conclusion. - They also tuned hyperparameters (like batch sizes), but with a different goal from “randomly trying to find a solution”: they tuned hyperparameters to ablate the effects of suboptimal configs on the task at hand. They concluded that these effects were orthogonal to the task, which allowed them to conduct experiments without worrying that a suboptimal config might confound the results. (Very often, the pressure to keep a model architecture at an optimal setting while scaling up runs into limited compute.) - They deep-dived into optimizers, e.g. adam vs muon's numerical ranges and their effects on epsilon. The two optimizers' comparisons always deliver insights... - Megatron's gradient handling behavior was a really fun read, plus the training ablation on this infrastructure deviation. - Something I really like about the blog is how it carries the mindset of "I care about what the *fundamental* problem is". E.g. identifying how weight norms are weird for early layer MoEs -> fixing this issue -> realizing how the real problem lies somewhere else -> keep validating

Something similar to this is the invention of QB: It is not that easy to understand the math part, but if you ignore the math derivation, the solution is extremely simple (with a litte bit infra challenge as well to find the exact top m*k/n score). Compared to auxfree bias, I also felt QB is good for optimization and scaling: you do not need to worry about the lr schedule for the bias update, and you do not need to worry about the bias RMS v.s. your gate score RMS, so your model never collapses to a static routing MoE at alll (which is bad as your topk is sort of dominated by the bias?) However, my personal taste is that QB is not as important as attention residual. Actually I am surprised that why auxfree bias is not working for K3's MoE balance...although a little bit more complicated, but I personally found auxfree surprisingly work very well for most cases

I’ve been working towards AGI my whole life, and as we enter this pivotal moment, I’m stepping into a new role as Chair of Google DeepMind & Chief Scientist of Alphabet. This will allow me to focus on long-term strategy, and accelerating scientific breakthroughs, including leaning into my work at Isomorphic to help cure disease. I’m excited that @koraykv will be stepping up to lead GDM as SVP, alongside @joshwoodward and our exec team. I could not be more excited and confident about our amazing next chapter! 🚀

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta! 🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive performance leap below! 👇 🔷 The official V4-Flash now natively supports the Responses API format and is fully adapted for Codex! Check out the configuration details in our official API docs:

Introducing Online KL Shampoo (OKLS), an optimizer that brings a KL-optimal approximation of full-matrix AdaGrad to language-model training. Diagonal optimizers ignore correlations between gradient coordinates. Full-matrix AdaGrad captures this geometry but requires quadratic state. Muon considers correlations but not their history. OKLS closes this gap using KL-optimal Kronecker factors, whitening matrix gradients across both row and column directions while remaining naturally scale-invariant. The main challenge is computing fresh inverse-square-root preconditioners at every step. Even one-step staleness can destabilize training. We make zero-staleness preconditioning practical with Scaled CANS Coupled Newton–Schulz: 10 iterations, 27 FP16 GEMMs, and FP32 accumulation. OKLS achieves 1.45× the parameter efficiency of Muon while retaining 98% of its training throughput. Across 200M–1B models, an OKLS model matches a Muon model roughly 1.5× larger.

man ignore the benchmarks and the race i just want there to always remain a frontier level intelligence model that is open and safe, available to everyone regardless of region it is truly incredible to have built such an open (and strong) model with a group of genuine people