This is very much a work log showing what it is like to do principled model architecture research! I especially like ideas like this, ones inspired by a problem that was well understood from the beginning. I think the insights and analyses here are far more valuable than the solution itself. Dropping a few things I liked about the blog! - To my surprise, the authors even validated the idea at a 180B-parameter scale. And since the model was quite undertrained due to limited compute, they further provided reasonable explanations & smaller-scaled ablations for why this may or may not affect the conclusion. - They also tuned hyperparameters (like batch sizes), but with a different goal from “randomly trying to find a solution”: they tuned hyperparameters to ablate the effects of suboptimal configs on the task at hand. They concluded that these effects were orthogonal to the task, which allowed them to conduct experiments without worrying that a suboptimal config might confound the results. (Very often, the pressure to keep a model architecture at an optimal setting while scaling up runs into limited compute.) - They deep-dived into optimizers, e.g. adam vs muon's numerical ranges and their effects on epsilon. The two optimizers' comparisons always deliver insights... - Megatron's gradient handling behavior was a really fun read, plus the training ablation on this infrastructure deviation. - Something I really like about the blog is how it carries the mindset of "I care about what the *fundamental* problem is". E.g. identifying how weight norms are weird for early layer MoEs -> fixing this issue -> realizing how the real problem lies somewhere else -> keep validating