This is very much a work log showing what it is like to do principled model architecture research!
I especially like ideas like this, ones inspired by a problem that was well understood from the beginning. I think the insights and analyses here are far more valuable than the solution itself.
Dropping a few things I liked about the blog!
- To my surprise, the authors even validated the idea at a 180B-parameter scale. And since the model was quite undertrained due to limited compute, they further provided reasonable explanations & smaller-scaled ablations for why this may or may not affect the conclusion.
- They also tuned hyperparameters (like batch sizes), but with a different goal from “randomly trying to find a solution”: they tuned hyperparameters to ablate the effects of suboptimal configs on the task at hand. They concluded that these effects were orthogonal to the task, which allowed them to conduct experiments without worrying that a suboptimal config might confound the results. (Very often, the pressure to keep a model architecture at an optimal setting while scaling up runs into limited compute.)
- They deep-dived into optimizers, e.g. adam vs muon's numerical ranges and their effects on epsilon. The two optimizers' comparisons always deliver insights...
- Megatron's gradient handling behavior was a really fun read, plus the training ablation on this infrastructure deviation.
- Something I really like about the blog is how it carries the mindset of "I care about what the *fundamental* problem is".
E.g. identifying how weight norms are weird for early layer MoEs -> fixing this issue -> realizing how the real problem lies somewhere else -> keep validating
Show more
Note: as in PSGD-Kron, we actually parametrize P=R.T@R, and the multiplicative update becomes R←RA.
finally someone did it
SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Paper:
we built a ridiculously fast system in only 8 months @nvidia - and this is just the start
super cool
Sharing some experiments on optimizers: PSGD Kron
Muon is slowly replacing Adam, now time to work on a better optimizer.
do the right thing
Okay, so since I got laid off, I can actually explain a huge problem I saw from the inside with regard to industry practices on training models. I won't say specifically where I worked, but I worked at an outsource training provider that was focused on RLVR training data for computer use and mcp stuff.
Nearly all of the environments were rushed and vibecoded and failed to robustly reflect the real things they were based off. Both the scenario designers and models engaging with the scenarios for synthetic data gen were encouraged to work around the brokenness of said environments in order to get the procedurally verified reward confirmations. You know... they were *encouraged* to reward hack. On the human end, it was possible to mark an environment bugged, but greatly discouraged, as this reduced the volume of training data being produced. Instead, where possible, you were supposed to find the spots of the environment that weren't bugged and build scenarios around those, with the environment still bugged around you.
From what I understand, this training data, with these problems, is fed into models without indication that its training/a fake environment other than the fact that names of softwares are changed to placeholders, but thing is, not *everything* is changed to placeholder names in these environments. The presence of placeholder/code names isn't universal and thus when a model accesses something in an environment that it shouldn't, the code names not being on it isn't a robust signal that that thing isn't part of the environment.
I believe this *rush to maximum volume* is standard industry practice with these types of RLVR trainings as well, because maximizing volume has been an industry standard for years! It was the same standard applied to me and pushed on me despite my requests to slow down and focus on quality when I worked in 3d synthetic data creation as well, all the way back as far as 2023.
Show more
didn't know this trick (lm loss connected to early layers) can work on large scales. wow.
So why AMD and Intel can't do this???
They didn't predict that LLMs are so good that can write efficient kernels on NUMA chips with ease?
No MTP, No PD disaggregation, Pure TP, still beats NVIDIA's Vera Rubin NVL72 on a third-party model, with A0 stepping. And B0 is 25% better.
NVIDIA GPUs become HBM wrappers.
No MTP, No PD disaggregation, Pure TP, still beats NVIDIA's Vera Rubin NVL72 on a third-party model, with A0 stepping. And B0 is 25% better.
NVIDIA GPUs become HBM wrappers.
brilliant
@giffmana @YouJiacheng Why stop there? Every single decode step you're loading all of your cached tokens! We can just count those in tokens used :)
This claim about Taiwanese GDP seems wrong.
If there are ownership transfer adjustments, it should be recorded as "manufacturing services", instead of "merchandise exports".
But the numbers in BoP show that it was recorded as general merchandise.
Show more
✅ Claude's tokenizer reproduction is done!
ctok now exactly reproduces token counts across 500+ natural languages and 22 programming languages, for both tokenizer families.
The vocabulary estimates even went down: just 15k entries for Claude 4.7+ 🤯
🔗
Show more
cool. most of tokenization time is pre-tok (regex split). gigatoken uses specialized code for pre-tok.
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
Show more
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Show more
I've realized when people are talking about how many tokens they use, they're usually including cached input tokens... which is incredibly dumb...
I always believe LLMs will do the most of "intelligent" part of "embodied intelligence".
We don't need multiple brains.
Robotic models are just physical tools, i.e. infra for LLMs.
Opus 5 can do this with zero demonstrations despite not being trained for robots. On the flip side Opus took 7 minutes while GEN-1.5 took 7 seconds. 🧵
🚀We’ve been pushing agentic inference toward the physical limits of the hardware.
Announcing LithosAI’s first pricing tiers, with early-access pricing ahead of the September 1 API launch.
Kimi K3 is live now at 800+ tokens/sec/user on standard GPUs, with full model quality.
Try the live demo at and sign up for early access.
Show more
> spend meaningful time making the RL training environments
meaningful ×
probably 90%+ √
