Sebastian Raschka
@rasbt
ML/AI research engineer. Ex stats professor.
Author of "Build a Large Language Model From Scratch" ( & reasoning (
1.2K Following 511K Followers
The main appeal of open-source agent harnesses isn't that they are free, but that we can inspect what they are doing on our computers.
MiMo-V2.6 is "simply" the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average).
With "simple," I mean a classic Grouped Query Attention (GQA) with Sliding Window Attention (SWA) at a tiny 128-token window size.
So, that underlines one of the points I've been trying to make in recent months: most of the progress still comes from the data and post-training recipe improvements. Fancy attention variants are just mostly efficiency tweaks.
What are some of the training data improvements and recipe improvements? The MiMo team shared a pretty detailed technical report. Lots to carefully digest there, but in short, there are a few things that stood out:
1. An increase in agent tasks; also training across different harnesses (the average DeepSWE pass@1 accuracy on held-out harnesses improved from approximately 50% -> 66%).
2. Better reward signals: they replaced a simple correctness verifier with an agentic grader that looks at the execution traces as well.
3. Large RL batches (1,568 prompts × 16 rollouts = 25,088 trajectories) and 2.7–3.7 billion training tokens per update (unclear, though, what the predecessor used).
Show more
MiMo-V2.6-Pro debuts as the top open weights model on the Artificial Analysis Intelligence Index (46). At $0.13 per Intelligence Index task, it lands on the Intelligence vs. Cost per Task Pareto frontier
@Xiaomi has just released MiMo-V2.6-Pro, an open weights model with major advances in intelligence over its predecessor, MiMo-V2.5-Pro (Intelligence Index: 26). Despite the improvement, it retains the same attractive pricing at $0.435 per 1M input tokens (with a 99% cache-hit discount) and $0.87 per 1M output tokens. This makes MiMo-V2.6-Pro one of the most cost-efficient models to deploy.
MiMo-V2.6-Pro is an MoE model with 1.02T total parameters and 42B active parameters. Stay tuned for additional analysis of the model.
Check out MiMo-V2.6-Pro full benchmarking breakdown here:
Show more
It’s easy to dismiss Jev it as “just a classifier”.
But people (me included) who have been training encoder-style models for classification for many years know they were usually special-purpose and limited in some way.
The breakthrough of Jev is that it generalizes well.
And I’d say the secret sauce is probably more in the data than in the training algorithm. (Plus a nice API design on top of it.)
Show more
poor guy claim to have built Jev a year ago but no one cared, and now Jev stole all the thunder
many people are saying “you gotta tell your story” or “marketing is important”, and they just completely missed what actually made the difference here
i just looked into this laya model and:
- it only supports 512-1k context… a lot of use cases won’t fit at all
- evaluating the model directly shows its accuracy is as good as a coin flip. in order to get good results, you need to first fine tune it
i’m sorry, but that’s not Jev
there’s a massive gap between an interesting research and a useful product
you can “tell your story” all you like, but you can’t blame Jev for stealing your thunder when Jev did all the work to make a well-packaged solution anyone can just grab and go
Jev is not completely new from an academic sense, just like how ChatGPT was not the first LLM
don’t underestimate the effort and value in putting together something that’s actually good enough for adoption - it makes all the difference
Show more
Interesting new insights into the old AdamW vs Muon debate: Muon seems to do better because it reduces memorization.
𝗔𝗱𝗮𝗺𝗪 𝘃𝘀. 𝗠𝘂𝗼𝗻: α, 𝗢𝘃𝗲𝗿𝗳𝗶𝘁𝘁𝗶𝗻𝗴, 𝗮𝗻𝗱 𝗠𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻. A quick update on our WeightWatcher experiments comparing AdamW with the newer Muon optimizer.
We trained a single-head nanoGPT model with both optimizers across five seeds. We then used WeightWatcher to study the weight-matrix spectra and several forms of memorization.
The top plot shows the average raw α.
AdamW produces relatively stable α values. Muon is much noisier. Its ESDs vary substantially across layers, which makes α harder to estimate reliably.
But the average hides something important:
👉 Individual layers do fall below α = 2.
Our spectral/RG theory predicts that this regime can be associated with pathological, example-specific learning. So we tested that prediction directly.
We inserted random sequences into the training set and measured whether the model remembered them using exact extraction, token recall, teacher-forced recall, and canary exposure.
The result was striking.
• AdamW memorized the planted sequences strongly. Earlier in training it reached ~96% exact recall of 32-token canaries.
• Muon strongly suppressed this kind of memorization. At 10,000 steps, teacher-forced recall was ~62% for AdamW versus <1% for Muon.
There is also a surprise: some Muon layers have α < 2 as well. So α < 2 by itself is not sufficient to explain memorization.
This is why we recommend examining individual WeightWatcher layers and their ESDs—not simply average α.
Muon appears to suppress memorization while also producing unusual ESDs that make conventional WeightWatcher analysis more difficult.
We are now running Muon much longer to understand how these spectra evolve.
If you are using WeightWatcher and trying to analyze models trained with Muon, feel free to reach out for help. #talkToChuck#
𝗔𝗱𝗮𝗺𝗪 𝘃𝘀. 𝗠𝘂𝗼𝗻: α, 𝗢𝘃𝗲𝗿𝗳𝗶𝘁𝘁𝗶𝗻𝗴, 𝗮𝗻𝗱 𝗠𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻. A quick update on our WeightWatcher experiments comparing AdamW with the newer Muon optimizer.
We trained a single-head nanoGPT model with both optimizers across five seeds. We then used WeightWatcher to study the weight-matrix spectra and several forms of memorization.
The top plot shows the average raw α.
AdamW produces relatively stable α values. Muon is much noisier. Its ESDs vary substantially across layers, which makes α harder to estimate reliably.
But the average hides something important:
👉 Individual layers do fall below α = 2.
Our spectral/RG theory predicts that this regime can be associated with pathological, example-specific learning. So we tested that prediction directly.
We inserted random sequences into the training set and measured whether the model remembered them using exact extraction, token recall, teacher-forced recall, and canary exposure.
The result was striking.
• AdamW memorized the planted sequences strongly. Earlier in training it reached ~96% exact recall of 32-token canaries.
• Muon strongly suppressed this kind of memorization. At 10,000 steps, teacher-forced recall was ~62% for AdamW versus <1% for Muon.
There is also a surprise: some Muon layers have α < 2 as well. So α < 2 by itself is not sufficient to explain memorization.
This is why we recommend examining individual WeightWatcher layers and their ESDs—not simply average α.
Muon appears to suppress memorization while also producing unusual ESDs that make conventional WeightWatcher analysis more difficult.
We are now running Muon much longer to understand how these spectra evolve.
If you are using WeightWatcher and trying to analyze models trained with Muon, feel free to reach out for help. #talkToChuck#
Show more
Some food for thought when designing benchmarks...
So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max.
The task here was to recreate the image in the center using the Paint UI.
Super interesting how the two different LLMs+Harnesses approached this totally differently by default.
I.e., Astra tried to approach this by drawing and layering geometric shapes. Qwen approached this pixel by pixel. (Of course, the pixel-by-pixel result looks closer to the original, it's essentially a low-res version of that by nature.)
So, the Qwen-generated image would surely score higher in the sense that it's closer to the original.
But I wouldn’t conclude from this example that one LLM generalizes better than the other on other tasks. Also, I wouldn't say Qwen has better compute-use capabilities or better visual understanding than Astra.
But it highlights an interesting point about how slippery benchmarks are when they only compare final results.
Show more
Nice showcase that interesting LLM work can be done on single GPU!
I extended the GPT-2-style code from @rasbt's "Build a Large Language Model (from Scratch)" so that it was a 6-expert (2 active) mixture-of-experts, and trained it from scratch over 8 days. It worked well! Full writeup with maths and code at
Show more
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
Show more
Now we know: The popular Ox Alpha LLM was GLM-5.3-Flash...
Compared to GLM-5.2, this new GLM-5.3-Flash model uses:
- a Kimi Linear-style 3:1 (super*) hybrid attention pattern with 34 Kimi Delta Attention layers (KDA) and 11 Multi-heat Latent Attention (MLA) / DeepSeek Sparse Attention (DSA) layers;
- a scaled-down GLM-5.2-style sparse MoE backbone, going from 744B-A40B to 320B-A18B;
- a DeepSeek V4-style mHC residual path with four parallel streams;
- plus a native vision encoder (not shown).
* "Super hybrid" because both KDA and MLA/DSA are "efficient" components. E.g., Kimi only uses KDA + full attention GQA, DeepSeek V3.2 uses DSA + full attention MLA.
PS: Sry for the excessive tech jargon. Explainers on all these components (MLA, DSA, KDA, mhC, etc.) in my LLM Architecture Gallery
PPS: Haha, maybe justification for getting that pricey Mac Studio M5 Ultra 256 GB / 512 GB to run this locally...
Show more
Introducing GLM-5.3-Flash
- Leading capabilities at a highly competitive price
- Natively multimodal with a 1M-token context window
- A 320B-A18B model released under the MIT License
- Previously previewed as Ox Alpha, running entirely on Chinese AI chips
Blog:
Available now across all official platforms:
Weights:
API:
Coding Plan:
ZCode:
Chat:
AutoClaw:
Show more
A couple of days ago, I did a quick explainer on Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be interesting to go into a bit more detail when explaining how it works.
So, instead of the usual text article, I recorded a little lecture on the topic (to change it up a bit from my usual articles).
It ended up a bit longer than intended, but I hope it clarifies a lot of things:
- Sampling the next token in an LLM and pseudorandom number generators
- How watermarking relates to the regular LLM sampling process
- Whether watermarking makes text "worse"
- How to remove watermarks
- Tournament sampling
- How new text is checked for watermarks without rerunning the LLM
I ended up with ~50 slides, but I hope that these explain it well, though! Happy watching!
Show more
Nice case study on using optimized functions whenever possible (except for educational purposes, though 😆)
By switching from a hand-rolled GELU to PyTorch's built-in one, I improved my LLM training speed -- and much more than I expected, from 21,000 tokens/second to 25,000!
8/19, join us for the #ACMTechTalk#, "From Conventional LLMs to Reasoning Models to Agents," w/AI & LLM Research Engineer @rasbt. ACM Practitioner Board Co-Chaior @marlene_zw (@Microsoft) will moderate.
Register (free) to attend live or on demand:
Show more
Whoa, Meta released a new open-weight LLM yesterday, something that hasn't happened since the good old Llama days.
Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like architecture design. (“Glimmer” is probably a wordplay on “Spark,” the more likely capable model from which Glimmer was distilled. Muse Spark is only available through Meta’s Model API, though.)
Architecture-wise, here are some of the main points:
1. "Only" a 131k context window, compared to Qwen3.6 and Gemma 4, which support 2x that natively; it's reasonable, but maybe on the shorter end in the age of agent harnesses
2. It's a dense model, not a mixture-of-experts. (So, it's fairer to compare it to Qwen3.6 27B than Qwen3.6 30B-A3B.)
3. Hybrid attention with grouped-query attention (GQA) and sliding window attention (SWA); the SWA:GQA pattern is a 3:1 local:global ratio. Other models like Gemma 4, which uses similar components, have a 5:1 ratio for comparison.
4. It adopts gated attention for both GQA and SWA; gated attention has become quite common in recent months. It basically applies a sigmoid gate to the attention output to decide how much of the attention information enters the residual connection. The interesting point is that it uses relatively standard GQA and SWA rather than hybrid attention mechanisms such as Nemotron or Qwen3.6.
5. A very extreme GQA ratio: 32 query heads and only 2 KV heads; for comparison, Gemma 4 31B uses 32 Q / 16 KV in the local heads and 32 Q / 4 KV in the global heads. This means that Meta Glimmer has a very small KV cache.
Overall, the probably most similar architecture is Gemma 3 27B (including the Gemma-style pre/post RMSNorm placement) and Gemma 4 31B, but with some tweaks like SwiGLU instead of GeGLU activations, gated attention, and the more extreme GQA:SWA pattern mentioned before.
What stands out is its extreme KV-cache efficiency.
I.e., the KV CACHE / TOKEN ratios (in BF16) are:
- Muse Glimmer: 52 KiB (lower is better)
- Qwen3.6 27B: 64 KiB
- Gemma 4 31B: 840 KiB
Modeling-performance-wise, their own benchmarks show that it's mostly ahead of Qwen3.6. According to the independent composite benchmarks on the Artificial Analysis Intelligence Index, it's slightly behind Qwen3.6 (see figure below). So, a few days of using it will tell where it really ranks.
Overall, it looks like a solid model, particularly for agentic workflows. What stands out most is its very low memory footprint and also pretty fast prefill and decode speed. It’s also just great to see Meta releasing open weights again :).
Show more
It’s been a long time since I’ve been excited to work through a technical book @rasbt
It’s time to bring more ML back into my life
Yes, LLM architectures are getting a little more complicated
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts.
1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)
2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).
4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.
5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.
6. Kimi K3 now also has native multimodal support, which is great!
There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
Show more
this scaling law is a piece of art, kimi K3 recipe improves by ~2.5x over kimi K2 recipe
the tech report is amazing
Looking forward to chatting with @hugobowne today (July 27) at 4 pm PT on the Vanishing Gradient livestream on YouTube.
Will cover open source, the newest LLMs & trends, agent frameworks, and whatever else comes up.
Basically, all the exciting stuff. Should be fun!
Show more
Next week I’m sitting down with @rasbt, independent AI researcher, author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch), and creator of Ahead of AI, which just crossed 200,000 subscribers.
Since we last spoke, Sebastian and I have been messaging about the insanity of DeepSeek-V4, then GLM-5.2, and now Kimi K3. Qwen3.8 is about to land as well.
I’m excited to ask him:
• What are these new models actually doing differently?
• What do stronger open-weight models mean for local coding agents?
• Which parts of our agent systems will survive the next model release?
• What did he learn by starting with a small Qwen3 model and adding evaluation, inference-time scaling, reinforcement learning, and distillation himself?
What do you want me to ask Sebastian? Reply with your questions and I’ll ask my favourites live.
@ManningBooks is giving away five ebook copies of Sebastian’s new book to the people who ask the best questions. They’re also providing a 45% discount for the audience.
We’re live July 28. Registration link below.
Show more
Yes, open-source / open-weight models are important for a healthy AI ecosystem. That's how we can verify things, check claims, and keep up outside the closed labs. Plus, it gives us the freedom to run AI on our own hardware if we are not ready to share personal data and IPs with closed labs through using their models. (Not that proprietary models are bad, actually I use them a lot as well, but it wouldn't healthy not to have any alternatives.)
Anyway, while pretty much everyone is waiting for the Kimi K3 and Ling 3.0 weights to land on the model hub any day now, there were quite a few other interesting new open-weight model releases the past week. Yes, one of those weeks!
So, here are the architecture pics along with some notes on what I found most interesting:
1) Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44-layers, but without duplicating the weights. (2x the transformer block compute but same memory footprint.)
Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive.
2) Laguna S 2.1 is poolside's Laguna model in a really nice size: 118B sparse MoE with 8B active parameters and a 1M-token context window. Otherwise, the architecture is pretty standard. It uses 36 sliding-window and 12 global (gated-)GQA layers. However, given this size, and the fact that it (just barely) runs on my DGX Spark (uses about <80 GB of RAM), this is right now the most interesting model for me personally. It's 3x bigger and thus a tad slower but maybe a good candidate as daily-driver-Qwen3.6-35B-replacement. (Still waiting on some more independent performance benchmarks though.)
3) Motif-3-Beta is a new 314B-A13B sparse MoE that is somewhat based on DeepSeek V4 in terms of mHC and latent attention. But it uses a new component, Grouped Differential Latent Attention, which is inspired by Multi-head Latent Attention. I probably should write an article about this some time, but for now, the tl;dr is as follows. Regular MLA compresses the keys and values into a smaller latent representation to mainly reduce the KV cache size. GDLA does a similar low-rank compression but puts the attention heads into groups and also learns a noise head for each group where the noise gets subtracted for filtering purposes... Anyway, a topic for another day!
4) Solar Open 2 is a new 250B-A15B hybrid MoE by Upstage that interleaves three Kimi Delta Attention layers with one GQA layer.
5) Antares 1B is a small model (and there is also an even smaller 0.3B variant) from Cisco starts that with the IBM Granite 4.0 1B backbone and uses SFT plus GRPO for terminal-based cybersecurity stuff. It is a nice example of task-specific post-training on a genuinely small model.
6) BTL-3 is a rank-32 LoRA adapter for Qwen3.6-27B aimed at coding agents and structured tool use. The really strong benchmark performance suggests that LoRA adapters are still a useful tool/technique in 2026.
I added all six to the LLM Architecture Gallery for some additional details:
Show more
Thanks everyone for all the kind words and feedback. Super happy that you are enjoying Build a Reasoning Model (From Scratch)!
Unfortunately, there's small typo in listing 6.5 on page 198 (see video below).
The line "torch.manual_seed(0)"
should be "torch.manual_seed(5)"
This correction is needed to reproduce the generated response in listing 6.5 and the corresponding log-probability outputs later in Chapter 6. If you use 0, the generated response and the results that follow will be different.
This will be fixed in the next printing. I am sorry about the oversight, and I hope this note saves you some debugging time.
Show more
How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less?
I put together a “little” article explaining how these effort levels are implemented at inference time and during training.
Show more
