Austin Lyons
@austinsemis
semis analyst. asking questions + explaining.
read by chipmakers, funds, model labs.
@chipstrat · @semidoped
1.8K Following 7.8K Followers
Beautiful morning for a 10k!
Maia 300's scale-up fabric allows prefill/decode disaggregation without crossing a scale-out network. The KV cache is passed all on the same fabric, from prefill to decode instances.
Show more
We just finished recording, “a micro explainer on micro LEDs and micro VCSELs” — should be a good episode!
The all new shiny Microsoft Maia 300, and the details brought straight to you by @semidoped
Check it out!
Maia 300 networking insights:
1. The NIC is on the die and built only for scale-up: low latency, high bandwidth between accelerators.
2. Tier 1 is copper inside the rack. Neighboring Maias talk the most (tensor parallelism), so about half of each chip's bandwidth stays in the server and half goes to the in-rack switch. 96 accelerators per rack.
3. Tier 2 is optics between racks, and it's still scale-up. Up to 128 racks, 12,288 accelerators, one domain, one communication stack.
4. Standard Ethernet switches and cables for the scale economics, with Microsoft's own transport on top. The NIC and transport are where Microsoft does its own engineering.
5. Why go past one rack? Bigger models, more aggressive sharding, and flexible job placement. The scheduler can put a job anywhere in 12,288 chips.
6. Fast data movement. Prefill and decode run as separate pools, and the KV cache moves between them over the scale-up fabric, never the slower scale-out network. For agents with long context, that's the expensive data movement.
Show more
Saw Saurabh at OCP yesterday!
Here's an insightful chat we had recently about the Maia 300.
$MSFT
🎙️ NEW EPISODE: @Microsoft Maia 300 with Saurabh Dighe
Why Maia 300 makes memory bandwidth and capacity a first-order design principle.
- Moving data to feed the compute engine can cost two orders of magnitude more energy than the compute itself
- Maia 200 had six HBM3E stacks at 7 TB/s. Maia 300 has eight HBM4 stacks at 18 TB/s
- Compute grows about 35% while rack-level memory bandwidth grows 5x
- 7x token throughput per rack over Maia 200 at 2.5x perf per watt
- Prefill and decode are disaggregated, and the KV cache moves over the scale-up fabric instead of the slower scale-out network
- Memory interface chiplets "extend the shoreline" so the TSMC N2P compute die stays one efficient die
This episode is presented by @CrusoeAI. Crusoe's Serverless Fine-Tuning trains an open model like Llama, DeepSeek, or GLM on your own data, and you pay per million tokens, not per GPU hour.
About $2 buys roughly 5M training tokens on smaller models. Get $5 in credit to start:
Chapters:
0:00 Intro: Saurabh Dighe
1:39 Microsoft's Silicon Philosophy
5:52 Workload Shift: Agentic AI
9:32 The Data Movement Problem
14:20 Crusoe Ad Read
17:47 Maia 300 Specs
24:07 Chiplets & HBM4
29:12 Prefill/Decode Disaggregation
30:33 Two-Tiered Network
35:45 NPO vs. CPO
36:46 Rack-Scale Power & Cooling
41:35 Agentic Kernel Optimization
49:08 What's Next: Maia 400
Get more of Austin and Vik daily, free!
Sign up:
Connect with Vik and Austin:
Vik's Paid Substack:
Austin's Paid Substack:
@austinsemis @vikramskr
Show more
We take you on an exclusive tour inside a secretive AWS networking lab. If you care about networking, timing, optics, and hyperscale, this is an awesome one
1.6T LPO from Linktel
No DSP, and you can see pre-FEC BER is about 1e-9.
Temps stay cool, at 35C due to no DSP
The deployment is LPO is expected to be limited to specific use cases.
LPO makes it difficult for the link to work with different vendors on either end. The link needs to be trained on a per port basis, and parts of the CTLE and amp are adjusted to close the links.
This extra work is what LPO difficult and limits its widespread use
Show more
Right silicon for the workload
All the way down to interface chiplets
Optimize all the things, even down to interface IP!!
When 1-2 applications drive all your revenue, it pays to optimize the full stack, from rack to chip. Standards-based IP gives way to application-optimized IP.
Show more
So many fibers
Lots of NPO
But you can still engineer reliability by over provisioning and having telemetry
What if you had an optical backplane
Zero flaps like copper
A vision from Credo
$CRDO 1.6T ALC at OCP
And @vikramskr documenting it
Interesting that OpenAI provides the FDEs here
How does the Synopsys-OpenAI deal actually work?
OpenAI brings:
• Specialized model
• Compute
• Forward deployed engineers
Synopsys brings:
• Tools
• Expertise
• Skills to make GPT Synopsys an expert user of their own software
Show more
Semi Doped hits up OCP!
@austinsemis and @vikramskr grabbing front row seats to the keynotes
Bunch of substackers went to OCP
Chip prices through the roof at #OCP2026#
This bag was $8
Introducing Maia 300, Microsoft’s next-generation accelerator. Building on the magic of Maia 200, it delivers 40% more inference tokens per megawatt compared to leading GPU-based systems, and up to 7x the rack-scale token throughput of Maia 200. It’s another step toward rethinking innovation and unlocking new possibilities.
Discover how Maia 300 will help co-create what’s next for Microsoft’s infrastructure and technology:
Show more
We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case).
I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves.
We're making it easy to fine tune models internally for our employees. We even have a skill for it in our internal harness that makes it easy to spin these up and start training them:
"/fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started"
I imagine we'll have some backend services that could be replaced entirely by this new architecture.
Show more
Spot on 😂
Bulls will think this is extremely bullish because user penetration is low and we are still early; bears will think this is extremely bearish because user penetration is low and adoption will be much slower than bulls expect.
Show more
50,000 podcast downloads and counting!
And we have so many more awesome guests planned on the show.
Can’t wait!
