Dropping a podcast with
@vikramskr, writer of Vik's Newsletter and co-host of Semi Doped, on AI agents, memory, and the hardware behind the data center buildout.
Buying more GPUs is only one piece. Getting data to them fast enough is another.
Vik explains why memory and interconnects can leave compute sitting idle, why AI data centers need more optics, and how training and inference create different hardware demands.
We also get into a question I found interesting. Do agents always need the highest possible speed, or should we be optimizing for how much useful work they get done per token?
In this conversation we discuss
- AI agents and what they actually need to run
- AI spending and the risk of overbuilding
- Training versus inference workloads
- Copper, optical interconnects, and co-packaged optics
- SRAM, DRAM, high bandwidth memory, and storage
- Useful work per token and agent context
- New inference chips and the memory bottleneck
Timestamps
00:00 Highlights
01:18 Vikram Sekar and Semi Doped
02:55 AI agents and hardware demand
18:57 AI spending and the risk of overbuilding
25:10 Training versus inference
36:25 Why distance limits copper bandwidth
41:18 Co-packaged optics and the push beyond copper
46:04 New approaches to optical interconnects
52:24 The AI memory hierarchy
56:39 Useful work per token
57:51 How much context does an agent need
1:03:35 DRAM, HBM, and memory on logic
1:05:54 The next generation of inference chips
1:08:25 AI's biggest engineering bottleneck
Enjoy.