One thing I will give the AI safety crowd is they are amazing at marketing these ideas. “Pacing the frontier” “warning shot” and the classic “paper clip maximizer” are all just such catchy phrases. I now get why people become doomers because it’s pretty fucking entertaining to read this stuff 😂😂😂

If you are going to take time to read anything today please read this lol. The Chinese couldn’t have conceived of more effective propaganda against data centers than what these hippies are coming up with on their own lol.

A big reason why people have been under the impression that TPUs or more generally “systolic array” architectures lag behind Nvidia GPUs for inference is because they notoriously struggled with running MoEs at first, while Nvidia’s SMs were an easier fit for small, dynamically shaped per expert matmuls that are a characteristic of MoEs. TPUs prefer static, compile time known in advance shapes, with dimensions that are multiples of the MXU size (128x128 or 256x256), and when you design your model you design with that in mind. You might sacrifice some albeit small amounts of flexibility (other ways around it) but the benefit is much higher op reuse and higher utilization on large GEMMs than GPUs. You also avoid register file access, shared memory access and the control overhead of GPUs because your weights stationary and the same data flow runs every time, you just load it fire and wait for the result. Ofc Google has since fixed MoE routing on TPUs with XLA & custom kernels & they run MoEs fine now. But to do this they basically had to turn the routing into dense GEMMs instead, or pad the matrices to fit the MXU dimensions. Which sounds like it would increase the runtime of complexity but it actually doesn’t cause it’s a matrix multiply machine anyway. So with that all being said, the fact this team can do higher tokens/sec on Kimi doesn’t surprise me at all. The trade off with Kimi is you get a fixed size KV that doesn’t grow with more context but in return you need to do prefill or checkpoint states more often since you lack precise context. Since decode isn’t bottlenecked by streaming KV cache out of memory every step you can do much bigger batches and serve more users. But the bottleneck now is redoing prefill which is just dense GEMMs which ofc as you guessed is good for a systolic array… it’s also why Moonshot in their reference design suggests disaggregating prefill and decode to different nodes. Long post to say nothing most of you didn’t already know. But the real takeaway to me is it’s looking likely we get a great schism where certain architectures that use hybrid attention will run disproportionately better on stuff like TPUs and other systolic array architectures like Tranium and will cost disproportionately less that many people will make the switch. This also leads me to believe that the next Gemini model will likely use a hybrid attention approach and the cost per token and intelligence per token will be much higher.