NVIDIA is continuing to push throughput, and I think that makes sense. We need more tokens, and we need them cheaper. But what used to feel fast at 100 to 200 tokens per second is quickly becoming the new batch mode. I spoke with @swyx on @latentspacepod about how expectations for inference speed are changing.

What excites me most about Jalapeño is the design methodology behind it. Using AI to help build chips faster and push both throughput and latency forward could benefit the entire industry. I spoke with @swyx on @latentspacepod about why I see this as the future of chip design.

When we started @cerebras, we knew we would have to solve difficult problems in yield, power, and cooling. What became clear was that making wafer-scale work would require rethinking almost the entire system. Today, we answered questions from the community about those engineering decisions, how we run larger models, and what it takes to program the hardware. 2:15 Why hasn't NVIDIA gone wafer-scale? 4:10 How we build a working chip out of an imperfect wafer 8:07 What happens when a model won't fit on one wafer? 11:34 Why we chose SRAM over HBM 14:26 How much of a barrier is NVIDIA's CUDA moat?

Ten years ago, we started @cerebras around an approach many believed was impossible. As a computer architect, it is hard for me to imagine a more exciting time. Model releases are accelerating, and hardware tapeout is compressing from multi-year roadmaps to annual launches. Hot Chips is my favorite conference, and it’s where I launched Cerebras 7 years ago. This year’s conference was especially exciting, and so much innovation was shared. I am watching the industry recreate itself: SRAM is mainstream, DRAM is moving into the third dimension, networks are being fundamentally redesigned, and AI is helping design and program the chips themselves. The industry has never moved faster and some of the hardest architectural questions are still wide open.