Prediction ML silicon started off unified (single GPUs/TPUs) Then grew into larger sharded clusters (NVL72s) And is now beginning to grow into more heterogeneous, disaggregated structures. Inference providers are realizing they can get better tok/watt using GPUs for prefill/FFNs and WSE/Raptors for decode/attention layers and CPUs for agent orchestration. I think that this branching will continue as we get independent chips for projections, FFNs, attention, normalizations, tool-use, etc. A dataflow-based rack entirely designed around the interconnect and data-locality to minimize power. I think we will look back in 5 years on the idea of using the same cores for both FFNs and attention layers as absurd. The science of silicon is extremely advanced, but the engineering putting it to use is still in its infancy. Much of this is because we were able to lean on Moores law for so long we could afford to not worry so much about our hardware designs. Now this naivety is beginning to show itself.

One of the main reasons we see hardware leaning towards 'shorter' HBM stacks (i.e. Nvidia dropping Rubin from 12Hi to 8Hi) is because HBM -> SRAM bandwidth is invariant to stack height. Imagine your goal is to get as much water out of an upside-down water bottle as fast as possible. Increasing the radius of the opening would help; making the water bottle taller would not (even though it can hold more water). In decode, models are not memory *footprint* constrained, they are predominantly memory *bandwidth* constrained. Adding more die layers of HBM (going from 8Hi to 12Hi) increases your memory footprint, but does not help your overall bandwidth as all data still must flow through the same number of I/O lanes, same base die, and same interconnect that has a constant signaling rate. Because of this, you actually get diminishing returns by stacking HBM higher and higher. One 8Hi HBM3e die has 24Gbs of capacity at a ~1.2TB/s bandwidth giving you ~50GB/s per GB of bandwidth. Comparing this to a 12Hi HBM3e die with 36GBs of capacity and the same 1.2TB/s of bandwidth, you only get ~33GB/s per GB of bandwidth. So stacking additional dies is not very attractive given the bandwidth is fixed and the fact that DDR offload is becoming an increasingly viable option for less latency-sensitive parts of model memory. This is likely why Nvidia decided to pursue 8Hi HBM for Rubin instead of the initially planned 12/16Hi because performance asymptotes but BOM doesn't (especially as HBM/DRAM prices skyrocket). Shoutout Dylan for the diagrams

No one wants an agent that can book their flight. Everyone wants an agent that can search instagram and compile a list of your Hinge dates ex bfs/gfs The Big Labs are trying to come up with moat after moat with respect to product and none of them are going to stick. Meta and Xai might be the only labs to get actual product sticky-ness because they have social + ads data. Nothing else is going to matter.

These people have literally lost their minds. They are extremely biased towards the reality they want rather than the reality we live in. Agents are slop. The huggingface incident is a nothing-burger being used as propoganda. The obsession with alignment is a weird fetish of human power dynamics the labs and researchers want to play. AI safety/alignment is fake and retarded. Do you know how much damage 1 person could do right now with a drone and dimethyl mercury? But we don't hear drone or chemical manufacturers talking about 'drone alignment' or 'chemicals alignment'. They have read way too much sci fi and/or done too many drugs. I hate sounding so negative, but sans being a better wikipedia and writing shitty code, what is AI doing? I believe in the long run media-gen (image/video models) will actually bring in more revenue than LLMs because that is a closer fit to what the median human wants; entertainment. Too many of these researchers circle-jerk themselves to oblivion thinking the world is as intelligent as they are and want a super-intelligent chatbot. Most people could not care less.

If Claude has 1 million haters, I am one of them If Claude has 5 haters, I am one of them If Claude has 1 hater, that hater is me If Claude has no haters, I am no longer alive If the world is with Claude, I am against the entire world Till my last breath, I will despise Claude