One of the main reasons we see hardware leaning towards 'shorter' HBM stacks (i.e. Nvidia dropping Rubin from 12Hi to 8Hi) is because HBM -> SRAM bandwidth is invariant to stack height. Imagine your goal is to get as much water out of an upside-down water bottle as fast as possible. Increasing the radius of the opening would help; making the water bottle taller would not (even though it can hold more water). In decode, models are not memory *footprint* constrained, they are predominantly memory *bandwidth* constrained. Adding more die layers of HBM (going from 8Hi to 12Hi) increases your memory footprint, but does not help your overall bandwidth as all data still must flow through the same number of I/O lanes, same base die, and same interconnect that has a constant signaling rate. Because of this, you actually get diminishing returns by stacking HBM higher and higher. One 8Hi HBM3e die has 24Gbs of capacity at a ~1.2TB/s bandwidth giving you ~50GB/s per GB of bandwidth. Comparing this to a 12Hi HBM3e die with 36GBs of capacity and the same 1.2TB/s of bandwidth, you only get ~33GB/s per GB of bandwidth. So stacking additional dies is not very attractive given the bandwidth is fixed and the fact that DDR offload is becoming an increasingly viable option for less latency-sensitive parts of model memory. This is likely why Nvidia decided to pursue 8Hi HBM for Rubin instead of the initially planned 12/16Hi because performance asymptotes but BOM doesn't (especially as HBM/DRAM prices skyrocket). Shoutout Dylan for the diagrams