After reading Huawei's paper, my takeaway: Huawei's innovation is fundamentally about using high design complexity + high manufacturing cost + bleeding-edge thermal solutions, to partially close the process gap Huawei's Tau Scaling Law marketing is just another way of saying More than Moore: the generalized Moore's Law that everyone already knows. What I'm actually interested in is whether Huawei's claimed density improvement brings real power efficiency gains. The 41% power efficiency number on their PPT , how is it actually achieved? --------------------- 1. Is Huawei's "equivalent density improvement" from chip stacking real or misleading? Is it a process breakthrough? Are there tangible benefits? The source of the equivalent density improvement is two pieces of silicon bonded together using hybrid bonding technology. In theory, the projected footprint could be cut in half. But the first generation is not full-chip double-layer folding — it selectively folds critical logic paths. Only about 53% of the chip area is actually folded (density goes from 155 to 238 MTr/mm²). In subsequent generations, folding coverage will gradually increase, approaching full-chip folding by 2030 (density 155→292). The 2026 first-generation equivalent density jumps from 155 MTr/mm² in 2025 to 238 MTr/mm² in 2026. Clock frequency also improves by 12.7%, and power efficiency improves by 41%. On the surface, this looks no different from a process node advancement. But there's one critical difference: Huawei never once mentions leakage power. As long as the process node doesn't change, I_off, gate leakage, and junction leakage won't improve from 3D stacking alone. The density discontinuity from 2030 to 2031 most likely comes from moving from 2-layer to 3-layer stacking, just as the 2025-to-2026 density and clock frequency discontinuity comes from going from single-layer to 2-layer folding. So clearly, the "1.4nm equivalent by 2031" claim has no connection to an actual process node breakthrough. What it really is: using high design complexity + high cost + bleeding-edge thermal management + early deployment of advanced packaging to partially compensate for the process gap. --------------------- So is this seemingly inflated equivalent density improvement actually useful? What are the real benefits? Yes, there are real benefits. The topological folding means that signal paths that previously ran several millimeters horizontally now become tens of micrometers vertically. This shortens super buffers and buses, reduces clock tree depth (clock depth -42%, clock wire -28%), and improves clock skew (-25%). These translate into genuine dynamic power savings. The shortening of critical paths also makes clock frequency increases easier to achieve. So the performance improvement shown on the PPT roadmap — the 12.7% gain from 2025 to 2026 — is almost entirely from clock frequency increase (12.7%). The benefits are fundamentally from topology-driven circuit design improvements. --------------------- Since there's no actual process improvement, what are the trade-off costs of Huawei's chip stacking? Three costs: thermal management gets harder, design complexity goes up, and manufacturing cost increases. The biggest cost is the simultaneous increase in thermal density. In theory, logic-on-logic stacking means the hottest CPU execution areas have their power density effectively doubled. But factoring in the 41% power efficiency improvement, the actual power density is only about 40-50% higher than the non-stacked design. That's why the first generation can only fold the most critical portions — roughly 53% of the chip area. This forces thermal technology to advance ahead of schedule. Huawei is deploying millimeter-scale MEMS fans as micro-cooling solutions. The second cost is design complexity. Which logic blocks can be folded? Once folded, the entire flow from front-end to back-end design has to be reworked. No existing EDA tools support 3D topology. The paper itself admits that full-scale LogicFolding requires an entirely new 3D-native EDA toolchain that treats multi-layer stacked dies as a single continuous design entity. Which logic can be folded, how to perform inter-die timing closure, and physical design (PD) are all significant challenges. Manufacturing cost also goes up significantly, forced into early deployment of advanced packaging. 1.5-2μm hybrid bonding combined with logic-on-logic is extremely challenging and substantially more expensive. Previously, one wafer layer required one lithography pass. Now two wafer layers are lithographed separately and then bonded, with hybrid bonding overlay control (the paper requires <0.5μm), TSVs, KOZ keep-out zones, redundancy and repair, and multiplicative yield losses. Per-chip manufacturing and testing costs increase significantly. --------------------- 2. What is Tau Scaling actually scaling? Is this a one-time design topology dividend? What's the potential? Where does continued improvement come from? The core claim of τ Scaling is: replace geometric linewidth with the time constant τ as the full-stack optimization target, compressing characteristic delay across four levels — device, circuit, chip, and system. The formula itself contains no new physics. "Focus on the bottleneck delay" is what every architect already does. The entire industry knows that interconnect RC is the delay bottleneck. TSMC has been using low-k dielectrics, semi-damascene, and other techniques to reduce RC with every process generation. Packaging a universally known optimization direction as a "law" is clearly a marketing move. It's essentially just another way of saying More than Moore — the generalized Moore's Law. Setting the marketing aside, Huawei's claimed RC delay improvement is fundamentally about topology distances shrinking after chip stacking, which reduces the effective RC along those paths — not the RC process constants themselves. As for what "scaling" means here, it refers to a sustainable roadmap of continued improvement. The path is: increasing the number of stacked layers across the full chip, from 2-layer stacking in 2025-2030, to 3-layer stacking starting around 2031, and potentially 4-layer stacking further out. The first-generation folding technology isn't even full-chip double-layer folding. It selectively folds critical logic, with only about 53% of the chip area folded (density 155→238). In subsequent generations, folding coverage will gradually increase, approaching full-chip folding by 2030 (density 155→292). The reason the 2031 roadmap shows a density discontinuity is precisely because that's the transition point from 2-layer to 3-layer folding. But note that the marginal returns of this scaling approach diminish with each layer. Folding from 1 to 2 layers yields up to 100% density gain. Going from 2 to 3 layers yields only 50%. If they go from 3 to 4 layers around 2035, the yield is only 33%. And as the number of stacked layers increases, all three challenges mentioned above — thermal, design complexity, and cost — get progressively worse. --------------------- 3. Is Huawei's chip stacking the same hybrid bonding technology that TSMC/AMD already have? Is it cache-on-logic, cache-on-cache, or logic-on-logic? How is the thermal problem of logic-on-logic solved? Yes, it uses existing technology. But it's also true that Huawei has pushed certain metrics to industry-leading levels. 3D stacking itself is not new, but TSMC's production hybrid bonding is still at 6μm pitch. Huawei's paper states that Kirin 2026 uses a 1.5μm hybrid bonding pitch. When I first saw the stacking news, my initial reaction was to suspect it was similar to AMD's 3D V-Cache, which primarily stacks SRAM cache on top of the existing L3 cache area and typically avoids stacking directly above the hottest CPU execution logic — precisely to avoid thermal issues. SRAM has different power density and thermal characteristics than high-activity logic. If you stack the hottest logic-on-logic, thermal management becomes extremely difficult. But after seeing more data — clock buffer -56%, clock depth -42%, clock wire -28% — these numbers are only possible if the core's internal clock distribution has been restructured. Pure SRAM stacking wouldn't touch the core's internal clock tree. Furthermore, cache-on-cache alone would likely not require a dedicated MEMS micro-fan for additional cooling. The evidence overwhelmingly points to a logic-on-logic approach. The elegant aspect of Huawei's approach is that logic-on-logic folding doesn't actually double the thermal density, because the topological benefits reduce power consumption by about 30%. This means the thermal density only increases by about 40-50%. And the first generation doesn't fully stack 100% of the hottest execution logic. The paper explicitly says it's "selectively applied along key critical paths." Only about 53% of selected critical paths are stacked, and the granularity may not even be that fine — it could be IP-block-on-IP-block stacking. In that case, the actual thermal density increase might stay within 20%. But as this path continues forward, bleeding-edge thermal solutions become inevitable. Right now it's millimeter-scale MEMS fans placed directly against the processor for high thermal conductivity. Like Huawei's phones in general, the thermal engineering is aggressive and leads the industry. Going forward, they may need to bring HBM7/8's microchannel cooling technology forward in time. After all, HBM7/8 will feature 24+ layer stacking, and Huawei may well need to deploy next-generation thermal solutions ahead of schedule. --------------------- 4. From an architecture perspective, the most important question: How is Huawei's 41% power efficiency improvement actually achieved? Why didn't AMD's 3D V-Cache produce a similar improvement? First, let's pin down what 41% means. The paper only says "SoC performance-core power efficiency improved by 41%," without specifying the benchmark, voltage/frequency operating point, temperature conditions, or power boundary. But there's a key clue on the PPT roadmap: the ISO-Power Performance numbers are 2.75 for 2025 and 3.1 for 2026 — a 12.7% improvement. This matches the clock frequency increase of 12.7% exactly. This can be interpreted as: the performance improvement at constant power is 12.7%, and it's almost entirely from clock frequency. My hypothesis for how the power efficiency improvement works: LogicFolding shortens the critical path → at a fixed Vdd, Fmax increases from 2.75GHz to 3.1GHz → this means at the original 2.75GHz, there's approximately 12.7% timing headroom → this headroom can be traded for lower Vdd in iso-performance mode. Additional power efficiency gains likely come from reduced cache hit latency after circuit folding. From industry experience, a 10% reduction in L2/L3 cache hit latency typically translates to at least a 5% overall CPU performance improvement. The PPT shows SRAM latency reduced by 30%, and some portion of that likely translates into lower cache hit latency. AMD's 3D V-Cache didn't produce a comparable improvement mainly because AMD's underlying logic die wasn't redesigned. The 3D cache latency actually increased rather than decreased — it only added cache capacity. The benefit of increased capacity is less dramatic than the benefit of reduced latency. On the other hand, the reduction in clock skew and shortening of critical paths improve circuit timing, meaning Huawei can use a lower Vdd (I estimate possibly 7-8% lower). Combined with the RC reduction from shorter paths (considering clock buffer -56%, wire -28%, SRAM pJ/bit -24%, a C_eff reduction of 10-15% seems reasonable), plus the overall shrinking of the clock tree, it's entirely possible to achieve a 30% power reduction at iso-performance on certain voltage/frequency operating points. And a 30% power reduction translates to exactly 41% power efficiency improvement. For comparison, Apple and Qualcomm typically see iso-power single-core performance improvements of 10-20% per generation, and iso-performance power reductions of 30-40%. This is determined by the shape of the V/F curve. So empirically, the numbers check out. The power efficiency improvement, based on the available data, can be derived from topology changes and appears to be plausible — it may genuinely have little to do with the process node. --------------------- 5. Is this technology path replicable? Will others follow? In the short term, no one will replicate this at scale, because the risk-reward ratio doesn't justify it. In the long term, everyone is heading in this direction — just under different names. Huawei's fundamental motivation for LogicFolding is the sanctions. With the process node capped at 7nm, the only option is to compensate through packaging and design. Huawei has paid a significant price for this: thermal engineering costs, design complexity, and higher manufacturing costs (including yield). This is a path born of necessity, not natural choice. Other players who can access TSMC can achieve normal economic iteration without taking on the risk of prematurely pushing thermal technology and design complexity forward. In the long term, Intel's Foveros, TSMC's SoIC, and AMD's MI300 3D stacking are all heading in the same direction. If the economics of chasing the most advanced nodes continue to deteriorate, then "fix a mature node + 3D topology optimization" will become increasingly attractive. On the thermal front, MEMS micro-fans and microchannels are likely to become mainstream for future HBM cooling as well. --------------------- In summary, Huawei's innovation here deserves genuine respect. Under sanctions, they've used extreme design complexity and cost to boldly redesign on a locked process node, extracting a significant one-time topology dividend — though it has a ceiling. The marginal returns diminish with each additional layer (1→2, 2→3, 3→4 layers yield progressively smaller percentage gains). Leakage remains unsolved. Thermal management gets harder. The 3D EDA toolchain is an entirely new challenge. Tau Scaling is not a path that sustains exponential growth for a decade. Each step up the staircase is harder to climb, and each step is shorter than the last. If Huawei wants to continue closing the gap, they'll need to find other routes beyond this one.

People always ask: where's the next structural opportunity in AI chips? One of the structural shifts driven by the paradigm change over the past few months is heterogeneous AI inference — and SRAM-route startups led by Cerebras is right at the frontier of this new trend. Every year, Nvidia's GTC conference introduces paradigm-shifting concepts in technology, setting the benchmark for the entire industry. Everyone scrambles to rewrite their roadmaps and copy the homework after GTC. ------------------------ To understand where SRAM-route companies fit in the ecosystem, you first need to look at the workload characteristics of different stages of genAI inference. It breaks down into three parts: Prefill: extremely high compute intensity, low demand on memory bandwidth, moderate-to-high demand on memory size. Decode-stage attention: moderate compute intensity, extremely high demand on memory bandwidth (repeated reads/writes to KV cache), extremely high demand on memory size, because KV cache grows linearly as batch size increases. Decode-stage FFN: moderate compute intensity, extremely high demand on memory bandwidth (repeated reads of model weights), moderate-to-high demand on memory size (model weights). The characteristics of SRAM-route chips are equally clear: They push memory bandwidth to the absolute extreme, but everything else is a severe weakness. They fundamentally trade off compute intensity and the inability to scale memory size for the ultimate in memory bandwidth speed. --------------------- Now let's look at SRAM's suitability across the three stages of AI inference: Prefill: SRAM can't achieve high compute intensity because SRAM takes up too much die area, leaving limited space for compute units. So prefill is a weak point. Decode-stage attention: SRAM can meet the high memory bandwidth requirement, but its memory size is too small to handle the batch size demands. So SRAM only satisfies half the requirements for attention. Decode-stage FFN: SRAM can meet the high memory bandwidth requirement, and the memory size requirement is moderate. Through optimized interconnect communication, SRAM chips can barely solve the memory size problem — the cost is steep, but the ROI can still pencil out in certain scenarios. ---------- So the applicable scope of SRAM-route accelerators in heterogeneous AI inference is crystal clear: Prefill — forget about it. Garbage performance, garbage economics. Decode-stage FFN — with enough effort and added cost, it's within reach. Decode-stage attention — KV cache demands on memory size are too extreme, and the cost of batch processing is prohibitively high. Letting Cerebras's $2.3 million-per-chip, 45-chip, $100-million luxury system serve as an exclusive ultra-VIP service — sure, that technically works. Imagine this: one or two users running agent flow on a coding task with 1–2M context length, and it takes the entire 44GB of SRAM on a single $2.3M Cerebras chip just for KV cache, otherwise the speed tanks. What kind of extravagant service is that? ------------- So the conclusion couldn't be more obvious: if Cerebras tries to do full-stack AI inference on its own (prefill + decode ATTN + decode FFN), the economics simply don't work. There's no future in it. Because the cost of Cerebras is staggering. Even with their gross margins squeezed razor-thin, the implied rental rate of each CS-3 system is still $41.96/hour — roughly ten times the rental of a B200. And that's for a single CS-3. You need to chain many of them together for LLM inference, so multiply that rental by a lot more. This is precisely why the economics of the SRAM route are so poor — Nvidia already made this point crystal clear at GTC (see chart). People hyping up SRAM as the future replacement for HBM? That's a pipe dream. With SRAM scaling already hitting a wall, SRAM density per generation of chips is already nearly impossible to improve. On the memory size dimension, HBM's exponential growth will only widen the gap with SRAM further. Even on the memory bandwidth dimension, HBM is growing exponentially too, narrowing the gap with SRAM. So Nvidia's solution is elegant and clean: hand off the decode-stage FFN to the SRAM route, keep everything else on traditional HBM GPUs, and push the entire Pareto frontier significantly toward the upper right. Rubin + LPX breaks through 1000 tokens/s at peak speed while still maintaining enough overall throughput to generate real commercial value (this is critically important). Remember — on Blackwell, if you wanted to hit 400–500 tokens/s at high speed, you could only process a tiny handful of concurrent requests. That's a massive waste of GPU resources. But now, even at 1000 tokens/s, you can still maintain a meaningful batch size (throughput), and it can finally generate commercial value. The chart shows that at 400 tokens/s, Rubin + LPX delivers a 35x improvement in throughput — classic token economics. At that token speed, it represents a 35x improvement in commercial value over Blackwell. --------------------- After GTC revealed this standard answer — and even earlier, after the Groq LPU acquisition — everyone has already started copying this homework on the heterogeneous inference front. Google's TPU tapped Marvell for the SRAM component. Amazon AWS's Trainium tapped Cerebras for the SRAM component. ByteDance's AI ASIC tapped Qualcomm for the SRAM component. We will absolutely see more announcements like these in the future. And this is the best path to economic sustainability for Cerebras: stop trying to muscle through full-stack AI inference, focus on what you're good at, partner with mainstream AI ASICs on inference, and fight to embed your SRAM chips into other companies' decode FFN pipelines. This is also why the key to Cerebras's long-term trajectory hinges on how deeply it can integrate with AWS Trainium's disaggregated inference. If it's just what's been reported so far — Trainium handles prefill and Cerebras handles decode as a simple split — the technical implementation difficulty is much lower, but the economics still don't pencil out. It can only be a strategic positioning play — enough for a slice of the market, but not enough to generate real scaled competitive advantage. To follow the Nvidia playbook, they need deep integration of both companies' strengths — and that will take real time and technical effort, with non-trivial difficulty, but the payoff would be worth it. Solution one: Trainium handles prefill and decode attention, Cerebras handles decode FFN. Solution two: Cerebras runs the draft model, Trainium runs verification. Either solution delivers dramatically more market competitiveness. -------------------- Does this kind of partnership with mainstream AI ASICs shrink the TAM for SRAM-route companies? No. This is the only long-term sustainable path for SRAM-route companies to grow their market. Heterogeneous AI inference is unequivocally the future. Finding your piece of the puzzle early in this growing blueprint is the only way to grow alongside the market. The moment an SRAM-route company embeds itself into any mainstream AI ASIC's heterogeneous inference pipeline, its valuation will skyrocket — because the TAM in terms of unit shipments isn't even in the same order of magnitude. Otherwise, heterogeneous AI inference will relentlessly erode the speed advantage of the SRAM route on the token speed dimension (not throughput). Full-stack inference on the SRAM route degenerating into an expensive toy is an inevitable outcome.

AI Semiconductor Endgame 2026 (Part 1) New Token Economics Computing Paradigm Shifts from GPU Compute to HBM This article starts from the essence of GPU architectural evolution to address a question the market has long worried about: Why must each GPU's HBM memory demand grow exponentially, and why won't this exponential growth in HBM demand stall? It then derives the first principle of token economics under the current architecture: token throughput = HBM size × HBM BW (bandwidth) It also discusses why the GPU ceiling is determined by HBM's two dimensions of progress. The topic of HBM cyclicality has long been controversial. Optimists argue that AI-driven demand is much greater than before, but the market mainstream still believes that previous up-cycles also saw 20%+ annual demand growth — so what's different this time? AI doesn't change the fact that HBM, like traditional DRAM, has commodity attributes. Once capacity expansion at the demand peak meets a downturn, history will repeat itself. We can take the perspective of compute-chip architecture, start from first principles, and unpack and reason through this question: why this time is genuinely different. ——————————————————————————————— History: The Era of CPU Compute For a very long time, we lived in the era of CPU-dominated compute. The CPU's top-level KPI was performance — running faster — and so each generation of CPUs deployed every method imaginable to push benchmark scores higher. First it was rising clock frequencies, then it was architectural evolution: superscalar designs, and so on. During this period, why didn't DDR need to advance technologically at high speed? DDR3 to DDR5 took a full 15 years. Because in this era, DDR's role was purely auxiliary — and only weakly so. By industry experience, even doubling DDR speed would generally only raise CPU performance by less than 20%. Why did improvements in DDR bandwidth and speed matter so little? Two reasons: 1. CPUs designed all kinds of architectural tricks to hide DDR latency — superscalar designs, wider issue widths, massive ROBs and register renaming to extract parallelism and hide latency, L1 caches, L2 caches — all of which weakened the demand for DDR bandwidth and speed. 2. CPU workloads don't have particularly demanding bandwidth requirements. For most everyday workloads — say, opening a webpage — DDR bandwidth is severely overprovisioned. Even cloud workloads often look the same. In other words, in the CPU era, DDR bandwidth and speed didn't really matter. There was virtually no difference between DDR4 and DDR5 except in a handful of games — and even the JEDEC standard advanced slowly. On top of that, only a small portion of any given app needs to permanently sit in DDR. Whatever is needed can be paged in from the hard drive on demand. App size grew slowly, and so DDR capacity demand grew slowly as well. That's why, over the past decade, the average PC went from 7–8GB of DDR to about 23GB — only 3× growth in ten years. This slow upgrade pace directly affected revenue. Capacity-based pricing was the main way of making money; speed improvements were just a technological upgrade that raised the unit price of capacity. With both of these dimensions advancing slowly, growth could only come from increases in PC/phone unit volumes. So along both dimensions — bandwidth/speed and capacity — DRAM was always a “nice-to-have” appendage to the chip industry. The marginal utility of DDR upgrades was very low, and almost completely disconnected from the CPU era's top-level KPI. ——————————————————————————————— The Paradigm Shift: GenAI's Top-Level KPI When we entered the era of GenAI large models, the computing paradigm shifted, and the top-level KPI changed fundamentally. By the time GPUs evolved into AI inference engines, the top-level KPI was no longer compute alone (TOPS/FLOPS), as it had been for CPUs — it became the cost of a token. Specifically: overall token throughput per unit cost / per unit power. A close second is token throughput speed — because in the agent era, many tasks have become serial, and token output speed has become a critical bottleneck for user experience. This is exactly why Jensen invented the concept of the AI factory: to produce the most tokens at the lowest cost, while pushing token throughput speed as high as possible. In the AI training era, Jensen's economics were TCO (Total Cost of Ownership): the more GPUs you buy, the more you save. In the inference era, Jensen's token economics flip the logic: AI inference has very healthy gross margins, so the logic now becomes: the NVIDIA GPU is the GPU that produces the cheapest token in the world, so the more you buy, the more you earn. The top-level KPI has become a Pareto frontier: along the two dimensions of token throughput and token speed, optimize as far as possible. Each generation of NVIDIA's token factory is essentially pushing the entire Pareto frontier up and to the right. This is the most important KPI of the AI inference era. ——————————————————————————————— From Token Throughput to HBM: The Core Logic Chain Below is the most important logical chain of this article: how to start from the exponential growth of token throughput and derive that the ceiling bottleneck lies in the exponential growth of HBM size and HBM speed. In the era of single-GPU inference with single-thread batch size = 1, token throughput had only one dimension: HBM bandwidth speed. Higher bandwidth = higher token throughput. But once we entered the NVL72 era, inference is no longer single-GPU. It is a system-level token factory composed of 72 GPUs + 36 CPUs, designed to fully saturate HBM bandwidth and compute simultaneously, in pursuit of the ultimate token throughput. Token throughput growth depends on two things: the number of requests batched simultaneously × the average token speed per request. That is: batch size × token speed. Take Rubin NVL72 as an example. At an average token speed of 100 tokens/s, processing 1,920 simultaneous requests yields a token throughput of 192,000 tokens/s. A Rubin NVL72 draws roughly 120kW (0.12MW), so per MW it can handle 1.6M tokens/s. So we need to find ways to push both parameters up: batch size and average token speed. Their product is our top-level KPI — token throughput. Parameter 1: Batch growth — bottleneck is HBM size Every request in the batch carries its own KV cache, which has to live in HBM, with sizes ranging from a few GB to tens of GB. Because hot KV cache must be read at high frequency and high speed at any moment, it must reside in HBM. For a model with, say, 80 layers, every token generation step requires reading the KV cache 80 times from HBM. As batch size grows, hot KV cache grows linearly. And because the hot KV cache for every request in the batch must sit in HBM, HBM size must grow linearly with batch size. Like an airport shuttle bus: the gate wants to move passengers to the plane as fast as possible. If HBM size is small, the shuttle is small, so you have to make extra trips. Conclusion: batch size growth bottlenecks on HBM size growth. Parameter 2: Average token speed per request — bottleneck is HBM bandwidth The decode-phase speed of a large model bottlenecks on HBM bandwidth, because every token generated requires reading the activated weights and KV cache many times over. The emergence of LPUs has, in cases where batch size isn't very large, moved the activated weights portion onto SRAM — but every generated token still requires many reads of the KV cache from HBM. The higher the HBM bandwidth, the faster each token is generated, in essentially linear correspondence. Like the airport shuttle bus: HBM bandwidth is like the width of the door — wider doors mean passengers board faster. The rest of the GPU's configuration is essentially adapted to support batch growth and to keep token compute speed in step with HBM growth. In some cases the GPU even spends excess compute to recover effective bandwidth (e.g., bandwidth compression techniques). —------- To return to the shuttle bus analogy: • Shuttle bus cabin size = HBM Size (capacity): determines how many passengers can fit at once (i.e., how many requests' KV caches can sit in HBM simultaneously). Bigger cabin = more passengers (higher batch size) per trip. If the bus is too small, moving 100 people takes two trips — and total throughput suffers. • Shuttle bus door width = HBM Bandwidth: determines how fast passengers get on and off. A wide door, and everyone piles on at once (decode/token generation is fast). A narrow door, and even with a giant cabin, people queue up and most of the time is spent boarding. • Passenger throughput = cabin size × door-width-determined boarding speed. —------- At this point, we've logically derived the first principle of token-economics hardware demand: Token throughput = HBM size × HBM Bandwidth The top-level KPI of the AI inference era is highly dependent on progress along both HBM dimensions. If we want to maintain 2× token throughput growth per generation, that means each generation of single GPU must grow HBM size × HBM BW speed by 2×! This is the first time in history that HBM memory size can influence the top-level KPI — token throughput. To validate this thesis, we can put NVIDIA's token throughput from A100 to Rubin Ultra on the same chart as HBM size × HBM BW speed. What you find is that the two curves track each other startlingly closely on log axes. HBM size × speed actually grows even faster than token throughput — which makes sense, because HBM defines the ceiling, and in practice utilization of that ceiling is very hard to push to 100%. Even if HBM size × HBM speed grew by 1,000×, with the supporting compute and architecture, it would be very hard to wring out the full 1,000× of headroom. This curve isn't a coincidence — it's the necessary solution of system optimization. throughput = batch × speed. This is the unavoidable first principle of token factory economics. —------- What about software? Won't software optimization reduce bandwidth demand? Reduce HBM demand? This is an independent dimension from hardware. It's like asking: if software on a CPU runs faster after optimization, does that mean the CPU doesn't need to advance for ten years? After all, software is faster now. If that were the case, would CPU vendors still make money? For a CPU vendor to survive, there's only one path: in standardized benchmarks, ignoring software optimization, every new CPU generation must score higher — otherwise it doesn't sell. GPUs are exactly the same. How well software is optimized, and the requirement that the GPU's own token-throughput KPI must improve dramatically every year, are two separate things. As long as token demand keeps growing, the pursuit of higher token throughput will not stop — and so neither will the pursuit of higher HBM size × HBM speed. If HBM size and HBM speed were to slow down, Jensen would personally fly to the Big Three and pressure them to accelerate, because that ishis GPU ceiling. If the ceiling stops rising, can his GPU still sell? Of course, NVIDIA also needs to wrack its brains to extract performance beyond the HBM ceiling through heterogeneous architectural angles. The LPU is a great example — it improved the Pareto frontier substantially from a different angle (the right-hand high-token-speed portion). —-------------------- HBM memory has now bid farewell to that old era of drifting with the tide. On this one-way road paved by exponential demand, it has, in something close to a destined fashion, walked onto the central stage of the industry's epic. When the inference paradigm's first principles evolve to this point, as long as Jensen still wants to sell GPUs, HBM must double — and it must double every generation. This is endogenous pressure from the supply side. It has nothing to do with AI demand, nothing to do with macro cycles, and nothing to do with the moods of the hyperscalers. The only remaining question is this: When demand has been physically locked into exponential growth, will the three players on the supply side — like they have for the past thirty years — once again drag themselves back into the mire of the cycle by their own hands?