🧠 Hot Chips 2026: AI's Memory Wall Is Becoming a Money and Packaging Wall Hot Chips 2026 is still producing the usual charts of larger chips and higher peak compute. But this year, memory vendors are becoming central to the AI hardware story. The reason is simple: adding more arithmetic is getting easier. Keeping those units fed is not. Zhihu contributor 乱序摸鱼 reviewed the conference's memory talks and traced a larger shift across Samsung, Micron, SK hynix, and emerging architectures such as HBF. His main takeaway: the boundary between compute and memory is starting to dissolve. 1️⃣ The Memory Wall has become a Money Wall AI is not merely consuming more memory. It is pushing the DRAM industry toward a far more silicon-intensive product. For the same capacity, HBM requires roughly 3× the DRAM die area of DDR. A wafer redirected to HBM can therefore deliver fewer total bits, even before packaging and yield are considered. At the same time, new DRAM capacity can take more than two years to become productive. AI accelerator demand changes much faster. The article cites a revealing mismatch: AI compute can grow around 3× every two years, while HBM bandwidth grows by less than 2× over the same period. HBM solves the bandwidth problem through massive parallelism: wider interfaces, more channels, more banks, TSVs, and advanced packaging. But the bandwidth is not free. Its cost appears elsewhere as silicon area, power, packaging complexity, and yield risk. 2️⃣ The HBM base die is turning into a SoC Traditional HBM had a clean division of labor. The upper DRAM dies stored data, while the base die handled interfaces, testing, and basic routing. HBM4 changes the economics. Once the base die moves to an advanced logic process, using it as an expensive wiring layer becomes increasingly difficult to justify. Samsung's roadmap therefore expands its role in stages. The shorter-distance PHY comes first. Then the memory controller moves down from the xPU, followed by telemetry, repair, testing, RAS, and potentially lightweight processing. This turns HBM from a passive device into something that understands its own banks, temperature, failure state, and access scheduling. But integration creates new tradeoffs. A shorter interface may reduce energy per bit while concentrating the same throughput into a smaller area, raising local power density and creating thermal hotspots. The base die is becoming smarter, but every new function brings more RTL, validation, thermal design, and software work. 3️⃣ Near-memory compute should be measured in bits avoided Putting compute near memory sounds attractive, but available silicon is not enough to justify doing it. The more useful question is: How much data movement disappears after this operator moves closer to memory? Filtering, compression, copying, search, and simple reductions can make sense near memory. They may scan a large dataset and return only a small result, eliminating enormous amounts of traffic. Dense matrix multiplication is different. Data moved into a GPU may be reused many times by Tensor Cores. Moving that computation into a constrained base die could create a weaker accelerator while adding a new compiler and runtime target. The author's rule is practical: move operations that significantly reduce data volume, not operations that merely fit into spare logic area. Samsung's zHBM takes this idea further by vertically stacking HBM above the xPU. The modeled design removes much of the millimeter-scale horizontal path between memory and compute. Samsung's public targets suggest around 70% lower I/O power and roughly 100 W saved in a modeled 1,200 W GPU system. Those are architectural targets rather than proven production results. The shorter electrical path also introduces harder problems in bonding yield, thermal design, power delivery, repair, and cross-die ownership. 4️⃣ HBF could add a cold tier, but software must make it work High Bandwidth Flash proposes another route: use stacked NAND to provide much more capacity at a lower cost than HBM. The tradeoff is severe. HBF may offer cheap capacity, but its bandwidth per gigabyte is roughly an order of magnitude lower. That makes it unsuitable as a drop-in replacement for HBM. Its strongest use cases are large datasets with consistently low access rates: 🔹 Cold MoE experts 🔹 Sparse KV cache 🔹 Prefix cache 🔹 Model state that must remain nearby but is not read every token Even MoE is not automatically a good fit. A single token activates few experts, but a larger batch combines expert requests from many users. The supposedly cold expert pool can become hot surprisingly quickly. The harder problem is system software. An HBM-HBF hierarchy needs placement, allocation, prefetching, request coalescing, cache policy, wear management, and runtime telemetry. The article's verdict is cautious: the architectural need is real, but HBF's path from an attractive model to a dependable product remains largely unproven. 5️⃣ HBM scaling is becoming a packaging problem SK hynix's presentation shows why adding more DRAM layers is no longer a simple capacity upgrade. Moving from 12Hi to 16Hi increases the number of layers by 33%, while the package height rises from roughly 720 μm to 775 μm, an increase of less than 8%. The remaining option is to compress everything. Dies become thinner, inter-die gaps narrower, and bump pitches denser. Warpage, underfill, thermal resistance, and bonding yield all become harder to control. Yield is also cumulative. A defect introduced early may only appear after many expensive processing and stacking steps have already been completed. For 20Hi and beyond, hybrid bonding becomes less of a futuristic option and more of a practical necessity. By removing conventional bumps and thick underfill gaps, hybrid bonding can reduce interconnect pitch and thermal resistance. It can also return part of the height budget to the DRAM itself. SK hynix estimates that, within the same overall stack height, core dies could be up to 24% thicker than with MR-MUF while using an interconnect pitch below 18 μm. That extra silicon thickness matters for mechanical strength, wafer handling, warpage, and manufacturability. 6️⃣ The real boundary being redesigned The most important Hot Chips 2026 memory story is not another increase in TB/s. It is the rising cost of the physical distance between data and compute. As arithmetic moves from FP16 to FP8 and FP4, chips can contain more MAC units than the system can consistently feed. Performance is increasingly lost to memory access, synchronization, interconnect power, thermal limits, and data placement. The optimization target is therefore expanding from one chip to the entire task path. HBM base dies are becoming logic devices. Memory controllers are moving closer to DRAM. Flash may become another managed AI memory tier. Compute and memory may eventually be vertically integrated. None of this eliminates complexity. It decides where that complexity should live so the whole system becomes more efficient. The author's final insight is a good one: Sometimes the best architecture does not make the road faster. It discovers that the trip never needed to happen. 🔗 Full analysis: #HotChips2026# #HBM# #AIInfrastructure# #Semiconductors# #MemorySystems# #AdvancedPackaging# #AIHardware#

lots of coding agent memory systems are unhelpful - at best i have said this a lot and some folks think I'm against agent memory in entirety which is not true - I just think that users should be on the loop that generates memories and should be able to enable / disable the system or individual memories at will and should be able to dismiss disruptive or unhelpful memories putting a user on the loop also lets you build a better data flywheel over time to improve the quality of recommendations live in @humanlayer_dev btw lmk if you want beta access

🧠 Code keeps changing, but wiki docs don't update themselves. Here's a memory system that automatically detects and self-corrects stale documentation. Title: Building Self-Correcting Memory in OpenWiki URL: OpenWiki links every claim in its wiki to the source code evidence behind it, then detects and repairs staleness whenever that code changes. Three highlights stand out. 📎 Claim-level evidence binding Every wiki assertion is stored explicitly as a "statement" paired with its "evidence" — the exact source location backing it up. Claims and their proof always travel together. 🔍 Deterministic staleness detection The runtime compares the evidence version a claim was written against with the current source version. Any mismatch flags the claim as stale — no model call needed. Stale doesn't mean wrong, just "needs reverification." 📈 Measured self-correction Across 2,000 evaluated claims, stale claims dropped from 80 to 9 (about 89% reduction) and hallucinated claims went from 15 to 0. At one checkpoint, a 17% stale rate recovered to 0% by the next update cycle. Memory systems that can say what they believe and why they believe it. Reframing forgetting as reassessing trust, not deletion, is the key idea here. #AIAgents# #KnowledgeManagement#

last saturday @vaibcode and I got together 50 of the best AI engineers, systems engineers, founders and founding engineers in the bay to chat about the frontier of software and AI. Some recurring topics: - @swyx and @vaibcode debating the future of git - sophisticated memory systems - now that you have all the context in the world, how do you organize it? What's the next generation of karpathy's LLM wiki? - Software factories, broad spectrum of "read the code" vs. "try to turn the lights off" and everything in between. One thing that I think the best builders are coming around on - for real production systems, you either have to read and police code quality, or you have to accept that your system will have slop in it. lots of great takes in both directions - what are the primitives of harness engineering and orchestration? Can we decompose the custom workflow problem into reusable building blocks? - what do we need to build great harnesses for non-coding tasks like hardware design, GTM analytics, and more, and how do you do this at massive scale (hundreds of GB/day ingested into embeddings in the context lake!?) and so much more - thanks to all our speakers, audience members and to our hosts Y Combinator for making this such an incredible event

AI can now remember, update, and forget facts without any external retrieval. A new paradigm for native memory in foundation models has just been published. Title: Metis: Memory Foundation Model 🔍 Overview Current external memory systems like RAG face three fundamental limitations: decoupling from the model backbone, inability to propagate gradients through discrete memory operations, and added inference latency. Metis integrates memory natively into Transformer blocks — analogous to how Chain-of-Thought became intrinsic to LLMs — eliminating external module dependency while enabling learned, end-to-end memory behavior. 🛠 Problem and Approach Two core components are introduced. The Local Memory Block maintains a dense memory network updated across inference steps via exponential moving average, with learned importance scoring and Top-ρ token selection. The Hyper Memory Block uses static learned parameters to enable memory transformation through the forward pass. Four memory operations — Remember, Update, Forget, and Reflect — are executed purely through forward computation without any background gradient updates. 📊 Experimental Results On the MemOps benchmark (no-context setting), Metis-27B achieves 24.76% average performance: · Baseline Qwen3.5-27B (no context): 1.69% · Test-time training Temp-LoRA-27B: 9.70% · Parametric memory δ-Mem: 4.38% On the Metis internal test set, the model reaches 73.77% average with Reflect (multi-hop reasoning) at 93.44%. It outperforms all no-context baselines across model sizes and tasks. 💡 Practical Significance Native memory avoids the retrieval, ranking, and prefilling overhead of RAG through parallelizable computation, keeping inference latency low. Domain adaptation via post-training is supported. Model checkpoints and code are publicly available on GitHub (MemTensor/Metis) and HuggingFace. #LLM# #AIAgent#

A single agentic task can burn around 100k tokens against maybe 1k for a simple prompt, and most of that gap is not new reasoning. It's the agent re-reading the same context every step because nothing persists between them. Fix memory and a big share of that token bill goes away before you ever reach for a bigger GPU. This is something we've been thinking a lot about as we build out persistent memory systems for our production agents.

"early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art. A detailed technical report on performance will be presented in the coming months. The architecture reduces data movement and balances compute, memory, and networking resources to achieve realized utilization much closer to theoretical peak performance. Jalapeño is a blank-slate design for modern LLM inference, not a general-purpose accelerator adapted from earlier AI workloads. It is informed by the systems OpenAI runs every day across ChatGPT, Codex, the API, and future agentic products, while also being designed for current and future LLMs across the industry. The goal is to combine the power and throughput of today’s leading AI accelerators with latency closer to the fastest specialized inference systems, making Jalapeño well suited for interactive LLM products at scale. That is the full-stack advantage. OpenAI is not only developing frontier models or building products on top of them; it is designing the infrastructure underneath them: chip architecture, kernels, memory systems, networking, scheduling, deployment systems, and product experience. Because OpenAI operates across the stack, each layer can be optimized around the same goal: making its models faster, more reliable, and more affordable for users."

Games have been an important testbed for our AI research for over 15 years. 🎮 From mastering Atari to reaching Grandmaster in StarCraft II, they have driven some of our biggest AI breakthroughs. Our work with SIMA taught agents how to understand 3D worlds, but learning to navigate real human dynamics takes a living, persistent universe. Through our research partnership with @FenrisCreations, we’re exploring how to tackle open challenges in AI: 🔵 Continual learning to acquire new skills without forgetting past knowledge. 🔵 Deep memory systems that store and retrieve information far beyond today’s context windows. 🔵 Long-horizon planning over weeks, months, or years. 🔵 Multi-agent dynamics spanning cooperation, negotiation, economics, and emergent behaviors. Our long-term goal is to use AI to discover entirely new gameplay experiences in partnership with game developers – making games more accessible and personalized – while applying what we've learned to problems in the real world and scientific discovery. Find out more →

One of our engineers got back hours of his week with a Littlebird routine that did boring financial research for him. I just became Littlebird's head of growth, so I needed to see what he was doing. I didn't believe what he'd rigged up and uh, how much money he made with it, so he showed me. There was nothing sophisticated. No model, no magic. He noticed a boring pattern: NVIDIA tended to hold after a new all-time high, then fade. A pattern like that only matters if you're glued to four broker tabs every morning for weeks, waiting for the one day it moves. Exhausting. So he stopped watching. He built a Littlebird routine to run the same research every morning. Momentum, insider activity, where institutional money was rotating. Delivered to him before coffee, no tabs, no babysitting. Most days, it said hold. Some days, it flagged an activity worth action on. That's the point of Littlebird. It does the assistant stuff every other tools promises but doesn't deliver on. The grunt work runs itself. A flooded inbox becomes a to-done list. A morning of manual research becomes a brief that's already waiting. You can chain routines like this together in Claude Code too, but Claude can't get your screen context and make suggestions based on our proprietary memory systems. One routine. One bored engineer. We're dying to see what you'd build. RT with "birdbrain" and I'll DM the first 100 people $20 of Littlebird credit. ***Littlebird is a productivity and automation tool. It does not provide investment, legal, or tax advice, and nothing here is a recommendation to buy or sell any security. Any results described are one person's experience and are not typical or guaranteed. Trading and investing carry risk, including loss of principal. Do your own research or consult a licensed professional.