Workers KV Instant delivers sub-2ms p99 read latencies and 250ms global replication across Cloudflare’s 300+ edge locations. KV Instant eliminates cold-read penalties and uses the familiar Workers KV API. #BirthdayWeek#

Model complexity and the KV cache are two primary drivers for memory, especially for inference requiring increasingly large context windows. Context windows for OpenAI’s leading models have risen 230-260X in three years to 1.05M tokens, while Meta’s Llama 4 Scout is 10X higher at 10M. $MSFT $AMZN $META $GOOG $MU

Agentic inference is making KV cache management and data movement increasingly central to the serving stack. The new AgentX / InferenceXv3 analysis from @SemiAnalysis_ captures this shift well. Glad to see Mooncake highlighted for external KV caching and disaggregated P/D data transfer, including support for hybrid-attention cache management and preserving speculative decoding state. Special thanks to AMD and SemiAnalysis for providing the CI resources and technical support that made Mooncake ROCm wheel packaging possible. We’re excited to ship it in our next release. Read more:

UBS Hot Chips 2026- Model size + KV cache keep going up. Roadmaps are now capacity, bandwidth, and scale-up — not peak FLOPS. $GOOG TPU 8i (inference, 2-die) is a memory-bandwidth chip: 384MB on-die SRAM, 288GB HBM3E, 19.2Tb/s inter-chip. Built because reasoning / MoE / agentic decode is now more memory-bound than training. TPU 8t ( $AVGO, training): 9,600 chips/Superpod, 121 EF FP4, 2PB shared HBM. Different product. Training and inference silicon have diverged enough to justify two architectures. Same conference, same bottleneck everywhere else: Samsung LPDDR5X-PIM: MAC inside the DRAM banks. 614GB/s PIM BW (8x LPDDR5X), ~3x Llama-3.1-8B tokens. Cheap inference vs HBM. CXL pooling (Samsung demo): 3.35x LLM decode at 100K context. d-Matrix 3D-DRAM: META engineer on stage. 32GB/card, ~100TB/s, 1M-token context in a 72-card rack. LPX (Groq/$NVDA): no HBM at all. SRAM-only decode. LPX + Rubin ~3-5x on long-context agentic. $CBRS CS-6: 3D DRAM on wafer-scale SRAM. ~2028. Net: TPU 8i is Google saying inference is a memory-movement problem. PIM / CXL / 3D-DRAM / SRAM decode are everyone else saying the same thing. AVGO still has the training Superpod. The duration is the memory stack sitting under both.