We've open-sourced AgentENV in collaboration with kvcache-ai. AgentENV is a distributed system for running agent environments at scale. Its components power agentic RL training for Kimi K3, with fast snapshot, resume, and fork support for large-scale parallel agent workflows. Explore on GitHub:

MI355X IS UP TO 1.7X BETTER 💰️PERF PER DOLLAR 💰️THAN DGX B300. The AMD Mainland China UMBP team co-designed, in collaboration with Alibaba & the @sgl_project community, a new feature in SGLang that removes the duplicated KVCache contained between local L2 DRAM & distributed L3 DRAM, allowing for up to 2x more KVCache to be stored in DRAM. This feature is called UnifiedRadixCache external cache. But importantly, this marks the trend of AMD increasingly being first-class co-designed for new features in widely used top production engines like SGLang.

For long-context LLM inference, should the KV cache be offloaded to disk or just recomputed on the GPU? There's no universal right answer, and this paper builds a system that decides quantitatively. Title: Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs URL: 📝 Overview The paper proposes py-kvcache, an external KV cache system for vLLM. Using io_uring for async I/O, even a Python implementation pulls near-full SSD read bandwidth of 13.5GB/s. ❗ Problem it solves Existing external KV caches like LMCache have no criteria for when to actually use them — they load from disk unconditionally even when a short prefix or a fast GPU makes recomputation cheaper. ⚙️ Methodology It introduces "scheduler-aware preloading," which starts disk reads while a request is still waiting to be scheduled, plus a "break-even gate" that rejects a load whenever it wouldn't improve time-to-first-token. 📊 Results On LongBench multi-document QA it's 6.02-7.43x faster than GPU recomputation and 2.77-3.64x faster than LMCache. On multi-turn SCBench, native vLLM read 3.4TB from disk with completion times over 1200s, while py-kvcache kept disk reads to 85GB and completion time to 480s. 🖥️ Use cases On a high-end H100, requests often don't even clear the break-even point, so skipping external caching is fine — but on a lower-end RTX 4000 Ada, external caching clearly wins, giving concrete hardware-specific guidance. #LLMInference# #vLLM#

AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing? $3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355, B200

TPU🚨 is working with the popular OSS inference optimization library Mooncake on integrating TPU with Mooncake Store. Similar to NVL72, KVCache DRAM P2P pooling will initially happen on the scale-out network via TENT instead of using ICI/NVLink. 🔥 Mooncake basically improves performance per TCO of production inference!

We’re excited to welcome Mooncake to the PyTorch Ecosystem! Mooncake is designed to solve the “memory wall” in LLM serving. By integrating Mooncake’s high performance KVCache transfer and storage capabilities with PyTorch native inference engines like SGLang, vLLM, and TensorRT-LLM, it unlocks new levels of throughput and scalability for large language model deployments. Mooncake enables prefill decode disaggregation, global KVCache reuse, elastic expert parallelism, and serves as a fault tolerant PyTorch distributed backend. 🔗 #PyTorch# #OpenSourceAI# #LLM# #AIInfrastructure#

Mizuho: Memory, CPO, 800VDC > Memory Bottlenecks & De-spec Risks: Potential significant DRAM de-spec risks are driving investor concern. GPU/ASIC OEMs may cap future HBM content at 4-8Hi, making 12Hi less appealing, while 16Hi/20Hi and hybrid bonding remain uncertain for future DRAM generations. > Impact on Shipments & Demand: GOOGL has scaled down TPUv8 HBM3e to 8Hi. Lower DRAM/GPU content could allow roughly 20-30% more GPU/ASIC shipments, driving higher aggregate DRAM bit demand for training, inference, and KVCache, alongside better margins. > Connectivity and Hardware Upside: Copper, NPO SiPho, and InP show upside for CRDO and LITE, whereas CPO is roughly 2 to 3 years away. A less-constrained memory environment points to upside for GPU/ASIC shipments and rack scale-up/out connectivity. > 800V Ramp Delays: Potential 800V delays are allowing for a smoother ramp. Data center installers are pushing back due to near-term power space changes and the need for personnel retraining, with DC racks moving toward OCP ORv3 Diablo bipolar dual-rail 400V racks. (Via @sean_________) $CRDO $LITE $GOOGL $NVDA $MU $DRAM $EWY

Mizuho on SemiAnalysis (rumor mill) channel checks: "Significant Feynman de-spec risk (potentially ~50%). 16/20Hi HBM + hybrid bonding still highly uncertain. GOOGL already scaled TPUv8 HBM3e to 8Hi." "Lower HBM content per GPU/ASIC = 20-30% more shipments possible. Aggregate DRAM bit demand for training/inference/KVCache actually rises. Better GPU/ASIC margins." "That extra silicon volume is pure upside for copper AEC, NPO, SiPho/InP. $CRDO and $LITE the clearest beneficiaries. CPO still 2-3 years out." "800V looks delayed — data centers pushing back hard on rapid power architecture change + staff retraining. Moving to OCP ORv3 Diablo 400V bipolar instead. More VRMs, smoother ramp." "2027 is still a monster year: xAI/ $SPCX 10GW + the $500B Neocloud financing facility = real tailwind. $GOOGL morphing into a major Neocloud supplier with accelerating TPU ramps. Mizuho modestly cuts PTs ( $MU $1,300 / $SNDK $1,875 / AMAT $590 / MKSI $330 / LRCX $365) on multiple compression from the de-spec uncertainty — but still sees memory tight and overall bit demand higher. Net: de-spec is real, but it unlocks more systems, more copper, more optics, and keeps the 2027 power + financing flywheel intact.

Inference Optimizations Behind the MiMo-V2.5 Series API Price Reductions Read the full technical blog: The V2.5 model family, including MiMo-V2.5 and MiMo-V2.5-Pro, is built on a Hybrid Sliding Window Attention (Hybrid SWA) architecture, which compresses KVCache storage to roughly 1/7 that of Full Attention. However, architectural advantages rarely translate directly into measurable gains in production serving. To realize these gains, we redesigned KVCache management, tiered caching, and the prefix-cache tree; addressed key challenges in SWA KVCache handling; and optimized scheduling as well as the Prefill/Decode pipeline. Validated on real production traffic, these optimizations have increased effective KVCache capacity by nearly 5x, with server-side cache hit rates averaging 93%–95% across mainstream harness frameworks. Together with MoE configuration tuning and multimodal inference optimizations, they enable more efficient long-context inference and form part of what makes the recent API price cuts possible.

🚀 A model that handles a million-token context while shrinking its KV cache to a quarter of the previous size just dropped. DeepSeek's latest. Title: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression URL: 💡 Overview A 552B multimodal MoE model built for input-heavy, long-running agent workloads, aggressively compressing the KV cache through both architecture and precision optimizations. 🎯 The problem it solves For agents working with long contexts, persistent KV cache storage — not just prefill compute — strains GPU memory, SSD, and I/O bandwidth, becoming the main bottleneck to lowering deployment costs. 🛠 Method A Causal Encoder-Decoder structure (first 20 layers as encoder, last 20 as decoder) halves prefill compute. Compressed Sparse Attention 2 reuses cross-layer KV in three modes, a hierarchical sparse indexer narrows search to a candidate pool, and FP4 quantization trims memory further. 📊 Results Global KV footprint drops to 890 bytes per token, about 1/4 of V4-Flash, with persistent KV around 1/8. Extending context 256x from 4K to 1M only increases decode FLOPs by 1/4. Benchmark gains hold too, e.g. 79.4% on HumanEval. #LLM# #KVCache#