We've open-sourced AgentENV in collaboration with kvcache-ai. AgentENV is a distributed system for running agent environments at scale. Its components power agentic RL training for Kimi K3, with fast snapshot, resume, and fork support for large-scale parallel agent workflows. Explore on GitHub:

We’re excited to welcome Mooncake to the PyTorch Ecosystem! Mooncake is designed to solve the “memory wall” in LLM serving. By integrating Mooncake’s high performance KVCache transfer and storage capabilities with PyTorch native inference engines like SGLang, vLLM, and TensorRT-LLM, it unlocks new levels of throughput and scalability for large language model deployments. Mooncake enables prefill decode disaggregation, global KVCache reuse, elastic expert parallelism, and serves as a fault tolerant PyTorch distributed backend. 🔗 #PyTorch# #OpenSourceAI# #LLM# #AIInfrastructure#

💡 Memory Is Buckling Under KV Cache Demand, Two Fixes Are in Play. Surging KV Cache demand against constrained memory supply created severe bottlenecks. Penguin Solutions, Marvell, and Meta are deploying CXL-based solutions to expand addressable memory capacity, while NVIDIA's KVTC and Google's TurboQuant compress demand from the other side. Get the full analysis 👉 Explore the memory bottleneck breakdown 📖 #TrendForce# #SelectedTopics#

This new Nvidia paper is huge for model routing and the future of inference: re-using the KV cache across different LLMs. One of the greatest challenge in model routing during long-horizon agent settings is that the cache does not transfer across different models. As a result, naively switching models without KV-cache-awareness will result in wasting more money. Conversely, transferring the KV cache across models significantly expands the total amount of savings that can be achieved. This paper is a great step in that direction. Congrats to Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, @MMKamani7, @Ritika_Borkar, Makesh Tarun Chandran, Pantea Zardoshti, and Bita Darvish Rouhani. Two questions: 1. Do you plan to investigate cross-family transfer? I saw this in the future work section but curious to learn more about your intuitions on the opportunities and challenges here. 2. What was the reason for focusing mostly on small to large model transfers and not large to small?

🌐 Microsoft brought 11 papers to NSDI 2026, the frontier of large-scale networked systems. This digest spans concrete advances for AI infrastructure: 4x throughput from KV-cache reuse, 3.2x over RDMA with CXL memory, and more. Title: Microsoft at NSDI 2026: Advances in large-scale networked systems URL: 📝 Overview NSDI is a leading venue for sharing new research on the design and operation of cloud, AI, and distributed application infrastructure. This post highlights Microsoft's 11 accepted papers spanning datacenter networks, wide-area networking, AI systems, and cloud infrastructure. ❓ Challenges Solved As AI workloads surge, scaling datacenter network, memory, and CPU resources while preserving reliability and cost efficiency keeps getting harder. The papers tackle that frontier from both production operation and research angles. 💡 Key Research Contributions ・DroidSpeak reuses KV caches across same-architecture language model variants for up to 4x throughput with minimal impact on output quality ・SONiC DASH SmartSwitch redesigns cloud network offloading and is deployed at scale in Azure, significantly improving power and space efficiency (community award winner) ・Octopus, a CXL disaggregated memory pod, achieves RPCs 3.2x faster than in-rack RDMA and 2.4x faster than CXL switches on three-server prototypes ・HarvestContainers dynamically harvests cores from latency-sensitive containers, using up to 75% of spare CPU while keeping tail latency within 4% of standalone 🎯 Use Cases It informs design decisions for faster LLM serving, memory disaggregation, container resource utilization, and network-protocol verification across cloud and AI infrastructure. 📊 Other Results ・Eywa used LLM-based model generation to find 33 bugs (16 previously unknown) in protocol implementations ・ForestColl generates theoretically optimal collective-communication schedules in polynomial time ・AVA is a video-analytics benchmark reaching 75.8% accuracy on 10+ hour videos ・KRAKENGUARD provides fine-grained eBPF isolation for multi-tenant security #NetworkedSystems# #AIInfrastructure#

The AI memory wall is costing you throughput. Diamond Partner @WEKA fixes it at the infrastructure layer: 1000x KV-cache expansion, 20x faster #TTFT#. See it at SuperAI Singapore, June 10–11.

As an AI Engineer. Please learn: - Harness engineering, not just prompt engineering - Prompt caching vs. semantic caching tradeoffs - KV cache management at scale - Speculative decoding vs quantization - Structured output failures & fallback chains - Evals (LLM-as-judge + human evals) - Cost attribution per feature, not just per model - Agent guardrails & loop budgets - LLM observability as a first-class discipline - Model routing & graceful fallback logic - Knowing when to fine-tune vs. in-context learning

Michael Goin (@mgoin_) walks through @vllm_project v0.20.0. 752 commits. 320 contributors. 123 new. 🚀 🎉 DeepSeek V4, TurboQuant 2-bit KV cache, MXFP4 for MoE on Blackwell, FA4 as MLA prefill default, @PyTorch 2.11 + CUDA 13.0, Transformers V5, and a lot more. ~8 minutes.

Most agentic stacks run into the same problems pretty quickly: reasoning and tool parsing drift across turns, KV cache reuse falls apart, or tools fire too late. We’ve been hardening Dynamo’s harness-facing path so @Claudeai Code, @OpenClaw, and @openai Codex-style agent patterns behave reliably on custom stacks and inference endpoints: • Stable prompts for KV reuse and lower TTFT • Interleaved reasoning + tool calls preserved across turns • Streaming tool dispatch instead of end-of-turn buffering • Harness behavior aligned with real multi-turn agent runtimes If you’re building your own agent stack or serving endpoint, this blog goes through the infrastructure issues that tend to show up in practice and the patterns we’ve been using to fix them. Tech blog ➡️