DeepSeek Opens DeepInfer: Inside V4.1 Flash’s Production Inference System @deepseek_ai just published an engineering deep dive on Zhihu, announcing DeepInfer, the open-source framework behind its large-scale V4.1 Flash service, with support for NVIDIA and Huawei Ascend hardware. DeepSeek reports 155 trillion tokens/day in service throughput from a single cluster. Its production statistics include a 96% input cache hit rate, average time to first token of 400 ms, and generation speeds of 230 tokens/s, reaching 300 tokens/s for coding. The engineering details explain how those numbers fit together. Separate the stages, then balance the actual work Vision encoding, prefill and decode run separately so resources can scale with each stage’s workload. The system targets standard eight-accelerator servers for prefill and 64–128-accelerator supernodes for decode, where wide expert parallelism improves batching efficiency. Prefill scheduling uses an estimated attention-compute cost to allocate work and size chunks within a time budget. Decode routing balances both request count and total KV-cache length. Popular MoE experts receive replicas that are periodically redistributed to reduce waiting between devices. Keep reusable prefixes for days Persistent block KV cache lives in a separate 3FS storage cluster with roughly 100 PiB of SSD capacity. Reported retention reaches three days for agent traffic and seven days for chatbot traffic. Any prefill worker can retrieve any cached prefix, avoiding a conflict between cache affinity and load balancing. Storage and compute can also scale independently. Cache loading is asynchronous, with host-to-device transfers designed to overlap layer execution. Rebuild short-lived attention state cheaply Sliding-window attention state has a different lifecycle: it is reused within active sessions, so DeepSeek keeps it in host memory rather than persistent storage. When that state is missing, SWA Bounded Replay reconstructs an approximation from just the latest 128 tokens, restricting attention to the replay segment while reusing the persistent block cache. Applied to the decoder, this avoids expensive state reconstruction across a much longer token span. DeepSeek says it nearly halves total prefill computation. Spend verification compute where it pays off DSpark predicts draft-token acceptance probabilities and varies verification lengths according to expected throughput. DeepSeek reports a 6% production throughput gain over always verifying five draft tokens. CPU scheduling also overlaps GPU execution to remove idle gaps between decode steps. Release the API layer alongside the engine The accompanying deepseek-recipe library handles Chat Completions, Responses and Messages APIs. It parallelizes tokenization and distinguishes actual control tokens from identical-looking user text, preventing that text from being interpreted as structural instructions. One release boundary matters: the KV-cache management and load-balancing components are not yet fully open-sourced, because they remain closely coupled to DeepSeek’s internal infrastructure.