deepseek做了一个创新,
将键值对暂存在解码 GPU 服务器DRAM内存
主要结果如下:
离线服务吞吐量最高可提升 1.87 倍
在线服务的并发性能平均提高了 1.96 倍
cool idea from DeepSeek in their DualPath paper!
instead of loading all KV's directly onto GPUs from local NVMe (or DRAM) and bottlenecking on the local PCIe bus, they can stage the KV's in the DRAM on the decode GPU servers, and then transfer the KV's to the prefill GPUs via GDRDMA
显示更多