注册并分享邀请链接,可获得视频播放与邀请奖励。

vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
加入 March 2024
37 正在关注    54.6K 粉丝
Quantization is easy. Fast quantization kernels for every format × every GPU × every workload are not. Today we're open-sourcing Humming with Ant Group: one kernel library for all of it — up to 5x kernel speedups, +14–43% E2E MoE throughput. Blog + benchmarks:
显示更多
0
23
479
50
转发到社区