vLLM
@vllm_project
A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!
๊ฐ์
March 2024
36 ํ๋ก์ ์ค 50.2K ํฌ
Excited to see @peano_ai run full-parameter RL for @XiaomiMiMo 310B MiMo-V2.6 on TPUs, across 1,000+ TPUs. ๐๐
vLLM drives the rollouts, bitwise-matched with the trainer in validation. All 310B params move across the ICI fabric in under 2s.
๐
๋ ๋ณด๊ธฐ
We enable full-parameter RL on TPUs: MiMo-V2.6 at 310B, plus other stable training runs of 1,000+ steps across 1,000+ TPUs.
With JAX, scaling up is a config change, not a rewrite.
We built on that with optimized vLLM inference for faster rollouts and full bitwise trainerโsampler agreement in validation.
Trainer and sampler share one TPU ICI fabric. All 310B MiMo-V2.6 parameters transfer in <2 seconds.
๋ ๋ณด๊ธฐ
