vLLM(@vllm_project):The hardware work reaches across the serving stack. NVIDIA's FlashInfer and vLLM optimizations include fused norm/gating/residual kernels, TEP pooler/unpooler sharding with fused MNNVL collectives, and an FP32-norm-to-TF32-router path. AMD MI355X supports FP8 and MXFP4 with TP/PP on ROCm. Initial TPU support uses TorchTPU. With DSpark (acceptance length = 6), Muse Spark Open megakernels reach over 650 tokens/sec on 4 TPU v7 chips and 960 tokens/sec on 8 chips; without speculation, over 270 and 350 tokens/sec. Code access is available through the team or Google. 4/6

vLLM

@vllm_project

A high-throughput and memory-efficient inference and serving engine for LLMs. Join to discuss together with the community!

参加 March 2024

37 フォロー中 54.1K ファン

vLLM@vllm_project

2026.10.15 18:13

The hardware work reaches across the serving stack. NVIDIA's FlashInfer and vLLM optimizations include fused norm/gating/residual kernels, TEP pooler/unpooler sharding with fused MNNVL collectives, and an FP32-norm-to-TF32-router path. AMD MI355X supports FP8 and MXFP4 with TP/PP on ROCm. Initial TPU support uses TorchTPU. With DSpark (acceptance length = 6), Muse Spark Open megakernels reach over 650 tokens/sec on 4 TPU v7 chips and 960 tokens/sec on 8 chips; without speculation, over 270 and 350 tokens/sec. Code access is available through the team or Google. 4/6