vLLM(@vllm_project):🎙️ @Alibaba_Qwen's Qwen3-Omni listens, reasons, and talks back. Serving that in real time is a pipeline problem, not a single model: a multimodal Thinker, then Talker → Code2Wav for the speech. Each stage bottlenecks differently, so the wins come from optimizing them layer by layer. One neat trick: under load, replicate only the two speech stages and let the heavy multimodal Thinker run once. At high concurrency that lands first audio in ~0.6s instead of ~6s, speech faster than real time, and ~5.4x the throughput on the same GPUs. Built with @AntGroup's Super Computing Technology (SCT) team and the vLLM-Omni team. The blog breaks down the full stack, one bottleneck at a time 👇 🔗

2026.07.03 07:15

🎙️ @Alibaba_Qwen's Qwen3-Omni listens, reasons, and talks back. Serving that in real time is a pipeline problem, not a single model: a multimodal Thinker, then Talker → Code2Wav for the speech. Each stage bottlenecks differently, so the wins come from optimizing them layer by layer. One neat trick: under load, replicate only the two speech stages and let the heavy multimodal Thinker run once. At high concurrency that lands first audio in ~0.6s instead of ~6s, speech faster than real time, and ~5.4x the throughput on the same GPUs. Built with @AntGroup's Super Computing Technology (SCT) team and the vLLM-Omni team. The blog breaks down the full stack, one bottleneck at a time 👇 🔗