Red Hat AI
@RedHat_AI
Accelerating AI innovation with open platforms and community.
The future of AI is open.
2.1K Following 12.9K Followers
come here me blab about microscaling formats in vllm at the end 🤠
You can now upgrade PyTorch without rebuilding vLLM. As of v0.33.0, vLLM ships LibTorch ABI Stable CUDA wheels, so you swap the torch wheel and import it, no vLLM rebuild to pick up a new PyTorch release.
How @vllm_project, @RedHat, and @AIatMeta pulled it off:
Show more
New vLLM Office Hours on advanced model compression, with Kyle Sayers and @_soyr_.
They cover:
- LLM Compressor's sequential pipeline and data parallel schemes
- Settings most users never change
- What observers do and how they affect quantization error
- Expert pruning with REAP and MoNE
Show more
@vllm_project @RedHat_AI 59 community-run sessions is a lot of free knowledge. Office hours like that make vLLM so much easier to learn.
The agentic era is here. Is your infrastructure ready for it? From zero trust operating systems to autonomous cluster management, follow along for Red Hat’s announcements live from #KubeCon#+#CloudNativeCon# and join us at booth #116# to learn more!
Show more
76 guest speakers. 24 organizations. 59 sessions and counting.
vLLM Office Hours is a bi-weekly livestream with the people building @vllm_project: a project update, a guest talk, and open Q&A. Open to all.
Get a calendar invite:
Show more
Love seeing what the community builds on vLLM-Omni 🤩 OpenDirector: a local short-film workspace — generate, edit, export — powered by vLLM-Omni with a wide range of models. Congrats @hsliuunc! 🎬
🎥 OpenDirector:
⚡ vLLM-Omni:
Show more
Join us in this livestream next Tuesday, learn how to shrink models to fit in your laptop and make them go brrr 💨
We will have @mervenoyann @maximelabonne @TadimetiAditya @bartowski1182 and @mgoin_ giving a primer and know-hows on quantization with llama.cpp and vLLM ⚡️
Show more
Hola Madrid! 🇪🇸 vLLM is coming to town — join us Nov 12 with @RedHat_AI and @NVIDIAAI for an evening of open-source inference: vLLM, vLLM-omni, LLM Compressor, and llm-d on Kubernetes. Register:
Show more
the strongest open model yet runs at 800+ tok/s/user on VR200 vLLM *on day 0*
can’t wait to see how much faster we make it over the coming weeks 🔥
Getting benchmark numbers is easy. Getting the right ones is hard. KV cache offloading can cut TTFT from 60s to 10s and lift throughput up to 5.8x, but only when your working set and storage tier line up. Here's how we benchmarked it, caveats and all:
Show more
Shift your open source security from reactive to ready.
Join our global webinar on November 12, 2026, to see how to evolve your security operating model from provenance to remediation.
Register for the virtual event:
Show more
vLLM v0.33.0 is out! 🚀
927 commits · 389 contributors · 122 first-timers
🤖 New models: Reflection Beam-1.0, Step-5 Preview, Nemotron Parse 3.5, MiniCPM-OCR 1.0
🔥 PyTorch 2.14 — CUDA 13.0-only wheels, libtorch ABI-stable
🎲 Spec decoding: XPress refiner, FLy lossy verify (1.04–1.31x), DSpark 1373→3390 tok/s on GB300
⚡ NVIDIA decode: 1.5–2x latency decode on GB300, FA4 sparse MLA, 39–48% lower TTFT
🚀 New AWS EFA image + shorter vllm/vllm name
🐧 ROCm 10.0 default, MonoKernel 2.32x P50
Full notes ➡️
Show more
Two optimized checkpoints for NVIDIA Nemotron 3.5 Super VL, a 120B multimodal MoE for long-running agents. Quantization and speculative decoding stack for close to 2x on Hopper, accuracy intact.
FP8 (LLM Compressor):
DSpark (Speculators):
Show more
More state-of-the-art MXFP4 ckpts ready for inference on AMD GPUs and @vllm_project available at:
Serving AI at scale brings real tradeoffs in cost, latency, and capacity.
Hear from vLLM core maintainer Tyler Michael Smith on inference economics in San Jose, Oct 19.
Thanks @RedHat, @NVIDIAAI, and @IBM for bringing the community together!
Show more
AI agents are driving the next evolution of enterprise apps. Learn how Red Hat is expanding its investment in @OpenClaw Enterprise to build an open agent control plane:
Show more
PyTorch moved from research labs to running enterprise AI. On Monday Oct 19, @RedHat, @NVIDIAAI, and @IBM host a day in San Jose: @vllm_project economics, @_llm_d_ at scale, and agents in production, with the @PyTorch maintainers behind them.
Register:
Show more
Want to learn more about self-hosting your own models and agents? This Thursday at 10am ET I’m joining @hacktoberfest for some live demos and tips on using @vllm_project 🎃
Bring your questions, I'll try to get to as many as I can :) and we’ll be streaming live from @MLHacks
🔗
Show more
vLLM Office Hours #58# recording is up:
- What's new in vLLM v0.29 and v0.30
- A new hardware backend from Tenstorrent
- An intro to llm-d Semantic Classifier, a lightweight classification service for high-throughput routing
- Why it isn't trying to be a guardrail
Watch + see slides:
Show more
