Accelerate inference without escalating operational spend. By pairing compact draft models with primary LLMs, speculative decoding speeds up auto-regressive generation. Learn how Red Hat OpenShift AI and Kubeflow optimize model serving for lower latency and better compute ROI.

Now stack it with quantization. Pair the speculator above with our NVFP4 checkpoint: a 4-bit, Blackwell-native target, with DSpark's faster decoding on top. Serve RedHatAI/Qwen3.8-27B-NVFP4 with the DSpark speculative-config (method dspark, 8 tokens).

Pair the DSpark speculator with our GLM-5.3 NVFP4 checkpoint for the best vLLM performance on Blackwell. NVFP4 gives you a 4-bit target that recovers 95%+ of accuracy across evals. DSpark adds faster decoding on top. To run both: serve RedHatAI/GLM-5.3-NVFP4 and set --speculative-config to method dspark with the GLM-5.3-speculator.dspark model (8 tokens). Full config on the card:

LLM Compressor v0.14.0 is out, and GPTQ just got its biggest speedup since launch. A new Triton kernel makes quantization ~15x faster end to end. Batching layers that share a shape pushes that to ~30x on some MoE workloads. Even the old eager path is 1.5-2x faster. Also new: expanded MSE/iMatrix observers that beat GPTQ for NVFP4 on internal benchmarks, REAP pruning with distributed DDP, and support for GLM 5.3 and Qwen3.8. Full release notes:

🚀 Our DSpark for Qwen3.8-27B beats native MTP with the same 8 speculative tokens on 4×H200. Up to 52% faster single-stream decoding and 23% higher peak throughput. On 8-needle MRCR, it averages 4.30 accepted tokens beyond 1M context. Model: Demo below, see for yourself!

DiffusionGemma-Jev now runs on vLLM 🚀 Ask yes/no, multiple-choice, or scored questions and get confidence with every answer. vLLM seeds a canvas with the response template, leaves only the answer slots noisy, then reads a probability distribution from every slot in a single denoising step. Huge thanks to @mmastrac for driving this upstream! 🙏

AgentOps Unlocked, episode 2. We start where every agent starts: on a laptop. @__gracecaroline builds a free agent running entirely local: open-weight Qwen3 on Ollama, orchestrated with LangGraph, three tools wired the same way, a Python calculator, web search via Firecrawl, and GitHub through an external MCP server. The model never phones home. Clone the repo and you're running in minutes:

Raleigh, Oct 17-18. Open models, 33B or smaller. Prizes include an RTX 5090. Small Models Hack with Red Hat and @NVIDIAAI. Fine-tune, build multi-agent systems, or go deep on tool-calling. Apply now:

Huge congrats to the @vllm_project team on pushing DSpark training for Kimi-K3 to this scale! 🚀 As the high-performance data plane behind the disaggregated pipeline, Mooncake moves hidden states between vLLM inference and training workers, fully utilizing RDMA bandwidth with topology-aware routing and multi-NIC aggregation. A great example of Mooncake powering large-scale AI data movement beyond KV cache transfer. Read more:

762 commits. 315 contributors. 104 first-timers. vLLM v0.30.0 is live. 🎉 Highlights: 🤖 Hybrid-attention hot paths: Kimi K3 streamlines KDA, AttnRes, and MLA; DeepSeek-V4.1-Flash adds MXFP8 KV and async Engram; Qwen3.8-Flash-Next fuses QSA/PLE and cuts sparse-GQA overhead 🗄️ HiSparse adds a host tier beneath sparse-MLA decode; under GPU pressure, only top-k misses return to a per-request hot buffer 🛠️ Model Runner V2 brings EAGLE3-style drafts to pipeline parallelism and extends adaptive verification to every draft-model speculator through online acceptance estimation (#50514#, #52228#) 🖋️ Dual-key Gumbel-max watermark generation and detection, with per-request opt-out and speculative-decoding support 🆕 New models include GLM-5.3-Flash, K2-Horizon, Cohere Compass, and Bailing V3 VL ⚡ Fast Start keeps post-quantized, TP-sharded weights in a per-GPU daemon; restarts map them over CUDA IPC with --load-format ipc_cache Thread 👇

If your agent can act on real systems, with real credentials, over long time horizons, what actually stops it from doing the wrong thing? Not the prompt. The model is probabilistic; your operational controls can't be. @LegareKerrison kicks off AgentOps Unlocked series with the four gaps between a laptop demo and production: execution containment, identity, observability, and lifecycle governance. Built on open source you already know: Kubernetes, SPIFFE/SPIRE, OpenTelemetry.

Traditional monitoring tells you an API responded successfully in five seconds. It can't tell you your agent called the wrong tool, or fed the model the wrong context. @LegareKerrison breaks down AI observability with @MLflow: tracing, LLM-judge evaluation, and OpenTelemetry for multi-agent systems.

Agents are easy to prototype. Hard to put into production. And as they multiply, so does everything you have to secure, govern, and pay for. Red Hat AI Elevate is a 3-hour virtual showcase unveiling Red Hat AI 3.6. Secure agent execution, models-as-a-service with per-tenant quotas and token metering, automated safety and security evals, and live demos. Nov 5, live or on demand:

Watch @MLflow capture a full trace of one agent request: the chain-of-thought reasoning, every tool call, the timing of each span, on the OpenShift AI dashboard. Here's @LegareKerrison on seeing what your agent actually did, not just what it answered:

Your AI assistant says 44% pull-through, 60 days to close. The dashboard says 43.2% and 52.5. Did the model hallucinate? Trace it with @MLflow and you see the agent got 44% and 60 straight from the tool. It summarized correctly. The bug is upstream: the tool and the dashboard just disagree. That's the shift, from "the AI is wrong" to a bug someone can actually fix. @cedricclyburn and @LegareKerrison on how AI observability works:

llm-d flow control, chapter 2: shared inference under burst pressure. A GPU pool can have spare capacity on average and still run out during a traffic burst. With @_llm_d_ flow control enabled, excess requests are queued until capacity becomes available.

Is an agent that gets it right every time better than one that's nearly as good in a quarter of the time? For agentic work, that's the real question. A 20-call task at ~2s per call is about 45 seconds of wall-clock time. Get each call under 1s and the same task lands around 13. Every step waits on the last, so small per-call latency compounds fast. Which is why "which model is smartest" is giving way to "which is smart enough and fast enough for this step." And you don't always have to trade off. Speculative decoding is lossless and cuts latency 1.5-3x. FP8 quantization keeps 99%+ accuracy. You can make a capable model fast on your own hardware. @_soyr_ on choosing models for agentic work: