Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware.
Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 💪 with llama.cpp.
Without speculative decoding
⚡ 166 tps
Add NVIDIA's LiLiCorr DFlash drafter
🚀 343 tps median
That's about a 2.07× jump.
And prompt processing
🔥 ~5,200 tps
VRAM is interesting too
🧠 Bolt only → 21.0 GiB
👁️ + vision projector → 22.7 GiB
🚀 + projector + LiLiCorr → 25.0 GiB
So the regular multimodal model really can squeeze onto a 24GB GPU.
Link in ALT.
Nvidia has delivered a Qwen3.6-35B killer and it's a 74B → 24GB model, actually, today.
And now we have real numbers. 👀
Nemotron 3.5 Bolt 74B-A9B
🧠 74B total / 9B active
💾 22.1GB quantized weights
🔢 ~2.06-bit experts + QAT
📚 up to 256K context
👁️ text + images + video
🤖 reasoning + tool use
🦙 official GGUF + llama.cpp
And NVIDIA reports on RTX Spark:
⚡ ~84 tps
That's a 74B multimodal agent model running locally.
Check out how much quality survived the 2-bit compression. BF16 → 24GB IQ2:
🎯 PinchBench
90.35 → 87.82
🔎 BrowseComp
70.0 → 66.75
🧠 GPQA Diamond
88.83 → 86.43
💻 Terminal-Bench 2.1
53.93 → 50.98
Nvidia calculates
💾 weights → 22.01GB
💾 cache + activations → 0.87GB
💾 LiLiCorr drafter → 0.96GB
Total → 23.83GB
The vision encoder/projector are not included in that number.
So Nvidia's official llama.cpp recipes actually target
🟢 32GB+ GPU → 65K context / 28GB config
🔥 48GB+ GPU → 256K context / 32GB config
🔗 Links in ALT