Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware.
Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 đĒ with llama.cpp.
Without speculative decoding
⥠166 tps
Add NVIDIA's LiLiCorr DFlash drafter
đ 343 tps median
That's about a 2.07Ã jump.
And prompt processing
đĨ ~5,200 tps
VRAM is interesting too
đ§ Bolt only â 21.0 GiB
đī¸ + vision projector â 22.7 GiB
đ + projector + LiLiCorr â 25.0 GiB
So the regular multimodal model really can squeeze onto a 24GB GPU.
Link in ALT.
Nvidia has delivered a Qwen3.6-35B killer and it's a 74B â 24GB model, actually, today.
And now we have real numbers. đ
Nemotron 3.5 Bolt 74B-A9B
đ§ 74B total / 9B active
đž 22.1GB quantized weights
đĸ ~2.06-bit experts + QAT
đ up to 256K context
đī¸ text + images + video
đ¤ reasoning + tool use
đĻ official GGUF + llama.cpp
And NVIDIA reports on RTX Spark:
⥠~84 tps
That's a 74B multimodal agent model running locally.
Check out how much quality survived the 2-bit compression. BF16 â 24GB IQ2:
đ¯ PinchBench
90.35 â 87.82
đ BrowseComp
70.0 â 66.75
đ§ GPQA Diamond
88.83 â 86.43
đģ Terminal-Bench 2.1
53.93 â 50.98
Nvidia calculates
đž weights â 22.01GB
đž cache + activations â 0.87GB
đž LiLiCorr drafter â 0.96GB
Total â 23.83GB
The vision encoder/projector are not included in that number.
So Nvidia's official llama.cpp recipes actually target
đĸ 32GB+ GPU â 65K context / 28GB config
đĨ 48GB+ GPU â 256K context / 32GB config
đ Links in ALT