Register and share your invite link to earn from video plays and referrals.

David Hendrickson
@TeksEdge
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | 🔔 Follow for AI & Vibe Coding Tips 👇
Joined July 2023
552 Following    12.5K Followers
Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware. Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 đŸ’Ē with llama.cpp. Without speculative decoding ⚡ 166 tps Add NVIDIA's LiLiCorr DFlash drafter 🚀 343 tps median That's about a 2.07× jump. And prompt processing đŸ”Ĩ ~5,200 tps VRAM is interesting too 🧠 Bolt only → 21.0 GiB đŸ‘ī¸ + vision projector → 22.7 GiB 🚀 + projector + LiLiCorr → 25.0 GiB So the regular multimodal model really can squeeze onto a 24GB GPU. Link in ALT.
Show more
Nvidia has delivered a Qwen3.6-35B killer and it's a 74B → 24GB model, actually, today. And now we have real numbers. 👀 Nemotron 3.5 Bolt 74B-A9B 🧠 74B total / 9B active 💾 22.1GB quantized weights đŸ”ĸ ~2.06-bit experts + QAT 📚 up to 256K context đŸ‘ī¸ text + images + video 🤖 reasoning + tool use đŸĻ™ official GGUF + llama.cpp And NVIDIA reports on RTX Spark: ⚡ ~84 tps That's a 74B multimodal agent model running locally. Check out how much quality survived the 2-bit compression. BF16 → 24GB IQ2: đŸŽ¯ PinchBench 90.35 → 87.82 🔎 BrowseComp 70.0 → 66.75 🧠 GPQA Diamond 88.83 → 86.43 đŸ’ģ Terminal-Bench 2.1 53.93 → 50.98 Nvidia calculates 💾 weights → 22.01GB 💾 cache + activations → 0.87GB 💾 LiLiCorr drafter → 0.96GB Total → 23.83GB The vision encoder/projector are not included in that number. So Nvidia's official llama.cpp recipes actually target đŸŸĸ 32GB+ GPU → 65K context / 28GB config đŸ”Ĩ 48GB+ GPU → 256K context / 32GB config 🔗 Links in ALT
Show more