註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

David Hendrickson
@TeksEdge
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | 🔔 Follow for AI & Vibe Coding Tips 👇
加入 July 2023
552 正在關注    12.5K 粉絲
Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware. Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 💪 with llama.cpp. Without speculative decoding ⚡ 166 tps Add NVIDIA's LiLiCorr DFlash drafter 🚀 343 tps median That's about a 2.07× jump. And prompt processing 🔥 ~5,200 tps VRAM is interesting too 🧠 Bolt only → 21.0 GiB 👁️ + vision projector → 22.7 GiB 🚀 + projector + LiLiCorr → 25.0 GiB So the regular multimodal model really can squeeze onto a 24GB GPU. Link in ALT.
顯示更多
Nvidia has delivered a Qwen3.6-35B killer and it's a 74B → 24GB model, actually, today. And now we have real numbers. 👀 Nemotron 3.5 Bolt 74B-A9B 🧠 74B total / 9B active 💾 22.1GB quantized weights 🔢 ~2.06-bit experts + QAT 📚 up to 256K context 👁️ text + images + video 🤖 reasoning + tool use 🦙 official GGUF + llama.cpp And NVIDIA reports on RTX Spark: ⚡ ~84 tps That's a 74B multimodal agent model running locally. Check out how much quality survived the 2-bit compression. BF16 → 24GB IQ2: 🎯 PinchBench 90.35 → 87.82 🔎 BrowseComp 70.0 → 66.75 🧠 GPQA Diamond 88.83 → 86.43 💻 Terminal-Bench 2.1 53.93 → 50.98 Nvidia calculates 💾 weights → 22.01GB 💾 cache + activations → 0.87GB 💾 LiLiCorr drafter → 0.96GB Total → 23.83GB The vision encoder/projector are not included in that number. So Nvidia's official llama.cpp recipes actually target 🟢 32GB+ GPU → 65K context / 28GB config 🔥 48GB+ GPU → 256K context / 32GB config 🔗 Links in ALT
顯示更多