登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

David Hendrickson
@TeksEdge
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | 🔔 Follow for AI & Vibe Coding Tips 👇
参加 July 2023
549 フォロー中    11.2K ファン
Interesting HuggingFace post. What if the easiest way to make long-context Local AI faster is to simply STOP feeding it everything? A new HF forum experiment tested 600 distractor-heavy long-context prompts with Qwen2.5-7B 4-bit. Instead of sending the entire ~14K-token context, a MiniLM semantic selector kept only the most relevant ~60%. The result ... 📚 Input tokens 13,124 → 7,702 ➡️ -41% ⚡ Prefill 871 ms → 463 ms ➡️ -47% 🚀 End-to-end latency 944 ms → 519 ms ➡️ -45% 💾 Peak VRAM 11.71 GiB → 9.16 GiB And here's the weird part... 🎯 Token F1 actually IMPROVED: 0.534 → 0.581 So the model got: ✅ less context ✅ lower VRAM use ✅ almost 2× faster prefill ✅ lower total latency ✅ BETTER answers Why? Because more context isn't always better context. Coding agents and RAG systems often drag around enormous histories filled with irrelevant junk. ⚠️ This was a controlled distractor-heavy HotpotQA stress test, not proof that deleting 40% of every real-world context will improve accuracy. 🔗 Link in ALT.
もっと見る