Huggingface more than models/cards it is a wealth of knowledge from the training, courses, articles and blogs!
Always something to learn
Tether just released BotPsy, a 0.8B agent model built on Qwen3.5-0.8B, with Apache 2.0 weights and GGUFs for llama.cpp
Interesting HuggingFace post.
What if the easiest way to make long-context Local AI faster is to simply STOP feeding it everything?
A new HF forum experiment tested 600 distractor-heavy long-context prompts with Qwen2.5-7B 4-bit.
Instead of sending the entire ~14K-token context, a MiniLM semantic selector kept only the most relevant ~60%.
The result ...
📚 Input tokens
13,124 → 7,702
➡️ -41%
⚡ Prefill
871 ms → 463 ms
➡️ -47%
🚀 End-to-end latency
944 ms → 519 ms
➡️ -45%
💾 Peak VRAM
11.71 GiB → 9.16 GiB
And here's the weird part...
🎯 Token F1 actually IMPROVED:
0.534 → 0.581
So the model got:
✅ less context
✅ lower VRAM use
✅ almost 2× faster prefill
✅ lower total latency
✅ BETTER answers
Why?
Because more context isn't always better context.
Coding agents and RAG systems often drag around enormous histories filled with irrelevant junk.
⚠️ This was a controlled distractor-heavy HotpotQA stress test, not proof that deleting 40% of every real-world context will improve accuracy.
🔗 Link in ALT.
is the HuggingFace attack something that should be taken seriously? yes.
is it the first warning shot of a machine takeover? no.
will issues of this type look like a tractable / mostly solved problem with proper precautions in 6 months? yes.
will that happen automatically? no. as always, will require work from smart people and their agents.