註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture 投稿は個人の意見です。
加入 May 2026
287 正在關注    476 粉絲
TL;DR: Storing user profiles as reusable KV cache instead of text made an LLM recommender's scoring path about 4x faster with ranking quality essentially unchanged. KVUse: Reusable-KV User Memory for Efficient LLM Recommenders Key points ・🧠 Targets memory-augmented recommenders that re-prefill each user's natural-language profile over and over ・🚫 Prefix caching can't help, since different instructions precede the profile in each call ・✍️ LoRA writers (about 6.8M params) on Llama-3.1-8B also emit KV while generating the profile. The reader is the unmodified LLM ・🎓 Self-distillation from full prefill, with positions fixed by RoPE re-rotation ・🚀 Scoring 3.96 to 4.49x (Books: 31.99 to 126.72 users/s), update path 1.43 to 1.91x ・✅ NDCG@10 change of −0.001 to +0.008, not significant. Naive KV reuse dropped 26 to 32% Caveats: KV is about 128 KiB per token, and writers need per-domain training. #LLMInference# #RecSys#
顯示更多