TL;DR: Storing user profiles as reusable KV cache instead of text made an LLM recommender's scoring path about 4x faster with ranking quality essentially unchanged.
KVUse: Reusable-KV User Memory for Efficient LLM Recommenders
Key points
・🧠 Targets memory-augmented recommenders that re-prefill each user's natural-language profile over and over
・🚫 Prefix caching can't help, since different instructions precede the profile in each call
・✍️ LoRA writers (about 6.8M params) on Llama-3.1-8B also emit KV while generating the profile. The reader is the unmodified LLM
・🎓 Self-distillation from full prefill, with positions fixed by RoPE re-rotation
・🚀 Scoring 3.96 to 4.49x (Books: 31.99 to 126.72 users/s), update path 1.43 to 1.91x
・✅ NDCG
@10 change of −0.001 to +0.008, not significant. Naive KV reuse dropped 26 to 32%
Caveats: KV is about 128 KiB per token, and writers need per-domain training.
#
LLMInference# #
RecSys#