TL;DR: Storing user profiles as reusable KV cache instead of text made an LLM recommender's scoring path about 4x faster with ranking quality essentially unchanged.
KVUse: Reusable-KV User Memory for Efficient LLM Recommenders
Key points
ใป๐ง Targets memory-augmented recommenders that re-prefill each user's natural-language profile over and over
ใป๐ซ Prefix caching can't help, since different instructions precede the profile in each call
ใปโ๏ธ LoRA writers (about 6.8M params) on Llama-3.1-8B also emit KV while generating the profile. The reader is the unmodified LLM
ใป๐ Self-distillation from full prefill, with positions fixed by RoPE re-rotation
ใป๐ Scoring 3.96 to 4.49x (Books: 31.99 to 126.72 users/s), update path 1.43 to 1.91x
ใปโ
NDCG
@10 change of โ0.001 to +0.008, not significant. Naive KV reuse dropped 26 to 32%
Caveats: KV is about 128 KiB per token, and writers need per-domain training.
#
LLMInference# #
RecSys#