Jeffrey Wang
@jeffreyycwang
LLMs @ Anyscale
加入 August 2022
110 正在關注 172 粉絲
A higher KV cache hit rate doesn’t always mean lower TTFT and TPOT, or higher throughput.
We studied LLM routers with different request scoring objectives under agentic and RL workloads. Here’s what we learned. 🧵
顯示更多
