Jeffrey Wang
@jeffreyycwang
LLMs @ Anyscale
加入 August 2022
110 正在关注 172 粉丝
A higher KV cache hit rate doesn’t always mean lower TTFT and TPOT, or higher throughput.
We studied LLM routers with different request scoring objectives under agentic and RL workloads. Here’s what we learned. 🧵
显示更多
