Jeffrey Wang
@jeffreyycwang
LLMs @ Anyscale
Joined August 2022
110 Following 172 Followers
A higher KV cache hit rate doesn’t always mean lower TTFT and TPOT, or higher throughput.
We studied LLM routers with different request scoring objectives under agentic and RL workloads. Here’s what we learned. 🧵
Show more
