Jeffrey Wang
@jeffreyycwang
LLMs @ Anyscale
가입 August 2022
110 팔로잉 중 172 팬
A higher KV cache hit rate doesn’t always mean lower TTFT and TPOT, or higher throughput.
We studied LLM routers with different request scoring objectives under agentic and RL workloads. Here’s what we learned. 🧵
더 보기
