Jeffrey Wang
@jeffreyycwang
LLMs @ Anyscale
参加 August 2022
110 フォロー中 172 ファン
A higher KV cache hit rate doesn’t always mean lower TTFT and TPOT, or higher throughput.
We studied LLM routers with different request scoring objectives under agentic and RL workloads. Here’s what we learned. 🧵
もっと見る
