alexa griffith(@alexa_griffith_):llm-d flow control, chapter 3: where control starts. Throughput can plateau while latency keeps rising. The admission point is where flow control decides to keep a request in the queue or admit it to the model server pod. Using a load test, you can find that cutoff point for your model and workload.

alexa griffith

@alexa_griffith_

book enthusiast, senior principal engineer at Red Hat, Inference FDE, @alexasinput podcast host

加入 November 2019

521 正在关注 504 粉丝

alexa griffith@alexa_griffith_

2026.09.23 04:26

llm-d flow control, chapter 3: where control starts. Throughput can plateau while latency keeps rising. The admission point is where flow control decides to keep a request in the queue or admit it to the model server pod. Using a load test, you can find that cutoff point for your model and workload.

显示更多