alexa griffith
@alexa_griffith_
book enthusiast, senior principal engineer at Red Hat, Inference FDE, @alexasinput podcast host
521 Following 504 Followers
Happy Thursday!! Went to the hat shop and got this adorably aggressive hat. Wearing weird shoes. And my inspo board. Hope ur doing well xx
llm-d flow control, chapter 3: where control starts.
Throughput can plateau while latency keeps rising. The admission point is where flow control decides to keep a request in the queue or admit it to the model server pod. Using a load test, you can find that cutoff point for your model and workload.
Show more
How live requests, batch jobs, and background apps reach the same inference pool in llm-d.
Try it yourself:
I love codex
Ladies and gentlemen... start... your... ENGINES. We are almost Tuesday and I promised a reset for Tuesday. Among some other things. See you soon.
I’m speaking at #KubeCon# + #CloudNativeCon# North America! My talk, “Cache Me If You Can,” breaks down llm-d: prefix-cache-aware routing, disaggregation, scheduling, and flow control. Salt Lake City, Nov 9–12: @CloudNativeFdn
Show more
Honestly, I’m working 5x harder since coding agents actually became useful. Weren't these supposed to take software engineers out of the critical path? Now it just feels like every part of the stack needs to be rebuilt, and it has to happen now.
Show more
Two agents start from the same commit, make different changes, and both report success. Which version should advance the branch? That's one question I'm exploring in Git with the times, my new Substack piece.
Show more
llm-d flow control, chapter 2: shared inference under burst pressure.
A GPU pool can have spare capacity on average and still run out during a traffic burst. With @_llm_d_ flow control enabled, excess requests are queued until capacity becomes available.
Show more
Gentlemen, you had my curiosity, but now you have my attention
llm-d flow control interactive learner, chapter 1: capacity utilization.
Here’s the same traffic in separate tenant-GPU reservations versus a shared pool. Sharing lets tenants use capacity that would otherwise sit idle.
Show more
Everything you need to start self-hosting an open LLM.
Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine.
The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models that cut an 8B from ~16GB of weights to a quarter of that while keeping 98-100% accuracy.
Walkthrough by @cedricclyburn.
Show more
I made an interactive learner to explain @_llm_d_'s flow control feature for shared inference.
Try it yourself:
Great article by @alexa_griffith_. When multiple teams share one GPU pool, arrival order shouldn't decide who gets served first.
@_llm_d_ flow control adds admission before routing. Under saturation, priority decides which traffic advances, fairness keeps one tenant from starving the rest, and lower priority work waits. Admission is when a request runs, routing is where, and keeping them separate is the whole idea. Worth a read!
Show more
flow control is an exciting feature in llm-d that enables sharedd, multi-tenant inference
Its never been easier to build the wrong thing
same
JUST IN: Anthropic says they’re highly profitable if you take out some of their biggest expenses.
Flow control blog out! This one details how llm-d's flow control feature in Red Hat AI Inference 3.5 provides priority queuing to protect SLAs on shared infrastructure. It's a really cool feature that opens up a lot of possibilities for running multitenant workloads and getting better use out of your GPUs. Make sure to check it out :) @RedHat_AI @_llm_d_
Show more
