we launched the most comprehensive ai performance engineering repo in the world last week now we'll be doing a deep dive on every single resource this is Wafer's AI Performance Engineering Series save this as your starting point. links in thread 🧵 Part 1: Prefill vs Decode a causal autoregressive Transformer computes next-token logits from the supplied prefix. a decoding policy selects one token and appends it before the model predicts the next. in a conventional causal decoder, each position attends only to itself and earlier positions. appending a token adds no allowed input to old positions. in evaluation mode, with the prefix, weights, mask and positional computation fixed, the old representations remain reusable. the KV cache stores earlier positions' per-layer keys and values for reuse in later forwards. prefill processes the known prompt under a causal mask and saves each layer's keys and values. the final prompt position produces logits for choosing the first output token. prompt positions can run together within a layer; the layers still depend on one another. the decoding policy selects a token. greedy decoding takes a highest-scoring token; sampling draws from a distribution. feeding the selected token back through the model creates its K/V and produces the logits for the next output. emitting a token and processing it are separate steps. to emit N ≥ 1 tokens, an ordinary loop needs one unchunked prefill and N−1 incremental forwards. emitting the final token does not require processing it through the model. the new token still runs through the layers, including its projections and MLP. its query computes new attention scores and a weighted sum over cached values. storing K/V saves repeated prefix computation; it leaves new attention work over the growing context. for fixed prompt length and model dimensions, caching changes total projection and MLP work from quadratic to linear in output length. full causal attention across generation remains quadratic. these work counts do not establish a measured latency improvement. prefill has many known token rows from one prompt. ordinary decode contributes one new row per active request, so batching it combines different requests. each request also needs its own logical KV state. more concurrent requests and longer retained contexts increase that state. choose the performance target around the workload. offline generation may prioritize completed work per dollar within a deadline. streaming chat also needs low time to first token and responsive delivery afterward. a single-user device may favor latency within its memory and compute limits. for a serving comparison, fix prompt/output lengths and concurrency. measure client time to first token, the intervals between later tokens, and total output tokens over a common measurement window. keep model-only prefill timing separate from queueing and delivery. a higher aggregate token rate alone cannot tell you whether one user's answer arrives sooner. chunked prefill and speculative generation change the execution pattern and are outside this ordinary-loop example.