[Attention] Attention is how one token pulls relevant information from itself and the tokens before it. At each layer, learned projections turn each token’s current representation into three vectors: a query, a key, and a value. The query represents what the current position is looking for. Keys describe what each token can be matched on, while values carry the information that can be pulled in. For “runs,” one attention head compares its query with the keys of “The,” “chip,” “Alice,” “designed,” and “runs” itself. Each comparison is a scaled dot product, q · k / √d. Softmax turns the scores into weights that add up to one, and the values are multiplied by those weights and summed. A transformer runs several heads in parallel. Each has its own learned projections, so it can view the same tokens differently. Their outputs are combined and passed through the remaining layers. A head may emphasize the subject while another captures a different relationship, but these are learned tendencies, not roles assigned in advance. But where did all those keys and values come from? And when the next token arrives, does the model have to build the earlier ones all over again? It does not. Tomorrow: KV cache. The paper that introduced the transformer: Vaswani et al., NeurIPS 2017.