Temperature, top-k and top-p, clearly explained!
(how an LLM picks its next token)
An LLM never picks a word directly. At every generation step, it gives a raw score to every token in its vocabulary.
It then turns those raw scores into probabilities and samples one token.
Temperature, top-k and top-p shape that sampling step. None of them touches the model's weights.
Here's how a typical sampler applies them:
↳ Temperature: it rescales the raw scores. Low values sharpen the distribution, so a few tokens dominate. High values flatten it, so unlikely tokens get a real chance.
↳ Top-k: it keeps only the k most likely tokens. With top_k at 50, it drops every token ranked below 50, whatever the distribution looks like.
↳ Top-p: it keeps the smallest set of tokens whose probabilities add up to at least p. With top_p at 0.9, an obvious next token leaves only a few tokens, and an open-ended one leaves many.
↳ Sample: the sampler picks one token from the tokens that remain.
The key difference: top-k always keeps the same number of tokens, while top-p adapts to how confident the model is.
Low temperature doesn't make a model more correct, though. It only reduces randomness, so a model without the right context can still be confidently wrong.
Good starting points for temperature:
- Classification and extraction: 0.0 to 0.2
- Code generation: 0.0 to 0.3
- General chat: 0.5 to 0.8
- Brainstorming: 0.7 to 1.2
Tune temperature first, then add top-p to trim the long tail of unlikely tokens.
While you evaluate, change one setting at a time instead of all three at once.
The diagram below shows the complete flow.
- Temperature reshapes the probabilities.
- Top-k keeps a fixed number of tokens.
- Top-p keeps just enough tokens to reach a set share of probability.
I wrote a full breakdown of LLM inference in the article below.