๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Akshay ๐Ÿš€
@akshay_pachaar
Simplifying LLMs, AI Agents, RAG, and Machine Learning for you! โ€ข Co-founder @dailydoseofds_โ€ข BITS Pilani โ€ข 3 Patents โ€ข ex-AI Engineer @ LightningAI
๊ฐ€์ž… July 2012
512 ํŒ”๋กœ์ž‰ ์ค‘    292.6K ํŒฌ
Temperature, top-k and top-p, clearly explained! (how an LLM picks its next token) An LLM never picks a word directly. At every generation step, it gives a raw score to every token in its vocabulary. It then turns those raw scores into probabilities and samples one token. Temperature, top-k and top-p shape that sampling step. None of them touches the model's weights. Here's how a typical sampler applies them: โ†ณ Temperature: it rescales the raw scores. Low values sharpen the distribution, so a few tokens dominate. High values flatten it, so unlikely tokens get a real chance. โ†ณ Top-k: it keeps only the k most likely tokens. With top_k at 50, it drops every token ranked below 50, whatever the distribution looks like. โ†ณ Top-p: it keeps the smallest set of tokens whose probabilities add up to at least p. With top_p at 0.9, an obvious next token leaves only a few tokens, and an open-ended one leaves many. โ†ณ Sample: the sampler picks one token from the tokens that remain. The key difference: top-k always keeps the same number of tokens, while top-p adapts to how confident the model is. Low temperature doesn't make a model more correct, though. It only reduces randomness, so a model without the right context can still be confidently wrong. Good starting points for temperature: - Classification and extraction: 0.0 to 0.2 - Code generation: 0.0 to 0.3 - General chat: 0.5 to 0.8 - Brainstorming: 0.7 to 1.2 Tune temperature first, then add top-p to trim the long tail of unlikely tokens. While you evaluate, change one setting at a time instead of all three at once. The diagram below shows the complete flow. - Temperature reshapes the probabilities. - Top-k keeps a fixed number of tokens. - Top-p keeps just enough tokens to reach a set share of probability. I wrote a full breakdown of LLM inference in the article below.
๋” ๋ณด๊ธฐ