Michael Wolfrath

Temperature, top-k and top-p, clearly explained! (how an LLM picks its next token) An LLM never picks a word directly. At every generation step, it gives a raw score to every token in its vocabulary. It then turns those raw scores into probabilities and samples one token. Temperature, top-k and top-p shape that sampling step. None of them touches the model's weights. Here's how a typical sampler applies them: ↳ Temperature: it rescales the raw scores. Low values sharpen the distribution, so a few tokens dominate. High values flatten it, so unlikely tokens get a real chance. ↳ Top-k: it keeps only the k most likely tokens. With top_k at 50, it drops every token ranked below 50, whatever the distribution looks like. ↳ Top-p: it keeps the smallest set of tokens whose probabilities add up to at least p. With top_p at 0.9, an obvious next token leaves only a few tokens, and an open-ended one leaves many. ↳ Sample: the sampler picks one token from the tokens that remain. The key difference: top-k always keeps the same number of tokens, while top-p adapts to how confident the model is. Low temperature doesn't make a model more correct, though. It only reduces randomness, so a model without the right context can still be confidently wrong. Good starting points for temperature: - Classification and extraction: 0.0 to 0.2 - Code generation: 0.0 to 0.3 - General chat: 0.5 to 0.8 - Brainstorming: 0.7 to 1.2 Tune temperature first, then add top-p to trim the long tail of unlikely tokens. While you evaluate, change one setting at a time instead of all three at once. The diagram below shows the complete flow. - Temperature reshapes the probabilities. - Top-k keeps a fixed number of tokens. - Top-p keeps just enough tokens to reach a set share of probability. I wrote a full breakdown of LLM inference in the article below.

Agent framework vs. agent harness, visually explained: