Temperature & sampling
The dials that control how the model picks the next token from its probability list - the difference between deterministic and creative output.
In plain terms
The model doesn't output an answer; it outputs odds - like 'Paris 92%, Lyon 3%, the 2%...'. Temperature decides how it rolls the dice: 0 ~= always take the favorite (predictable), 1 = roll honestly (varied), higher = give underdogs a chance (creative, then chaotic).
Why it matters
Different jobs need different randomness. Extracting a date from an invoice should be boring and repeatable; brainstorming taglines should surprise you. One dial serves both.
How it works
Temperature reshapes the probability distribution before picking (low = sharpen toward the top choice, high = flatten). top_p (nucleus sampling) instead trims the list to the smallest set covering e.g. 90% probability, then samples within it. Set temperature OR top_p, not both. Note: even temperature 0 isn't perfectly deterministic on most providers - close, not guaranteed.
When you use it
Every API call chooses a value. Defaults: ~0-0.3 for extraction, classification, code, tool use; ~0.7 for chat; ~1.0 for creative writing and brainstorming.
Common mistakes
- Leaving temperature at default 1.0 for structured tasks, then wondering why JSON output randomly breaks.
- Cranking temperature to 'fix' a boring prompt - fix the prompt; temperature adds noise, not ideas.
- Tweaking both temperature and top_p at once, making results impossible to reason about.
Best practices
- Treat temperature as part of your app's config, chosen per task, recorded with your evals.
- When debugging output quality, pin temperature to 0 first so you're comparing prompts, not dice rolls.
Try it yourself
In any API playground, run 'Suggest a name for a coffee shop' 5x at temperature 0, then 5x at 1.0. Watch determinism appear and disappear.
Resources
- OpenAI playground Slide the temperature dial and watch the same prompt change character.