What is temperature in LLM sampling?
Temperature is a number, typically between 0 and 2, that controls how much randomness a language model uses when choosing its next token. At each step the model computes a probability for every possible next token, and temperature reshapes that distribution before a token is sampled. Low temperature sharpens it toward the most likely tokens, so output gets more deterministic and repetitive. High temperature flattens it, giving less likely tokens a real chance, so output gets more varied and less predictable.
What actually happens at the sampling step
The model's raw output for each step is a set of scores (logits) across its whole vocabulary, one per possible next token. Temperature divides those logits before they're converted to probabilities: dividing by a value below 1 exaggerates the gap between the top candidates, making the model's favorite token even more dominant. Dividing by a value above 1 compresses that gap, letting less likely tokens compete more evenly. At temperature 0, sampling becomes deterministic: the model always picks its single highest-probability token, known as greedy decoding.
What it feels like in practice
Low-temperature output reads as consistent and safe, close to the most conventional phrasing, and the same prompt run twice gives near-identical answers. High-temperature output is more varied, but past a point it loses coherence or drifts off-topic, because the model keeps picking tokens it rates as unlikely. Neither end is better in general. Pick by whether the task rewards consistency (extraction, code, factual answers) or variety (brainstorming, creative writing, several distinct options).
Top-p and other sampling controls
Temperature is usually paired with other sampling parameters rather than used alone. Top-p (nucleus sampling) restricts the candidate pool to the smallest set of tokens whose cumulative probability crosses a threshold, cutting off the long unlikely tail regardless of how temperature reshaped it. Top-k caps the candidate pool at a fixed number of tokens. The settings interact, so changing one changes how the others behave; tune them together for a task, not one at a time.
A note on reliability
This post cites no benchmark figures on purpose. The effect of a given temperature is specific to the model and the task, and GPUwerk has not run controlled measurements across models. Before production, test a few values on your own task and data instead of reusing a default from an unrelated model or vendor.
Where temperature is set
In most serving setups temperature is a request-level parameter, not part of the model weights, so you can change it per call without reloading or retraining anything. It travels with the prompt in the API request, on a hosted API or a self-hosted inference server alike. Because it is cheap to change, many production systems set it per request type (low for a structured-output endpoint, higher for a creative-writing one) instead of one value for the whole deployment.
Why this matters for self-hosted deployment
Hosted API providers sometimes cap or hide sampling parameters, or change their defaults with little notice. When you run the serving engine yourself, on your own hardware or a dedicated rented machine, every parameter it exposes (temperature, top-p, top-k, repetition penalties) is yours to set, and it stays put until you change it. That matters if you are tuning output behavior for a specific task instead of accepting a provider's default.