Sampling: temperature, top-k, top-p, and min-p

1

Concept

After the final Transformer layer processes the current context, the language-model head produces one logit for every token in the vocabulary. Logits are unrestricted scores, not probabilities. Softmax converts them into a distribution. Generation still needs one more decision: which token should be appended? The family of rules that makes this choice is called decoding.

The simplest rule is greedy decoding: select the token with the highest probability every time. It is deterministic and inexpensive, but locally best choices do not guarantee the best complete sequence. Greedy output can become repetitive or bland, and an early choice cannot be reconsidered. At the opposite extreme, sampling from the full distribution preserves every tiny tail probability. That can introduce surprising language, but it can also select tokens that the model itself considered implausible.

Temperature changes the shape of the distribution before sampling. For logits $z_i$ and positive temperature $T$, probabilities are computed as

When $T<1$, dividing by a small number magnifies logit differences. The distribution sharpens, and high-ranked tokens dominate. As $T$ approaches zero, behavior approaches greedy decoding. When $T>1$, differences shrink and the distribution flattens, giving lower-ranked tokens more chance. A temperature of one leaves the original softmax unchanged. Temperature is not a creativity dial in any semantic sense; it is a precise rescaling of confidence.

Top-k sampling sorts tokens by probability and keeps only the highest $k$. All other probabilities become zero, and the survivors are renormalized before sampling. This prevents selection from the extreme tail. Its weakness is the fixed candidate count. At an obvious step such as “two plus two equals,” even ten candidates may be too permissive. In an open-ended list of names, ten may be unnecessarily restrictive.

Top-p sampling, also called nucleus sampling, adapts to the distribution. Sort tokens from most to least probable, then keep the smallest prefix whose cumulative probability reaches at least $p$. If the model is confident, a few tokens may cover 90 percent of the mass. If uncertainty is broad, many tokens may be needed. The candidate set therefore expands and contracts with context. Implementations differ in edge details, such as whether the token that crosses the threshold is always included, so reproducibility requires knowing the exact sampler.

Min-p sampling uses the most probable token as a reference. A common formulation removes candidates whose probability is below a fraction of the maximum probability. With threshold $\alpha$, keep tokens satisfying $p_i \geq \alpha p_{\max}$. This also adapts to confidence: when the best token is overwhelmingly likely, the cutoff rises; when the distribution is flat, more candidates survive. Min-p should not be confused with a fixed absolute minimum probability.

Samplers can be combined, and order matters. A runtime might apply temperature, top-k, top-p, min-p, repetition penalties, and other processors before renormalizing. The same labels do not guarantee identical outputs across libraries if operations, defaults, random-number generators, or numerical precision differ. A random seed helps reproducibility only when the entire execution path is deterministic.

Decoding parameters interact with the task. Factual extraction, code completion, and structured output often benefit from constrained or low-variance choices. Brainstorming can tolerate a wider candidate set. Long-form generation may need enough diversity to avoid loops without admitting the incoherent tail. There is no universally correct temperature. Parameters should be evaluated on representative prompts with task-specific quality measures, not chosen from folklore.

Sampling also does not add knowledge. If the model assigns negligible probability to the correct answer, top-p may remove it and higher temperature may merely elevate many wrong tail tokens alongside it. Conversely, a low temperature can make a confidently wrong mode even more dominant. Retrieval, tools, better training, or explicit constraints address different failure modes. Decoding only redistributes and selects among scores the model already produced.

For production systems, record the model version, tokenizer, full decoding configuration, seed policy, and runtime implementation. Measure not only average quality but failure rates, repetition, format validity, latency, and token usage. A setting that looks lively in one anecdote can be unstable across thousands of requests.

The reliable mental sequence is: the model outputs logits; temperature reshapes them; truncation methods remove candidates; renormalization restores a distribution; a pseudorandom draw chooses one token; the token joins the context; and inference repeats. Separating model scoring from decoder choice makes generation behavior much easier to reason about.