Sampling: temperature, top-k, top-p, and min-p

01 · Concept

Concept

The Qwen3.8-27B model card recommends two sampling configurations; it does not mandate them. Thinking mode uses temperature 1.0, top-p 0.95, top-k 20, min-p 0.0, presence penalty 0.0, and repetition penalty 1.0. Instruct mode uses 0.7, 0.80, 20, 0.0, 1.5, and 1.0 respectively. Framework support varies, which the card states explicitly. This lesson explains what the controls do and presents a testable hypothesis for the different temperature choices; the model card does not publish the vendor’s causal rationale.

Start from what the model hands the sampler. After the final layer, the language-model head projects the last position’s hidden state into one logit per vocabulary entry — 248,320 unrestricted scores. Softmax turns them into a distribution; decoding is the family of rules that then picks one token. Greedy decoding always takes the argmax: deterministic, cheap, and prone to repetitive, locally-safe text. Sampling from the untouched distribution preserves every tail token — including ones the model itself scored as implausible.

Temperature rescales logits before softmax. For logits $z_i$ and $T>0$,

Watch it work on two candidates with logits 2.0 and 1.0. At $T=1$ the gap is $1.0$, so the leader’s probability is $1/(1+e^{-1.0}) = 1/1.368 \approx 0.731$. At the instruct preset’s $T=0.7$ the gap becomes $1.0/0.7 \approx 1.429$, giving $1/(1+e^{-1.429}) = 1/1.240 \approx 0.807$. Cooling from 1.0 to 0.7 moved the leader from 73 percent to 81 percent — a precise sharpening of confidence, nothing more. Below 1 sharpens toward greedy; above 1 flattens; exactly 1 leaves the model’s distribution untouched.

Top-k keeps only the $k$ highest-probability tokens and renormalizes. Both recommended configurations set $k=20$: every draw happens among at most twenty survivors out of 248,320 — about 0.008 percent of the vocabulary. Top-p (nucleus sampling) adapts instead of fixing a count: keep the smallest prefix of the sorted tokens whose cumulative mass reaches $p$. A confident distribution may cross $p=0.95$ with two tokens; a flat one may need many (still capped at 20 here by top-k). Min-p keeps tokens with $p_i \geq \alpha\, p_{\max}$, a cutoff relative to the leader; Qwen sets it to 0.0, effectively disabling that extra filter. The recommended presence penalty differs materially: 0.0 for thinking and 1.5 for instruct. The card says it can reduce endless repetition but may also cause language mixing and a slight performance decrease. Repetition penalty stays neutral at 1.0 in both modes.

Why temperature 1.0 for thinking mode? The card does not say. One plausible hypothesis starts from degeneration research: greedy decoding can yield repetitive loops, while a longer reasoning trace may benefit from exploring alternatives that later tokens can revisit. Top-k 20 and top-p 0.95 still fence off much of the tail. The instruct configuration instead sharpens the distribution ($T=0.7$), tightens the nucleus ($p=0.80$), and adds presence penalty 1.5. That mechanism is coherent, but it is not a vendor-documented explanation and it does not guarantee self-correction. Treat it as a hypothesis: compare task success, truncation, latency, and variance under a fixed output budget before choosing a deployment default.

Order of operations matters in practice. A runtime may apply temperature, top-k, top-p, penalties, and other processors in sequence before renormalizing, and identical labels do not guarantee identical outputs across libraries if the order, edge cases, or RNG differ. A seed reproduces output only when the whole execution path is deterministic. When you deploy the model through vLLM, SGLang, or Workers AI, pin the preset explicitly rather than trusting defaults — and record model version, tokenizer, and the full decoding configuration with every request you might need to replay.

The reliable mental sequence: the model outputs 248,320 logits; temperature reshapes them; truncation removes candidates; renormalization restores a distribution; a pseudorandom draw picks one token; the token joins the context; inference repeats. Separate the model’s scoring from the decoder’s choosing, and the two presets stop looking like folklore and start looking like two deliberate positions on one dial.