Reasoning models: test-time compute & long CoT

01 · Concept

Concept

Three facts about Qwen3.8-27B, taken straight from its model card, contain most of what this lesson has to teach. Thinking mode is the default. It has its own sampling preset — temperature 1.0, top_p 0.95, top_k 20 — while instruct mode uses temperature 0.7, top_p 0.80, top_k 20. And a caller can switch the thinking segment off per request by setting enable_thinking to False. Test-time compute has stopped being a research technique and become a shipped product surface with a default, a configuration, and an off switch. Each of those three decisions is defensible, and each one costs something.

The extra wrinkle is that “thinking mode” is not one portable setting. In Qwen’s pinned open chat template, enable_thinking defaults to on, unset reasoning_effort renders the same dedicated instruction as xhigh, low and xhigh have their own effort instructions, medium is accepted but does not inject a dedicated medium instruction, and high is rejected. preserve_thinking also defaults to preserving previous reasoning in the rendered conversation, while enable_thinking=false suppresses the thinking segment cleanly. Those are template semantics, not universal semantics of every hosted Qwen service. If a benchmark says only “Qwen3.8-27B, reasoning medium,” it has not yet described the experiment.

Start with the axis itself. Traditional scaling spends compute before deployment: train a larger model on more data. Test-time compute spends it afterward, on the specific problem in front of the system — more tokens, several sampled solutions, a search over a tree, tool calls, self-critique, verifiers. Quality can sometimes improve without touching the checkpoint at all. The 2024 work by Snell and colleagues framed this as an allocation problem: different strategies and budgets suit different problem difficulties, and compute should go where it pays. That is a claim about allocation, not a promise that inference scaling always beats a bigger model.

The simplest pattern is best-of-$N$: generate $N$ candidates and select one with a reward model, an exact checker, or another scorer. The selector is decisive — if it cannot recognize correctness, more candidates simply produce more persuasive errors. Correlated samples cap the gains, too: ten near-identical chains are not ten attempts. Sequential methods spend the budget differently, letting one trajectory propose a plan, execute part of it, detect a contradiction, backtrack, and revise. Tool-using systems spend it on retrieval, code execution, or symbolic solvers. “Reasoning tokens” is not one algorithm.

Now the question the presets pose. Why would a vendor sample the reasoning trace hotter than the final answer? The intuition most people bring is that reasoning is the delicate part, so it should be the conservative part. That reads the situation backwards, and working through the arithmetic shows why.

Temperature rescales logits before the softmax. Take two candidate continuations at some step, with logits 4.0 and 3.0 — one obvious next move and one plausible alternative. Considering only these two, at temperature 0.7 the gap of 1.0 is divided by 0.7 and becomes about 1.43 in logit units, so the odds ratio is $e^{1.43}\approx 4.17$ and the alternative takes about 19 percent of the mass. At temperature 1.0 the gap stays 1.0, the odds ratio is $e^{1.0}\approx 2.72$, and the alternative rises to about 27 percent. That is a relative increase of roughly 39 percent in the chance of exploring the second branch, from a change that looks small on the dial.

In a reasoning trace, that extra 8 points of probability is the entire product. The trace is a search, and a search that always takes the modal step is not searching. More importantly, a mistake inside a trace is recoverable: the model reads its own previous tokens, and a wrong turn at step four can be contradicted at step nine, with the final answer conditioned on the whole corrected sequence. A mistake in the final answer has no downstream repair. Nothing comes after it. So the vendor pushes temperature up exactly where errors are cheap and pulls it down where they are terminal. That is the principle, and it generalizes far beyond this model.

One detail keeps temperature 1.0 from being reckless: both presets hold top_k at 20. Out of a vocabulary of 248,320 rows, only twenty candidates are ever eligible at any step, and top_p then trims further — to 0.95 in thinking mode, 0.80 in instruct. Raising temperature redistributes mass within a hard-truncated shortlist; it does not open the tail. Thinking mode is not permissive sampling, it is a slightly flatter distribution over a very small set.

Which sets up the classic wrong turn. A team wants maximum accuracy and reproducibility, so they decode the thinking segment greedily at temperature 0. Two things break. First, long greedy traces are prone to repetition loops and to committing irreversibly to the first branch, which is precisely the failure mode a trace exists to avoid. Second — and this one is fatal — if they layer best-of-8 on top for extra accuracy, all eight samples are byte-identical. They pay eight times the tokens for one answer. Lesson 6.8 showed the training-side twin of this: a GRPO group in which every sample earns the same reward has zero variance and produces no gradient. Diversity is not a side effect of sampling in either regime; it is the resource being spent. The correction is to use the published preset for the mode you are running, and to introduce determinism, if you need it, through a fixed seed rather than through a collapsed distribution.

Long chain-of-thought is the workspace this compute buys: an intermediate textual trace that decomposes a problem and leaves working memory in the context. Post-training on verifiable rewards makes extended traces more useful, because successful strategies get reinforced without anyone authoring the path. But the trace is still generated text. It can rationalize a guess, conceal an invalid step, or repeat itself with perfect fluency. Trace length is a cost, not a quality metric.

The evidence Qwen offers for the mode is a reported GPQA Diamond score of 89.2 — vendor-reported, from the model card, and awaiting independent reproduction. It is a graduate-level science benchmark of exactly the kind where extended reasoning should pay, which makes it a reasonable place to look for the mode’s value and a poor place to stop looking. Without a technical report, without an independent run, and without knowing the harness, a single number establishes a claim rather than a fact.

An independent Qwen3.8-27B deployment study makes the operating-point problem concrete. On one 50-question hard pack with a 4,096-token ceiling, thinking off scored 68 percent with one truncation, low scored 94 percent with no truncations, and xhigh scored 88 percent with five truncations. Looking only at xhigh answers that actually finished, it was 44 out of 45; the paired comparison did not support the slogan “low is smarter than xhigh” (McNemar p = 0.25). The better reading is narrower: under that fixed output budget, low finished the work and xhigh often ran out of page. On easy prompts in the same campaign, thinking off had p50 latency around 0.93 s, while low, unset/default, and xhigh clustered around 8.35-8.51 s. And preserving about 50,000 characters of old reasoning inflated a later prompt to 23,654 tokens versus 55 tokens with preservation disabled. Those are serving facts, not changes to the weights.

The off switch is a cost lever as much as a behaviour lever. Thinking tokens are output tokens, and output tokens are the expensive kind — on Cloudflare Workers AI this model bills at USD 3.20 per million output tokens against USD 0.45 per million input tokens, as of August 2026. A thinking segment that runs several thousand tokens before a two-hundred-token answer inverts the economics of a simple request. Thinking tokens also occupy the KV cache at the 64 KiB per token derived in lesson 7.2, and they extend the request’s occupancy of a serving slot. Setting enable_thinking to False on routine work is not a downgrade; it is spending the budget where it earns something.

Serving reasoning workloads changes the operational picture as well. Request durations become highly variable, output lengths unpredictable, and branching uneven. Static batching turns inefficient, and schedulers need cancellation, priorities, and hard budget caps — a runaway trace is simultaneously a cost incident and a denial-of-service vector. Evaluation has to change too: plot success against generated tokens, latency, and money; separate proposal quality from selection quality by asking whether any candidate was right and whether the selector picked it; and test robustness to reworded prompts and adversarial distractors, or a system will look like it reasons better when it has merely recognized a pattern in your benchmark.

The durable model is compute allocation under uncertainty. Extra inference work creates opportunities to search and to verify; it does not manufacture truth. The proposal mechanism explores, the verifier discriminates, the scheduler decides what the answer is worth — and long chain-of-thought is one possible workspace inside that system, not the definition of reasoning.