Distillation: making small models punch up

01 · Concept

Concept

You have built something good on Qwen3.8-27B and now someone asks for it on a phone, offline. Do the arithmetic before answering. The weights are 54 GB in bf16. Lesson 7.10 pins the community Q4_K_M text-model artifact at 17.1 GB (15.93 GiB), before its separate multimodal projector, cache, and runtime memory. A flagship handset’s entire memory budget is below that practical requirement. More aggressive quantization can shrink storage further, but it never reduces the number of parameters or changes the architecture; once quality or runtime support makes the target infeasible, you need a smaller model. Lesson 9.7 works through edge hardware; this lesson is about how to train that smaller model.

Knowledge distillation is learned model compression: it trains a smaller student to imitate signals from a teacher, aiming to preserve useful behaviour at lower memory, latency, and serving cost. It is not lossless file compression or a re-encoding of the same weights; the student has a different parameterization and learns an approximation over a chosen input distribution.

The classic form uses teacher logits. A temperature $T$ softens the teacher’s distribution, and the student minimizes a divergence from it, usually mixed with ordinary loss on ground-truth labels:

The non-target probabilities carry relational information. If the teacher assigns moderate mass to two semantically related tokens and almost none to the rest, the student learns something a one-hot label never encodes. This route is normally closed for frontier teachers served behind an API, which is why most practical distillation is sequence-level. Qwen3.8-27B is the exception that makes the choice real: it ships open weights under Apache 2.0, so its logits over all 248,320 vocabulary rows are yours to read. That is a genuine advantage of an open-weights teacher, though it comes with a storage problem of its own — cached soft targets over a quarter-million-row vocabulary are enormous, so implementations usually keep only the top few hundred entries per position.

Sequence distillation is the other route: the teacher answers a curated prompt set, the outputs are filtered, verified, or ranked, and the student runs SFT on the survivors. Walk one build through, step by step, because the interesting decisions are all in the data generation.

First, pick the mode and its preset. If you want the student to produce direct answers, generate with instruct mode — enable_thinking set to False, temperature 0.7, top_p 0.80, top_k 20. If you want reasoning traces in the curriculum, generate under the thinking preset of temperature 1.0, top_p 0.95, top_k 20. Off-preset generation produces a teacher distribution the vendor never shipped, and you would be distilling a model nobody validated.

Second, verify before accepting. Wherever a checker exists — tests, an exact-answer comparison, a schema validator — run it and discard failures, exactly as lesson 6.8 argued for training signals. Filtering teacher output with the same teacher is circular approval and teaches its errors with extra confidence.

Third, price it. Suppose 200,000 prompts averaging 800 output tokens: that is 160 million output tokens. Served through Cloudflare Workers AI at USD 3.20 per million output tokens, generation costs about USD 512, plus input tokens at USD 0.45 per million. Illustrative and specific to August 2026 pricing, but the shape is the point — synthetic curricula are cheap enough to iterate on and expensive enough that generating ten times more data than you filtered for is a real waste.

Now the classic wrong turn, and it is tempting precisely because thinking mode makes it easy. The teacher produces long, articulate reasoning traces, so a team distils the traces into a small student expecting the student to reason too. What arrives is a student that has learned the form of a long trace without the capacity to run the search underneath it: fluent multi-step deliberation, confidently wrong conclusions, and a token bill several times higher than the direct-answer student it replaced. You paid for reasoning twice — once to generate the traces, once on every inference — and bought a worse model. The correction is to filter traces by outcome rather than by appearance, keeping only those whose final answer passed a checker, and to evaluate the student on accuracy per output token rather than on how much its traces resemble the teacher’s. Often the right answer is that a small student should not reason aloud at all, and should instead be taught the direct answers that the teacher’s reasoning produced.

Prompt coverage defines what transfers. A curriculum full of arithmetic improves arithmetic and leaves multilingual instruction-following exactly where it was. Easy prompts saturate without touching the student’s decision boundary; prompts far beyond the student’s capacity become noise. Adaptive generation targets the cases where student and teacher disagree, or where external verification can separate the strategies that worked.

Capacity imposes a real ceiling, and it does not show up in an average benchmark score. A smaller student typically retains frequent surface behaviour while losing rare knowledge, long-context competence, multilingual nuance, and robust composition of several instructions. Evaluate by slice — capability, difficulty, language, safety boundary, prompt distribution — and compare against a student trained on original data, so improvements can be attributed to the teacher signal rather than to extra tokens.

Distillation can be offline, from a fixed synthetic dataset, or online, querying the teacher as the student changes. Offline data is reproducible and cheap to reuse but goes stale as the student moves. Online teaching targets current errors at higher cost and worse reproducibility. Intermediate-feature distillation is available when architectures and activations are accessible, though matching internal representations tends to overconstrain a student whose geometry differs — and it is a poor fit here, since Qwen3.8-27B’s hybrid arrangement of 48 Gated DeltaNet layers and 16 full-attention layers is unlikely to match whatever the student is.

The techniques compose. Distil first, then quantize the student (lessons 7.9 and 7.10), or use teacher signals during quantization-aware training to recover what quantization cost. They solve different problems: distillation changes what a smaller model learns, quantization changes how its weights are represented, and the edge deployments in lesson 9.7 usually need both. The durable model is curriculum-driven behavioural transfer — the teacher supplies richer targets, the student compresses what fits inside its capacity, evaluation reveals what survived. A small model punches up not because it secretly contains the teacher, but because carefully chosen supervision spent its limited capacity on the behaviours that mattered.