Distillation: making small models punch up

1

Concept

Large models can produce valuable data for smaller ones. Knowledge distillation trains a student model to imitate signals from a teacher, aiming to preserve useful behavior with lower memory, latency, or serving cost. Distillation is not file compression: the student has a different parameterization and learns an approximation over a selected distribution of inputs.

The classic form uses teacher logits. For an input, the teacher produces a probability distribution over classes or tokens. A temperature $T$ softens that distribution, exposing probabilities assigned to alternatives. The student minimizes a divergence between teacher and student distributions, often combined with ordinary loss on ground-truth labels:

The non-target probabilities contain relational information. If a teacher assigns moderate probability to two semantically related tokens and almost none to irrelevant ones, the student learns more than a one-hot answer reveals. Temperature and scaling conventions matter; the displayed formula is a common template, not the only implementation.

For closed teachers whose logits are unavailable, sequence distillation uses generated outputs as supervised targets. A teacher answers a curated prompt set; outputs are filtered, verified, or ranked; the student performs SFT on the resulting conversations. The teacher can also produce explanations, tool traces, preference pairs, critiques, or multiple diverse candidates. In this setting, data generation and validation become the distillation algorithm’s most important components.

Prompt coverage defines the transferred behavior. A curriculum focused on arithmetic may improve arithmetic while leaving multilingual instruction following untouched. Easy prompts can saturate without teaching the student’s decision boundary. Difficult prompts can become noise if the teacher fails or the student lacks capacity. Adaptive generation targets cases where the student disagrees with the teacher or where external verification can separate successful strategies.

Teacher outputs are not ground truth. Hallucinations, unsafe instructions, cultural bias, verbosity, and formatting quirks can all be copied. Filtering by the same teacher creates circular approval. Strong pipelines use external tools, trusted datasets, independent judges, human audits, and provenance records. Synthetic data should retain teacher version, decoding settings, prompt source, filtering decisions, and date.

Capacity imposes a real ceiling. A smaller student may learn frequent surface behavior while losing rare knowledge, long-context performance, multilingual nuance, or robust composition. Average benchmark scores can hide these losses. Evaluate slices by capability, difficulty, language, safety boundary, and prompt distribution. Compare against a student trained on original data so improvements are attributable to the teacher signal rather than extra tokens alone.

Distillation can be offline, using a fixed synthetic dataset, or online, querying the teacher as the student changes. Offline data is reproducible and cheap to reuse but becomes stale. Online teaching can target current student errors but costs more and complicates reproducibility. Intermediate-feature distillation is possible when architectures and activations are accessible, though matching internal representations can overconstrain a student with different geometry.

Distillation also complements quantization and pruning. Distill first, then compress the student; or use teacher signals during quantization-aware training to recover quality. These techniques solve different problems: distillation changes what a smaller model learns, while quantization changes how its weights and activations are represented.

The durable mental model is curriculum-driven behavioral transfer. The teacher supplies richer targets, the student compresses patterns within its capacity, and evaluation reveals what survived. A small model “punches up” not because it secretly contains the teacher, but because carefully selected supervision allocates its limited capacity to the behaviors that matter most.