Supervised fine-tuning & instruction data

01 · Concept

Concept

Here is the artifact, before the theory. When you call Qwen3.8-27B you do not hand it a string; you hand it a conversation, and a chat template turns that conversation into one flat token sequence. Roles are rendered as marker tokens — system, user, assistant — around your text. By default the assistant turn begins with a delimited thinking segment before the answer proper. And the model card documents a request-level switch, enable_thinking set to False, which suppresses that segment. All of it — the markers, the segment, the switch — is post-training output. None of it exists in a pretrained checkpoint, and none of it is architecture. Those role markers are ordinary token ids drawn from the same 248,320-row vocabulary every other token comes from.

Supervised fine-tuning (SFT) is where most of that protocol gets installed. It takes a pretrained checkpoint and continues training it on demonstrations of the desired behaviour: a system instruction, a user request, and an assistant response written or approved by a human. The goal is not to reteach grammar or world knowledge. Lesson 6.1 argued the capability is already present; SFT makes the conversational contract, the output patterns, and the task boundaries statistically familiar so the right behaviour becomes the default with no in-context demonstration.

The loss is the same token-level cross-entropy used in causal pretraining, with one modification that carries most of the weight. Given conversation tokens $x_1,\ldots,x_T$,

where $A$ denotes the assistant-response positions. System and user tokens stay in the input context — the model must attend over them — but their loss is masked.

Walk one record through the arithmetic, because the masking is where implementations quietly go wrong. Suppose the rendered conversation tokenizes to 460 positions: 18 template marker tokens, 96 system tokens, 226 user tokens, and 120 assistant tokens. The forward pass runs over all 460. The backward pass sums log-probabilities over 120 of them. Every gradient the optimizer sees therefore answers a single question — given everything before it, was this the right next token of the assistant’s reply? — and never the question of what the user should have asked. If the conversation has three assistant turns, the mask follows role boundaries and covers all three, not merely the final suffix.

Now the classic wrong turn: train on all 460 positions with no mask, on the theory that more supervised tokens must mean more learning. It trains fine. The loss drops. Then the deployed model finishes its answer and cheerfully writes the next user question, because you spent 340 of every 460 gradient signals teaching it to be a good predictor of user turns and template markers. The fix is not more data, it is the mask. Roughly three quarters of the tokens in a typical instruction record are context to be conditioned on, not behaviour to be imitated.

An instruction example teaches far more than its literal answer. It demonstrates how to read a request, which assumptions to surface, what format to use, when to ask a clarifying question, and — critically — when to stop. A safety example teaches a bounded refusal followed by a benign alternative. A tool-use example teaches a structured call instead of prose. For a model like Qwen3.8-27B that ships two modes in one set of weights, the demonstrations must cover both branches: conversations that contain a thinking segment and conversations that do not, so that the switch actually selects a well-practised behaviour instead of pushing the model off its training distribution.

Data quality is therefore the central design problem, and it is not a volume problem. Thousands of near-duplicate templates create the illusion of scale while teaching a brittle surface pattern. Verbose demonstrations make the model verbose even when users ask for brevity. Answers with invented citations reward invention. Contradictory labels teach unstable boundaries. Filtering has to inspect prompts as well as responses, because a clean answer paired with an ambiguous instruction still teaches the wrong mapping. Mixture design matters for the same reason: if most examples are easy rewriting tasks, the model becomes pleasant without becoming better at reasoning or at composing several instructions. Hold out evaluation sets by task family and source, since a random split of duplicated records leaks across train and test.

SFT also risks catastrophic forgetting when learning rates run high, training runs long, or the data is too homogeneous. The checkpoint slides toward the supervised distribution and loses probability mass on useful pretrained behaviour. Conservative optimization, broad mixtures, and evaluation against base capabilities expose the trade-off. Parameter-efficient methods (lesson 6.3) shrink which weights move, but they do not rescue bad supervision.

Finally, SFT is not preference learning. A single target response says “imitate this” without saying how it compares to plausible alternatives, so token-level imitation over-penalizes harmless wording differences whenever several answers would have been fine. Preference methods teach relative ranking instead, and they normally run after SFT has established a stable conversation policy. A sound pipeline records dataset lineage, chat template, tokenizer, masking policy, mixture weights, optimizer configuration, and checkpoint ancestry — because when behaviour regresses six weeks later, exactly one of those changed.