Supervised fine-tuning & instruction data

1

Concept

Supervised fine-tuning (SFT) takes a pretrained checkpoint and continues training it on demonstrations of desired behavior. A record might contain a system instruction, a user request, and an assistant response written or approved by a human. The goal is not to reteach grammar or world knowledge from scratch. It is to make the conversational contract, output patterns, and task boundaries statistically familiar.

The underlying loss is usually the same token-level cross-entropy used in causal pretraining. Given conversation tokens $x_1,\ldots,x_T$, the model minimizes negative log-probability for target tokens:

where $A$ commonly denotes assistant-response positions. System and user tokens remain in the input context, but their loss is masked. This distinction prevents the optimizer from spending capacity learning to reproduce the prompt and focuses the gradient on what the assistant should say. Some datasets deliberately train on multiple assistant turns, so the mask follows role boundaries rather than simply covering the final suffix.

An instruction example teaches more than its literal answer. It demonstrates how to interpret a request, which assumptions to state, what format to use, when to ask a question, and how to stop. A code example can teach fenced output and explanation order. A safety example can teach a bounded refusal followed by a benign alternative. A tool-use example can teach a structured call rather than prose. These conventions become part of the learned policy.

Data quality is therefore the central design problem. Correctness, diversity, provenance, and coverage matter. Thousands of near-duplicate templates can create the illusion of scale while teaching a brittle surface pattern. Verbose demonstrations can make the model verbose even when users ask for brevity. Answers containing fabricated citations reward fabrication. Contradictory labels teach unstable boundaries. Filtering must examine both prompts and responses, because a clean response paired with an ambiguous instruction may still teach the wrong behavior.

Mixture design also matters. If most examples are easy rewriting tasks, the model may become pleasant without becoming better at reasoning or instruction composition. If the mixture overweights one language, format, or refusal pattern, performance can drift elsewhere. Teams commonly hold out evaluation sets by task family and source, rather than randomly splitting duplicated examples that could leak across train and test.

SFT can cause catastrophic forgetting or narrower behavior when learning rates are too high, training continues too long, or data is too homogeneous. The checkpoint moves toward the supervised distribution and may lose probability on useful pretrained behaviors. Regularization, conservative optimization, broad mixtures, and evaluation against base capabilities help reveal this trade-off. Parameter-efficient methods reduce which weights change, but they do not eliminate bad supervision.

SFT is also not preference learning. A single target response says “imitate this,” but it does not show how it compares with plausible alternatives. When several answers are acceptable, token-level imitation may over-penalize harmless wording differences. Preference methods instead teach relative ranking, often after SFT has established a stable conversation policy.

A sound SFT pipeline records dataset lineage, chat template, tokenizer, masking policy, mixture weights, optimization configuration, and checkpoint ancestry. It tests factuality, task success, refusal precision, multilingual behavior, formatting, and retained base capabilities. The durable mental model is rehearsal: pretraining supplies general competence; curated demonstrations teach how that competence should appear in the product’s actual interactions.