Objectives: causal LM, MLM, span corruption, FIM

01 · Concept

Concept

You have a cluster, a corpus, and a 27-billion-parameter network. Nothing yet tells the optimizer what a mistake is. Before a single gradient exists, someone must write a rule that turns raw text into inputs, targets, and a loss. That rule is the pretraining objective, and it decides two things: what each prediction is allowed to see, and which error gets corrected. Two models trained on the same corpus under different objectives practice different skills, and no amount of compute converts one skill into the other.

Start with the objective that Qwen3.8-27B actually uses, because it is the simplest one and the whole course rests on it. It is causal language modelling. The sequence is factorized left to right: at position $i$ the model predicts $x_{i+1}$ from $x_{\le i}$. A triangular mask blocks future leakage, and shifted targets supply a loss at nearly every position. That is exactly the mini-GPT you built in lesson 4.13, scaled up. The output distribution has one class per row of the output embedding, and this model’s is 248,320 rows wide, untied from the input embedding — the model card and config give both matrices roughly 1.271 billion parameters each.

Work through what that vocabulary size means for the loss. Cross-entropy is $-\ln p(\text{correct token})$. A model that has learned nothing and spreads mass uniformly over 248,320 classes assigns each one $1/248{,}320$, so its loss is

Run one step of the same arithmetic backwards from a plausible converged value. For a trained model at mean loss 2.0 nats, $e^{-2.0}\approx 0.135$ is the geometric mean of the probabilities assigned to the correct next tokens — roughly one in seven as an effective probability — while uniform prediction gives $1/248{,}320\approx 0.000004$. The ratio is about 33,000 in geometric-mean assigned probability. It is not the arithmetic mean of per-token probabilities.

Two model-card facts extend the picture for this specimen. First, it is trained with multi-token prediction (MTP): alongside the next token, additional heads predict tokens further ahead. Second, it is natively multimodal — image and video patches, produced by the vision tower of lesson 4.17 and projected into the same 5120-dimensional stream, sit in the sequence beside text tokens and are conditioned on identically. The prediction target remains text.

The alternatives are worth knowing because they define what causal LM gives up. Masked language modelling, associated with BERT, corrupts selected input tokens and asks a bidirectional encoder to recover the originals; ordinary positions attend both directions, producing full-context representations, but only selected targets carry the reconstruction loss, so the signal is sparser than next-token loss. Span corruption, used by T5, removes contiguous regions marked by sentinels and trains an encoder–decoder to emit the missing passages, which resembles summarization or completion more than isolated blanks do. Fill-in-the-middle keeps next-token cross-entropy but rearranges each document into prefix, suffix, then middle, so an autoregressive decoder learns insertion — the shape of most real code editing.

These are not interchangeable labels. Masks, special tokens, packing, target construction, and loss masks have to agree. A visible MLM target leaks the task. FIM boundary tokens that differ between training and inference make the arrangement unrecognizable. Concatenating unrelated documents without separators teaches transitions that never occur. Loss computed on padding wastes signal. Tokenizer configuration is therefore part of the objective contract, not preprocessing that happens beforehand.

Mixtures are possible and common: a decoder can see ordinary causal examples plus a fraction of FIM-transformed ones. Mixture weights decide how scarce compute is split between skills, so they belong in the run log and the evaluation, not in a preprocessing script nobody reads. Evaluate against the practiced contract — causal perplexity for left-to-right models, reconstruction for corruption objectives, infilling exactness or code tests for FIM — and remember that losses under different objectives are not comparable, because the targets and the visible information differ.

The lasting question is the one to carry into the rest of this track: what prediction game consumes every unit of compute? For this model the answer is next-token prediction over 248,320 classes, with MTP heads looking further ahead and vision patches sharing the sequence. Architecture constrains which games are possible; the objective picks the one that gets rewarded on every training token there was — however many that turned out to be, which this model has never published.