Objectives: causal LM, MLM, span corruption, FIM

1

Concept

Pretraining needs more than text and a network. It needs a rule that turns text into inputs, targets, and loss. That objective determines what each prediction is allowed to see and what mistake the optimizer corrects. Two models trained on the same corpus with different corruption and visibility patterns practice different skills.

Causal language modelling factorizes a sequence from left to right. At position $i$, the model predicts $x_{i+1}$ from $x_{\le i}$. A triangular attention mask prevents future leakage, while shifted targets provide a loss at almost every position. The objective matches autoregressive generation directly: inference supplies a prefix, predicts one token, and repeats. Its weakness is that a representation at an early position cannot use right context.

Masked language modelling (MLM), associated with BERT, selects input tokens for corruption and asks a bidirectional encoder to recover the originals. Ordinary positions can attend left and right, so the model builds full-context representations. Only selected targets contribute the primary reconstruction loss, making the learning signal less dense than next-token loss. The special corruption process also creates a difference between pretraining inputs and ordinary downstream text, which implementations mitigate with a mixture of mask, random, and unchanged selected tokens.

Span corruption, used by T5, removes contiguous spans rather than mostly independent tokens. Each missing region is replaced in the input by a sentinel marker. The decoder emits sentinel markers and the corresponding missing text. This trains an encoder–decoder to compress the visible source and generate variable-length missing passages. Contiguous gaps better resemble summarization or completion than isolated blanks.

Fill-in-the-middle (FIM) teaches an autoregressive decoder to generate a missing interior span. A document is divided into prefix, middle, and suffix. Special tokens rearrange the training sequence so the model receives prefix and suffix before predicting the middle. The underlying loss remains next-token cross-entropy; data transformation changes what “previous context” contains. This is especially useful for code, where a developer often edits inside an existing file.

The objectives are not interchangeable labels. Attention masks, special tokens, packing, target construction, and loss masks must agree. If an MLM target accidentally remains visible, the task leaks. If FIM boundary tokens are inconsistent with inference, the model cannot recognize the requested arrangement. Tokenizer configuration therefore belongs to the objective contract.

Mixtures are possible. A decoder can see ordinary causal examples and a chosen fraction of FIM-transformed examples. An encoder–decoder can mix span lengths or denoising patterns. Mixture weights decide how scarce compute is allocated between skills, so they should be logged and evaluated rather than treated as harmless preprocessing.

Objective quality also depends on data boundaries. Concatenating unrelated documents without a separator teaches transitions that never occur naturally. Masking across document boundaries creates artificial reconstruction. Loss on padding wastes signal. A sound pipeline carries document identity, boundary tokens, and loss masks all the way to the batch.

Evaluation should match the practiced contract and also probe transfer. Measure causal perplexity for left-to-right models, reconstruction on held-out corruptions, FIM exactness or code tests for infilling, and downstream tasks that matter. A lower training loss across different objectives is not directly comparable because targets and visible information differ.

The lasting question is: what prediction game consumes every unit of compute? Causal LM practices continuation; MLM practices bidirectional token recovery; span corruption practices conditional passage reconstruction; FIM practices insertion into known surroundings. Architecture constrains the possible games, and the objective selects the behavior optimization repeatedly rewards.