Causal masking & why order matters

1

Concept

A next-token language model learns the factorization

The probability for token $x_i$ may depend on earlier tokens, but not on $x_i$ itself or later tokens. During training, however, the complete sequence is already in memory. Without a constraint, self-attention at an early position could read the future answer and achieve a deceptively low loss. Causal masking makes the computation obey the autoregressive factorization.

For a sequence of length four, the allowed pattern is lower triangular:

Row $i$ is the receiving query position; column $j$ is the key being read. A one means the edge is allowed. A zero means it is forbidden. Implementations usually create an additive mask containing zero for allowed scores and negative infinity for forbidden scores. The mask is added to $QK^\top/\sqrt{d_k}$ before softmax. Since $\exp(-\infty)=0$, forbidden positions receive zero attention weight.

In finite-precision code, a library may use the most negative representable value rather than literal infinity. The intent is the same, but care matters: a merely “large negative” constant can be insufficient in unusual dtypes or after other transformations. Modern attention APIs often accept an explicit causal flag and handle the correct kernel-specific representation.

The mask provides a useful combination: sequential information flow with parallel training computation. All rows of the attention matrix are produced together, yet row three can only depend on columns one through three. An RNN enforces causality by its execution order. A Transformer enforces causality by its connectivity pattern. During generation the situation changes: the model truly receives only the existing prefix, produces one new token, appends it, and repeats.

Targets are shifted relative to inputs. If the input tokens are [BOS, cats, sleep], the corresponding targets can be [cats, sleep, EOS]. The hidden state at the first input position predicts “cats”; the state at “cats” predicts “sleep.” The mask lets a position attend to itself because that state is used to predict the next token, not the current input token. Off-by-one errors here can create label leakage or train a model against the wrong targets.

Do not confuse a causal mask with a padding mask. Padding is used when sequences of different lengths share a batch. Placeholder positions should neither contribute as keys nor count toward the loss. A causal mask hides future positions based on order; a padding mask hides invalid positions based on sequence length. A batch can require both. Packed-sequence training may need an additional document-boundary mask so one document does not attend into the previous packed document.

Causality is also different from the model knowing absolute order. If attention had no positional signal, swapping two earlier tokens would present the same set of content vectors. The triangular mask says which positions are earlier; positional encodings or biases provide richer information about where and how far apart they are.

Bidirectional encoders such as BERT use a different visibility pattern for masked-language modelling: ordinary tokens can attend both left and right, while selected input tokens are hidden or corrupted. Encoder–decoder models combine patterns: bidirectional attention in the encoder, causal attention in the decoder, and cross-attention from decoder positions to all encoder states.

The durable rule is simple: the training tensor may contain the future, but the computation graph for a prediction must not. A triangular mask turns that rule into matrix structure, preserving the Transformer’s parallel training advantage while protecting the legitimacy of the learning objective.