Modelling sequences: the setup

1

Concept

A sequence is an ordered collection $x_1,x_2,\ldots,x_T$. Order carries information: “dog bites person” and “person bites dog” contain the same words but describe different events. Sequence modelling asks how a shared system can process variable lengths, retain relevant context, and produce outputs whose relationship to positions depends on the task.

In sequence classification, many inputs produce one output, such as a sentiment label. In sequence labelling, each input position receives an aligned output, such as a part-of-speech tag. In sequence generation, a prefix produces a distribution over the next item repeatedly. Translation maps a source sequence to a target sequence with a different length. These shapes determine architecture, masks, and loss.

A basic recurrent formulation maintains state:

The same parameters $\theta$ are reused at every time step. This sharing lets the model apply learned patterns at different positions and to lengths not seen exactly during training. The hidden state $h_t$ is a learned summary, not a hand-written log of all previous tokens.

What context is legal depends on the application. A causal language model predicting position $t+1$ cannot use future tokens; training must enforce that boundary even when all targets are available in the file. A bidirectional encoder for classifying an already complete sentence may use left and right context. Using future information in an offline benchmark can silently invalidate a system intended for live prediction.

Sequences in a batch usually have different lengths. Padding fills a rectangular tensor, while a mask marks which positions are real. Padding must not alter hidden states, receive attention, or contribute to the loss unless explicitly intended. Packed representations or length-aware kernels can avoid some wasted computation. A length bug can produce excellent metrics by leaking labels or averaging over easy padding.

Training autoregressive models commonly uses teacher forcing: at step $t$, the model receives the actual previous training token rather than its own sampled output. This enables parallel target construction and stable learning, but inference conditions on the model’s earlier choices. Errors can change later context, creating exposure mismatch. Alternatives exist, but their benefits must be demonstrated rather than assumed.

State capacity creates a bottleneck. A fixed-size recurrent state must preserve every detail needed later while continually incorporating new input. Long sequences make credit assignment difficult because a distant event influences loss through many transformations. Attention later changes this setup by giving the decoder direct, content-dependent access to multiple stored states rather than relying on one compressed path.

Evaluation must match sequence structure. Token accuracy can hide poor whole-sequence validity. Averaging loss across padded and unpadded positions differently changes results. Generated sequences require decoding choices and may have several acceptable targets. Report directionality, context limit, truncation, masking, teacher-forcing policy, and how metrics aggregate. The setup is not boilerplate: it defines what information the model may use and what problem the score actually measures.

One reliable test constructs two examples identical up to time $t$ but different afterward. A causal representation at $t$ must remain identical in evaluation mode. If it changes, future information leaked through masking, preprocessing, normalization, or target construction.