Modelling sequences: the setup

01 · Concept

Concept

A model is reading a document and has arrived at token 40,000. To predict token 40,001 it may need something that happened at token 12, or at token 39,999, or nothing at all. The architecture cannot know in advance which. So it must decide, once and for all, what it is allowed to carry forward from everything it has already read.

That is the question this entire track turns on, and it is worth stating as bluntly as possible: how much state may a model carry between tokens? Not “which architecture is best” — that framing hides the trade. Every design in tracks 3 and 4, and the hybrid layout of this course’s specimen, is a different answer to the state question, and each answer buys one thing and pays for it with another.

Two extreme answers bracket the space.

The first says the state is a fixed-size vector. At every position the model computes a new state from the old state and the current token, and the old one is discarded:

The same parameters $\theta$ act at every position, which is what lets one model handle any length. Memory is constant, work per token is constant, and token 40,000 costs exactly what token 1 cost. Lesson 3.2 develops this design.

The second says keep one key/value record for every position. When predicting, use learned content scores to mix those records. This avoids compressing the whole past into one recurrent state and preserves a direct path to every stored position, but the projected records and their weighted blend do not guarantee exact retrieval. Memory grows with every token, and the work of consulting the store grows too. Lesson 3.5 introduces the mechanism, and track 4 builds the whole Transformer on it.

Now make the fixed-state limit concrete, because it is not a vague worry about “forgetting” — it is arithmetic. Consider a copy task: show the model a random sequence of tokens, then a delimiter, then require it to reproduce the sequence exactly. Take a vocabulary of 1000 items, so each random token carries $\log_2 1000\approx9.97$ bits of information, and take a state of 256 numbers stored in bf16, that is 512 bytes, or 4096 bits, as an absolute upper bound on what it can hold. A 100-token sequence to copy carries about 997 bits. That fits inside 4096, so the task is at least possible — whether training finds the solution is a separate question. A 1000-token sequence carries about 9970 bits. That does not fit. No architecture with this state size, no learning rate, no dataset, and no amount of patience produces a model that copies 1000 random tokens through a 512-byte channel, because the information is not there to be recovered.

Here is the classic wrong turn. A model fails on a task requiring information from far back, and the reflex is to train longer, widen the network, or blame the optimizer. Sometimes that is right. But before spending a week on it, do the counting: estimate how many bits the task requires to cross the gap, and compare against what the carried state can physically hold. If the required information exceeds the channel, stop tuning and change the architecture. If it fits comfortably and the model still fails, then you have a genuine learning problem — which lesson 3.2’s vanishing gradients will explain.

The other extreme has a cost too, and it is not subtle at production scale. Storing every position means memory that grows without bound. Lesson 7.2 works out the exact bill for Qwen3.8-27B: at its native 262,144-token context, the stored keys and values of a single sequence reach 16 GiB. Against that, the layers of the same model that carry a fixed recurrent state instead hold roughly 144 MiB in total in the reference float32 path, and that figure does not change whether the sequence is one hundred tokens or a quarter of a million (Qwen3.8-27B Model Card, 2026). Sixteen gibibytes against one hundred forty-four mebibytes, for the same sequence, inside the same model. Hold that contrast; it is the argument the rest of this course keeps returning to, and lessons 3.6, 4.15, and 4.16 explain how one model came to contain both answers at once.

Some housekeeping is genuinely part of the problem rather than around it. What context is legal depends on the task: a causal model predicting position $t+1$ may not see future tokens, and that boundary must be enforced during training even though every target sits in the file already. A bidirectional encoder classifying a complete sentence may read both sides. Using future information in an offline benchmark silently invalidates a system meant to run live, and it is one of the easiest mistakes to make and the hardest to notice.

Sequences in a batch have different lengths, so padding fills a rectangular tensor and a mask marks which positions are real. Padding must not change any state, receive any attention weight, or contribute to the loss. A masking bug can produce excellent metrics either by leaking labels or by averaging over easy padded positions, and the metric will look wonderful throughout.

Training autoregressive models normally uses teacher forcing: at step $t$ the model receives the true previous token rather than its own sample. This makes targets constructible in parallel and stabilizes learning, but inference conditions on the model’s own earlier choices, so an early error changes all later context. The mismatch is real and the alternatives are not free.

Finally, evaluation must match the structure. Per-token accuracy can hide poor whole-sequence validity; averaging loss across padded and unpadded positions differently changes the number; generated sequences need decoding choices and often have several acceptable targets. Report directionality, context limit, truncation, masking, and how metrics aggregate. A reliable leakage test is worth building once and keeping: construct two examples identical up to time $t$ and different afterward, and confirm that the causal representation at $t$ is bit-identical in evaluation mode. If it moves, future information reached it through masking, preprocessing, normalization, or target construction.