A 60-second history: n-grams to the GPT-5 era

01 · Concept

Concept

Suppose it is 1990 and you have a large collection of English text and a simple goal: predict the next word. The obvious plan is to count. Take the two preceding words as context, look up how often each word followed that pair in your corpus, and divide. This is a trigram model and it is a perfectly respectable language model in the sense of lesson 0.1: it returns a probability distribution over the vocabulary given a context.

Work an example. Suppose your corpus contains the pair butter and eighty times, followed by jelly fifty-two times, jam nine times, sugar four times, and fifteen other continuations once each. The estimate for jelly is $52/80 = 0.65$, for jam $9/80 \approx 0.11$, and so on. The distribution is honest, inspectable, and required no training run. So far the counting approach looks not merely adequate but attractive.

Now give it the context quantum peanut. Your corpus has never contained that pair. The maximum-likelihood estimate tries to divide each zero continuation count by a zero context count: $0/0$, which is undefined, so no next-word distribution exists. A related failure occurs when the context was observed but a particular continuation was not: that continuation receives probability zero, and one zero factor annihilates the probability of any sequence containing it. Smoothing repairs both cases by reserving probability mass for unseen events; backoff uses the shorter context peanut when the longer one is missing. Production speech recognizers ran on smoothed n-grams for two decades.

But smoothing patches the symptom. The deeper defect is that the table cannot generalize sideways. Having learned a great deal about the context the cat sat on the, a trigram model knows precisely nothing about the dog sat on the, because those are different table keys. Nothing in the representation says the two contexts are almost the same. And the table grows exponentially: with a vocabulary of a hundred thousand words, a four-gram table has $10^{20}$ conceivable entries, of which your corpus fills a vanishing fraction. Larger context meant an exponentially emptier table. That is the wall.

Bengio and colleagues walked through it in 2003 by replacing the table with learned vectors. Each vocabulary item gets a continuous representation; a neural network maps a window of those representations to a distribution. Because cat and dog can be assigned nearby vectors, evidence gathered about one transfers to the other automatically, without anybody writing a rule. This paper also fixed the shape of everything that followed: embeddings in, a parameterized function in the middle, logits over the vocabulary, softmax, and training by maximizing the probability of the observed text. Every model in this course, Qwen included, still has exactly that silhouette.

Recurrent networks then removed the fixed window, at least on paper. At each step an RNN mixes the current token with a hidden state carried from the previous step, and that state can in principle summarize any amount of prefix. In practice, gradients shrink or explode across long chains of updates, so the early parts of the summary decay. LSTM and GRU cells added gates that decide what to keep and what to overwrite, which helped a great deal. What could not be fixed was the sequencing: step twenty needs the state from step nineteen, so training could not use a large machine efficiently.

Attention arrived first as a repair inside machine translation, letting a decoder consult many encoder states instead of compressing a whole source sentence into one vector. The 2017 Transformer promoted it to the central operation and deleted recurrence entirely. Every position forms queries, keys, and values; every position can consult every earlier position; and because the whole thing is matrix multiplication, training runs across all positions at once. Causal masking keeps a language model from peeking at the answers even while all positions compute in parallel.

Scaling did the rest. Parameters, data, context length, and training infrastructure all grew, and post-training methods such as instruction tuning and preference optimization turned a raw predictor into something that behaves like an assistant. Labels such as the GPT-5 era are marketing chronology, not scientific boundaries; deployed products bundle models with routers, tools, safety layers, and caches, and vendors disclose different amounts of it.

Which brings the story to where this course picks it up. In August 2026 the Qwen team released Qwen3.8-27B under Apache 2.0: a dense model of roughly 27 billion parameters that accepts text, images, and video and emits text, with a native context of 262,144 tokens. Two of its properties matter for this history. It is multimodal, so the sequence it attends over is no longer purely linguistic. And it is hybrid: rather than 64 identical attention blocks, it interleaves full attention with a recurrent-state mechanism called Gated DeltaNet, which is a direct descendant of the RNN idea that attention supposedly retired. The wheel did not turn backwards. The recurrent state came back because attention’s cost grows with context and a fixed-size state’s does not, and at a quarter of a million tokens that difference decides what can be served at all. Lesson 4.16 tells that story properly.

Notably, there is no dedicated technical report for this model, only a model card. Lesson 0.8 treats that absence as its subject rather than an inconvenience.