A 60-second history: n-grams to the GPT-5 era

1

Concept

Language modelling is older than modern neural networks. The enduring task is to assign probabilities to sequences, or equivalently to estimate a next item from what came before. What changed across generations was the machinery used to represent context and share evidence. The history is therefore not a parade of unrelated inventions. It is a sequence of answers to one question: how can a model use more context without making estimation or computation impossible?

An n-gram model answers with counts. A trigram model estimates the next word from the previous two. If “peanut butter and” is frequent in a corpus, “jelly” may receive high probability. Smoothing reserves probability for combinations not observed in training. The approach is inspectable and can work well when data matches the task, but the table is sparse: similar contexts do not automatically share knowledge, and a fixed window cannot distinguish histories that differ further back.

Neural language models replaced most of that table with learned continuous representations. Bengio and colleagues’ 2003 model mapped vocabulary items to vectors and fed a fixed window through a neural network. Similar words could acquire nearby representations, letting evidence transfer between combinations. This introduced the basic pattern still visible today: embeddings, a parameterized function, logits over a vocabulary, softmax, and training by maximizing the probability of observed text.

Recurrent neural networks removed the fixed input window in principle. At each step an RNN combines the current token with a hidden state carried from the preceding step. The state can summarize an arbitrarily long prefix, but in practice gradients can vanish or explode across many recurrent updates. LSTM and GRU architectures added gates that help preserve or replace information. Recurrent computation also makes training inherently sequential: step twenty depends on the state produced at step nineteen.

Attention first became prominent in neural translation as a way for a decoder to consult different encoder states instead of squeezing an entire source sentence into one vector. The 2017 Transformer then made attention the central sequence-mixing operation and removed recurrence. During training, all positions can form queries, keys, and values through large matrix operations. Causal masking prevents a language model from seeing future answers even though the computations for positions run in parallel.

GPT-style models use the decoder side of this architecture for autoregressive prediction. Successive generations expanded parameter counts, datasets, context windows, training infrastructure, and post-training methods. Instruction tuning and preference optimization changed how a base predictor behaves as an assistant. Tools, retrieval, multimodal inputs, and reasoning-oriented training added capabilities around the same core interface: given tokens so far, produce logits for what may come next.

Names such as “the GPT-5 era” are historical labels, not clean scientific boundaries. Public systems combine models, routers, tools, safety layers, caches, and product logic, and vendors disclose different amounts. Parameter count alone therefore cannot describe capability. Dataset quality, compute allocation, architecture, inference budget, and evaluation design all matter. A timeline should resist turning marketing releases into a simple law of progress.

The useful historical lens is continuity plus changing representation. Counts represented local regularities explicitly. Neural networks compressed them into shared parameters. Recurrence carried context through a state. Attention exposed pairwise routes between positions. Scaling made these mechanisms broadly capable, while post-training made them easier to direct. None of those steps converts probability into guaranteed truth. The objective can learn remarkable structure, but factuality and reliability still require evidence, evaluation, and often external tools.