"Attention Is All You Need" in context

1

Concept

In 2017, sequence modelling was dominated by recurrent networks. An RNN reads one position, updates a hidden state, then passes that state to the next position. LSTMs and GRUs made the state more controllable, and attention already let a translation decoder consult encoder states instead of squeezing a sentence into one vector. The limitation was not that recurrence could never work. It was that its dependency chain made training hard to parallelize and made distant information travel through many sequential updates.

The paper Attention Is All You Need, published in June 2017, proposed an encoder–decoder architecture whose main route between sequence positions used attention rather than recurrence or convolution. During training, every position in a layer could be computed together with matrix operations. The path between two positions also became short: information could move directly through an attention edge instead of crossing every intervening recurrent step.

The slogan is memorable but easy to overread. A Transformer is not only attention. The original system contained token embeddings, positional encodings, multi-head attention, position-wise feed-forward networks, residual connections, layer normalization, an output projection, softmax, and a training objective. “All you need” meant that attention could replace recurrent and convolutional sequence-processing layers in that design. It did not mean that embeddings, nonlinear transformations, optimization, or order information disappeared.

The encoder and decoder had different jobs. Each encoder layer used bidirectional self-attention: a source position could read every source position. Each decoder layer used causal self-attention so a target position could not see later target tokens, followed by cross-attention into encoder outputs. This supported machine translation, the paper’s central task. Later families kept different subsets. BERT popularized encoder-only masked modelling; GPT-style models use a decoder-only causal stack; T5 retained an encoder–decoder form.

Why did the design scale so effectively? Parallelism is part of the answer, not all of it. Accelerators are excellent at large matrix multiplications, and attention expresses many comparisons as dense matrices. Residual connections provide a stable route through deep stacks. Feed-forward blocks supply substantial nonlinear capacity at every position. Multiple heads let the layer form several learned routing systems. None guarantees intelligence, but together they make a highly trainable architecture that can absorb more data and compute.

There is a cost. Conventional self-attention forms a score for every allowed pair of positions, so its score matrix grows quadratically with sequence length. For a sequence of length $n$, there are on the order of $n^2$ pairwise scores per head. Recurrence has a sequential path but does not materialize that same square matrix. The Transformer therefore exchanged one bottleneck—sequential dependency—for another—attention memory and computation at long context. Later kernels and architectures attack that cost.

The 2017 results also should not be treated as a timeless leaderboard claim. Hardware, datasets, evaluation, and architectures have changed. The durable fact is architectural: a model can represent sequence relationships with content-dependent routing while processing training positions in parallel. That idea became a platform on which later work changed normalization, positional schemes, objectives, scale, and deployment.

Read the block as two alternating operations. Attention lets positions exchange information. The feed-forward network transforms the information independently at each position. Residual paths carry the evolving state through both operations, while normalization controls scale. Stack this pattern, train it to predict missing or next tokens, and a representation can progress from surface identity toward context-sensitive features.

The best historical interpretation is neither “one paper invented modern AI” nor “it was only a small engineering tweak.” The Transformer recombined known ideas into a system whose parallel training path matched accelerator hardware and whose interfaces were unusually reusable. The rest of this track opens that system part by part, then puts it back together and implements a small causal language model.