"Attention Is All You Need" in context

01 · Concept

Concept

Start with the concrete problem of 2016-era machine translation. To translate a hundred-token sentence, a recurrent network reads token one, updates a hidden state, then reads token two, and so on: one hundred strictly sequential steps. Information from the first token reaches the hundredth only by surviving ninety-nine state updates. Training is slow because no step can start before the previous one finishes, and long-range signal is fragile because it travels through every intermediate step.

The obvious first fix — the one the field actually tried — was to make recurrence stronger: LSTM and GRU gates to protect the state, deeper stacks, bigger hidden vectors. It helped, but it attacked the wrong term. The gates improved what survives each handoff; they did nothing about the number of handoffs. The dependency chain, not the cell design, was the bottleneck. The correction was to remove the chain entirely.

Attention Is All You Need, published in June 2017, proposed an encoder–decoder architecture whose main route between sequence positions used attention rather than recurrence or convolution. Two properties follow, and a small worked example makes both visible. Take a four-token sequence. An RNN needs 3 sequential state updates before the last token is processed, and the path from token 1 to token 4 crosses all 3 of them. Attention instead computes a score for every allowed pair of positions: $4 \times 4 = 16$ scores, all produced together in one matrix multiplication, and the path from token 1 to token 4 is a single direct edge. Scale that to length $n$: the sequential depth stays constant while the pairwise work grows as $n^2$. The Transformer traded a sequential bottleneck for a quadratic one — a trade that made sense because accelerators are excellent at large parallel matrix multiplications and terrible at long dependent chains.

The slogan is memorable but easy to overread. A Transformer is not only attention. The original system contained token embeddings, positional encodings, multi-head attention, position-wise feed-forward networks, residual connections, layer normalization, an output projection, softmax, and a training objective. “All you need” meant that attention could replace recurrent and convolutional sequence-mixing layers in that design. It did not mean that embeddings, nonlinear transformations, optimization, or order information disappeared.

The encoder and decoder had different jobs: bidirectional self-attention in the encoder, causal self-attention plus cross-attention in the decoder. Later families kept different subsets — encoder-only BERT, decoder-only GPT, encoder–decoder T5 — and the decoder-only causal stack became the standard shape for large language models. Read any of these as two alternating operations: attention lets positions exchange information, and the feed-forward network transforms each position independently, with residual paths carrying the evolving state through both.

The quadratic cost was the price. For a sequence of length $n$, conventional attention forms on the order of $n^2$ pairwise scores per head, and that cost returns with force when context windows reach hundreds of thousands of tokens. Track 7 is largely about paying, dodging, or restructuring this bill.

This course studies one shipped 2026 model against that 2017 baseline, and the comparison is the cleanest summary of what lasted. Qwen3.8-27B keeps the three load-bearing ideas: attention as content-dependent routing, a feed-forward network at every layer, and a pre-norm residual stream running through all 64 layers. It replaces the uniform stack: instead of attention in every layer, it repeats a 3