The full block, assembled

1

Concept

The pieces now fit into one repeated interface. A decoder-only, pre-norm Transformer block receives a tensor $x$ with shape $B\times T\times d_{model}$ and returns a tensor of the same shape. Everything inside may expand, split into heads, and form square attention matrices, but the boundary stays stable so dozens or hundreds of blocks can stack.

Conceptually, the block is two equations:

Dropout or residual scaling may appear on the update branches. Post-norm architectures move normalization after each sum. The interface remains an existing state plus a learned update.

Open the attention branch. Normalized features are projected into $Q$, $K$, and $V$, reshaped into heads, and compared with scaled dot products. A causal mask removes future edges. Softmax turns each query row into routing weights, which mix values. Heads are concatenated, an output projection restores $d_{model}$, and the result is written into the residual stream.

Open the FFN branch. The updated stream is normalized, expanded into a wider dimension, transformed by an activation or gate, and projected back to $d_{model}$. This operation is shared across positions but does not exchange information between them. It recognizes and transforms feature combinations delivered by the stream.

The alternation is the heart of the architecture. Attention is token mixing: each position can receive information from other allowed positions. The FFN is channel mixing: each position can recombine its own features nonlinearly. Repeating the pair allows routes and transformations to compose. A first layer may collect a local syntactic cue; a later FFN can encode it; another head can carry that feature to a distant position.

Embeddings sit before the stack. Token IDs become vectors, and the model supplies position through an absolute embedding, RoPE, a relative bias, or another scheme. A final normalization often follows the stack. The language-model head then maps each final position to vocabulary logits. Weight tying may reuse the token embedding matrix for this output projection.

Shape tracing prevents many mistakes. With $B=8$, $T=512$, $d_{model}=768$, and 12 heads, each usual head has width 64. Scores have shape $8\times12\times512\times512$. After value mixing, the head tensor returns to $8\times12\times512\times64$, then concatenates to $8\times512\times768$. The FFN may expand the final dimension, but must project back before its residual addition.

Memory use does not follow parameter count alone. The attention score or its backward intermediates scale with $T^2$ in conventional implementations. FFN activations scale with $T d_{ff}$. Training must retain or recompute information for gradients, while decoding caches keys and values across generated positions. The same block stresses hardware differently in pretraining, prefill, and token-by-token decode.

A useful anatomy view should preserve causality and ownership: the residual stream is central; normalization reads it; attention and FFN are branches that return updates. Drawing the stream as if it flows through attention and loses its prior state obscures why residual learning works.

The assembled block is surprisingly compact. Its power comes from learned width, depth, data, and repeated composition rather than a long list of operations. Understanding the interface lets you read real implementations without mistaking fused kernels or renamed variables for different mathematics.