Encoder-only, decoder-only, encoder–decoder

1

Concept

“Transformer” describes a toolkit, not one visibility pattern. The three classic families—encoder-only, decoder-only, and encoder–decoder—reuse attention and FFN blocks but connect information differently and train on different prediction problems. Those choices create different defaults for understanding, generation, and conditional transformation.

An encoder-only model lets every ordinary position attend left and right across the available input. BERT’s masked-language objective replaces or corrupts selected tokens and trains the model to reconstruct them using both sides. The output is a contextual representation for each input position rather than an inherently autoregressive generator. Classification can read a pooled position; tagging uses per-token states; retrieval systems can derive embeddings. Generation is possible with special procedures, but it is not the architecture’s native factorization.

A decoder-only model uses causal self-attention. Position $i$ reads only positions up to $i$, and the state predicts token $i+1$. Training can evaluate losses for all positions in parallel under a triangular mask. Inference appends one sampled or selected token and repeats. GPT-style models belong here. A prompt and its continuation simply form one token sequence, so tasks can be represented as text completion.

An encoder–decoder model has two stacks. The encoder reads the source bidirectionally. The decoder reads prior target tokens causally and contains an extra cross-attention sublayer. Decoder queries come from the evolving target; keys and values come from final encoder states. Translation is the canonical example: the full source sentence is available, while the target must still be generated left to right.

The distinction can be expressed through masks. Encoder self-attention usually has full visibility except padding. Decoder self-attention is lower triangular. Cross-attention lets every target query read valid source keys, because the source is already known. Packed data and prefix objectives can create hybrid masks, so the family label alone does not reveal every edge.

Objectives matter as much as topology. A bidirectional encoder trained on token corruption learns to recover hidden content. A causal decoder learns a normalized distribution over continuations. T5 casts many tasks as text-to-text span corruption, replacing spans with sentinel tokens and decoding the missing text. Architecture and objective jointly decide what training signal reaches each position.

Computational tradeoffs follow. An encoder processes its source once. An encoder–decoder system can cache encoder outputs while generating the target, but the decoder pays both self-attention and cross-attention. A decoder-only system may repeat prompt representations across output tasks, though KV caching avoids recomputing all prior keys and values during one generation. Parameter allocation between stacks also affects quality and serving cost.

Prefix language modelling blurs the boundary: a single stack may permit bidirectional attention within an input prefix and causal attention in the generated suffix. Retrieval and multimodal systems can add cross-attention modules to a mainly decoder-only backbone. Real architectures are compositions, but the three families remain useful reference points.

When choosing a family, begin with the information contract. Is the complete input known before any output? Must the model produce a free-form sequence? Should each input token receive a contextual feature? How many times will one encoded source be reused? These questions are more reliable than fashion.

The family tree is therefore a tree of visibility and objectives. Encoder-only models read a complete field; decoder-only models extend a prefix; encoder–decoders first comprehend a source representation and then condition a causal generator on it. The blocks are related, but the masks determine what each prediction is allowed to know.