Seq2seq, encoder-decoder, and the bottleneck

01 · Concept

Concept

Translate a forty-word German sentence into English. The words do not appear in the same order, one language marks a distinction the other does not, and the correct English article for a noun near the end depends on a case ending near the beginning. Both sequences vary in length, and their lengths differ from each other. What architecture handles that?

The 2014 answer separates the job in two. An encoder reads the source and builds a representation. An autoregressive decoder generates the target one token at a time, conditioned on that representation and on what it has already produced. The pattern covers far more than translation — transcription, summarization, structured generation — and the encoder-decoder split remains one of the field’s durable ideas.

In the recurrent version, the encoder updates a hidden state across every source token, and its final state, or a transformation of it, becomes the fixed representation of the whole source. The decoder initializes from that representation, consumes a beginning-of-sequence token, predicts a distribution over the first target token, and feeds its choice back in until an end token appears. Source and target lengths need not match; the model learns alignment only indirectly, through the training objective. Cross-entropy scores every target position and masks exclude padding, with the decoder input holding the start token followed by all but the last target while the labels hold the target followed by its end marker.

Teacher forcing supplies the true previous target during training, which stabilizes optimization and lets all decoder inputs be prepared in advance. At inference the true continuation is unavailable, so the decoder conditions on its own choices and one early error changes every later input. Beam search and sampling change how that space is explored; they cannot restore source information that never made it into the representation.

Which brings us to the interface itself. Look at where the entire source has to fit: one vector, produced before decoding begins, unchanged for the rest of the generation. Everything the decoder will ever need about the source — every name, every negation, every number, the order of clauses — must be inside it. This is the fixed-vector bottleneck, and it fails in two separate ways that are worth keeping apart, because they have different symptoms and only one obvious fix.

The first failure is capacity, and lesson 3.1 already showed how to count it. A bounded vector holds a bounded number of bits; a long or information-dense source has more; something is lost, and the model chooses what without knowing what the decoder will ask for.

The second failure is credit assignment, and it is easy to miss. Trace the route from an early source token to a late target loss. The gradient must travel forward through every remaining encoder step to reach the final state, and then through every decoder step to reach the loss — there is no other path, because the final state is the only channel. For a forty-token source and a forty-five-token target that is about eighty-five recurrent transitions. Apply lesson 3.3’s arithmetic directly: at a per-step retention of $0.99$ the surviving factor is $0.99^{85}\approx0.43$, which is fine, but at $0.95$ it is $0.95^{85}\approx0.013$, and the first source token is effectively untrainable against a late target error. Gated cells stretched this; they did not remove it.

Here is the classic wrong turn, and the field genuinely took it before finding the real answer. Quality degrades on long sources, so you widen the encoder state — a larger summary vector should hold more. It helps a little, and it costs parameters, and the degradation curve keeps its shape. Then you try a bidirectional encoder, which does improve the representations because the full source is available in both directions. Then stacked layers. Then, in one memorable early result, reversing the source token order, which shortens the distance from the first source words to the first target words and measurably improved translation. Every one of these is a real improvement and every one leaves the interface intact: one summary, computed once, serving all output steps. Reversal in particular is the tell — it improves results by shuffling which dependencies are short, which only makes sense if path length is the binding constraint.

The modular separation the design introduced was still worth keeping. The encoder builds representations of observed input, the decoder models target history and generation. But separation guarantees nothing about faithfulness: a strong decoder can emit fluent, grammatical target-language text that omits or invents source content, which is exactly the failure mode a fixed bottleneck encourages. Evaluation therefore needs adequacy checks and error analysis — omissions, repetitions, entity substitutions — not fluency and not a single aggregate overlap score.

A clean bottleneck experiment holds target complexity fixed while varying source length in two ways: adding relevant detail, and adding irrelevant filler. If quality falls only when relevant details become numerous or distant, the compression interface is implicated. If irrelevant padding causes equal collapse, suspect masking or optimization instead. Inspect the error types rather than the aggregate score, because the whole point is the difference between a decoder that is fluent and a decoder that is informed.