Seq2seq, encoder-decoder, and the bottleneck

1

Concept

Sequence-to-sequence learning maps one variable-length sequence to another. Translation is the canonical example, but the pattern also covers transcription, summarization, and structured generation. The classic architecture separates an encoder, which reads the source, from an autoregressive decoder, which produces the target.

In a recurrent seq2seq model, the encoder updates hidden state for every source token. Its final state, or a transformation of it, becomes a fixed representation of the whole source. The decoder initializes from that representation, consumes a beginning-of-sequence token, and predicts a distribution for the first target token. The selected token then becomes input to the next step until an end token is emitted.

Source and target lengths need not match. The model learns alignment only indirectly through the final training objective. Cross-entropy scores each target position, and masks exclude padding. During training, targets can be shifted: decoder input contains the start token followed by all but the last target, while labels contain the target followed by its end marker.

Teacher forcing supplies the true previous target token during training. It makes optimization easier and lets all known decoder inputs be prepared, though an RNN still advances sequentially. At inference, the true continuation is unavailable, so the decoder conditions on its own selected tokens. One early error can change every later input. Beam search or sampling changes exploration but cannot restore source information absent from the representation.

The central weakness is the fixed-vector bottleneck. Every source detail needed at any future decoder step must pass through one bounded final state. A short sentence may fit; a long source forces names, negation, order, and modifiers to compete. Later encoder updates can overwrite earlier information, and gradients from early source positions to late target losses travel through long recurrent chains.

Bidirectional encoders improve source representations by reading both directions because the full source is available. Stacked recurrent layers add capacity. Reversing source order shortened some dependencies in early work. These measures help but preserve the single-vector interface. Increasing vector width spends more parameters without changing the requirement that one summary serve every target step.

Encoder and decoder form useful modular roles. The encoder builds representations of observed input; the decoder models target history and generation. But the separation does not guarantee faithful translation. A strong decoder can produce fluent target-language text while omitting or inventing source content. Evaluation needs adequacy and error analysis, not fluency alone, and automatic overlap metrics have limited scope.

The bottleneck prepared the conceptual leap to attention. Instead of demanding that the final encoder state contain everything, store the sequence of encoder states. At every decoder step, compute a content-dependent weighted combination relevant to the next output. The encoder-decoder contract remains, but the information channel widens from one card to an addressable set of notes. That change made alignment visible and long-source learning substantially more direct.

A useful bottleneck experiment holds target complexity fixed while increasing irrelevant and relevant source length separately. If quality falls only when relevant details become distant or numerous, the compression interface is implicated; if irrelevant padding causes equal collapse, masking or optimization may be the culprit. Inspect omissions, repetitions, and entity substitutions rather than trusting one aggregate overlap score. These error types reveal whether the decoder is fluent but source-poor.