Bahdanau attention: the idea that changed everything

1

Concept

Classic encoder-decoder models force one fixed vector to carry the entire source. Bahdanau attention replaces that interface with a collection of encoder states and a learned addressing operation. At every target step, the decoder asks which source positions are useful now. The answer is a differentiable distribution, so alignment and translation can be learned jointly from the translation objective.

Let the encoder produce states $h_1,\ldots,h_S$. Before target step $t$, the decoder has previous state $s_{t-1}$. Additive attention computes one energy for every source position:

The learned scorer can compare decoder needs and source content after projecting them into a shared alignment space.

Softmax across source positions turns energies into weights:

For a fixed target step, weights are non-negative and sum to one. A mask gives padded source positions effectively zero weight. Temperature is not normally an extra free decoding parameter here; the scale is learned through scorer parameters.

The context vector is

The decoder uses $c_t$ with its state and target input to update itself and predict the next token. At the next target position, it computes new energies and a new context. A name, verb, or agreement marker can therefore retrieve different source evidence without demanding that one final encoder state preserve all details equally.

The full matrix of $\alpha_{ti}$ values forms a soft alignment map: target positions on one axis, source positions on the other. Translation often produces roughly diagonal patterns with jumps or spreads for reordered phrases and multiword expressions. The alignment is learned without word-level alignment labels. “Soft” means the model can mix positions and gradients flow through every nonzero weight.

This design shortens paths for both information and learning. A source state can influence a late target directly through one attention edge rather than only through the encoder’s final state and many decoder transitions. The recurrent decoder remains sequential, and computing scores for all source positions at every target step adds work, but the fixed bottleneck is removed.

Attention weights are not complete explanations. A high weight shows that a stored value contributed strongly under one normalized routing operation. Encoder states already mix context; downstream decoder dynamics can amplify or cancel the contribution; alternate weights can sometimes yield similar outputs. Alignment plots are useful diagnostics and historical evidence, but causal claims require interventions rather than visual inspection alone.

Bahdanau attention changed the architecture’s central question from “how can the encoder compress everything?” to “how can the decoder retrieve what it needs?” Later dot-product attention altered the scoring rule, and the Transformer removed recurrence by using attention for representation mixing throughout. Queries, keys, and values generalize the same address-and-retrieve pattern. The enduring breakthrough is not a heatmap: it is a learned, differentiable memory interface whose address depends on the current computation.

When implementing the alignment, assert that every unmasked row sums to one and every padded position receives zero probability within numerical tolerance. These invariants catch a softmax applied over the wrong axis.