Bahdanau attention: the idea that changed everything

01 · Concept

Concept

Lesson 3.4 ended with a diagnosis in two parts. One fixed vector cannot hold everything a long source contains, and even if it could, it is computed before the decoder knows which word it is about to produce. Widening it addressed neither. The 2015 answer changes the interface instead of its size, and the change is small enough to state in one sentence: stop throwing the encoder states away.

Let the encoder produce states $h_1,\ldots,h_S$, one per source position, and keep all of them. Before target step $t$, with the decoder holding previous state $s_{t-1}$, a learned scorer produces one energy for every source position:

Both the decoder’s current need and the source position’s content are projected into a shared space, added, squashed, and scored. Nothing here is a similarity metric chosen in advance; the comparison itself is learned.

A softmax across the source positions turns those energies into weights:

For a fixed target step the weights are non-negative and sum to one, and a mask drives padded source positions to effectively zero. The context vector is then the weighted blend of what was stored:

The decoder combines $c_t$ with its own state to predict the next token and update itself — and at the next position it computes fresh energies and a fresh context.

Work an example through by hand, because the arithmetic makes the behavior obvious. Take a three-position source and suppose the scorer produces energies $e=(2.0,\,0.5,\,-1.0)$ at some decoder step. Exponentiate: $7.389$, $1.649$, $0.368$, summing to $9.406$. Divide: the weights are approximately $0.786$, $0.175$, and $0.039$. So $c_t\approx0.786\,h_1+0.175\,h_2+0.039\,h_3$ — this step is reading mostly from source position one, with a glance at position two. Now change one energy: raise the second from $0.5$ to $2.5$ and the weights become roughly $0.371$, $0.611$, $0.018$. A difference of two in the exponent moved the majority of the read from one position to another. Softmax is sharply sensitive to energy gaps, which is what lets the mechanism behave like a lookup while remaining perfectly differentiable.

Now count what this bought. The capacity problem is gone, because nothing must be compressed into a single fixed vector and the summary is tailored to the current step. And the credit-assignment problem is gone with it, which is the half people usually forget: source state $h_i$ now influences a late target loss through one weighted edge. Lesson 3.4 measured that path at roughly eighty-five recurrent transitions for a modest sentence pair, with a retention factor raised to the eighty-fifth power. Attention replaces the product with a single multiplication by $\alpha_{ti}$. Two distinct failures, one mechanism.

Here is the classic wrong turn, and it is the single most common attention bug ever written. You have a matrix of energies with target steps on one axis and source positions on the other, and you apply softmax to it. Along which axis? Normalize across target steps by mistake and the code runs, the shapes are identical, the loss decreases, and the model produces fluent output — because the decoder’s own language modelling can carry a great deal on its own. What you have built is not attention: each source position now distributes a fixed budget of one across the target steps, so a source word that matters to three output words has its influence diluted, and a target step’s weights no longer sum to anything meaningful. The correction is an assertion, not a code review: check that every unmasked row over source positions sums to one within numerical tolerance and that every padded position receives zero. Two lines, and they catch an error that a decreasing loss curve will hide indefinitely.

The full matrix of $\alpha_{ti}$ values forms a soft alignment map, and plotting it was a genuine milestone — translation produces roughly diagonal patterns with jumps and spreads where phrases reorder, learned without any word-level alignment supervision. “Soft” means the model may blend positions and gradients flow through every nonzero weight.

One practical note before that. The design keeps the recurrent decoder, so generation is still sequential, and computing energies for every source position at every target step adds real work that grows with the product of the two lengths. That cost was worth paying to remove the bottleneck, and it is the same cost that reappears, magnified, when attention is applied not between two sequences but within one — which is where track 4 begins.