The feed-forward block & where knowledge lives

01 · Concept

Concept

Start with an accounting question. Qwen3.8-27B is marketed at roughly 27 billion parameters — so where do they physically sit? Attention earns the publicity because it draws visible links between tokens, but if you open the checkpoint and add up tensor sizes, the answer is unglamorous: around 17 billion parameters, the clear majority of the model, live in the feed-forward network (FFN) repeated in every one of the 64 layers. Attention routes information; the FFN transforms it; and the FFN is where most of the learned capacity resides.

The original Transformer block applies two linear maps with a nonlinearity between them:

The first projection expands from model width $d_{model}$ to a larger intermediate width $d_{ff}$; the second projects back so the result can rejoin the residual stream. The same weights apply at every sequence position, positions are processed in parallel, and no information moves between tokens inside the block.

Qwen3.8-27B uses the gated descendant of this idea. Its FFN forms three projections instead of two:

$W_{gate}$ and $W_{up}$ both expand the 5120-dimensional hidden state to an intermediate width of 17,408. The gate path passes through SiLU — the smooth activation $z\,\sigma(z)$, where $\sigma$ is the logistic sigmoid — and multiplies the up path element-wise. One projection proposes candidate features; the other decides, feature by feature, how strongly each one fires. $W_{down}$ then compresses the 17,408 activations back to 5120 for the residual addition.

Now count parameters, and count them the way most people first get it wrong. The instinct trained on the two-matrix formula says: two matrices of $5120\times 17408\approx 89.1$ million entries each, so about 178 million parameters per layer. That misses the gate. The gated block holds three matrices — gate and up at $5120\times 17408$, down at $17408\times 5120$ — each around 89.1 million parameters, giving approximately 267 million per layer. Multiply by 64 layers and the FFNs alone account for roughly 17 billion parameters of the ~27 billion total. The attention-side projections, by comparison, are a small fraction of a layer’s budget. Each FFN matrix also costs about 89.1 million multiply-adds per token, so this block dominates compute as well as storage for typical context lengths.

Why spend the budget on width? A wide intermediate space is a large bank of pattern detectors. A row of $W_{gate}$ or $W_{up}$ responds strongly when the incoming residual vector aligns with its learned direction; the gate suppresses or admits that response; columns of $W_{down}$ then write combinations of surviving features back into the 5120-dimensional model space. The block behaves less like a generic “extra layer” and more like a learned feature dictionary consulted at every position.

Research has sharpened that intuition into the key–value memory lens: input directions act as keys that recognize patterns, output directions as values that write associated information. It is a productive picture. A hidden activation can respond to a pattern like “capital of France is” assembled by attention at the token “Paris” and add a feature that the output head associates with a likely continuation. It also helps explain why targeted FFN edits can change factual behavior.

Notice what the FFN never sees: position. The block applies identical weights at token 3 and token 200,000, and it performs no operation across positions. Everything positional you studied in the last lessons — RoPE rotating queries and keys, ALiBi adding distance-dependent penalties to attention scores — acts inside the mixing sublayer, never here. If the FFN at one position behaves differently from another, that is because attention delivered different residual content, not because the FFN knows where it is.

Later architectural choices reshape this block’s economics rather than its role. Sparse mixture-of-experts models replace one dense FFN with a router and several expert FFNs, activating a subset per token; Qwen3.8-27B does not do this — its FFNs are dense, and every token pays the full 267 million parameters per layer. When you inspect a real FFN, track both shapes and residual effects: which directions activate, what they write, how large the update is against the existing stream, and whether later layers preserve it. Activation alone is not causation; ablation and controlled intervention give stronger evidence.

The stable mental model is a factory rhythm. Attention is the courier bringing each address its selected notes. The FFN is the machine shop: it expands the local state across 17,408 workbenches, gates each bench with SiLU, and compresses an update back onto the shared conveyor. Sixty-four shops in a row, holding seventeen billion of the twenty-seven billion parts — this is, quite literally, where most of the model lives.