The classic block — and Qwen's two block types

01 · Concept

Concept

The pieces now assemble into one repeated interface. A decoder-only, pre-norm Transformer block receives a tensor $x$ of shape $B\times T\times d_{model}$ and returns a tensor of the same shape. Everything inside may expand, split into heads, or form attention matrices, but the boundary stays stable so dozens of blocks can stack. Conceptually, the block is two equations:

In the classic block the mixer is causal multi-head self-attention: normalized features are projected into $Q$, $K$, and $V$, reshaped into heads, compared with scaled dot products under a causal mask, mixed by softmax weights, concatenated, and projected back to $d_{model}$. The FFN branch normalizes the updated stream, expands it, gates or activates it, and projects back. Attention is token mixing — positions exchange information; the FFN is channel mixing — each position recombines its own features nonlinearly. Repeating the pair lets routes and transformations compose across depth.

Now trace shapes through a real block, because this is where a confident wrong turn lives. Take Qwen3.8-27B’s attention block, with hidden size 5120 and 24 query heads at head dimension 256. The reflex answer for the query projection is “5120 in, 5120 out — projections are square.” Check it: width out equals heads times head dimension, $24\times 256 = 6144$. That is not 5120, and it is not a bug. That 6144 is the query state; because these layers are gated, the q_proj weight emits it twice over — 5120 → 12288 — and splits into a 6144 query and a 6144 gate (lesson 4.2). Keys and values, with only 4 KV heads under grouped-query attention, map 5120 → $4\times 256 = 1024$ each; and the output projection maps the concatenated 6144 back to 5120 so the update can rejoin the stream. Nothing in the architecture forces the head subspace to match the model width — only the block’s boundary must return to 5120. If you “fix” a shape mismatch by forcing square projections, you have silently changed the architecture.

Here is the larger structural fact. Qwen3.8-27B contains two block types, and both are instances of the interface above. They share everything outside the mixer: pre-norm RMSNorm ($\epsilon=10^{-6}$) in the same positions, the same residual additions, and an identical SiLU-gated FFN expanding 5120 → 17,408 → 5120 at a cost of roughly 267 million parameters per layer.

The Attention block is the classic design with modern refinements: gated attention using 24 query heads and 4 shared KV heads at head dimension 256, rotary position information applied to a 64-dimensional slice of each head, softmax scores over the causal past, an output projection from 6144 back to 5120. It can look at any earlier position with exact, content-addressed precision. Sixteen of the 64 layers are this block.

The DeltaNet block replaces softmax attention entirely. Its mixer is Gated DeltaNet, a recurrent layer that carries a fixed-size matrix state from token to token, updating it with a gated delta rule rather than storing every past key and value. It never materializes a $T\times T$ score matrix and never grows memory with sequence length. Forty-eight of the 64 layers — three out of every four — are this block. Lesson 4.15 opens the mechanism; here, what matters is that from the residual stream’s point of view the DeltaNet block is just another mixer honoring the same contract: read a normalized 5120-wide state, return a 5120-wide update.

The two types alternate on a strict pattern: three DeltaNet blocks, then one Attention block, repeated 16 times down the stack. Why a hybrid at all is the subject of lesson 4.16; the assembly-level point is that the residual stream makes the mixture possible. Because no block owns the representation — each only adds a bounded update to a shared workspace — a recurrent mixer and an attention mixer can interleave freely, each reading whatever the other wrote.

Around the stack sit the bookends. Token IDs index a 248,320 × 5120 embedding table before layer 0; a final RMSNorm follows layer 63; and the language-model head maps each final 5120-dimensional state to 248,320 logits. Qwen3.8-27B does not tie this head to the embedding table — they are separate matrices of about 1.271 billion parameters each — a fact you will meet again when reading the checkpoint in lesson 4.14.

Memory behavior differs sharply between the block types. The attention block’s score computation scales with context in conventional implementations, and at decode time it must retain past keys and values; the DeltaNet block carries only its fixed-size state regardless of length. The same stack therefore stresses hardware very differently depending on which sixteen layers you are measuring — a thread picked up in track 7.

The assembled picture is compact: one stream, one contract, two mixers. Attention where direct access to per-token keys and values earns its cost, DeltaNet where cheap constant-state mixing suffices, and an identical gated FFN doing the heavy transformation on every floor. Understanding the interface — not memorizing any one block’s internals — is what lets you read a 64-layer heterogeneous model as easily as a uniform twelve-layer one.