Activation functions: ReLU, GELU, and SwiGLU

01 · Concept

Concept

Open the parameter list of any recent large language model and count the matrices in one feed-forward block. Lesson 2.2 described that block as up-project, activate, down-project — two matrices. Qwen3.8-27B has three per block, named gate, up, and down (Qwen3.8-27B Model Card, 2026). Something changed between the textbook picture and the shipped model, and the third matrix is where this lesson lives.

Begin with what the textbook version does. An activation function sits between two learned affine maps and makes their composition nonlinear. ReLU is $\operatorname{ReLU}(x)=\max(0,x)$: positive inputs pass unchanged, negative inputs become zero. It is cheap, piecewise linear, has derivative one on the positive side and zero on the negative side, and it avoided the broad saturation of sigmoid and tanh that had made deep networks hard to train. Its known hazard is that flat negative side. If a unit’s pre-activation stays negative across all relevant examples, no gradient reaches it through the ReLU path and the unit is effectively dead; leaky variants keep a small negative slope to preserve a recovery route.

GELU smooths the corner. It is usually written $x\Phi(x)$, where $\Phi$ is the standard normal cumulative distribution function, so the input is scaled by how far into the distribution it sits: large positives pass almost untouched, large negatives are almost fully suppressed, and the transition near zero is gradual rather than a kink. It produces small negative outputs instead of hard zeros. SiLU, also called swish in this form, is the close cousin $x\sigma(x)$ with the logistic sigmoid in place of $\Phi$ — same shape, cheaper to evaluate, and the one that ended up inside gated blocks.

Every function so far is a fixed curve applied elementwise. Feed it a number and it returns a number; the rule never depends on anything else. Here is the move that broke that pattern.

A gated linear unit projects the same input twice. One projection is the content, the other is a gate, and the two are multiplied together element by element. With SiLU on the gate branch this is SwiGLU:

followed by the down projection $W_{d}$ back to the model width. Three matrices, not two. The gate branch does not carry content forward; its only job is to decide, per feature and per input, how much of the content branch survives.

Work one coordinate through with numbers. Suppose for some token the gate projection produces $-3.0$ at coordinate $j$ while the content projection produces $+8.0$ there. The gate value is $\operatorname{SiLU}(-3.0)=-3.0\times\sigma(-3.0)\approx-3.0\times0.047\approx-0.14$, so the output at that coordinate is about $-0.14\times8.0\approx-1.1$: a strong content signal has been almost entirely suppressed, and slightly inverted. Now take a different token where the same coordinate’s gate projection is $+3.0$ instead. The gate becomes $3.0\times\sigma(3.0)\approx3.0\times0.953\approx2.86$, and the same content value of $8.0$ leaves as roughly $22.9$. Identical content, opposite fate, decided by the other branch. A fixed curve cannot do that — feed ReLU the value $8.0$ and it returns $8.0$ every single time, regardless of what the rest of the vector said.

Here is the classic wrong turn in evaluating this. You have a feed-forward block with intermediate width $d_{ff}$ and a GELU. You swap in SwiGLU, keep $d_{ff}$ the same, retrain, and measure a quality gain. It is tempting to report that gating helped. It is not a clean claim: you also added an entire extra matrix of shape $d_{\text{model}}\times d_{ff}$, so parameters and compute rose by roughly fifty percent for that block. The gain may be gating, may be capacity, and the experiment cannot separate them. The standard correction is to shrink the intermediate width when adopting a gate, commonly to about two-thirds of the ungated value, so that the three-matrix block costs what the two-matrix block cost. Only then does the comparison isolate the mechanism.

Now put the numbers on the block. In Qwen3.8-27B each of the 64 layers holds one gated feed-forward block that takes the 5120-dimensional hidden state, projects to an intermediate width of 17408 along both the gate and the up path, multiplies, and projects back to 5120. That is roughly 267 million parameters per layer, and near 17 billion across the model (Qwen3.8-27B Model Card, 2026). Whatever else gating is, it is where the majority of this model’s parameters chose to live.

Two engineering cautions close the lesson. First, activations shape distributions across depth: large inputs amplify outputs and gradients, saturated regions suppress learning, and nonzero means shift what following layers see — which is precisely why lesson 2.8’s normalization and residual paths exist. Second, always check the exact formula and dtype the implementation uses. GELU ships in exact and approximate forms, fused kernels reassociate arithmetic, and bf16 rounding differs from the symbolic curve. Two blocks sharing an activation’s name can compute measurably different functions.