Activation functions: ReLU, GELU, and SwiGLU

1

Concept

An activation function sits between learned affine maps and makes their composition nonlinear. It also controls which signals and gradients travel through the network. A useful activation is inexpensive, behaves predictably over expected input scales, and supports optimization. Its effects cannot be judged from the curve alone: initialization, normalization, width, hardware kernels, and parameter budget interact with the choice.

ReLU is $\operatorname{ReLU}(x)=\max(0,x)$. Positive inputs pass unchanged; negative inputs become zero. It is continuous, cheap, and piecewise linear. On the positive side its derivative is one; on the negative side it is zero. At exactly zero, software adopts a conventional derivative. ReLU avoids the broad saturation of sigmoid and tanh on positive inputs, which helped train deeper networks.

Zero output can create sparse activations, but zero negative gradient has a risk. If a unit’s pre-activation stays negative for all relevant examples, gradient through that ReLU path vanishes and the unit may become “dead.” Bias, learning rate, input scale, and initialization affect this outcome. Leaky ReLU and related functions keep a small negative slope, trading hard sparsity for a possible recovery path.

GELU applies a smooth, magnitude-dependent gate, commonly written $x\Phi(x)$, where $\Phi$ is the standard normal cumulative distribution function. Unlike ReLU, it can produce small negative outputs and transitions smoothly around zero. Practical implementations may use an approximation. The probabilistic-looking formula motivates an intuition of softly weighting inputs, but GELU is a deterministic function during ordinary inference.

SiLU, also called swish in this form, computes $x\sigma(x)$. It is another smooth non-monotonic activation with a small negative region. SwiGLU uses SiLU inside a gated linear unit. A common feed-forward form is

followed by an output projection. One branch decides how strongly another content branch passes, making gating learned and input-dependent.

Comparing SwiGLU to a plain ReLU or GELU MLP requires matching budgets. SwiGLU uses additional input projections, so keeping the same hidden width increases parameters and compute. Architectures often adjust the intermediate width to keep totals comparable. A quality gain at unmatched parameter count cannot be attributed cleanly to the activation. Kernel availability also changes wall-clock performance even when FLOP estimates look similar.

Activations influence distributions across depth. Very large inputs can amplify outputs or gradients; saturated regions can suppress learning; nonzero means can shift following layers. Normalization and residual paths constrain these effects in Transformers. Mixed-precision arithmetic adds practical concerns: stable formulations and fused kernels can avoid overflow, rounding, or unnecessary memory traffic.

The right question is not “which activation is best?” but “under this architecture and budget, what behavior does the activation enable, and what did the experiment control?” ReLU offers a transparent baseline. GELU provides smooth gating used in many Transformer families. SwiGLU adds multiplicative, learned feature selection and is common in modern feed-forward blocks. All three are mechanisms, not explanations of model intelligence; they shape the function and the optimization path through which useful representations are learned.

Always compare the exact formula and dtype used by the implementation; an approximation and a fused kernel can differ from the symbolic curve while sharing its name.