Abstract

Model code is full of names that are rarely explained: ReLU, GELU, SwiGLU, RMSNorm, AdamW. Each is a small function with a specific job, and each replaced something earlier for a reason that is easy to state. This article goes through them in plain terms — what the thing computes, what problem it fixes, and what it costs. It is deliberately the gentlest article in the series; LLM Training: The End-to-End Flow puts the same components in their systems context.

1. Why Any Activation Function at All

A neural network layer is a matrix multiply. Stack two of them and you get

(XW1)W2=X(W1W2)=XW′, (XW_1)W_2 = X(W_1W_2) = XW',

which is just one matrix multiply with a different matrix. Without something nonlinear in between, a hundred layers collapse into one, and the model can only represent straight-line relationships no matter how large it is.

An activation function is that something. It is applied to every element independently, and its only requirement is to be nonlinear. Everything else — which shape, how smooth, how cheap — is engineering.

2. The Activation Lineage

2.1 Sigmoid and tanh: the ones that stopped being used

Early networks squashed every value into a fixed range: sigmoid σ(x)=1/(1+e−x)\sigma(x) = 1/(1+e^{-x}) into (0,1)(0,1), tanh into (−1,1)(-1,1).

The problem is what happens to the slope. For ∣x∣>4|x| > 4 or so, both are almost flat, so their derivative is almost zero. During the backward pass, gradients are multiplied by that derivative at every layer, and a chain of small numbers multiplied together vanishes. Deep networks built from these trained very slowly or not at all — the vanishing-gradient problem.

2.2 ReLU: the one that fixed it

ReLU⁡(x)=max⁡(0,x). \operatorname{ReLU}(x) = \max(0, x).

Negative inputs become zero; positive inputs pass through unchanged. Two properties made it the default for a decade. Its derivative is exactly 1 for positive inputs, so gradients pass through deep stacks without shrinking. And it is a comparison and a select — as cheap as an operation gets.

Its weakness is the flat half. A unit whose input is negative for every example in the data has zero gradient forever and never recovers, a failure mode called a dying ReLU.

2.3 GELU and SiLU: smoothing the corner

ReLU has a sharp corner at zero — the derivative jumps from 0 to 1. Two smooth alternatives dominate modern models:

GELU⁡(x)=x Φ(x),SiLU⁡(x)=x σ(x), \operatorname{GELU}(x) = x\,\Phi(x), \qquad \operatorname{SiLU}(x) = x\,\sigma(x),

where Φ\Phi is the Gaussian cumulative distribution function and σ\sigma the sigmoid. Both read as “scale xx by how positive it probably is”. Both are near zero for very negative inputs, near xx for very positive inputs, and smooth in between — and both allow small negative outputs near zero, which keeps a unit from dying outright. SiLU is also called Swish.

Column charts of ReLU, GELU and SiLU sampled from minus four to four, showing the hard corner of ReLU against the smooth dip below zero in the other two.
Figure 1. The three shapes, sampled at integer inputs. All three are flat-ish on the left and pass values through on the right; the difference is the corner. ReLU is exactly zero for every negative input, while GELU and SiLU dip slightly below zero before flattening, so a unit that goes negative still receives gradient.
Input xx−2-2−1-1001122
ReLU00012
GELU−0.05-0.05−0.16-0.1600.841.95
SiLU−0.24-0.24−0.27-0.2700.731.76

2.4 SwiGLU: adding a gate

Everything above is one function of one number. Gated activations use two:

SwiGLU⁡(x)=SiLU⁡(xWgate)⊙(xWup), \operatorname{SwiGLU}(x) = \operatorname{SiLU}(xW_{\text{gate}}) \odot (xW_{\text{up}}),

where ⊙\odot is elementwise multiplication. One projection produces a value, the other produces a gate that scales it. Because the two are multiplied, the layer can suppress or amplify each channel depending on the input, which a single fixed curve cannot do.

A plain feed-forward block with an up projection, activation and down projection, beside a gated block where two parallel projections are multiplied before the down projection.
Figure 2. The feed-forward block, ungated and gated. The gated version splits the input into two projections and multiplies them elementwise, so the third matrix is the cost of making the activation input-dependent. Hidden width drops to roughly two-thirds so the two blocks have the same parameter count.

SwiGLU is what Llama, PaLM and most recent models use. The evidence for it is empirical: swapping it in improves loss at equal parameter count, and no one has a first-principles explanation. The paper that introduced the family says so outright, ending with the observation that the improvement is offered “without explanation”.

3. Normalization

Activations drifting in scale as they pass through dozens of layers makes training unstable: too large and the loss overflows, too small and gradients vanish. Normalization rescales them at every block.

LayerNorm standardizes each token’s activation vector to zero mean and unit variance, then applies a learned scale and shift:

LN⁡(x)=x−μσ2+ϵ⋅γ+β. \operatorname{LN}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta .

RMSNorm drops the mean entirely and divides by the root-mean-square:

RMSNorm⁡(x)=x1d∑ixi2+ϵ⋅γ. \operatorname{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}} \cdot \gamma .

Removing the mean subtraction turns out to cost nothing in quality while saving a pass over the vector and a learned parameter, which is why current models use RMSNorm. Both are memory-bandwidth-bound: they read a vector, compute a statistic, and write a vector, with almost no arithmetic per byte.

One placement detail matters more than the choice between them. Pre-norm — normalizing the input to each sublayer, rather than its output — leaves the residual path free of any normalization, so the gradient reaches early layers unattenuated. That is what makes very deep transformers trainable without a warmup-sensitive schedule.

4. Optimizers, One Idea at a Time

An optimizer decides what to do with the gradient once you have it. Each step below adds exactly one idea to the one before.

4.1 SGD: follow the gradient

θ←θ−η g. \theta \leftarrow \theta - \eta\, g .

Move each parameter a little way downhill. Simple, and it works — but every step follows the gradient of one noisy mini-batch, so the path is jittery, and one learning rate η\eta has to suit parameters whose gradients differ in magnitude by orders.

4.2 Momentum: smooth the direction

m←β1m+(1−β1)g,θ←θ−η m. m \leftarrow \beta_1 m + (1 - \beta_1) g, \qquad \theta \leftarrow \theta - \eta\, m .

Keep a running average of recent gradients and step along that. Directions that keep reappearing accumulate; noise that flips sign cancels out. With β1=0.9\beta_1 = 0.9, mm is roughly an average of the last ten gradients.

4.3 Per-coordinate scaling: fix the step size

v←β2v+(1−β2)g2,θ←θ−η gv+ϵ. v \leftarrow \beta_2 v + (1 - \beta_2) g^2, \qquad \theta \leftarrow \theta - \eta\, \frac{g}{\sqrt{v} + \epsilon} .

Track the recent magnitude of each coordinate’s gradient and divide by it. A parameter with consistently tiny gradients gets its step scaled up; one with huge gradients gets scaled down. The learning rate stops having to be right for every parameter at once.

4.4 Adam: both at the same time

Adam keeps both averages and divides one by the other:

m^=m1−β1 t,v^=v1−β2 t,θ←θ−η m^v^+ϵ. \hat{m} = \frac{m}{1 - \beta_1^{\,t}}, \qquad \hat{v} = \frac{v}{1 - \beta_2^{\,t}}, \qquad \theta \leftarrow \theta - \eta\, \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon} .

The hats are bias correction. Both averages start at zero, so early on they are biased toward zero and would produce tiny first steps; dividing by 1−βt1 - \beta^t cancels that exactly, and the correction fades as tt grows.

4.5 AdamW: fix the weight decay

Weight decay keeps parameters small, which helps generalization. The old way was to add λθ\lambda\theta to the gradient — mathematically identical to L2 regularization for plain SGD.

With Adam it is not identical, because everything in the gradient gets divided by v^\sqrt{\hat v}. A parameter with large gradients has its decay divided away to nothing; one with small gradients gets decayed hard. The regularization ends up depending on gradient history, which was never the intent.

AdamW decouples them, applying the decay to the parameter directly:

θ←θ−η(m^v^+ϵ+λ θ). \theta \leftarrow \theta - \eta\left( \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon} + \lambda\, \theta \right).

That one change is the whole difference between Adam and AdamW, and it is why every large model is trained with the latter.

A chain from SGD to momentum to per-coordinate scaling to Adam to AdamW, each step annotated with the idea it adds and the state it keeps per parameter.
Figure 3. Each optimizer is the previous one plus a single idea, and every idea that helps costs memory. The right-hand column is what makes optimizer state the largest single term in the training memory budget.
OptimizerExtra state per parameterIdea it adds
SGDnonefollow the gradient
SGD + momentummmsmooth the direction
RMSPropvvper-coordinate step size
Adammm, vvboth, with bias correction
AdamWmm, vvdecoupled weight decay

That middle column is not free. In mixed precision, mm and vv are kept in fp32 — 8 of the 16 bytes per parameter a training run has to hold, and the reason optimizer state dominates the memory budget.

5. What Gets Used Today

A current model, and why each choice is there:

ComponentTypical choiceBecause
ActivationSwiGLUBest measured loss at equal parameter count
NormalizationRMSNorm, pre-normCheaper than LayerNorm; pre-norm keeps the residual path clean
OptimizerAdamWScale-free steps, decay that behaves as intended
β1,β2\beta_1, \beta_20.9, 0.95Slightly faster-adapting second moment than the 0.999 default
ScheduleWarmup, then cosine decayAdam’s v^\hat v is unreliable in the first steps
Weight decay0.1Applied to weights, not to biases or normalization gains

6. Summary

  • Activations exist because stacked linear layers collapse into one. Everything after that is engineering.
  • ReLU fixed vanishing gradients with a hard corner; GELU and SiLU smooth that corner and let units recover from being negative.
  • SwiGLU multiplies two projections so the gate is learned and input-dependent, at the cost of a third matrix and a narrower hidden dimension.
  • RMSNorm is LayerNorm without the mean subtraction, and pre-norm placement is what keeps deep stacks trainable.
  • SGD → momentum → per-coordinate scaling → Adam → AdamW is one idea per step. Adam’s step is about η\eta regardless of gradient scale; AdamW’s only change is to apply weight decay to the parameter instead of the gradient.

References

  1. Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML 2010.
  2. Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). 2016.
  3. Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. 2017.
  4. Noam Shazeer. GLU Variants Improve Transformer. 2020.
  5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. 2016.
  6. Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. NeurIPS 2019.
  7. Ruibin Xiong et al. On Layer Normalization in the Transformer Architecture. ICML 2020.
  8. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ICLR 2015.
  9. Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. ICLR 2019.