What Is Layer Normalization? A Simple Explanation for Beginners

Updated · By Viviai

Layer normalization (LayerNorm) is a technique that rescales the numbers inside one neural network layer so they stay in a steady range, using only the current data sample rather than a whole batch. It computes a mean and standard deviation across the features of a single sample, normalizes each feature against them, and then applies optional learned scale and shift values. LayerNorm is widely used in transformers and recurrent networks because its result does not depend on batch size.

If you have met batch normalization, the easiest way to understand LayerNorm is to ask one question: where do the mean and variance come from? BatchNorm gathers them across many examples in a batch, while LayerNorm gathers them across the features of one example. That single difference explains most of its behavior.

The Problem Layer Normalization Addresses

Deep networks stack many layers, and the numbers flowing between them can drift. Some activations grow large, others shrink, and the distribution seen by each layer keeps moving as earlier weights update. This makes gradient-based training harder: learning rates become difficult to set, and progress can slow down or become unstable.

Normalization keeps the inputs to each layer roughly centered around zero with a comparable spread. The original Layer Normalization paper by Ba, Kiros, and Hinton (2016) introduced it as a way to reduce training time, and specifically as a design that works when batch normalization is awkward — for example with small batches or recurrent networks.

How Layer Normalization Works

Suppose a layer outputs a vector of dimension (D) for one sample, with values (x_1, x_2, \dots, x_D). LayerNorm follows four steps.

  1. Compute the mean across the features.

[ \mu = \frac{1}{D}\sum_{i=1}^{D} x_i ]

  1. Compute the variance across the same features.

[ \sigma^2 = \frac{1}{D}\sum_{i=1}^{D}(x_i - \mu)^2 ]

  1. Normalize each value, adding a small constant (\epsilon) to avoid dividing by zero.

[ \hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} ]

  1. Apply learned scale and shift (also called gain and bias), so the network can restore a useful distribution rather than being forced into exactly zero mean and unit variance.

[ y_i = \gamma_i \hat{x}_i + \beta_i ]

The normalized vector now has mean close to zero and standard deviation close to one, while (\gamma) and (\beta), both learned during training, give the layer flexibility.

Where LayerNorm Sits in a Transformer

The name can be misleading. In a transformer, LayerNorm is not applied to an entire layer at once. It is applied to the embedding vector of a single token, a vector of hidden dimension (d), as Stanford's Speech and Language Processing textbook notes. The statistics for one token are computed independently from the statistics for every other token.

You will see two common placements:

Many recent transformer implementations lean toward pre-norm because it tends to make training deeper models more stable, but both arrangements remain in use.

LayerNorm vs Batch Normalization

The two methods normalize different axes, which leads to practical trade-offs.

Dimension Layer Normalization Batch Normalization
Statistics computed over Features of one sample A batch of samples, per feature
Depends on batch size No Yes; small batches give noisy statistics
Training vs inference Same computation Uses running averages at inference
Common in Transformers, RNNs CNNs, computer vision feed-forward nets
Works with variable sequence length Yes, statistics per step Harder to apply across time steps

Because LayerNorm normalizes each sample independently, it performs the same calculation during training and inference — there are no running population statistics to maintain. This also makes it straightforward to apply in a recurrent network by computing the statistics separately at each time step.

Why Does It Help? The Research Is Still Evolving

The early explanation was that normalization steadies the forward activations. Later work questioned whether that is the main reason. A 2019 NeurIPS paper, Understanding and Improving Layer Normalization, found that the derivatives of the mean and variance matter more than the forward normalization itself, by re-centering and re-scaling the backward gradients. The same study reported that the learned bias and gain can raise the risk of overfitting and often add little; a simplified version without them performed well on several tasks.

The practical takeaway is modest: LayerNorm reliably helps training in many architectures, but the precise mechanism is a research question, and the affine parameters are not always essential. Similar to how the original explanation for batch normalization was later debated, it is useful not to treat any single intuition as settled.

Frequently Asked Questions

Does layer normalization require a large batch?

No. Because statistics come from a single sample's features, the calculation is independent of batch size. A batch of one works the same way as a larger batch, which is one reason LayerNorm is common when batch sizes are small.

Is layer normalization the same as batch normalization?

No. BatchNorm normalizes each feature across a batch of samples and keeps running averages for inference. LayerNorm normalizes across the features within each sample and uses the identical computation at training and inference time.

Why do transformers use LayerNorm?

Transformers process sequences where each token is normalized independently, and LayerNorm does not rely on batch size or running statistics. It helps keep the attention and feed-forward stacks trainable, including in deep models.

Can I remove the scale and shift parameters?

Sometimes. Research such as the 2019 NeurIPS study found a version without them can work comparably, but keeping them remains the standard in most frameworks because they restore representational flexibility.

Summary

Layer normalization stabilizes training by rescaling each sample's features against its own mean and variance, then optionally applying learned scale and shift. It is independent of batch size, identical in training and inference, and a standard component of transformers and recurrent networks. For the broader picture of how networks learn, see our explanations of neural networks, transformers, and regularization.

Sources

  1. https://arxiv.org/pdf/1607.06450
  2. https://web.stanford.edu/~jurafsky/slp3/8.pdf
  3. https://papers.nips.cc/paper_files/paper/2019/file/2f4fe03d77724a7217006e5d16728874-Paper.pdf
  4. https://en.wikipedia.org/wiki/Normalization_(machine_learning)

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free