Vanishing and Exploding Gradients: Why Deep Networks Stop Learning

Updated · By Viviai

Vanishing and exploding gradients are two failure modes in deep networks where the error signal that flows backward during training becomes either so small that the early layers stop updating, or so large that the weights become unstable. Both come from the same root cause: backpropagation multiplies many layer-by-layer factors together, and repeated multiplication of numbers below one shrinks toward zero while repeated multiplication of numbers above one grows without bound.

The chain rule is a product, and products are fragile

When a network learns, backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule. For a network with many layers, the gradient reaching an early layer is a product of terms, one from each layer it passes through:

[ \frac{\partial L}{\partial W^{1}} = \frac{\partial L}{\partial g} \prod_{l=1}^{L-1} \frac{\partial A^{l}}{\partial Z^{l}} \frac{\partial Z^{l+1}}{\partial A^{l}} ]

Each factor is usually near or below one. Raise a number like 0.9 to the power of 50 and it collapses to about 0.005; raise a number like 1.1 to the power of 50 and it balloons to about 117. Depth turns a small, harmless per-layer deviation into an exponential effect. MIT lecture notes describe the shrinking case as magnitude (\exp(-\ell)) and the growing case as (\exp(\ell)), where (\ell) is the number of layers.

The vanishing gradient problem

A vanishing gradient is what happens when that product shrinks toward zero. The early layers receive almost no error signal, so their weights barely change, even though the later layers are still learning. The network appears to train, but its deepest features never improve.

The classic trigger is an old, saturating activation function such as the sigmoid. A sigmoid squashes every input into the range between 0 and 1, and its derivative is at its largest around 0.25, falling close to zero in the flat regions. Once a neuron saturates, its local gradient is tiny, and multiplying many such derivatives together crushes the signal. This was especially damaging in early recurrent networks, where long sequences meant chaining through dozens or hundreds of time steps.

The exploding gradient problem

The mirror image occurs when the factors stay above one. The gradient grows as it travels back, producing enormous weight updates that overshoot and destabilize training. In the worst case the values overflow into NaN and the run collapses entirely.

Exploding gradients are common in recurrent networks over long sequences and in very deep networks with weights initialized too large. The Toronto course on the topic notes that gradients can explode on some inputs while vanishing on others, so a single average loss can hide the problem.

Side by side

Vanishing gradients Exploding gradients
Gradient magnitude in early layers Approaches zero Grows without bound
Effect on early weights Hardly update Swing wildly or become NaN
Common cause Saturating activations, poor initialization Weights too large, long unrolled sequences
Observable symptom Loss plateaus and early layers feel frozen Loss spikes, oscillates, or turns NaN
Typical fix Better initialization, ReLU-type activations, normalization, stable architectures Gradient clipping, lower learning rate, smaller weights

How the problems are addressed

Careful weight initialization

If the weights start at a scale that keeps the activations and gradients in a healthy range, the product neither shrinks nor explodes. Initialization schemes such as Xavier/Glorot and He set the variance of each layer according to its number of inputs and outputs, so the signal magnitude stays roughly stable as it moves through the network.

Non-saturating activations

Replacing sigmoid and tanh with ReLU and its variants removes the near-zero derivative over most of the input range, which eases vanishing in deep stacks. Modern transformers often use smoother choices such as GELU and SwiGLU.

Normalization

Batch normalization and related techniques rescale activations during training, which keeps values in a range where gradients stay usable and reduces the pressure on initialization.

Gradient clipping (for exploding gradients)

Clipping caps the gradient at a fixed maximum value before the weights are updated. With norm clipping, if the gradient's total norm exceeds a threshold, it is rescaled down to that threshold. This is a simple, widely used safeguard against spikes, especially in recurrent and language models.

Architecture changes

Some of the strongest fixes are structural. Gated recurrent cells such as LSTM and GRU add pathways that let gradients flow over long time windows without repeated multiplication by small factors. Residual connections in deep networks create direct shortcuts for gradients, so a signal can skip layers rather than passing through every factor. The recurrent neural network guide explains the gated designs.

How to detect it in your own training

Frequently asked questions

Do ReLU networks still have vanishing gradients? Less often, but it can still happen. ReLU has a derivative of zero for negative inputs, so a neuron can become "dead" and stop updating. Variants such as Leaky ReLU and the normalization and residual techniques above reduce this risk.

Does gradient clipping fix vanishing gradients? No. Clipping only limits gradients that are too large, so it addresses the exploding case. Vanishing requires changes on the other side, such as initialization, activations, normalization, or architecture.

Are these problems only in recurrent networks? They became famous in recurrent networks because long sequences create very long gradient chains, but any sufficiently deep feedforward network can show the same behavior through repeated layer multiplication.

How does the learning rate relate to exploding gradients? A high learning rate amplifies large updates and can make an unstable gradient explode sooner. Lowering the rate and adding clipping often work together; see the learning rate explainer.

Key takeaways

For the broader picture, read how gradient descent updates weights and how a neural network is structured.

Sources

  1. https://people.csail.mit.edu/madry/6.S979/files/lecture_3.pdf
  2. http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/slides/lec14.pdf
  3. https://arxiv.org/pdf/2106.09475v1
  4. http://d2l.ai/chapter_multilayer-perceptrons/numerical-stability-and-init.html
  5. https://introml.mit.edu/_static/spring26/lectures/lec06/lecture06.pdf

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free