Vanishing and Exploding Gradients: Why Deep Networks Stop Learning
Vanishing and exploding gradients are two failure modes in deep networks where the error signal that flows backward during training becomes either so small that the early layers stop updating, or so large that the weights become unstable. Both come from the same root cause: backpropagation multiplies many layer-by-layer factors together, and repeated multiplication of numbers below one shrinks toward zero while repeated multiplication of numbers above one grows without bound.
The chain rule is a product, and products are fragile
When a network learns, backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule. For a network with many layers, the gradient reaching an early layer is a product of terms, one from each layer it passes through:
[ \frac{\partial L}{\partial W^{1}} = \frac{\partial L}{\partial g} \prod_{l=1}^{L-1} \frac{\partial A^{l}}{\partial Z^{l}} \frac{\partial Z^{l+1}}{\partial A^{l}} ]
Each factor is usually near or below one. Raise a number like 0.9 to the power of 50 and it collapses to about 0.005; raise a number like 1.1 to the power of 50 and it balloons to about 117. Depth turns a small, harmless per-layer deviation into an exponential effect. MIT lecture notes describe the shrinking case as magnitude (\exp(-\ell)) and the growing case as (\exp(\ell)), where (\ell) is the number of layers.
The vanishing gradient problem
A vanishing gradient is what happens when that product shrinks toward zero. The early layers receive almost no error signal, so their weights barely change, even though the later layers are still learning. The network appears to train, but its deepest features never improve.
The classic trigger is an old, saturating activation function such as the sigmoid. A sigmoid squashes every input into the range between 0 and 1, and its derivative is at its largest around 0.25, falling close to zero in the flat regions. Once a neuron saturates, its local gradient is tiny, and multiplying many such derivatives together crushes the signal. This was especially damaging in early recurrent networks, where long sequences meant chaining through dozens or hundreds of time steps.
The exploding gradient problem
The mirror image occurs when the factors stay above one. The gradient grows as it travels back, producing enormous weight updates that overshoot and destabilize training. In the worst case the values overflow into NaN and the run collapses entirely.
Exploding gradients are common in recurrent networks over long sequences and in very deep networks with weights initialized too large. The Toronto course on the topic notes that gradients can explode on some inputs while vanishing on others, so a single average loss can hide the problem.
Side by side
| Vanishing gradients | Exploding gradients | |
|---|---|---|
| Gradient magnitude in early layers | Approaches zero | Grows without bound |
| Effect on early weights | Hardly update | Swing wildly or become NaN |
| Common cause | Saturating activations, poor initialization | Weights too large, long unrolled sequences |
| Observable symptom | Loss plateaus and early layers feel frozen | Loss spikes, oscillates, or turns NaN |
| Typical fix | Better initialization, ReLU-type activations, normalization, stable architectures | Gradient clipping, lower learning rate, smaller weights |
How the problems are addressed
Careful weight initialization
If the weights start at a scale that keeps the activations and gradients in a healthy range, the product neither shrinks nor explodes. Initialization schemes such as Xavier/Glorot and He set the variance of each layer according to its number of inputs and outputs, so the signal magnitude stays roughly stable as it moves through the network.
Non-saturating activations
Replacing sigmoid and tanh with ReLU and its variants removes the near-zero derivative over most of the input range, which eases vanishing in deep stacks. Modern transformers often use smoother choices such as GELU and SwiGLU.
Normalization
Batch normalization and related techniques rescale activations during training, which keeps values in a range where gradients stay usable and reduces the pressure on initialization.
Gradient clipping (for exploding gradients)
Clipping caps the gradient at a fixed maximum value before the weights are updated. With norm clipping, if the gradient's total norm exceeds a threshold, it is rescaled down to that threshold. This is a simple, widely used safeguard against spikes, especially in recurrent and language models.
Architecture changes
Some of the strongest fixes are structural. Gated recurrent cells such as LSTM and GRU add pathways that let gradients flow over long time windows without repeated multiplication by small factors. Residual connections in deep networks create direct shortcuts for gradients, so a signal can skip layers rather than passing through every factor. The recurrent neural network guide explains the gated designs.
How to detect it in your own training
- Watch the gradient norms per layer. If early-layer norms trend toward zero while later layers stay normal, suspect vanishing.
- If norms spike and the loss jumps or becomes NaN, suspect exploding.
- Compare how fast early versus late layers update. Frozen early layers are a practical tell.
- Check whether the problem appears only on long sequences or very deep models, which points at the chain-rule product.
Frequently asked questions
Do ReLU networks still have vanishing gradients? Less often, but it can still happen. ReLU has a derivative of zero for negative inputs, so a neuron can become "dead" and stop updating. Variants such as Leaky ReLU and the normalization and residual techniques above reduce this risk.
Does gradient clipping fix vanishing gradients? No. Clipping only limits gradients that are too large, so it addresses the exploding case. Vanishing requires changes on the other side, such as initialization, activations, normalization, or architecture.
Are these problems only in recurrent networks? They became famous in recurrent networks because long sequences create very long gradient chains, but any sufficiently deep feedforward network can show the same behavior through repeated layer multiplication.
How does the learning rate relate to exploding gradients? A high learning rate amplifies large updates and can make an unstable gradient explode sooner. Lowering the rate and adding clipping often work together; see the learning rate explainer.
Key takeaways
- Backpropagation builds a long product of per-layer factors, which is sensitive to depth.
- Factors below one shrink the gradient to vanishing; factors above one grow it to exploding.
- Saturating sigmoids and poor initialization drive vanishing; oversized weights and long sequences drive exploding.
- Initialization, non-saturating activations, normalization, residual and gated architectures target vanishing, while clipping targets exploding.
- Inspect per-layer gradient norms and the loss curve to tell the two apart.
For the broader picture, read how gradient descent updates weights and how a neural network is structured.
Sources
- https://people.csail.mit.edu/madry/6.S979/files/lecture_3.pdf
- http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/slides/lec14.pdf
- https://arxiv.org/pdf/2106.09475v1
- http://d2l.ai/chapter_multilayer-perceptrons/numerical-stability-and-init.html
- https://introml.mit.edu/_static/spring26/lectures/lec06/lecture06.pdf
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free