What Is Backpropagation? A Plain-Language Guide

Updated · By Viviai

Backpropagation (short for "backward propagation of errors") is an algorithm that calculates how much each weight in a neural network contributes to the prediction error, using the chain rule from calculus; it works backward from the output layer so an optimizer such as gradient descent can adjust the weights and reduce future error.

What backpropagation actually does

A neural network makes a prediction by passing data forward through layers of weighted connections. When the prediction is wrong, the network needs a way to assign blame: which weights should change, and by how much. Backpropagation answers that question by computing the gradient of the loss with respect to every weight and bias. Strictly, it is a gradient-computation method; the actual step that changes weights is usually a separate optimizer such as gradient descent.

The problem it solved

Adjusting weights in a single-layer model is straightforward, but a network with several hidden layers creates a nested chain of calculations. Working out each weight's influence one by one would repeat a lot of arithmetic. Backpropagation reuses intermediate results as it moves backward, which makes training multi-layer networks computationally practical. The approach became widely known after a 1986 paper by Rumelhart, Hinton, and Williams, although related ideas in automatic differentiation appeared earlier.

Forward pass versus backward pass

Pass Direction What it computes
Forward pass Input to output Activations and a prediction, then the loss
Backward pass Output to input Gradients of the loss for each weight and bias

In the forward pass, each neuron applies its weights and an activation function to produce an output. The loss function then compares that output with the target. In the backward pass, the error signal travels from the output layer back toward the input, one layer at a time.

The chain rule, step by step

Backpropagation is an efficient use of the chain rule. For a layer (l), an error term often written (\delta) is derived from the layer in front of it:

[ \delta^{l} = (f^{l})' \circ (W^{l+1})^{T}\delta^{l+1} ]

Here ((f^{l})') is the derivative of the activation function and (W^{l+1}) holds the weights of the next layer. Once (\delta) is known, the gradient for a weight follows from the activation that fed into it. Because each layer reuses the gradient of the layer ahead, the same quantities are not recalculated.

How a weight is then updated

Backpropagation supplies the gradients; an optimizer uses them. With a basic gradient-descent update, a weight (w) moves a small step opposite its gradient:

[ w_{\text{new}} = w_{\text{old}} - \eta\frac{\partial L}{\partial w} ]

The symbol (\eta) is the learning rate, which controls step size. One full run of the training data, split into forward and backward updates, is often organized into epochs and batches.

Terms you will meet

Term Meaning in this context
Gradient The direction and rate at which loss changes as a weight changes
Epoch One complete pass over the training data
Batch A subset of examples used in one update
Learning rate A factor controlling how large each update is
Chain rule The calculus rule for differentiating nested functions

Honest caveats

Backpropagation does not guarantee a useful model on its own, and it depends on the data, the loss, and the chosen optimizer. In deep networks, gradients can become very small as they move back through many layers, a situation called the vanishing-gradient problem, which activation choices and network design aim to address. It also relies on functions that are differentiable, so parts of a model that have no gradient cannot be trained this way.

FAQ

Is backpropagation the same as gradient descent?

No. Backpropagation computes the gradients, while gradient descent is an optimizer that uses those gradients to update the weights. They usually work together but are separate steps.

Why does backpropagation move backward?

Because the error is known at the output, and the chain rule lets each hidden layer infer its own contribution from the layer directly in front of it, reusing results along the way.

What is the role of the activation function's derivative?

It controls how much of the error signal passes through a neuron during the backward pass, which is why a near-zero derivative can slow learning.

Does backpropagation work for any neural network?

It applies to feed-forward networks with differentiable operations; architectures with loops or non-differentiable steps need additional methods or modified handling.

Key takeaways

Continue with what is a neural network and what is gradient descent.

Sources

  1. https://www.ibm.com/think/topics/backpropagation
  2. https://en.wikipedia.org/wiki/Backpropagation
  3. https://www.geeksforgeeks.org/backpropagation-in-neural-network/

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free