What Is Gradient Descent? A Plain-Language Explanation
Gradient descent is an iterative optimization algorithm used to train machine learning models by repeatedly adjusting parameters to reduce a loss function, moving each step in the direction that lowers error. It is a common way to find parameter values that make predictions more accurate.
The simple analogy
Imagine walking down a hilly valley while blindfolded, feeling the slope beneath your feet and taking a step in the steepest downhill direction. You repeat this until the ground levels out and you reach the bottom. The loss function is the landscape, and each step moves you toward lower error.
Key terms
| Term | Meaning |
|---|---|
| Loss function | Measures the gap between predictions and actual values |
| Gradient | The slope indicating the direction of increasing error |
| Learning rate | Size of each step |
| Parameters | Values the model adjusts while learning |
How it works
- Start with initial parameter values.
- Calculate the loss and the gradient.
- Update parameters in the direction opposite the gradient.
- Repeat until the loss stops meaningfully decreasing.
The learning-rate balance
- A learning rate that is too large may overshoot or fail to settle.
- One that is too small may converge slowly.
- Practitioners often test values or use schedules that change the step size over time.
Common variants
- Batch gradient descent uses the whole data set for each step.
- Stochastic gradient descent (SGD) uses one example at a time.
- Mini-batch gradient descent uses small groups, a common practical middle ground.
Honest caveats
Gradient descent can settle into a local minimum rather than the global minimum, and results depend on the shape of the loss function and the chosen learning rate. It is a widely used method, not a guarantee of the single optimum.
FAQ
What is a gradient in plain terms?
A signal showing which way error increases, so the model steps the opposite way to reduce it.
Why does the learning rate matter?
It controls step size: too large can overshoot, too small can make learning very slow.
Is gradient descent used in deep learning?
Yes, it is commonly used to train neural networks, often in mini-batch form.
Does it always reach the lowest point?
Not always. It may stop at a local minimum depending on the landscape and starting point.
Key takeaways
- Gradient descent minimizes a loss function through repeated steps.
- It moves opposite the gradient, guided by a learning rate.
- Batch, stochastic, and mini-batch are common variants.
- Local minima and learning-rate choice are practical limitations.
Continue with what is a neural network and what is overfitting.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free