What Is Gradient Boosting? A Plain-Language Explanation
Gradient boosting is an ensemble machine learning technique that builds a sequence of weak models, usually shallow decision trees, where each new model focuses on correcting the errors of the previous ones, combining them into one stronger prediction model.
The simple analogy
Think of a student who reworks only the questions they got wrong, then repeats the process. Each round targets the remaining mistakes, so errors shrink step by step. Gradient boosting trains models in this sequential, error-correcting way.
How it differs from a random forest
Both combine decision trees, but a random forest builds independent trees and averages them, while gradient boosting builds trees one after another, each learning from the errors of the ensemble so far.
How it works
| Step | Description |
|---|---|
| Start with a simple model | Produce an initial prediction |
| Compute errors | Find the residual, or remaining error |
| Add a weak model | Train a tree to predict the error |
| Update predictions | Combine with a learning rate |
| Repeat | Continue until a stopping point |
Key terms
- Weak learner: a simple model, often a shallow tree.
- Learning rate: controls how much each new tree contributes.
- Residual: the difference between actual and predicted values.
Strengths
- Often performs well on structured, tabular data.
- Handles classification and regression.
- Captures complex patterns through sequential correction.
Honest caveats
It can overfit if too many trees or a high learning rate are used, and tuning takes care. It is also generally slower to train than a single tree because models are built sequentially.
FAQ
What is a weak learner?
A simple model that performs slightly better than a basic guess, often a shallow tree.
What is the learning rate?
A parameter that scales each tree's contribution to keep learning gradual.
Is gradient boosting supervised?
Yes, it is trained on labeled data for classification or regression.
How is it different from bagging?
Boosting corrects errors sequentially, while bagging trains independent models and averages them.
Key takeaways
- Gradient boosting adds trees that correct earlier errors.
- It is a sequential ensemble, not independent trees.
- It works well on tabular data.
- Learning rate and tree count need careful tuning.
Continue with what is a random forest and what is a decision tree.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free