What Is Gradient Boosting? A Plain-Language Explanation

Updated · By Viviai

Gradient boosting is an ensemble machine learning technique that builds a sequence of weak models, usually shallow decision trees, where each new model focuses on correcting the errors of the previous ones, combining them into one stronger prediction model.

The simple analogy

Think of a student who reworks only the questions they got wrong, then repeats the process. Each round targets the remaining mistakes, so errors shrink step by step. Gradient boosting trains models in this sequential, error-correcting way.

How it differs from a random forest

Both combine decision trees, but a random forest builds independent trees and averages them, while gradient boosting builds trees one after another, each learning from the errors of the ensemble so far.

How it works

Step Description
Start with a simple model Produce an initial prediction
Compute errors Find the residual, or remaining error
Add a weak model Train a tree to predict the error
Update predictions Combine with a learning rate
Repeat Continue until a stopping point

Key terms

Strengths

Honest caveats

It can overfit if too many trees or a high learning rate are used, and tuning takes care. It is also generally slower to train than a single tree because models are built sequentially.

FAQ

What is a weak learner?

A simple model that performs slightly better than a basic guess, often a shallow tree.

What is the learning rate?

A parameter that scales each tree's contribution to keep learning gradual.

Is gradient boosting supervised?

Yes, it is trained on labeled data for classification or regression.

How is it different from bagging?

Boosting corrects errors sequentially, while bagging trains independent models and averages them.

Key takeaways

Continue with what is a random forest and what is a decision tree.

Sources

  1. https://scikit-learn.org/stable/modules/ensemble.html
  2. http://www.shturl.cc/fe25075e9dba480d1ac53ea5aae0cea0
  3. https://ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/materials/Making%20Sense%20of%20Gradient%20Boosting%20in%20Classification_%20A%20Clear%20Guide%20__%20Reader%20View.pdf

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free