What Is Regularization? A Plain-Language Explanation
Regularization is a technique that adds a penalty to the training loss to keep model weights small or sparse, reducing overfitting and helping the model generalize to unseen data.
The simple analogy
A student who memorizes every detail performs well on practice questions but struggles on new ones. Regularization acts like a gentle rule that discourages relying too heavily on any single detail, favoring simpler, more balanced answers.
L1 and L2 compared
| Type | Penalty | Typical effect | Also called |
|---|---|---|---|
| L1 | Sum of absolute weights | Drives some weights to zero | Lasso |
| L2 | Sum of squared weights | Shrinks weights toward zero | Ridge |
Why L1 creates sparsity
L1 can set weights exactly to zero, which effectively removes features and acts as feature selection. L2 keeps weights small but rarely reaches exactly zero.
The role of the penalty strength
A larger penalty simplifies the model more, while a smaller one allows more flexibility. The value is typically chosen through validation.
Honest caveats
Too much regularization can cause underfitting, where the model becomes too simple to capture the pattern. The penalty is usually tuned rather than guessed.
FAQ
What problem does regularization address?
It reduces overfitting by limiting model complexity.
What is the difference between L1 and L2?
L1 can produce sparse weights and feature selection, while L2 shrinks weights without setting them to zero.
What is weight decay?
A name often used for L2 regularization in neural networks.
Can regularization hurt performance?
Excessive regularization can lead to underfitting.
Key takeaways
- Regularization penalizes large weights.
- L1 sparsifies, L2 shrinks.
- It helps generalization.
- The strength needs tuning.
Continue with what is the bias-variance tradeoff and what is overfitting.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free