What Is the Adam Optimizer? A Simple Explanation for Beginners
The Adam optimizer (short for adaptive moment estimation) is a training algorithm that gives every weight its own adjustable step size by keeping two running averages: one of past gradients and one of past squared gradients. It was introduced by Diederik Kingma and Jimmy Ba in 2014 and combines ideas from two older methods — momentum and RMSProp — into one algorithm that usually works with little manual tuning. If you have read about gradient descent, think of Adam as a version that both remembers the direction it has been moving and scales each step according to how noisy that weight's gradients are.
Because Adam needs little memory and handles sparse or noisy gradients well, it is a common starting choice when training a neural network, including many deep learning models.
The Problem Adam Solves
Plain gradient descent uses one shared learning rate for every weight and takes a step based only on the current gradient. This creates two practical issues:
- A single learning rate may be too large for some weights and too small for others.
- Gradients can be noisy or point in conflicting directions from one batch to the next.
Adam addresses both by looking at recent history rather than a single gradient. It adapts the step for each weight separately, which removes much of the need to hand-tune a global learning rate, though hyperparameter tuning still matters for difficult tasks.
The Two Moments
Adam maintains two values for every parameter, both starting at zero:
- First moment (m): an exponential moving average of the gradients, similar to momentum. It smooths the direction of updates.
- Second moment (v): an exponential moving average of the squared gradients. It measures how large and noisy the gradients have been.
At each step, with gradient (g), the two moments are updated as
[ m = \beta_1 m + (1-\beta_1)g, \qquad v = \beta_2 v + (1-\beta_2)g^2 ]
where the square is applied element-wise. The final step divides the smoothed gradient by the square root of the smoothed squared gradient, so weights with large or noisy gradients take smaller steps.
Why Bias Correction Is Needed
Because (m) and (v) start at zero and the decay rates are close to one, their early values are biased toward zero. Adam corrects this by dividing each moment by (1-\beta^t),
[ \hat{m} = \frac{m}{1-\beta_1^t}, \qquad \hat{v} = \frac{v}{1-\beta_2^t} ]
The correction matters most in the first few iterations, when the averages otherwise underestimate the true gradient. As training continues, the factor approaches one and has little effect.
Common Default Settings
The original paper proposed values that most frameworks, such as Keras and PyTorch, use as defaults.
| Hyperparameter | Common value | Role |
|---|---|---|
| Learning rate (\eta) | 0.001 | Overall size of each update |
| Beta 1 (\beta_1) | 0.9 | Decay for the first moment (gradient) |
| Beta 2 (\beta_2) | 0.999 | Decay for the second moment (squared gradient) |
| Epsilon (\epsilon) | (10^{-8}) | Small constant that prevents division by zero |
These defaults are a reasonable starting point, not fixed rules; some models benefit from a smaller learning rate or adjusted betas.
Adam vs SGD
| Aspect | Adam | SGD (with momentum) |
|---|---|---|
| Step size | Adapted per weight | One shared learning rate |
| Tuning effort | Often works with defaults | Usually needs more manual tuning |
| Noisy or sparse gradients | Handles them well | Can be more sensitive |
| Early progress | Often fast | Can be slower to start |
| Generalization | Usually strong, sometimes weaker | Can generalize well when tuned |
In practice Adam often reaches a low training error quickly, while some studies find well-tuned SGD can produce comparable or slightly better generalization on certain tasks.
Variants and Limitations
- AdamW separates weight decay from the gradient-based update and is a common choice for training transformer models, including those used in the attention mechanism.
- Adam does not remove the need to check results: a fast-falling training loss does not by itself prove the model generalizes, so a separate loss function on validation data is still useful.
- Like other gradient-based methods, Adam relies on backpropagation to compute the gradients it uses.
Frequently Asked Questions
What is Adam optimizer in simple words?
It is an optimizer that keeps a smoothed average of past gradients and of past squared gradients, then uses both to choose a separate step size for each weight. This makes it robust and reduces manual tuning.
Why is it called Adam?
The name comes from adaptive moment estimation, because the algorithm estimates the first and second moments of the gradient.
What is the default learning rate for Adam?
Many frameworks use 0.001 as the default, along with beta 1 of 0.9 and beta 2 of 0.999, but these are starting values rather than universal rules.
Is Adam better than SGD?
Adam often converges with less tuning and handles noisy gradients well, but a carefully tuned SGD can sometimes generalize comparably. The choice depends on the task and how much tuning is practical.
What is the difference between Adam and AdamW?
AdamW changes how weight decay is applied, keeping it separate from the momentum-style update. This often works better for transformer and large language model training.
Summary
Adam is an adaptive optimizer that combines a momentum-like average of gradients with an RMSProp-like average of squared gradients, then applies bias correction so early updates are not underestimated. Its default settings make it a practical starting choice for many neural networks, while variants such as AdamW are widely used for transformers. For related ideas, see gradient descent, backpropagation, and batch normalization.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free