What Is a Learning Rate? Step Size in Machine Learning Explained
A learning rate is a hyperparameter that controls how large a step a machine learning model takes each time it updates its weights during training. It scales the gradient: a small value moves the weights gently, while a large value moves them aggressively. The learning rate is often written as the Greek letter eta (η) or alpha (α), and it sits at the center of whether a model converges slowly, trains smoothly, or diverges.
The one equation that matters
In plain gradient descent, the weights at the next step follow this rule:
[ \theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t) ]
Here (\theta_t) is the current set of weights, (\nabla L(\theta_t)) is the gradient of the loss (the direction that would make the loss go up), and (\eta) is the learning rate. The minus sign flips the direction so the model walks downhill, toward lower loss. The gradient points the way; the learning rate decides how far to walk.
You can find the same idea in more detail in the gradient descent explainer.
Why the number matters: walking down a valley
Think of training as walking down the side of a valley toward the lowest point in thick fog. You can feel which way the ground slopes, but you cannot see the bottom.
- Take tiny, cautious steps and you will get there eventually, but the walk takes a long time and you may stall on a small dip along the way.
- Take long, bold strides and you cover ground fast, but you can leap straight over the valley floor, bounce up the opposite wall, and even climb higher with every step.
The learning rate is the length of those strides. The same gradient can produce a stable model or a broken one depending only on this multiplier.
Too high vs. too low
| Learning rate | What happens during training | Typical sign |
|---|---|---|
| Too small | Weight updates are tiny; progress is slow and the model may stop early in a poor local minimum | Loss falls, but very slowly, and barely moves after many epochs |
| In a sensible range | Loss decreases steadily and settles near a low value without wild swings | Smooth loss curve with small, shrinking oscillations |
| Too large | Updates overshoot the minimum; the loss oscillates, plateaus at a high value, or grows without bound | Loss jumps around or turns into NaN |
The numerical value alone tells you nothing. A rate that works for one model can be far too large for another, because the effect depends on the scale of the loss, how the features are normalized, and the optimizer. A value of 0.01 is common, not universal.
Common starting points
There is no single correct number, but these ranges show up often enough to be useful starting guesses:
| Setup | Typical starting learning rate |
|---|---|
| Plain gradient descent | 0.01 to 0.1 |
| SGD with momentum | 0.01 to 0.05 |
| Adam and other adaptive optimizers | 0.001 (1e-3) |
| Fine-tuning a large pretrained model | 1e-5 to 5e-5 |
Treat these as first guesses to test, not constants. Many practitioners start with a default such as 0.001 for Adam and then adjust based on the loss curve. If the loss explodes, lower it by roughly a factor of 10; if it barely moves, raise it.
Learning rate schedules
A constant rate is the simplest choice, but most deep networks do better when the rate changes over time. Early on, larger steps help the model approach the region of a good solution; later, smaller steps let it settle without overshooting.
- Step decay: keep the rate flat, then multiply it by a factor (such as 0.1) at set epochs.
- Exponential decay: shrink the rate by a fixed percentage each epoch.
- Cosine decay: lower the rate following a cosine curve, with a slow start, a long steady decline, and a rapid drop near the end.
- Warmup: start with a very small rate and raise it over the first few batches. This is common when training transformers, where a large early rate can destabilize the fresh weights.
A 2019 study catalogued 13 different learning rate functions and found that dynamic schedules often reach higher accuracy than a fixed constant, at the cost of being harder to tune.
Adaptive optimizers
Some algorithms do not use one global learning rate. They keep a separate, self-adjusting rate for every parameter, based on the history of its gradients.
- AdaGrad shrinks the rate for parameters with large, frequent gradients.
- RMSProp tracks a moving average of squared gradients to keep the adaptation from shrinking to zero.
- Adam combines moving averages of both the gradients and their squares, which is why a default around 0.001 works across many tasks.
The Adam optimizer guide walks through those moving averages. Even adaptive methods still expose a base learning rate, so the concept never disappears; it simply becomes a starting point that the algorithm modifies.
How to choose one in practice
- Start with a known default for your optimizer (for example, 1e-3 for Adam).
- Plot training and validation loss each epoch rather than watching a single number.
- If the loss is flat, try a rate roughly 3 to 10 times larger.
- If the loss oscillates or becomes NaN, divide the rate by 10 and check that input features are normalized.
- Once a stable range works, add a schedule such as cosine decay and compare the result.
A learning rate finder can automate the first step: it runs a short training while steadily increasing the rate and records where the loss falls fastest, suggesting a value just below that point.
Frequently asked questions
Is a smaller learning rate always safer? Not always. A very small rate is unlikely to diverge, but it can make training painfully slow and leave the model stuck in a poor local minimum. The goal is a rate that makes steady progress, not the smallest number you can find.
What is the difference between learning rate and step size? They refer to the same quantity. "Step size" is the older term from numerical optimization, while "learning rate" is the common name in machine learning. Both describe how far each weight update moves.
Does the learning rate affect model accuracy? Yes, indirectly. It controls how completely and how stably the optimization converges. A poorly chosen rate can stop the model far below the accuracy it is capable of, even when the data and architecture are sound.
How does learning rate relate to batch size? They interact. Larger batches produce smoother gradients and often tolerate a slightly larger learning rate. When you scale the batch size up, you sometimes scale the learning rate up as well, then verify on the loss curve.
Key takeaways
- The learning rate scales each weight update and is written as η or α.
- Too large and the loss overshoots or diverges; too small and training stalls or crawls.
- The value is meaningful only relative to the optimizer, loss scale, and data.
- Schedules and adaptive optimizers change the rate over time rather than holding it fixed.
- Choose it by watching the loss curve and adjusting in roughly 10-fold steps, not by guessing once.
For the surrounding concepts, see the loss function and RMSNorm explainers, and the Adam optimizer guide for adaptive updates.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free