What Is a Loss Function? A Plain-Language Explanation

Updated · By Viviai

A loss function is a mathematical rule that measures how far a model's predictions are from the correct values, producing a single number that is larger when the model is wrong; training adjusts the model's parameters to make that number as small as practical.

What a loss function does

During training, the model makes a prediction for each input, and the loss function compares that prediction with the known answer, often called the ground truth. If the prediction is accurate, the loss is small; if it is far off, the loss grows. A loss of zero means the model makes no error on that sample.

The loss is often divided by the number of samples so it can be read as an average error per sample, independent of how large the dataset is.

Loss, cost, and objective: the terms

The terms are closely related and sometimes used interchangeably.

Term Meaning
Loss Error on a single prediction or sample
Cost The aggregated loss over the whole training set
Objective function Any function training aims to minimize or maximize

A loss function is the specific case where minimizing the value is the training goal. Some objectives also include a regularization term that penalizes overly complex models.

Common loss functions

Different problems use different loss functions.

Loss function Typical task Idea
Mean squared error (L2) Regression Squares the difference between prediction and target
Binary cross-entropy Two-class classification Penalizes confident wrong probabilities
Categorical cross-entropy Multi-class classification Compares predicted class probabilities with labels
Hinge loss Support vector machines Penalizes points on the wrong side of a margin
Log-loss Logistic regression Rewards well-calibrated probabilities

For a regression problem, the mean squared error is written as:

[ \mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}\left(\hat{y}_i - y_i\right)^2 ]

where (\hat{y}_i) is the prediction and (y_i) is the correct value.

How loss drives training

The loss alone only reports the error. An optimization algorithm, commonly gradient descent, uses the loss to update each parameter in the direction that reduces the error. Repeating this cycle over many iterations gradually lowers the loss, which is how the model learns.

Training loss versus validation loss

Training loss is measured on the data the model learns from. Validation loss is measured on data the model has not seen, and it helps reveal overfitting. If training loss keeps falling while validation loss rises, the model may be memorizing the training data rather than generalizing.

Honest caveats

A low training loss does not by itself indicate a useful model, because the number can be driven down by overfitting. Loss values are also not directly comparable across different functions, so a model with a smaller MSE is not automatically judged the same way as one with a smaller cross-entropy.

FAQ

What is the difference between a loss function and a cost function?

Loss usually refers to the error on one sample, while cost refers to the aggregated loss over the entire dataset. In practice, the two terms are often used interchangeably.

Why do classification models use cross-entropy?

Cross-entropy penalizes confident wrong answers heavily, which encourages the model to produce calibrated class probabilities.

Can a loss of zero be achieved on real data?

A loss of zero means error-free predictions, which is rare on real-world data and can indicate overfitting if it occurs on the training set.

Is the loss function the same as the optimization algorithm?

No. The loss function measures the error; the optimization algorithm, such as gradient descent, uses that measurement to update the parameters.

Key takeaways

Continue with what is gradient descent and what is regularization.

Sources

  1. https://www.ibm.com/think/topics/loss-function
  2. https://www.cs.cornell.edu/courses/cs3780/2026sp/lectures/lecturenote01_MLsetup.html
  3. https://deeplearning.cs.cmu.edu/S26/documents/Recitation_0_Series/0.20/S26_Rec0.20_Losses_Part1.pptx.pdf

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free