What Is Softmax? A Simple Explanation of the Softmax Function

Updated · By Viviai

Softmax is a function that converts a list of raw numbers, called logits, into a list of probabilities where every value lies between 0 and 1 and all the values add up to 1. It is commonly placed at the output layer of a neural network that chooses among several classes, because the result can be read as the model's confidence for each option. If you have met an activation function before, softmax is the one used at the final layer rather than inside the hidden layers.

The same function also appears inside the attention mechanism, where it turns scores into weights that decide how much each part of the input contributes.

How Softmax Works

For a vector of logits (z) with (K) elements, softmax applies the exponential function to each value and divides by the sum of all exponentials,

[ \sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} ]

The exponential makes larger logits produce larger probabilities, while the division guarantees that everything sums to one. Even a very negative logit still produces a small positive number rather than zero.

A Small Example

Suppose a classifier sorting images into three classes produces the logits in the table below.

Class Raw logit Softmax probability
Cat 2.0 About 0.66
Dog 1.0 About 0.24
Bird 0.1 About 0.10

The raw numbers are not probabilities because they do not sum to one and can be negative or larger than one. After softmax, the same ordering is kept, but the values now share a total of one and are easier to interpret.

Why Use Exponentials

The exponential function has two useful properties here:

  1. It is always positive, so every output is greater than zero.
  2. It amplifies gaps: a slightly larger logit can become a noticeably larger probability.

This makes the model's preference clearer than a simple linear rescaling would, while keeping the operation smooth and differentiable so that backpropagation can still update the network.

Numerical Stability

Computing (e^{z}) directly can produce very large numbers when a logit is big. A common implementation subtracts the largest logit from every value first,

[ z'_i = z_i - \max(z) ]

This does not change the result, because adding or subtracting a constant cancels out in the division, but it keeps the exponentials within a safer range.

Softmax vs Sigmoid

Aspect Softmax Sigmoid
Output Probabilities across several classes One value between 0 and 1
Sum Outputs add to one No sum constraint
Typical use Multiclass output layer Binary choice or hidden units
Relationship With two classes it reduces to a sigmoid Generalizes to softmax for many classes

For a problem with only two outcomes, softmax over two logits gives the same kind of result as a single sigmoid, which is why sigmoid is often described as the two-class version. This also connects softmax to multinomial logistic regression.

Temperature and Cross-Entropy

Limitations

Frequently Asked Questions

What is softmax in simple words?

It is a function that turns a set of raw scores into probabilities that are all between zero and one and sum to one, so the largest score gets the highest probability.

Why is softmax used in neural networks?

It converts the final layer's logits into interpretable probabilities for each class, and it works smoothly with cross-entropy loss during training.

What is the difference between softmax and sigmoid?

Softmax produces a set of probabilities that sum to one across several classes, while sigmoid gives a single value for a binary choice. Softmax with two classes behaves like a sigmoid.

Does softmax always pick the largest value?

It gives the largest probability to the largest logit, but it keeps small probabilities for the other options rather than returning a hard choice, which is why it is called a soft version of the maximum.

What is temperature in softmax?

Temperature is a divisor applied to the logits that controls how sharp the distribution is: low temperature concentrates probability, high temperature spreads it out.

Summary

Softmax normalizes a vector of logits into a probability distribution that sums to one, using exponentials and a stable implementation that subtracts the maximum. It is a standard output for multiclass classification and a key step in attention, often paired with cross-entropy loss. For related concepts, see activation functions, logistic regression, and the transformer.

Sources

  1. https://web.stanford.edu/~jurafsky/slp3/old_sep21/7.pdf
  2. https://en.wikipedia.org/wiki/Softmax_function
  3. https://courses.grainger.illinois.edu/ECE448/sp2026/slides/lec13.pdf
  4. https://aiwiki.ai/wiki/softmax

Learn by doing

Read a guide, then practice with bite-sized AI lessons you can take offline.

Start learning free