Logits and Softmax Explainer
Logits are the raw scores a model produces before turning them into a normalized distribution. In a classifier, there may be one logit for each class. In a language model, there is typically one for every token in the output vocabulary at each generation step.
Softmax transforms those scores by exponentiating them and dividing each result by the total. The outputs are positive and add up to one, so they can be used to rank alternatives or sample a next token. Increasing one logit relative to the others increases its softmax share.
The transformation depends on differences among scores, not their absolute level. A temperature parameter can make the distribution sharper or flatter before sampling. Lower temperature concentrates weight on higher-scoring options; higher temperature spreads weight across more alternatives.
Softmax values are often described as probabilities, but they are not automatically well calibrated. A model can assign a high value to a wrong class, especially on unfamiliar data. Reliability must be tested against observed outcomes, and decision thresholds should reflect the real costs of different errors.
Logits are a model's raw scores, while softmax converts a set of scores into values that sum to one.
Key terms include logit, softmax, class score, normalization, vocabulary, temperature, sampling, ranking and calibration.
Softmax compares every logit with every other logit in the set and returns normalized values whose total is one.
Because exponentiation magnifies score differences, a modest lead in logits can become a much larger lead after normalization.
A high softmax value means an option dominates the model's current alternatives; it does not prove that the option is true or correct.
Calibration tests compare predicted confidence with actual accuracy and can reveal when a model is systematically overconfident or underconfident.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.