AI Model Calibration Explainer
Model calibration asks whether predicted probabilities correspond to observed frequencies. If a well-calibrated classifier assigns 70 percent confidence to many comparable cases, roughly 70 percent of those cases should be correct. Calibration concerns probability quality, not only the winning label.
A model can be accurate but overconfident, or less accurate yet honest about uncertainty. Reliability diagrams group predictions by confidence and compare average confidence with actual outcomes. Metrics such as expected calibration error summarize gaps, though the grouping choices affect the number.
Post-processing methods such as temperature scaling can adjust probabilities using a separate validation set without changing the predicted class ordering. Other approaches retrain or recalibrate for specific model families and operational conditions.
Calibration can degrade under distribution shift and can differ across classes or groups. A single global score may hide harmful local errors. Decisions also require thresholds, costs, and the option to defer uncertain cases; a calibrated probability does not decide what risk is acceptable.
Model calibration measures whether predicted probabilities match how often outcomes actually occur, such as whether 70-percent predictions are right about 70 percent of the time.
Key terms include confidence, probability, reliability diagram, calibration curve, expected calibration error, temperature scaling, threshold and abstention.
Predictions are binned by confidence, then the average predicted probability in each bin is compared with the observed accuracy for those cases.
Post-hoc calibration fits a transformation on validation data. That data must be separate from training and representative of the deployment environment.
A model can be calibrated overall while remaining overconfident for one class, subgroup, location, device, or range of inputs.
Teams should examine class-conditional and subgroup curves, retest after drift, preserve an abstention route, and connect probability thresholds to real consequences.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.