A probability is not a mood, although plenty of dashboards treat it like one.

When a model says 0.8, it is making a frequency claim:

among cases like this, the event should happen about 80 percent of the time

That claim is different from being accurate. It is different from ranking examples correctly. It is different from having a high AUC. A model can sort cases well while systematically overpricing every probability. Another model can be calibrated by always predicting the base rate and still be useless for ranking.

Calibration is the discipline of asking whether the numbers mean what they claim to mean.

The catch is that calibration is not directly observed. We never get infinitely many copies of exactly the same 0.73 forecast under exactly the same conditions. We group forecasts into bins, smooth them, score them, slice them by subgroup, and hope the diagnostic we drew is close to the property we care about.

The reliability diagram has a bin width. That tiny measuring choice is where many confident calibration claims start to wobble.

What 0.8 Is Allowed to Mean

For a binary event \(Y\in\{0,1\}\) and a forecast \(P\in[0,1]\), perfect calibration means:

\[E[Y \mid P=p] = p.\]

Equivalently:

among all cases assigned probability p,
the empirical event rate should be p

Dawid’s “well-calibrated Bayesian” paper made calibration a central property of sequential probability forecasts, while also showing that coherence and calibration have a subtle relationship.1 Forecast verification had already been using this idea for weather probabilities: if you say 30 percent rain often enough, roughly 30 percent of those days should rain.

In machine learning, the same question shows up whenever a probability becomes an action: triage, lending, fraud review, recommender throttles, medical screening, abstention, tool-use gates, human handoff, or position sizing.

If 0.8 means “pretty high” rather than “80 percent,” downstream systems will eventually charge interest.

A Small Calibration Bench

The lab below simulates a binary classifier. There is a true latent probability, a model score, a calibration split used to fit one temperature parameter, and an evaluation split used for diagnostics.

The raw model is intentionally overconfident by default. Temperature scaling divides logits by a fitted scalar \(T\):

\[\tilde p = \sigma\left(\frac{z}{T}\right).\]

For an overconfident model, the fitted \(T\) is usually greater than one. This softens probabilities without changing the score ordering.

Try four moves:

  1. raise Confidence scale and watch the red reliability curve bend away from the diagonal;
  2. change Reliability bins and watch ECE move even though the data did not;
  3. raise Group skew and notice that a tolerable global curve can hide subgroup damage;
  4. change Decision threshold and remember that calibrated probabilities are inputs to decisions, not decisions by themselves.
Raw model Temperature scaled Perfect reliability Group A Group B Threshold / bins

Deterministic synthetic experiment. A temperature is fitted on the calibration split by minimizing log loss, then reliability, ECE, Brier score, and subgroup diagnostics are computed on the evaluation split. The displayed Brier decomposition uses equal-width bins, so the residual term is a warning about binning error rather than a new law of nature.

In the default run, the raw model is a familiar neural-network caricature: mostly right about ranking, too confident about magnitude. The red curve bends away from the diagonal. Temperature scaling, fitted on held-out calibration rows, softens logits and pulls the green curve closer to the diagonal.

Guo, Pleiss, Sun, and Weinberger documented this pattern in modern neural networks and found that temperature scaling is a surprisingly strong post-processing baseline across several architectures and datasets.2

The word “baseline” matters. Temperature scaling can fix a global confidence scale. It cannot repair every local pathology. If a subgroup is overpredicted and another is underpredicted, one global temperature may make the average plot look calmer while the local plots remain sick.

ECE Moves When the Ruler Moves

Expected calibration error is often written as:

\[\mathrm{ECE} = \sum_{b=1}^B \frac{n_b}{n} \left| \operatorname{acc}(b)-\operatorname{conf}(b) \right|.\]

For each bin \(b\), \(\operatorname{conf}(b)\) is the average predicted probability and \(\operatorname{acc}(b)\) is the observed event rate. The formula is useful because it compresses a reliability diagram into one number.

It is also an estimate with knobs:

  • how many bins;
  • equal-width or equal-frequency bins;
  • whether empty bins count;
  • whether labels are scarce in the tails;
  • whether subgroups are pooled;
  • whether the same sample was used for fitting and evaluation.

Move Reliability bins. The underlying forecasts do not change. The ECE does. That does not make ECE useless. It makes it a measurement with a resolution limit.

Naeini, Cooper, and Hauskrecht proposed Bayesian binning into quantiles partly because calibration by naive binning can be unstable and sensitive to bin choice.3 More recent work on reliability diagrams makes the same practical point: the diagram is a statistical estimator, not a photograph of a population truth.4

The right habit is to show the curve, the histogram or point sizes, the sample size, the binning scheme, and subgroup slices. A single ECE number without those details is a ruler with the tick marks cropped out.

Brier Keeps Three Ledgers

Brier’s original probability score is the mean squared error of probability forecasts:5

\[\mathrm{Brier} = \frac{1}{n} \sum_i (p_i-y_i)^2.\]

It is a proper scoring rule for binary probabilities: truthful probabilities minimize expected score. But it is not only a calibration metric. Murphy’s decomposition separates the Brier score into uncertainty, reliability, and resolution terms:6

\[\mathrm{Brier} = \mathrm{reliability} - \mathrm{resolution} + \mathrm{uncertainty}.\]

Reliability penalizes probabilities that do not match observed frequencies. Resolution rewards forecasts that separate cases into bins with different event rates. Uncertainty is the irreducible base-rate term.

This matters because a model can improve calibration while losing useful resolution. A forecaster that always predicts the base rate can be calibrated and still dull. Gneiting and Raftery’s review of proper scoring rules emphasizes the old forecast-verification distinction: calibration is necessary, but sharpness or refinement is also part of useful probabilistic forecasting.7

In the lab’s Brier panel, the green temperature-scaled forecast usually improves both reliability and total Brier score under the default overconfidence setting. But do not promote this into a universal theorem. A calibration method is a post-processing model. It can overfit, underfit, or improve the wrong slice.

The Average Can Wear a Mask

The subgroup panel is there because global calibration can average away local harm.

Suppose group A is underpredicted and group B is overpredicted. If the groups appear in the right proportions, the global reliability curve may look acceptable. The model is not calibrated for either group; it is calibrated for a mixture.

That distinction matters whenever decisions are made on individuals or subpopulations. A global statement like:

our 80 percent scores happen 80 percent of the time

does not imply:

our 80 percent scores mean the same thing for every slice that matters

This is the same reason calibration monitoring should be sliced by deployment regime, language, device, geography, market condition, data source, label delay, and any subgroup where error has asymmetric cost.

Probabilities Still Need a Policy

A calibrated probability still needs a utility function.

If false negatives cost three times as much as false positives, the optimal threshold is not necessarily 0.5. If review capacity is fixed, the rule might be “send the top 2 percent to humans.” If abstention is allowed, the rule might depend on the value of information. If the forecast enters a Kelly sizing rule, small probability errors can become large leverage errors.

Calibration makes probabilities usable. It does not decide what use is wise.

This is why I like to separate four layers:

score ranking
probability calibration
decision threshold or policy
outcome monitoring

Mixing them creates confusion. Accuracy can improve while calibration worsens. Calibration can improve while utility falls under a bad threshold. Utility can look good in aggregate while a subgroup is systematically mispriced.

Questions for a Calibration Report

Before I trust a probability-calibration report, I want to know:

  • whether calibration was fit on data separate from evaluation;
  • which scoring rules were reported, not only ECE;
  • the binning scheme and bin count used for reliability diagrams;
  • whether point sizes or confidence intervals were shown;
  • whether ECE is stable across plausible bin counts;
  • whether Brier reliability, resolution, and uncertainty were inspected;
  • whether subgroups and deployment regimes were sliced separately;
  • whether calibration was monitored after distribution shift;
  • whether the decision threshold was chosen from a cost model;
  • whether the model is calibrated where the action boundary actually lies.

The reliability diagram is one of the most useful plots in machine learning evaluation. It is also easy to overread. It estimates a conditional frequency function from finite data using bins that you chose.

That does not make it weak. It makes it honest. A probability forecast is a contract with future frequencies, and every contract needs an audit trail: sample size, slices, scores, bin widths, and the downstream decisions that will spend the numbers.

Paper Trail

  1. A. Philip Dawid, “The Well-Calibrated Bayesian”, Journal of the American Statistical Association, 1982. ↩

  2. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, Proceedings of Machine Learning Research, 2017. ↩

  3. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, “Obtaining Well Calibrated Probabilities Using Bayesian Binning”, AAAI, 2015. ↩

  4. Mikhail Popoviciu, Michael P. Kim, and S. Matthew Weinberg, “Metrics of Calibration for Probabilistic Predictions”, Journal of Machine Learning Research, 2022. ↩

  5. Glenn W. Brier, “Verification of Forecasts Expressed in Terms of Probability”, Monthly Weather Review, 1950. ↩

  6. Allan H. Murphy, “A New Vector Partition of the Probability Score”, Journal of Applied Meteorology, 1973. ↩

  7. Tilmann Gneiting and Adrian E. Raftery, “Strictly Proper Scoring Rules, Prediction, and Estimation”, Journal of the American Statistical Association, 2007. ↩