Most machine learning systems are trained to answer. Production systems need a less glamorous skill: knowing when the answer is not worth buying.

That sounds like calibration. It is partly calibration. A model that cannot rank its own errors is dangerous. But abstention is not merely a confidence threshold. It is a decision about consequences. The same probability of being wrong means different things when the action is tagging a photo, approving a trade, denying credit, escalating a security alert, or giving medical advice.

The older literature knew this before neural networks existed. Chow’s reject option framed recognition as statistical decision theory: a classifier may accept a label or reject the case for exceptional handling, and the optimal rule depends on posterior probabilities and costs.1, 2 Modern selective classification gives this a clean learning-theoretic vocabulary: coverage is how often the system answers; selective risk is the error rate on the answered subset; the frontier between them is the risk-coverage tradeoff.3

Deep learning rediscovered the same shape under new pressure. Geifman and El-Yaniv showed how to equip neural networks with selective prediction rules at a desired risk level, and later proposed SelectiveNet, which learns prediction and rejection jointly rather than bolting rejection on afterward.4, 5

Thesis: abstention is not a failure mode. It is an action. The right policy compares answering, abstaining, deferring, and computing longer in units of expected utility, not in units of raw confidence.

This is why “only answer when confidence is above 90%” is not a general safety rule. It hides the part of the problem that actually matters: the bill for being wrong, the bill for silence, and the price of one more piece of evidence.

Confidence Does Not Pay the Bill

Suppose a model predicts an action and estimates that it has error probability \(r(x)\). Let a correct action produce benefit \(B\), a wrong action cost \(L\), and abstention cost \(A\). Acting has estimated utility

\[U_{\text{act}}(x) = (1-r(x))B - r(x)L,\]

while abstaining has utility

\[U_{\text{abstain}} = -A.\]

The decision boundary is therefore

\[r(x) \leq \frac{B + A}{B + L}.\]

If errors are cheap, the model should answer even when it is not very sure. If errors are catastrophic, it should abstain at error probabilities that would be perfectly acceptable elsewhere. This boundary is not a property of the model alone. It is a property of the model embedded in a decision problem.

Franc, Prusa, and Voracek make this point in a modern unified treatment of reject-option classifiers: cost-based, bounded-risk, and bounded-coverage versions all lead back to selection rules ordered by conditional expected risk.6 A good uncertainty score is useful because it preserves that ordering. It is not useful because “uncertainty” is intrinsically sacred.

The Curve Is Only the Front Door

Risk-coverage curves are the right first picture. Sort examples from easiest to hardest according to an uncertainty score. Answer the easy ones. Reject the hard ones. As coverage falls, risk on the answered subset should improve.

That curve is elegant because it separates discrimination from deployment. It asks: can the system put likely mistakes later in the queue? A badly calibrated model can still have a useful risk-coverage curve if its ranking of easy versus hard cases is good. A numerically calibrated model can still be operationally weak if the risk curve is flat.

But a deployment policy needs more than a curve:

  • What is the cost of a false positive versus a false negative?
  • Is abstention a cheap no-op, an expensive human review, or a lost opportunity?
  • Does the system have another action, such as retrieving more context, running more samples, asking a verifier, or escalating to a specialist?
  • Does extra computation help uniformly, or mainly on a narrow band of cases?

The last question is the bridge to value of information. Howard’s information value theory asks how much a decision-maker should pay to reduce uncertainty before acting.7 Russell and Wefald brought the same idea into AI as rational metareasoning: computation itself is an action with costs and expected benefits.8

That is the missing layer in many LLM and agent systems. “Answer or abstain” is already a decision problem. “Think longer or stop” is the same problem one level up.

Five Policies in the Sandbox

The simulator below creates synthetic binary decisions. Each case has a true probability that the base model’s proposed action is correct. The model observes a noisy estimate of its own error probability. Extra computation can improve the answer and the error estimate, but it costs resources and helps most in the middle: easy cases do not need it, and some hard cases are traps.

The chart compares five policies:

  • always answer;
  • a fixed confidence cutoff;
  • a selective-risk rule calibrated to a target answered-risk;
  • a cost-aware rule that compares action utility with abstention utility;
  • a value-of-information rule that computes longer only when the estimated improvement is worth its cost.
Mean utility Answered risk Target risk

Deterministic simulation, 1,600 test cases and a separate calibration set. Utility is per case: correct answers gain 1, wrong answers lose the selected error cost, abstentions pay the selected abstention cost, and extra computation pays the selected compute cost.

Try raising the error cost. The cost-aware policy becomes more conservative without changing the model. Try raising the abstention cost. The same model now answers more often because silence has become expensive. Lower score quality and watch selective risk become harder to control: abstention depends on ranking mistakes before they happen.

The value-of-information policy is deliberately modest. It does not spend more compute on every hard case. It spends compute where the estimated post-compute decision is likely to cross a utility boundary. This is the operational version of “think longer”: not more tokens everywhere, but more evidence where the evidence can change the action.

Ship a Stop Policy

For an AI system that can harm users, money, or trust, I would not ship a single confidence threshold. I would ship a small decision policy:

  1. Estimate the conditional risk of the proposed action.
  2. Estimate the utility of acting, abstaining, deferring, and computing longer.
  3. Calibrate those estimates on held-out data that resembles deployment.
  4. Report coverage, selective risk, utility, compute rate, and subgroup slices.
  5. Recalibrate when the population, action set, or cost model changes.

The cost model does not need to be perfect to be useful. The point is to make it explicit. A hidden cost model is still a cost model; it is just harder to audit.

Learning-to-defer research is pushing this idea toward human-AI teams. Recent benchmarks emphasize that deferral policies depend on expert quality, capacity, and heterogeneity, not just on model confidence.9 That should sound obvious to anyone who has seen a real operations queue. It is also exactly why the abstention problem cannot be solved inside the softmax layer.

More Than One Pause

The next interesting frontier is not merely “better uncertainty.” It is decision-calibrated uncertainty for agentic systems.

An agent does not have one reject option. It has many possible pauses: retrieve, sample again, run a verifier, ask a human, narrow the action, defer execution, hedge the position, or refuse. Each pause has a cost. Each cost depends on the state. Each state is changing because the agent’s own computations reveal information.

That suggests a practical research program:

  • learn risk scores that preserve the ordering of conditional loss, not just probability calibration;
  • learn compute-value scores that predict when additional reasoning changes the final decision;
  • evaluate policies by utility under explicit cost models, not just accuracy;
  • stress-test under shifted costs, shifted populations, and limited human capacity;
  • expose the cost model in logs so the system can be audited after failures.

Abstention is sometimes framed as humility. That is emotionally right, but technically incomplete. A system abstains well when it has learned the economics of being wrong, being silent, and buying one more piece of evidence.

Source Notes

  1. C. K. Chow, “An Optimum Character Recognition System Using Decision Functions,” IRE Transactions on Electronic Computers, 1957. PDF. ↩

  2. C. K. Chow, “On Optimum Recognition Error and Reject Tradeoff,” IEEE Transactions on Information Theory, 1970. DOI summary. ↩

  3. Ran El-Yaniv and Yair Wiener, “On the Foundations of Noise-free Selective Classification,” Journal of Machine Learning Research, 2010. JMLR. ↩

  4. Yonatan Geifman and Ran El-Yaniv, “Selective Classification for Deep Neural Networks,” NeurIPS 2017. Proceedings. ↩

  5. Yonatan Geifman and Ran El-Yaniv, “SelectiveNet: A Deep Neural Network with an Integrated Reject Option,” ICML 2019. PMLR. ↩

  6. Vojtech Franc, Daniel Prusa, and Vaclav Voracek, “Optimal Strategies for Reject Option Classifiers,” Journal of Machine Learning Research, 2023. PDF. ↩

  7. Ronald A. Howard, “Information Value Theory,” IEEE Transactions on Systems Science and Cybernetics, 1966. NASA ADS. ↩

  8. Stuart Russell, “Rationality and Intelligence,” IJCAI Computers and Thought Award Lecture, 1995; see also Russell and Wefald’s work on rational metareasoning. PDF. ↩

  9. Jean V. Alves et al., “A benchmarking framework and dataset for learning to defer in human-AI decision-making,” Scientific Data, 2025. Project page. ↩