The most common AI evaluation picture is a leaderboard: model names down the left, scores across the right, winners at the top.

That picture is convenient. It is also a little too smooth. A benchmark is not a scoreboard in the abstract. It is a measuring instrument. It has items, difficulty, discrimination, coverage, reliability, contamination risk, and a range where it is informative. When the best systems crowd together near the top, the right diagnosis may not be “the models are tied.” It may be “the instrument is saturated.”

Psychometrics has lived with this problem for a long time. Rasch’s measurement work and later item response theory ask a deceptively simple question: what can we infer about a person’s latent ability from a pattern of correct and incorrect responses to items with different difficulties?1, 2 The same question now applies to language models.

Thesis: a benchmark should be treated like a test instrument, not a trophy case. Raw accuracy is only a summary after we have asked what the items measure, which ability range they separate, and whether the test has been exposed to the systems being measured.

This post is not a complaint about benchmarks. MMLU, BIG-bench, HELM, MT-Bench, and Chatbot Arena all improved the evaluation culture in different ways: breadth, task diversity, multi-metric reporting, human preference collection, and transparency.3, 4, 5, 6, 7 The point is that better instruments deserve better interpretation.

Accuracy Pretends Items Weigh the Same

The raw score on a multiple-choice benchmark is usually

\[\widehat{p}_m = \frac{1}{n}\sum_{i=1}^{n} Y_{mi},\]

where \(Y_{mi}=1\) if model \(m\) answers item \(i\) correctly. This treats every item as equally informative. A trivial item, a saturated item, a noisy item, a memorized item, and a sharp diagnostic item all receive the same weight.

Item response theory uses a different lens. In a two-parameter logistic model,

\[\Pr(Y_{mi}=1) = \sigma(a_i(\theta_m-b_i)),\]

where \(\theta_m\) is latent model ability, \(b_i\) is item difficulty, and \(a_i\) is item discrimination. The item is most informative near the ability region where the response probability is changing rapidly. Items that everyone gets right or everyone gets wrong contribute little to ranking nearby models. For multiple-choice tasks, a guessing or chance parameter can be added; the simulator below uses a 25% chance floor to mimic four-option questions.

This is the key difference between a score and a measurement model. The score says what happened. The measurement model asks where the evidence came from.

Every Instrument Has a Range

MMLU was introduced to measure broad multitask academic and professional knowledge across 57 tasks.3 BIG-bench was designed as a broad collection of tasks believed to be beyond the capabilities of then-current language models.4 HELM argued for holistic evaluation across scenarios and metrics, because language models serve many purposes and a single accuracy number hides tradeoffs.5

These projects were responses to real measurement gaps. But instruments age. Easy items become saturated. Public items can leak into training data. Model families improve in lopsided ways. A benchmark that once separated models can become a historical artifact unless it keeps enough high-information items in the region where current models differ.

This is familiar in human testing. A test made only of elementary arithmetic is not a good instrument for ranking graduate mathematicians. The issue is not that the questions are “bad.” They are simply aimed at the wrong ability range.

Exposure Is Part of the Score

Data contamination turns a benchmark from a test of generalization into a mixed test of ability and exposure. Deng et al. study contamination in modern LLM benchmarks and introduce methods such as Testset Slot Guessing to detect whether models can reconstruct held-out benchmark artifacts.8

The uncomfortable part is that contamination does not need to be malicious. Many benchmark examples are public. Training corpora are huge. Instruction-tuning and evaluation-driven model development create feedback loops. Once a benchmark is important, it becomes part of the environment it is supposed to measure.

Item response theory helps with difficulty and discrimination. It does not, by itself, make a contaminated item clean. Psychometrics gives us a language for instrument quality, not a ritual that purifies bad data.

Arena Is Still Measurement

Pairwise preference platforms such as Chatbot Arena move away from static ground-truth questions toward live user prompts and comparative judgments. The Arena paper frames this as a response to the fact that many current benchmarks miss open-ended, real-world preference behavior.7 MT-Bench and LLM-as-a-judge methods try to scale open-ended evaluation, while documenting biases such as position and verbosity effects.6

This is still psychometrics in spirit. The item is now a prompt, the response is a preference comparison, and the measurement model may look more like Bradley-Terry or Elo than a multiple-choice IRT model. But the question remains: which prompts discriminate among which models, with what noise, and for which population of users?

The answer cannot be compressed into one global rank without losing something.

Item Bank Simulator

The simulator below creates four synthetic language models and a benchmark item bank. Each item has a difficulty and discrimination. One non-frontier model can receive a contamination boost on leaked items. The chart compares raw leaderboard accuracy against an IRT-style latent ability estimate computed on clean anchor items.

The point is not that IRT magically solves evaluation. It does not know what “reasoning” means. The point is narrower and useful: once items differ in difficulty, discrimination, and contamination, equal-weight accuracy can answer the wrong measurement question.

Raw accuracy Estimated ability True ability Item information

Deterministic simulation. "Exposed" is not the true frontier model, but it receives a boost on leaked items. Ability is estimated from clean anchor items using the generated item parameters.

Try moving the difficulty center far left. The benchmark becomes too easy: accuracy compresses near the ceiling, and item information moves below the frontier models. Move it far right. Now many items are too hard, and the test starts measuring guessing and noise. Increase leaked items. The exposed model can climb the raw leaderboard even though its clean latent ability did not improve.

The most useful benchmark is not the one with the highest average difficulty. It is the one with enough reliable, discriminating items around the ability range where decisions are being made.

Benchmark Report Card

A serious model evaluation should report more than a headline score:

  1. Item difficulty and discrimination distributions.
  2. Reliability or uncertainty intervals for model differences.
  3. Score sensitivity to item subsets and task mixture.
  4. Saturation: the fraction of items near ceiling or chance.
  5. Contamination and exposure audit results.
  6. Preference-population metadata for open-ended evaluations.
  7. A statement of the benchmark’s intended measurement range.

The last point is easy to skip and hard to fake. A benchmark cannot measure everything. It should say what kind of evidence it is designed to create.

Adaptive Evaluation, Not Forever Exams

The next generation of AI evaluation should look less like a static exam and more like an adaptive measurement system.

Psychometrics already knows the pattern: calibrate items, estimate ability, choose the next informative item, update uncertainty, and stop when the decision is reliable enough. Lalor, Wu, and Yu show how item response theory can be learned from machine response patterns in NLP tasks.9 Recent work is pushing IRT-style ideas directly into LLM benchmark analysis and efficient model evaluation.10, 11

That suggests a better leaderboard:

  • not one score, but a capability profile over calibrated domains;
  • not all items, but informative anchor items plus fresh adaptive probes;
  • not a permanent public test set, but rotating public examples and protected measurement items;
  • not rank without uncertainty, but pairwise decisions with confidence;
  • not “the model is best,” but “the model is best for this construct, in this range, under this cost and exposure model.”

Benchmarks are how a field looks at itself. If the instrument is blurry, the field can mistake saturation for progress, leakage for intelligence, and leaderboard motion for measurement.

Further Reading

  1. Georg Rasch, Probabilistic Models for Some Intelligence and Attainment Tests, 1960. ERIC record. ↩

  2. Frederic M. Lord, Applications of Item Response Theory to Practical Testing Problems, 1980. ETS record. ↩

  3. Dan Hendrycks et al., “Measuring Massive Multitask Language Understanding,” ICLR 2021. arXiv. ↩ ↩2

  4. Aarohi Srivastava et al., “Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models,” TMLR 2022. arXiv. ↩ ↩2

  5. Percy Liang et al., “Holistic Evaluation of Language Models,” 2022. arXiv. ↩ ↩2

  6. Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023. arXiv. ↩ ↩2

  7. Wei-Lin Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,” ICML 2024. arXiv. ↩ ↩2

  8. Chunyuan Deng et al., “Investigating Data Contamination in Modern Benchmarks for Large Language Models,” NAACL 2024. ACL Anthology. ↩

  9. John P. Lalor, Hao Wu, and Hong Yu, “Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds,” EMNLP-IJCNLP 2019. ACL Anthology. ↩

  10. Hongli Zhou et al., “Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory,” 2025. arXiv. ↩

  11. Sang Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo, “Reliable and Efficient Amortized Model-Based Evaluation,” ICLR 2025. arXiv. ↩