The leaderboard number likes to pose as a fact.

1847.

It has the aesthetic of measurement: precise, sortable, public. It fits in a badge. It can be used to seed tournaments, match opponents, gate ranked queues, grant titles, and start arguments.

But a rating is not a prize.

It is an estimate of a hidden variable.

The hidden variable is skill. We do not observe it directly. We observe noisy outcomes of games between players whose skill may be changing. A rating system is an online inference machine that turns those outcomes into a belief about skill.

Once you say it that way, the missing number becomes obvious.

rating = estimate + uncertainty

Elo gives the estimate. Glicko and TrueSkill made the uncertainty harder to ignore.12

A Game Is Noisy Evidence

Most rating systems start with a paired-comparison model. Player i plays player j. The system sees a result:

win, loss, draw, rank, score, or team outcome

The simplest binary version says:

\[\Pr(i \text{ beats } j) = \sigma(\theta_i - \theta_j),\]

where \(\theta_i\) and \(\theta_j\) are latent skills and \(\sigma\) is a monotone link function. Bradley and Terry’s 1952 model is the classic logistic form.3 Thurstone-style models use Gaussian performance noise. The distinction matters, but the shared idea is more important: outcomes are probabilistic evidence about relative strength.

Elo’s chess rating system is an online version of this grammar.4 In a common logistic Elo form:

\[E_i = \frac{1}{1 + 10^{-(R_i - R_j)/400}},\]

and after the game:

\[R_i \leftarrow R_i + K(S_i - E_i).\]

Here \(S_i\) is the observed score, usually 1 for win, 0.5 for draw, and 0 for loss. The term \(S_i - E_i\) is the surprise. Beat a stronger player and the surprise is positive. Lose to a weaker player and it is negative. The parameter \(K\) decides how much the new evidence moves the estimate.

This is beautifully compact. It is also hiding several questions inside one knob:

  1. How uncertain is the player’s current rating?
  2. How uncertain is the opponent’s rating?
  3. How noisy is this game mode?
  4. How quickly can skill change?
  5. Is this player new, inactive, returning, sandbagging, improving, or stable?

A fixed \(K\) has to answer all of them with one number.

That is too much responsibility for one letter.

The Error Bar Is Part of the Number

Glickman’s Glicko system makes the reliability problem explicit. Its technical description starts from the weakness of Elo when two players have the same rating but very different evidence behind that rating.1

Imagine two 1700-rated players:

  1. one has played 400 recent games against known opponents;
  2. one has played 3 games after returning from a long break.

The displayed rating is the same. The belief should not be.

Glicko attaches a rating deviation to the rating. TrueSkill uses a Gaussian belief over player skill, often summarized as a mean \(\mu\) and standard deviation \(\sigma\).2 The public number can even be conservative:

\[\mathrm{exposure} = \mu - c\sigma.\]

That conservative value says:

do not rank someone as elite until the uncertainty has been paid down

This matters for online games. New accounts, returning players, smurfs, specialists, team modes, asymmetric roles, balance patches, and seasonal resets all create uncertainty. A visible number without uncertainty encourages the wrong social interpretation:

1847 means this person is exactly 1847-good

No. It means the current evidence has been compressed into a point estimate. Sometimes that compression is harmless. Sometimes it is the bug.

Let a Strong Newcomer In

The lab below simulates a small ladder. Each player has a hidden true skill. Match outcomes are noisy paired comparisons. True skill drifts slowly over time. A strong new player enters partway through the season with no rating history.

Two online systems see the same matches:

  1. Fixed-K Elo: one rating per player, one update size.
  2. Uncertainty-aware filter: a toy Gaussian-style mean and uncertainty. New or uncertain players move faster; stable players move slower; uncertainty shrinks with informative games and grows with drift.

The second system is not exact Glicko or exact TrueSkill. It is a deliberately small executable model of their central lesson: a rating should carry reliability.

True skill Elo Uncertainty mean Conservative rating Late entrant

Deterministic toy model. The uncertainty-aware filter is inspired by Glicko/TrueSkill ideas, but it is not a faithful implementation of either production system.

Start with the default. Final RMSE is about 62 for fixed-K Elo and about 54 for the uncertainty-aware mean. The late strong player gets 26 games, moves from an initial mean of 1500 to about 1796, and still carries uncertainty of about 96 rating points. The conservative public rating is therefore only about 1604, even though the mean has moved much faster than Elo’s 1728.

I swept 729 knob combinations. The uncertainty-aware toy had lower final RMSE than fixed-K Elo in 481 of them, but not all of them. With aggressive update settings and little actual drift, it can overreact. That is the honest lesson: uncertainty is not fairy dust. It is another model assumption that needs calibration.

That is the right tension:

matchmaking wants the mean
leaderboards want caution

Raise late entrant strength. Elo adapts, but slowly, because the same \(K\) is used for a new account and a well-measured veteran. The uncertainty-aware filter gives the new account larger updates, then shrinks uncertainty as games arrive.

Raise skill drift. Stable historical evidence becomes less trustworthy. A rating system that never re-opens uncertainty becomes overconfident about old information. This is one reason seasons, decay, inactivity inflation, and volatility parameters exist.

Raise outcome noise. A game with high randomness should move ratings less per match than a game with low randomness. If outcomes are noisy, a single upset is weak evidence. If outcomes are sharp, the same upset is stronger evidence that the rating was wrong.

Raise matchmaking tightness. Close matches are fun and informative near the decision boundary, but they also make it harder to sort the very top if strong players rarely meet enough diverse opponents. A ladder is not just a model. It is also an experimental design.

The Matchmaker and the Trophy Case

A common mistake is to use one public number for every job.

For matchmaking, the system wants a predictive distribution:

\[\Pr(i \text{ beats } j \mid \mathcal{D}).\]

The mean matters because it predicts game balance. The uncertainty matters because a wide distribution says the next game is also a measurement.

For a public leaderboard, the product question changes. The cost of over-ranking an uncertain player can be high. A conservative exposure like \(\mu - 2\sigma\) or \(\mu - 3\sigma\) says: put the player where the evidence can defend them.

This is not cowardice. It is epistemic accounting.

The same player can have:

  1. a matchmaking mean,
  2. a conservative public rank,
  3. a provisional badge,
  4. a role-specific rating,
  5. a team-mode posterior,
  6. a decay/inactivity state.

Compressing all of that into one integer is sometimes good interface design. It is rarely good statistical thinking.

The Ladder Chooses Its Data

Ratings are not learned from neutral data. The ladder chooses who plays whom.

If the matchmaker mostly pairs equal visible ratings, the system gets many close games, which are good for player experience. But it may get fewer direct tests between separated regions of the ladder. If parties, regions, queues, roles, time zones, smurf detection, or avoidance systems constrain pairings, the comparison graph becomes uneven.

Bradley-Terry is a model for paired comparisons. The ladder decides which pairs exist.

That is why rating-system engineering is not only about the update formula. It is about data collection:

  1. How connected is the comparison graph?
  2. Are new players tested broadly or only against other uncertain players?
  3. Do high-rated players mostly play each other?
  4. Does the system observe teams, roles, maps, patches, and parties?
  5. How fast does evidence expire after balance changes?

A rating can be mathematically elegant and operationally blind.

Online Games Move the Target

Chess is already difficult: draws, color advantage, inactivity, junior improvement, rating inflation, changing pools. Online games add more trouble.

Players can queue in teams. A support player and a carry player may contribute in different ways. Maps and patches change the game. Matchmaking itself changes the data distribution. A player can have different strengths by character, role, region, input device, mode, or time of day. Anti-cheat and smurf detection become part of measurement.

TrueSkill was built for this world: online multiplayer games with teams, multiple competitors, draws, and the need to infer individual skill from team results.2 Herbrich’s summary of the system emphasizes that the goal is not merely ranking, but identifying and tracking skills so players can be matched into competitive games.5

The key word is tracking.

Skill is not a fixed label. It is a state.

That makes rating closer to filtering than accounting. The system observes a noisy stream, updates a belief, and decides what action to take next.

Questions for a Ladder Audit

If I were auditing a ranked ladder, I would not start with the prettiness of the leaderboard. I would ask for calibration and uncertainty evidence.

  1. For players rated near 1600, do they win about the predicted amount against 1500, 1600, and 1700 opponents?
  2. Do 90% intervals contain the future estimated skill about 90% of the time?
  3. How many games does a strong new player need before matchmaking stops harming opponents?
  4. How often does the conservative leaderboard later reverse itself?
  5. How does rating error vary by games played, role, party size, region, queue, and patch?
  6. How much of the update comes from skill evidence versus matchmaker selection?

The public argument is usually about whether someone “deserves” a number. The engineering question is better:

what decisions is this number safe enough to support?

Matchmaking, prestige, tournament seeding, anti-smurfing, rewards, and balance analytics may need different safety thresholds.

The Number Is a Promise

A rating system makes a promise to players:

we will put you in games that make sense

The leaderboard is only the visible surface of that promise. Underneath it is a model of uncertainty, a policy for exploration, a memory of drift, and a graph of who was allowed to compare against whom.

Elo’s great gift was not the exact constant 400 or the ritual of a K factor. It was the idea that pairwise outcomes could be turned into a portable measurement system. The modern lesson is that the measurement is incomplete without reliability.

The rating is not the player.

The rating is the current posterior summary of what the ladder has managed to learn.

  1. Mark E. Glickman, “The Glicko System,” technical description, 1995/1999. PDF: glicko.net/research/gdescrip.pdf. See also Glickman, “Rating the Chess Rating System,” Chance, 1999: math.bu.edu/people/mg/research/chance.pdf. ↩ ↩2

  2. Ralf Herbrich, Tom Minka, and Thore Graepel, “TrueSkill: A Bayesian Skill Rating System,” Advances in Neural Information Processing Systems 19, 2006/2007. PDF: papers.neurips.cc/paper/3079-trueskilltm-a-bayesian-skill-rating-system.pdf. ↩ ↩2 ↩3

  3. Ralph Allan Bradley and Milton E. Terry, “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons,” Biometrika 39, no. 3/4, 1952, 324-345. DOI: 10.1093/biomet/39.3-4.324. JSTOR: 2334029. ↩

  4. Arpad E. Elo, The Rating of Chessplayers, Past and Present, Arco, 1978. Google Books record: The Rating of Chessplayers, Past and Present. A scanned copy is mirrored at gwern.net. ↩

  5. Ralf Herbrich, “Computer Gaming: Ranking and Matchmaking,” summary of TrueSkill research and gaming applications: herbrich.me/computer-gaming. ↩