The ordinary A/B test asks two rankers to fight through fog.

Half the users see ranker A. Half see ranker B. We compare clicks, purchases, dwell time, reformulations, saves, or some product metric downstream. This is the right experiment when the question is:

Which whole experience should we ship?

But it is an expensive way to answer a narrower question:

Which ranker places the better documents above the worse documents?

For that question, the query is a nuisance variable. Some queries are easy. Some are hopeless. Some users click everything. Some click nothing. Some days are strange. A split-user A/B test has to average over all of that before the ranking difference becomes visible.

Interleaving changes the unit of comparison.

Instead of showing one user ranker A and another user ranker B, it builds one shared result list from both rankers. The user clicks on that shared page. The clicked documents are then credited back to the ranker that contributed them.

The rankers duel on the same query, in front of the same user, under the same position bias.

That paired design is the whole trick: make the nuisance variables sit in the same room as both contestants.

Clicks Are Witnesses With Bad Lighting

Search has always been tempted by clicks. They are abundant, cheap, timely, and produced by real users with real information needs. Joachims’s 2002 KDD paper made clickthrough data a central object for learning retrieval functions from preferences rather than from expensive expert judgments.1

The warning arrived almost immediately: clicks are informative, but not clean.

Joachims, Granka, Pan, Hembrooke, and Gay combined eye-tracking with manual relevance judgments and concluded that click feedback is biased. Highly ranked results get extra trust. A click depends on the surrounding page, not only on the clicked document. Their useful conclusion was not “ignore clicks.” It was that relative preferences derived from clicks can be much more reliable than absolute click-as-relevance judgments.2

Position bias is one reason. Craswell, Zoeter, Taylor, and Ramsey perturbed a major search engine’s rankings and compared models of click position bias. Their best explanation for early ranks was a cascade-like user model: users move from top to bottom and may stop after finding a worthwhile result.3

This makes naive click counts dangerous. A document can get more clicks because it is better, because it is higher, because its snippet is more attractive, or because everything above it was bad enough to keep the user scanning.

Interleaving does not remove these biases from the universe. It tries to make the two rankers experience them on the same page.

Pair the Query, Not the Traffic

In a split-user A/B test, a session contributes something like:

Y_i(A)  or  Y_i(B)

where Y might be clicks per search. The estimate is a difference of two groups:

mean(Y | B) - mean(Y | A)

Its variance includes query mix, user mix, session context, and ranker quality. Randomization balances those terms in expectation, but the experiment may need a lot of traffic before a small ranker improvement rises above them.

In an interleaving test, the session contributes a paired score:

S_i = clicks credited to B - clicks credited to A

The same query and user produced both sides of the comparison. Query difficulty has less room to masquerade as ranker quality.

Team Draft Interleaving is the clean folk algorithm. Two rankers are captains drafting a team. A coin decides who picks first, then rankers alternate taking their highest-ranked document that has not already appeared. The result page keeps a hidden owner label for each document. When the user clicks, the owner gets credit.4

Radlinski, Kurup, and Joachims introduced Team Draft Interleaving to avoid problems with earlier balanced interleaving when rankings are nearly identical. In their arXiv user study, interleaving methods reflected known retrieval quality differences more reliably than a set of absolute click metrics at the sample sizes they considered.4

Chapelle, Joachims, Radlinski, and Yue later analyzed interleaved evaluation using data from two large commercial search engines and a scientific-literature retrieval system, studying agreement with other quality signals and statistical efficiency.5 The broader survey by Hofmann, Li, and Radlinski places interleaving inside the family of online information-retrieval experiments: online evaluation measures real users in situ, complements offline judgments, and supports both absolute and relative quality assessments.6

Recent large-system work is still finding the same pattern. Bi and coauthors describe interleaving as far more sensitive than ordinary A/B testing in industrial retrieval systems, while also emphasizing practical issues: comparing many systems, interpreting raw interleaving magnitudes, and operationalizing the method at scale.7

That last clause matters. A fast test is not automatically an easy test.

Same-Page Duel

The lab below simulates a search product with a hidden relevance landscape. Each query has a set of candidate documents. Ranker A sees relevance through a noisy scoring function. Ranker B is a controlled improvement: it uses slightly less noisy evidence and therefore lifts offline NDCG when Ranker B lift is positive.

Users follow a simple position-biased click model. Higher positions are examined more often; attractive and relevant documents are clicked more often; after a satisfying click, later positions receive less attention.

The same finite user sessions are evaluated two ways:

  • A/B: split users between A and B and compare clicks per search.
  • Interleaving: show a Team Draft page and compare click credit inside that page.
Ranker A Ranker B A/B evidence A wins / loss Tie

Deterministic synthetic experiment. Clicks are generated from a simple position-biased examination model, not from real users. The useful object is the comparison between estimators under the same simulated traffic.

With the default settings, ranker B improves offline top-10 NDCG by about five points. The ordinary A/B click delta is positive but just under the usual two-sigma line in this finite sample. The interleaved duel sees the same traffic and produces a much stronger preference for B.

Set Ranker B lift to none. Both tests become neutral. That is important: the interleaving procedure is not simply biased toward the challenger in this toy. It gets power by pairing, not by magic.

Now make Query mix very skewed. A/B has to fight the fact that the traffic stream is dominated by a few query types. Interleaving still sees the same query on both sides of the comparison. Increase Position bias. The absolute click metric becomes more fragile, because top positions dominate. Team Draft does not abolish position bias, but it alternates ownership on the same page, so the bias is shared more symmetrically.

This is why interleaving can feel unfairly effective. It is not measuring everything. It is measuring a narrower thing with less nuisance variance.

What the Duel Cannot Ship

Interleaving answers:

When these two rankers share a page, whose contributed documents attract more
click credit?

That is not the same as:

What is the product impact of shipping ranker B?

Shipping can change latency, result diversity, freshness, ad load, trust, long-term satisfaction, downstream conversion, abandonment, and user learning. Interleaving is a sensitive ranker-preference test, not a complete business experiment.

It also needs careful engineering:

  • Both rankers should draw from comparable candidate sets.
  • Duplicate handling must be deterministic and audited.
  • Snippets, badges, prices, and media treatments should not reveal ownership.
  • Credit assignment should match the interaction surface.
  • Guardrail metrics still need ordinary A/B tests.
  • The result should be interpreted as a direction-of-preference signal, not as a direct revenue or retention estimate.

Generalized and optimized interleaving work makes this explicit. Kharitonov, Macdonald, Serdyukov, and Ounis frame sensitivity as a central property of an interleaving method and study ways to optimize both the interleaving policy and credit assignment.8 Industrial work makes the same point from the other direction: interleaving is powerful enough to be worth operational complexity, but the raw measurement needs translation before it becomes a launch decision.7

Three-Layer Ranking Evidence

The evaluation stack I want for ranking systems has three layers:

  1. Offline judgments and counterfactual replay to catch obvious regressions before touching users.
  2. Interleaved online duels to detect ranker preference quickly under real traffic.
  3. Full A/B tests for product-level metrics, guardrails, and long-term effects.

Those layers should not compete for philosophical authority. They answer different questions.

Offline evaluation asks whether the ranking agrees with a relevance model. Interleaving asks whether users prefer one ranker’s contributions under a shared page. A/B testing asks whether the shipped experience changes the system.

The failure mode is pretending that one layer answers all three.

Search, recommendations, marketplaces, feeds, and retrieval-augmented generation all now contain rankers inside larger products. A reranker can improve pairwise relevance while harming diversity. A diversity policy can reduce clicks while improving satisfaction. A citation ranker can win clicks and lose factuality. We need evaluators that say exactly what they measured.

Let the rankers share a page when the question is ranking preference.

Then make them face the whole product when the question is shipping.

Reading Trail

  1. Thorsten Joachims, “Optimizing Search Engines using Clickthrough Data,” KDD, 2002. PDF, ACM. ↩

  2. Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay, “Accurately Interpreting Clickthrough Data as Implicit Feedback,” SIGIR, 2005. PDF, ACM. ↩

  3. Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey, “An Experimental Comparison of Click Position-Bias Models,” WSDM, 2008. Microsoft Research, ACM. ↩

  4. Filip Radlinski, Madhu Kurup, and Thorsten Joachims, “How Does Clickthrough Data Reflect Retrieval Quality?” CIKM, 2008. PDF. ↩ ↩2

  5. Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue, “Large-scale Validation and Analysis of Interleaved Search Evaluation,” ACM Transactions on Information Systems, 2012. Microsoft Research, ACM PDF. ↩

  6. Katja Hofmann, Lihong Li, and Filip Radlinski, “Online Evaluation for Information Retrieval,” Foundations and Trends in Information Retrieval, 2016. PDF. ↩

  7. Nan Bi, Bai Li, Ruoyuan Gao, Graham Edge, and Sachin Ahuja, “Interleaved Online Testing in Large-Scale Systems,” WWW Companion, 2023. PDF, ACM. ↩ ↩2

  8. Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis, “Generalized Team Draft Interleaving,” CIKM, 2015. PDF, ACM. ↩