The aggregate table has a dangerous authority.

It is bigger. It uses all the data. It feels less fussy than subgroup analysis. When a debate has too many slices, the aggregate promises relief:

just tell me what happened overall

Sometimes that is exactly the wrong request.

Simpson’s paradox is the case where an association points one way inside every stratum and the other way after the strata are pooled. Simpson’s 1951 paper gave the contingency-table version, and Blyth later gave the reversal its familiar name in a discussion of the sure-thing principle.12 A treatment helps both high-risk and low-risk patients, but the pooled treatment group does worse. An app feature improves retention in every country, but global retention falls. A school admits women at higher rates within departments, but the university-wide admission rate is lower.

The aggregate did not make an arithmetic error. It changed the mixture.

The Reversal in Two Buckets

Suppose there are two strata: easy cases and hard cases. Treatment improves the success probability in both:

\[P(Y=1\mid D=1,S=s) > P(Y=1\mid D=0,S=s)\]

for \(s \in \{\text{easy},\text{hard}\}\).

The marginal comparison is different:

\[P(Y=1\mid D=1) - P(Y=1\mid D=0).\]

That comparison averages over the stratum distribution inside each treatment group. If the treated group contains many more hard cases than the control group, the aggregate can reverse sign even when the conditional effect is positive in every stratum.

This is not a contradiction. It is two estimands:

within-stratum effect

versus

effect plus change in case mix

Only one of them may answer the causal question.

A Mixture You Can Flip

The lab below uses a binary outcome called success. There are easy and hard strata. Treatment has the same conditional odds ratio in both strata. The mix slider changes which stratum is overrepresented in the treatment group.

At the default setting, treatment improves success in both strata. But treatment also gets mostly hard cases while control gets mostly easy cases, so the marginal success gap reverses.

Control Treatment Marginal effect Balanced OR Conditional OR

Deterministic two-stratum binary-outcome table. Treatment has the same conditional odds ratio in both strata. The mix imbalance slider changes the hard-case share in treatment versus control. The balanced-OR panel sets the treatment and control stratum mix equal, showing odds-ratio non-collapsibility without confounding by stratum mix.

Move Mix imbalance to zero. The reversal disappears. Treatment and control now have the same easy/hard composition, so the marginal risk difference is a reasonable standardized summary of the within-stratum improvements.

But notice the third panel. Even with balanced mix, the marginal odds ratio is smaller than the conditional odds ratio when the strata have different baseline risks. That is not Simpson’s paradox. It is non-collapsibility of the odds ratio.

The distinction matters:

  • Simpson reversal is about changing mixture weights.
  • Odds-ratio non-collapsibility can occur without confounding.

Both are ways an aggregate can surprise you. They are not the same surprise.

Berkeley, the Famous Warning

The most famous real example is the 1973 University of California, Berkeley graduate admissions data. The aggregate table suggested women had lower admission rates. Bickel, Hammel, and O’Connell showed that the pattern largely changed after department was considered: women applied more often to departments with lower admission rates, while within many departments the apparent bias was smaller or reversed.3

The lesson is often told as “always stratify.” That is too blunt.

Department mattered because it was connected to both applicant sex and admission probability. It changed the case mix. But conditioning on every available variable is not a law. Some variables are confounders, some are mediators, some are colliders, and some are irrelevant bookkeeping. The right adjustment set is a causal question, not a dashboard preference.

Pearl’s treatment of Simpson’s paradox makes this point sharply: the resolution depends on whether the stratifying variable is a cause of treatment, a cause of outcome, an effect of treatment, or something else in the causal graph.4 Probability tables tell you where the reversal is. They do not by themselves tell you which table answers the intervention question.

Odds Ratios Have Their Own Trick

The word collapsibility asks whether a measure of association keeps the same value after averaging over a covariate distribution, assuming the relevant conditional measure is constant across strata.

Risk differences are collapsible under common conditions. If the treatment raises success by 10 percentage points in every stratum, then a standardized average over a common stratum distribution also raises success by 10 points.

Risk ratios are also collapsible in a similar standardized sense when the conditional risk ratio is constant.

Odds ratios are different. Even when the treatment-control stratum mix is the same, and even when there is no confounding by that stratum, the marginal odds ratio can be closer to 1 than the common conditional odds ratio. Greenland, Robins, and Pearl’s review of confounding and collapsibility is useful precisely because it separates these ideas: confounding is not the same as non-collapsibility.5

Why does the odds ratio behave this way?

Odds are nonlinear:

\[\operatorname{odds}(p)=\frac{p}{1-p}.\]

If you average probabilities first and then take odds, you do not get the same answer as taking odds ratios inside strata and then averaging. The denominator \(1-p\) changes shape as baseline risk changes.

In the lab, set Mix imbalance to zero and raise Stratum gap. The treatment and control groups have the same stratum mix, so this is not a Simpson reversal. The balanced odds ratio still drifts away from the conditional odds ratio. The aggregate odds ratio is answering a marginal association question on a nonlinear scale.

Randomization Does Not Freeze the Mix

Randomization protects the treatment assignment, not every subgroup comparison you might later make.

In a large randomized A/B test, treatment and control should have the same expected distribution of risk strata. That makes the marginal risk difference a legitimate estimand for the randomized population. But if product teams compare countries, acquisition channels, device classes, or tenure buckets after traffic allocation changes, the aggregate can move because the user mix moved.

There are two common mistakes.

The first is treating a stratified reversal as proof the aggregate is fake. The aggregate may be the policy estimand if the policy truly changes the mix in that way. If opening a clinic changes which patients arrive, the marginal outcome includes that composition change.

The second is treating the aggregate as the final referee when the decision is about a conditional intervention. If a treatment will be assigned within departments, hospitals, countries, or customer segments, the relevant estimand may be standardized within those strata.

The right question is not:

aggregate or stratified?

It is:

which population distribution should define the intervention?

Questions Before You Trust the Average

Before trusting an aggregate binary-outcome comparison, I want to know:

  • What is the target estimand: marginal, conditional, or standardized?
  • Which variables change the baseline risk?
  • Which variables change treatment assignment or exposure?
  • Which variables are downstream of treatment?
  • Are the treatment and control groups using the same stratum distribution?
  • Are the reported effects risk differences, risk ratios, odds ratios, or model coefficients?
  • Would the qualitative claim survive standardization to a common population?
  • If an odds ratio is reported, is the non-collapsibility being interpreted as confounding?
  • Does the decision act on individuals within strata or on a policy that changes the mixture itself?

The last item is the important one. Statistics does not decide whether the aggregate or the strata are “right” in the abstract. The decision does.

The Sentence to Keep

Simpson’s paradox is not a magic trick. It is a warning label on mixtures.

An aggregate comparison combines within-stratum effects with differences in stratum composition. If the composition is part of the intervention, keep it. If the composition is a nuisance path, standardize it. If the effect measure is an odds ratio, remember that even balanced aggregation can move the number.

The aggregate is useful. It is just not a referee.

before asking what the average says, ask what population it averages over

That is where the paradox usually goes to rest.

Reading Trail

  1. Edward H. Simpson, “The Interpretation of Interaction in Contingency Tables”, Journal of the Royal Statistical Society, Series B, 1951. DOI page: Wiley. ↩

  2. Colin R. Blyth, “On Simpson’s Paradox and the Sure-Thing Principle”, Journal of the American Statistical Association, 1972. ↩

  3. Peter J. Bickel, Eugene A. Hammel, and J. W. O’Connell, “Sex Bias in Graduate Admissions: Data from Berkeley”, Science, 1975. DOI: 10.1126/science.187.4175.398. ↩

  4. Judea Pearl, “Understanding Simpson’s Paradox”, The American Statistician, 2014. DOI: 10.1080/00031305.2014.876829. ↩

  5. Sander Greenland, James M. Robins, and Judea Pearl, “Confounding and Collapsibility in Causal Inference”, Statistical Science, 1999. PDF mirror: UBC. ↩