Averages Lose Innocence in 3-D

The average is innocent in one dimension.

Observe a noisy number. Estimate its mean by the number you saw. Under squared error, that rule is not only natural; it is admissible. Nobody can build another estimator whose risk is never larger and sometimes smaller.

In two dimensions, the same story survives.

In three dimensions, it breaks.

That is Stein’s paradox. If

X ~ Normal_p(theta, sigma^2 I)

and loss is total squared error,

loss(delta, theta) = ||delta(X) - theta||^2,

then the usual estimator

delta(X) = X

has constant risk p sigma^2. For p >= 3, James and Stein gave a simple estimator with lower risk for every value of theta.¹ Stein’s earlier Berkeley Symposium paper had already shown the shock: the familiar best-invariant estimator is inadmissible once the normal mean has dimension three or more.² Brown’s decision-theoretic treatment makes the border especially crisp: the one- and two-dimensional location cases are admissible; the three-and-up case is not.³

The theorem sounds like a trick because we are used to judging means one at a time. Stein changes the unit of judgment. He asks for simultaneous accuracy across the vector.

The coordinates may be baseball players, hospital effects, school effects, factor returns, benchmark task accuracies, or model deltas across many slices. If you score the whole vector, every coordinate gets to learn a little from the others.

The Shrinkage Rule

The zero-target James-Stein estimator is

delta_JS(X) =
  (1 - ((p - 2) sigma^2) / ||X||^2) X.

The multiplier is the shrinkage factor. Large observed norm, little shrinkage. Small observed norm, heavy shrinkage toward zero.

More generally, with an a priori center c,

delta_c(X) =
  c + (1 - ((p - 2) sigma^2) / ||X - c||^2) (X - c).

The phrase “a priori” is not decoration. If you pick c after looking at the same data, you changed the procedure and the dominance theorem no longer says what you want it to say. Shrinkage is not permission to aim at the answer after you have seen it.

There is also a practical variant:

delta_+(X) =
  max(0, 1 - ((p - 2) sigma^2) / ||X||^2) X.

This positive-part rule refuses to let the shrinkage factor go negative. The ordinary James-Stein rule can overshoot through the origin when ||X||^2 is tiny; the positive-part rule clamps that behavior. The lab below compares all three rules instead of asking you to take the geometry on faith.

The Coordinates Share a Risk Budget

The usual estimator has zero bias:

E[X] = theta.

James-Stein gives up that purity. It introduces bias toward the target. The payoff is variance reduction in the aggregate loss. In dimensions three and above, the variance you save beats the bias you introduce for every theta.

This is the part that should feel slightly unfair. If the first coordinate is a product metric and the second is a hospital effect, why should they help each other?

They help only because the risk function groups them. The theorem is about estimating the vector under total squared error, not about improving every coordinate on every realized data set. One coordinate can get worse. A single sample can get worse. The expectation over noise, summed across coordinates, gets better.

Efron and Morris’s empirical Bayes papers are still the cleanest bridge from the paradox to practice.⁴ If the coordinates are exchangeable effects drawn from some population, then shrinking noisy extremes toward a center is not a gimmick. It is the posterior mean instinct wearing a frequentist guarantee.⁵

That is why this old theorem still appears under modern names:

• partial pooling in hierarchical models;
• regularization in high-dimensional regression;
• small-area estimation;
• factor shrinkage in portfolio construction;
• leaderboard and benchmark de-noising;
• slice-level evaluation of ML systems.

Different models, same warning: the noisiest extreme coordinate is often extreme partly because it is noisy.

A Risk Lab for a Weird Theorem

The simulator uses a known-variance normal means problem. The true vector has norm

||theta|| = radius * sigma * sqrt(p).

That makes Signal radius roughly a per-coordinate signal-to-noise scale. The chart shows risk per coordinate across radii. The metrics report total risk, because the theorem is about the whole vector.

Try this sequence:

1. set Dimension to 2;
2. move Signal radius around and notice that the James-Stein rule is just the usual estimator because p - 2 = 0;
3. set Dimension to 3 or 10;
4. make Signal radius small and watch shrinkage save a lot of risk;
5. make Signal radius large and watch the shrinkage factor approach one.

Dimension 2 3 5 10 25 50 Signal radius 0.0 0.5 1.0 2.0 3.0 4.0 5.0 Noise sigma 0.5 1.0 1.5 Monte Carlo trials 1,500 4,000 8,000

Usual mean James-Stein Positive-part Observed noisy vector More loss on that trial

Deterministic Monte Carlo. The plotted risk is loss averaged over repeated noisy observations from the same true vector. The bottom histogram is trial-by-trial, which is why some red bars remain even when expected risk is lower.

At p = 10, radius = 1, the default run should show an obvious risk reduction. At radius = 5, shrinkage still helps in expectation, but the improvement is small because the observed vector usually points in nearly the right direction. At p = 2, the shrinkage coefficient vanishes and the paradox does not appear.

This is why I like the theorem as a mental calibration device. It is not saying “always shrink everything.” It is saying:

unbiased coordinatewise obviousness is not the same thing
as decision-theoretic optimality under aggregate loss

That sentence applies far beyond normal means.

Where Should the Pooling Stop?

The modern version of Stein’s question is not whether shrinkage exists. We know it does. The harder question is where the pooling boundary belongs.

Suppose a lab reports 80 task-level improvements for a model. Suppose a fund reports 200 strategy sleeves. Suppose a marketplace reports seller-quality effects across thousands of shops. Some effects should be pooled hard. Some should barely be pooled. Some should not share a prior at all.

A useful research system would keep an explicit shrinkage ledger:

• What is the estimand: coordinate loss, total loss, ranking loss, downstream utility?
• Which coordinates are exchangeable enough to borrow strength?
• Was the shrinkage target chosen before looking at the noisy estimates?
• How much risk was saved, and where did bias enter?
• Which outliers remain outliers after accounting for selection and noise?

Stein’s paradox is old, but this ledger is still rare. We have leaderboards, dashboards, experiment readouts, factor libraries, and model-eval slices full of noisy coordinates. We still often publish the raw X.

Sometimes the raw average is admissible.

Past dimension two, it has to defend itself.

Primary Sources

1. W. James and Charles Stein, “Estimation with Quadratic Loss,” Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, 1961. PDF. ↩

2. Charles Stein, “Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution,” Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1956. Project Euclid, PDF. ↩

3. Lawrence D. Brown, “Admissibility of Invariant Estimators of One or More Location Parameters,” The Annals of Mathematical Statistics, 1966. PDF. ↩

4. Bradley Efron and Carl Morris, “Stein’s Estimation Rule and its Competitors: An Empirical Bayes Approach,” Stanford Technical Report, 1972. Stanford report page. ↩

5. Bradley Efron and Carl Morris, “Data Analysis Using Stein’s Estimator and its Generalizations,” Journal of the American Statistical Association, 1975. PDF. ↩