A table of discoveries is not a table of facts.

It is a filtered list produced by a machine: collect many hypotheses, compute many test statistics, sort the evidence, choose a boundary, and publish the rows that cross it. The quiet failure mode is pretending that the rows became independent again after sorting.

They did not. The denominator moved.

Thesis: false discovery rate control is not a nicer p-value. It is a promise about a whole discovery list. The unit of reasoning is the family of tests, the filtering rule, and the expected dirt in the rows that survive.

This is why a single clean p-value can still live inside a bad scientific process. If a model team tries 2,000 features, 300 prompts, 90 treatment slices, or a garden of trading signals, the final row inherits the search. The question is no longer “is this one p-value below 0.05?” It is “how much false signal are we willing to carry in the list we are about to act on?”

The List Has Accounting Columns

Suppose we test \(m\) null hypotheses. After the procedure runs:

  • \(R\) hypotheses are rejected and become “discoveries”;
  • \(V\) of those discoveries are actually null;
  • \(S = R - V\) are genuine discoveries.

The realized false discovery proportion is

\[\operatorname{FDP} = \begin{cases} V/R, & R > 0, \\ 0, & R = 0. \end{cases}\]

The false discovery rate is the expectation of that random fraction:

\[\operatorname{FDR} = \mathbb{E}[\operatorname{FDP}].\]

That expectation matters. FDR is not a guarantee that every published table is less than 10% wrong. Some tables will be cleaner; some will be dirtier. The control statement lives over repeated full tables generated by the same procedure.

This is weaker than familywise error rate control, which tries to limit the probability of even one false rejection. But it is often the right weakness. In genomics, recommender experiments, model diagnostics, feature search, security alerts, or factor research, the goal is rarely “make at most one claim.” The goal is to build a candidate list whose expected contamination is acceptable for the next stage.

Benjamini and Hochberg introduced FDR as that middle ground: less conservative than guarding against any false positive, but explicit about the cost of turning many tests into a discovery table.1

The Step-Up Line

Sort the p-values:

\[p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)}.\]

For target FDR \(q\), the Benjamini-Hochberg step-up rule finds the largest rank \(k\) such that

\[p_{(k)} \le \frac{k}{m}q.\]

Then it rejects everything up through \(k\). The boundary is generous near the bottom of the list because a larger discovery set can absorb some false rows. It is severe at the first row because when \(R=1\), a single false row makes the whole list false.

This geometry is the thing people forget. BH is not “adjust every p-value by a constant.” It is a sorted-list rule. A p-value can be too large to be the first discovery and still small enough to be the fortieth discovery, because the denominator has changed.

There is also a useful edge case. If every null hypothesis is true, then any rejection is false. In that all-null world, FDR becomes the probability of at least one false rejection, so it coincides with weak familywise error control. When real signals exist, FDR can spend some error budget to find more of them.

Dependence Is Not a Footnote

The original BH proof covers independent test statistics. Many real discovery tables are not independent. Adjacent genes co-express, experiments share users, features share upstream transformations, prompts share benchmarks, and trading signals share market regimes.

Benjamini and Yekutieli later studied FDR control under dependence. The headline is subtle: BH is still justified under certain positive dependence conditions, but arbitrary dependence needs a more conservative line.2 One common version replaces \(q\) with

\[\frac{q}{H_m}, \qquad H_m = \sum_{i=1}^m \frac{1}{i}.\]

That harmonic term is not small. At \(m=1200\), \(H_m \approx 7.67\). In other words, a dependence-robust guarantee can be dramatically less powerful than the ordinary BH line.

This is not an argument to always use the harmonic penalty. It is an argument to stop saying “we used FDR correction” as if dependence structure were irrelevant. The correction is attached to a stochastic story.

Estimating the Null Shelf

Storey’s view turns the list around. Instead of fixing the FDR level and asking where the rejection boundary lands, estimate how much null mass is in the p-value distribution.3 Under the null, p-values are uniform. If the right side of the histogram still looks like a flat shelf, it can estimate \(\pi_0\), the fraction of true nulls:

\[\widehat{\pi}_0(\lambda) = \frac{\#\{p_i > \lambda\}}{(1-\lambda)m}.\]

If many hypotheses are truly non-null, \(\widehat{\pi}_0\) can be below one, and the procedure can reject more rows at the same estimated error rate. This is the intuition behind q-values: each row can be associated with the minimum FDR level at which it would enter the discovery list.4

This extra power is not free. The null shelf has to be real. A broken p-value model, unmodeled dependence, optional reporting, or a post hoc family definition can make the shelf lie.

A Discovery Table You Can Shake

The lab below builds repeated discovery tables. Each run creates \(m\) one-sided z-tests. A chosen fraction of tests are real signals; the rest are nulls. Tests are grouped into correlated clusters, so the p-values are marginally valid but not independent.

The procedures compared are:

  • raw thresholding at \(p < q\);
  • Bonferroni at \(p < q/m\);
  • Benjamini-Hochberg;
  • Benjamini-Yekutieli with the harmonic dependence penalty;
  • Storey-style adaptive BH using \(\widehat{\pi}_0\) from the upper half of the p-value histogram.
Raw p<q Bonferroni Benjamini-Hochberg Benjamini-Yekutieli Storey-style adaptive BH

Deterministic simulation. Each operating-frontier point averages 180 repeated discovery tables. Circle area scales with the average number of rejected hypotheses.

At the default setting, the simulator tests 1,200 hypotheses with about 8% real signals, effect size 2.4, 30% within-cluster dependence, and target \(q=10\%\). Across 180 repeated discovery tables, raw thresholding finds about 195 rows but has empirical FDR around 56.5%. Bonferroni finds about 9 rows with very low FDR, but misses most signals. BH finds about 42 rows with empirical FDR around 9.1% and power around 38.8%. BY finds about 11 rows because the harmonic penalty is expensive. Storey-style adaptive BH finds about 45 rows in this setting and runs near the target.

Before publishing, I ran the lab through its Node API. The default case passes finite-output, sorted-p-value, and nested-rejection checks. A 54-case sweep over test count, signal sparsity, dependence, and target \(q\) passed the same invariants; in the all-null cases, the largest empirical BH FDR in the sweep was 9.86%.

The Promise Is Conditional

The useful sentence is not “we corrected the p-values.” It is:

For this predeclared family of tests, with this p-value model, this dependence assumption, and this selection rule, the expected false fraction among published discoveries is controlled at this level.

Each clause can fail.

If the family is defined after seeing the data, the denominator is fiction. If the p-values are anti-conservative, the null shelf tilts downward and every FDR procedure inherits the bad calibration. If dependence is strong and not covered by the method, the guarantee may be optimistic. If the team silently filters discoveries again after correction, the published list is no longer the list the procedure controlled.

This is why FDR belongs in the design document, not just the appendix.

How I Would Log a Discovery Run

For an experiment platform, model evaluation suite, or research pipeline, the ledger should include:

  1. The exact hypothesis family before results are inspected.
  2. The p-value construction and its calibration checks.
  3. The dependence story: independent, positively dependent, blocked, clustered, resampled, or unknown.
  4. The chosen procedure and target \(q\).
  5. The number of discoveries, estimated FDP, and power proxy if available.
  6. The rows that failed to survive correction.
  7. Any human filtering applied after the statistical rule.

The failed rows matter. They show the denominator. A discovery without its denominator is not a discovery; it is a selected anecdote with mathematical lighting.

  1. Yoav Benjamini and Yosef Hochberg, “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society: Series B, 1995. DOI. A public PDF is mirrored by Purdue’s statistics course materials: PDF. ↩

  2. Yoav Benjamini and Daniel Yekutieli, “The Control of the False Discovery Rate in Multiple Testing Under Dependency,” The Annals of Statistics, 2001. Project Euclid. ↩

  3. John D. Storey, “A Direct Approach to False Discovery Rates,” Journal of the Royal Statistical Society: Series B, 2002. PDF. ↩

  4. John D. Storey and Robert Tibshirani, “Statistical Significance for Genomewide Studies,” Proceedings of the National Academy of Sciences, 2003. DOI, PDF. ↩