A backtest is not one experiment if it came out of a research process.

It is the surviving artifact of many experiments: universes, lags, filters, rebalance rules, volatility targets, transaction-cost assumptions, start dates, outlier rules, factor definitions, neutralization choices, and parameter grids. The strategy that reaches the pitch deck is not sampled at random from that garden. It is selected because it looked good.

That selection step changes the statistics before anyone opens the slide deck.

Thesis: the correct question is not "what was the Sharpe ratio of this backtest?" The correct question is "what was the distribution of the best Sharpe ratio after the entire search procedure?"

This is not a cynical point. Some strategies are real. Some market anomalies survive costs, crowding, publication, and regime shifts. But the evidence has to be priced correctly. A clean-looking equity curve can be either discovery or data reuse with a good costume.

White’s Reality Check made the data-snooping problem explicit for model selection: when the same data are used repeatedly for specification search, a strong-looking winner may owe more to chance than to predictive ability.1 Sullivan, Timmermann, and White applied the idea to technical trading rules and showed how conclusions change when the full rule universe is treated as part of the test.2 Hansen’s superior predictive ability test refined the Reality Check by improving power against relevant alternatives.3

The modern quant version is sharper. Bailey, Borwein, Lopez de Prado, and Zhu define the probability of backtest overfitting in investment simulations, using combinatorially symmetric cross-validation to estimate how often in-sample winners disappoint out of sample.4 Bailey and Lopez de Prado’s deflated Sharpe ratio adjusts for selection bias, non-normal returns, and the number of trials behind a reported Sharpe.5 Harvey, Liu, and Zhu make the same argument at the factor zoo level: once hundreds of factors have been searched, the familiar t-statistic hurdle near 2 is too low.6

The common theme is simple: a backtest is evidence only after the search budget has been put on the table.

The Winner Came From a Crowd

Suppose a team tests \(m\) strategies that have no true edge. Each test produces a one-sided p-value. If the team reports the best one, the relevant null is not

\[\Pr(p < 0.05) = 0.05.\]

It is

\[\Pr(\min_i p_i < 0.05) = 1 - 0.95^m,\]

under independent null tests. With \(m=100\), the probability of at least one nominally significant false discovery is about 99.4%. With \(m=1000\), it is essentially certain.

Real strategy libraries are not independent. Variants share signals, assets, features, code, and assumptions. Correlation reduces the effective number of independent trials, but it does not restore the single-test interpretation. If you tried 500 versions of momentum, the winner is not “a momentum strategy with a Sharpe of 1.8.” It is the maximum of a correlated family of noisy estimates.

That is why the phrase “we only show the final model” should make an investor more nervous, not less. The discarded variants are part of the evidence.

Sharpe Has Sampling Noise

Even before multiple testing, Sharpe ratios are noisy. Lo derives the sampling behavior of Sharpe ratios under several return assumptions and shows why naive annualization can fail when returns are serially correlated.7 A five-year monthly backtest is not a microscope. It is 60 observations, often smoothed, clustered, regime-dependent, and measured after many design choices.

A useful mental model is

\[\widehat{S}_j = S_j + \epsilon_j,\]

where \(S_j\) is a strategy’s true Sharpe and \(\epsilon_j\) is estimation noise. The strategy selected by a backtest is

\[j^\star = \arg\max_j \widehat{S}_j.\]

Selection makes \(\epsilon_{j^\star}\) special. It is likely positive. The winner won partly because its noise term helped it. This is the same reason the tallest child in a classroom is likely to be less exceptional on the next measurement if yesterday’s height included measurement error. In finance, the next measurement is called live trading.

Corrections Are Receipts

There are several ways to respond. None is magic.

Bonferroni-style correction controls the probability of making even one false claim across a family of tests. It is simple and often conservative.

False discovery rate procedures, introduced by Benjamini and Hochberg, control the expected fraction of false discoveries among the discoveries.8 This is often a better scientific target when many hypotheses may be real and the goal is to build a research program rather than approve a single trade.

Reality Check and SPA-style tests ask whether the best model in a search outperforms a benchmark after accounting for the search. Probability of backtest overfitting asks how often an in-sample winner turns into an out-of-sample loser under cross-validation. Deflated Sharpe asks whether a reported Sharpe still clears a hurdle after selection bias and non-normality are priced in.

The practical rule is not “use this one correction.” It is:

  1. Define the research universe before looking at the winner.
  2. Count failed variants as part of the experiment.
  3. Separate model selection from final evaluation.
  4. Penalize the selected statistic for search.
  5. Treat costs, turnover, capacity, and decay as part of the null, not as a footnote after the signal looks good.

Research Department Simulator

The simulator below creates a research department. In each campaign, the department tests many candidate strategies. A small fraction may have genuine pre-cost edge. The rest are null strategies. Each candidate receives a noisy in-sample Sharpe estimate, a noisy validation estimate, and an expected live Sharpe after implementation drag.

The chart compares five selection policies:

  • always publish the best in-sample backtest;
  • publish the best strategy with nominal one-sided \(p < 0.05\);
  • publish only if the best strategy survives Bonferroni correction;
  • publish using Benjamini-Hochberg false discovery control;
  • reserve a holdout slice, pick the research-half winner, and require validation significance.

The Audit tile is generated by the same JavaScript as the chart. It runs 20 deterministic checks over parameter bounds, p-value arithmetic, the Benjamini-Hochberg step-up rule, policy ledgers, headline consistency, selected noise inflation, null-search live decay, sample-length effects, real-edge recovery, live-drag sensitivity, and correlation as a smaller effective search budget. It is a simulator contract, not evidence that any trading rule works.

Selected backtest Sharpe Expected live Sharpe Selection rate False discovery rate

Deterministic simulation, 260 independent research campaigns. The audit checks simulation contracts. A false discovery is a selected strategy whose true live Sharpe after drag is not positive.

The static-site audit runner replays the default research department and checks the policy ledger against fixed deterministic outputs:

node - <<'NODE'
const lab = require("./assets/js/backtest-lab.js");
const EXPECTED_AUDIT = { checked: 20, failureCount: 0, ok: true, passed: 20 };
const EXPECTED_DEFAULT = [
  "400:6:4:70:35:20:260",
  "1.450498:0.425884:0.188462:0.380769:0.604551:0.211538",
  "best=1.000000/0.188462/1.450498/0.425884",
  "nominal=1.000000/0.188462/1.450498/0.425884",
  "bonferroni=0.380769/0.070707/1.675096/0.547244",
  "bh=0.553846/0.111111/1.603352/0.501656",
  "holdout=0.211538/0.000000/0.830600/0.604551"
].join(":");

function fixed(value) {
  return Number(value).toFixed(6);
}

function sameJson(actual, expected) {
  return JSON.stringify(actual) === JSON.stringify(expected);
}

const audit = lab.runAudit();
const result = lab.evaluate({
  strategies: 400,
  years: 6,
  edgeRate: 4,
  edgeSharpe: 70,
  correlation: 35,
  liveDrag: 20
});
const auditShape = {
  checked: audit.checked,
  failureCount: audit.failures.length,
  ok: audit.ok,
  passed: audit.passed
};
const paramsShape = [
  result.params.strategies,
  result.params.years,
  result.params.edgeRate,
  result.params.edgeSharpe,
  result.params.correlation,
  result.params.liveDrag,
  result.campaigns
].join(":");
const headlineShape = [
  result.headline.bestBacktest,
  result.headline.bestLive,
  result.headline.bestFalse,
  result.headline.bonferroniSelection,
  result.headline.holdoutLive,
  result.headline.holdoutSelection
].map(fixed).join(":");
const policyShape = result.policies.map((row) => `${row.key}=${[
  row.selectedRate,
  row.falseRate,
  row.backtest,
  row.live
].map(fixed).join("/")}`).join(":");
const defaultShape = [paramsShape, headlineShape, policyShape].join(":");

if (!sameJson(auditShape, EXPECTED_AUDIT) || defaultShape !== EXPECTED_DEFAULT) {
  throw new Error(JSON.stringify({ auditShape, defaultShape }, null, 2));
}

console.log(`${audit.passed}/${audit.checked} backtest lab checks passed`);
NODE

Try increasing the number of tested strategies while leaving the edge rate near zero. The best in-sample Sharpe improves, but the expected live Sharpe does not magically improve with it. The search is finding noise.

Now increase the true edge rate. Multiple testing correction is no longer just a scold; it becomes a way to separate a crowded discovery set from a random winner. Finally, increase live drag. A strategy can be statistically real and economically dead. That distinction matters because markets pay after costs.

The Search Ledger

The audit trail of a quant strategy should look more like an experiment registry than a slide deck. I would want to know:

  • the total number of tested variants, including abandoned branches;
  • the exact family of hypotheses covered by the search;
  • the rules for train, validation, and final holdout periods;
  • the transaction-cost and capacity assumptions before seeing performance;
  • the selected statistic and its multiple-testing correction;
  • the sensitivity of performance to small design changes;
  • the live monitoring rule that will retire the strategy.

The last item matters. Backtest overfitting is not only a launch problem. It is also a monitoring problem. If a strategy was selected from a large library, its initial live underperformance is not surprising evidence of “bad luck.” It may be the expected regression from selected noise.

A Better Research Notebook

The strongest version of quant research is not less creative. It is more honest about the denominator.

I would like to see strategy notebooks expose a “search ledger” by default: each idea, parameter branch, data filter, and rejection should become part of a machine-readable experiment graph. The final backtest would then be accompanied by an automatically computed search-adjusted evidence report: effective trial count, family definition, holdout hygiene, deflated Sharpe, PBO-style cross-validation diagnostics, cost sensitivity, and live decay triggers.

That sounds bureaucratic until you realize what the alternative is. Without a ledger, the research process becomes a private multiple-testing machine and the investor receives only the maximum.

A backtest is a microscope if the experiment was pre-registered. It is a mirror if the experiment was searched until it smiled back.

Primary Sources

  1. Halbert White, “A Reality Check for Data Snooping,” Econometrica, 2000. PDF. ↩

  2. Ryan Sullivan, Allan Timmermann, and Halbert White, “Data-Snooping, Technical Trading Rule Performance, and the Bootstrap,” Journal of Finance, 1999. SSRN. ↩

  3. Peter Reinhard Hansen, “A Test for Superior Predictive Ability,” Journal of Business & Economic Statistics, 2005. SSRN. ↩

  4. David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu, “The Probability of Backtest Overfitting,” Journal of Computational Finance, 2015. SSRN. ↩

  5. David H. Bailey and Marcos Lopez de Prado, “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality,” Journal of Portfolio Management, 2014. SSRN. ↩

  6. Campbell R. Harvey, Yan Liu, and Caroline Zhu, “…and the Cross-Section of Expected Returns,” Review of Financial Studies, 2016. SSRN. ↩

  7. Andrew W. Lo, “The Statistics of Sharpe Ratios,” Financial Analysts Journal, 2002. SSRN. ↩

  8. Yoav Benjamini and Yosef Hochberg, “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society: Series B, 1995. DOI. ↩