A quantitative researcher has tested 500 different trading strategies over a 10-year historical dataset using the same underlying data. The best-performing strategy achieves an annualized Sharpe ratio of 2.4 with a maximum drawdown of 8%. A risk committee member argues the result should be heavily discounted. Which concept BEST justifies the risk committee member's skepticism?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Testing 500 strategies on the same data is like flipping a coin 500 times and getting excited when you hit heads 12 times in a row — the probability is irrelevant without accounting for how many times you tried. This is data snooping, and a Sharpe of 2.4 out of 500 attempts is roughly what you'd expect by pure chance. Harvey, Liu, and Zhu showed that the Sharpe ratio discovery hurdle should be raised substantially when many strategies have been tested on the same dataset.
Full explanation below image
Full Explanation
The multiple comparisons problem, also called data snooping bias or p-hacking, is the dominant validity threat when testing many strategies on the same historical dataset. If 500 strategies are evaluated, and all are compared to the same market realization (historical returns with identical noise), the best-performing strategy's apparent Sharpe ratio reflects optimization to the noise in that specific sample, not generalizable signal.
Harvey, Liu, and Zhu (2016, Review of Financial Studies) derived that the minimum t-statistic required to declare a factor 'significant' should be raised from the conventional 1.96 to approximately 3.0 given the cumulative number of factor discoveries reported in the academic and practitioner literature. For 500 strategies tested in-house, the analogous hurdle is even higher. Alternatively, the Bonferroni correction divides the significance threshold by the number of tests (500), yielding a required individual p-value of 0.0001 rather than 0.05.
Option A (transaction cost underestimation) is a real and common backtesting flaw but is secondary to the data snooping problem. Transaction costs might reduce the Sharpe from 2.4 to 1.9, but data snooping can explain the entire observed Sharpe as noise.
Option C (benchmark selection bias) is a performance attribution concern, not a validity concern. A Sharpe ratio is already risk-adjusted and does not require a benchmark; the risk committee's concern is about whether the number is real, not how to interpret it relative to a benchmark.
Option D (regime dependency) is a valid concern for strategy robustness but is a secondary issue. Regime dependency affects whether the strategy's historical performance predicts future performance; data snooping affects whether the historical performance is even meaningful in isolation.
Best practice is to pre-register the primary strategy hypothesis before testing, reserve a hold-out dataset (true out-of-sample), and apply a Bonferroni or False Discovery Rate correction if multiple strategies are evaluated.