Combinatorial Purged Cross-Validation: Why Standard Backtests Overfit

1. Why standard backtests overfit

Three problems compound each other. The first is overlapping labels. If a label is the return over the next 20 bars, an observation at time t and one at t + 5 share most of their label. Put one in training and the other in the test set, and the model has already seen the answer to the question it is being tested on.

The second is serial correlation. Price series, volatility estimates and moving-average features change slowly, so observations just before or after a test window are close relatives of the test data even when their labels do not overlap.

The third is that history provides one path. A walk-forward test trains on the past and tests on the next block, which respects time order but produces a single sequence of results. One path is one draw. It cannot distinguish a robust edge from a fortunate ordering of events, and every parameter adjustment made after looking at that path adds to the selection bias.

2. Purging and embargo

Purging and embargo are the two repairs applied to the training set around each test block.

• Purging: every training observation whose label window overlaps the time span of the test set is deleted. The model never trains on a label that was partly determined by test-period prices.
• Embargo: a further buffer of observations immediately after the test set is also dropped from training. It covers leakage through serial correlation that persists beyond the label horizon, which a purge on label overlap alone does not catch.

The embargo is sized to the memory of the data rather than fixed in principle. The note on causal factor-absence placebo testing describes the protocol Qlumina applies, with a buffer of 10 trading days after every test set.

3. How combinatorial purged cross-validation works

CPCV divides the history into N contiguous groups of observations. For each split, k of those groups are held out as the test set and the other N minus k form the training set, with purging and embargo applied around every test block. Repeating this for every possible choice of k groups gives C(N, k) splits.

Because each group appears in the test set in several different splits, the out-of-sample predictions can be stitched together into complete paths, each covering the whole history exactly once but assembled from different models.

The distinction between splits and paths matters. With N = 6 and k = 2 there are 15 train and test splits, and from them 5 complete backtest paths can be assembled. The model count is large. The independent paths are fewer.

4. What to do with a distribution of paths

The point of many paths is to stop reporting a single number. A strategy whose Sharpe ratio is high on every path is a different proposition from one whose average is the same but whose paths range from strongly positive to negative. Report the dispersion, the share of paths with a loss, and the worst path.

Two related measures carry the logic further. The Probability of Backtest Overfitting estimates how often the configuration that looks best in sample ranks poorly out of sample. The Deflated Sharpe Ratio adjusts an observed Sharpe ratio for the number of trials, the length of the sample and the shape of the returns. Both depend on an honest count of every strategy and parameter set tried, which is why a trial record is part of the evidence an allocator should ask for.

5. Where CPCV still goes wrong

The method is only as good as its implementation. These are the failure modes worth checking:

6. CPCV inside a wider validation stack

No cross-validation scheme substitutes for data the research process has never seen. CPCV, however carefully built, is still run on the history the researchers can inspect. For that reason it is one layer among several: placebo controls that test whether a signal survives when the factor is removed, a Deflated Sharpe Ratio that charges for trials, and a physical quarantine of a long out-of-sample window. Qlumina describes the last of these in The 20-Year Blind Out-of-Sample Air-Gap, and the wider cost of a flattering simulation in Microstructure Feasibility.

The same applies to the models themselves: leakage is also how language-model alpha claims fail, as discussed in why LLM alpha models suffer regime collapse.

7. What allocators should ask

An allocator does not need to run the validation to test whether it was done properly. Useful questions for a manager presenting a backtest:

• How many strategies and parameter sets were tried before this one was chosen?
• Were training observations purged for label overlap, and what embargo was used and why?
• Is the result a single walk-forward path or a distribution across many paths?
• Was any part of the data held back and never examined during research?
• How do live fills compare with what the simulation assumed?

These sit inside a wider checklist, set out in how to verify a quant track record and, for the broader audit, the 38-point forensic due diligence framework. The investor guide summarises the validation controls applied to programs in its risk and operations section.

8. Further reading

• Marcos López de Prado, Advances in Financial Machine Learning (Wiley, 2018): the chapters on cross-validation in finance and on backtesting.
• David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, “The Probability of Backtest Overfitting”, Journal of Computational Finance, 2017.
• David H. Bailey and Marcos López de Prado, “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality”, Journal of Portfolio Management, 2014.

Frequently asked questions