Folds Remember Label Windows

A validation split is not just a partition.

It is a firewall.

When the observations are independent, the firewall is easy to draw: shuffle the rows, hold out some rows, train on the rest. But financial labels often have a different geometry. A sample is not a point in time. It is an interval.

At trade time t, you observe features. The label might be the sign of the next 20 trading days:

y_t = sign(r_{t+1} + r_{t+2} + ... + r_{t+20})

That label consumes the future interval (t, t + 20]. The next sample, y_{t+1}, consumes almost the same interval. If y_t is in the test fold and y_{t+1} is in the training fold, the training set has not used a forbidden feature column. It has still seen nearly the same realized future path.

The leak is not in the matrix.

It is in the label.

The Row Is Not the Event

Most machine-learning tooling thinks in rows. Time-series validation has to think in events.

For a financial event i, keep two timestamps:

trade_time_i = when the features are known
event_time_i = when the label is fully known

For a fixed-horizon return, event_time_i = trade_time_i + H. For a triple barrier label, the event time is whichever comes first: profit target, stop loss, or vertical barrier. In both cases, the label is formed over an interval:

[trade_time_i, event_time_i]

Two rows can be far apart in a shuffled dataframe and adjacent in information time. Two labels overlap if their event intervals intersect. That overlap is the object the split has to police.

This is why Lopez de Prado’s financial ML treatment puts labeling, overlapping outcomes, sample uniqueness, and purged K-fold cross-validation in the same pipeline.¹ The cross-validation chapter is not merely about respecting chronological order. It is about preventing the test label’s event interval from being represented inside the training labels.

Scikit-learn’s TimeSeriesSplit documentation gives the ordinary chronological warning: other cross-validation methods can train on future data and evaluate on past data; the splitter also exposes a gap parameter that excludes samples from the end of each training fold before the test fold.² That is a useful primitive. It is not the whole contract. A gap between trade times is not always the same thing as a purge between event intervals.

A Split Makes a Promise

Here is the strongest way I know to say it:

Validation is a conditional independence claim.

It claims the evaluation rows were not made easier by information available in the training rows. For independent tabular data, row disjointness may be a reasonable approximation. For time series, spatial data, panels, recommender logs, user histories, and market events, row disjointness can be theater.

This is an old problem with many dialects.

Burman, Chow, and Nolan proposed h-block cross-validation for dependent data: leave a neighborhood around the test observation out of the training set so nearby dependent observations cannot make the evaluation too easy.³ Racine later proposed hv-block cross-validation for dependent-data model selection.⁴ Roberts and coauthors reviewed cross-validation strategies for temporal, spatial, hierarchical, and phylogenetic structure and argued for blocking when dependence structures matter.⁵

There is also a useful caution against over-correction. Bergmeir, Hyndman, and Koo show that standard K-fold can be valid for purely autoregressive time-series forecasting when the candidate models have uncorrelated errors.⁶ So the right lesson is not “never use K-fold on time series.” The right lesson is:

validate the split against the dependence created by your prediction problem

For a one-step autoregression with clean residuals, the danger is different from a 20-day overlapping financial label. For a panel of users, the danger is different again. The split has to match the data-generating graph.

Purge the Overlap, Embargo the Echo

Suppose a test fold contains events with intervals:

[trade_time_j, event_time_j], for j in test

Purging removes every training event whose interval overlaps any test interval:

[trade_time_i, event_time_i] intersects [trade_time_j, event_time_j]

Embargoing removes a buffer after the test fold. It is a cruder defense against downstream dependence: market impact, delayed reaction, feature construction, data-vendor revisions, overlapping holdings, or simply residual serial correlation that a strict interval-overlap rule does not capture.

They solve different problems.

Purging says: “Do not let the training labels share the test label’s realized future.”

Embargoing says: “Do not assume dependence stops exactly at the boundary I drew on the chart.”

Walk-forward validation is even stricter in one dimension: it trains only on the past and evaluates in the future. That matches deployment time, but it can use less data per fold and gives fewer paths through market history. Purged K-fold uses more historical slices while trying to remove event-overlap contamination. Combinatorial purged CV extends this idea by creating multiple out-of-sample paths from combinations of test groups, which is useful when the market gives you only one history.¹

No splitter is a sacrament. Each one encodes a different claim.

A Little Leak Detector

The lab below generates a synthetic market. There is a persistent latent signal, noisy features observed at trade time, and a forward return label over H days. A simple kernel classifier predicts the sign of the forward return.

The important quantity is not only accuracy. The lab also counts whether a test label’s event interval overlaps any training label’s event interval.

The four splitters are:

• Shuffled K-fold, which randomly distributes rows across folds.
• Blocked K-fold, which uses contiguous test blocks but still trains on both sides of the test block.
• Purged + embargo, which starts with blocked folds and removes overlapping event intervals plus a post-test buffer.
• Walk-forward, which trains on earlier history only, with the same gap.

The final holdout is the last 18 percent of the synthetic history, evaluated as if time only moves forward. It is not perfect ground truth. It is a useful anchor because it was not part of the cross-validation folds.

Label horizon 5 days 10 days 20 days 30 days Serial persistence low medium high very high Feature noise low medium high Folds 4 5 6 8 Embargo 0 days 5 days 10 days 20 days Model bandwidth narrow medium wide Highlight Shuffled K-fold Blocked K-fold Purged + embargo Walk-forward

Train Test or leakage Shuffled Blocked Walk-forward

Deterministic synthetic experiment. The point is not that the kernel model is a good trading model; the point is that the same legal feature matrix can produce very different validation estimates when labels overlap in event time.

With the default settings, shuffled K-fold looks wonderful. It usually reports around 72 percent accuracy while the final forward holdout is around 56 percent. The giveaway is the leakage counter: almost every shuffled test label has at least one overlapping training label.

Blocked K-fold fixes the most cartoonish error: it stops scattering adjacent rows into every fold. But it still trains on both sides of a test block. The labels near the test boundary can overlap. Purged + embargo removes those training rows. Walk-forward removes even more history because it refuses to train on anything after the test fold.

Now change Label horizon from 5 to 30 days. Longer horizons create longer event intervals. The same row split becomes more contaminated. Increase Serial persistence. Neighboring features become more similar, so label overlap becomes easier for the model to exploit. Increase Embargo. The purged splitter keeps less training data, but its leakage counter remains at zero.

The loss of training data is not a bug. It is the price of the claim.

Make the Split Show Its Papers

A serious time-series validation report should not only print a score. It should print the split hygiene:

• trade time and event time for every labeled sample;
• number of test labels whose event interval overlaps training labels;
• fraction of train/test event-interval pairs that overlap;
• embargo length and why it was chosen;
• number of training samples discarded by purging and embargoing;
• whether any fold trains on observations after the test fold;
• final untouched chronological holdout;
• sensitivity of the score to a larger gap.

That report changes the tone of a research conversation. Instead of arguing whether random K-fold is “allowed” for time series in the abstract, you can ask:

Which test labels are represented in the training labels?

If the answer is “many,” the validation score is not a forecast of deployment. It is a measurement of a contaminated experiment.

Honesty Has a Variance Bill

There is a real trade-off here. Purging and embargoing throw away data. A walk-forward split can have high variance because early folds train on little history. A very large embargo can make the model look worse simply because it starves the learner. That is why the correct output is not one blessed number.

The correct output is a curve:

score as a function of purge width, embargo width, horizon, and regime

If the research conclusion survives that curve, I trust it more. If the alpha exists only when the gap is zero, it is probably not alpha. It is the shape of the split.

This is especially important for machine-learning finance because the model is often flexible enough to exploit weak artifacts. A linear model may merely be fooled. A nearest-neighbor model, random forest, boosted tree, or transformer can quietly become an index of local history. The model does not need to know that it is cheating. It only needs a feature representation in which nearby events are close.

Make Label Intervals Real Objects

I would like backtesting libraries to treat label intervals as first-class objects. Not optional metadata. Not comments in a notebook. Objects.

Every labeled sample should carry:

feature_time
label_start_time
label_end_time
asset_or_entity_id
construction_version

Then splitters could refuse invalid folds by construction. Model-selection reports could include a leakage matrix. Hyperparameter searches could carry the same event-time firewall that the final backtest uses. Feature-importance plots could be annotated with the actual split contract under which they were estimated.

The most dangerous validation bugs are the ones that look statistically professional. A five-fold score with confidence intervals feels rigorous. If the folds share label intervals, the rigor is decorative.

A fold has a memory. The validation job is to decide how much of that memory must be erased before the score deserves to be believed.

Primary Sources

1. Marcos Lopez de Prado, Advances in Financial Machine Learning, Wiley, 2018. The table of contents includes chapters on labeling, overlapping outcomes, sample uniqueness, and “Why K-Fold CV Fails in Finance” with “Purged K-Fold CV,” purging, and embargo. PhilPapers entry, Wiley. ↩ ↩²

2. scikit-learn, TimeSeriesSplit documentation. The splitter is for time-ordered data and includes a gap parameter excluding samples before the test set. Documentation. ↩

3. Prabir Burman, Edmond Chow, and Deborah Nolan, “A Cross-Validatory Method for Dependent Data,” Biometrika, 1994. PDF. ↩

4. Jeff Racine, “Consistent Cross-Validatory Model-Selection for Dependent Data: hv-Block Cross-Validation,” Journal of Econometrics, 2000. IDEAS/RePEc. ↩

5. David R. Roberts et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure,” Ecography, 2017. Publication page, DOI. ↩

6. Christoph Bergmeir, Rob J. Hyndman, and Bonsoo Koo, “A Note on the Validity of Cross-Validation for Evaluating Autoregressive Time Series Prediction,” Computational Statistics & Data Analysis, 2018. Rob Hyndman’s publication page, PDF. ↩