Learn · Case study
Why your backtest looks great and your live account doesn’t
We ran 21,888 backtests across five years of NQ and ES futures data. The best result turned $100,000 into $227,000 with a clean, rising equity curve, the kind of chart that sells trading courses. It was also completely worthless, and proving that took three checks most backtesting software never runs.
This is a case study in backtest overfitting: what it looks like from the inside, why it is the default outcome of any serious strategy search, and the specific tests that separate a real edge from a beautiful accident.
The setup: an honest search, run at scale
The data: Databento 1-minute NQ and ES futures, June 2021 to June 2026, rolled to regular-session 5-minute bars, about 99,000 bars per symbol. The strategy space: every 1–3 candle price pattern (Strat-style labeling), each forward-simulated with pre-specified triple-barrier exits on every occurrence: winners and losers, no outcome filtering. Costs were modeled at 2 ticks round-trip plus 1 tick of slippage. 288 parameter combinations × 84 patterns = 24,192 configurations tested; the 21,888 with a large-enough sample (at least 150 trades each) form the multiple-testing trial count.
The winner, and why it was a lie
The top result: a long NQ pattern earning +0.15 to +0.22R per trade. Equity curve from $100k to $227k. Here is what the same strategy looked like once we stopped admiring the curve:
| Check | Result | Verdict |
|---|---|---|
| Equity curve, NQ | $100k → $227k | Looks brilliant |
| Same pattern on ES (~0.9 correlated) | Negative | Fails the twin test |
| Deflated Sharpe Ratio at 21,888 trials | 0.21 | Far below the 0.95 bar |
| Maximum drawdown | −31% | Untradeable at size |
Three separate autopsies, one cause of death. NQ rose from ~14,150 to ~30,647 over the window; every long pattern “worked” because the market went up, not because the pattern meant anything. The backtest wasn’t measuring an edge. It was measuring a bull market wearing a costume.
Why the best of 21,888 is impressive by construction
This is the core statistical trap. Run enough random strategies on the same history and the best one will look spectacular. Guaranteed. The expected maximum Sharpe ratio of N skill-free strategies grows roughly like √(2 ln N): with 1,000 trials, a backtest Sharpe above 1 is expected on pure noise. If your tool shows you the winner without telling you how big the search was, it is showing you a lottery ticket and calling it skill. The fix is the Deflated Sharpe Ratio: it re-scores the winner against the luck of a search that size.
Round two: doing it properly, and still finding nothing
We re-ran the study the disciplined way: a small pre-registered set of 96 direction-matched patterns, each scored on its excess return over blindly trading the same direction on every bar (which subtracts the trend). The only drift-independent signal was on the short side, worth about +0.04-0.05R over baseline, and it still failed: Deflated Sharpe 0.53 against the 0.95 bar we require, with 70–86% drawdowns from shorting into a five-year uptrend.
Final score: 21,984 trials, zero tradeable edges. The zero is the finding. Most pattern sets on liquid markets contain nothing that survives honest testing, and a tool that can tell you so is the only kind whose “yes” means anything.
The five checks that catch overfitting
- Count your trials, all of them. Every parameter tweak, every indicator swap, every timeframe you eyeballed. Then penalize the winner for the whole search (DSR).
- Seal a vault before you search. Lock away the most recent slice of data before the first test, and open it once, at the end. If the edge dies on data it never saw, it was never an edge (how to structure this).
- Beat the honest baseline, not zero. In a trending market, score every long strategy against “just be long.” The excess is your edge; the rest is drift.
- Run the twin test. A structural edge on NQ shouldn’t lose money on ES. If it works on one 0.9-correlated twin and fails on the other, it’s noise.
- Look at the real drawdown, with costs in. The smoothed curve hides the valley. A 31% drawdown ends most accounts before the “edge” can pay.
What this means for your own testing
None of this says backtesting is useless. It says naive backtesting is worse than useless: it produces confident, precise, wrong answers. Testing honestly mostly produces “no,” and each honest no is money you didn’t lose trading a mirage. The rare survivor of all five checks is the thing worth sizing.
TrueEdge runs these checks by default: sealed vault, Deflated Sharpe at the true trial count, out-of-sample persistence folds, real drawdowns with costs in. The free tier runs the honest core (sealed vault + the Deflated Sharpe verdict) on your own data, 1 discovery run a day, no card required; Pro unlocks the full validation detail.
Try the free tierNext: The Deflated Sharpe Ratio, explained for traders who hate math · Walk-forward analysis and out-of-sample testing