I already wrote about auditing every backtest file I have — 152,484 results, 9 pairs I'd actually trust. That was the wide-angle view: everything I've ever run, one final pass/fail line each. This is the narrow-angle view of one specific batch, and it's uglier. It's what happens to the configs that already looked good enough to keep testing.
My pipeline has three stages. Broad screening throws out most things fast. Anything that survives goes to deep validation — walk-forward folds on held-out data, plus two checks that don't show up in a normal backtest report: how much of the P&L clusters into a handful of good days versus how many days are actually red, and how many trades secretly needed a third round of averaging down to stay alive. I ran a batch of 2,331 broad configs through this. Here's where they went.
The funnel
| Stage | Configs | % of 2,331 |
|---|---|---|
| Entered broad screening | 2,331 | 100% |
| Rejected — too many bad trading days | 1,299 | 55.7% |
| Rejected — relied on a 3rd DCA level | 1,005 | 43.1% |
| Marginal pass (weak / too few trades) | 27 | 1.2% |
| Reached deep walk-forward validation | 20 | 0.86% |
| Passed deep validation | 0 | 0.0% |
2,331 stage10 broad-screening configs and their 20 deep-validation descendants, this batch's run.
The 20 that made it this far — and still didn't make it
Getting to deep validation already means a config survived the two broad filters above. These 20 weren't junk. Eighteen of them still put up a per-trade win rate over 60%, median 67.8%, topping out at 79.2% on a 1000LUNCUSDT micro_momentum config with 500 trades. And every single one of the 20 was still net negative — that config's net sat at -83.09, and the worst of the batch, a 1000LUNCUSDT pullback_reclaim config with a 68% win rate, closed at -177.34 net over 500 trades.
The rejections split two ways. Eighteen were flagged for how their daily P&L was distributed — a high per-trade win rate doesn't stop a strategy from spending most of its trading days red, and that's a different failure mode than a bad win rate. The other two — both 1000000MOGUSDT configs, volume_reversal and range_bottom_reclaim — were flagged because more than 11% of their trades only worked because the system quietly modeled a third averaging-down entry. Take that rescue level away, which is exactly what a real drawdown eventually forces, and the trade doesn't recover.
The one result that actually surprised me: the two LLYUSDT range_bounce configs had the best fold-pass-rate in the whole batch, 80% — meaning they held up in 4 of 5 out-of-sample walk-forward folds, which is normally the hard part. Both still got rejected, on the day-level check. Surviving the folds tells you the edge isn't a fluke of one time window. It doesn't tell you whether the wins and losses land in a shape you could actually sit through.
What this changes about how I read "made it to final testing"
It's tempting to treat "reached deep validation" as most of the way to done. In this batch it was 0.86% of the starting pool and still a 0% pass rate once it got there. A config that clears broad screening has already beaten 99%+ of what I threw at it and can still be worthless. The number that kept fooling me was win rate — 18 of 20 finalists cleared 60%, and none of them were fundable. Fold-pass-rate fooled me too, in the other direction: the best fold performer in the batch was also rejected.
What I actually trust now, in order: net P&L across real out-of-sample folds first, then how many of the calendar days along the way were red, then how much of the win rate depends on a DCA level I haven't actually tested surviving. Win rate on its own goes last. It's the number every one of these 20 rejects would have looked best on if I'd stopped checking early.
Before you trust a backtest that "passed"
- Ask what stage it passed, not just that it passed. 2,331 configs entered this batch's screening; only 20 (0.86%) reached the deep walk-forward stage, and none of those 20 passed either.
- Don't stop at win rate. 18 of the 20 finalists here had a win rate above 60%, one hit 79.2%, and every one was still net negative.
- Check how P&L lands across calendar days, not just the total. 55.7% of the broad pool and 18 of 20 finalists were rejected specifically for this, separate from win rate or profit factor.
- Ask what the strategy needs to survive a drawdown. Two finalists needed a third DCA level on over 11% of trades just to stay net positive on paper.
→ Risk of ruin calculator · → DCA survival calculator · → Profit factor calculator
FAQ
What percentage of backtest configs make it from screening to final validation?
In this batch, 0.86% — 20 out of 2,331 broad configs reached my deepest walk-forward fold validation stage. Getting that far isn't the same as passing it: all 20 of those finalists were still rejected once tested.
Why do high win-rate backtests still fail validation?
Because win rate is a per-trade number and it says nothing about how P&L is distributed across calendar days or how much the result depends on averaging into losers. Of my 20 finalist configs, 18 had a per-trade win rate above 60% (one hit 79.2%) and every single one was still net-loss and still rejected.
What is a DCA shadow rate in backtesting?
It's the share of trades that would have needed an extra round of averaging down to survive, even though that extra DCA level wasn't actually taken in the live simulation. It's a stress-test on the strategy's actual reliance on rescue orders. Two of my 20 finalists were rejected specifically because upward of 11% of their trades needed a third DCA level to hold together.
Does passing out-of-sample folds guarantee a strategy is safe to trade?
No. The best fold-pass-rate in this batch was 80% — a config that held up in 4 of 5 out-of-sample folds — and it still got rejected on a separate day-level check. Surviving walk-forward folds is necessary but not sufficient; a strategy can be stable fold-to-fold and still be unsafe in how its wins and losses land across real trading days.