Published
My backtest table logs a column called max_mae on every one of its 12,331 rows -- the deepest unrealized loss a config's trades sank into, on average, before closing. It isn't part of how a config earns my is_top_candidate flag. That flag runs off win rate, profit factor and trade-count thresholds. So I never actually checked whether the 50 configs that cleared those gates also had shallow drawdowns, the thing I'd assumed came along for free with a strong profit factor. It doesn't. I ranked every config in the pool by MAE and found where the 50 flagged ones land: the 20.2th percentile. Worst-fifth territory, not best.
Two percentile ranks, same 50 configs
| Metric | Flagged configs (n=50) | Rest of pool (n=12,281) | Flagged avg percentile* |
|---|---|---|---|
| Profit factor (avg) | 2.61 | 1.31 | 95.8th |
| Win rate (avg) | 83.7% | 66.8% | n/a |
| Net profit (avg, pooled units) | 451.65 | 77.60 | n/a |
| Max adverse excursion (avg) | -11.82% | -7.31% | 20.2th |
*Percentile computed by ranking all 12,331 configs on each metric and averaging the rank of the 50 flagged configs; 100th = shallowest/best MAE in the pool, 0th = deepest/worst.
The pool-wide correlation between max_mae and is_top_candidate is -0.07 -- near zero on its own, the kind of number that would normally say "no relationship, move on." But a flat correlation across 12,331 rows can hide a sharp effect concentrated in a 50-row tail, and that's exactly what's happening here. The pool's overall median MAE is -6.51%. Thirty-nine of the 50 flagged configs sit worse than that median. Eleven don't -- the best of them closes at -5.00% MAE, genuinely shallow. But the median flagged config's MAE is -14.24%, more than double the pool median, because 30 of the 50 flags share that exact figure: parameter variations on one MEGAUSDT setup, same final-invalidation level, same drawdown to the third decimal.
Why a near-zero correlation still produces a skewed flag
Run the same percentile check on profit factor and the picture flips completely: flagged configs average the 95.8th percentile, just about where you'd expect a filter that screens on profit factor directly to land its winners. That's the control case -- it confirms the percentile method works and that the MAE result isn't a quirk of how I'm ranking things. The flag does exactly what it was built to do on the two numbers it checks. It just has nothing to say about a third number that happens to move in the opposite direction for the family that wins on the first two. All 50 flagged configs come from one strategy family, range_bottom_reclaim, and that family's edge seems to run through accepting a deeper average drawdown in exchange for a much higher win rate and profit factor -- not by avoiding drawdown at all.
What I'm changing
Not the flag logic -- win rate and profit factor are still the right first gate, and this isn't a case where the flagged configs are secretly bad; their profit factor and win rate numbers are real. But I was reading "top candidate" as shorthand for "good on every axis," and that's the assumption this breaks. Before sizing a position off any flagged config now, I pull its own max_mae next to the pool average, not just its profit factor, because the flag alone doesn't tell me which bucket I landed in. A config that wins on profit factor and loses on drawdown needs smaller size than one that wins on both -- same flag, different risk budget. If you're running your own backtest exports through a similar top-N filter, check whatever field measures worst-case excursion separately before trusting the shortlist; a screen that doesn't name drawdown in its rules isn't controlling for it, no matter how clean the win rate looks.
Method: 12,331 rows from our own backtest results table (the same pipeline behind backtest-reports.html), covering 5 strategy families across 176 live-tracked symbols. max_mae, pf, wr, net and is_top_candidate are the pipeline's own logged fields; percentiles computed by full-pool rank, not bucket averages.