Monte Carlo and overfitting over a finished run

A finished run gives you one equity path and one Sharpe ratio. Neither is a result on its own: the path is the order the trades happened to land in, and the Sharpe is the maximum of however many configurations you searched. vike-analytics answers both questions, and this tutorial runs them over a run you already have.

Where the inputs are — and are not

Spell the Monte Carlo module vike_analytics::montecarlo. vike-backtest re-exports most of that crate's modules at its own root but deliberately not this one — re-adding a name is only correct once a call site needs the shorter spelling, and none ever has.

How much of the path was ordering

mc_resample(trade_pnls, start_equity, n_sims, seed, method) builds n_sims synthetic equity paths from the observed trade list. Two methods, each documented by what it preserves:

• Shuffle reorders the trade list, preserving the exact multiset of P&Ls per path. Terminal equity is therefore the same on every path; what varies is the shape — the sequence, and so the drawdown.
• Bootstrap samples with replacement, so a path may draw one trade several times and another never. Terminal equity varies too.

You get back terminal (each path's final equity), max_drawdowns (each path's max-drawdown fraction, through the shared metrics::max_drawdown) and curves_sample — at most 20 full curves, thinned evenly across the simulations, for plotting. confidence_bands(curves, qs) turns equal-length curves into per-step percentile bands.

mc_summary(trade_pnls, start_equity, n_sims, seed, ruin_pct) is the one-call version, running the shuffle method and returning terminal equity at the 5th, 50th and 95th percentiles, max drawdown at the 50th and 95th, prob_loss (the share of paths ending at or below the starting equity) and risk_of_ruin (the share ending at or below ruin_pct × start_equity).

Two things to decide before you read a number from it.

Which P&L you feed. Trade.pnl is gross price P&L and Trade.fees is the round-trip cost, carried separately. Feeding pnl gives you a pre-cost distribution; feeding pnl − fees gives you the one you can spend. Pick deliberately and say which in the write-up.

What the seed does and does not promise. The module is seeded for run-to-run reproducibility within this implementation, and it says outright that there is no golden fixture for it: porting CPython's exact random number generator was ruled out as not worth it for one file, so no exporter ever froze one. The provenance survives the decision — when the two implementations were run side by side, the same seed produced statistically equivalent, not bit-identical, paths.

Whether the Sharpe survives the search

crates/vike-analytics/src/overfit.rs has one sanctioned entry: sharpe_moments(equity_curve), which produces the four inputs everything else takes — sr_per_obs, n_obs, skew, kurt. It exists because two unit conventions each carry a footgun, and the annualized-Sharpe one has already shipped as a real bug. Never hand overfit:: an annualized figure; the annualized Sharpe is a display quantity for a different reader.

From one run you can compute the probabilistic Sharpe and nothing more — the deflated Sharpe and the probability of backtest overfitting both need the set of trials. That is the whole point, and audit_selection is the function that refuses to let you forget it.

The audit reports selected_trades beside the Sharpe, and its doc says why: a mean-reversion family dies by starvation — too few divergences — before it dies by unprofitability, and a Sharpe-ranked sweep cannot see that. It matters most for pair selection, a maximum over N(N−1)/2 correlated candidates, a search far larger than any parameter sweep and one that happens before the sweep.

Reading the verdict

overfit_verdict(pbo, deflated_sr, wf_consistency) folds the three into a level plus its reasons, on a fixed point scale: PBO above 0.5 scores two points and above 0.2 scores one; a deflated Sharpe below 0.5 scores two and below 0.9 scores one; walk-forward consistency below 0.5 scores one. Three points or more is High, one or two is Medium, zero is Low. A NaN PBO is not assessed — it gets its own reason row rather than a vote toward Low.

The Studio has this wired end to end, and it is the fastest way to see the numbers move. Its sweep computes the deflated Sharpe over every trial's per-period Sharpe, and PBO over the full transposed returns matrix at the canonical 16 splits — computed off the UI thread, with the PBO left NaN unless there are at least two entries and at least 16 observations. The results pane then shows the single-run half (Sharpe, observations, skew, kurtosis, PSR) and the set half (deflated Sharpe, PBO, overfit risk, walk-forward consistency when a walk-forward ran) with the verdict's reasons as notes.

One habit is worth more than any of these numbers: read n_candidates first. It is the first number to check when a result looks too good.

Next: Monte Carlo and overfitting for the reference pages, sweeps and walk-forward for producing the trial set honestly, and results for the pane that renders it.