Overfitting

A Sharpe ratio is a number; whether it means anything is a different question. crates/vike-analytics/src/overfit.rs answers it three ways, each accounting for something the raw figure ignores: track-record length and the non-normality of returns (the probabilistic Sharpe), the number of configurations that were tried (the deflated Sharpe), and whether the configuration that won in-sample keeps winning out of it (the probability of backtest overfitting). A fourth function folds two of them — the deflated Sharpe and the probability of backtest overfitting — together with an optional walk-forward consistency, into a verdict. The module is a port of the statistics in Advances in Financial ML, and every entry point is pure over an equity curve or a trial matrix.

One derivation feeds everything

Every function on this page takes the same four inputs, and there is exactly one sanctioned way to produce them: overfit::sharpe_moments, which turns an equity curve into a SharpeMoments — sr_per_obs, n_obs, skew, kurt. The struct exists because two unit conventions each carry a footgun, and one of the two has already shipped as a real bug.

The second convention is kurtosis. metrics::returns_kurtosis reports excess kurtosis (the pandas convention, normal = 0) while the PSR formula wants the non-excess form (normal = 3), so sharpe_moments rebases by +3.0 exactly once, there. n_obs is deliberately left unclamped: a curve with fewer than two returns yields n_obs < 2, and probabilistic_sharpe_ratio's own guard turns that into an honest 0.0 rather than a number derived from a degenerate curve.

The three statistics

probabilistic_sharpe_ratio(sr, n, sr_star, skew, kurt) is the probability that the true Sharpe exceeds sr_star, given the observed per-period Sharpe over n observations. With sr_star = 0 it is the single-run question — is this edge distinguishable from zero — and that is exactly what the Studio results pane shows for one run (vike-studio/src/results.rs's validation_rows, which also prints the annualized Sharpe as a separate display row).

deflated_sharpe_ratio(observed_sr, trial_sharpes, n_obs, skew, kurt) is the PSR benchmarked not against zero but against expected_max_sharpe(var_trials, n_trials) — the maximum Sharpe you would expect from that many independent trials with that much spread between them, which is what a search produces from pure noise. expected_max_sharpe returns 0.0 for fewer than two trials or non-positive variance, and that detail matters below. deflated_sharpe_with_effective_n is the same test with the trial count replaced by effective_n_trials(trial_return_series): N / (1 + (N−1)·max(mean_corr, 0)), clamped to [1, N], computed from pairwise Pearson correlation of the trials' return series. Perfectly correlated trials collapse to about one effective trial; uncorrelated or anti-correlated ones stay near N.

pbo_cscv(matrix, n_splits) is the probability of backtest overfitting by combinatorially symmetric cross-validation. matrix is T observations by N trials of per-observation performance. The observations are cut into n_splits groups (which must be even — an odd count panics); for every way of choosing half the groups as in-sample, the in-sample best column is found (first-argmax, mirroring the oracle), its out-of-sample rank among all columns becomes a logit, and PBO is the share of logits at or below zero — the fraction of paths on which the in-sample winner ranked at or below the out-of-sample median. A non-finite value anywhere in the matrix, or an empty one, returns NaN: uncomputable and honest, never 0.0, which would read as "no overfit".

The verdict

overfit_verdict(pbo, deflated_sr, wf_consistency) scores points and returns a Verdict with an OverfitLevel and a list of reasons. PBO above 0.5 scores two points, above 0.2 one; a deflated Sharpe below 0.5 scores two, below 0.9 one; a walk-forward consistency below 0.5 — the fraction of windows profitable out-of-sample, which vike_backtest::walkforward::WalkForwardReport::wf_consistency supplies (see walk-forward) — scores one. Three or more points is High, one or two Medium, none Low, with "No major overfitting flags." as the sole reason. A NaN PBO adds a "not assessed" reason and no points, so an unassessed PBO is neither a pass nor a vote toward low risk.

In the Studio, vike-studio-core/src/run.rs's run_sweep_slice computes a sweep's DSR with the effective trial count over every trial, and its PBO with N_SPLITS = 16 — C(16, 8) = 12,870 combinations, off the UI thread — only when there are at least two trials and sixteen observations, leaving the PBO NaN otherwise; the sweep's ranking sort is deliberately the annualized Sharpe, a display quantity that never reaches this module. validation_verdict_rows renders the verdict, with a NaN PBO shown as "—".

The selection audit

audit_selection exists because the gates above are only as good as the trial count they are given, and the wrong count is easy to supply.

The returned SelectionAudit carries n_candidates, effective_trials, selected_sharpe, deflated_sr, pbo, the verdict, and selected_trades. The trade count rides alongside Sharpe on purpose: a mean-reversion family dies by starvation — too few divergences — before it dies by unprofitability, and a Sharpe-ranked sweep cannot see that. Inconsistent inputs (an empty set, a length mismatch, an out-of-range index) return None: an unauditable selection is not a passing one.

It matters most for pair selection, a maximum over N(N−1)/2 correlated candidate spreads — a search larger than any parameter sweep, and one that happens before it. The doc cites published work measuring the probability that an in-sample cointegrated pair is still cointegrated out-of-sample at about 4.9% against a 5.0% unconditional base rate: indistinguishable from a 5%-level test's false-positive rate. Pair search is exactly where a spurious winner enters.

Related

Survival statistics answers the same selection question by bootstrap, for signal panels rather than equity curves; stability scores the parameter neighbourhood of a sweep's optimum; validation supplies the splitters and the combinations helper pbo_cscv iterates.