Stability
A single score in [0, 1] for how broad the optimum of a parameter sweep is, and the three-way label that reads it.
An optimizer returns the best point. A parameter-stability graph shows what surrounds it: a broad plateau where the neighbouring configurations perform nearly as well, or a lonely spike where one configuration wins and its neighbours do not. The plateau is robust and the spike is fragile — a spike is the signature of a parameter fitted to the noise of one sample. crates/vike-analytics/src/stability.rs quantifies that reading into one number and one word.
The score
parameter_stability(scores, top_frac) takes the trial scores of a sweep, in any order, and returns a value in [0, 1]. Only a plain score list is needed — no optimizer, no trial-generation machinery, nothing about which parameters produced which score. Any sweep that yields a scalar per point is an input; a sweep ranked by an objective is the natural one.
The procedure, from the module header:
- Sort the scores best-first.
- Take the top
top_fracfraction of trials as the neighbourhood of the optimum — at least one trial, however small the fraction. - Compute the plateau ratio:
mean(top scores) / best_score. - Clamp to
[0, 1].
A ratio of 1.0 means every score in the neighbourhood equals the best — a flat plateau. A ratio near 0.0 means the neighbourhood averages far below its peak — an isolated spike. Zero or one trial returns 1.0, since there is no evidence of spikiness to report. The module's own tests fix the reading: [10, 10, 9.9, 9.8, 9.7, 1.0] at top_frac = 0.25 scores at least 0.9, while [10, 1, 1, 1, 1, 1, 1, 1] at top_frac = 0.5 takes the top four — [10, 1, 1, 1], mean 3.25 — for a ratio of 0.325.
Negative scores are handled by sign: the ratio is inverted as well as negated — (−best) / (−mean) — so that "least negative" reads as best, and a negative plateau such as [−1, −1.1, −1.2, −1.3, −10, −20] still scores at or above 0.8. A best score of exactly zero returns 1.0 when the neighbourhood mean is also zero and 0.0 otherwise, and never divides.
The neighbourhood count is rounded the oracle's way
k = round(n · top_frac) is computed with the module's round_half_even — round-half-to-even, the behaviour of Python's round() — not with f64::round, which rounds half away from zero. The doc is explicit about why: k must match exactly, or the number of trials entering the "top" neighbourhood silently diverges from the oracle the port was verified against. 2.5 rounds to 2; 1.5 rounds to 2; 0.75 rounds to 1.
The label
stability_label(stability) turns the score into a word: "plateau" at or above 0.8 (a broad, robust optimum), "ridge" at or above 0.5 (moderate sensitivity), and "spike" below that (a fragile, likely overfit optimum). The boundaries are inclusive at the top of each band — 0.8 is a plateau, 0.79 a ridge, 0.5 a ridge, 0.49 a spike.
How to read it beside the other gates
Stability is a statement about the shape of the sweep, not about the significance of any one result. A plateau says the strategy's behaviour does not hinge on a precise parameter value; it does not say the plateau is above zero. That is the job of the deflated Sharpe and the probability of backtest overfitting, which take the same sweep and ask whether its winner beats what a search of that size produces from noise. Read them together: stability says whether the result depends on hitting one parameter value, and the deflated Sharpe says whether the result is distinguishable from what the search would have found in noise.
Overfitting
The probabilistic and deflated Sharpe ratios, the probability of backtest overfitting, a plain-language verdict, and the selection audit that refuses to be fed only the survivor.
Validation
The time-series cross-validation splitters — walk-forward by count or by duration, purged k-fold, and combinatorial purged paths — and where the embargo sits.