Bayesian parameter search

An exhaustive grid costs exponentially in resolution. Two searches in the harness buy the same final resolution for far fewer backtests, and they buy it in different ways: --search euler refines locally around the grid's winner, while --optimizer tpe models where good scores live and proposes there. Both read the SAME [sweep] table, both score through the same objective seam, and both return the same ranked SweepReport the grid returns.

They live on the standalone runner, which opens the store itself:

backtest --profile user_data/profiles/sweep.toml --optimizer tpe --trials 120 --seed 7
backtest --profile user_data/profiles/sweep.toml --search euler --euler-depth 4

The rest of this page runs all three paths over ONE profile so the numbers are comparable, and prints what each returned. That profile sweeps two numeric axes of an RSI mean-reversion strategy on BTCUSDT 1h bars.

The grid these searches are measured against

A search is only interesting against the number it is trying to beat, so start with the exhaustive grid. Three lengths by two oversold thresholds is six backtests:

backtest --profile tut_rsi_sweep.toml --rank-by sharpe

sweep ranked by sharpe (6 points)
*#1  len=14.0 lo=30.0               ret=0.2129 sharpe=1.7800 max_dd=0.0835 trades=33
#2   len=21.0 lo=30.0               ret=0.1542 sharpe=1.1885 max_dd=0.1341 trades=15
#3   len=14.0 lo=20.0               ret=0.0755 sharpe=0.6565 max_dd=0.1402 trades=12
#4   len=7.0 lo=30.0                ret=0.0246 sharpe=0.2807 max_dd=0.0873 trades=103
#5   len=7.0 lo=20.0                ret=-0.0242 sharpe=-0.1026 max_dd=0.1137 trades=50
#6   len=21.0 lo=20.0               ret=-0.0992 sharpe=-0.6544 max_dd=0.1738 trades=2

Six evaluations, best sharpe 1.7800 at len=14 lo=30. Two of the six points LOSE money — #5 returns -2.42% and #6 returns -9.92% — and the grid keeps them and ranks them last.

That matters more than it looks. The same two rows reappear at the bottom of the euler run below, and a reader who only ever sees a winner never learns how much of this space is bad.

The grid rows print no score= column. Both search outputs below do, and on every one of their rows the score equals that row's sharpe, because --rank-by picks the objective as well as the ranking.

The space both searches read

numeric_sweep_axes is the one [sweep]-table reader, shared by the grid and both searches. Every axis must be numeric AND type-homogeneous — an all-integer or an all-float TOML array. A string or bool axis has no notion of halfway between two values, and a mixed [1, 2.5] axis would silently change the TOML type the strategy receives versus the grid path, so a strategy reading its param through as_integer() would behave differently under a search than under --search grid. Both are a clean validation error telling you to use the grid.

For TPE, tpe_space_from_profile then turns each axis into a CONTINUOUS range bounded by that axis's min and max, with integral axes staying whole. So fast = [5.0, 10.0, 20.0] does not mean "try these three" — it means "search 5 to 20". TPE explores BETWEEN the grid points, the same philosophy as euler's halving.

The grid above prints len=14.0 and lo=30.0, decimal points and all, so both axes here are float arrays. TPE therefore receives a continuous len over 7 to 21 and a continuous lo over 20 to 30. The six grid points are six particular points inside that box, not the box itself.

What TPE actually does

harness::tpe is a clean-room implementation of Bergstra, Bardenet, Bengio and Kégl's Tree-structured Parzen Estimator (NeurIPS 2011, §4), restated for this workspace's maximize convention:

1. Keep the observation history {(x, score)}.
2. Split it at the gamma quantile of score — the top fraction is GOOD, the rest BAD.
3. Per parameter, independently, build two Parzen density estimates: l(x) from the good observations and g(x) from the bad ones.
4. Expected improvement is monotone in l(x)/g(x), so propose the x that maximises the ratio. For a continuous parameter that means drawing n_candidates samples from l and taking the argmax of l/g.

A continuous parameter uses an adaptive Parzen estimator — one Gaussian per observation plus a broad prior Gaussian at the domain midpoint, each kernel's bandwidth the maximum distance to its sorted neighbours, clipped so it can neither collapse to a spike nor exceed the domain width. The first n_startup asks are seeded-random, to give the model something to fit.

TpeConfig publishes the defaults: DEFAULT_TRIALS 64, DEFAULT_GAMMA 0.25, DEFAULT_STARTUP 10, DEFAULT_CANDIDATES 24. --trials and --seed (default 0) are the two the CLI exposes.

Running the TPE search

Same profile, same objective, twelve trials at seed 7:

backtest --profile tut_rsi_sweep.toml --optimizer tpe --trials 12 --seed 7

tpe: 12 trials (seed 7), best score 1.4950
sweep ranked by sharpe (12 points)
*#1  len=19.024470417556497 lo=26.62415622170128 ret=0.1986 sharpe=1.4950 max_dd=0.1083 trades=16 score=1.4950
#2   len=19.610649528496367 lo=25.829302930280782 ret=0.1582 sharpe=1.2327 max_dd=0.1058 trades=14 score=1.2327
#3   len=19.096107272110384 lo=25.4828741659996 ret=0.1582 sharpe=1.2327 max_dd=0.1058 trades=14 score=1.2327
#4   len=19.852274192045854 lo=28.71331759876744 ret=0.1536 sharpe=1.1804 max_dd=0.1102 trades=16 score=1.1804
#5   len=15.667688416361274 lo=27.573219822649627 ret=0.1213 sharpe=1.0531 max_dd=0.1313 trades=23 score=1.0531
#6   len=13.334186530160558 lo=22.49431522282743 ret=0.1224 sharpe=0.9732 max_dd=0.1434 trades=19 score=0.9732
#7   len=13.551342059120229 lo=23.28076739152503 ret=0.1155 sharpe=0.9328 max_dd=0.1441 trades=21 score=0.9328
#8   len=21.0 lo=27.077044141294163 ret=0.1059 sharpe=0.8671 max_dd=0.1285 trades=12 score=0.8671
#9   len=8.87961618331828 lo=24.131413974177793 ret=0.0763 sharpe=0.6776 max_dd=0.1019 trades=57 score=0.6776
#10  len=8.449839262830166 lo=29.598740765730916 ret=0.0504 sharpe=0.4758 max_dd=0.1085 trades=77 score=0.4758
#11  len=19.31459176678944 lo=23.263613015537427 ret=0.0410 sharpe=0.3962 max_dd=0.1278 trades=7 score=0.3962
#12  len=12.457616477477801 lo=20.167882945281562 ret=0.0307 sharpe=0.3180 max_dd=0.1497 trades=17 score=0.3180

The first line is the run summary. It goes to stderr, so a --json stdout stays a clean document.

Read the coordinates before you read the scores. Every row is a fresh real number rather than a grid value, which is the continuous box doing exactly what the section above said it would. The one round coordinate is len=21.0 on row #8, and that is the axis's upper bound.

Twelve trials is a small budget deliberately, and it is smaller than it looks. DEFAULT_STARTUP is 10, and TpeOptimizer::ask draws seeded-random coordinates until the history reaches that many. So ten of these twelve asks were random and exactly two were model-driven proposals. This run sits much closer to random search than to a TPE run at its intended budget.

Two coincidences in the table are worth a second look. Rows #2 and #3 sit at different coordinates and report identical ret, sharpe, max_dd and trades. So does row #5, against a point the euler run below reaches from the other direction.

A period has to become a whole number of bars somewhere, and a threshold only bites when an observation crosses it, so the continuous box contains flat plateaus. Some of the resolution TPE is paying for is resolution this strategy cannot use.

Reproducibility, and the bug that was in this contract

Every stochastic step draws from a seeded SplitMix64 owned by the optimiser, so the same seed over the same space and the same data produces the same proposal sequence. SplitMix64 rather than rand's StdRng is deliberate: StdRng's algorithm is not guaranteed stable across versions, while a ten-line generator is fixed forever.

Comparing against successive halving

--search euler evaluates the coarse grid once, then repeatedly halves the per-axis step and re-searches only the immediate neighbourhood of the current best point, clamped to each axis's range and deduped against everything already evaluated. After D halvings the effective resolution is that of a grid with 2^D·(n−1)+1 values per axis; euler pays at most N + D·(3^d − 1). For a two-axis 5 × 5 grid at depth 3 that is at most 49 evaluations against 1089 for the equivalent-resolution grid.

The same profile, halved twice:

backtest --profile tut_rsi_sweep.toml --search euler --euler-depth 2

euler: 16 evaluations at depth 2/2 (grid at the depth reached would be 45)
sweep ranked by sharpe (16 points)
*#1  len=14.0 lo=30.0               ret=0.2129 sharpe=1.7800 max_dd=0.0835 trades=33 score=1.7800
#2   len=17.5 lo=25.0               ret=0.1811 sharpe=1.4293 max_dd=0.1336 trades=16 score=1.4293
#3   len=10.5 lo=30.0               ret=0.1600 sharpe=1.3415 max_dd=0.0830 trades=57 score=1.3415
#4   len=12.25 lo=27.5              ret=0.1578 sharpe=1.3370 max_dd=0.0830 trades=36 score=1.3370
#5   len=12.25 lo=30.0              ret=0.1668 sharpe=1.3085 max_dd=0.0833 trades=43 score=1.3085
#6   len=14.0 lo=27.5               ret=0.1406 sharpe=1.2094 max_dd=0.1148 trades=27 score=1.2094
#7   len=21.0 lo=30.0               ret=0.1542 sharpe=1.1885 max_dd=0.1341 trades=15 score=1.1885
#8   len=14.0 lo=25.0               ret=0.1273 sharpe=1.0656 max_dd=0.1365 trades=21 score=1.0656
#9   len=15.75 lo=27.5              ret=0.1213 sharpe=1.0531 max_dd=0.1313 trades=23 score=1.0531
#10  len=17.5 lo=30.0               ret=0.0980 sharpe=0.8521 max_dd=0.1192 trades=21 score=0.8521
#11  len=15.75 lo=30.0              ret=0.0785 sharpe=0.7099 max_dd=0.0934 trades=26 score=0.7099
#12  len=14.0 lo=20.0               ret=0.0755 sharpe=0.6565 max_dd=0.1402 trades=12 score=0.6565
#13  len=10.5 lo=25.0               ret=0.0606 sharpe=0.5592 max_dd=0.0880 trades=38 score=0.5592
#14  len=7.0 lo=30.0                ret=0.0246 sharpe=0.2807 max_dd=0.0873 trades=103 score=0.2807
#15  len=7.0 lo=20.0                ret=-0.0242 sharpe=-0.1026 max_dd=0.1137 trades=50 score=-0.1026
#16  len=21.0 lo=20.0               ret=-0.0992 sharpe=-0.6544 max_dd=0.1738 trades=2 score=-0.6544

The budget line is those two formulas made concrete. Three values and two values per axis at depth 2 reach a grid of 9 × 5 = 45 points, the bound N + D·(3^d − 1) is 22, and the run spent 16.

EulerBudget's equivalent_grid is derived from the depth REACHED rather than the depth asked for, so that 45 is a comparison the run actually earned.

--euler-depth defaults to EulerConfig::DEFAULT_MAX_DEPTH (3) and is capped at 16. A halving that fails to improve the incumbent is NOT a stop condition — only a smaller step can reach the points between the incumbent and the ones just rejected — so the search runs its full budget and stops early only when a depth produces no unseen candidate at all. The EulerBudget line on stderr compares against the grid matching the depth ACTUALLY reached, so an early-stopped run cannot advertise a saving it did not buy.

The price is that euler is a LOCAL refinement: it can only descend into the basin its coarse grid already found. Keep the coarse grid wide and treat it as a refinement OF a grid, not a replacement for one. TPE has no such constraint, and correspondingly no guarantee that it visited the coarse grid at all.

Scoring is shared: --rank-by doubles as the objective for both searches, a row whose backtest succeeded carries a score while a failed one keeps None and serializes no score key at all, a failed point steers as NaN and is unrankable, and a NaN candidate never displaces a finite incumbent — though euler's first evaluated point becomes the incumbent whatever it scored, so an all-NaN search still refines around one. Note that none of the three search paths PERSISTS a run — only a single run writes manifest.json and report.json under user_data/runs/<run_id>/.

What the three runs actually say

Neither search beat the grid. TPE spent twice the grid's budget and came back roughly a third of a sharpe point worse. Euler spent nearly three times the budget and returned the grid's own answer — the same coordinates, the same 1.7800 — so its ten extra half-step evaluations found nothing better than the point it started from.

That is the result, and it should stay uncomfortable. A page that ran these three searches and reported an optimiser win would be reporting a run that did not happen.

Why TPE lost is not mysterious. The best point in this space IS a grid point, and TPE searches the continuous box between grid points, so it can approach len=14.0 lo=30.0 and never land on it. Its two nearest rows, #5 and #6, both score under 1.1.

Turn that around and you have the case FOR TPE: had the good point sat between grid points, the grid could not have reported it at all, and both searches could.

Four things these runs do NOT say:

• Not that TPE is worse than a grid. One seed, one two-axis space, one tape, one objective. Every proposal moves when the seed moves, which is the whole point of the reproducibility section above, so this is a single draw rather than a measurement.
• Not that TPE was given a fair budget. DEFAULT_TRIALS is 64. This ran 12, of which 10 were the random startup, so the model made two proposals.
• Not that euler wasted ten evaluations. A refinement that confirms the incumbent has told you the incumbent survives a finer sweep. That is information; it is just not a better number.
• Not that 1.7800 is an edge. It is the best of six in-sample numbers on one symbol, one interval and one window, picked by looking at all six. Taking the maximum of a set is how you buy a number that does not survive contact with new data.

The next step is therefore not a bigger search. It is holding that winner out of sample — see Sweep, then walk forward.

Where next

• Search, Sweeps and Objectives.
• Sweep, then walk forward — validate what a search found.
• Overfitting.