Strategy backtesting

Two engines, one result type

There are exactly two simulation APIs, and they return the same result type — the numbers are the same shape whichever produced them.

The event engine

The general one. It steps aligned per-symbol bar series timestamp by timestamp, or — in tick mode — merges recorded quote, trade and book events into one ordered stream, and hands each event to your strategy. Anything with a resting order, a stop, a bracket, a cash gate, a margin call, a position sizer or a maker quote runs here.

The vectorized kernel

Deliberately narrow and deliberately fast: the market-order, target-weight path with next-open fills, shared-cash application in column order, taker fees, slippage, funding and cost-basis averaging. If your strategy is a matrix of target weights and nothing else, this is the fast path. Stops, brackets, the cash gate, position caps, leverage capping, membership masks, liquidation, sub-bars and sizers are explicitly out of its scope.

They are not alternatives that may quietly drift apart. Engine-versus-kernel agreement is a merge gate: the two run over identical synthetic bars and must produce the same trade count and a bit-identical final equity. It is a second oracle that keeps working as strategies are added, because it compares two things that both run rather than one thing against a frozen file.

What the fill actually models

"Realistic" is not a specification. Here is the list, with what each one is for. Every item is a real setting on the run profile, and several are off by default because turning one on changes results and you should know when it did.

Where a fill price comes from

Bar model

No ticks available. Fill prices come from the same fill-trigger oracle the live conditional-order emulator uses, so a stop triggers in a backtest the way it triggers live.

L1 quote model

With quote bars, a market order crosses the real spread — buys at the ask, sells at the bid. Single-quote-tick bars trigger resting orders off the quote side.

L2 book walk

With a recorded book, a taker is priced by consuming the resting book: the volume-weighted price of your own size, best level outward. A resting limit caps the walk at its own limit price, so it can only be filled by liquidity it would genuinely have crossed. And when the displayed book cannot cover the size, the order does not fill — it stays resting, waiting for liquidity, rather than filling size the market never showed.

The book tier exists because the tiers above it price a taker at a price nobody was offering, and the error is one-directional and scales with size. On one measured prediction-market tape the trade prints averaged materially below the resting best ask at the same instant, and in the large majority of cases there was no resting ask at the printed price at all — roughly four cents per share on a strategy whose whole edge was under six. That is an engine defect, not one strategy's bug, which is why the fix is in the engine.

Costs

Adverse slippage

Multiplicative and always against you: buys fill up, sells fill down. The factor is floored so an extreme adverse fraction can never carry a price through zero and invert its sign, which the original arithmetic this was ported from could do.

Maker and taker fees

Separate rates, each falling back to a flat rate when unset. A fee schedule can override the whole chain where a flat rate cannot express the venue — the prediction-market curve that scales commission by the outcome probability is the shape that exists today.

Funding

The perpetual-futures funding cash flow on a held position: longs pay positive funding, shorts receive it. The stored funding series is window-joined onto the price bars, one rate per bar whose window contains the event — never forward-filled.

Market impact

Opt-in, bar mode only, and taker-only. The flat slippage number charges the same whether an order is one share or a large fraction of the day's volume; the impact model adds the size-dependent part, estimated from bar OHLCV with the published Almgren–Chriss cost function and its published exponents and coefficients — a permanent term that is horizon-independent plus a temporary term that sees the execution horizon through the participation rate. One term of the paper is deliberately omitted, with the reason recorded: it needs a per-issuer shares-outstanding series that does not exist at all for crypto symbols. A resting limit that was hit supplied the liquidity this cost pays for, so it is never charged impact.

Timing and queue

Queue position

Opt-in, on the tick and book replay path. Without it, a resting limit fills the instant price touches it — as if your order were always first in the queue, which over-fills makers. With it, a touch is queue-gated: a trade at your price first consumes the estimated size ahead of you, and only the excess beyond that fills your order. Recorded book updates drive the estimate, so cancellations ahead of you shorten the wait. Two estimators ship: a conservative bound that shrinks the queue only on hard evidence, and a probabilistic one that splits a net depth decrease between the orders ahead of you and behind you.

Order latency

Opt-in, on the tick path. Without it a strategy sees a tick and its order is already resting at the matching engine on that same tick, and a cancel issued on the tick that would have filled it always wins — impossible reflexes, and the largest remaining source of optimism after the queue model. Two legs are modelled: the entry leg, from your submit to matching-engine visibility, and the response leg, from an exchange-side event to your strategy seeing it. Either a constant, or interpolated from a recorded series of real round trips. A recorded rejection is encoded as a negative entry latency: the request never reached the matching engine, so the action is dropped and counted, and a calibrated series with a rejection burst cannot silently swallow a maker's orders and leave you unable to diagnose it.

Intrabar resolution

OHLC cannot say what happened first inside a bar. When several resting orders trigger on one bar the adverse ones — stops and trailing stops — are applied before the favourable ones, which is the pessimistic ordering. When more than one order reduces the position in the same bar, the total reduction is capped to the position size, so a take-profit cannot also fill after a stop already flattened you. The ambiguity is not swallowed: the result carries a counter of how many bars hit the cap, which is how you discover your bar interval is too coarse for your brackets.

Stale prices

Opt-in. A repeated last price is not a tradeable price: on a sparse feed a resampled series emits flat, forward-filled bars, and a market order that fills against one manufactures edge that never existed. With a staleness bound set, market orders wait for a fresh print — an event carrying real volume or a live two-sided quote. Price-conditional orders pass through untouched, because a repeated stale price cannot spuriously satisfy a condition it did not already satisfy, and gating a protective stop would strand live risk. Deferrals are counted, so "zero trades because the tape never printed" stays distinguishable from "zero trades because the strategy never traded".

Point-in-time instrument grid

Opt-in. Orders are rounded onto the tick and lot grid the venue actually published at that point in the history, rather than onto today's grid.

Parity is structural

Naming the modelled factors is the easy half. The harder claim is that the engine that computed them is the engine that trades — and that is a property of how the code is arranged, not a list of approximations.

The fill primitives — the fill-price tiers, the intrabar resolution, the three cost scalars, the staleness discipline — live in a crate below both the simulator and the live paper exchange, and both call into it. There is one definition of what a fill costs and when it happens, so "backtest equals paper" holds by construction rather than by two implementations happening to agree. A merge gate then proves it: the same strategy over the same bars, run through the backtest engine and through the live core in paper mode, must produce bit-identical fills — timestamp, side, quantity, price, fee and maker flag — plus identical realized profit, fees and final position, every float compared as a raw bit pattern.

Underneath that sit frozen golden fixtures. Floats are stored as IEEE-754 hex bit patterns and compared exactly, and the rule around them is the one that outlived everything else: never widen a tolerance to make a failing test pass — investigate the divergence. The full argument, and what happens when a session is replayed rather than re-simulated, is on replay and parity.

What you can backtest

A strategy resolves by name out of a registry shared with the live daemon, so the profile that was backtested is the profile that trades. The portable templates are buy_hold, grid, dca_accumulate, spread_maker, gueant_maker, trailing_scalper, momentum, funding_carry, funding_capture, pairs_zscore.

The simulator-only names are not a gap left open. Each one carries a written reason in the registry as data any consumer can read, so a daemon refusing one can say why rather than "unknown strategy" — most of them reach machinery the live stack genuinely does not have, such as the position sizer, the shared-cash gate's weight argument, the protective stop attached to an entry, the symbol universe or the bar-count schedule.

Beyond the templates you write your own, two ways. A Rhai script is the same code in a backtest and on the daemon by decision rather than accident, and its result is held byte-identical to a Rust twin by a third parity gate. A Rust strategy written against the portable broker trait runs unchanged on both stacks, and portability is enforced by compilation rather than by assertion: narrowing an implementation back to one concrete broker stops compiling.

One profile file

A backtest is described by one TOML file: a data slice, an engine cost and cash configuration, and a strategy by name plus its parameters. Unknown top-level keys are a hard parse error, so a misspelt section fails the profile rather than silently doing nothing.

[data]
venue    = "binance"
symbols  = ["BTCUSDT"]
kind     = "bar"
interval = "1h"
from     = "2026-01-01T00"
to       = "2026-02-01T00"

[engine]
cash     = 100000.0
fee_rate = 0.001
slippage = 0.0

[strategy]
name = "momentum"
[strategy.params]
size = 1.0

Two details are worth knowing before you write one. Knobs that only make sense in one mode are rejected rather than ignored when set in the wrong one — a market-impact table on a tick profile is a validation error, because silently ignoring it would leave you reading a result you believed was impact-adjusted. And the risk table carries the same schema and the same converter the live run profile uses, so a limit proven here by a real simulated denial carries unchanged into paper and live.

The same profile text runs a single backtest, a parameter sweep or a walk-forward, and the command-line and Studio paths ship the profile verbatim to a data service so the server honours the whole engine surface rather than a subset of it.

Parameter sweeps

Add a sweep table naming parameters and the values to try, and the profile expands into the cartesian product of single runs — each expanded point is itself a plain runnable profile, so nothing about a swept run differs from a run you launched by hand.

Rows are ranked by Sharpe, total return, maximum drawdown or final equity, or by a composite objective. A point that fails does not fail the sweep: its error is recorded on that row so the report still shows every point and which ones broke. A non-finite ranking key sorts last rather than first, so a Sharpe computed over a zero-variance curve cannot win a sweep and feed downstream selection.

Points run in parallel by default on a bounded pool, and the parallel and sequential paths are byte-identical — an order-preserving collect and a stable sort, both named as load-bearing in the source and pinned by a test. The pool is bounded rather than taking every core because each point materialises its own slice of data, so peak memory scales with the thread count and these sweeps run on boxes that also host live trading. Search strategies beyond the full grid — successive halving and a tree-structured optimizer — reuse the same expansion, evaluation and ranking code, so they cannot drift apart from the grid on what a point scores.

Walk-forward

A walk-forward run splits the series into successive windows, evaluates the strategy on each window's out-of-sample range, and stitches those curves into one equity path. It reports each window's out-of-sample return, the stitched curve, the overall out-of-sample return and Sharpe, and a consistency figure: the fraction of windows that were profitable out of sample.

Nothing is re-fitted. Each split carries a training range as well as a test range and the runner never reads the training half — what walks forward is one fixed parameter set evaluated on successive unseen windows. To have parameters chosen on the earlier data, choose them with a sweep first and walk the winner forward. The mode is anchored — the training window expands rather than rolling — and the annualisation is derived exactly as the Studio derives it, so a profile and the GUI cannot disagree about what "walk-forward" means.

The overfitting audit

A Sharpe ratio is a number; whether it means anything is a different question. The analytics crate answers it three ways, each accounting for something the raw figure ignores.

Probabilistic Sharpe ratio

The probability that the true Sharpe exceeds a benchmark, given the observed per-period Sharpe, the length of the track record, and the skew and kurtosis of the returns. Against a benchmark of zero it answers the single-run question: is this edge distinguishable from nothing.

Deflated Sharpe ratio

The same test, benchmarked not against zero but against the maximum Sharpe you would expect from that many trials with that much spread between them — which is what a search produces from pure noise. It can also use an effective trial count derived from the pairwise correlation of the trials' return series, so highly correlated trials collapse toward one and uncorrelated ones stay near the full count.

Probability of backtest overfitting

Combinatorially symmetric cross-validation over the full trial matrix: for every way of choosing half the observation groups as in-sample, find the in-sample winner and rank it out of sample. The reported figure is the share of paths on which the in-sample winner landed at or below the out-of-sample median. An uncomputable input returns not-a-number rather than zero, because zero would read as "no overfit".

Those fold, with the walk-forward consistency, into a plain-language verdict with its reasons listed. And a selection audit exists for the mistake that defeats all three: hand the deflated Sharpe only the survivor of a search and you read a reassuring number, because a single trial has no trial variance and gets benchmarked against zero. The audit takes the full candidate set and the selected index precisely so a survivors-only call is visible — it reports a candidate count of one and says so in the verdict.

Alongside them, Monte Carlo resampling characterises how much of a result was the order the trades happened to land in: shuffling preserves the exact set of trade outcomes and varies only the path and therefore the drawdown, while bootstrapping samples with replacement so terminal equity varies too. Both report percentile bands across the paths and a risk-of-ruin figure.

More of what it does