Run a sweep on the data service
Ship a profile to the box that holds the history, run the grid next to the data, and get back only the ranked report.
vike-datahub is a headless compute-to-data backtest server. A client sends a small profile
config over a localhost TCP socket; the server runs the existing vike-backtest engine right next to
the data and returns just the report. The heavy history never crosses the wire — only the profile in
and the metrics summary out (crates/vike-datahub/src/lib.rs). That is what makes a sweep on a box
holding a large recorded tape practical from a laptop.
Start the service
The server takes no options: the listen address and the store root come from the environment.
VIKE_DATAHUB_ADDR— the listen address, default127.0.0.1:7878.VIKE_DATAHUB_STORE— the history-store root. Unset, it falls through the shared store-root precedence; the old CWD-relative last resort is gone, and it was the whole bug — launching from anywhere but the repo root created an empty store beside the shell.
Only a build with the serve-datafusion feature can actually serve, because that is what makes the
concrete DataFusion backend nameable; a default build still produces a runnable binary that answers
--help and --version identically. That is not a detail: this binary once parsed no argv at all,
so under the feature vike-datahub --help started a server.
A non-loopback bind is refused, and one opt-in is not enough
Binding a non-loopback address is refused at startup unless VIKE_DATAHUB_ALLOW_PUBLIC_BIND is set
to the exact string "1" and at least one node key is configured. The opt-in alone used to be
enough, behind a warning, and crates/vike-datahub/src/datahub_cli.rs's header records why that
was not sufficient: the opt-in consents to being REACHABLE, and that is not consent to serve a
store write and a script compiler unauthenticated. The guard did not soften when authentication landed: the
handshake is plaintext and authenticates the connection, not each frame, so a tunnel is still what
supplies confidentiality and integrity. Reach a remote service with
ssh -L 7878:localhost:7878 <box>.
Authentication is opt-in and the credential is the switch — this workspace's absent-credentials
idiom pointed at a server. With neither VIKE_DATAHUB_OBSERVE_KEY nor VIKE_DATAHUB_CONTROL_KEY in
the credential store, the server authenticates nothing and behaves exactly as it did before
authentication existed: Hello is optional and informs rather than gates. With either set, Hello
becomes mandatory and must be the first frame, the Welcome answering it carries a fresh 32-byte
per-connection nonce, an Auth frame carrying the HMAC over that nonce must follow, and any verb
before AuthOk is refused and closes the connection. Observe reads history and the catalog;
Control additionally admits the backfill write and every Run* verb — those compile
client-supplied Rhai server-side, so they are not reads whatever they return.
crates/vike-datahub/src/server.rs's required_scope is the one authority for the split and its
match is exhaustive, with no _ arm, so a new verb cannot be added unclassified.
Write the [sweep] table
A sweep is an ordinary backtest profile with one extra table. Each key overrides
strategy.params.<key> across a value grid, and crates/vike-backtest/src/harness/sweep.rs's
expand_sweep takes the cartesian product, clearing each expanded point's own sweep field so every
point is itself a plain, runnable single-backtest profile.
[sweep]
rsi_period = [7, 14, 21]
threshold = [1.5, 2.0, 2.5]A profile with no [sweep] table, or an empty one, expands to exactly one point with no overrides.
Shipped to the service as a sweep, it comes back as a clean server-side error instead.
Ship it
vike-cli sweep --profile <run.toml> [--addr 127.0.0.1:7878]
[--rank-by sharpe|return|max_dd|equity] [--json]The profile's text is read locally and shipped verbatim; the server parses and validates it with
BacktestProfile::from_toml_str, expands the grid, and runs it. --rank-by defaults to sharpe;
the name is validated locally against a four-entry roster so a typo costs no round trip, and the
metric itself is computed and applied server-side by the canonical comparator, so rows arrive already
ordered best-first. That local list is a spelling check, not a second implementation.
Why the whole TOML crosses the wire
This command used to re-parse the profile into wire DTOs — a second profile parser and a second date
parser in the workspace — and that mapping could not carry [engine].fee, [engine.impact],
[engine.resolution], [risk], snap_to_properties, a tick slice or a cross-venue
[[data.series]] slice. A sweep therefore ran quietly under different costs than the same
profile's vike-cli backtest. Shipping the text leaves one profile parser in the workspace and makes
the sweep honour the whole profile; it also removed a local reimplementation of Sharpe, max drawdown
and total return, so every printed statistic is now the server's own report number
(crates/vike-backtest/src/harness/sweep.rs).
What the service does with the grid
Every expanded point is fully independent over one shared store handle, so points run on a rayon
pool. Determinism does not rest on scheduling: par_iter().map().collect() is order-preserving, so
rows come back in expansion order regardless of which point finished first, and the ranking sort that
follows is stable over that same input — the identical bytes the one-at-a-time loop produced, pinned
by parallel_and_sequential_sweeps_are_byte_identical.
Parallelism multiplies the data slice
The shared store handle shares the handle, not the loaded data: every point independently re-scans
and fully materializes its own slice, so peak RSS and store I/O are N_threads × slice, not
1 × slice. On a large recorded tick slice an uncapped one-worker-per-core sweep is an OOM shape,
and these sweeps run on boxes that also host live trading. So the points get a bounded pool of their
own rather than rayon's global one: it defaults to min(4, available_parallelism()), overridable
with VIKE_SWEEP_THREADS (if the OS refuses those threads the sweep warns and falls back to the
ambient pool — the same rows, without the bound). Raising that raises the multiplier — size it
against the slice, not the core count. VIKE_SWEEP_SEQUENTIAL=1 pins the whole process back to one
at a time.
The server is deliberately hard to wedge: each connection runs on its own thread, so a panic inside a handler drops that one socket rather than the accept loop; framing and decoding are split, so a well-framed body from a newer client that this build cannot decode is answered with an error and the loop continues; an idle or half-open peer hits a read timeout that always closes rather than resuming a desynced stream; and every request produces a response, so a client always learns the outcome.
Reading what comes back is read a backtest report. The expansion itself is sweeps; when a full grid is too expensive, the same profile feeds search and the pluggable objectives seam.
Replay a recorded session
Two different replays — a recorded market tape through the backtest engine, and a command journal through the real core against a state-hash fence.
Read a backtest report
Every row of the summary table, what the annualization factor actually is, and why a flat run prints a diagnosis instead of zeros.