Record a feed, then replay it

Accumulate your own market tape with the headless recorder, stop it without losing the buffer, and replay the result deterministically into a backtest.

If you want to backtest a market you have actually watched, you have to record it first. crates/vike-recorder/src/lib.rs states the reason plainly: a customer has no production box and no warehouse, the recorders behind the data product share no code with the terminal, so the terminal has to accumulate the tape itself.

It is its own binary on purpose. Recording needs venue feeds, and those drag a signer, a WebSocket stack and TLS into whatever links them — keeping them out of the backtest server means a panic in feed parsing cannot take serving down. Not the desktop app, because a tape that exists only while a window is open has a hole wherever you closed it. Not the trading daemon, because recording must run whether or not you are trading, and a daemon restart must not punch a hole in the tape.

The profile

One reviewable TOML names the store and what to record. From the shipped crates/vike-recorder/recorder.example.toml:

store = "market_data/hist"

[[subscribe]]
venue = "polymarket"
family = "btc-updown-5m"
backfill = "off"

[maintenance]
interval_secs = 300
min_parts = 4

A family is general across venues, not a Polymarket-shaped concept: Polymarket resolves one to live tokens over its Gamma API because its token ids rotate every five minutes, while a static venue resolves the same key as a glob filter over its instrument list. Explicit symbols stay per-symbol with no grouping — grouping only pays across a wide subscription. A subscription names one or the other, never both.

Two things about that store line

The directory name hist is not a free choice: it is the name every binary's own default resolves to (vike_model::state_path's HIST_SUBDIR under PROJECT_DATA_DIR). A store called anything else is one the code cannot name, and a reader run without --store opens <project>/market_data/hist, finds nothing, and reports zero rows rather than failing. And the relative path resolves against the process working directory — right for a daemon whose unit sets WorkingDirectory, wrong for every command you run by hand. Measured: a hand-run single-tick run from a source checkout wrote its tape into the checkout while the project's own market_data/ stayed empty.

Every struct in this file is deny_unknown_fields, so a misspelled key is refused by name, with the line number and the keys that would have been accepted. That is a refusal rather than a nicety: webhook for webhooks left an operator believing a pager was armed, a misspelled retention_days kept the tape forever after they asked for thirty days, and backfil silently fell back to venue-REST backfill — which on Polymarket cannot restore the book at all.

Building and dry-running it

Venue support is a Cargo feature, because a bridge is heavy: a build recording only static-symbol venues should not link an EIP-712 signer. A venue this build has no feature for is a startup error naming the feature to rebuild with, never a silent no-record. Build vike-recorder with --features polymarket (or binance), then run it with --profile <file> and --once.

--once runs a single tick and exits: it proves the profile parses, the store opens and each family resolves to live symbols, without leaving a daemon running. Read its exit code, not its log — 0 only if every feed ended that tick with a live subscription, 4 otherwise, naming each feed that would record nothing. It exited 0 on a total resolve failure until 2026-08-08.

One capability note before you plan a study around it: Binance's declared caps are bars, trades and depth — quotes: false, book: false — so a Binance recording is the trade tape plus the conflating depth lane under its own kind=depth series. A lossless Binance L2 book is not obtainable in this workspace by any means, and depth is deliberately not a rename of one.

Stopping it without losing the tape

One stop flag, many triggers: a control word on stdin (quit / shutdown / stop / exit), and SIGTERM and SIGINT through vike_ops::stop's install_handlers, so a service-manager stop, a bare kill and Ctrl-C all reach the same teardown.

Until that handler existed SIGTERM reached none of it — and the flush is where the tape was lost, on every restart of every box. On Windows there is no POSIX signal, so the same install_handlers call bridges the console control events instead — Ctrl-C, Ctrl-Break and a window close all raise the same flag, and a recorder running in a console stops as gracefully there as a service stop does on Linux. The caveat is that a console event needs a console: a recorder started detached has neither one nor a stdin anybody can type into, and the stop-file trigger vike-ops also offers is armed by vike-tradehub alone — so on that path taskkill is still a kill, not a stop.

Four words, not three: crates/vike-datahub/src/recorder.rs's control_loop matches exit alongside the other three, while that file's module doc and its reply to an unrecognised word both list only three. Follow the code — exit works.

Teardown order is load-bearing: unsubscribe every feed first, then shut the sink down. The other order leaves feed threads pushing rows into a channel whose writer is gone, and those rows are counted as dropped with no reporter left to announce them. The whole sequence is budgeted end to end, because a service manager follows SIGTERM with SIGKILL — a feed-stop budget plus a bounded flush tail, summing to what the shipped unit's TimeoutStopSec is checked against, ordered flush-first so a hard cap abandons a compaction pass and never the rows.

[maintenance] is optional and its absence means defaults, not off: the recorder commits once per buffer flush, and a busy series was measured writing 23 parts in 150 seconds for one family's book at about 57 KB each — roughly 13,000 files a day if nothing merges them, and every later scan pays for all of them.

What lands, and how

Rows arrive through RecorderSink (crates/vike-data/src/live_rec.rs), which never blocks a feed thread — it only ever try_sends a row and bumps a dropped counter on a full or disconnected channel — and does every store write on one dedicated writer thread that owns the store handle. Book recording is the raw book_update lane; the folded-state book verb is a documented no-op, because a folded L2Book is not serialisable and carries no delta or replay information. Buffers flush on row count, on age, on a UTC-date rollover, or at shutdown, so a buffer never straddles a day and never outlives the process.

The silence watchdog is a pair: one module decides which subscribed series stopped receiving rows, another decides who is told. Both are always on — a silent series is logged and alerted — while exiting on silence is opt-in, because a venue can be legitimately quiet.

Replaying it

Point a tick run profile at that store root, passing --store <project>/market_data/hist to the backtest bin. That flag falls back to $VIKE_HIST_STORE, then to the <project>/market_data/hist of a project $VIKE_SETTINGS_DIR declares outright, then to <repo>/market_data/hist while the checkout that built the binary still exists, then to the project the working directory sits in, and finally to a per-user data directory — the same precedence the collector bins resolve through, and the root and the rung that chose it are logged.

The replay merges each symbol's quotes, trades and book events into one timestamp-ordered stream with the tie-break Book, then Quote, then Trade — book first because a live derived quote follows the book application that produced it — and a symbol with zero rows of all three kinds is skipped with a warning rather than failing the run, so an all-sparse window degrades to an empty but valid result. The streaming loader is pinned bit-identical to the bounded one by streaming_equals_bounded_bit_eq, so which of the two you reach for changes memory shape and never numbers.

Next: record a live feed and replay a recorded session for the task versions, tick replay for the loader, and tick replay with a queue model for what having the book on disk buys you.