Record a live feed

vike-recorder accumulates a live tape into your own history store, so you can backtest on markets you have actually watched without buying an archive. It is a separate binary on purpose (crates/vike-recorder/src/lib.rs): recording drags the signer, WebSocket and TLS stack into whatever links it, so keeping it out of vike-datahub means a panic in feed parsing cannot take serving down. Not the desktop app either — a tape that exists only while a window is open has a hole wherever you closed it — and not vike-tradehub, because a trading restart must not punch a hole in the tape.

The profile

One TOML file names the store and what to record. Every tick re-resolves each subscription's current symbols, publishes their family membership, and drives the venue feed to exactly that set; rows land in your DataFusion+Parquet store through RecorderSink, one commit per family rather than one per symbol (crates/vike-datahub/src/recorder.rs).

store = "market_data/hist"

[[subscribe]]
venue = "polymarket"
family = "btc-updown-5m"
backfill = "off"

A rolling family is recorded as one grouped series, which is why a 5-minute rotation costs one commit per flush instead of one per token. A subscription may name a family or symbols, never both. A misspelled key is refused by name, with the line number and the keys that would have been accepted — webhook for webhooks once left an operator believing a pager was armed.

Which venues this build can record

crates/vike-recorder/src/venues/mod.rs is the only venue-aware code in the crate, one module per recordable venue, each behind its own Cargo feature — enabling polymarket pulls an EIP-712 signer into the binary, and a build recording only static-symbol venues should not link one. A feature-absent build answers with a startup error naming the missing feature, never a silent no-record.

Families are general, not a Polymarket shape: Polymarket asks what is live right now because its token ids rotate every five minutes, while Binance resolves the same key as a glob (*USDT, BTC*, *USDT.P) over an instrument list that does not change from tick to tick. What a Binance recording contains is worth stating plainly — its declared capabilities are bars, trades and depth, with quotes: false and book: false, so the tape is the trade tape plus a conflated depth series, and the trade lane is the only path to that data at all (crates/vike-recorder/src/venues/binance.rs).

Prove the profile before you daemonize

--once runs a single tick and exits: it proves the profile parses, the store opens, and each family resolves to live symbols, without leaving a daemon running.

Two watchdogs, because one is blind to the other

A feed that is subscribed and connected but receiving nothing is invisible from every other vantage point: subscribe_* returned Ok, the socket is established, no error is raised, and the loss counters stay at zero because there are no rows to lose. A production recorder ran 95 minutes in exactly that state, writing nothing for binance while logging a clean startup and zero warnings; it was noticed by accident, days later, by reading row counts by hand (crates/vike-recorder/src/liveness.rs).

• SilenceWatch watches series — rows arriving on something already subscribed. --silent-secs defaults to 300 s, chosen against the slowest thing a healthy recorder does: a Polymarket up/down family rotates every five minutes, so a shorter threshold cries wolf every cycle. On by default, deliberately — the failure is silent by nature, so an operator who has to know to switch it on is exactly the operator who will not.
• ResolveWatch watches feeds — a venue that has never produced a symbol to subscribe at all. It exists because the first is structurally blind to it: a venue that resolved nothing subscribed nothing, so the series watch iterates an empty set and reports nothing at two seconds or at two days. A misconfigured Polymarket proxy recorded nothing for a whole run while every watchdog stayed green.

A silent series is always logged and always alerted. The third reaction, --exit-on-silence (EXIT_SILENT = 3, distinct from a startup failure so a supervisor can tell them apart), is opt-in: a venue can be legitimately quiet, and a quiet market must not kill a daemon recording five healthy ones.

The [alerting] table is optional and its absence means defaults, not off — with no webhooks the alert still fires and still reaches the log; naming a target escalates it to a pager. series_prefix scopes it by a prefix on the {kind}/{venue}/{symbol} key, because a Polymarket token id does not exist yet when the file is written. No credential goes in this file: webhooks names targets, and the token behind a name comes from the credential store or the process environment.

Maintenance, and the one memory knob

[maintenance] is likewise optional-with-defaults rather than optional-and-off. The recorder commits once per buffer flush, so a busy series writes a part every few seconds: 23 parts in 150 s for one family's book, about 57 KB each, roughly 13,000 files a day if nothing merges them. Compaction runs in its own thread and takes the same per-series lock the appends do, so a racing flush serializes and loses nothing.

max_merge_rows is the memory knob, and it is rows rather than bytes for a measured reason: a merge is materialized whole (its output is timestamp-sorted, and a sort cannot stream), and on a live Polymarket book series the parts were only 2.5× compressed, so a 64 MB budget peaked at 8.95 GB. Size it against your MemoryMax. retention_days omitted means keep forever — a recorder exists to accumulate tape, so deleting any of it is always an explicit choice — and 0 is rejected, because it would prune the tape as it is being written.

Stopping without losing the tape

One stop flag, many triggers: a control word on stdin (quit/shutdown/stop), and SIGTERM/SIGINT through vike_ops::stop's install_handlers, so systemctl stop, a bare kill and Ctrl-C all reach the same teardown. EOF stops only on a TTY — under systemd stdin is /dev/null, which reads EOF immediately. On Windows there is no signal, but install_handlers bridges the console control events onto the same flag — Ctrl-C, Ctrl-Break, window close. A detached recorder has neither a console nor a TTY stdin, and unlike vike-tradehub it watches for no SHUTDOWN file, so nothing short of taskkill reaches it — and that loses the buffered tape.

Teardown order is load-bearing: unsubscribe every feed first, then shut the sink down. The other order leaves feed threads pushing rows into a channel whose writer is gone, and those rows are counted as dropped with no reporter left to announce them. The sequence is budgeted end to end — FEED_STOP_BUDGET_SECS plus SHUTDOWN_DEADLINE_SECS, summing to TOTAL_STOP_BUDGET_SECS, which is what the shipped unit's TimeoutStopSec= is checked against — and ordered flush first with the maintenance join last, so what a hard cap abandons is a restartable compaction pass and never the rows. Until that handler existed SIGTERM reached none of it, and the flush is where the tape was lost.

Once there is tape, replay it; the store's layout and its per-kind contracts are in the history store and concepts/data. A long run also wants VIKE_LOG_FILE_LEVEL.