vike-recorder

The headless live-tape recorder — one profile, a tick that re-resolves what to record, and a budgeted teardown that does not lose the flush.

The headless live-tape recorder. A trader with no warehouse behind them still needs to backtest on markets they have actually watched, so the terminal has to accumulate that tape for them. That is this binary (crates/vike-recorder/src/lib.rs).

usage: vike-recorder --profile PATH [--tick-secs N] [--silent-secs N] [--exit-on-silence] [--once]

  --profile PATH    the recorder profile TOML (store root + subscriptions)
  --tick-secs N     how often to re-resolve the desired symbol set (default 30)
  --silent-secs N   warn when a subscribed series has received no rows for N seconds
                    (default 300; 0 disables)
  --exit-on-silence stop with status 3 when a series is silent, so Restart=on-failure can act
  --once            run ONE tick and exit — the dry run
  -h, --help        print this and exit 0
  -V, --version     print the version and exit 0

One TOML profile names the store and what to record. Every tick resolves each subscription's current symbols, publishes their family membership, and drives the venue feed to exactly that set; rows land in your own DataFusion + Parquet store through RecorderSink, one commit per family rather than one per symbol.

Why it is its own daemon

Recording needs venue feeds, and those drag the signer, WebSocket and TLS crates into whatever links them. Keeping them out of vike-datahub means a panic in feed parsing cannot take serving down, and the server's dependency tree — audited on the same box that signs real orders — stays small. The accepted cost is that datahub cannot report recorder lag; if that ever matters the fix is an IPC surface, not co-hosting.

Not vike-desktop either: a tape that only exists while a desktop app is open has a hole wherever you closed it. And not vike-tradehub: recording must run whether or not you are trading, and a trading daemon restart must not punch a hole in the tape.

Everything venue-free — the subscription model, the family membership, the subscription driver and the daemon tick — is tested without a network, because the driver speaks the vike_data::live::DataClient trait and the runtime drives a VenueFeed. The concrete venue is the binary's choice.

Venue features

crates/vike-recorder/src/venues/mod.rs is the only venue-aware code in the crate, one module per recordable venue, each behind its own Cargo feature — features rather than plain deps because a bridge is heavy: enabling polymarket pulls the EIP-712 signer stack into the recorder binary, and a build recording only static-symbol venues should not link one. A venue module implements VenueFeed by assembling what the bridge crate already owns, never by reimplementing venue logic.

A feature-absent build errors instead of recording nothing

build_feed answers a profile naming a venue this build cannot serve with a startup error that names the Cargo feature to rebuild with. A recorder that silently skipped an unsupported venue would look healthy while recording nothing — the same failure the profile-level config validation already exists to prevent.

The binance feature records the trade tape only: the venue serves no quotes and no lossless book, and its L2 is the conflating depth lane, which RecorderSink does not persist (crates/vike-recorder/Cargo.toml's binance).

Stopping without losing the tape

One stop flag, many triggers: a control word on stdin (quit/shutdown/stop) raises it, and so do SIGTERM and SIGINT through vike_ops::stop::install_handlers, so systemctl stop, a bare kill and Ctrl-C all reach the same teardown the typed word does. EOF stops only on a TTY — under systemd stdin is /dev/null, which reads EOF immediately, so treating that as a stop would make the daemon exit on startup. On Windows there is no POSIX signal, but install_handlers bridges the console control events — Ctrl-C, Ctrl-Break, a window close — onto that same flag; only a recorder started detached, with no console to deliver them, is left with the stdio word. Until that handler existed, SIGTERM reached none of the teardown — and the flush is where the tape was lost, on every restart of every box.

The teardown is budgeted end to end, because systemd follows SIGTERM with SIGKILL after TimeoutStopSec= and a teardown that can outrun that budget is killed mid-write instead of finishing. There are two numbers, not one: FEED_STOP_BUDGET_SECS for the feed unsubscribe (measured, warned on, and subtracted from what follows, because a non-Send VenueFeed cannot move onto the orchestration thread) plus SHUTDOWN_DEADLINE_SECS for the bounded flush-and-compaction tail, summing to TOTAL_STOP_BUDGET_SECS, which is what the shipped unit's TimeoutStopSec= is checked against.

Teardown order is load-bearing

Unsubscribe every feed first, then shut the sink down. The other order leaves feed threads pushing rows into a channel whose writer is gone, and those rows are counted as dropped with no reporter left to announce them. The sequence is flush-first with the maintenance join last, so what a hard cap abandons is a compaction pass — restartable, costs nothing — and never the rows.

The two watchdogs

The silence watchdog catches a feed that is subscribed, connected and receiving nothing, which raises no error anywhere. It has three possible reactions and only two are on by default: a silent series is always logged and always alerted (crate::alerts::watchdog_tick, in the library so a test drives it; delivery is log-only until the [alerting] profile table names a target). Exiting is opt-in via --exit-on-silence, because a venue can be legitimately quiet — an illiquid market, a closed session, a window between rotations — and letting a quiet venue kill the daemon recording five healthy ones would be a worse failure, crash-looping under Restart=on-failure. The 300-second default was chosen against the slowest thing a healthy recorder does — a Polymarket up/down family rotates its tokens every five minutes and a freshly opened window can be genuinely quiet for a stretch, so a threshold at or below the rotation period would cry wolf every cycle. The exit status is EXIT_SILENT = 3, distinct from the generic failure 1 so a supervisor can tell "a series went silent" apart from "the profile was bad".

The resolve watchdog is the feed-level twin and is on by default: a venue that has never resolved a symbol since startup escalates, past the same --silent-secs grace, to an error plus a real alert, while a venue that resolved before and is failing now stays the per-tick warning it always was. The series watchdog is structurally blind to that state — a venue that resolved nothing subscribed nothing, so its expected-series set is empty — and a clean install once produced exactly it while the daemon ran on recording nothing.

--once is the dry run and the only reaction that exits on that judgement, with EXIT_DRY_RUN. It exits 0 only if every feed ended that tick with at least one live subscription and no refused subscribe; otherwise status 4, naming each feed that would record nothing. It is stricter than the daemon on purpose — it is a commissioning check of your own profile, with you watching.

vike-recorder

Why it is its own daemon

Venue features

Stopping without losing the tape

The two watchdogs

On this page