The live data seam
DataClient and LiveDataSink — the live mirror of the history store that every venue feed implements, where subscribing returns nothing and every datum arrives through a sink chosen at construction.
crates/vike-data/src/live.rs is the live twin of the history store:
DataClient is what a venue feed implements to start and stop streams, and LiveDataSink is where
everything a stream produces lands. The contract carried over from an earlier prototype is
fire-and-forget. subscribe_bars, subscribe_quotes, subscribe_trades, subscribe_book and
subscribe_depth start a stream and return a SubscriptionId immediately; they never carry data.
All data returns through the one sink handed to the client at construction — the trait methods take
no sink parameter, and there is no subscribe-with-a-callback shape. An Err from a subscribe call
means the stream could not be started; a disconnect, a reconnect or a resubscribe after a drop
surfaces on the feed's own status path, never through this trait. Reconnect is the feed's concern,
not the seam's.
Capabilities, not obligations
A client that cannot serve a verb returns LiveDataError::Unsupported rather than pretending to
subscribe. require_live_verb drives that refusal off the venue's declared VenueCaps.live_data
row, so a feed that calls it first in every subscribe_* verb cannot drift from the capability
matrix, and an unknown venue string refuses every verb. Most bridge feeds do call it; a few still
hand-roll their own decision — hyperliquid's snapshot-only subscribe_book and polymarket's
subscribe_bars are the ones left — and routing those through the seam is the noted follow-up.
subscribe_depth — the DOM's conflating snapshot lane, distinct from the lossless tick-lane
book — is Unsupported by default and only venues with a live depth feed override it.
Two lanes, and which price verb a feed must use
Sink methods are called from feed worker threads on every closed bar, intrabar update and tick, so
they must be cheap and effectively non-blocking — a slow sink stalls the feed. Downstream is a
lossless bar lane, seed_bars and close_bar, where a missed close is a series hole, plus a
latest-wins conflating lane, forming_bar, bar_close_tick and mark_tick, where drops are fine
and only the newest snapshot matters.
The price verbs are not interchangeable, because each fills a different rung of the
price board's resolver chain, and that chain is strictly ordered:
mark above side-aware bid/ask above last_trade above bar_close. A feed choosing the wrong
verb does not mislabel a number; it re-ranks it against every other price for the symbol. The
mapping is fixed:
| verb | what may call it | rung |
|---|---|---|
mark_tick | only a venue-published mark or index stream — binance-family @markPrice@1s, bybit tickers.markPrice, okx mark-price, hyperliquid activeAssetCtx.markPx | mark |
quote | a BBO or two-sided book top | bid / ask |
trade | a real print | last trade |
bar_close_tick | a kline feed's candle close, from every venue that serves klines | bar close — the lowest rung, up to one full interval stale |
Mark streams are default on for perps; aster's is default off, its grammar unverified, and opts in
with VIKE_MARK_STREAMS_ASTER=1; VIKE_MARK_STREAMS=0 is the master kill. The classes overlap
freely: a venue with no mark stream still fills the quote and trade rungs whenever those
subscriptions are open, so such a symbol is valued at its live quote or last trade rather than its
candle close — the intended law.
The L2 verbs split the same way. book carries folded state behind an Arc — a pump keeping one
standing book hands out clones and mutates through Arc::make_mut, so an applied delta no longer
deep-clones two whole maps per emission — while book_update is the recordable twin carrying the
raw delta, snapshot anchor or status marker that only recording sinks opt into; today the polymarket
pump in book mode is the one feed that emits it. l2_snapshot carries the instrument's tick_size
so a book-consuming sink quantizes on the grid the feed used rather than re-inferring it from a
truncated level set.
Stream health
stream_status is a default no-op that a feed's idle watchdog and reconnect path fire. GapStart
says the stream can no longer be trusted from a timestamp: for tick lanes the data in the gap is
missing and unrecoverable, for the forming-bar lane it is stale, and the lossless bar lane is
repaired by the re-seed before Live fires. Stale is the case a socket-liveness watchdog cannot
see — the transport is alive, keepalives arrive, and no fresh data has come inside the
subscription's freshness threshold. Both can be true at once, and only the consumer can decide what
staleness means: flatten, re-arm, widen.
Teardown costs one timeout, not the sum of them
A feed thread learns it should stop on its next socket read timeout, 2 s on every on-driver venue.
DataClient::begin_shutdown raises every stop flag and joins nothing, so a caller holding several
clients runs that phase across all of them before any join; shutdown then joins. vike-recorder's
teardown used to stop one stream at a time, outside its bounded region, and its total stop time was
on a collision course with the unit's TimeoutStopSec= as a profile grew — a binance subscription
is two threads per symbol. FeedRegistry is the shared per-subscription registry behind that: one
stop flag and one JoinHandle per stream, nothing ever detached, with an optional hook that runs on
each spawned thread — the seam a venue threads its core-affinity pin through without this crate
depending on vike-exec. Bybit, binance, okx, hyperliquid and polymarket use it, and so do
deribit, ig and oanda.
TeeSink fans one feed's sink slot out to several destinations in order — the core ingest adapter
plus a tick-recording sink is the shipped composition — and must forward every verb explicitly:
inheriting a trait default for bar_close_tick, l2_snapshot or book_update would silently
drop candle-close ticks, depth snapshots or recordable book events. On the consuming side, the
desktop app's CoreSinkAdapter bridges the sink onto the core's bar and market
ingest lanes, and the RecorderSink writer actor behind the
hist-datafusion feature buffers live quotes, trades and book events into the store — the path
recording a feed rides.
Startup preflight
A pure go/no-go gate that runs once before the first live order — clock skew, credential validity, disk headroom and network — with per-venue thresholds derived from measurement, a bound on every leg, and a demotion to paper that is enforced.
The node protocol
How the desktop app and the CLI talk to a running daemon — length-prefixed JSON frames, an HMAC nonce-challenge handshake with capability bound into the signature, and a wire vocabulary that mirrors the core without depending on it.