Databento
The paid equities-and-futures vendor — one schema, symbol and window from the historical HTTP API, decoded by pure fixture-tested parsers into bars, trades, quotes or the L2 book.
databento_backfill is the crate's adapter for Databento's historical HTTP API. Its module,
crates/vike-backfill/src/databento/mod.rs, describes the lane in one sentence: intraday
bars/trades/quotes/L2-book from hist.databento.com/v0/timeseries.get_range in the raw CSV
encoding, into the vike-data hist store. It is a data source rather than an execution venue, which
is why it lives in vike-backfill and not under crates/bridges/. The bin is built only under the
databento feature (crates/vike-backfill/Cargo.toml), so the invocation is
cargo run -p vike-backfill --features databento --bin databento_backfill.
What a run does
From crates/vike-backfill/src/bin/databento_backfill.rs's usage text:
databento_backfill --dataset GLBX.MDP3 --symbol ESZ4 --schema trades \
--start 2024-01-02T00:00:00 --end 2024-01-02T00:01:00 [--store DIR] [--venue NAME]--dataset is the Databento dataset code, --symbol the instrument within it, and --schema one
of trades, ohlcv-1m, ohlcv-1h, ohlcv-1d, mbp-1, mbp-10 — each mapping to a store series.
--start/--end are in Databento's own timestamp form. --store resolves through the shared
ladder, where $VIKE_HIST_STORE and a project declared by
$VIKE_SETTINGS_DIR both outrank <repo>/market_data/hist, and --venue is the label the series is
stored under, defaulting to databento.
The transport (crates/vike-backfill/src/databento/client.rs) builds the timeseries.get_range
URL, authenticates with HTTP Basic — the API key as username, empty password — and streams the CSV
body to a temporary file rather than buffering it, because tick and book ranges are large. The
ingest (crates/vike-backfill/src/databento/ingest.rs) drives the parsers into the store under the
commit key databento:{kind}:{symbol}:{start}-{end}, so re-running the same window is a store
no-op.
The parsers
crates/vike-backfill/src/databento/parse.rs is pure — no I/O, no clock, fixture-tested — and its
header carries the wire facts a reader of the stored rows needs: prices arrive as 1e-9 fixed-point
integers and are divided by PX_SCALE; timestamps are nanoseconds and are reduced to milliseconds;
the trade side is the aggressor, A meaning a sell aggressor (the buyer is the maker) and B a
buy aggressor. Columns are resolved by name through crates/vike-backfill/src/csvutil.rs's
Header, which the two vendor parsers share, so column-order drift between schema versions is
tolerated. csvutil also explains why no CSV crate was pulled: both vendors emit a header row and
only numeric or string fields with no embedded commas, so a plain split is sufficient.
Credentials
DATABENTO_API_KEY is read by the bin from the credential store — <project>/settings/secrets.env
— then the process environment, never argv (bin header). client.rs's header records the
boundary: the key arrives as a plain &str parameter, and the module reads neither the environment
nor the workspace .env. It used to own an api_key_from_env() while its Tardis sibling already
had the bin do the read; the two premium-vendor adapters are now symmetric. The live smoke for this
adapter, crates/vike-backfill/tests/databento_smoke.rs, is #[ignore]d and self-skipping — its
ignore reason reads "live: needs DATABENTO_API_KEY, spends a tiny amount".
Where it sits in the capability table
crates/vike-backfill/src/caps.rs's Databento row is NONE for every roster venue, and the comment
on it is the point rather than a gap: the vendor has zero crypto coverage (verified in the vendor
study), so it is an equities and futures answer, keyed off its own vendor-namespaced venue label
rather than a roster venue — which is why no roster row is true for it, and why the --venue
default is databento. A gap plan over a roster venue will therefore never offer Databento; you
reach it by naming the dataset yourself.
Overlapping windows are two commit keys
The key names {start}-{end}, so re-running the identical window is a no-op, but two runs over
windows that overlap are two keys, and the store compares keys, never row values
(crates/vike-backfill/src/vikedata/ingest.rs names this as the same contract
databento/ingest.rs has). Schedule non-overlapping windows rather than widening a trailing one;
where two writers already cover the same rows, the store's source-precedence policy on
the Data API page is the repair.
The store kinds a schema produces are documented under concepts/data; the comparable crypto-only vendor lane is Tardis.
Funding rates
Two collectors that look alike and write different things — the venue-wide market funding rate a strategy trades on, and one account's realized funding payments.
Tardis
The paid crypto vendor — per-day CSV datasets of trades, quotes and the incremental L2 book, one UTC day at a time, with bars derived by resampling the ingested trades.