Data providers

Every backfilled series in the store was written by a collector, and every collector lives in crates/vike-backfill. Its crates/vike-backfill/src/lib.rs header states the contract: venue history fetchers live in the per-venue bridge crates, and what vike-backfill adds on top is the historical backfill path — one-shot paged REST or .bi5 pulls plus ingest into the vike-data hist store, with commit-window bookkeeping — kept as its own crate so that machinery does not bloat every bridge's live-feed code. Every collector is a one-shot binary an operator runs; the long-running counterpart, collector_supervisor, dispatches the same backfill_* functions and never a second implementation (crates/vike-backfill/src/supervisor/mod.rs's contract). It has its own page at collector-supervisor.

Two facts shape everything below. Every bin here is a composition root — one of the few places allowed to read the process environment and the credential store directly; that is the shape crates/vike-ops/src/settings.rs's Layer::Binary calls "the correct place to read process env", and it is why each family's page names the variable its bin reads. And the crate is excluded from CI's derived roster — xtask/src/ci/tables.rs's EXCLUDE_FROM_CI carries it with the reason, that it "only builds/tests under features" — so its families are compiled by the union backfill feature lane rather than the default build.

The families

The feature column is not decoration. crates/vike-backfill/Cargo.toml's venue-backfill note explains why the venue collectors are off by default: every bin links the whole library, so while the bridge crates were unconditional dependencies, all twenty-two bins compiled all of them — including the fourteen that touch no venue at all. Each bin that needs a bridge now says so through required-features. What cannot be gated is vike-data/hist-datafusion: every collector ingests into the store, so DataFusion is the compile cost you always pay.

What a source can and cannot serve

crates/vike-backfill/src/caps.rs is the capability table, and its header opens with the finding it exists to make visible: no venue-direct backfill in this workspace serves trade or book at all. Every *_backfill bin that talks to a venue writes kind=bar, except Dukascopy, which writes kind=quote from .bi5 ticks and resamples bars from them. Trades and the L2 book come only from archives and paid vendors — pmxt, data.vike.io, a local ClickHouse capture, Databento, Tardis. That is exactly inverted from what "backfill from the venue" sounds like, and the consequence is stated in the same header: the recorder writes trade/quote/book, so a hole in a recorded tape generally cannot be repaired from the venue.

The table's Source enum names the five places a fill can come from, each with a one-line verdict in its doc comment: Venue (the venue's own API — free, and only as complete as the venue exposes), Archive (the data.vike.io / pmxt archives — paid per GB or keyless CC-BY, complete L2 plus trades), ClickHouse (a local capture, prod-side), and the two paid vendors Databento and Tardis. backfill_caps(source, venue) answers per pair; sources_for(venue, kind) is the Data Manager's answer to a gap — which sources are even worth offering — and is_unfillable is the honest empty result. Polymarket is the sharpest case and was verified rather than assumed: the bridge implements no price-history endpoint, so its venue row is empty across every kind.

plan_fill is day-granular off the date=-partitioned file index, so a day the recorder covered for six hours reads as present and is never planned; within-day trust is a different question, answered by vike_data::quality::DayQuality over scanned rows (gapfill.rs's header).

What every bin shares

crates/vike-backfill/src/cli.rs is the harness the *_backfill bins used to re-implement, and its store_root is the one resolution law for every bin that has a default to resolve — the five klines bins and dukascopy_backfill take the root as a required argument, and ingest_bench_bars resolves its own market_data/bench_hist on purpose. The ladder: an explicit --store, then $VIKE_HIST_STORE, then <project>/market_data/hist when $VIKE_SETTINGS_DIR names that project outright, then <repo>/market_data/hist when the box still has that checkout, then the project the working directory sits in, then the per-user …/vike-data directory. The header records why it had to become one function: the four "verbatim" bins honoured the environment variable while pmxt/databento/tardis silently ignored it, so one operator variable routed half the collectors to one store and half to another. The resolved root and the rung that produced it are logged, because a store does not merge — if the default moves, the old store is simply no longer read and a backfill reports zero rows into a fresh empty one.

Three more shared rules, each cited to cli.rs. short_circuit runs first: --help and --version are answered on stdout with exit 0 before logging is initialised, before a store is opened, before any network call — one bin's --help used to perform the entire ingest. log_config sets every batch bin's file-log level to warn rather than vike-log's trace default, and the measurement behind it is in the same doc: one hour of pmxt_backfill wrote 6.4 GB, and a long range once wrote 341 GB. VIKE_LOG_FILE_LEVEL still wins over that default; log volume carries the rule for any long run. And every value flag refuses a token beginning with -- as its value (is_flag_token), so a mistyped command names the flag that went unfed rather than swallowing the next one.

Reaching a collector without a terminal

Two crates depend back on vike-backfill through optional features (lib.rs header): the desktop app — which took it under the deleted fat feature and calls nothing directly any more, its bulk Backfill action now taking only backfill_route's WIRE arm against the datahub's store — and vike-datahub's backfill-serve feature for backfill-on-demand. Backfill from the GUI covers the first; the store layouts every family writes into are on the hist store and, kind by kind, under concepts/data.