Store maintenance
Regrouping stray per-symbol tick series into their grouped family, seeding the benchmark store, and where most backfill bins decide their store root.
Two vike-backfill bins maintain a history store rather than fill it from a source. The supervisor that keeps the collectors themselves running is a separate binary, documented at collector-supervisor.
Where the store is
Most bins in the crate resolve their root through crates/vike-backfill/src/cli.rs's store_root, with one precedence: an explicit --store, then $VIKE_HIST_STORE, then the project's own <project>/market_data/hist when $VIKE_SETTINGS_DIR names that project outright, then the repo-root default <repo>/market_data/hist when that checkout exists on the box, then the <project>/market_data/hist found by walking up from the working directory, then the per-user vike-data directory. The resolution is logged — root and rung both — for a reason its doc states: a store does not merge. If the default moves, the old store is simply no longer read, and a backfill reports zero rows into a fresh empty one, indistinguishable from "the venue returned nothing" until somebody goes looking. $VIKE_SETTINGS_DIR is the override that pins the project, and the same resolver honours it.
migrate_to_group
migrate_to_group --venue NAME [--store DIR] [--group NAME] [--dry-run]Tick series have two layouts — one series per symbol, or one grouped series per family — and this bin folds stray per-symbol series into their family's grouped one. crates/vike-backfill/src/bin/migrate_to_group.rs names the two real causes of a store holding the same instrument in both layouts: a family recorded before grouping existed, and two recorder bugs, since fixed, that let rows escape (a max-rows flush that skipped the group resolver, and a rotation that unmapped a symbol while its rows were still buffered).
Such a store is not wrong: scan_* reads both layouts, so no row is lost or hidden. It is untidy in ways that cost later — the coverage report in the Data Manager lists one instrument twice, maintenance treats them as unrelated series, and every scan opens both.
It refuses when the target is ambiguous
The store cannot know which family a stray token belonged to — that knowledge lived in the recorder's in-memory membership map and was never persisted. So a fold happens only when the venue has exactly one grouped series for that kind. With none there is nothing to fold into; with several, picking one would write rows into the wrong family, which no later pass can untangle. --group NAME names the target explicitly for the several case.
crates/vike-backfill/src/regroup.rs's plan_regroup is the pure half: per kind it yields KindPlan::Fold, NoTarget or Ambiguous, and kinds with no strays are absent. GROUPABLE_KINDS is quote, trade, book and depth; bar has no grouped append and never folds. A forced --group overrides the target for every kind, including a kind that had no group at all, which is what lets a fold into a brand-new group name work.
The I/O is vike_data::DataFusionHist::migrate_series_to_group, and its order is deliberate: copy, verify, then delete. Rows are appended under a migrate:{kind}:{symbol} commit key, so re-running is a no-op rather than a double-append; the grouped read is checked to contain them; only then is the per-symbol source removed. A failure anywhere before that leaves both copies — untidy, which is what it already was, never missing. One series failing never aborts the rest; the run prints series planned, rows moved and series skipped. --dry-run prints the plan and writes nothing.
ingest_bench_bars
ingest_bench_barsThe seed for the two-engine benchmark, cargo bench -p vike-backtest --features bench-hist. It ingests a Python parquet bar cache into a self-contained bench store — the pure-Rust replacement for a retired Python exporter, with no venv, pandas or SQLite — and is idempotent by commit key (bench:{sym}:{month}), so a re-run is a no-op.
It takes no options; both paths are fixed at compile time. The source is <vike-trader-app>/storage/parquet/<SYMBOL>/1m/<MONTH>.parquet with columns ts,open,high,low,close,volume; the destination is <this checkout>/market_data/bench_hist, written as kind=bar, venue=binance, interval 1m, and derived from the crate directory at compile time so the ingest writes into its own checkout from any worktree rather than a hardcoded absolute path. SYMBOLS is the five-symbol canonical order (BTCUSDT, ETHUSDT, SOLUSDT, DOGEUSDT, ADAUSDT; BTCUSDT doubles as the single-symbol bench series) and MONTH is 2026-06.
The --help that ran the job
This bin once read no argv at all, so ingest_bench_bars --help opened a store, created a directory and wrote bars — the worst shape in the family, because --help is the safest-looking thing a person types first. The triage now runs before the log subscriber and before the store, and crates/vike-backfill/tests/help_cli.rs asserts both the exit status and that no store directory appeared. crates/vike-backfill/src/cli.rs's short_circuit generalises the rule in its own doc — call it first, before vike_log::init, before opening a store, before any network call, because answering a question about the command line must not create a log file, create a store directory, spawn a subprocess or ingest anything.
Both bins log through the crate's shared log_config, which caps the JSON file at warn; see Log volume.
EOD, events and exec trades
Three small, unrelated collectors on one page — daily index bars from Yahoo, the data.vike.io live events API, and a venue account's own trade-history export.
Log volume
Why a long backfill or an open desktop window can write gigabytes of JSON log, and the knobs — VIKE_LOG_FILE_LEVEL, the file-level preference, and retention — that bound it.