collector_supervisor

Every collector in data providers is a one-shot process: you invoke it, it fetches a range, it exits. collector_supervisor is the long-running counterpart — a declared roster of series, a cadence per source, and a per-pass gap heal, running forever.

Its module doc states what was missing before it existed. Gap detection already existed (vike_data::DataFusionHist's series_gaps derives a series' missing UTC days from its manifest date= index, with no Parquet scan) and a coverage dashboard already existed (the Data Manager). What did not exist was anything that runs on its own: the only in-repo scheduler was store maintenance — compaction and retention, not collection. crates/vike-backfill/src/supervisor/mod.rs is that missing third leg; the binary is crates/vike-backfill/src/bin/collector_supervisor.rs, behind the venue-backfill feature.

It reimplements no collector

The first clause of the module's stated contract: every dispatch goes through the same backfill_* function the matching one-shot bin calls (crates/vike-backfill/src/supervisor/run.rs), so idempotency, paging and the still-forming-candle guard are inherited verbatim — a supervised backfill and a hand-run one are the same code path.

Wired today are the three keyless crypto kline collectors — binance, bybit and okx — the only library entry points that already share one signature and need no credentials. That is the same trio the Data Manager's bulk-backfill button offers; the supervisor is the unattended twin of that button. A collector with a different arity needs a small adapter first, deliberately left out rather than papered over with a lossy default mapping.

Why the freshness window is anchored, not rolling

So heal.rs's fresh_window starts at the series' stored watermark — max(now - lookback, last_ts + step) — and lookback_ms is only the outer bound for a cold or long-stalled series. A healthy series fetches the sliver since its last stored bar and nothing else; when the watermark is already at or past "now", no freshness job is planned at all. The planner, not the store, is what keeps the supervisor from duplicating its own data.

Two more invariants live in the same module. Heal work rotates. A gap the venue genuinely has no data for can never close: the fetch returns zero rows, commit_rows returns before recording the key, and series_gaps re-reports the identical hole forever. Taking the oldest N ranges every pass would re-attempt those same N holes and never reach hole N+1, so plan_heal_jobs walks the gap set round-robin from a rotating cursor. And max_heal_jobs bounds work, not range count: consecutive missing days collapse into one range, so a 300-day hole is a single range spanning hundreds of paged REST calls. Every range is split into HEAL_CHUNK_MS sub-windows — seven days — before the cap applies, which is what keeps a stop prompt.

Order is load-bearing: freshness first, heal after. An always-on collector's primary duty is that "now" is present; healing 2019 must never delay today's bars.

Failures, and the two lanes

Nothing inside a pass can end the loop. A series_gaps failure degrades that symbol to freshness-only, a backfill failure is recorded and the next symbol runs, and a status-write failure is logged and ignored — in the resident loop; under --once that write error is propagated and the run exits 2. Only the failure count changes behaviour, through crates/vike-backfill/src/supervisor/schedule.rs's effective_interval_ms — a bounded exponential backoff of cadence * 2^failures, capped at max_backoff_ms — plus the optional is_parked hard stop after max_consecutive_failures. A recovered source snaps straight back to its cadence.

Only the freshness lane feeds that. crates/vike-backfill/src/supervisor/run.rs explains why: a source whose freshness fetch is healthy but whose historical heal window returns 429s must keep collecting "now" at its configured cadence, or healing 2019 pushes today's bars out to the one-hour backoff cap — exactly the priority inversion the ordering above exists to prevent. Heal-lane failures increment a separate heal_failures counter that is published and changes no schedule.

The loop runs on the shared vike_data::worker::Worker harness — the same one the store's maintenance scheduler uses: a condvar-interruptible inter-pass sleep, stop() as flag-then-join, and a Drop that mirrors it. Sources are visited sequentially and a pass is never re-entered, so a supervised backfill never races another; racing an unrelated live append is safe because every store write serialises on the per-series manifest lock.

Running it

collector_supervisor --config FILE.toml [--store DIR] [--status FILE] [--once]

--config is required and is a TOML roster of [[source]] tables — name, collector, symbols, kind, interval, cadence_secs, lookback_ms, heal, max_heal_jobs, max_backoff_secs, max_consecutive_failures. It is parsed with deny_unknown_fields, so a typo'd key is a hard error rather than a silently ignored setting. There is deliberately no per-source venue override: each collector writes under its own hard-coded venue, so an override would make the gap lookup and the ingest target disagree.

--store resolves through the shared ladder in crates/vike-backfill/src/cli.rs's store_root, with one wrinkle worth knowing: the config's store = line is folded into that function's explicit argument before the ladder is entered, so it outranks $VIKE_HIST_STORE rather than sitting below it. Beneath the flag and that line the rungs are $VIKE_HIST_STORE, a project named outright by $VIKE_SETTINGS_DIR, the repo-root <repo>/market_data/hist on a box that still has that checkout, the project's own <project>/market_data/hist, and finally the per-user data directory.

--status publishes a JSON document carrying last-run / next-run / heal-queue / last-error per source, rewritten after every pass by serialise → write .tmp → rename, so a reader always sees a complete document. It is deliberately a plain file rather than a socket: no new transport, no port to secure, and cat/jq can read it while the loop runs. A status write failure never stops the resident loop — it is logged and collecting goes on; under --once it is fatal and exits 2.

--once runs a single pass and exits, for a cron or timer deployment. The resident mode stops on EOF on stdin or the word quit, finishing the in-flight pass first; nothing is lost either way, because every store write is already committed and idempotent by the time a pass returns.

Set VIKE_LOG_FILE_LEVEL for any long run — see log volume.