The vike.io Data API

data.vike.io is one vendor reached two ways, and the two are different tools with different features, keys and store kinds. vikedata_backfill reads the /v1 REST API — cohort metrics and the hourly asset panel — behind the vikedata feature. vike_archive_backfill reads the Polymarket L2 Parquet archive behind the vike-archive feature. crates/vike-backfill/Cargo.toml explains why they were deliberately not folded into one feature: the archive lane pulls a ranged-HTTP Parquet reader and a worker pool that a cohort-metrics build would compile for nothing.

Route one: vikedata_backfill and the /v1 API

crates/vike-backfill/src/vikedata/mod.rs describes what it fetches: one (exchange, asset, axis, grading) ladder over one hourly window from GET {base}/{exchange}/coins/{asset}/cohort-metrics, paged by cursor — for each hour and each cohort label, the long-side and total notional the grading service reported — landing as kind=cohort rows through vike_data::CohortRecorder. The module builds no commit key of its own: crates/vike-data/src/cohort_rec.rs's CohortFetch is the one place cohort:{venue}:{asset}:{axis}:{grading}:{label_basis}:{start}-{end} is spelled.

The bin header (crates/vike-backfill/src/bin/vikedata_backfill.rs) defines three run shapes, and which one a timer may use:

vikedata_backfill --asset BTC --schedule --catch-up 3          # SCHEDULED — the only timer shape
vikedata_backfill --asset BTC --axis size --days 30            # periodic — MANUAL ONLY
vikedata_backfill --asset BTC --axis pnl --grading unrealized --start 2026-04-01 --end 2026-04-08

--axis is size, pnl or tier (default size); --grading is realized, realized-pit or unrealized and is refused off the pnl axis; --exchange defaults to hyperliquid; --api-base defaults to $VIKE_API_BASE, else https://data.vike.io/v1. --panel switches to the hourly asset panel (/v1/{exchange}/assets/hourly) and writes kind=perp_metrics rows — (ts, premium, open_interest) per hour — under perp_panel:{venue}:{asset}:{start}-{end}; the usage text names it the only producer that fills open_interest, and crates/vike-backfill/src/vikedata/panel.rs states why: the venue's own surface serves OI as a current snapshot and nothing else.

The guards are the value, and vikedata/mod.rs lists them as ported rules: label_basis must agree across every page and an absent one is a hard failure on the pnl axis (the endpoint's default resolves a wallet's cohort at query time, which is lookahead, and it shipped once); the grading echo must match what was asked; a (ts, cohort) seen twice across a page overlap is first-wins; the cursor walk has three stops, including a window break, because the endpoint keeps serving up to its own now regardless of the start sent; both notionals are snapped before anything is derived from them (measured: 430 of 8,190 rows differing at ~1.4 ULP between two GETs of a byte-identical URL); and the hour label is multiplied from seconds to milliseconds, because an unmultiplied row lands in the 1970 partition for good.

The header also says what is measured and what is not. Against the live endpoint on 2026-08-25, the pnl label_basis echo is real and the auth header is case-insensitive; a --axis size --days 1 run stored 300 rows. Still unmeasured: the pnl and tier axes end to end, and any cursor walk past one page.

VIKE_API_KEY comes from the credential store and is sent as an X-API-KEY header (crates/vike-backfill/src/vikedata/client.rs). Absent is a refusal, not a fallback: an unauthenticated fetch 403s part-way through a window and would store a partial batch under a key claiming the whole one, indistinguishable from a thin tape afterwards (bin header).

Route two: vike_archive_backfill and the Parquet archive

crates/vike-backfill/src/vike_archive.rs describes the archive: Polymarket full L2 (book_events), taker tape (trades) and derived top-of-book (l1_quotes), one UTC day per Hive-style partition at {base}/venue=polymarket/date=YYYY-MM-DD/{stream}.parquet, plus an open manifest.json needing no key. Auth is an X-API-Key header; the bin reads VIKE_ARCHIVE_API_KEY from <project>/settings/secrets.env and passes it in as a parameter. The sizes are the vendor's own: a book_events day runs ~7.4 GB, 465M rows, 453 row groups.

vike_archive_backfill --from YYYY-MM-DD --to YYYY-MM-DD [--kind book|trade|quote|all]
    [--tokens T1,T2,...] [--tokens-file PATH] [--store DIR] [--dry-run]
    [--asset NAME --tenor NAME] [--bulk ...] [--file PATH [--jobs N]]

The module's differentiator is that it does not need the whole file: HttpRangeReader implements the parquet crate's ChunkReader over ranged GETs, so the reader pulls the footer (~1.17 MB in two range reads) and then only the column chunks of row groups whose token_id min/max statistics bracket a requested token (select_row_groups). A filtered single-token scan costs one row group — tens of MB — rather than the day. Ingest streams one row group at a time and commits under vikearchive:{kind}:{date}:rg{idx}, a per-row-group key that makes a partial run resumable.

The bin header (crates/vike-backfill/src/bin/vike_archive_backfill.rs) then records where the time actually goes. A book_events day costs ~419 row groups × one HTTP round trip each through the CDN — hundreds of seconds of network latency, not compute. --file PATH is the measured fix: download the day file once, then ingest it locally through the identical decode/map/commit pipeline (LocalFileReader in place of HttpRangeReader), committing byte-identical rows under the same keys. --file needs an unambiguous target — --from equal to --to, and an explicit --kind — and it is mutually exclusive with --asset/--tenor, the pair that switches to the family layout (venue=polymarket/asset=<asset>/tenor=<tenor>/date=.../{stream}.parquet, resolved through the /v1/archive/datasets and /v1/archive/url discovery calls). --jobs N fans the local ingest across worker threads (default min(4, available_parallelism)), and --bulk switches to vike_data::BulkIngestSession's staged commit profile (defaults 20 row groups / 2M rows / 256 MB per window). The two compose.

Two siblings share this lane. events_api_backfill reads the live /v1/events endpoint under the same feature and key and is on EOD, events and exec trades. And crates/vike-data/src/archive_store.rs's ArchiveParquetHistStore skips the import entirely, answering HistStore reads straight off downloaded day files — the route the Polymarket page recommends for research.

When two writers cover one window

crates/vike-data/src/datafusion_hist/sources.rs is where the store keeps the rule this vendor makes necessary. Commit-key idempotency stops the same writer re-ingesting a window; it does nothing about two different writers — the recorder's live-… key and a backfill's — covering one window, and the store never dedups by row value, so a scan returns both copies. StoreSourcePolicy — the ranked prefix list plus its strict flag, which builds the call-time SourceRankPolicy — is persisted next to the data as _sources.json so DataFusionHist::run_maintenance can resolve duplicates on its own, and the header lists what makes a row-dropping pass safe on a timer: an absent policy file is byte-identical to today; an empty prefix list supersedes nothing; a series whose commit key matches no listed prefix is skipped, with the key named, unless strict: false is set; and every drop is reported through rows_superseded. It is not a claim about which source is better — only which one this operator decided wins.