Downloading

There is one rule for the bulk archive:

Why, exactly

Every request to a gated /archive/ path is authorised on its own: the credential is validated and one usage row is written under the archive_download label. That happens per HTTP request. The hourly counter is not checked, so no number of archive requests can rate-limit you; the row is there so your usage view shows what you downloaded.

A Parquet reader working over HTTP fetches the footer, reads the metadata, then fetches each column chunk of each row group it needs as its own Range: request. Columns multiply by row groups, and a large partition has many of both.

Measured on the live archive: one 1.26 GB partition, read remotely, produced 20,237 requests — more than an entire hour's ceiling, for a single file. Before 2026-09-02 every one of them counted against that ceiling, and of 573 rate-limit rejections in the platform's history, 572 were archive downloads. That is why the archive stopped spending the budget. It is not a reason to read remotely: each of those requests is still a round trip.

The contrast is the whole of this page:

A complete pull of every date and every stream in the archive is roughly 120 whole-file requests for hundreds of gigabytes.

Do this

curl

# 1. The manifest is open — no key, and it lists every partition with size and row count.
curl -s https://data.vike.io/archive/manifest.json -o manifest.json

# 2. One whole file, one request.
curl -H "X-API-Key: $VIKE_API_KEY" \
     -o polymarket-trades-2026-09-01.parquet \
     "https://data.vike.io/archive/venue=polymarket/date=2026-09-01/trades.parquet"

# 3. A multi-gigabyte file, resumable. `-C -` sends one Range request to pick up where the
#    partial file stopped, so each resume is one more request -- not thousands.
curl -H "X-API-Key: $VIKE_API_KEY" -C - --retry 5 --retry-delay 10 \
     -o polymarket-book-2026-09-01.parquet \
     "https://data.vike.io/archive/venue=polymarket/date=2026-09-01/book_events.parquet"

Use -o with an explicit name, not -O. The last path segment is trades.parquet for every date, so -O overwrites the previous day's file with the next one.

A bounded fan-out is fine — it is one request per file either way:

jq -r '.dates[].date' manifest.json | \
  xargs -P 4 -I{} curl -sS -H "X-API-Key: $VIKE_API_KEY" \
    -o "l1-{}.parquet" \
    "https://data.vike.io/archive/venue=polymarket/date={}/l1_quotes.parquet"

Python

import requests
import pyarrow.parquet as pq

KEY = "vk_..."
URL = "https://data.vike.io/archive/venue=polymarket/date=2026-09-01/trades.parquet"
LOCAL = "polymarket-trades-2026-09-01.parquet"

# Download: ONE request, streamed to disk so a multi-GB file never has to fit in memory.
with requests.get(URL, headers={"X-API-Key": KEY}, stream=True, timeout=300) as r:
    r.raise_for_status()
    with open(LOCAL, "wb") as f:
        for chunk in r.iter_content(chunk_size=1 << 20):
            f.write(chunk)

# Parse: LOCAL path, so column pruning and predicate pushdown cost nothing at all.
table = pq.read_table(LOCAL, columns=["ts", "token_id", "price", "size", "side"])

Verify what landed against the manifest's own bytes for that partition before parsing — a truncated download is a valid-looking file until the footer read fails.

DuckDB, polars, pandas

Same shape: fetch first, then open the local path.

-- Right: the files are already on disk.
SELECT token_id, count(*) FROM read_parquet('polymarket-trades-*.parquet') GROUP BY 1;

Do not do this

# WRONG. Every column chunk of every row group becomes its own Range request over the network.
table = pq.read_table("https://data.vike.io/archive/venue=polymarket/date=2026-09-01/book_events.parquet")

-- WRONG, for the same reason: httpfs turns one query into thousands of requests.
INSTALL httpfs; LOAD httpfs;
SELECT * FROM read_parquet('https://data.vike.io/archive/venue=polymarket/date=.../book_events.parquet');

The files do advertise Accept-Ranges: bytes, and the manifest sets range_requests: true. That is there so an interrupted download can resume — not so a query engine can page a remote file.

Fetch less by choosing a smaller file

The supported way to narrow a pull is to pick a narrower file, never a byte range inside a wide one:

• Only need the tape? Take trades.parquet — around 14 MB a day, against several gigabytes of full L2.
• Only need top-of-book? Take l1_quotes.parquet, a few hundred megabytes a day.
• Only need one crypto up/down family? Use the family layout — venue=polymarket/asset=btc/tenor=5m/date=YYYY-MM-DD/… — where a day is tens to hundreds of megabytes instead of gigabytes. The layouts and their caveats are on Bulk archive.
• Only need one day? Partitions are per UTC day already; ask the manifest which dates exist rather than probing for them.

Every one of those is still one request per file.

If you get a 429 or a 401

The archive never returns a rate-limit 429: an /archive/ request is not checked against the hourly counter. A missing or invalid credential is a 401. The budget the same key's /v1 and MCP calls share is unaffected by downloads, and those calls are what a 429 comes from; see Rate limits and Errors.