Page through large reads

The three ways the /v1 routes page, a cursor, an offset and a moving time window, each walked in one script with its stop condition.

There is no single paging scheme. Which one a route uses is on its endpoint page, and using the wrong one either repeats the first page or silently stops short. The recipe walks one route of each kind for three pages:

Style	Example route	Move to the next page	Stop when
Cursor	`/liquidations`	pass `nextCursor` back as `cursor`, every other parameter unchanged	`nextCursor` is `null`, or a page comes back empty
Offset	`/archive/datasets`	add the rows received to `offset`	`offset + len(datasets) >= total`
Time window	`/ohlcv`	restart `start` one bar after the last candle you hold	a page shorter than `limit`

Run

export VIKE_API_KEY=...
python paging-large-reads.py             # real run: needs the key
python paging-large-reads.py --dry-run   # prints the first request of each walk, calls nothing

MAX_PAGES keeps the script to three pages per walk. Remove it, or raise it, for a real pull.

paging-large-reads.py

"""The three ways /v1 pages, each walked for a few pages: a cursor, an offset, and a moving time window.

Run:  VIKE_API_KEY=... python paging-large-reads.py    (--dry-run lists the requests and makes none)
A real walk drops MAX_PAGES and runs to the end.
"""
import os
import sys
from urllib.parse import urlencode

NAME = "paging-large-reads"
BASE = "https://data.vike.io"
DRY = "--dry-run" in sys.argv
MAX_PAGES = 3  # keeps the demo short; a real walk loops until the stop condition alone ends it
DAY_START = 1786924800000  # 2026-08-17T00:00:00Z in epoch ms
MINUTE_MS = 60_000


def fail(why):
    sys.exit(f"{NAME}: {why}")


def call(path, method="GET", **params):
    url = BASE + path + ("?" + urlencode(params) if params else "")
    if DRY:
        print(method, url)
        return None
    key = os.environ.get("VIKE_API_KEY") or fail("set VIKE_API_KEY")
    import requests

    r = requests.request(method, url, headers={"X-API-Key": key}, timeout=60)
    if r.status_code != 200:
        fail(f"{method} {path} answered HTTP {r.status_code}: {r.text[:200]}")
    return r.json()


def by_cursor():
    """HL liquidations: pass nextCursor back verbatim, with every other parameter unchanged, until it is null."""
    rows, cursor = 0, ""
    for _ in range(MAX_PAGES):
        params = dict(coin="BTC", start="2026-08-17T00:00:00Z", end="2026-08-18T00:00:00Z", limit=500)
        if cursor:
            params["cursor"] = cursor
        page = call("/v1/hyperliquid/liquidations", **params)
        if page is None:
            return 0
        rows += len(page["liquidations"])
        cursor = page["nextCursor"]
        # A full last page still carries a cursor; the page after it is empty.
        if not cursor or not page["liquidations"]:
            break
    return rows


def by_offset():
    """Polymarket archive index: offset += rows received, until offset reaches total."""
    rows, offset = 0, 0
    for _ in range(MAX_PAGES):
        page = call("/v1/polymarket/archive/datasets", stream="trades", limit=100, offset=offset)
        if page is None:
            return 0
        rows += len(page["datasets"])
        offset += len(page["datasets"])
        # total is the filtered count before limit and offset, the only field that says when to stop.
        if not page["datasets"] or offset >= page["total"]:
            break
    return rows


def by_window():
    """HL candles have no cursor: restart one bar after the last candle you hold."""
    rows, start = 0, DAY_START
    for _ in range(MAX_PAGES):
        page = call("/v1/hyperliquid/ohlcv", symbol="BTC", interval="1m", start=start,
                    end=DAY_START + 6 * 3_600_000, limit=100)
        if page is None:
            return 0
        candles = page["candles"]
        rows += len(candles)
        if len(candles) < 100:  # a short page is the last one
            break
        start = candles[-1]["ts"] + MINUTE_MS
    return rows


def main():
    counts = {"cursor": by_cursor(), "offset": by_offset(), "window": by_window()}
    if DRY:
        return
    for style, n in counts.items():
        if not n:
            fail(f"the {style} walk returned no rows")
        print(f"{style:6} paging: {n} rows over at most {MAX_PAGES} pages")


if __name__ == "__main__":
    try:
        main()
    except Exception as e:  # a shape the recipe does not expect fails by name, not by traceback
        fail(f"{type(e).__name__}: {e}")

Details that decide whether the walk is complete

A full last page still carries a cursor. /liquidations sets nextCursor whenever the page holds limit rows, so the last page of a result whose size is an exact multiple is followed by an empty one. Stop on either.
total is the filtered count before paging. On the archive index it follows your filters, not the page, and it is the only field that says when to stop. limit defaults to 200 and is capped at 1000.
/ohlcv has no cursor. It returns one page of at most 5000 bars, oldest first, and start accepts epoch milliseconds, so the next start is the last candle's ts plus one bar.
Not every list route pages. /traders stops at 20 rows with no cursor, and /markets at 100 with no offset; for those, narrow the filter or use the richer route named on the page.

A cursor records where to resume, not what you asked for

/liquidations builds its cursor from the last row's timestamp and wallet alone. The filters come from the request, so send the same coin, start, end and limit with every page; the script does, and a page with different ones answers a different question from the position the cursor names.

Page through large reads

Details that decide whether the walk is complete

On this page