vike
ConceptsData

book

The book store kind: its columns, partitioning, what venue and symbol mean for it, and the producers that write it.

book is one of the shapes the historical store holds. A stored series lives under a kind= path segment, and that segment is the layout contract its producers and its consumers have to agree on: the columns, how the leaf is partitioned, and what venue and symbol mean for this particular kind. All of it is below, rendered from the same table the runtime dispatches on.

One decoded row becomes a BookRow (exploded from vike_model::BookUpdate — ONE ROW PER LEVEL). The codec that owns the schema is BookCodec; the store writes through append_book_updates and reads through scan_book_updates, and parts carry the Parquet metadata key vike.schema.book.

Identity

venue = the bridge venue id; symbol = the instrument. The LOSSLESS L2 lane: every delta present, seq contiguous, gaps detectable. ⚠ CONTRADICTED by the pmxt Polymarket archive, and PINNED here rather than quietly fixed: crates/vike-data/src/cheap_np_book.rs — which moved into this crate in #2017 and so now sits beside this row — records that the pmxt mapper assigns seq per asset per HOURLY FILE, so it restarts at 0 every hour. This row describes the LIVE recorder's output; the archive ingested from pmxt does not satisfy it, and that file's a_recorded_seq_restart_does_not_drop_later_deltas is what holds the fold correct across the restart

This is the field most often assumed and most often wrong, which is why it is stated per kind rather than once for the store.

Columns

Every Arrow column today's writer emits, in schema order, with the codec flavor that picks its Arrow type and nullability.

ColumnFlavorNote
tsi64—
local_tsi64—
seqi64—
kindi8—
is_bidbool—
pricef64—
sizef64—
tick_sizef64—
symbol_colstr_addmay be ABSENT ENTIRELY from a part sealed before the column was added

Layout

PropertyValue
Partitioningsymbol
Grouped formyes — many symbols in one part, told apart by a row-level symbol column
In the cross-kind tick reportyes

Traps

⚠ The APPEND and the SCAN count different things: append_book_updates returns EVENTS while the part holds one row per LEVEL, and a zero-level event (a status marker, or a degenerate empty snapshot) is stored as ONE placeholder row with is_bid=true, price=0.0, size=0.0 that decodes back to empty level vectors. A consumer counting rows is not counting events. The kind column is an Int8 CODE, not the series kind — crates/vike-data/src/datafusion_hist/codec.rs's book_kind_code is the mapping, and the §B stream-health markers (GapStart/Stale/LiveResume) ride this lane as those codes.

Pinned rather than fixed: the contradiction is recorded with both sides named, so a reader meets it here instead of in production.

Known producers

Idempotency in this store is batch-level and keyed on a commit-key string, never on per-row value dedup. Two producers that collide on one template silently no-op each other's appends, and a producer that changes its template re-admits history it already wrote — both invisible unless the shapes are written side by side.

ProducerCommit-key template
crates/vike-data/src/live_rec.rslive-{venue}-{symbol}-{}-{first_ts}-{last_ts}-{seq}
crates/vike-backfill/src/pmxt/ingest.rspmxt:book:{asset}:{hour}
crates/vike-backfill/src/vike_archive.rsvikearchive:{}:{date}:rg{rg_idx}
crates/vike-backfill/src/events_api.rseventsapi:book:{symbol}:{day}
crates/vike-backfill/src/tardis/ingest.rstardis:{}:{symbol}:{day}
crates/vike-backfill/src/databento/ingest.rsdatabento:{}:{symbol}:{window}

⚠ That list is observed, not exhaustive, and no gate can make it so: a commit key is an ordinary string argument, so proving the list complete would mean following every call in the workspace. What is gated per listed entry is that the producer exists and spells the template verbatim. Read it as "these producers write this kind", never as "only these do".

Where this comes from

vike_data::store_kind::STORE_KINDS, exported as rosters.json at every release. The store's own gate derives the live kind set from the code — the series directory literals, the compaction dispatch and the write verbs — and fails until a new kind has a row, so this page cannot describe a shape the store does not write.

On this page