The history store
One Parquet store under the project, thirteen declared layouts, a gate that derives the live set from the code, many parts per day, and the resolver that decides where the store lives when nobody said.
Every historical series Vike Trader records, backfills or replays goes through one seam:
HistStore, in crates/vike-data/src/hist.rs — one store for bars and ticks. The trait is
always compiled and depends on vike-model only; the concrete backend, DataFusionHist, is
DataFusion over Parquet behind the hist-datafusion feature — no SQLite, no server — partitioned
by UTC day, with compaction, retention and a manifest file-index that also drives reads, so nothing
lists a directory. Ingest is idempotent by commit key, never by row value: a key names the
source batch, and a batch already ingested for a series appends nothing, because trades carry no
id and idempotency has to live at the batch level. The same crate owns the
live mirror of this seam, so a strategy reads bars the same way
whether they came off a socket or off disk.
The authority for the rest of this page is crates/vike-data/src/store_kind.rs.
Thirteen kinds, one declared row each
The on-disk layout is a contract between many producers — the live recorder, the backfill
collectors, the journal materializer, a bridge writing a derived series — and three consumer
families: tick replay in the backtester, served reads in the data service, and the inventory views
in Data Manager and Studio. Until STORE_KINDS existed no site named that contract; each end
learned it by reading another end, and several shipped defects were exactly a producer and a
consumer disagreeing about the shape — a kind missing from the coverage view while its rows sat on
disk, five Data Manager verbs silently operating on a series that did not exist, a part-name
collision overwriting live parts.
STORE_KINDS declares one row per kind= naming what both ends must agree on: the domain row
type, the Arrow columns today's writer emits (with the codec flavour that decides nullability and
old-part tolerance), the Parquet schema-metadata key, the partitioning, whether a grouped form
exists, what venue and symbol actually mean for that kind, and the commit-key shapes its
known producers use.
| kind | grouped form | what it holds |
|---|---|---|
bar | no | OHLCV bars, sub-partitioned by interval — the only kind that is |
quote | yes | top-of-book quotes |
trade | yes | the market print tape |
book | yes | the lossless L2 lane: every delta, snapshot anchors and stream-health markers, seq contiguous |
depth | no | the conflating L2 lane: periodic full snapshots with every intermediate state discarded — a different series on purpose, so a book reader never receives conflated data |
properties | no | point-in-time SymbolProperties grids, about one row per symbol per day |
equity | no | equity-curve samples — venue is the fixed namespace portfolio and symbol is a venue name or the TOTAL rollup |
exec_fill | no | the account's own fills, a distinct kind from trade so the two never collide |
exec_order | no | the account's own orders, as lifecycle snapshots |
funding | no | the account's realized funding payments, keyed by coin |
chain | no | option-chain snapshots, hundreds of rows sharing one ts |
cohort | no | a graded positioning panel from a metrics service |
perp_metrics | no | a perpetual's premium, plus open interest where a source supplies it — the narrowest kind in the store |
Each kind has its own page under data, generated from this table; what fills them is the data providers tree.
Two disagreements are pinned, not fixed
depth is listed as groupable by the backfill crate's regroup table and has no grouped form
anywhere in the store — and no backfill source serves it at all, so a depth gap can only ever be
re-recorded live. And two unrelated series are both called funding: kind=funding is the
account's realized payments, while the market funding-rate history is a kind=bar series
under the reserved interval=funding label. Each is asserted by a test, so whoever resolves one is
told to update the table rather than leaving a stale warning behind. That is the workspace's
playbook for any per-thing capability table: declare today's reality first, byte-identical, and
flip behaviour one row at a time.
The gate derives the live set from the code
crates/vike-data/tests/store_kind_gate.rs does not trust the table. It derives the set of kinds
the code actually writes three independent ways, because each is blind to a different mistake:
the first argument of every path-helper call in the backend, the kind strings each compaction
dispatch accepts, and the append_* verbs on the trait. It fails until every derived kind has a
row, then checks each row's columns and schema-metadata key against the codec they were read from.
Adding a kind the ordinary way reddens the trait check first, before anything writes a directory.
Its residuals are declared on the gate rather than implied: it cannot see dataflow, so a literal
reaching a directory through a helper it does not follow is invisible, and every such site carries
a written reason.
A partition holds many parts
Parts are date=-partitioned identically for every kind, and a part never spans a UTC day. That
does not mean one file per day: a writer seals one part per commit, the live recorder commits
per buffer flush, and compaction merges the results alongside. Measured read-only on a live tape,
one day's kind=book partition held 103 Parquet files and the previous day's 130. A consumer built
for one-file-per-day mishandles the real case, which is why a_date_partition_holds_many_parts
pins both halves. One more trap for a reader of old parts: a column with an _add flavour may be
absent entirely from a part sealed before it existed, so a consumer must never assume it is
present.
Where the store lives when nobody said
DataFusionHist::open takes the root it is given and creates it — it has no opinion about where
data belongs. That question is answered one layer up by
crates/vike-model/src/store_path.rs's resolve_store_root, the one precedence every binary
resolves through, born from a cleanup that found six stores scattered over three directories on
two disks. From the top: an explicit --store or a profile's store =; then $VIKE_HIST_STORE,
the box's standing answer; then <project>/market_data/hist for a project declared outright by
$VIKE_SETTINGS_DIR; then a developer's existing checkout store, if the checkout that built the
binary still exists; then <project>/market_data/hist for a project discovered by walking up
from the working directory — the same walk that finds <project>/settings/, so settings,
credentials and data can never disagree about which project this is, and the default a fresh
install gets; then a per-user data directory for a binary with no project above it at all. Every
answer comes back with the StoreRootRung that chose it, and every binary that resolves through it
logs both before opening anything — a tool handed a root outright, as vike-study and tearsheet
are by --store, opens it with no such line.
Nothing validates that a root is the one you meant
Point a tool at an empty directory and it creates a fresh, valid, empty store; the run then
reports zero rows, which is indistinguishable from "the backfill found nothing". A store does not
merge — if the answer moves, the old one is simply no longer read. That is why the resolver never
invents a path from the working directory, why every rung above the default is something a human
stated, and why the declared project outranks the checkout hinge: a declaration must not lose to
an inference. Under systemd's ProtectSystem=strict a root outside ReadWritePaths= fails at
open, and the error is decorated with those two words — a diagnosis, deliberately not a pre-flight
probe.
The data model
The value types every crate shares — bars and ticks, the L2 book and its one delta-apply law, the per-symbol instrument grid, the fill primitive, the twenty-variant Event enum the core consumes — and the interning rule that decides which fields may be a pointer.
The paper exchange
Paper trading is the backtest fill model running behind the live core's own seam, built from the same primitives — so backtest equals paper by construction, and a gate proves it bit-for-bit rather than asserting it.