Deterministic replay and live/backtest parity

The command journal

The journal is off by default and costs nothing when off: no journal is opened and the fold path is byte-identical to a build that never had one. Turned on, it appends every execution-lane message — venue events and commands — to pre-allocated memory-mapped segments, one frame per record, each frame carrying a length prefix, a checksum over its payload, and the payload itself. A zero length means the end of written data; a record that fails its checksum is a torn tail, and reading stops at the last valid record rather than panicking.

The append sits before the fold, which has one consequence worth stating plainly:

Only the execution lane is journaled. Market, bar and tick lanes are not command state and never touch it — which is also why replaying a journal cannot move an equity curve on its own. Session topology such as a runtime mount or unmount is recorded in a sidecar instead. Full-state checkpoints are written on a record cadence and always on shutdown, so a journal always has a base to restore from.

Durability, stated honestly

Appending is a bounded copy into a mapped page. Forcing those pages to disk costs time proportional to the dirty bytes — measurably tens to hundreds of milliseconds for a dirty segment — and the append happens on the thread with the latency budget, so every blocking sync runs on a dedicated syncer thread the appender never waits on. What makes a late sync safe is what the mapped pages are: they belong to the operating system's page cache, so a process crash, an out-of-memory kill, memory-pressure eviction and a clean reboot all lose nothing with no sync at all. Only a hard power cut loses the un-synced tail, and that tail is exactly what the post-restart venue reconcile repairs. The sync is latency shaping — it bounds how much dirty data any one flush has to move — not the durability mechanism.

One writer at a time: opening a journal takes an exclusive advisory lock on its directory, so a double-launched instance fails fast instead of interleaving frames into a live journal. The read-only entry points take no lock at all, which is what lets you inspect and replay a directory a live core currently owns.

The format carries a version per segment and reads and resumes anything inside a supported range, so upgrading the binary never costs an operator their live journal. The rule that keeps the range safe is narrow: a version step may only add a record variant, or add a defaulted field to an existing one. Anything that renames, removes or reshapes has to raise the floor in the same change.

Replaying a session

Offline replay restores the engine from the journal's first checkpoint, re-folds the entire tail after it, and compares the reproduced final state against the source journal's last checkpoint. Restoring from the first is what makes the comparison worth anything: restoring from the latest would leave nothing to replay on a clean run, because the exit checkpoint already reflects everything.

From that engine a real core is spawned, with three substitutions and no others:

A no-op execution client

It suppresses every venue side effect and synthesizes no events. The original venue events are already in the journal and are re-pumped from it, so a client that also emitted them would double the state — and correctly break the comparison.

A queue clock

It replays the tail's recorded timestamps in order, so timestamp-derived state such as an order's creation time reproduces rather than being stamped with today's wall clock.

A checkpoint-free journal

The replay's own temporary journal is configured so only the exit checkpoint fires, and that checkpoint is what carries the reproduced hash back out.

The tail is pumped through one queue in journal order, so dispatch order equals record order. The replay core runs no strategy hook at all: only a strategy's already-decided orders need to reproduce, and those are on disk. Extraction of what re-applies is an exhaustive match with no catch-all, so a new record variant fails to compile until somebody decides what replay should do with it — the class of omission that would otherwise be discovered as a mismatch months later. Records that are decision markers, whose effects arrive as their own records, are deliberately replay-neutral: re-issuing them would double a cancel or a batch of submits.

It runs from a command line over a journal directory and prints the source record count, the number of checkpoints verified, the reproduced hash and one line per engine with its open orders, positions and balance. A failure prints the error and exits non-zero.

The three fences

The primary fence is a hash over the canonical serialization of the engine snapshots. Two fields are cleared before hashing, and the reason each is cleared is the reason the fence is credible: the top-level clock stamp, which is written by every ingest message including non-journaled market ticks, and the account's mark-price cache, which mark ticks mutate. Neither is reproducible by re-folding the execution lane alone, so hashing them would fail the fence on a perfectly valid journal. Everything else stays hashed.

That covered one thing while the restore hands back several more, so two further fences sit over the same oracle:

• The resting conditional books are hashed and compared, with a trailing stop's ratcheted extreme cleared for the same reason the mark cache is — it moves on market data the journal does not carry.
• The two id counters are fenced as floors rather than as equalities. An overshoot only wastes ids; an undercount means a restored session re-mints an id the pre-crash run already spent — and because an arm id is also the disarm key, a duplicate would let a stale disarm remove a fresh protective stop.

The two failure modes surface as distinct errors on purpose, because they have different triage paths. And a truncated journal is reported as truncated rather than as a divergence: when the frame walk halts at a torn frame the last readable checkpoint is a mid-session one, so an honestly re-folded tail cannot match it. The error even distinguishes the causes — a torn length prefix is the ordinary consequence of a crash, while a checksum-valid frame that does not parse came from a different build.

Crash restore

The same engine backs a restart. Restore reads from the latest checkpoint, re-folds only the tail after it — empty on a clean shutdown — and applies no fence, because it is a restore rather than a proof. What it hands back is the whole of what a restarted process needs to not collide with its own past: the engines, the client-order-id session and counter so post-restart ids never collide with a still-open prior id, the arm counter, the resting conditional books, the contingent-order book, the per-mount attribution ledgers, and the map from client order id to the mount that placed it — which is what lets an order still resting across the restart book its eventual fill into the right mount.

The caller pattern is the obvious one and is written on the function itself: rebuild the engine from the snapshot with the real client, spawn the core with the restored session and books, point the journal at the same directory so appends resume, and let venue reconciliation repair whatever the venue saw while the process was down. Two residuals are declared rather than discovered: a trailing stop re-arms at its last-checkpoint extreme — slightly looser than it died, re-tightening as live data resumes — and on an unclean crash the attribution ledgers are the base checkpoint's values, with the gap showing in the published residual row rather than vanishing.

The parity gates

Replay proves a session reproduces. Parity proves the arithmetic has not moved. Four gates, each comparing something different:

Frozen golden fixtures

The pure cores — the fill primitive, the fill triggers, the cost scalars, the event engine, the vectorized kernel, the sizing maths, the account read-model — are held to bit-identical floating-point results against committed fixtures. Floats are stored as IEEE-754 hex bit patterns and compared exactly, never through a tolerance. Every exporter that produced them was deleted from the tree, so the committed bytes are the oracle: there is no environment to regenerate from and nothing to drift.

Engine against kernel

The event engine and the vectorized kernel run over identical synthetic bars and must agree on trade count and produce a bit-identical final equity. A second oracle that keeps working as strategies are added, because it compares two things that both run.

Script against Rust

A scripted strategy and its Rust twin must produce a byte-identical result.

Backtest against paper

The same strategy and the same bars, run through the backtest engine and through the live core in paper mode, must produce bit-identical fills — timestamp, side, quantity, price, fee and maker flag — plus identical realized profit, fees and final position. The strategy logic is defined once and mounted through each runtime's own trait, so the gate proves the runtimes agree rather than that two copies of a strategy happen to.

Why this is structural, not a claim

Parity is a property of how the code is arranged. Three arrangements carry it, and each was learned from a specific failure:

• One fill definition, in a crate below both consumers. The simulator and the live paper exchange do not each own a cost model kept in step by review; they share one, in a crate beneath them. When two sides must not disagree, the cure is a shared crate below both — not a shared crate containing both.
• Operation order is part of the contract. Bit parity is a property of the order operations happen in, so the pure cores forbid reordering, fused multiply-add and "simplification". Even container choice is part of it: the account keeps insertion-ordered maps rather than hash maps, because insertion order is the floating-point fold order the fixtures pin against. The most instructive divergence the gates ever caught was a compensated summation in one language's built-in sum function against a naive fold — a guaranteed last-bit difference at every ported site that used it.
• Transcendentals are pinned rather than platform-supplied. Floating-point standards require nothing of logarithms, exponentials or powers, so two platforms disagree in the last bit; the pure cores the frozen fixtures drive reach none of them at all, and everywhere else in the tree a single pinned implementation is called instead of the platform's. The property bought is reproducibility, not accuracy: two platforms are differently wrong at the last bit, and pinning makes them identically wrong everywhere.

And where randomization is used to find what fixtures cannot, a counterexample found in CI is committed as a regression seed, so the failure reproduces on every machine rather than depending on the random draw that found it.

Where the equivalence stops

A law with an unstated exception is worth less than no law, so here are the scopes as the source states them.

More of what it does