Running the latency gate

The core-hop tail gate is a microsecond timing assertion over the live runtime's fold loop, and running it is not an ordinary test invocation. This page is the procedure. What it means — which numbers gate, which only measure, and why the split is deliberate — is The latency gate.

Run it

crates/vike-core/tests/runtime_latency.rs is a release-only harness. Every test in it carries #[ignore = "release-only latency harness (see module doc)"], so a default test run compiles it and executes nothing.

cargo test -p vike-core --release --test runtime_latency -- --ignored --test-threads=1 --nocapture

Two of those flags are load-bearing; the third is not:

• --release. The module header says it outright: timing-sensitive, skipped in the debug run.
• --test-threads=1. Correctness, not tidiness. Every harness thread busy-spins waiting for its core worker, and the gate runs pinned to a small core set, so the default parallelism puts N spinners plus N workers on those few cores and the spinners starve the workers they wait on. Priority inversion, not slowness — measured 2026-07-29, three tests ran in 0.94 s and a fourth livelocked the binary at 380 % CPU for five minutes, the incident crates/vike-core/tests/runtime_latency.rs's SPIN_DEADLINE records. It also makes the numbers honest: run in parallel, these tests contend for the very cores whose p99 they report, noise that once hid a 27× tail regression.
• --nocapture. Habit, not a requirement — nothing in the harness depends on it. HopStats::report writes its LATENCY-GATE line straight to the stderr handle rather than through eprintln!, precisely so the numbers survive libtest's capture on a pass: a gate whose figures are legible only when it is red cannot be trended. The CI step has never passed the flag, and every variant's line is in its logs regardless.

Reading a red run

Only three tests assert a derived ceiling — p99_core_hop_under_10us, p99_core_hop_under_10us_with_journal and p99_core_hop_under_10us_with_journal_snapshots, reporting the variants baseline, journal and journal-snap. The submit-lane, book-lane and coid harnesses beside them assert nothing about timing, deliberately, and snapshot-build only the 1 ms p50 sanity ceiling it inherited — a liveness check with ~100× headroom, not a derived budget: deriving JOURNAL_P99_NS cost 98 CI attempts, and an uncalibrated ceiling is a flaky gate — worse than no gate, the header says, because people learn to re-run it rather than read it.

Each variant reports its whole distribution over HOP_SAMPLES hops, so read the failure line rather than the first assertion. assert_hop_budget checks p99 first, which on the journal variants is the least informative number in the file: 15 of the 18 red attempts in the 2026-08-05 calibration carried a hops > 100 µs and/or p99.9 cause instead, and JOURNAL_HOPS_OVER_100US is the ceiling that binds today. JOURNAL_P99_NS was derived at 2.0× the worst green observation, from green observations only — a figure from a red attempt is evidence of how badly the gate can be disturbed, never of how the code behaves when healthy.

Reading the persisted series

HopStats::report writes one LATENCY-GATE line per variant straight to the stderr handle, and the CI latency step appends one JSON row per (run, attempt, variant) to a durable series file on the measurement box — on pass and fail alike, every k=v after the marker carried verbatim. Read that series; do not re-scrape job logs. Three traps come with it:

• The schema is flat, with a variant field. .journal.p999_ns matches nothing and fails silently — the query is select(.variant=="journal") | .p999_ns.
• A pull-request row's sha is an ephemeral merge commit, so the code trend filters event=="push" and branch=="main"; and a row whose pin differs from its pin_requested was measured on the wrong cores.
• p50_ns is indifferent to box load — 1 353–1 523 ns across every load bucket measured 08-15 to 08-24 — while only the tail moves. A four- or six-figure "p50" quoted anywhere is a p99.9 that has been mislabelled.

Adding a variant

A new measurement variant needs no workflow change: printing the report line is the whole of joining the series. Two things about the label are gated rather than commented, at the one choke point every label passes through — check_report_line, called from HopStats::report. A label containing whitespace does not produce a mangled row, it produces a plausible row filed under a truncated name; and two harnesses sharing a label merge two distributions into one trend line with nothing to separate them afterwards.

The trap a new harness hits first is growth. The runtime publishes whenever the core is about to go idle, and this file's request/response pacing makes it idle between every message, so CoreSnapshot::build runs once per message and walks the whole order registry. A harness that submits N orders under N distinct client-order-ids is O(N²), and its tail reports index-map growth rather than the lane it claims to — run_submit_hop pins the registry by reusing one coid.

An honest A/B

The shape is what matters, not the exact commands: dedicate a small core set by shrinking the system slices onto the rest, run the built test binary in a scope of its own holding those cores, and release them afterwards — releasing is mandatory. The scope needs its own top-level slice, because a child cgroup cannot hold CPUs its parent lacks, and scopes reject exec properties, so real-time priority comes from chrt. That fence is the one the harness already assumes: crates/vike-core/tests/runtime_latency.rs's module doc describes the binary as running "under chrt -f 50 on four reserved cores" of a box that also carries ClickHouse and the live recorders.

Then interleave the variants and take medians of at least eight reps: single runs and four-rep samples produced contradictory rankings all day. A real A/B in this tree was measured that way and records it — crates/vike-core/src/runtime/mod.rs's note_event cites "shielded cores, SCHED_FIFO 50, --test-threads=1, 8 interleaved reps" for the change that took the baseline core-hop p99 from 2,024 ns to 802 ns. Shielding does not make the gate deterministic either — the dominant noise is memory bandwidth and last-level cache, shared box-wide regardless of pinning, which is why the CI job retries three times with a settle between attempts.