Segments

A journal directory holds numbered segment files, journal-{idx:08}.vjl, each pre-sized to segment_bytes (64 MiB at the default) and zero-filled. crates/vike-core/src/journal/segment.rs owns the naming and the block reservation; the lifecycle — mapping, preparing the successor, the roll — stays with the writer in crates/vike-core/src/journal/writer.rs, because it is inseparable from the CommandJournal state it swaps. Sequence numbers are global across segments: each segment's header stamps its first_seq, and each roll carries the seq forward.

The staging name is load-bearing

Every scanner in the module resolves a segment index by stripping the journal- prefix and the .vjl suffix — the resume scan in open, latest_segment_version, read_since, prune_before_latest_snap. A segment under construction lives at journal-{idx:08}.vjl.prep (prep_path), which fails that suffix and is invisible to all of them. Were it named .vjl, a crash between preparation and roll would leave an empty highest-index segment; open would resume into it, find no magic, treat it as fresh and stamp first_seq = 0 — restarting the sequence at zero while earlier segments hold higher seqs. A leftover .prep after a crash is inert; the next preparation truncates and reuses it. The LOCK sentinel from Locking is invisible for the same reason.

Pre-allocation, and why it is a window

std::fs::File::set_len only moves the size marker; the extents stay unallocated, so the filesystem defers each block allocation to the first write fault on that page — landing it on the append path the latency gate measures. reserve_blocks calls posix_fallocate at open so that work moves to one metadata operation. Measured against this journal, a sparse segment took 3–9 appends over 100 µs with a 220 µs maximum; reserved, 0–1 and about 40 µs. It is deliberately infallible: a filesystem with no fallocate degrades to sparse rather than refusing to open, and on Windows it is a no-op.

Reservation removes the allocation from the fault, not the fault itself. A shared mapping starts with no page-table entries, so the first store into each 4 KiB page still traps into the kernel. warm_ahead populates those entries writable with MADV_POPULATE_WRITE — not WILLNEED or MAP_POPULATE, which populate for reading and still trap on the first write. Warming a 256 MiB segment whole cost 694 ms at open against a 1.7 ms baseline, to serve a run that touched 9 MiB of it, which is why it is a window. warm_ahead_bytes sizes that window from the sync chunk — four chunks, floored at 8 MiB — so it outlasts one blocking msync plus the chunk that follows. The fixed 8 MiB it replaced was outrun every chunk: the appender runs at roughly 430 MB/s while the syncer's msync blocks for 36–141 ms, and the refresh was aimed at the synced watermark, behind a cursor that had already run past it. The A/B of that fix, medians over eight interleaved reps on the reserved-core box, took the journal variant's p99.9 from 37 150 ns to 3 342 ns — where the journal-free baseline's own p99.9 of 3–7 µs sits. The cost is at open, 41 ms to 135 ms on a 256 MiB segment as the window grew from 8 to 64 MiB and one eighth of that for the production segment, paid once at session start and on the syncer thread for every segment after the first. A kernel below 5.14 has no MADV_POPULATE_WRITE and degrades to first-touch faults; Windows has no madvise at all.

The roll

When a frame would not fit, write_framed calls roll. Reserving a 64 MB segment inline measured a median 1.7 ms (max 2.8 ms) against about 160 µs for a bare size-and-map, and roll runs inside the append path — so map_segment is called with reserve = false there, and the reservation happens ahead of time instead. Once the live segment crosses PREP_WATERMARK_PCT (75 %), prepare_next spawns one worker thread, vjl-prep, to size and reserve the successor under its staging name; on a 64 MB segment that leaves roughly 150 000 appends against about 1.7 ms of work, and a journal that never fills a segment never spawns it. The roll joins the worker (so nothing is still resizing a file about to be mapped), renames the .prep file into place if it is the right size, maps it, and moves two things across: the directory lock — without that move the guard would be released on the first roll — and the syncer, which is handed the new mapping and told to seal the old one. What is no longer in roll is the whole-segment msync that used to open it, 292 ms (max 432 ms) on a dirty 64 MB segment, on the fold thread. The same msync still happens on the same bytes; only the thread changed — see Sync policy, which also warms the rolled segment's window after the seal.

Resume

open finds the highest existing index, maps it, and walks its frames to find the cursor and the next seq. The sync debt is rebased onto that cursor on both sides, because those bytes were already written by whoever appended them and re-syncing a nearly full segment at startup would move the very stall the chunking exists to prevent. The warm window is re-aimed at the resumed cursor for the same reason. A supported older version is restamped to the current one, since every frame already in it is a valid frame under the additive rule on the index.

Pruning

prune_before_latest_snap deletes whole segment files that the latest checkpoint fully supersedes: those whose maximum record seq is strictly below the latest Snap's seq. It never touches the segment holding that Snap or any later one, which carry the restore base and its tail. It is conservative — a segment whose records cannot be positively bounded (unreadable, no magic, torn from the start, empty) is kept — and it honours a second floor: when a materializer checkpoint exists, no segment holding records past that checkpoint is dropped, so the trade log can still be fed from any surviving segment.