The watchdog
The core's cold-path sweeps — a confirm-grace ladder for orders stuck before acknowledgement, a reject-free early confirm, managed expiry, safe-state entry after a panic, the margin-call sweep and the drawdown latch.
crates/vike-core/src/runtime/watchdog.rs holds the core's cold-path sweeps: the stuck-order
ladder, the fast in-flight confirm, managed good-till-date expiry, safe-state entry, the
margin-call sweep and the equity-drawdown latch. None runs on the per-message fold; the timed
ones fire from the timer wheel at the drain-loop boundary and the
others on a closed bar, so the latency gate is untouched. This
page is written from that module and from the CoreConfig fields in
crates/vike-core/src/runtime/mod.rs that arm each sweep.
The stuck-order ladder
sweep_stuck_orders is the last-resort backstop for an order stuck before acknowledgement
(Initialized or Submitted): an adapter that accepted submit but whose thread wedged before
emitting any terminal. It must never race a slow-but-real venue ack into a phantom reject — if
the venue actually accepted or filled the order, a synthesized OrderRejected strands a real
position, because the later real OrderAccepted or OrderFilled is an illegal transition from
Rejected and is dropped (counted as stranded_terminal_drops). So it is a ladder, armed by
CoreConfig::submit_ack_timeout (None by default) with submit_ack_confirm_grace (default
15 s):
- Stage 1 — at
created + submit_ack_timeout, soft-warn and activelyconfirmonce: the venue is prodded into a status re-query whose authoritative terminal folds back on the normal event lane. Nothing is terminalized, and the time of the confirm is recorded. - Stage 2 — the plain reject deadline is
created + submit_ack_timeout + grace, but a candidate whose stage-1 confirm is presumed in flight (it is still pre-ack) is deferred one further grace window, to+ 2·grace. Only past that, still pre-ack, is theOrderRejectedsynthesized, with the reasonwatchdog: no venue ack within confirm-grace. An order that reaches the deadline without ever passing stage 1 — a busy core skipped ticks, or the clock jumped — is confirmed now and deferred, so no backstop reject without at least one active confirm plus its grace window holds.
The core fold thread issues no network call: confirm_order is fire-and-forget onto the
adapter's own thread, and the sweep only waits for the terminal. Reconcile-seeded orders, which
carry no creation time, are never swept. The bookkeeping is pruned to still-pre-ack orders at
the end of every sweep.
Sizing the two knobs
submit_ack_timeout must exceed the venue's worst-case order-visibility latency and the
adapter's own submit-plus-requery budget; the doc suggests a conservative value such as 30 s for
reconcile-less venues. The grace must clear the confirm's own round trip — the stated bound is
2·grace > submit_ack_timeout/2 + N·requery_timeout, with a re-query timeout of about 5 s and
N = 2 for Bybit (a realtime-then-history re-query, worst case about 10 s) or 1 for OKX. A
zero grace collapses the ladder back to immediate reject and reintroduces exactly the
phantom-reject risk the window exists to remove.
The reject-free early rung
sweep_inflight_confirms, armed by CoreConfig::inflight_confirm (None by default,
suggested live value about 2 s), re-queries an order stuck pre-ack on a fast cadence, catching a
wedged adapter in seconds instead of waiting the conservative timeout for the ladder's first
confirm. Three guarantees make it safe: it calls only the publish-nothing confirm_order and
can strand nothing; it is band-limited to ages in [inflight_confirm, submit_ack_timeout),
handing off the instant an order crosses the timeout so the two rungs never act on one order at
one age; and it keeps its own dedup map, re-confirming a still-stuck order at most once per
window. With the ladder off the band is unbounded and this becomes the only stuck-order signal
— still reject-free, because without a configured reject timeout the core must never invent a
terminal.
Managed expiry
sweep_gtd_expiry, armed by CoreConfig::gtd_sweep, cancels every resting order whose
time-in-force deadline has passed, for venues that do not expire GTD or DAY orders themselves.
The decision is vike_model::tif_expired, the one expiry law shared with the paper book; the
terminal vocabulary differs on purpose. The paper book owns its resting order and mints
OrderExpired; this sweep asks the real venue to cancel and lets the venue's authoritative
OrderCanceled fold back, because publishing OrderExpired locally would desync local state
from a venue order still live. Each expiry is journaled write-ahead as GtdExpire — a marker,
never re-applied on replay — and the cancel is fire-once per order.
Safe-state entry
enter_safe_state is what a fold panic reaches. It logs the reason once at error, sets
trading_state = Halted on the primary and every extra engine so the risk gate denies every
new order, records the fault for the snapshot, and issues a best-effort cancel for every
non-terminal working order tagged CancelIntent::RiskOff — the core has faulted and is pulling
its whole book, so a venue metering its cancels must not hold any back. The sweep is itself
guarded: a client that panics during it cannot take the loop down. What the
execution core does afterwards — read for display, never
traded on again until restart — is that page's subject.
The margin-call sweep and the drawdown latch
Both run once per closed bar of the engine's own series, where marks are fresh, and both read
equity through the resolver the snapshot displays. sweep_margin_call runs the
margin-call model per engine and submits each liquidation as a
reduce_only market order through the normal gate, routed to the breaching engine by index.
sweep_drawdown_latch, armed by CoreConfig::max_drawdown (None or any value at or below
zero disables it), folds a high-water mark of the daemon's own equity curve and, when the drop
exceeds the threshold, sets every engine to TradingState::Reducing so only reduce-only orders
pass. It mints no orders, pushes one warning naming the mark, the curve and the drawdown into
the recent-events ring, and never un-latches on its own — recovery is a deliberate manual
SetTradingState.
A fraction of what
The curve is Σ seed_of(engine) + Σ resolved_own_pnl(engine) — configured capital plus the
daemon's own realized and unrealized P&L — and deliberately not observed account equity, which
on a venue-attested account is the wallet for the whole account the credentials open. Measured
on prod2 on 2026-08-17: nine paper mounts at 1000 seed each plus one bybit mount carrying
53647.10600813 of a shared demo wallet latched a high-water mark near 62647, where a third party
withdrawing would trip liquidate-only with no trading at all, and a genuine 25% loss on the
daemon's own ~9000 of book read as 3.6%. Across a restart the peak seeds to max(capital base, curve), so the curve resumes underwater rather than forgiving earlier losses; a non-positive
peak cannot express a fraction, so a profile with max_drawdown set is refused unless its seed
cash is positive.
An operator facing an order that will not leave Submitted starts at
triage a stuck order; the venue-side truth the ladder
defers to is reconciliation.
The dead-man's switches
Two of them, with opposite defaults because they read opposite evidence — the connection switch is on unless you turn it off and trips on a link the venue's own bridge reports dead; the silence switch is off unless you write the key and trips on a feed that has gone quiet, which is also what a closed market looks like.
Timers
One allocation-free, deterministic deadline wheel, advanced once per drain-loop boundary and never per message, carries every periodic sweep the live core runs.