vike
Concepts

The watchdog

The core's cold-path sweeps — a confirm-grace ladder for orders stuck before acknowledgement, a reject-free early confirm, managed expiry, safe-state entry after a panic, the margin-call sweep and the drawdown latch.

crates/vike-core/src/runtime/watchdog.rs holds the core's cold-path sweeps: the stuck-order ladder, the fast in-flight confirm, managed good-till-date expiry, safe-state entry, the margin-call sweep and the equity-drawdown latch. None runs on the per-message fold; the timed ones fire from the timer wheel at the drain-loop boundary and the others on a closed bar, so the latency gate is untouched. This page is written from that module and from the CoreConfig fields in crates/vike-core/src/runtime/mod.rs that arm each sweep.

The stuck-order ladder

sweep_stuck_orders is the last-resort backstop for an order stuck before acknowledgement (Initialized or Submitted): an adapter that accepted submit but whose thread wedged before emitting any terminal. It must never race a slow-but-real venue ack into a phantom reject — if the venue actually accepted or filled the order, a synthesized OrderRejected strands a real position, because the later real OrderAccepted or OrderFilled is an illegal transition from Rejected and is dropped (counted as stranded_terminal_drops). So it is a ladder, armed by CoreConfig::submit_ack_timeout (None by default) with submit_ack_confirm_grace (default 15 s):

  • Stage 1 — at created + submit_ack_timeout, soft-warn and actively confirm once: the venue is prodded into a status re-query whose authoritative terminal folds back on the normal event lane. Nothing is terminalized, and the time of the confirm is recorded.
  • Stage 2 — the plain reject deadline is created + submit_ack_timeout + grace, but a candidate whose stage-1 confirm is presumed in flight (it is still pre-ack) is deferred one further grace window, to + 2·grace. Only past that, still pre-ack, is the OrderRejected synthesized, with the reason watchdog: no venue ack within confirm-grace. An order that reaches the deadline without ever passing stage 1 — a busy core skipped ticks, or the clock jumped — is confirmed now and deferred, so no backstop reject without at least one active confirm plus its grace window holds.

The core fold thread issues no network call: confirm_order is fire-and-forget onto the adapter's own thread, and the sweep only waits for the terminal. Reconcile-seeded orders, which carry no creation time, are never swept. The bookkeeping is pruned to still-pre-ack orders at the end of every sweep.

Sizing the two knobs

submit_ack_timeout must exceed the venue's worst-case order-visibility latency and the adapter's own submit-plus-requery budget; the doc suggests a conservative value such as 30 s for reconcile-less venues. The grace must clear the confirm's own round trip — the stated bound is 2·grace > submit_ack_timeout/2 + N·requery_timeout, with a re-query timeout of about 5 s and N = 2 for Bybit (a realtime-then-history re-query, worst case about 10 s) or 1 for OKX. A zero grace collapses the ladder back to immediate reject and reintroduces exactly the phantom-reject risk the window exists to remove.

The reject-free early rung

sweep_inflight_confirms, armed by CoreConfig::inflight_confirm (None by default, suggested live value about 2 s), re-queries an order stuck pre-ack on a fast cadence, catching a wedged adapter in seconds instead of waiting the conservative timeout for the ladder's first confirm. Three guarantees make it safe: it calls only the publish-nothing confirm_order and can strand nothing; it is band-limited to ages in [inflight_confirm, submit_ack_timeout), handing off the instant an order crosses the timeout so the two rungs never act on one order at one age; and it keeps its own dedup map, re-confirming a still-stuck order at most once per window. With the ladder off the band is unbounded and this becomes the only stuck-order signal — still reject-free, because without a configured reject timeout the core must never invent a terminal.

Managed expiry

sweep_gtd_expiry, armed by CoreConfig::gtd_sweep, cancels every resting order whose time-in-force deadline has passed, for venues that do not expire GTD or DAY orders themselves. The decision is vike_model::tif_expired, the one expiry law shared with the paper book; the terminal vocabulary differs on purpose. The paper book owns its resting order and mints OrderExpired; this sweep asks the real venue to cancel and lets the venue's authoritative OrderCanceled fold back, because publishing OrderExpired locally would desync local state from a venue order still live. Each expiry is journaled write-ahead as GtdExpire — a marker, never re-applied on replay — and the cancel is fire-once per order.

Safe-state entry

enter_safe_state is what a fold panic reaches. It logs the reason once at error, sets trading_state = Halted on the primary and every extra engine so the risk gate denies every new order, records the fault for the snapshot, and issues a best-effort cancel for every non-terminal working order tagged CancelIntent::RiskOff — the core has faulted and is pulling its whole book, so a venue metering its cancels must not hold any back. The sweep is itself guarded: a client that panics during it cannot take the loop down. What the execution core does afterwards — read for display, never traded on again until restart — is that page's subject.

The margin-call sweep and the drawdown latch

Both run once per closed bar of the engine's own series, where marks are fresh, and both read equity through the resolver the snapshot displays. sweep_margin_call runs the margin-call model per engine and submits each liquidation as a reduce_only market order through the normal gate, routed to the breaching engine by index. sweep_drawdown_latch, armed by CoreConfig::max_drawdown (None or any value at or below zero disables it), folds a high-water mark of the daemon's own equity curve and, when the drop exceeds the threshold, sets every engine to TradingState::Reducing so only reduce-only orders pass. It mints no orders, pushes one warning naming the mark, the curve and the drawdown into the recent-events ring, and never un-latches on its own — recovery is a deliberate manual SetTradingState.

A fraction of what

The curve is Σ seed_of(engine) + Σ resolved_own_pnl(engine) — configured capital plus the daemon's own realized and unrealized P&L — and deliberately not observed account equity, which on a venue-attested account is the wallet for the whole account the credentials open. Measured on prod2 on 2026-08-17: nine paper mounts at 1000 seed each plus one bybit mount carrying 53647.10600813 of a shared demo wallet latched a high-water mark near 62647, where a third party withdrawing would trip liquidate-only with no trading at all, and a genuine 25% loss on the daemon's own ~9000 of book read as 3.6%. Across a restart the peak seeds to max(capital base, curve), so the curve resumes underwater rather than forgiving earlier losses; a non-positive peak cannot express a fraction, so a profile with max_drawdown set is refused unless its seed cash is positive.

An operator facing an order that will not leave Submitted starts at triage a stuck order; the venue-side truth the ladder defers to is reconciliation.

On this page