vike
Guides

Triage a stuck order

The core's own confirm-grace ladder, what a venue confirm can return, and which reconciliation divergence a still-missing order becomes.

"Stuck" is two different failures wearing one word, and they are triaged in opposite directions. An order pre-ack — Initialized or Submitted, with no terminal from the venue — may or may not exist at the venue at all. An order the venue never mentions in a reconcile pass is the reverse: your book says it is live and the venue's report is silent. Start by asking which one you have.

The core already has a ladder

crates/vike-core/src/runtime/watchdog.rs's sweep_stuck_orders is the last-resort backstop for the first case: an adapter that accepted submit and whose thread wedged before emitting any terminal. Its whole design constraint is stated up front — it must never race a slow-but-real venue ack into a phantom reject, because a synthesized OrderRejected would strand a real position, and the later genuine OrderAccepted or OrderFilled is then an illegal transition out of Rejected and is dropped (counted as stranded_terminal_drops in the exec fold, with the position still stranded).

So it is a ladder, not a timeout:

  • Stage 1, at created + submit_ack_timeout: the order is flagged un-acked and the venue is actively prodded — the core issues a confirm once and records when. Nothing is terminalized.
  • Stage 2, at created + submit_ack_timeout + grace: the hard reject — unless the in-flight confirm guard defers it. A reject candidate that had a stage-1 confirm and is still pre-ack (a real ack or fill would have moved it out) has its confirm presumed in flight, so the deadline extends to created + submit_ack_timeout + 2·grace. Only past that, still pre-ack, is the backstop OrderRejected synthesized.

An order that somehow reaches the reject deadline without a prior confirm — a busy core skipped ticks, or a clock jumped — is not rejected on the spot either: it is confirmed now and deferred, so the invariant no backstop reject without at least one active confirm and its grace window holds. The fold thread issues no network call itself: confirm_order is fire-and-forget onto the adapter's own thread, and the sweep only waits for the terminal to arrive on the event lane. Reconcile-seeded orders carry no creation timestamp and are never swept.

The watchdog is off unless something armed it

CoreConfig::submit_ack_timeout is None by default, which disables the ladder entirely. A run profile's [guards] table arms it: submit_ack_timeout_ms is stage 1 and submit_ack_confirm_grace_ms is stage 2's window. Leaving the grace key out leaves the caller's value exactly as the binary built it — that struct carries no default of its own, and it used to, which is the incident: its stale 5 s was written over the hardened 15 s default whenever the timeout key was named, silently halving the confirm window on a live mount.

What a confirm actually does

vike_exec::ExecutionClient's confirm re-queries one order's status by client-order-id. Like every method on that seam it is fire-and-forget: the venue's authoritative terminal returns over the ingest lane, and the call returns nothing. Its default is a no-op — a client with no order-status re-query does nothing, and the backstop reject still covers it — so what a confirm buys you depends on the venue adapter overriding it.

Where one is implemented, the mapping is shared and venue-neutral (crates/vike-bridge-core/src/rest.rs's resolve_ambiguous_submit), with three outcomes and one rule:

  • the venue has the order, live or already filled → a managed OrderAccepted carrying the venue order id, with the fill following on the normal user-data lane;
  • the venue confirms it never landed → the true terminal OrderRejected;
  • the re-query itself was inconclusive (transport error, double timeout) → an optimistic OrderAccepted with no venue id, never a false terminal, because a phantom reject here would strand a position the venue actually opened.

That last arm is why the grace matters. If you tune it, the documented lower bound is that the grace should exceed submit_ack_timeout/2 plus the adapter's re-query round trips — the halved timeout is watchdog tick jitter, and the re-query term is the number of sequential REST calls a venue's confirm makes (two for Bybit's realtime-then-history hop, one for OKX). A grace of zero collapses the ladder back to immediate-reject-at-timeout, making both the window and the guard no-ops, and reinstating exactly the phantom-reject risk they exist to remove.

When the order is not pre-ack

If the order is past Submitted and simply not moving, the watchdog will never touch it — it only sweeps Initialized and Submitted. This is where reconciliation takes over, and the divergence kind names your situation (crates/vike-exec/src/recon/types.rs's DivergenceKind):

  • MissingTerminal — the venue's order is terminal and yours is not. Local origin, and it auto-applies under hybrid, but it resolves to no events: auto-apply is not the same as doing something.
  • OrphanLocalOrder — a live local open order this pass's venue report never mentions. It folds nothing under any policy, deliberately: a synthesized cancel would terminalize an order that may still be resting at the venue. Its classification decides only whether a held policy surfaces it, which before the reclassification it did not — hybrid folded an empty event list and raised nothing at all.
  • UnknownOrder — the venue reports an order with no local match, the mirror image. It quarantines with nothing to adopt unless generate_missing_orders is on, and even then adoption is narrowed to a terminal unknown order whose fills fell outside the lookback, so a live one can never book the same activity twice.

Two of these three fold nothing, whatever the policy

Which is the useful part of the diagnosis: under a policy that HOLDS the kind — quarantine for all three, hybrid for OrphanLocalOrder and UnknownOrder — reconciliation will tell you about a stuck order and will not clear it for you. A hybrid MissingTerminal auto-applies, so it neither folds nor alerts, and synthesize surfaces none of the three. Cancelling it is your action, from the trade REPL or an agent tool, and if the venue has genuinely lost it the cancel is what proves so.

Confirming a held divergence

A held divergence appears in the published snapshot's reconciliation block as a ReconAlertView — its kind, a detail string, and how many events are held pending confirm. It deliberately never carries the raw proposed events; those stay fold-thread state until you confirm. You confirm by the id that block carries, which the reconcile pass assigns monotonically within this process run and is not stable across a journal replay.

Under quarantine every divergence waits for that confirm; under external-quarantine the foreign-origin ones do, including a coid-less MissingFill — a venue fill echoing no client order id, which names no local order at all. See policies.

Do not help by hand

Cancelling a stuck order in the venue's own web UI while a mount runs does not settle it quietly — it moves the disagreement to the next reconcile pass: MissingTerminal while the venue report still carries the now-terminal order, OrphanLocalOrder once it has aged out of that report. Neither reads as foreign activity — crates/vike-exec/src/recon/types.rs's DivergenceKind::origin classifies them ReconciliationMaterialized and Absence, never External — so external-quarantine treats both exactly as hybrid does (it differs only where the origin is External), and neither folds an event under any policy. If you must, halt first — the procedure is stop, liquidate, restart, and the states an order can legally reach are the order lifecycle.

On this page