Triage a stuck order
The core's own confirm-grace ladder, what a venue confirm can return, and which reconciliation divergence a still-missing order becomes.
"Stuck" is two different failures wearing one word, and they are triaged in opposite directions. An
order pre-ack — Initialized or Submitted, with no terminal from the venue — may or may not
exist at the venue at all. An order the venue never mentions in a reconcile pass is the reverse:
your book says it is live and the venue's report is silent. Start by asking which one you have.
The core already has a ladder
crates/vike-core/src/runtime/watchdog.rs's sweep_stuck_orders is the last-resort backstop for the
first case: an adapter that accepted submit and whose thread wedged before emitting any terminal.
Its whole design constraint is stated up front — it must never race a slow-but-real venue ack into
a phantom reject, because a synthesized OrderRejected would strand a real position, and the later
genuine OrderAccepted or OrderFilled is then an illegal transition out of Rejected and is
dropped (counted as stranded_terminal_drops in the exec fold, with the position still stranded).
So it is a ladder, not a timeout:
- Stage 1, at
created + submit_ack_timeout: the order is flagged un-acked and the venue is actively prodded — the core issues aconfirmonce and records when. Nothing is terminalized. - Stage 2, at
created + submit_ack_timeout + grace: the hard reject — unless the in-flight confirm guard defers it. A reject candidate that had a stage-1 confirm and is still pre-ack (a real ack or fill would have moved it out) has its confirm presumed in flight, so the deadline extends tocreated + submit_ack_timeout + 2·grace. Only past that, still pre-ack, is the backstopOrderRejectedsynthesized.
An order that somehow reaches the reject deadline without a prior confirm — a busy core skipped
ticks, or a clock jumped — is not rejected on the spot either: it is confirmed now and deferred, so
the invariant no backstop reject without at least one active confirm and its grace window holds.
The fold thread issues no network call itself: confirm_order is fire-and-forget onto the adapter's
own thread, and the sweep only waits for the terminal to arrive on the event lane. Reconcile-seeded
orders carry no creation timestamp and are never swept.
The watchdog is off unless something armed it
CoreConfig::submit_ack_timeout is None by default, which disables the ladder entirely. A run
profile's [guards] table arms it: submit_ack_timeout_ms is stage 1 and
submit_ack_confirm_grace_ms is stage 2's window. Leaving the grace key out leaves the caller's
value exactly as the binary built it — that struct carries no default of its own, and it used to,
which is the incident: its stale 5 s was written over the hardened 15 s default whenever the timeout
key was named, silently halving the confirm window on a live mount.
What a confirm actually does
vike_exec::ExecutionClient's confirm re-queries one order's status by client-order-id. Like every
method on that seam it is fire-and-forget: the venue's authoritative terminal returns over the ingest
lane, and the call returns nothing. Its default is a no-op — a client with no order-status
re-query does nothing, and the backstop reject still covers it — so what a confirm buys you depends
on the venue adapter overriding it.
Where one is implemented, the mapping is shared and venue-neutral
(crates/vike-bridge-core/src/rest.rs's resolve_ambiguous_submit), with three outcomes and one
rule:
- the venue has the order, live or already filled → a managed
OrderAcceptedcarrying the venue order id, with the fill following on the normal user-data lane; - the venue confirms it never landed → the true terminal
OrderRejected; - the re-query itself was inconclusive (transport error, double timeout) → an optimistic
OrderAcceptedwith no venue id, never a false terminal, because a phantom reject here would strand a position the venue actually opened.
That last arm is why the grace matters. If you tune it, the documented lower bound is that the grace
should exceed submit_ack_timeout/2 plus the adapter's re-query round trips — the halved timeout is
watchdog tick jitter, and the re-query term is the number of sequential REST calls a venue's confirm
makes (two for Bybit's realtime-then-history hop, one for OKX). A grace of zero collapses the ladder
back to immediate-reject-at-timeout, making both the window and the guard no-ops, and reinstating
exactly the phantom-reject risk they exist to remove.
When the order is not pre-ack
If the order is past Submitted and simply not moving, the watchdog will never touch it — it only
sweeps Initialized and Submitted. This is where reconciliation
takes over, and the divergence kind names your situation
(crates/vike-exec/src/recon/types.rs's DivergenceKind):
MissingTerminal— the venue's order is terminal and yours is not. Local origin, and it auto-applies underhybrid, but it resolves to no events: auto-apply is not the same as doing something.OrphanLocalOrder— a live local open order this pass's venue report never mentions. It folds nothing under any policy, deliberately: a synthesized cancel would terminalize an order that may still be resting at the venue. Its classification decides only whether a held policy surfaces it, which before the reclassification it did not —hybridfolded an empty event list and raised nothing at all.UnknownOrder— the venue reports an order with no local match, the mirror image. It quarantines with nothing to adopt unlessgenerate_missing_ordersis on, and even then adoption is narrowed to a terminal unknown order whose fills fell outside the lookback, so a live one can never book the same activity twice.
Two of these three fold nothing, whatever the policy
Which is the useful part of the diagnosis: under a policy that HOLDS the kind — quarantine for
all three, hybrid for OrphanLocalOrder and UnknownOrder — reconciliation will tell you
about a stuck order and will not clear it for you. A hybrid MissingTerminal auto-applies, so it
neither folds nor alerts, and synthesize surfaces none of the three. Cancelling it is your action,
from the trade REPL or an agent tool, and if the venue has genuinely lost it the cancel is what
proves so.
Confirming a held divergence
A held divergence appears in the published snapshot's reconciliation block as a ReconAlertView —
its kind, a detail string, and how many events are held pending confirm. It deliberately never
carries the raw proposed events; those stay fold-thread state until you confirm. You confirm by the
id that block carries, which the reconcile pass assigns monotonically within this process run
and is not stable across a journal replay.
Under quarantine every divergence waits for that confirm; under external-quarantine the
foreign-origin ones do, including a coid-less MissingFill — a venue fill echoing no client order id,
which names no local order at all. See policies.
Do not help by hand
Cancelling a stuck order in the venue's own web UI while a mount runs does not settle it quietly — it
moves the disagreement to the next reconcile pass: MissingTerminal while the venue report still
carries the now-terminal order, OrphanLocalOrder once it has aged out of that report. Neither reads
as foreign activity — crates/vike-exec/src/recon/types.rs's DivergenceKind::origin classifies
them ReconciliationMaterialized and Absence, never External — so external-quarantine treats
both exactly as hybrid does (it differs only where the origin is External), and neither folds an
event under any policy. If you must, halt first — the procedure is
stop, liquidate, restart, and the states an order can
legally reach are the order lifecycle.