vike-datahub

The headless compute-to-data backtest server — a profile in, a report out, with the history never crossing the wire.

A headless, compute-to-data backtest server. A client sends a small vike_backtest::harness::BacktestProfile config over a TCP socket; the server runs the existing vike-backtest engine right next to the data and returns just the vike_backtest::harness::BacktestReport answer. The heavy history never crosses the wire — only the profile in and the metrics summary out. The crate owns no engine of its own: it is assembly of vike-backtest for the engine, config and report, and vike-data for the store seam (crates/vike-datahub/src/lib.rs).

usage: vike-datahub

The headless compute-to-data backtest server. It takes no options: the listen address and the
hist-store root come from the environment.

  -h, --help      print this and exit 0
  -V, --version   print the version and exit 0

Transport and backends

Blocking std::net — TcpListener/TcpStream, thread-per-connection — with length-prefixed serde_json frames from vike_datahub_client::proto. There is no async runtime and no second HTTP/RPC/WS stack: the workspace deny.toml bans one, and this stays inside bare std::net plus serde_json. The server holds the store as Arc<dyn HistStore + Send + Sync>, so the same serve runs over DataFusionHist in production and over the in-memory MemHistStore double in tests.

The wire protocol, the thin DatahubClient and RemoteHistStore (a HistStore served over this protocol) live in the light vike-datahub-client crate, which is DataFusion-free so the GUI can link it without the Arrow/DataFusion tree; it is the one home of the wire schema, and every consumer — this server included — imports it directly.

A default build does not serve

The real DataFusionHist-backed server is compiled only under the serve-datafusion feature, which is what makes the concrete backend nameable. A default build compiles an inert stub main that prints a rebuild hint and exits, so cargo build -p vike-datahub always produces a runnable binary — and that build is DataFusion-free. backfill-serve additionally compiles the real collector table and makes serve advertise and answer Request::Backfill; it requires serve-datafusion, because the collectors write a concrete DataFusionHist.

--help and --version are answered in both builds, identically, before anything else. This binary parsed no argv at all once: under the feature it went straight to opening a store and binding a listener, so vike-datahub --help started a server; without it, every invocation printed the missing-feature message and exited 2. Help is a question about the command line, which both builds have; the missing backend is a question about running, which is why only the run path still fails (crates/vike-datahub/src/datahub_cli.rs).

Where it listens

VIKE_DATAHUB_ADDR is the listen address, default 127.0.0.1:7878.

A non-loopback bind is refused twice over

A non-loopback address is refused at startup unless VIKE_DATAHUB_ALLOW_PUBLIC_BIND is set to the exact string 1 and at least one node key is configured. The opt-in alone used to be enough behind a warning; that is the state crates/vike-datahub/src/datahub_cli.rs's header names as the one the refusal exists to prevent — the opt-in consents to being reachable, and is not consent to serve a store write and a Rhai compiler unauthenticated. The guard did not soften when authentication landed: the handshake is plaintext and authenticates the connection, not each frame, so an SSH tunnel is still what supplies confidentiality and integrity.

Authentication — the credential is the switch

Two scoped HMAC node keys live in the credential store (<project>/settings/secrets.env): VIKE_DATAHUB_OBSERVE_KEY and VIKE_DATAHUB_CONTROL_KEY. Their absence is the switch. With neither set — the default — the server authenticates nothing and behaves exactly as every build before them did: Hello is optional and informs rather than gates, every verb is served to whoever can reach the socket, and the Welcome bytes are identical. That is the same credential-is-the-gate idiom the venue mounts use, pointed at a server, and it is why that mode is confined to a loopback bind.

With either key set the handshake becomes mandatory. Hello must be the first frame; the Welcome answering it carries a fresh 32-byte per-connection nonce; a Request::Auth carrying the HMAC over that nonce must follow; and every subsequent verb is checked against the authenticated scope. Any verb before AuthOk is refused and the connection closes.

crates/vike-datahub/src/server.rs's required_scope is the one authority for which verb needs which scope, and its match is exhaustive — no _ arm — so a new wire verb cannot be added unclassified; it fails to compile instead. Scope::Observe covers the history and catalog reads (LoadBars, ScanQuotes, ScanTrades, PropertiesAsOf, ListSeries, Inventory, SeriesGaps, Coverage, ListStrategies) and Ping — a liveness probe is Observe rather than a handshake frame, because answering one unauthenticated hands out a free oracle for "is this address a datahub". Scope::Control additionally admits the Backfill store write and every Run* verb — those compile client-supplied Rhai server-side, so they are not reads whatever they return.

Connection hygiene

A read error or clean EOF ends only that connection's loop, never the accept loop; each connection runs on its own thread, so even a panic inside a handler drops one socket rather than taking down the server; and every request produces a response, so a client always learns the outcome. Framing and decoding are split, so a well-framed body that fails to decode into a known Request — an unknown verb from a newer client — is answered with an error and the loop continues: one bad request is a bad request, not a bad connection. Each accepted stream gets a generous IDLE_READ_TIMEOUT, and a timeout on an idle or half-open peer closes the connection rather than resuming the loop, which would desync the stream.

The rest of the environment

VIKE_DATAHUB_STORE names the hist-store root; unset, it falls through the shared precedence resolve_store_root_from documents at its call site — the CWD-relative market_data/hist this line used to name as the last resort is gone, and it was the whole bug (launching from anywhere but the repo root created an empty store beside the shell). VIKE_USER_DATA_DIR is the user-content root whose indicators/ this server installs, falling back to the project walk from the working directory; a client script resolves against the set this box installs.

Under serve-datafusion the RunSlice boundary (crates/vike-studio-core/src/wire_run.rs) is compiled too — it is the only place this crate names vike-studio-core, and the parity test drives the same run_slice_local the server arm does, keeping "local" and "over the wire" one computation. A default build answers RunSlice with a clean error instead.