RFC: Live Instrument Add Without Restart (EP3 + Bitnomial)

Date: 2026-07-03

Status: Draft

Listing a new instrument today ends with a coordinated restart of every venue-facing service, on both venues. Each loader reads its catalog once at startup and never again:

The Bitnomial (AIEX) side mirrors the same shape across its five loaders — api-gateway (rs/api-gateway/src/instruments/bitnomial.rs), md-btp (ProductCatalog, whose doc comment is explicit: "No refresh — adding/removing/re-pointing instruments needs a restart, kept in lockstep with the api-gateway", rs/md-btp/src/btp/catalog.rs), bitnomial-order-gateway, risk-engine-bitnomial, and trade-engine-bn. A-4040 documented the restart-at-roll runbook and added a startup-log symbol(product_id) inventory to each loader; the Bitnomial expiring-contracts RFC (bitnomial-expiring-contracts.md, work item 2) deliberately deferred the refresh mechanism — to this RFC.

Two process-wide cells make "just reload" a non-starter as written:

• The symbol↔︎InstrumentId map is a OnceLock that may be set exactly once per process (InstrumentIds::set_global, rs/sdk-internal/src/instrument_ids.rs:50-57).
• The order-gateway's margin computer is seeded from the catalog once at startup (rs/order-gateway/src/lib.rs:251-276) with no later SetInstrumentParams source.

And one gap makes the listing flow itself unsafe to do live: admin-cli creates EP3 instruments directly in InstrumentState::Open (rs/admin-cli/src/instruments.rs:220). The moment EP3 knows the instrument, it matches orders for it — there is no window in which the instrument exists but cannot trade.

The safety-critical property this RFC needs is mostly already true, just undocumented and untested as a contract:

• Order admission rejects an unknown symbol: order_margin_effect errors when instrument_id_for_symbol or the instrument lookup misses (rs/order-gateway/src/check_margin.rs:46-55).
• Margin math refuses to run without params: MarginComputerError::MissingInstrumentParams (rs/risk-engine2/src/open_orders_margin.rs:73-80).
• trade-engine2 refuses to book a fill without a multiplier: multiplier_for bails (rs/trade-engine2/src/state_machine.rs:60-63), which kills the driver task and restarts the engine. On restart it reloads the catalog and replays from the resume token — so a fill for an unknown instrument is held (head-of-line, with the engine crash-looping until the catalog includes it), never dropped and never booked with a defaulted multiplier.

That last behavior is the accidental version of this whole RFC: today the only catalog-refresh mechanism is process restart, and trade-engine2 already uses restart-as-refresh to self-heal. The design below turns that into a deliberate, graceful mechanism.

• Postgres logical replication already streams users, api_keys, trading_accounts, account_permissions, and instruments into in-memory replicas (btreemapped::BTreeMapReplica); index_instrument_specs_task is the proof that live instrument-row updates can reach a running loader.
• EP3/cortex models the same idea: cortex's database_monitor replicates the instruments table via cortex_publication, and the internal-reforms RFC already specifies "order-gateway loads at startup and on instrument refresh" (dated-futures-and-options-internal-reforms.md).
• md-btp's ProductCatalog is already interior-mutable with a replace_all swap — refresh-ready in shape, just never called after startup.
• EP3's instrument lifecycle already has the not-yet-tradable state we need (Pending, InstrumentState in rs/ep3/src/protocol/connamara.ep3.instruments.v1beta1.rs) and an admin RPC to leave it (UpdateInstrumentStateRequest). Bitnomial's product lifecycle has the equivalent (product_status: forthcoming → active).

1. A newly-listed instrument reaches every loader with no restart, on both venues.
2. The instrument is not orderable until every engine that admits or books orders has loaded its spec — and this is provable from a fail-closed guarantee, not from deployment choreography.
3. One code path: the refresh path is the startup path, re-run. No second "diff" loader that can drift from what a restart would produce.
4. Additive only: reconcile may add instruments, never remove or mutate loaded ones.

• Live removal or mutation of a live instrument (margin changes, re-pointing, delisting). Gated on expiry-lifecycle enforcement and final-settlement mirroring (see bitnomial-expiring-contracts.md items 3–4); until then those still take the restart runbook.
• Venue-side changes. EP3 and Bitnomial behave as they do today.
• New infrastructure. No new service, datastore, or protocol; the reconcile loop reuses the queries each loader already runs at startup.

Four pieces: (1) refresh = re-run the startup load; (2) a self-paced reconcile schedule; (3) the lifecycle gate separating "in the catalog" from "open for orders"; (4) the fail-closed contract plus an adoption check. Safety comes from (4) alone; (1)–(3) are what make the system converge quickly enough that fail-closed rejects are a transient, not an outage.

Each loader gains a reconcile_instruments_task that calls the same function it already calls at startup (load_from_ep3, instrument_multipliers_from_ep3, load_ax_mapped_catalog, …) and installs the result additively into its in-memory catalog.

Why this shape and not an incremental diff protocol:

• The result of a reconcile is, by construction, exactly what a restart at that moment would have loaded. Restart and refresh converge to the same state — this is the lemma the state-space section rests on, and it is free if there is only one loader code path.
• The startup loaders already handle every partial-listing case: a Postgres row without a venue instrument is skipped (non-strict mode), a venue instrument without a row is an error. Reconcile inherits those semantics instead of re-deciding them.

Additive-apply rule. Reconcile only inserts symbols not already loaded. If the freshly-loaded catalog disagrees with a loaded entry (changed spec) or omits one (deleted row), the reconciler logs and raises an alert but does not touch the loaded entry — mutation and removal are out of scope, and a silent in-place change to a live instrument's margin or multiplier is exactly the bug class this rule exists to make impossible. (additional_product_specs keeps its existing live-patch path in api-gateway, which predates this RFC.)

Failure rule. A reconcile that errors or times out installs nothing and retries next tick. Never install a partial result; the previous catalog stays authoritative.

What "install" means per loader:

No monthly/quarterly cadence is hardcoded anywhere. Each loader polls at a pace derived from its own catalog's data:

• Baseline tick (order of minutes): re-run the load unconditionally. This is cheap — the same one DbInstrument::query_all plus one venue list call a restart makes.
• Eager tick (order of seconds): while the loader can see a reason to expect change — any loaded instrument in a not-yet-tradable state (EP3 Pending/Preopen, Bitnomial forthcoming), or any contract whose expiration / first-trading-day falls within the next baseline window — poll fast until the state settles. Contract rolls thereby drive their own refresh cadence: the data says when to look, not a calendar constant.
• Nudge (optional, deferred): the instruments logical-replication replica already delivers row inserts to api-gateway; wiring the same replica (or LISTEN/NOTIFY) into the reconciler as a wake-up makes adoption near-instant. A nudge is strictly a latency optimization — the poll is the correctness mechanism, so a dead replication slot degrades latency, never safety. Ship poll-only first.

Both ticks are bounded: a reconcile carries a timeout, and eager mode has a cap so a stuck forthcoming contract cannot hot-loop a loader forever.

Reuse each venue's own lifecycle; do not invent an AX-side state machine.

EP3. admin-cli's synchronize changes to create instruments with state: Pending instead of Open (rs/admin-cli/src/instruments.rs:220). EP3 itself rejects orders for Pending instruments, so closure is venue-enforced — no AX component needs to consult a flag. Opening is a deliberate second step: flip Pending → Open via UpdateInstrumentStateRequest once the adoption check (below) passes.

Bitnomial. The venue owns forthcoming → active; AX cannot gate it. The AX-side gate is the fail-closed rule: an instrument absent from the bitnomial-order-gateway's catalog is rejected at admission before any order reaches the venue. The listing flow inserts the AX rows while the venue product is still forthcoming wherever possible, so adoption completes before the venue starts matching. If a contract is listed late (already active), nothing breaks — clients receive rejects until adoption completes, never a defaulted margin.

Listing order (normative).

1. Insert the Postgres products/instruments rows.
2. Ensure the venue instrument exists — EP3: create as Pending; Bitnomial: the venue already lists it.
3. Wait for every loader to adopt (reconcile ticks).
4. Open — EP3: state flip to Open; Bitnomial: the venue flips active on its own schedule.

The Postgres row always precedes venue-open. This ordering is compatible with the loaders as they stand: load_from_ep3 skips a DB row with no EP3 instrument and errors on an EP3 instrument with no DB row, so at every point in the sequence each loader either loads the full spec or cleanly skips it.

The contract (normative): no engine may default a spec it has not loaded. An order for an unknown instrument is rejected before matching; a fill for an unknown instrument is held, never booked with assumed parameters. This is the entire safety argument — everything else in this RFC is liveness.

Per engine, this means pinning today's accidental behavior as tested contract:

• order-gateway admission — unknown symbol → reject (rs/order-gateway/src/check_margin.rs:46-55). Add the test.
• risk-engine2 margin — MissingInstrumentParams stays an error, and RiskState applies the same rule to fills/marks for unknown ids.
• trade-engine2 booking — the graceful version of today's crash-loop-hold: an unknown-symbol fill first triggers an immediate reconcile; only if the symbol is still unknown after a fresh load does the engine bail (that is a genuine invariant violation — a fill exists for an instrument no catalog contains — and should page). Holding is head-of-line blocking on the dropcopy stream, which is correct: booking later fills first would reorder position history.
• Bitnomial counterparts mirror each of the above.

Adoption check. Extend A-4040's startup-log inventory into a queryable surface: each loader exposes its loaded instrument set (a GET /health/instruments inventory — symbols + loaded-at). The listing flow gains admin-cli instruments await-adoption <symbol>, which polls every loader's inventory until all contain the symbol, then (EP3) flips Pending → Open, or (Bitnomial) reports ready ahead of the venue's own flip.

The adoption check is deliberately not safety-critical: if an operator flips early, or the venue opens before adoption completes, engines that haven't adopted still fail closed — rejects, not corruption. The check exists so that opening an instrument doesn't greet the first customers with a reject storm. Belt (fail-closed, per engine, enforced in code) and suspenders (adoption gate, operational, enforced in the listing flow).

The claim to prove: the order path stays closed until every engine has loaded the new instrument's spec, under any interleaving of order arrivals, per-engine reconcile ticks, and engine restarts.

Axes (independent):

• L — instrument lifecycle: L0 no Postgres row · L1 row present, venue not open (EP3 Pending / Bitnomial forthcoming) · L2 venue open (Open / active).
• E — a given engine's catalog: E− not loaded · E+ loaded. The engines that matter for safety are the order-gateway (admission; hosts the risk-engine2 margin params, so OG⁺ implies margin params present) and trade-engine2 (booking). Read-only loaders (api-gateway listing, marketdata-publisher, md-btp) are pruned: their worst case is an invisible symbol or a missing ticker — no order admitted, no money moved.
• V — event: order submission · fill delivery · reconcile tick · engine restart · open-flip.

Monotonicity lemma (prunes restarts). Per engine, catalog adoption is monotone. Reconcile only adds (additive-apply rule). A restart reloads the full current Postgres + venue set, which is a superset of anything the engine had loaded before (rows are never deleted in scope — removal is a non-goal). So the only E-transition is E− → E+, E+ is absorbing while the row exists, and restart is just reconcile with an empty prior — it can only advance adoption. "Every engine E+" is therefore a stable property: once reached, no interleaving of restarts and ticks regresses it. This lemma is what lets the grid below treat "engine restarts mid-rollout" as one row instead of multiplying every other row by it.

The surviving grid:

Walking the standard hazard questions over the two interesting cells:

• Cell 5 (the crux). Causality: nothing guarantees TE2 loads before OG admits — and nothing needs to; the hold is the answer. Authority: only the venue matches; a fill reaching TE2 proves the venue considered the instrument open, and the Postgres row must exist (listing order), so TE2's triggered reconcile finds it. Idempotency: reconcile twice = reconcile once (same load, additive apply). Adversarial timing: a fill arriving hours after the flip is unreachable if the adoption gate ran (TE2 was E+ before the flip); if an operator bypassed the gate, the hold path bounds the damage to delayed booking, and the bail-after-reload pages on the truly impossible case.
• Cell 8. An order in flight while the EP3 state flips Pending → Open lands on whichever side EP3 says it lands on. No AX component caches instrument state on the admission path today, and this RFC keeps it that way — the only openness authority is the venue.

Closure claim, assembled: rows 1–4 show no order is admitted while any admission-side engine is E−, and no fill can exist before L2. Row 5 is the only cell where an order is admitted while some engine is E−; reaching it requires the open-flip to have preceded full adoption, which the listing flow forbids and fail-closed contains. Rows 7–10 show ticks, restarts, and failures only move engines toward E+ and never install partial state. Once all engines are E+, the lemma makes that stable. ∎

Making the cells reachable in tests. Per-cell coverage needs the test harness to gate reconcile ticks and dropcopy delivery deterministically — lifecycle_hooks pause/resume (rs/sdk-internal/async/src/lifecycle_hooks.rs) already provides the primitive, and btp-mock/EP3 test doubles need a "list instrument at runtime" affordance. The two must-have integration tests: cell 5 (admit with TE2 gated E−, assert hold then adoption then booking) and cell 7 (kill and restart one engine mid-rollout, assert convergence to E+ and identical catalogs).

Explicitly unaffected: venue behavior, the InstrumentId/products identity model, TigerBeetle, and the public SDK.

1. Pin fail-closed (no behavior change): tests for the three existing choke points, plus the RiskState unknown-id rule. This makes the safety property a contract before any refresh machinery exists.
2. Registry mutability: InstrumentIds grow-only registry; additive margin-param path in order-gateway. Still no schedule — a restart still loads everything, proving the additive path against the startup path.
3. EP3 side live: reconcile tasks in the six EP3 loaders, inventory endpoints, admin-cli Pending + await-adoption + open-flip. First live add with no restart on AX.
4. Bitnomial side live: the same reconcile in the five AIEX loaders, eager-tick driven by product_status/expiry. Retires the A-4040 restart-at-roll runbook for adds; the runbook remains for removal and mutation until the expiry-lifecycle work lands.

Per the project's lifecycle-testing rule, the reconcile tasks ship only after exercising client disconnect, server-initiated disconnect, crash/restart, and recovery paths (grid rows 7 and 10 are the restart/recovery cells).

• Push-based refresh as the primary mechanism (logical replication or LISTEN/NOTIFY driving catalog updates). Rejected as the guarantee: a dead replication slot or dropped subscription silently stalls adoption, and the existing replica tasks' own failure mode ("replica stream ended prematurely") would become a listing outage. Poll is self-healing; push is kept as the optional nudge on top.
• A central catalog coordinator that loads once and pushes catalogs to every engine. Rejected: a new service, a new protocol, and a new failure domain to solve a problem each engine already solves at startup; the coordinator's crash semantics reintroduce exactly the skew it exists to prevent.
• Restart runbook forever (status quo, per A-4040). Acceptable at EP3-listing frequency; untenable at contract-roll frequency, and a restart of the order-gateway is not free — it drops live sessions and triggers cancel-on-disconnect churn for every connected client, to list one symbol.
• Versioned catalog epochs with cross-engine two-phase commit (every engine acks epoch N before any engine serves it). Rejected: distributed coordination to obtain a guarantee the fail-closed rule already provides per-engine, with none of the availability cost.

• Poll intervals (baseline and eager) per loader — pick at implementation time; the constraint is query cost on Postgres and the venue REST APIs, both of which already absorb one full load per service restart today.
• Inventory surface: HTTP health endpoint vs a Redis key per service. Leaning endpoint (no new dependency for services that don't touch Redis; admin-cli can reach all of them).
• Should await-adoption auto-flip EP3 to Open or stop at "ready" and leave the flip to a human? Leaning auto-flip behind an explicit --open flag.
• When cortex promotes risk-engine2 to a standalone binary, its database_monitor task becomes that engine's reconciler — confirm the additive-apply rule carries over there.