RFC: Live Instrument Mutation on the Read Surface

Date: 2026-07-13

Status: Draft

The product-delisting work introduced an is_closing flag on the instrument row that flips at runtime (admin sets it; the order path must freeze new orders for that instrument). It was correctly read from instruments_replica (btreemapped::BTreeMapReplica<DbInstrument, 1>), the live Postgres logical-replication view — because the other instrument store, InstrumentManager, cannot see runtime changes.

InstrumentManager (rs/sdk-internal/src/instrument_manager.rs) is a static startup snapshot: order-gateway loads it once (rs/order-gateway/src/lib.rs:230-246), wraps it in Arc<InstrumentManager>, and never mutates it (a plain HashMap<String, Instrument> with no interior mutability). It holds the venue-authoritative microstructure a matching engine owns — multiplier, tick size, price bands, trading schedule — merged with the Postgres identity map (InstrumentIds).

The result is a bifurcated read surface:

• Static, startup-era: multiplier / tick / bands / schedule, via InstrumentManager.
• Live, replicated: is_closing (and whatever lifecycle field comes next), via instruments_replica.

Every order now reads both stores — validate_order_size_and_price hits the lock-free manager for tick/size, check_instrument_not_closing takes the replica's RwLock and allocates a (String,) key for is_closing. Two concerns compound as more lifecycle state lands:

1. A growing foot-gun. Each new dynamic field forces every consumer to know which of the two stores is authoritative for it. is_closing is the second field bolted onto the replica for this reason (the first was api-gateway's additional_product_specs live-patch, rs/api-gateway/src/index.rs:88-112).
2. Cross-store inconsistency. The two stores are read at different points in time and can disagree: microstructure is frozen at process boot while is_closing is live. There is no single coherent instrument snapshot.

• live-instrument-add-without-restart.md handles live addition of instruments (grow-only, additive-apply) and explicitly lists live removal or mutation of a live instrument (margin changes, re-pointing, delisting) as a non-goal. This RFC picks up that deferred mutation case for the read surface. The two share machinery: that RFC already proposes a per-loader reconcile that re-runs the startup load and installs into a swappable InstrumentManager, and turns InstrumentIds from a set-once OnceLock into a grow-only swappable registry. This RFC rides the same reconcile loop and relaxes the additive-only rule for a restricted set of mutable lifecycle fields on already-loaded entries.
• bitnomial-expiring-contracts.md (items 3–4) and dated-futures-and-options-internal-reforms.md are the forcing functions: expiry/roll lifecycle makes intraday mutation of existing instruments routine, not exceptional.

1. A single authoritative in-process read surface for instrument state, so a consumer never has to know which store holds a given field.
2. Runtime lifecycle-field changes (starting with is_closing) reach that surface with no restart, driven by the existing Postgres replication.
3. Readers keep the current hot-path cost: lock-free and allocation-free lookups keyed off &str. Tick/size is read on every order; it must not regress onto a per-read RwLock + key allocation.
4. Readers see a coherent point-in-time snapshot — no startup-vs-live skew.

• Live addition/removal of instruments — owned by live-instrument-add-without-restart.md.
• Mutating fields with downstream settlement/margin consequences (multiplier, re-pointing). Those stay on the restart runbook until the expiry-lifecycle work enforces them end-to-end. This RFC's mutable set is deliberately narrow.
• Venue, SDK, GUI, or schema changes.

Arc<InstrumentManager> → arc_swap::ArcSwap<InstrumentManager>. Reads become .load() — a couple of atomics, no lock, no allocation, so the hot path is unchanged. Writes swap a whole freshly-built snapshot, so every reader sees one coherent point-in-time view (strictly better than today's split).

Rebuild is driven by the replica, not a second code path. A small task subscribes to the instruments_replica update stream (the BTreeUpdate broadcast the btreemapped sink already publishes) and, on change, re-runs the same enrichment the startup loader runs (Ep3InstrumentBackend / Bitnomial load_ax_mapped_catalog), then ArcSwap::stores the result. This is the "refresh = startup load, re-run" shape the add-RFC already argues for; the only addition here is that a changed field on an existing entry is applied (for the mutable set) rather than logged-and-skipped.

The mutable set is a closed, explicit list. v1: is_closing (and sibling delisting/lifecycle flags). Changes to fields with settlement or margin consequences remain a non-goal and continue to raise the add-RFC's "disagreement" alert rather than silently apply.

Concrete payoff. Once is_closing lives on the unified manager, check_instrument_not_closing reads it from the same lock-free &str lookup validate_order_size_and_price already does — the second store, the RwLock, and the per-order (String,) allocation all go away, and the two reads can no longer be inconsistent.

Shares the reconcile/registry machinery with live-instrument-add-without-restart.md; ideally landed after or alongside it so there is one reconcile loop, not two.

• Exact mutable field set for v1, and where the "safe to mutate live vs. restart-only" line sits relative to the expiring-contracts work.
• Whether the rebuild is a full re-enrich (simple, fine at hundreds of instruments) or an incremental patch of the changed rows (cheaper, more code) — start with full.
• Ordering against an in-flight order during a swap: ArcSwap gives each order a consistent snapshot for its whole admission, but an order that loaded the pre-swap snapshot admits against the old is_closing. Confirm that's acceptable (it is for a freeze — worst case is one extra order admitted a few ms after the flip, same race the replica read has today).
• Interaction with the add-RFC's grow-only InstrumentIds registry: mutation must not perturb the identity map, only the per-symbol microstructure/flags.