Runbook: Refreshing AIEX Bitnomial Catalogs at Contract Roll

Last updated: 2026-07-03

Every AIEX (Bitnomial-backed) service loads its instrument/product catalog once at startup and never refreshes. A single perpetual never rolls, so this is invisible today. Dated contracts roll on a schedule: a newly-listed front month, or a back month that flips to expired at Bitnomial, only reaches the running services after a restart. This runbook is the coordinated restart that carries a roll into every loader at once.

It does not cover the database-side listing or delisting itself — creating the new instruments row and retiring the old one. Do that first, per listing-a-new-product.md and delisting-a-product.md; this runbook picks up once the rows exist and only the running processes are stale.

This is the deliberate YAGNI choice for a small, known set of contracts on a predictable calendar. The refresh-vs-restart trade-off, and the point at which a distributed refresh mechanism becomes worth building, are laid out in work item 2 of ../rfc/bitnomial-expiring-contracts.md.

When to run

After listing a new dated front month. The instruments row exists but no loader has it yet.
After a contract's Bitnomial product_status flips to expired. Loaders that filter on active drop it on restart; admission should stop routing to it.
At a quarterly/monthly roll, both happen together: list the new front month and the back month expires. Run this once, after both DB changes are in.

The loaders and the lockstep invariant

Five AIEX-edition services cache a Bitnomial catalog at startup. All of them derive it from the same two sources — the Postgres instruments rows (btnl_product_config.product_id) joined against the Bitnomial REST product/specs catalog:

Service (logical)	Caches	Restart is safe because
api-gateway	public `/instruments` catalog + internal Bitnomial wiring	reload is deterministic from DB + Bitnomial
market-data publisher (`md_pub`)	product_id → (AX symbol, tick, price band)	idem; keeps only `active` products
bitnomial-order-gateway	admission/routing by symbol + product	idem
risk-engine (Bitnomial)	margins / position mapping	idem
trade-engine (Bitnomial)	instrument snapshot / multipliers	idem

Warning — restart them as one batch, not in a trickle. The market-data publisher's catalog is "kept in lockstep with the api-gateway" by design. If one service is restarted and another is not, a mid-roll skew opens: one service prices or routes a contract another has already expired (or has not yet listed). Do the whole set inside one short window.

Note — a benign filter asymmetry. The market-data publisher keeps only products whose Bitnomial product_status is active; the api-gateway lists any mapped product present in the spec catalog regardless of status. So a freshly-expired contract can still appear on /instruments (api-gateway) while having no market data (the market-data publisher) until its Postgres row is delisted. That gap — full removal and order-path blocking of expired contracts — is the delisting runbook plus expiry-lifecycle enforcement (RFC item 3), not this runbook.

Procedure

The examples target aiex-demo; the same shape applies to any AIEX environment.

Phase 1 — Preconditions: DB and Bitnomial agree

Confirm the instruments rows. The new contract's row must exist and the expiring one must still be present (delisting keeps the row).
```
just connect-db aiex-demo   # choose 1) PostgreSQL
```
```
SELECT symbol, product_id, contract_date, expiration, btnl_product_config
FROM instruments ORDER BY symbol;
```
The new front month should show a real expiration and contract_date (YYYYMMDD), and a btnl_product_config.product_id pointing at the Bitnomial contract.
Confirm Bitnomial status. Query the Bitnomial public REST spec for each affected product id:
```
GET https://<btnl-rest-base>/product/spec/{product_id}
```
The new front month must read product_status: active — not still forthcoming, or the market-data publisher will restart into nothing for it. The expiring one should read expired.

Phase 2 — Coordinated restart (single window)

Restart data/identity producers before consumers, so no consumer resolves against a map its producer has not yet republished. On the infra box:

ssh ec2-user@aiex-demo
status                 # confirm the exact compose service names first
restart md_pub         # 1. market-data publisher (BBO/tick/band by new symbol)
restart api_gateway    # 2. public catalog + Bitnomial wiring clients read
restart order_gateway  # 3. bitnomial order-gateway admission/routing
restart risk_engine    # 4. risk (margins/positions)
restart trade_engine   # 5. trade engine (multipliers/snapshot)

Warning: service names above are logical. Confirm the real compose names with status before restarting — the AIEX stack renames some services (e.g. the market-data publisher runs as md_pub).

Do all five within one short window; do not pause for verification between them.

Phase 3 — Verify each loader picked up the roll

aiex-demo logs are reachable only over SSH (logs <svc>), not the logs MCP. Each loader emits a startup inventory line listing exactly what it cached, as sorted symbol(product_id) pairs — diff these across services to confirm lockstep:

logs md_pub        | grep "loaded .* AX-mapped products"
logs api_gateway   | grep "loaded .* Bitnomial-mapped instruments"
logs order_gateway | grep "loaded .* Bitnomial-mapped instruments"
logs risk_engine   | grep "loaded .* product-id symbol mappings"
logs trade_engine  | grep "loaded .* product-id symbol mappings"

Expect:

The new dated symbol present in every list.
The expired symbol absent from the market-data publisher (the active-only filter) and, once its row is delisted, absent everywhere.
The api-gateway and order-gateway inventories otherwise identical.

Then cross-check behavior:

GET /instruments shows the new dated symbol with its real expiration.
A test order via the sims (start maker2 && start taker2) fills on the new symbol.
An order on the expired symbol rejects cleanly (no fills).
The risk snapshot shows sane margin/position for the new symbol.

Skew recovery

If one loader missed the batch (its inventory diff disagrees with the others), just restart that one service again. The load is deterministic and idempotent from DB + Bitnomial — there is no state migration and no ordering dependency beyond the producers-before-consumers preference above.

If a bad instruments row produced a skipping warning at load (mapped product absent from the catalog, non-positive tick, etc.), fix the row and re-restart the affected services.

When this runbook stops being enough

The manual coordinated restart is acceptable while AIEX lists a small, known set of contracts on a predictable calendar — rolls are scheduled events, not surprises. When contract count, roll frequency, or a no-downtime requirement makes a per-roll manual restart untenable, build the deferred refresh mechanism described in RFC item 2: a catalog-reload signal (poll product/specs on an interval, or Postgres LISTEN/NOTIFY on instruments) that every loader honors while preserving the same cross-loader lockstep. The market-data publisher already exposes the atomic-swap seam (ProductCatalog::replace_all); the order-gateway, risk, and trade engines hold their catalog immutably and would need swappable state plus a reload path first.