RFC: Automatic Price-Band Recentering on Underlying Prices

Date: 2026-07-02

Status: Draft

Related: A-3255 (PR #2509, in flight), A-3256, A-3257 (parent A-3254); prior design doc PR #1886, stale PR #1891; stacked on PR #2413 (/underlying-prices); sibling RFC equity-reference-price-5min-twap

Author: Andrew Lee (with Claude)

When the underlying moves, our price bands don't. Today a human notices the admin dashboard turning red, looks up the underlying price somewhere (Yahoo Finance, usually), and writes it into EP3 as a preliminary settlement price — because settlement is the anchor EP3 uses for relative price limits, and rewriting it is the only recenter lever we've wired up. This RFC proposes closing the loop: use EP3's purpose-built trading reference price as the band anchor (A-3255, already in flight as PR #2509), and add a recon-engine check that pages when a band is about to bind — with a per-instrument opt-in mode that remediates automatically, recentering the anchor on a rolling TWAP of the underlying — equities first, priced off Databento. Alerting comes first and stays regardless: automatic remediation isn't always possible, and moves past a sanity cap are deliberately left to a human. Settlement goes back to meaning settlement, and nobody has to watch a dashboard card on a Sunday night.

EP3 enforces price bands as relative price limits on the instrument: PriceLimit.relative_low/relative_high (rs/ep3/api/protos/connamara/ep3/instruments/v1beta1/instruments.proto:27-37, stored as decimal fractions — 0.1 = 10%). The limits are computed by EP3 relative to an anchor price. By default the anchor is the settlement price; if a trading reference price is set on the book, it replaces settlement as the anchor (see below).

On the AX side:

• POST /admin/price-bands (rs/api-gateway/src/admin_routes.rs:964, set_price_bands) writes the relative percentages to EP3 via UpdateActiveInstrument and mirrors them into Redis (price_band_lower/upper keys, rs/sdk-internal/src/redis/redis_keys.rs:22-26).
• monitor_price_bands (rs/marketdata-publisher/src/tasks/monitor_price_bands.rs) recomputes the displayed absolute limits every second: lower = anchor × (1 − pct/100) rounded up to tick, upper = anchor × (1 + pct/100) rounded down to tick (calculate_price_band_limits, lines 89-112). Its anchor today is settlement_price only (line 63).

There is no automated recentering. The end-to-end loop is:

1. The admin GUI computes band proximity client-side: priceBandProximity() flags a market when best bid/ask is within PROXIMITY_PCT = 2% of a band (gui/packages/app/util/markets.ts:18,167). The admin dashboard shows a "Markets Near Price Bands" card that turns red (gui/apps/admin/src/components/Dashboard.tsx), and MarketsTable.tsx shows a per-row warning triangle. This is the entire alerting story — GUI-only, human-in-the-loop. There is no pager or Grafana alert.
2. An operator looks up the underlying price out-of-band (Yahoo Finance; the only Yahoo code in-repo is mock-data seeding in py/scripts/upsert_mock_candles.py, confirming the source is informal).
3. The operator opens "Set Settlement Price" in MarketsTable.tsx, enters the price with preliminary = true, which hits POST /admin/settlement-price (admin_routes.rs:1072, set_settlement_price) → EP3 UpdateSettlementPrice with SettlementPriceCalculationMethod::Override (rs/ep3/src/conversions.rs:117-147).
4. EP3 re-anchors the relative limits on the new settlement; monitor_price_bands republishes the displayed limits within ~1s.

The problems, in decreasing order of severity:

• It pollutes settlement. Settlement price drives MTM, EOD reports, estimated funding, and clearing. The settlement engine writes the real settlement daily (rs/settlement-engine/src/common.rs:176, update_ep3_settlement_price); every intraday band nudge overwrites that field with a number chosen for a different purpose.
• Human latency and coverage. The loop only closes while someone is watching the dashboard. Overnight and weekend equity-session moves (which the underlying feed covers — see below) can pin a market against its band until morning.
• Error-prone input. A hand-typed price from a browser tab, with no staleness or sanity checks.

EP3 has a first-class mechanism for exactly this. UpdateTradingReferencePrice on the InstrumentsAPI (instruments_api.proto:79-80; request at :323-338: {symbol, trading_reference_price: int64, transact_time, clear}) sets an "optional trading reference price to be used instead of settlement price as the basis for relative price limits" (doc comment, verbatim, at books.proto:80 and instruments.proto:551). EP3 reports it back on BookStats as trading_reference_price / _set / _set_time (books.proto:80-90) and publishes it over market data as trading_reference_px.

Until PR #2509, this RPC was entirely unwired in AX — generated bindings only, mock stub returning unimplemented.

This was designed a year of Fridays ago. The A-3255 design doc (PR #1886, May 2026) laid out the two-part plan: (1) wire UpdateTradingReferencePrice behind an admin endpoint and make monitor_price_bands prefer it as its displayed anchor; (2) publish underlying prices for stocks. Its stated end-state: "VWAP-driven auto-recenter becomes a thin job that calls one admin endpoint on a schedule." The first implementation (PR #1891) went stale and auto-closed; a fresh one (PR #2509, loc/a-3255-trading-reference-price) is open now and covers the admin endpoint, the ax_ep3 client wrapper, and the EP3-mock RPC — but not the marketdata-publisher precedence change.

The auto-recenter itself is ticketed as A-3257, but its design assumed a Databento trades feed + rolling VWAP computed inside the publisher. That's not what got built. What exists today (post-A-3256, renamed in PR #2413) is ax-underlying-publisher:

• Pyth Hermes streaming source for crypto (Direct), CME-blend futures (CmeBlend, roll-schedule-weighted), and US equities (UsEquitySessions, DST-aware session gating incl. overnight) — aggregated and written to ClickHouse every 10s, staleness threshold 120s (rs/underlying-publisher/src/pyth/).
• Ornn GPU-index polling source for Compute instruments, every 5 min (rs/underlying-publisher/src/ornn.rs).
• Rows land in ClickHouse underlying_prices (ReplacingMergeTree, keyed (symbol, timestamp, source)), queryable via ChUnderlyingPrice::query_for_symbol (rs/sdk-internal/clickhouse/src/schema.rs). These are points, not candles: one aggregated price per instrument per write interval (rs/underlying-publisher/src/pyth/streamer.rs:529-635 — the streamer blends the fresh legs and inserts a single (source, symbol, price, timestamp) row every 10s); any windowed statistic (median, TWAP) is derived at query time.
• Per-leg staleness health published to Redis every second and alerted through Grafana (rs/underlying-publisher/src/health.rs, configs/grafana/alerter-{demo,prod}.yaml.tpl).

The vendor landscape has also settled since: for the equity benchmark, the equity-reference-price-5min-twap RFC standardized on Databento EQUS.MINI (consolidated, intraday, zero-license) — and explicitly considered and declined Pyth for provenance. A-3257's scope should be re-cut against this reality; this RFC is that re-cut: equities first, priced off Databento, and a plain rolling TWAP instead of a trades-ingest VWAP — the time-weighted average of evenly-spaced consolidated bars, not a new streaming pipeline or volume-weighting machinery.

• Band-anchor drift is alerted on server-side and pageable — replacing the watch-the-dashboard model. Alerting exists whether or not remediation is enabled, because remediation isn't always possible.
• Instruments with an underlying feed can opt in to automatic recentering, bounded by a hard sanity cap — moves beyond it alert and wait for a human, never auto-apply.
• Settlement price is never again written for band-recentering purposes.
• The displayed band limits (price_band_lower_limit/upper_limit on the ticker) always track EP3's actual enforcement anchor.
• Rollout is per-instrument and reversible (alert-only by default; EP3's clear flag restores settlement anchoring).

• Moving band enforcement out of EP3. Enforcement stays in the matching engine; we only change what it anchors on and who updates the anchor.
• The funding/benchmark reference price. How the benchmark price for equities is computed (5-min TWAP etc.) is equity-reference-price-5min-twap; this RFC only consumes the published underlying series.
• Changing band widths. The relative percentages remain operator-set via POST /admin/price-bands.
• Bitnomial/Aiex venue bands. Those are venue-controlled and mined from rejects (rs/api-gateway/src/instruments/bitnomial.rs); nothing here applies.
• Non-equity underlyings in v1. Crypto, CME-blend, and GPU instruments keep manual anchoring until the check is extended to their existing Pyth/Ornn series — a config-level follow-up, not part of this rollout.
• Instruments without an underlying feed. They keep today's manual anchoring — though operators should use the new trading-reference dialog, not preliminary settlement.

PR #2509 lands the write path: POST /admin/trading-reference-price → ax_ep3 admin client → EP3 UpdateTradingReferencePrice, with clear support and EP3-mock coverage. This RFC treats that as a dependency and does not restate it.

What #2509 does not include is the read-side precedence from the original design: monitor_market_data must extract trading_reference_px from EP3 book stats alongside settlement, and calculate_price_band_limits must prefer it over settlement_price as the displayed anchor — mirroring EP3's own precedence so the ticker's published bounds match actual enforcement. That is milestone 2 below, and it is a hard prerequisite for auto mode: without it, every automatic recenter makes the GUI's displayed bands (and its proximity warnings) silently wrong.

"The band anchor tracks the underlying" is an invariant, and the alert-on-violation / optionally-remediate shape is exactly what recon-engine exists for: cron-scheduled InvariantChecks (rs/recon-engine/src/check.rs) over Postgres + ClickHouse (CheckContext carries both pools), with an incident.io fire/resolve sink (rs/recon-engine/src/incident_io.rs) and per-check schedules (rs/recon-engine/src/schedule.rs). A new check — underlying_price_band_drift — joins the existing ledger invariants in rs/recon-engine/src/checks/.

Inputs and one new dependency: band percentages come from Redis (price_band_lower/upper; recon-engine already depends on redis), the underlying series from ClickHouse, and the current anchor from EP3 book stats — for which recon-engine gains an ax-ep3 client (read-only for the check; the same client's admin channel, via the #2509 wrapper, performs the remediation in auto mode).

The price series (v1 = equities, Databento). The check TWAPs over underlying_prices rows in ClickHouse; v1 populates them for equity perps from Databento EQUS.MINI — 1-minute consolidated bars written as points (source = "DATABENTO") by a new source module in underlying-publisher that holds a live EQUS.MINI subscription (databento::LiveClient — the settlement runner already holds one for same-day closes) rather than polling the historical API, whose availability lag at intraday freshness is unverified. Bars arrive as the market prints them; the module follows the same durable-task/staleness pattern as the Pyth streamer, and shares client plumbing with the settlement engine's existing downloader (rs/settlement-engine/src/downloaders/databento.rs). Using the same consolidated source the benchmark work standardized on means bands recenter on the price the contract actually settles against, and the intraday-bar plumbing is shared groundwork for the 5-min-TWAP benchmark RFC and the A-3876 intraday-funding experiment. Non-equity underlyings (crypto, CME-blend, GPU) already have Pyth/Ornn series in the same table — extending the check to them later is a config change, not new machinery — but they are out of v1.

Notably not marketdata-publisher, even though it has most inputs in hand: there may be multiple marketdata-publishers in the future, and anything that writes exchange-admin state must be an obligate singleton with one clear owner — replicating a publisher must never replicate a writer. Publishing market data and remediating exchange configuration are different remits; see §Alternatives for the other placements considered.

Each enabled instrument carries a mode: alert (default) or auto. The check body is identical in both; auto additionally remediates when it's safe to. Per instrument, on the check's cron schedule (default every 60s):

The remediation never sets clear; automatic transitions back to settlement anchoring don't exist (an operator can clear manually via the A-3255 dialog).

The staleness gate doubles as the market-closed check: outside its sessions the equity publisher is intentionally idle and writes no rows, so a closed market simply never has fresh data and the check skips it. No separate market-hours calendar needed in this service.

The hard cap is the important one, and it is a cap, not a clamp: a move larger than 10% is never partially applied or walked toward — automation stands down entirely and the page stays open. A genuine >10% gap (earnings, halt, news) is precisely the situation where a human should be looking before the exchange's own risk rail gets moved; and if the underlying feed is what's broken, the 5-minute TWAP of consolidated bars dilutes single bad prints, while a fully poisoned window results in zero moves rather than a capped-but-wrong one.

A new nullable column on instruments (Postgres, db/postgres/1.sql), following the existing per-instrument config pattern (pyth_config, ornn_config, cme_future_config on DbInstrument, rs/sdk-internal/db/src/entities.rs:747):

Column absent → the instrument is invisible to the check. mode: Alert fires incidents only; mode: Auto also remediates within the guards. The check reads instrument config fresh on each scheduled run, so enabling a symbol — or flipping alert→auto — is a DB write away and needs no deploy.

• Incidents are the primary channel: check-fail fires through recon-engine's incident.io sink and auto-resolves when the check passes again (post-remediation or post-human-fix). One caveat to budget for: IncidentIoSink::from_env is env-gated and the check-failure ping is currently disabled in demo and prod — enabling delivery (at least for this check) is part of the rollout, not an afterthought.
• Structured log per action: check result (drift, anchor, smoothed), remediated (old → new anchor), stood-down (cap exceeded), skipped (reason), EP3 write failed.
• Metrics per symbol: fail count, remediation count, cap-exceeded count, skip count by reason, EP3 write failures, drift at fire time.
• The GUI proximity card stays. For an auto-remediated instrument it should approximately never be red; when it is, that itself is signal. MarketsTable.tsx should additionally show the current anchor kind (trading-ref vs settlement) and last recenter time — EP3 gives us trading_reference_price_set_time for free on book stats.

The check is stateless: the anchor and its set-time live in EP3 (BookStats), the config lives in Postgres, and the price series lives in ClickHouse. Each scheduled run re-derives everything; min_interval is enforced against EP3's trading_reference_price_set_time, so a crash-loop can't machine-gun recenters. If recon-engine is down, bands freeze at their last anchor and nothing pages from this check — which is today's behavior when no operator is watching, and is covered by recon-engine's own liveness monitoring rather than anything new here. If ClickHouse is down, all symbols skip on staleness. If EP3 is down, there's no market to protect.

Testing per the house rule on lifecycle paths: EP3-mock gains enough UpdateTradingReferencePrice state (already in #2509) to integration-test the check, including restart-with-anchor-set, EP3-disconnect mid-run, cap-exceeded stand-down, and stale-feed skip.

1. Land PR #2509 (A-3255 endpoint + manual dialog). Immediately usable by operators — manual recentering stops touching settlement.
2. Milestone 2: AX precedence in marketdata-publisher + SDK ticker field. Displayed bands now track whichever anchor is live. Verify on demo by manually setting a trading reference and watching the ticker limits move.
3. Land the Databento source in underlying-publisher: EQUS.MINI 1-minute bars → underlying_prices points for equity perps. Verify on demo against the Pyth series for the same symbols.
4. Land the check, alert-only, enabled for every equity perp, demo then prod — including making sure incident delivery is actually on (§6 caveat). This alone retires the watch-the-dashboard model and is valuable even if auto mode never ships.
5. Flip one demo equity symbol to auto, soak a week; watch remediation/skip/cap metrics against what operators would have done manually.
6. Auto on demo broadly, then prod symbol-by-symbol.
7. Retire the habit: settlement dialog relabeled; runbook updated so a band-drift page → check remediation health or recenter via the trading-reference dialog, not "type a Yahoo price into settlement". Extending the check to non-equity underlyings (Pyth/Ornn series) follows as config once the equity rollout proves out.

• Automate the preliminary-settlement write instead (no EP3 changes needed). Rejected: it institutionalizes the pollution this RFC exists to stop, and every automated write would overwrite real settlement data on a schedule.
• Periodic unconditional recenter (set anchor = underlying every N minutes). Simpler trigger logic, but constant EP3 admin writes, constant anchor churn for clients watching trading_reference_px, and no natural place for the drift-based trigger that makes the sanity cap meaningful. Drift-triggered keeps writes near zero in calm markets — and it's the same computation the alert needs anyway.
• Rolling trade-VWAP (original A-3257 scope). Right instinct, heavier than needed: it means ingesting a trades feed and maintaining volume-weighting state, when a plain time-weighted average of evenly-spaced 1-minute consolidated bars answers the same question with a avg() over ClickHouse rows. If trade-weighted centering ever matters, it slots into the smoothed computation without changing anything else.
• Pyth as the v1 equity price source. The series already exists in underlying_prices (session-gated, staleness-alerted) and would need zero new ingest. Declined for equities for the same provenance reason the benchmark RFC declined it: recenter the exchange's risk rail on the same consolidated source the contract settles against. The Pyth/Ornn series remain the path for extending the check to non-equity underlyings.
• Task in marketdata-publisher: it has most inputs in hand (band state, EP3 client, ClickHouse pool) and was this RFC's first-draft answer. Rejected because marketdata-publisher may become multiple instances, and an exchange-admin writer must be an obligate singleton — scaling out the publisher must never scale out the writer. It keeps the read-side precedence work (§1) and nothing else.
• Expanding underlying-publisher's remit: the credible runner-up — it's a singleton with the freshest prices in memory, plus the session/staleness machinery. But it has no EP3 dependency and no band/anchor state, and it would put exchange-admin credentials inside a vendor-ingest service; the ≤10s ClickHouse lag recon-engine pays instead is irrelevant at a ~60s check cadence. If recon-engine turns out to be the wrong host in practice, this is the fallback.
• Job in api-gateway (it owns the admin routes): a request-serving gateway, not a job host; it also doesn't watch EP3 book stats.
• Standalone cron/service: maximal isolation, but a fourth copy of EP3/ClickHouse/Postgres wiring for what is ~one check module inside a service purpose-built for scheduled invariant checks with alerting.
• Widen the bands so they bind less often. Orthogonal and worse: bands exist to stop fat fingers and runaway matches; widening them trades that protection for operator convenience.

1. What does EP3 do to resting orders left outside the new limits after a recenter? Cancel, leave-but-block-matching, or reject-on-touch? The protos don't say. Needs a Connamara answer or a sandbox experiment (docs/internal/operations/ep3/sandbox-reset.mdx) before prod enablement — it determines whether we owe users a heads-up in docs.
2. Should EOD settlement ever re-take the anchor? Once a trading reference is set it anchors forever (until cleared). Proposed answer: no auto-clear — auto mode keeps the anchor fresher than daily settlement ever did, and alert-mode instruments get it refreshed by the operator answering the page. But if ops wants "clear on settlement" semantics for some instruments, it's a small addition to the settlement engine's EOD flow.
3. EP3 credential scope for recon-engine — the check needs book-stats read; auto mode needs admin auth on the instruments API for UpdateTradingReferencePrice. Confirm what tier to mint, and whether alert-only deployments should carry write-capable credentials at all (ideally not).
4. Expose trading_reference_price on the public SDK ticker? Original design said yes (clients currently can't tell which anchor produced the limits). Confirm we're comfortable with it in the public protocol; mind the rs/sdk internal-leak rule when documenting it.
5. Bar statistic — BBO-mid vs bar close for the ingested points. The benchmark RFC recommends mid; matching it keeps the recenter TWAP and the funding-benchmark TWAP comparable. (Delivery mode is decided — the source module holds a live EQUS.MINI subscription, §2 — so this is the only remaining ingest question.)
6. Do we fold this into A-3257 or re-ticket? The scope diverges enough from A-3257's original framing (trade-VWAP, all underlyings at once) that a rewrite of that issue against this RFC is probably cleaner than a new ticket.