RFC: Order Gateway Latency on the Admin Console

Date: 2026-09-17

Status: Draft

Scope: surfacing the order-gateway latency series as min / p50 / p99 timeseries on the admin console homepage. Builds on the as-built record Order gateway latency statistics and is the admin-console counterpart to the Grafana deliverable in Performance Tracking & Regression Protection. EP3 edition only; btnl-order-gateway follows once it emits equivalent series.

The order gateway now records six latency histograms and exports them to the observability ClickHouse. The admin homepage shows none of them: every number on it comes from the exchange ClickHouse, Postgres, or Redis, and api-gateway has no connection to the observability cluster at all.

Wiring a chart to the data as it stands would produce a misleading chart. Two problems at the source have to be fixed first:

1. The buckets are too coarse to separate p50 from p99. On ax-demo today, 100% of og_ws_order_to_ack_ms samples land in two buckets (5–10 ms and 10–25 ms). Every minute of the last hour has p50 and p99 in the same bucket, so both lines would be interpolations inside 10–25 ms and move only with the bucket's fill ratio.
2. There is no min. A Prometheus histogram carries no minimum; the Min/Max columns of otel_metrics_histogram are always 0.

The plan has three parts:

Nothing is written to the exchange database, and the gateway's hot path gains one lock-free record call per sample. The as-built's three rules still hold.

gui/apps/admin/src/components/Dashboard.tsx makes one call, useExchangeStats(timeframe, { groupBy: 'day' }) → GET /api/admin/exchange-stats, and renders MetricCard tiles plus two recharts bar charts (TradingVolumeChart, DailyFillsChart). Timeframe tabs are 1h / 24h / 30d / YTD. There is no polling; the query runs once per mount or tab change.

The handler (rs/api-gateway/src/admin_routes.rs, get_exchange_stats) reads the exchange ClickHouse (AppState.klickhouse), Postgres, and Redis tickers. Request and response types live in rs/sdk-internal/src/protocol.rs. Auth is by router: anything registered in admin_routes::read_router gets authorize_admin.

api-gateway has exactly one ClickHouse pool, configured by CLICKHOUSE_*. No Rust service reads the observability cluster. The only consumers of otel_metrics_* are Grafana (configs/grafana/) and the alerter, and neither queries otel_metrics_histogram. The admin /monitor page is three external links.

Facts that shape the design, checked against the live ax-demo cluster:

• Rows are cumulative (AggregationTemporality = 2). Each 15 s row holds process-lifetime BucketCounts (per-bucket, not running totals). A window's distribution is the difference between two rows. StartTimeUnix changes when the gateway restarts.
• One gateway per environment (container_name plus host networking rule out a second), so there is one service.instance.id. The histograms stay mergeable if that changes.
• Table TTL is 30 days. ClickHouse is 26.4.
• About 40 histogram series exist in total on ax-demo, across all services.
• ax-prod runs 16.0.0 and emits none of the six series yet, only the pre-rename og_order_to_ack_ms.
• Every other og_* distribution is already exported as a rolling summary with quantiles 0, 0.5, 0.9, 0.95, 0.99, 0.999, 1 (otel_metrics_summary), because a metric without configured buckets renders that way. The Grafana order-gateway dashboard reads those.

Per-minute bucket deltas for og_ws_order_to_ack_ms{req="place",result="acked"} on ax-demo, 2026-09-17 17:13–17:18 UTC:

The as-built states the limit plainly: a quantile's error bound is the width of its bucket. A 15 ms-wide bucket is fine for "are we in the right decade" and useless for a chart whose purpose is to show p99 pulling away from p50.

• Min, p50, and p99 for the key latency series, as timeseries and as a headline number, on the admin homepage, following the homepage's timeframe tabs.
• Numbers that are correct under re-aggregation: the 30-day headline p99 is the p99 of 30 days of samples, not an average of per-minute p99s.
• A stated error bound on every number shown.
• The homepage keeps working, minus this section, when the observability cluster is down or unconfigured.

• Replacing Grafana. Label breakdowns, ad-hoc ranges, and the other ~30 og_* metrics stay there.
• A generic metrics query API in api-gateway.
• Client round-trip time. The gateway cannot see it (as-built, "Limits").
• Alerting on latency. The alerter can adopt the same SQL later.
• Per-account or per-session latency. The series have no such labels by design.

Replace GATEWAY_INTERNAL_BUCKETS_MS, ORDER_PATH_BUCKETS_MS, and DELIVERY_BUCKETS_MS in rs/order-gateway/src/in_situ_latency_measurement.rs with one generated set: four buckets per octave (ratio 2^¼ ≈ 1.19) from 0.05 ms to 30 s, about 78 bounds.

• Worst-case error of a linearly interpolated quantile is half a bucket, about ±9% of the value. Around 15 ms the bucket is 2.6 ms wide instead of 15 ms.
• One set for all six series removes the per-series range judgement and lets any two series be compared bucket for bucket.
• Cost: /metrics grows by about 3,000 lines for the current ~40 label sets, scraped every 15 s. In ClickHouse the row count is unchanged; BucketCounts grows from 16 to 79 elements and compresses well because most are zero.
• Recording cost does not change. The exporter buckets samples at render time, not on the hot path.

A bound change is safe for readers. It ships in a release, the restart changes StartTimeUnix, and the query never differences rows across a StartTimeUnix boundary. The query groups by ExplicitBounds, so a range that spans the change comes back as two histograms. The endpoint does not rebucket one onto the other. For the one step that contains both, and for the range headline, it uses the newer set only.

Alternative considered: OTLP exponential histograms. Better resolution per byte, and the collector already has the table. The metrics Prometheus exporter cannot emit them, so this means replacing the exporter for one service. Not worth it for this feature.

Record each of the six series a second time under <name>_window with no configured buckets. The exporter renders it as a rolling summary (60 s window), which lands in otel_metrics_summary with quantile 0 (min) and quantile 1 (max).

Only those two values are used. Min-of-mins and max-of-maxes are exact under any re-aggregation (over time, labels, or replicas), which is the property that the summary's p50/p99 lack and the reason those still come from the histogram.

• Cost: a second lock-free push per sample, the same operation the histogram record already performs. The perf RFC's probe budget (≤ ~200 ns per order, ≤ 2%) applies; extend benches/ack_path.rs when it exists, or land a before/after criterion run with this change.
• The 60 s rolling window smears a min by up to a minute at the edges of a step. At 1-minute steps a min can appear in two adjacent points. Acceptable; documented on the chart tooltip.
• The HistogramSpec table grows a second name per entry so the twin cannot drift from its histogram. describe_metrics covers both.

Alternative considered: read min from the lowest non-empty bucket. No second record, but the value is a bucket bound, so the min line is a staircase with ~19% steps and is always an underestimate. Rejected because min is the one statistic where operators read the absolute value ("what is our floor").

Alternative considered: an atomic min per series, swapped to +∞ when /metrics renders, exported as a gauge. Exact per scrape window, but any second reader of /metrics (a curl during an incident) silently steals a window. Rejected.

Optional config read from the O11Y_CLICKHOUSE_* variables, which are already provisioned on ax-prod. The config struct's fields follow the existing variable names rather than introducing new ones; it needs the HTTPS endpoint (port 8443, the PrivateLink host the collector also uses; api_gateway runs on the same host with host networking), a username, and a password. The credentials must be read-only. If the provisioned user can write, swap it for one granted SELECT on default.otel_metrics_histogram and default.otel_metrics_summary before the endpoint ships.

The client is reqwest against the ClickHouse HTTP interface with FORMAT JSONEachRow, parameters bound with param_* query arguments, readonly=1, and max_execution_time=5. klickhouse is not reused: the exchange pool speaks the native protocol without TLS, and a handful of read-only aggregate queries do not justify a second native pool.

When the config is absent, AppState.o11y_clickhouse is None. This keeps local dev, CI, and the AIEX edition unchanged.

Failure isolation. The observability cluster is a separate vendor-hosted system with weaker availability expectations than the exchange. The new endpoint is the only code that touches it, it is a separate request from exchange-stats, and it has a 5 s timeout. A failure degrades one homepage section.

Alternative considered: have the gateway write per-minute rollups to the exchange ClickHouse, so the homepage reads them through the existing pool. Rejected. It contradicts the as-built's rule that these numbers use the observability pipeline, it needs a second in-process recorder because the Prometheus handle only renders text, and it adds an insert path to the most latency-sensitive service to feed a dashboard.

Alternative considered: embed Grafana panels. Grafana is Tailscale-only and anonymous; the admin console is Clerk-authenticated and IP-allowlisted. Bridging the two is more work than one endpoint, and the result would not look or behave like the rest of the homepage.

GET /admin/latency-stats, registered in admin_routes::read_router. Types in rs/sdk-internal/src/protocol.rs:

The server owns the catalog. LatencySeriesKey is an enum; each variant maps to a metric name and a label filter:

ws_place_to_ack minus og_ws_order_to_ep3_ack_ms (gateway delivery) and the REST series are left to Grafana for now. Adding a key is a one-line catalog change plus a GUI title.

Behaviour:

• The server picks the step from the range so a response is at most ~200 points per series: ≤ 2 h → 60 s, ≤ 36 h → 15 min, otherwise 6 h.
• A range reaching past the 30-day TTL is clamped and retention_clamped is set. The GUI's YTD tab shows 30 days with a note.
• window is computed from the merged buckets of the whole range. Points with count = 0 have None statistics so the chart draws a gap, not a zero.
• Responses are cached in-process for 15 s per (start bucketed to the step, end bucketed to the step). The source only changes every 15 s, and this bounds cluster load at one query set per 15 s regardless of how many admins have the page open.
• No config → 503 with the standard ErrorResponse. Query failure → 502. Errors are classified by typed reqwest/status errors, not message text.

One query per call for the histogram side, one for the summary side. The histogram side, for one catalog entry:

• The range is widened by one step so the first in-range step has a predecessor to difference against. A process born inside the range differences against zero instead.
• Partitioning by StartTimeUnix makes restarts safe. As a guard against a missed reset, a step whose delta has any negative element is dropped and logged.
• sumForEach merges label sets (and replicas, should they exist).

Quantiles are computed in Rust from buckets and ExplicitBounds: linear interpolation inside the bucket, lower edge 0 for the first bucket, the last finite bound for the overflow bucket. This is ~30 lines, is where the "newer bounds win" rule lives, and is unit-tested against known distributions. quantilePrometheusHistogram is available on 26.4 but expects running-total buckets and would still need the delta step; doing the last step in Rust keeps one tested implementation for both points and window.

The summary side takes, per step, min of quantile 0 and max of quantile 1 from otel_metrics_summary for <name>_window, ignoring rows whose window was empty.

The histogram query was run by hand against ax-demo for 15-minute ranges and returns the expected per-minute deltas; it has not been timed. A 30-day range reads about 170k rows per label set. If that proves slow, see §7.

• gui/packages/admin/src/api/admin.ts: fetchLatencyStats(timeframe).
• gui/packages/admin/src/hooks/useLatencyStats.ts: react-query, key ['latency-stats', timeframe], refetchInterval: paused ? false : 15000 through usePausedPolling for the 1h and 24h tabs, no polling for 30d/YTD.
• gui/apps/admin/src/components/LatencySection.tsx, rendered by Dashboard.tsx below the existing charts and sharing its timeframe state:
  • Four MetricCard tiles. Headline is window.p50_ms; beneath it p99 and min as secondary numbers, laid out like the count/notional pair on the deposit tile; sample count as the caption. Sub-millisecond values render in µs.
  • One recharts LineChart for the selected tile (default ws_place_to_ack) with min, p50, and p99 lines, log-scale y-axis, connectNulls={false}. Clicking a tile selects it. Chart colours and the tooltip follow TradingVolumeChart.
  • The tooltip states the error bound: "p50/p99 ±9% (histogram); min exact, 60 s window".
• 503 hides the section. 502 shows the section with an inline "latency data unavailable" state. Neither affects the rest of the homepage.
• Hidden on the AIEX edition through isSectionVisible() until that gateway has the series.
• Check @architect-xyz/ui-components and @architect/ui for an existing duration formatter before writing one.

Phases 1 and 2 are independent. Phase 4 can be built against demo as soon as 1–3 are there.

Disconnect and restart paths to test before prod: gateway restart mid-range (new StartTimeUnix), gateway down for a whole step (gap, not zero), collector down (gap), observability cluster unreachable (502, homepage otherwise intact), config absent (503, section hidden).

Deferred until either the 30-day query is too slow or someone needs more than 30 days.

A refreshable materialized view in the observability cluster computes the §5.5 deltas once a minute into og_latency_1m (t, metric, attributes, bounds, buckets, min_ms, max_ms) with a 13-month TTL. The endpoint switches its FROM; nothing else changes, and YTD becomes real. The observability schema is created by the collector and is not under Atlas, so this needs a home for its DDL first (see open questions).

1. Prod dependency. Is a read-only dependency from api-gateway to the vendor-hosted logs cluster acceptable on ax-prod, given the isolation in §5.3? The alternative is the rejected gateway-writes-rollups design.
2. Bucket density. Four per octave (±9%, 78 bounds) or eight (±4.4%, 155 bounds)? The exposition and storage cost doubles; nothing else changes.
3. Headline statistic. p50 as the tile's big number with p99 beneath, or p99 as the headline because that is what the market-maker conversation is about?
4. DDL home for the observability cluster. Needed for the §7 rollup, and for any read-only grant phase 3 turns out to need. Proposal: db/observability/*.sql, applied by hand, outside Atlas.
5. REST on the homepage. og_rest_request_to_response_ms is left out of the catalog because route × status makes "the" REST number ambiguous. Add rest_place_order (route /place-order, status 200) if REST order flow is material.
6. AIEX. btnl-order-gateway has no equivalent series. Same names with a btnl_og_ prefix (as the perf RFC suggests) would make the catalog edition-keyed rather than duplicated.