RFC: Performance Tracking & Regression Protection

Date: 2026-07-02

Status: Draft

Scope: order-path latency measurement (order-gateway, aiex-order-gateway) and a regression-gate harness. Co-leads: Loc Nguyen · Tin Chung.

Project: Performance tracking & Regression protection

A market maker reports 50–75 ms order response times and asked if we can do better. Today we can't confirm, refute, or attribute that number — the request→ack envelope and the time spent waiting on the venue are measured nowhere. Past performance leaks (EP3 drop copies, ClickHouse inserts, race conditions) were each found in production, by symptom. This RFC supplies the missing instrumentation to answer the client question and a harness to catch the next leak before prod.

It deliberately builds less than the last attempt. PR #1955 (ax-perf-harness, A-3358) prototyped a 65-file load-test framework — Stack/Session/Recorder traits, a TUI, saturation-knee detection, capacity binary-search. It was auto-closed by the stale-bot: a single mega-PR that no qualified reviewer ever engaged with. We keep three of its ideas (coordinated-omission-aware latency, server-histogram reconciliation, baseline diff) and drop the rest of its scope. The framework was over-built for the two questions that actually matter.

Q1 — "Is the MM's 50–75 ms real, and where does it go?" Answered by passive measurement, no harness. Instrument the real order path and read it off production traffic. No synthetic load, no blast radius. This is M0/M1.

Q2 — "Will the next dropcopy/ClickHouse-class leak get caught before prod?" Answered by a thin local harness. A deterministic in-process boot is the only place with a noise floor tight enough to gate regressions. This is M2–M5.

The two never mix: production is measured, never load-tested.

The harness accepts --target dev|local only. There is deliberately no --target prod, and therefore no abort-switch / isolated-firm-on-prod machinery to build — synthetic load never reaches prod, so the safeguards that would gate it are unnecessary.

Why a Rust harness, not k6/drill, for the order path. The order path is WebSocket JSON-RPC (with EP3 gRPC / Bitnomial BTP behind the gateway). The harness must (a) boot the real gateway in-process for the deterministic regression gate and (b) speak the exact wire protocol and auth that clients use. drill is HTTP-only and cannot open the order WS at all. k6 can drive WebSocket, but you would hand-reimplement the AX auth handshake and the JSON-RPC order protocol in JavaScript and re-sync it every SDK release. The Rust harness reuses rs/sdk types directly, so it cannot drift from the protocol it measures.

Why drill, not custom code, for REST. api-gateway and onboarding are plain HTTP. drill is a single-binary Rust load tester driven by YAML — it matches the stack, costs nothing to maintain, and runs cheaply in CI. Custom Rust would be pure duplication here. (k6 is held only as a fallback if WS/gRPC breadth coverage is later needed — drill can't do those.)

Why the harness stays small. It is the existing OgTestStack::boot() (rs/order-gateway/tests/integration_tests.rs) plus a fixed-rate load loop, a /metrics scrape, and a baseline compare. No Stack/Session/Recorder framework, no TUI, no knee detection. Those were what made #1955 unreviewable.

• Server ack histogram. In code today as og_order_to_ack_ms (label result="accepted"), emitted in rs/order-gateway/src/lib.rs, bucketed via ORDER_ACK_BUCKETS_MS in rs/order-gateway/src/main.rs, exposed at /metrics in rs/order-gateway/src/rest_service.rs. The project milestone M0 refers to ax_og_order_ack_seconds (the reverted PR #2360). These are the same metric under two names; M0's first task is to reconcile them into one canonical name rather than land a second histogram measuring the same span. The Bitnomial edition emits the mirror family btnl_og_* in rs/btnl-order-gateway/.
• Full-stack in-process boot OgTestStack::boot() — testcontainers (Postgres/ClickHouse/Redis)
  • Ep3Mock + the real ax_order_gateway::run() on ephemeral ports, seeded. The harness reuses this; §5 extracts it so the harness and the integration test share one boot path.
• Metrics scrape-and-parse pattern test_order_ack_histogram_metrics in rs/order-gateway/tests/sections/test_ws_order_events.rs.
• Client-side p50/p95/p99 latency report record_latency / print_latencies in rs/admin-cli/src/cancel_replace_by_cid_probe.rs.
• Venue mocks rs/ep3-mock (gRPC), rs/btp-mock (raw TCP). Criterion benches rs/order-gateway/benches/. Grafana provisioning configs/grafana/. Edition switch EXCHANGE_EDITION / AX_MATCHING_ENGINE in rs/sdk-internal/src/edition.rs.

New crate rs/perf-harness (ax-perf-harness), subcommands load / baseline / report.

Boot reuse. Extract the reusable core of OgTestStack::boot() into ax-test-utils (rs/test-utils/src/harness_boot.rs, re-exported from lib.rs), returning a BootedExchange handle. The existing integration test and the harness then call the same code — the single highest-leverage refactor, because it stops the harness's boot from drifting from the test's.

Load generation. An open-loop fixed-rate generator (tokio::time::interval with MissedTickBehavior::Skip, the pattern already in rs/sdk-examples/src/bin/taker2.rs) — send cadence is decoupled from ack receipt so a slow server surfaces queueing delay instead of silently throttling the offered rate. Latency is measured from each tick's scheduled send time, not the actual send instant: if the sender itself falls behind under load, that queueing delay is captured rather than hidden. This is the coordinated-omission fix carried over from #1955 — measuring elapsed() from the actual send would silently omit exactly the tail we care about. Each order carries its scheduled-send timestamp keyed by ClientOrderId; a reader task matches OrderAcked and records the delta into an HDR histogram.

Concrete --rate / --sessions values in this RFC are illustrative; M2 fixes the standing load level from the observed prod peak order rate (via the api-gateway order-rate metric) so the gate exercises a realistic envelope rather than an arbitrary one.

Dual measurement. Client latency (network + gateway + venue-mock round-trip) and the server og_order_to_ack_ms histogram (gateway-internal request→ack only) measure different spans by construction. Report both; assert the invariant client_p99 >= server_p99; assert og_order_to_ack_ms_count{result="accepted"} equals orders sent (drop detection). The server histogram is the regression signal of record — it isolates the gateway from harness and network noise; client latency is the SLA-facing sanity check. They are never averaged together.

Edition. An AX target boots ep3-mock + ax_order_gateway::run() and scrapes og_order_to_ack_ms; an AIEX target boots btp-mock + ax_bitnomial_order_gateway::run() and scrapes btnl_og_*. Both inherit the same load driver and reporting.

What the gate protects, and what it can't. The local gate runs against ep3-mock — a fixed-latency stand-in, not the real venue. So the og_order_to_ack_ms it diffs is gateway-internal code-path time, and the regressions it catches are ours: a new lock, an allocation on the hot path, a serialization change, a probe that costs too much. Real venue-side latency is deliberately out of its reach — that is what the prod og_ep3_rpc_duration_ms histogram (§6.4) measures on live traffic. A green gate means "we did not slow the gateway down," not "end-to-end latency is fine." The two questions in §1 are answered by two different mechanisms precisely because neither can answer the other.

Compare (regression gate). baseline save writes a JSON record (schema version, git SHA, target, machine, rate, duration, server {p50,p90,p99,p999,count}, client {p50,p99,n}). baseline diff computes per-metric relative delta against a saved baseline, prints a rustc-perf-style table, and exits non-zero on any metric past tolerance. Two tolerance classes: a static fallback for early milestones (e.g. p50 ±10 %, p99 ±15 %) and, after M4, per-metric thresholds derived from the measured noise floor. diff refuses when target / machine / rate / duration differ between baseline and candidate — an A/B comparison must be same-machine.

Baseline provenance. The blessed baseline lives as a committed JSON file under rs/perf-harness/baselines/, and the nightly always diffs against it — never against the previous night, which would let slow drift accumulate unnoticed. A legitimate perf change (or an accepted regression) lands with an explicit baseline bump in the same PR, so every movement in the number is a reviewed diff rather than a silent shift. Updating the baseline is therefore a deliberate, visible act.

Validating the gate. A test-only --rtt-ms flag injects a fixed delay into the harness's send/ack path, producing a known regression the diff must flag. It is the mechanism behind M3's acceptance ("synthetic regression flagged; null comparison passes") — not a shipping feature.

Five flows, one per question/surface. Commands are illustrative of the shape, not final CLI.

The everyday developer + CI path. Fully in-process, deterministic, no external services.

1. Micro gate (per PR, fast). Any diff touching the order hot path runs the probe-cost bench: cargo bench -p ax-order-gateway --bench ack_path. Criterion compares against the committed baseline and fails if an added probe moved the ack path past budget. This is the first line and runs in seconds — it catches "the metric I added is itself the regression" before any load test.
2. Boot + load. cargo run -p ax-perf-harness -- --target local load --module order-gateway --sessions 16 --rate 50 --duration-secs 30. Under the hood: harness_boot spins testcontainers (Postgres/ClickHouse/Redis) + ep3-mock, starts the real ax_order_gateway::run() on ephemeral ports, seeds fixtures, waits for readiness. The open-loop generator then fires PlaceOrder over the SDK WS client at the fixed rate; a reader task matches OrderAcked by ClientOrderId.
3. Measure + reconcile. At drain, the harness scrapes GET /metrics from the booted gateway, parses the og_order_to_ack_ms histogram, and emits a report: client p50/p99 (HDR) alongside server p50/p99, with the client_p99 >= server_p99 invariant and count == sent drop check. Writes a JSONL artifact + text report.
4. Baseline. First run of a known-good build: ... baseline save --name og-local. Later runs on a candidate build: ... baseline diff --name og-local. diff prints the rustc-perf-style delta table and exits non-zero if any server percentile regresses past tolerance. That non-zero exit is the gate.

Same flow, AIEX edition: --module aiex-order-gateway swaps ep3-mock→btp-mock, ax_order_gateway→ax_bitnomial_order_gateway, and the scraped metric to btnl_og_*. Everything else is identical.

.github/workflows/perf-nightly.yml, schedule cron, on the isolated Depot runner:

1. Matrix module ∈ {order-gateway, aiex-order-gateway}. Each cell runs the §6.1 boot+load with fixed rate/duration, --target local, record-only.
2. Upload the baseline JSON + JSONL as build artifacts. M4 stops here — no gate — for ~2 weeks, accumulating runs so we can compute each metric's stddev = the CI noise floor.
3. M5 turns the recorded baselines into thresholds and adds a final baseline diff --tolerance noise step; a regression past the noise-derived threshold fails the job and pages the alert channel. The triage runbook (A-4023) says what to do when it fires.

Same harness binary, no in-process boot: ... --target dev load --rate 5 --duration-secs 300 points the SDK client at the sandbox gateway (env + isolated-test-firm creds resolved via just env <environment> + configs/). Used for soak and for load levels the laptop can't reach, against real Postgres/ClickHouse/network. Reconciliation still works — it scrapes the deployed gateway's /metrics. Never run against prod (§2). Not part of the pass/fail gate; it's for exploration and pre-launch capacity checks.

The client-latency question is answered here, and no synthetic load is involved:

1. M0 histograms (ack, ep3_rpc_duration) and M1 sampled OTEL spans ship in the gateway and run against real production order flow.
2. To answer "is 50–75 ms real, where does it go": read the histograms off the prod Grafana dashboard (configs/grafana/) — network vs gateway-envelope vs EP3-RPC hop breakdown at p50/p99.
3. To answer "why was this order 75 ms": pull the sampled trace via the timeline-for-order-X runbook (A-4014).

No CLI, no load generator — the "tool" is the deployed instrumentation plus Grafana. This is deliberately the highest-value, lowest-risk flow.

Orthogonal to the order path. just perf-rest dev runs drill --benchmark rs/perf-harness/drill/api_gateway.yml (and onboarding.yml): a YAML script of REST calls (auth → list instruments → history queries, etc.) at a configured concurrency, emitting RPS + latency percentiles. Record-only, not gated initially. drill is HTTP-only and cannot open the order WS — it exists purely to cover the api-gateway/onboarding HTTP surface the order-path harness doesn't touch.

Instrumentation can regress the very latency it measures, especially the tail. Standing rule: every probe added to the order path ships with a benchmark proving it didn't move p99. Two gates:

1. Criterion micro-gate (PR-blocking, primary). rs/order-gateway/benches/ack_path.rs benches the instrumented ack path with the probe off vs on (feature/env-driven no-op recorder). Budget: an added probe (the M0 RPC histogram, the M1 span) costs ≤ ~200 ns/order and ≤ 2 %.
2. Harness A/B (nightly, secondary). Because M1 spans are sampled and exported out-of-band, prove p99 is unmoved at 0 % vs the target sample rate: two harness runs on one boot, baseline diff the server og_order_to_ack_ms p99, require within the noise tolerance. This catches export-thread contention a synchronous micro-bench misses.

M1's out-of-band export design is what makes this budget passable; a kill switch (runtime toggle) sits on all hot-path instrumentation.

• M0 — Attribution histograms + dashboards. Reconcile the ack histogram to one canonical name; add og_ep3_rpc_duration_ms (and the BTP analogue); Grafana panels; the ack_path probe-cost bench. Deliverable: answer the MM thread with a production hop breakdown (network / gateway envelope / EP3 RPC).
• M1 — Per-order timelines. Sampled OTEL spans on the OG order path, head-sampled before span creation, bounded out-of-band export, runtime kill switch; a "timeline for order X" query/runbook.
• M2 — Harness: measure. The thin crate; local + dev load; dual measurement; extract the shared boot. Acceptance: two back-to-back local runs agree within a stated tolerance.
• M2b — AIEX/BTP support (tracked under the M2 milestone; the project has only M0–M5). The AIEX target over btp-mock + ax_bitnomial_order_gateway.
• M3 — Harness: compare. baseline save|diff with the mismatch guards. Acceptance: a synthetic regression (--rtt-ms / injected delay) is flagged; a null comparison passes.
• M4 — Silent nightly. .github/workflows/perf-nightly.yml, schedule + workflow_dispatch, matrix over {order-gateway, aiex-order-gateway}, runs-on: depot-ubuntu-24.04-arm-8 (never a shared runner — p99 variance is unusable), record-only, upload baselines/JSONL, concurrency group, permissions: contents: read + artifact write. ~2-week soak to establish the noise floor. This is the one high-risk .github/workflows/ review.
• M5 — Alert. A post-soak baseline diff step with thresholds from M4's measured noise, an alert channel, and a triage runbook. Acceptance: a synthetic regression is caught; zero unexplained false positives over the soak window.

The REST-breadth drill benches (rs/perf-harness/drill/*.yml, a just perf-rest <env> recipe, dev/local only) run record-only alongside; they are not gated initially.

• A load-test framework. No Stack/Session/Recorder abstraction, TUI, or capacity search. If a second order-path target needs the harness and duplication hurts, factor then — not before.
• Synthetic load against production. Prod is measured passively; §2.
• "Make it faster." This is measurement and regression protection. Optimization becomes justifiable only once these numbers exist.
• Network / private-link attribution, MM-report methodology, EP3-internal instrumentation. Handed off (infra / report tooling / black-boxed venue), not silently dropped.

Prior art: PR #1955 (ax-perf-harness, A-3358), reference only. Code references as of 486582ede.