Order gateway latency statistics

As built: 2026-09-17, commit 04582813b

The order gateway records six latency histograms at its own boundaries. They measure:

• a WebSocket place or replace, from the socket read to the EP3 RPC, to the drop-copy execution that answers it, and to the socket write of the event that answers it;
• a WebSocket cancel, from the socket read to the socket write of the OrderCanceled event;
• every drop-copy event, from its arrival at the gateway to the socket write on each session;
• a REST order request, from the start of authorization to the handler's response.

The gateway times a WebSocket request by storing a timestamp on the open-orders entry of the order the request names. The drop-copy execution that answers the request names the same order, so the gateway finds the timestamp there. There is no separate table of in-flight requests.

The histograms are Prometheus histograms. They export over OTLP to the observability ClickHouse with every other gateway metric. Nothing is written to the exchange database.

The design follows three rules, in priority order:

1. Measurement does not delay order handling. Measurement allocates nothing. It adds one lock acquisition: a cancel or replace takes the open-orders write lock briefly to store its stamp. A place stores its stamp in the insert that the order path already performs.
2. Every request is measured the same way. Clocks start and stop at the boundaries stated in this document. Load on a session does not change whether its requests are measured.
3. Numbers use the existing observability pipeline. A quantile is read from a histogram, and its error bound is the width of the bucket it falls in.

No series is the client's round trip. TLS terminates at nginx. A completed gateway socket write means the gateway handed the bytes to its connection toward nginx. It does not mean the client received them. Only the client can measure its own round trip.

In these series, "ack" means the event that answers a place or replace: OrderAcked, OrderRejected, or OrderReplacedOrAmended.

The three order series start at the same frame read, and each ends later than the one before:

• og_ws_order_to_ep3_ms covers the gateway's work before it calls EP3.
• og_ws_order_to_ep3_ack_ms adds EP3's processing and the drop-copy transit.
• og_ws_order_to_ack_ms adds the gateway's fan-out and socket write.

og_ws_order_to_ack_ms minus og_ws_order_to_ep3_ack_ms is the gateway's delivery time for that order. og_dropcopy_to_ws_ms measures the same interval for every drop-copy event.

The start of a WebSocket series is the instant the session loop takes the frame off the socket. It is not the network arrival time. A frame can wait in kernel buffers while the loop is busy, for example while the loop is blocked on a write. That wait is outside the measured span.

No series has an account, client, order ID, or request ID label.

A request stamp holds three values: the request kind, the instant the session loop read the frame, and a process-local ID for the session that read it. The session loop creates a stamp for every place, cancel, replace, and cancel-all request and passes it to the spawned handler. REST requests have no stamp.

The stamp follows the request through these steps:

1. Store. The handler stores the stamp on an open-orders entry before it issues the EP3 RPC, so the stamp is in place before any execution can arrive.
   • A place inserts the new order with its stamp.
   • A replace stamps the replacement order's entry after begin_replace creates it.
   • A cancel stamps the order being cancelled, in a separate cancel field. If that field already holds a stamp, the gateway keeps it. A repeated cancel therefore measures from the first cancel.
   • A cancel-all stores no stamp.
2. Issue the RPC. Just before the EP3 RPC, the handler records og_ws_order_to_ep3_ms. This is the only series a cancel-all records.
3. Match the execution. The drop-copy processor removes the stamp from the entry when it applies the answering execution, and attaches the stamp to the routed event:
   • A New or Rejected execution takes the order's stamp and records og_ws_order_to_ep3_ack_ms. The stamp goes on the OrderAcked or OrderRejected event.
   • A Replace execution takes the replacement order's stamp and records og_ws_order_to_ep3_ack_ms. The stamp goes on the OrderReplacedOrAmended event.
   • A Canceled execution takes the cancel stamp. The stamp goes on the OrderCanceled event.
4. Write the event. Every session of the account writes the event. The session whose ID matches the stamp records og_ws_order_to_ack_ms or og_ws_cancel_to_out_ms after its write returns. Other sessions record nothing for the stamp.

The handler sometimes rejects a request itself, for example when the client order ID is a duplicate or the EP3 RPC returns a definitive rejection. The OrderRejected event it broadcasts carries the handler's stamp. The requesting session records og_ws_order_to_ack_ms with result="rejected" when it writes that event. No og_ws_order_to_ep3_ack_ms sample is recorded, because no drop-copy execution answered the request.

Each stamp produces at most one sample per series, because the drop-copy processor removes the stamp when it uses it.

A stamp that no execution takes stays on its open-orders entry and is deleted with the entry. Stamps exist only in memory, so a gateway restart discards them.

• A request that fails before it has an order to stamp and broadcasts no event. Examples are a malformed frame, a permission refusal, and an internal error.
• A cancel whose order ends another way. After a fill, an expiry, or a CancelRejected, the cancel stamp is unused and is deleted with the entry.
• A replace that EP3 rejects asynchronously. The rejection arrives as a drop-copy cancel-reject. The gateway rolls back the replacement order, which deletes its stamp, and sends a CancelRejected event with no stamp.
• A replace whose original order terminates while the replace is pending. The gateway rejects the replacement order itself, and that OrderRejected event has no stamp.
• A request whose session closes before the answer arrives. og_ws_order_to_ep3_ack_ms is recorded. og_ws_order_to_ack_ms and og_ws_cancel_to_out_ms are not.
• An order recovered at startup.

The drop-copy reader records a monotonic instant when a batch arrives. The drop-copy processor sets that instant on every RoutedEvent it broadcasts for the batch. This includes the CancelRejected events and the rejections of orphaned replacement orders that the processor generates while it handles the batch. After a session loop's write of the event returns, the loop records og_dropcopy_to_ws_ms under the event's label.

Events that the gateway generates outside the drop-copy path, such as a handler's own reject, have no arrival instant and produce no delivery sample. The label set includes heartbeat because the label function covers every event type. The session heartbeat is generated in the session loop, so that label has no samples.

The gateway closes a session that falls too far behind the event broadcast, and counts the closure in og_ws_slow_consumer_evictions_total. The events that session had not yet read produce no delivery samples. The counter gives the number of closed sessions, not the number of missing samples.

The public event wire shape does not include the arrival instant or the stamp.

A timing middleware wraps the REST order write routes. It is the outermost layer, so a sample covers authorization, the full-access check, and the handler. The sample ends when the handler returns a response, before the body is written.

The route label is the matched pattern from a fixed set: /place-order, /cancel-order, /replace-order, /cancel-all-orders, /place_order, /cancel_order, /cancel_all_orders. Any other matched route is labelled other. A request with no matched route is labelled unmatched. A unit test asserts that every write route has its own label.

The status label is the exact code for 200, 400, 401, 403, 404, 409, 422, 429, 500, 502, and 503. Any other code is labelled with its class: 2xx, 3xx, 4xx, or 5xx.

REST requests have no request stamp, so they record none of the WebSocket series.

All six histograms share one set of explicit bucket boundaries, in milliseconds. The set is log-spaced at eight buckets per octave (ratio 2^⅛ ≈ 1.09), from 0.05 ms until a bound covers 30 s: 155 bounds, each rounded to four significant figures. A linearly interpolated quantile is off by at most half a bucket, about ±4.5% of its value. One shared set also lets any two series be compared bucket for bucket. Explicit buckets make the Prometheus exporter emit a histogram, which can be aggregated across time, labels, and gateway replicas.

A histogram carries no minimum or maximum. Each series is therefore recorded a second time under <name>_window, with the same labels and no configured buckets. The exporter renders the _window series as a rolling summary over a 60 s window. Only its quantile 0 (min) and quantile 1 (max) are meant to be read: a min of mins and a max of maxes are exact under any re-aggregation, which the summary's other quantiles are not. Quantiles such as p50 and p99 come from the histogram. The rolling window means a min or max can appear for up to a minute after the sample that set it.

Each series is one LatencySeries constant that holds its name, its _window name, and its description. LatencySeries::record writes a sample to both the histogram and the _window summary. The Prometheus builder and the metric descriptions both read from the list of these constants.

The second record costs about the same as the first. benches/latency_probe.rs measures one probe with and without the _window record.

The gateway stores no individual samples. A gateway crash loses the observations made since the last scrape.

On every heartbeat tick, the session loop sends a WebSocket ping with an empty payload, then the application heartbeat event. The gateway does not measure the ping round trip.

The WebSocket library answers client pings. The session loop sends no pong of its own, so a client receives exactly one pong for each ping it sends.

• The series do not include time before the session loop reads a frame or after the gateway's socket write returns. The gateway cannot see the client's network path, region, or identity behind nginx.
• og_rest_request_to_response_ms does not include writing the response body.
• The gateway does not measure how long a CancelRejected answer takes, how long a cancel-all takes to complete, or how long an order takes to fill.
• An acknowledged order is still open. The ack series measure the time to the answer, not the life of the order.

The public SDK protocol has no latency fields. A client can measure its own round trip by timing each response against its request ID.

Unit tests cover:

• which request kind each event can answer;
• that session IDs are distinct;
• that the bucket set has 155 log-spaced bounds from 0.05 ms past 30 s;
• that every _window name is the series name plus _window;
• that every REST write route has a label.

Integration tests over the real WebSocket stack cover:

• a WebSocket place and cancel populate every series, as histograms with the expected labels, and every _window series renders as a summary with quantiles 0 and 1;
• a client ping gets exactly one pong.