RFC: Consuming EP3 executions directly from Redpanda

Date: 2026-08-28, revised 2026-09-10 Status: In review. Production uses the gRPC drop copy. Paired dev captures confirm every native field mapping. In production, EP3's drop-copy delivery accounts for about 69% of mean acknowledgement latency and most of its tail.

For most of 2026 we have been chasing order-acknowledgement latency on EP3. A client places an order, the InsertOrder RPC returns almost immediately -- and then the client waits for the ack. The wait is long enough that market makers have reduced or ceased trading over it.

The wait is structural. InsertOrder's response carries only an exchange order id (rs/order-gateway/src/rest_service.rs:641 uses it for nothing else; the synchronous-return fields are only populated when max_block_time is set on the request, and nothing in the codebase sets it). The actual acknowledgement -- the Accepted state transition, the OrderAcked WebSocket event, the og_ws_order_to_ep3_ack_ms measurement -- fires from the EP3 drop-copy stream, when an execution with ExecutionType::New arrives at rs/order-gateway/src/lib.rs:1541.

And the drop copy is not a direct line. Connamara confirmed (December 2025) that EP3's drop-copy service is itself a consumer of EP3's internal Kafka bus, a Redpanda cluster. We have since confirmed that directly: EP3's own drop-copy resume token is a plain vector of Kafka offsets, twelve pairs of a 4-byte partition and an 8-byte offset, one per partition of the events topic. The drop copy is a Kafka consumer with a serialisation layer bolted on, and we repeat that last hop four times over, because order-gateway, trade-engine2, risk-engine and marketdata-publisher each hold their own subscription.

The acknowledgement a client waits on is not the reply to InsertOrder. It is a message that was sitting on EP3's internal Kafka bus milliseconds after the match, and everything between that bus and our order-gateway is overhead we can remove.

Connamara supports reading the bus directly and documents the contract at https://connamara-ep3.readme.io/docs/consuming-directly-from-kafka. A probe on the branch michael/ep3-kafka-direct consumed both feeds side by side and put the expected p99 ack time below 10 ms. That figure was never reproduced, and it is not what this RFC rests on -- see "Where the time actually goes" below, which measures the same claim from production data.

Consume the events topic from EP3's Redpanda brokers in a shared ax-ep3 module that reassembles, deduplicates, and decodes the raw records behind the same subscription contract the gRPC drop copy serves today -- and switch consumers over one at a time, by per-service configuration, starting and possibly ending with order-gateway.

"Kafka" names the protocol, not a different server: Redpanda is a drop-in implementation of the Kafka wire protocol, which is why an ordinary Kafka client library talks to it natively.

events topic. EP3's internal firehose: executions, order events, requests and market data all pass through it. Twelve partitions on the dev exchange. It is strictly read-only for us, and it carries far more than we need, so we filter client-side.

TransactionEvent. The envelope in every record value (connamara.ep3.wire.v1beta1, vendored at rs/ep3/api/protos/connamara/ep3/wire/v1beta1/wire.proto). It holds a trace_route and repeated google.protobuf.Any payloads; the type_url says what each one is.

The two execution-bearing shapes. This is the trap. Executions arrive as books.v1beta1.BookEvent and as orders.v1beta1.OrderEvent, which carry the same orders, executions, transact_time and cancel_reject fields. On a sample of live dev order flow, order events were the more common of the two. A consumer that decodes only book events looks perfectly healthy and silently drops about half the executions. The topic also carries orders.v1beta1.FindAndCancelOrdersRequest and trades.v1beta1.TradeCaptureReport; neither carries executions, and both are ignored.

The two Execution types. The drop copy delivers ep3.v1beta1.Execution: an execution with the full Order snapshot embedded (clord_id, cum_qty, leaves_qty, state, price) plus its own transact_time. An event carries the leaner orders.v1beta1.Execution: an order_id string pointing at the affected order carried once in the same event, its own order_status (the order's cum, leaves and state as of this execution), and no timestamp. The two Order types share exactly one field tag -- id -- because the lean one nests everything under attributes and status. Our consumer performs the same join EP3's drop copy does, because order-gateway reads the embedded order everywhere (cum_qty/leaves_qty at lib.rs:1438, replace resolution at lib.rs:1392). Connamara confirmed on 2026-09-10 that the per-execution status is a contract: every book-side execution clones the order's live status at creation, mutated for fills immediately before the clone, and executions are appended in the order they are applied to the book.

id header. Every record carries an 8-byte big-endian signed 64-bit id, globally unique, assigned by EP3 at publish time. Delivery is at-least-once, so this is the deduplication key. Ordering is per partition only.

Chunked message. Brokers cap record size, so EP3 splits any larger message across records carrying chunkedMessageId / chunkedMessageIndex (a byte offset, not a sequence number) / chunkedMessageTotalSize. Reassembly is copying bytes into a buffer of the stated size until it is full. Not an edge case: our own drop-copy client raises its gRPC frame limit to 64 MB (rs/ep3/src/client/admin.rs:506) precisely because single messages -- mass cancels, expirations -- get that large.

Sentinel record. After the final chunk of a split message EP3 publishes one ordinary record with the same id and an empty payload. Consumers skip any TransactionEvent with an empty payload. The sentinel reuses the split message's id deliberately, so what keeps it safe is not the order of the two checks but that chunks never feed id-based dedup at all: chunked records are deduplicated on chunkedMessageId, which leaves the sentinel's id unseen and lets it reach the empty-payload check on its own.

Offset vs resume token. The drop copy's cursor is an opaque resume_token with a known failure mode, vendor-confirmed in December 2025: a token older than retention is silently clamped to the start of what remains, with no error. A Kafka offset fails loudly with OffsetOutOfRange instead. Loud beats silent: a stalled consumer is a bad morning, a consumer that silently skipped a fill is a wrong position nobody finds for a month.

What the diagram does not show is why each step is mandatory:

• Never produce. The topic is the exchange's spinal cord; an injected record can disrupt matching. Enforced in code by exposing only a fetch API from the module, and ideally at the broker by a read-only ACL.
• Decode both execution-bearing shapes. Handling only book events is the failure this spike actually hit, and every local test passed while it was wrong.
• Deduplicate before the state machine sees anything. The gRPC stream never redelivers, so order-gateway has no dedup today; a redelivered fill would re-broadcast WebSocket events and re-write order_log rows. Chunks are idempotent by construction -- the same bytes at the same offset -- but whole records are not.
• Stop on a record the feed cannot read or synthesize exactly. Skipping it can lose executions. A decode or synthesis error on a BookEvent or OrderEvent fails the subscription and does not advance its position. Other payload types carry no executions and are skipped. A retry at the same position fails the same way, so the supervisor must latch the failure and stop automatic retries until the cause is repaired. The gateway retry task does not do this yet; do not serve acknowledgements from direct Kafka until it does.

A client sends a limit order on EURUSD-PERP over the order-gateway WebSocket.

Today: order-gateway calls InsertOrder; EP3 matches or rests it and publishes the resulting event to its internal Redpanda bus; EP3's drop-copy service consumes that record, joins the order snapshot into an ep3.v1beta1.Execution, waits out its own batching, and streams a gRPC response to each of our four subscriptions; order-gateway's reader receives the batch, and only now records og_dropcopy_delivery_latency_ms, transitions the order to Accepted, and emits OrderAcked. Every hop after Redpanda re-reads and re-sends a message that already existed.

Proposed: the same record lands on the events topic; order-gateway's fetch (already long-polling, so the broker responds the moment data arrives) returns it directly; the module dedups, decodes the event in whichever shape it arrived, synthesizes the identical execution batch, and hands it to the same channel the gRPC reader fills today. The drop-copy service, its batching and the gRPC hop are gone; og_ws_order_to_ep3_ack_ms is the before/after scorecard.

The whole value of this change is capped by one quantity: how long EP3's drop copy takes to tell us about a match. Everything before it (our validation, the network to EP3, the match itself) and everything after it (our state update, our acknowledgement) is untouched by switching transport. That quantity is measured in production, per execution, by og_dropcopy_delivery_latency_ms: EP3 stamps TransactTime when it matches, order-gateway stamps arrival, and the difference is the drop copy's contribution.

Over 24 hours of production, 14.9M executions and 6.3M accepted orders:

Delivery is about 69% of mean acknowledgement latency, and our entire share of that same return path stays under a millisecond at p95.

The tail is the sharper finding. Half of all executions reach us in 2.3 ms, but the slowest 5% take 158 ms -- roughly seventy times worse -- while our own handling of those same executions never exceeds a millisecond. A tail that severe on the far side of the wire, and flat on ours, is what drags acknowledgement p99 to 303 ms. Production has never acknowledged an order in under 5 ms.

This is also a test the proposal could have failed. Had delivery come back at half a millisecond, the 24 ms would have been our own code and no change of transport would have helped. It did not.

What replaces those 16.8 ms is known: the record is on the topic 0.1 ms after the match, and reading it from inside the exchange's network costs about 1 ms.

Two caveats on the numbers. Delivery compares EP3's clock to ours, so the split between the outbound and return legs carries unknown clock skew; the tail argument is unaffected, because 158 ms dwarfs any plausible skew. And delivery is measured per execution while acknowledgement is per order, so their means and shapes compare but their percentiles do not.

Direct Kafka delivers the same execution and cancel-reject fields as native gRPC, or it fails with an error. It does not exclude fields, substitute end states, deliver partial batches, or log and skip. A position that can never be served again is different: the feed reports it as unservable, and the gateway cold starts, resyncs and counts the gap (see "Recovery paths" below).

Paired captures from EP3 dev confirm every field mapping: a historical comparison of 27,598 executions for the order and execution fields, and live captures of ten commission rule shapes for the four commission fields. Synthesis fails an event whose native rendering was not observed: spread and expression commission rules, and rate strings that do not parse.

The measurements below have different scopes. Demo samples and a retained-dev report identified candidate mappings; the paired dev captures of 2026-09-09 confirmed them. Raw captures, comparison reports and the findings write-up live outside the repository (ep3-demo-capture-20260908/evidence-20260909/DEV_KAFKA_FINDINGS.md).

• Two shapes carry executions. Live dev records: 5 OrderEvent, 4 BookEvent, plus requests and trade-capture reports that carry none. This found a real bug in the spike's own code.
• Each event carries the orders its executions reference. Every sampled execution's order is in the same event, with zero unresolved joins, and both legs of a trade arrive together: 1000 demo trades shared an identical nanosecond transact_time across both legs. The batch_count = 1 framing depends on this.
• avg_px is derivable. It is the quantity-weighted average of price_to_quantity_filled, rounded half away from zero -- matched on 2000 demo fills including 79 that spanned more than one price level, where truncation disagreed on 27.
• create_time is initial_order_receive_time. The retained dev events window (partition 4, 41,325 records, 2026-08-28/29) was captured raw and synthesized, then compared with native AdminAPI reports for the same 27,598 execution ids. All 81 serialized field paths matched on every execution. create_time matched on new, partial fill, fill, cancel, replace (where insert_time moves and create_time does not), reject (no insert_time) and expiry. Connamara confirmed on 2026-09-10 that the drop copy copies this field literally for every execution type. Native renders the two basis-point strings as "0" for an order without commissions, and synthesis does the same. This environment had no commissions configured, and historical reports do not show live delivery or ordering.
• Each lean execution carries its own order_status. Field 5 of orders.Execution holds quantities, state, last trade ID, the fill map and the commission maps. Synthesis joins that status with the affected order's attributes and metadata. It does not read the event's final order status, undo fills, infer terminal or replacement states, or search earlier events for trade IDs. An execution without a status fails the event. Connamara confirmed on 2026-09-10 (question 1 below) that every book-side execution clones the order's live status when it is created.
• Live paired captures on dev cover the lifecycle edge cases. A paired recorder (ax-admin-cli test ep3-paired-capture) pins every partition's live edge and the native drop copy before a scenario driver (ep3-scenario-drive) runs FOK kill and fill, market-to-limit, post-only reject and reprice behaviours, self-match prevention (all three instructions), all-or-none, minimum quantity, cancel of unknown/filled/foreign orders, replace of a filled order, mass cancels with and without resting orders, and good-till-time expiry, plus the exec snapshot probe's IOC partial, replace-with-immediate-fill and replace-after-partial-fill. All executions matched on all 81 field paths (9 of 9 and 46 of 46), and all cancel rejects matched (1 of 1 and 4 of 4).
• Commissions are mapped, on ten rule shapes. A recipient account and rules on the participant firm were configured on dev and fills captured under each: single 10/5 bps, a global 100 bps rule stacked on it, decimal rates ("2.50" + "100"), flat fee, minimum and maximum notional, skipped balance adjustment, an order-entry override, currency rounding, and a negative maker rebate. Native commissions_basis_points and maker_commissions_basis_points are the normalized decimal sums over the lean order's rules (a rule without a maker rate charges its ordinary rate to the maker; flat-fee rules contribute 0; "110"/"105", "102.5"/"101.25", "10"/"-2"). execution.commission_notional_collected is the sum of the lean execution's commission_notional_calculated map, already minimum-adjusted; the order total is the sum of the status commission_notional_total_collected map. 82 executions across the ten captures equal on every field. Spread rules cannot be configured on dev ("applying commissions as spreads not allowed"), so spread and expression rules fail synthesis instead of being rendered.
• Only BookEvent and OrderEvent carry executions. The other three types seen on the topic do not: TradeCaptureReport repeats executions already delivered in a book event, FindAndCancelOrdersRequest echoes carry only the request, and FirmsNotification is broadcast on any account or firm change. Connamara confirmed on 2026-09-15 that the set of execution-bearing types will not change (envelopes may gain fields, which the decoder tolerates), so the feed decodes those two and skips everything else.
• Synthesis covers every field. The mock end-to-end tests check direct-feed parity against the mock's own drop copy, and the gateway suite passes on both transports.
• Chunk reassembly is exercised against the real exchange. A 301,500-lot sweep across 3015 resting makers produced the first chunked message we have seen on a real topic: one ~1.9 MB event split into two chunks and followed by a sentinel. The production pipeline rebuilt it into a single event carrying 3016 orders, with both legs of all 3016 trades together and zero unresolved joins across 25,861 executions; the drop copy's own count for the same window, 6031 executions, accounts for it exactly.
• Every record carries the id header deduplication depends on, 8 bytes, on 100% of records sampled natively. The consumer treats a missing id as "not applicable", so this needed confirming rather than assuming. Records also carry a traceparent the vendor guide does not mention.
• Retention is time-based, not positional. An earlier reading of "trimmed to roughly the current position" was an artifact of an idle exchange: under load a busy partition held 19,516 records while quiet ones held none.

A shared module in ax-ep3, behind the contract consumers already use: create_drop_copy_subscription keeps its request and response types, and the transport behind it is selected per service by env (EP3_EXECUTION_FEED=dropcopy|kafka plus a broker list). Call sites keep their request and response types, but the method now returns a boxed stream rather than tonic::Streaming, so the few helpers that named that type were retyped, and order-gateway gained explicit handling for the one failure only the direct feed can raise. This is also what makes migration gradual: every service links the same library, but each process picks its own transport, so flipping a consumer is a one-variable config change and rolling back is changing it back.

Not a relay service that re-serves the gRPC API: a relay is a new deployable and a single point of failure in front of every consumer, and the vendor checklist lives in one place either way.

Order-gateway is where the latency is felt, and it is the easiest correct migration: it skips block-trade executions outright, and keeps no durable cursor -- its token lives in a local, so a restart subscribes with an empty token, tails live, and repopulates open orders from a SearchOrders snapshot (sync_initial_open_orders, filtered to OrderStateFilter::Open). That maps exactly onto "start at latest offsets, resync on OffsetOutOfRange", including the gap it leaves: the snapshot restores resting orders, never terminal ones. trade-engine2 is the opposite: post-trade, latency-insensitive, and its Postgres-atomic resume-token machinery is the most carefully tested code on this stream. It should move late or never.

rskafka is the right client: pure Rust, no librdkafka C toolchain in a workspace whose builds already wedge on quickfix-ffi, and a manual partition/offset model with no consumer-group rebalancing to reason about.

The swap is signature-preserving because everything the drop copy does server-side is either emulable in the library or deliberately improved:

• Symbol filters -> applied client-side to the synthesized batches. The server-side filter only ever saved bandwidth. Cancel rejects bypass it, as they do on the wire, because they carry no symbol. The firms filter is refused at subscribe time rather than emulated: nothing uses it, and an event does not say which firm an order belongs to.
• resume_time -> refused at subscribe time, like firms. It asks the exchange to resume from a wall-clock instant, while this cursor is a Kafka offset, and rskafka exposes only earliest and latest, not offset-for-timestamp. No service sets it; only ep3-cli admin-subscribe-dropcopy --restart and --since do, and they stay on the gRPC transport.
• Batch framing -> everything yields as batch_count = 1. Consumers already treat the single-chunk batch as the base case; multi-chunk framing was a gRPC transport artifact.
• Heartbeats -> synthesized periodic empty batches, so liveness logic sees no difference. Note these are minted locally on a timer and do not prove the broker is alive -- and neither does the absence of an error, because rskafka retries a broken connection indefinitely, so an unreachable broker yields green heartbeats rather than a stream failure. The feed keeps no lag metric of its own, and no consumer reads drop-copy heartbeats for staleness today, so staleness detection is unbuilt on both transports rather than lost in the swap.
• Resume tokens -> the Kafka transport writes its own token: a source tag and, for each partition, a fetch position and an emit position. A resume must re-read from the first chunk of a partly reassembled message, but must not deliver executions that the consumer already has. So the fetch starts at the fetch position, and the feed suppresses records before the emit position. The token also advances over records that deliver nothing (request traffic, filtered symbols), so that a healthy consumer's token does not age out of retention. Each transport refuses the other's token with FailedPrecondition, because a translation or a discard would lose history.

Two differences are deliberate:

• An unservable position recovers automatically and is visible; an integrity failure stops delivery.
  • Feed. A partition that the stored position does not name starts at its live edge. A partition's position is unservable when its offset is older than retention, when a split message is still incomplete after the reassembly lag limit, or when the reassembly buffer is full. For an unservable position, the feed:
    • resets only that partition to its live edge;
    • publishes a token with the reset, with no change to the other partitions;
    • closes the stream with OutOfRange;
    • increments ep3_kafka_partition_repositioned_total{partition}.
  • Gateway. The gateway keeps that token, rebuilds open orders from a snapshot once, resumes from the token, and increments og_ep3_partition_repositioned_resyncs_total. Only the gap on the reset partition is lost.
  • Cost of a gap. The feed does not deliver executions inside the gap. A resting order shows its current filled quantity and has no fill events for the gap.
  • Other refusals.
    • The feed refuses a token that it cannot decode, including one with a different version byte, with FailedPrecondition. The consumer keeps the token and stops.
    • The gRPC drop copy clamps an aged-out token to the oldest retained record, with no error.
    • A corrupt record or a synthesis error raises DataLoss and ends the stream. The feed does not acknowledge the failed event, and the gateway keeps its last token, so the next read starts at the failed record.
    • The feed skips a payload type that it does not decode (see "Stop on a record the feed cannot read or synthesize exactly" above).
• Ordering is per partition only. The drop copy delivers one merged stream; this merges twelve partitions and preserves only per-partition order. Measured, not assumed: records carry NO Kafka key -- the producer assigns partitions by product group (on the sandbox, FX perps share one partition, metals another, energy a third; on dev everything rides one), so all of a symbol's events are totally ordered on a single partition and per-order sequencing holds. What backs it is a deployment convention rather than a keying contract, so the partition-to-product mapping is on the remaining vendor questions.