RFC: Cortex — consolidating trade/transaction/risk onto one transactional core

Date: 2026-06-23 Status: Draft for decision Author: Andrew Lee (with Claude)

The original pitch for cortex ("a TigerBeetle-based risk engine") undersold the actual problem. The real motivation is not that TigerBeetle (TB) is a tidy ledger — it is that our dropcopy → money-state pipeline is split across ClickHouse (CH) and Postgres (PG) with no cross-database atomicity, and we pay for that split in two recurring ways:

1. A recurring bug class with a deferred real fix, plus a standing reconciliation tax. Each torn-CH/PG-write incident does get fixed (A-3112 is in prod; A-3790 merged) — but they get fixed individually, with band-aids, while the architecture keeps manufacturing new ones. The correct idempotency fix is explicitly blocked on a migration (see Pillar 1). Meanwhile we maintain a whole coping apparatus — a recon-engine posting invariant breaches to incident.io, an admin-cli reconcile suite (positions/trades/transactions/open_orders), a DropcopyRestartFence, three per-batch dedup sets — none of which would exist if the write were atomic.

2. A forced durability-vs-latency tradeoff. To keep order placement fast we made the CH write fire-and-forget (wait_for_async_insert=0) after place_order p50 hit ~275ms at the cash open (A-3508). Money mutations funnel through a single global Postgres advisory lock. We are trading away durability and throughput to hold latency — a tradeoff a purpose-built transactional store removes.

This RFC reframes the decision as three options, separating the durable win (consolidation onto one transactional source of truth) from the contentious choice of which store:

• Option 0 — status quo; keep paying the reconciliation tax.
• Option 1 — consolidate writers into one service with Postgres as the single money SoT and CH as a rebuildable projection. No new datastore.
• Option 2 — consolidate onto TigerBeetle as the money SoT (cortex as designed). Best invariants/throughput; new pre-1.0 datastore; migration risk.

Recommendation: Commit to the consolidation (shared by Options 1 and 2): one writer owns the trade→ledger→projection path, money has exactly one transactional SoT, and CH becomes a rebuildable projection rather than a place data can be lost. Then settle TB-vs-Postgres empirically by running TB in shadow mode and measuring it against a hardened Postgres path before granting either authority. Do not do a flag-day cutover of money state.

Naming first, because three things wear the word "risk"/"trade":

• ax-risk-engine (legacy) — the deployed risk/monitoring service (compose.yml risk_engine → ax-risk-engine); ClickHouse/Postgres-based.
• ax-risk-engine2 — a library (no main.rs) used by order-gateway and api-gateway for in-memory margin math (RiskState, OpenOrdersMarginComputer, stress). The order-gateway is the real hot-path margin authority and publishes og:margin:{account_id} to Redis itself.
• ax-trade-engine2 — consumes the EP3 dropcopy stream, pairs executions into trades, and writes positions/trades/transactions to ClickHouse and a resume token to Postgres.
• ax-transaction-engine — applies money movements to Postgres current_balances and logs to ClickHouse transactions; the shared write path for deposits/withdrawals (api-gateway) and funding (settlement-engine).
• TigerBeetle — driver + micro-crate exist but behind a non-default cargo feature and an opt-in compose profile (image 0.16.66, pre-1.0). Not running in production. The README's "TB will be the source of truth" is aspirational.
• Cortex — origin/michael/cortex-reborn (rs/cortex/): promotes risk-engine2 into a standalone binary (main.rs + tasks/{ep3_dropcopy, state_updater, transaction_server, mark_price_calculator, database_monitor}) with the TB driver wired live. Stalled: rebased past ~25 releases, tests failing, and per its own commit it "does not handle inserting transactions from the API gateway or the settlement engine."

The two write paths and where atomicity breaks:

There is no transaction spanning CH and PG (there can't be), and ClickHouse has no multi-table transactions either. Every numbered/lettered step above is an independent failure point.

Caveat up front (these are not open bugs): each incident below has been fixed. The argument is not "we have live torn-write outages." It is that the same bug class keeps recurring — across engines and across the years — that each fix is a point patch rather than the structural fix, and that the structural fix is explicitly deferred because the current architecture makes it migration-heavy. That is the signature of an architecture fighting itself, not of a few stray bugs.

Three things this table shows that "is it fixed?" misses:

1. The real fix is blocked on a migration. A-3112's own writeup says the correct idempotency fix is to "make sequence_number content-deterministic … [so] CH ReplacingMergeTree actually work[s] as the idempotency layer, per the original intent. Requires a migration." So the intended design (CH dedups by content) has never actually held; what ships instead is an ever-growing pile of in-memory dedup sets. The architecture is why the right fix keeps getting punted.

2. Every new writer re-pays the tax. A-3790 is A-3112's cousin reappearing in the Bitnomial port, because each engine re-implements torn-write recovery by hand and a port forgot a branch. The next venue will re-pay it again.

3. A fix for one torn-write bug introduced another (A-2801, reverted). This logic is subtle enough that patching it is itself a source of correctness bugs.

• recon-engine — scheduled invariant checks (fees_zero_sum, pnl_zero_sum, transactions_balance_per_account, …) posting drift to incident.io, with snooze/disable controls (i.e. ops have to silence false positives under load).
• admin-cli reconcile/* — transactions.rs (dedupe CH dups + re-derive PG balances), positions.rs (replay positions from trades), trades.rs (CH-vs-EP3 divergence), open_orders.rs, position_ledger.rs. A whole repair toolbox.
• DropcopyRestartFence (trade-engine2/.../dropcopy_monitor.rs) — exists solely to drain in-flight CH flushes before a reconnect reads PG, to prevent the A-3112 double-commit. Quote: "Without this, a proactive reconnect during a flush lets us read eager CH writes … with a stale PG resume_token … and the ensuing replay double-commits transactions."
• Per-batch dedup scans — every batch runs SELECT DISTINCT execution_id … and trade-id checks; lib.rs carries an explicit TODO: "at what point can we stop incurring the cost of checking prior_trade_ids and prior_execution_ids…?"
• Eager-CH disclaimer — transaction-engine: "ClickHouse inserts are performed eagerly (before the PG transaction is committed). If the caller later rolls back … there may be ClickHouse rows that don't correspond to committed PG state. … Readers … should verify that they only read to the latest sequence number in Postgres." Every reader inherits this caveat.
• In-memory↔︎CH drift — state_machine.rs: "CR alee: seems wrong, no round_decimals? Will drift from CH over time…"

Characterizing the cost: it is not one outage; it is (a) a recurring stream of silent-money correctness bugs that are hard to even detect, (b) the engineering cost of building/maintaining recon + repair tooling, and (c) an on-call/operational tax (drift alerts, manual reconciliation runs, snoozing false positives). Every new writer (Bitnomial, future venues) re-pays it — A-3790 is the Bitnomial trade engine re-hitting the same torn-write class trade-engine2 already fought.

The user's intuition ("CH+PG writes are slow; TB might be much faster") is directionally right, but the precise shape matters.

• The latency event was real. A-3508 (9e19494e2, 2026-06-02): "At the US cash open, client place_order p50 sat at ~275ms. Root cause: the REST hot path wrote the Pending order_log row with wait_for_async_insert=1, blocking the client response on ClickHouse's async-insert buffer flush — a time-triggered flush of hundreds of ms, inflated further when the orderflow cluster is busy at the open."
• The "fix" was to give up durability. The resolution was to move recovery metadata to Redis and make the CH insert fire-and-forget (async_insert=1, wait_for_async_insert=0, order_log_writer.rs:24). So the CH write is "fast" today only because nothing waits for it to be durable — which is also what widens the torn-write window in Pillar 1. The two pillars are the same root cause seen from two sides.
• The money path has a global serialization point. Every balance mutation takes pg_advisory_xact_lock(8675309) — one lock for the whole system (transaction-engine/.../database.rs:74) — then a synchronous PG round-trip with optimistic WHERE sequence_number = $5. That caps balance-write throughput and puts a PG-round-trip floor under latency, regardless of CH.
• Reads pay too. Correctness-sensitive reads use SELECT … FINAL to force ReplacingMergeTree dedup (expensive), and risk-snapshot computation needed a current_positions materialized-view cache to avoid full-log scans (ae5bbeecf).

What we don't yet have: a clean steady-state number for "CH+PG write per fill." We have the metric hooks (og_dropcopy_ch_insert_ms, og_dropcopy_batch_processing_ms) and the transaction-engine PG path, but no documented p50/p99. Action: pull these from prod before sizing the bet (see Next steps).

The TB comparison. TB commits batched linked transfers (5–6 per trade: qty, basis, maker/taker fee, maker/taker PnL/basis-reset — all-or-nothing via AccountFlags::Linked), is designed for ~10^6 transfers/sec, has no global advisory lock, and is durable on commit. The point is not that AX needs 10^6/s — it doesn't. The point is that TB lets us have durability AND low latency AND atomicity simultaneously, instead of picking two. That is the actual performance argument, and it is stronger than "TB is faster."

The structural move — independent of store — is:

1. One service owns dropcopy → trade pairing → money mutation → projection (subsuming trade-engine2's write path and transaction-engine into cortex).
2. One transactional SoT for money: a single atomic commit per trade containing qty + basis + fees + realized/finalized PnL. No torn write because there is one write.
3. Idempotent, deterministically-keyed mutations (trade_id-derived) so replay is a no-op, not a double-commit — deleting the dedup-scan / fence machinery.
4. ClickHouse demoted to a rebuildable projection of the SoT + the dropcopy stream. A lost CH insert becomes a replay, not data loss. This is the single most important reframe: today CH holds authoritative-ish data (realized PnL on trades) that cannot be reconstructed, which is why a torn write loses money.

What it does NOT fix / must be designed around:

• CH still exists, so "SoT committed but CH projection lagging/lost" is still possible — the win is that it's now recoverable by replay rather than lost. The projector must be a deterministic, restartable consumer of the SoT.
• The order-gateway hot path stays in-memory. The margin check is microsecond- scale over cached positions/marks and must not block on any DB. Consolidation changes durability/recovery, not the latency-critical check. (This is a feature: it bounds the blast radius.)
• Business logic that reads-before-write (withdrawal limits, funding zero-sum verify_net_funding_zero, check_not_already_settled) still lives in the service layer above the ledger.

Consolidation is necessary but store-agnostic. The genuinely debatable question is which store is the single SoT.

Option 1 — Postgres as the single money SoT.

• Pros: no new datastore to operate/back up/upgrade; mature ACID; transactions can span the relational data we already keep in PG; the balance SoT is already Postgres. The global advisory lock is a design choice, not inherent — replace it with per-account advisory locks or row locks to recover throughput. CH becomes a projection rebuilt from PG + stream.
• Cons: we hand-roll double-entry invariants and idempotency in SQL; PG write throughput/latency for a high-rate ledger is real engineering; per-trade multi-row commits under contention need care.

Option 2 — TigerBeetle as the single money SoT (cortex as designed).

• Pros: double-entry correctness enforced by the store (debit=credit, debits_must_not_exceed_credits, linked all-or-nothing batches) — a near-direct mechanization of our G1–G3/recon invariants; idempotency free via deterministic transfer IDs; designed-for-this throughput with durability; clean resume()-from-ledger recovery.
• Cons: a new pre-1.0 consensus datastore (0.16.x) running as custody-adjacent infrastructure — formatting, replication, backups, version- matched client libs, submodule/build plumbing (already visible in CI history); no guaranteed on-disk format stability across versions → migration risk for the SoT; TB stores integer u128 only, so all Decimal risk math (marks, IM/MM rates, multipliers, VWAP) stays in Rust and prices get smuggled as f64 in user_data_128 (see realized-drift and "apply contract multiplier to TB basis/recovery" fixes); identity packed into IDs (user_id|ledger_id, trade_id<<64|nibble) couples our type layout to TB (already forced the ProductId/InstrumentId split in the dated-futures RFC).

The honest framing: Option 1 fixes most of the pain (atomicity, recoverable projection, idempotency) with zero new infra — most of our incidents (A-3790, A-3112, A-2801) are torn CH/PG writes that a single-SoT + rebuildable-projection design closes regardless of which store. Option 2 additionally buys invariant-enforcement-in-the-store and a throughput ceiling we don't currently need, at a real and permanent operational cost. TB's marginal benefit over a hardened Postgres path is not yet quantified.

1. Commit to consolidation now (the shared win). Make one service own dropcopy → trade → money-mutation → projection; make money have exactly one transactional SoT; make ClickHouse a deterministic, rebuildable projection. This kills the torn-write class and lets us start deleting recon/fence/dedup machinery. Reuse the risk-engine2 math crate so the service and the OG hot path compute margin identically.

2. Decide TB-vs-Postgres empirically, not by vibe. Stand up the SoT behind a trait so the projection/consumer code is store-agnostic. Run TigerBeetle in shadow mode (write the ledger in parallel; reconcile continuously against the authoritative path; non-authoritative, doesn't block trading). Measure: commit latency/throughput vs. a per-account-locked Postgres path, recovery time, and f64/Decimal fidelity under real load.

3. Gate granting TB authority on: (i) a sustained clean shadow-reconciliation window, (ii) finishing the api-gateway + settlement-engine transaction integration cortex skipped, and (iii) a TB version we accept for custody-adjacent state (≥ 1.0 or an explicit risk acceptance). No flag-day cutover of money state.

4. Either way, retire the divergent branch. If we proceed, rebuild cortex on main incrementally rather than re-syncing michael/cortex-reborn (we've paid that tax once). If we don't proceed within a quarter, shelve explicitly and delete the branch; keeping the feature-flagged TB code on main (as today) is a fine zero-cost resting state that preserves the option.

• Quantify Pillar 2. Pull prod p50/p99 for og_dropcopy_ch_insert_ms, og_dropcopy_batch_processing_ms, and the transaction-engine PG commit latency (incl. advisory-lock wait) from the logs/metrics stack. Confirm whether the bottleneck is the global lock, the PG round-trip, or CH.
• Quantify Pillar 1. Count recon-engine drift alerts and admin-cli reconcile runs over the last quarter; estimate the standing on-call/reconciliation tax in hours. Tally torn-write incidents (A-37xx/A-31xx series) and dup rates.
• Spike the abstraction. Define the SoT trait + a deterministic CH projector; prove a single atomic per-trade commit on the Postgres path first (cheapest path to killing the torn write), then add the TB shadow implementation behind it.

• Does liquidation/auto-deleverage need a transactional guarantee per-account PG locking can't give cheaply? (If yes, strengthens Option 2.)
• Is CH + the dropcopy stream a strong enough oracle to validate the SoT in shadow mode, or do we need a third independent check?
• What is our real appetite for operating a pre-1.0 consensus datastore as custody-adjacent infra, given Postgres is already the balance SoT?
• How do future venues (Bitnomial and beyond) plug into the consolidated writer so they stop re-paying the torn-write tax (cf. A-3790)?

• Write paths: rs/trade-engine2/src/{tasks/dropcopy_monitor,tasks/state_machine_driver,state_machine,database}.rs; rs/transaction-engine/src/{lib,database}.rs
• Atomicity comments: transaction-engine eager-CH disclaimer (lib.rs:226-231); DropcopyRestartFence (dropcopy_monitor.rs); round-drift CR (state_machine.rs); dedup-cost TODO (trade-engine2/src/lib.rs)
• Incidents: A-3790 1d7485653/a15d56c4f; A-3112 a8b76e2df/49221815b; A-2801 27ebaafbe→5ed4bd85b; A-2478 PR #977 / tlaplus/LESSON_24.md
• Perf: A-3508 9e19494e2; async CH inserts 632a34753,f09bb4142; permit-pool b1c4474e6; positions cache ae5bbeecf; order_log_writer.rs:24; metrics in order-gateway/src/lib.rs:94-125
• Coping machinery: rs/recon-engine/; rs/admin-cli/src/reconcile/{transactions,positions,trades,open_orders,position_ledger}.rs
• Money SoT today: db/postgres/1.sql (current_balances); advisory lock transaction-engine/src/database.rs:74
• TB: rs/risk-engine2/src/tigerbeetle_driver.rs (batched linked transfers); rs/sdk-internal/tigerbeetle/; compose.yml (profile tigerbeetle, image 0.16.66); rs/risk-engine2/Cargo.toml (non-default feature)
• Cortex branch: origin/michael/cortex-reborn (rs/cortex/, incl. src/main.rs, src/tasks/)
• Identity coupling: docs/rfc/dated-futures-and-options-internal-reforms.md; rs/risk-engine2/2025_01_04_REALIZED_DRIFT.md