When margin
goes negative,
something
must act.
— Liquidation engine, v1
Pure functional core driving a single‑actor imperative shell. Detects MMA breach, cancels open orders, places IOC liquidations at a protective limit, parks accounts it cannot unwind. Dark by default behind a global kill‑switch and a per‑user gate. Every suppressed trigger is itself an audit row — the action is suppressed, the breach is observed. FTX/Alameda invariant (no per‑account bypass in engine code paths) enforced by proptest. Nine PRs, four weeks, dark until #9.
Kill switch
Source of truth lives in Redis at liquidation_engine:kill_switch. The shell re‑reads the key at every dispatch (belt‑and‑braces). Off ships at first deploy. The engine is dark, harmless, and observable for one full soak week before a deliberate config flip turns it on.
Engine is dark. A trigger from risk‑monitor opens an incident, persists, audits, and posts to Slack — but no order ever leaves the box. This is the only mode an institutional account is ever in unless explicitly opted in.
The blast radius map
Click any node. Risk‑monitor is the canonical breach detector; the engine never recomputes MM. Redis pubsub wakes the engine; margin_status validates the breach before any incident opens. Dashed red edges are best‑effort transports guarded by Postgres authority.
Liquidation Engine
single mutator · pure core + shellThe driver is the only task that touches live EngineState. Everything else — Redis wakeup listener, PG validator, OG client, persistence writers, Slack — pushes validated events into one bounded mpsc. handle_event(state, event) → (state, Vec<Effect>) is pure, replayable, and proptestable. Redis wakes; Postgres validates; the driver dispatches returned effects in deliberate order: persist → audit → cancel → place → notify.
Drive the account
Click events. Watch state transition, effects emit, and the per‑instrument attempts counter climb. The pure core makes invalid states unrepresentable: "Liquidating with no incident" cannot be expressed in the type system. Five consecutive rejects on the same instrument enters a pre‑park probe; fresh healthy MMA resolves instead of paging ops.
Event / effect log
The playground above is the live thing. This is the contract behind it: every event the pure core accepts, every effect the driver dispatches, in the order §5.2 prescribes. next = select_next_action(state) — emitted only when an eligible instrument exists, the kill-switch is Live, the mark is fresh, margin_status validated the breach, and auto_liquidation_enabled was true at trigger ingress.
| Event | PersistAccountpostgres | RemovePersistpostgres | CancelAllOrdersorder-gateway | PlaceLiquidationOrderorder-gateway | AppendAuditLogclickhouse | EmitSlackslack |
|---|---|---|---|---|---|---|
| TriggerCandidateValidated StaleMarginStatus | — | — | — | — | StaleTriggerDropped | — |
| TriggerReceived validated MarginBreach | Liquidating | — | ✓ | next | TriggerReceived | BreachDetected |
| TriggerReceived AdminManual | Liquidating | — | ✓ | next | TriggerReceived | BreachDetected |
| TriggerReceived duplicate · §6.5 | — | — | — | — | DuplicateTriggerDropped | — |
| TriggerReceived auto_liq=false · §6.2 | Parked AutoLiquidationDisabled | — | — | — | AccountParked | AccountParked |
| OrdersCancelled | Liquidating | — | — | next | OrdersCancelled | — |
| OrderAcked | Liquidating inflight[i].exchange_id | — | — | — | OrderAcked | — |
| OrderFilled | Liquidating attempts[i]=0, drop inflight[i] | — | — | next | OrderFilled | — |
| OrderRejected attempts < N | Liquidating attempts[i]++ | — | — | next | OrderRejected | — |
| OrderRejected attempts ≥ N | PreParkProbe until now+safety_tick | — | — | — | PreParkProbeStarted | — |
| Tick probe timeout / MMA < 0 | Parked AttemptsExhausted | — | — | — | AccountParked | AccountParked |
| RiskSnapshotUpdated MMA ≥ 0 · liquidating/probe/recoverable parked | Resolved | ✓ next tick | — | — | AccountResolved | Resolved |
| RiskSnapshotUpdated freshness=Stale · §6.3 | — | — | — | — | StaleMark | StaleMark |
| RiskSnapshotUpdated freshness=SkewViolation · §6.3 | — | — | — | — | ClockSkewViolation | StaleMark |
| Tick | — | — | — | next | — | — |
| KillSwitchChanged Off | — | — | — | — | KillSwitchChanged | EngineHalted |
| KillSwitchChanged DryRun | Live | — | — | — | — | KillSwitchChanged | — |
| ConfigUpdated | — | — | — | — | ConfigUpdated | — |
The account FSM above hides a second lifecycle: each outstanding liquidation order has its own short life. Per §10.8, the attempts counter is scoped per (user, instrument) — a stuck market parks only itself, never poisons sibling instruments. A Filled exit resets that instrument's counter to zero; a Rejected exit increments it and, on the fifth consecutive miss, starts the pre‑park probe.
PlaceLiquidationOrder is the only entry. The driver pre-generates client_order_id, the shell POSTs og.admin_place_liquidation_order, and the order goes into inflight[instrument].
OrderAcked upgrades the inflight entry with the OG-stamped L-id exchange order id. State stays in Acked until the dropcopy stream reports terminal outcome.
OrderFilled ⇒ drop inflight[i], set attempts[i] = 0. The account stays Liquidating; select_next_action picks the next (user, instrument) by largest MMR.
OrderRejected ⇒ drop inflight[i], attempts[i]++. Hard rejects (no liquidity, halted) arrive once; transient errors (network, 5xx) arrive only after og_client exhausts 100ms / 500ms / 2s backoff.
attempts[i] ≥ max_attempts_before_park the account FSM enters PreParkProbe for one safety tick. Fresh MMA ≥ 0 resolves; otherwise it parks with AttemptsExhausted. That park reason can still self‑resolve on a later healthy snapshot.
Who gets liquidated first
Multiple accounts can breach at the same minute. The pure select_next_action(state) → Option<Effect> picks the (user, instrument) with the largest current MMR across every Liquidating account. Invoked on every Tick, OrderFilled, and OrderRejected. ▷ Click Fill or Reject to drive the queue. Watch the next pick re‑sort live.
Alternatives considered (§10.3)
- rejected per‑account round‑robin · fair, but ignores RFP guidance to bias toward largest MM first
- rejected pure global priority queue · head‑of‑line risk — one stuck account starves others
- chosen per‑account worker + global priority selection · head‑of‑line isolation and global fairness. A parked
(user, instrument)is skipped; siblings keep winding down.
The mark beneath the limit
Every protective limit is mark ± liquidation_limit_bps. The shell reads mark_cache at limit‑build time — a per‑instrument RwLock<HashMap> hydrated at boot via SCAN MATCH instrument:*:mark, then kept warm by Redis keyspace notifications. Same key risk‑engine v1 uses to compute MMA, so breach detection and limit pricing share a basis. Freshness is local recv‑time (the key carries no timestamp). ▷ Press play, then kill the publisher. Watch the held instrument go stale, watch the engine skip‑and‑retry, watch the streak escalate to Park{StaleMark}.
stale_mark_streak[BTC‑USD]
Consecutive ticks the shell observed a stale or missing mark for the position the engine is trying to liquidate. Resets on first successful OrderAcked (§5.4). At mark_stale_park_after, instrument parks per §5.4.
EngineConfig
mark_freshness_max — shell refuses to act if older
3s
mark_stale_park_after — consecutive stale ticks → Park{StaleMark}
30
safety_tick
1s
mark_cache_boot_timeout
5s
▶ Run clock to start ageing marks. The instrument with the pink stripe is the one the engine is actively liquidating — that is the row whose mark gates PlaceLiquidationOrder. Other rows still age (publisher health signal), but their freshness only matters when they become the next target.
Where it can bite
The single chokepoint inventory. Every field in §4 has exactly one enforcement site. Click any card to reveal what the failure mode looks like in prod. These are the surfaces auditors and on‑call should both be staring at.
The flag that can't become Alameda
auto_liquidation_enabled is the only per‑user exemption in v1. The FTX failure mode was not that a flag existed — it was that the flag was invisible, unaudited, flippable by one engineer, and suppressed the trigger itself. Three primitives close that gap: (1) the suppression is its own audit event (TriggerSuppressedByGate); (2) every flip writes an immutable row to liquidation_gate_changes with actor + timestamp; (3) the architectural invariant in §6.8 forbids any code path keyed on a specific account. ▷ Flip the gate · ▷ fire a trigger. Watch the suppression land as a row, not as silence.
usersprop_assert_eq!(
handle_event(state, evt_with_user("usr_x")),
handle_event(state, evt_with_user("usr_y")),
);
Two synthetic user ids → identical trace. Any code path that special‑cases an account fails CI. The exemption is a snapshotted boolean read at ingress; the engine never looks up an id again.
append‑only audit · clickhouse liquidation_log
change history · postgres liquidation_gate_changes
Written in the same transaction as the users UPDATE. PG is system of record; the row is immutable; quarterly export answers compliance without discovery. Reason‑code enum, dual‑control, client‑facing exemption surface deferred → see §09.
Where the industry agrees
Eleven venues read end‑to‑end — Binance, OKX, Bybit, Bitget, BitMEX, Deribit, Hyperliquid, dYdX, Paradex, Drift, GMX/Synthetix, plus the Lighter Oct‑2025 post‑mortem and the FTX/Alameda evidence. Where five‑plus venues converge, AX copies. Where venues diverge or fail, AX picks the simplest defensible choice and writes the failure mode into the doc. Three blast‑radius reductions hide here: (a) JELLY (Hyperliquid, Mar 2025) → notional cap as a prerequisite, not a feature; (b) BitMEX (Mar 2020) → cadence throttle + bounded inflight; (c) FTX/Alameda (Nov 2022) → §6.8 + §6.9. Citations and full memo: docs/plans/liquidation-landscape-research.md.
cancel_all_orders(user) before any IOCmark ± k·bps / bankruptcy price. Deribit, OKX, Binance, HL.mark ± liquidation_limit_bps (default 50)liquidation_increment_qty + 1s safety_tickPark{state} + ops manual‑close is the fallbacksafety_tick, bounded mpscWhat v1 refuses to ship
Each deferred item carries a revisit trigger — the observable that flips it from "later" to "now". Not vibes; not roadmap. The non‑goal stays in §1; the alternative (if relevant) sits once in §10; the follow‑up lives in §11 with the trigger attached. No triple‑statement, no drift. ▷ Hover a card for the failure mode being accepted.
Tiered MM ladder
Insurance fund
ADL · socialized loss · pool‑absorbs
Park{state}Book‑depth probing
Gate governance · phase 2
RiskSnapshot protocol extension
MLL · max liquidation latency
safety_tick + book conditions. v2: explicit deadline + escalation path (operator page, route widening).breach_to_first_action_seconds p99 trends above target across rolling windowHuman approval gate · bps widening
liquidation_limit_bps with operator approval. Today: park after 5 rejects, ops manual decision.TWAP · time‑sliced liquidation
safety_tick; size is one parameter.Notional cap (prerequisite, not engine)
Named on‑call for parked accounts
AccountParked Slack is the first observable; manual unwind is the only path out.Cross‑margin handling
Nine PRs, four weeks, dark until #9
Each PR ≤ ~1k LOC. App functional after every merge. No PR depends on a future PR's behavior. Engine binary ships dark; flipping to Live is a config change, not a code change. Hard cutoff: code‑complete end of W3; soak window is non‑compressible. Bottleneck is PR #4 — strongest engineer + paired review on the mark‑cache / publisher‑liveness co‑gate. ▷ Tap any row for sandbox state at that stage.