SoW: Bitnomial ledger reconciliation

Date: 2026-09-24

Tracking: A-4816 (statement ingest), A-4817 (planner).

Replaces: the btnl-clearing-live-sync RFC, which is deleted. Uses: the Support Queue (price-band-automation), the bitnomial-reconciler service, and the recon-engine check framework. Related: aiex-venue-fee-accrual, btnl-pg-dropcopy-pipeline, continuous-recon-checks. Applies to: the AIEX edition only.

Objective. Every business day, AX compares the cash, positions, fees and funding of each introduced account with Bitnomial's statement for that account. When they differ, AX writes a plan of corrections and opens a task in the Support Queue. An admin approves the plan, and AX applies it while order entry on the account is frozen. An admin can also run the same comparison for one account at any time, against Bitnomial's live API. Bitnomial is the source of truth for settled cash, positions, fees and funding.

Terms

• Bracket. The period that one statement covers: 4PM to 4PM US/Central (America/Chicago, so 21:00Z during CDT). A bracket is named by the date on which it ends. A Monday bracket starts on Friday.
• Statement. Bitnomial's daily record of one account for one bracket.
• IB API. Bitnomial's Introducing Broker endpoints in the clearing API. They return live account state and activity.
• Drop copy. The FIX feed of fills that btnl-trade-engine books into our ledger.
• Plan. The corrections for one account and one bracket, as a list of actions.
• Parked action. An action that only a person can decide.
• Bracket close. An account's bracket is closed when its plan is empty or applied, or when a newer plan for the account is applied.
• Freeze. A per-account block on new orders while the executor applies a plan.

The problem

AX can lose fills, and nothing repairs the ledger when it does. btnl-trade-engine drops a fill when its clearing firm and account are not in the account index, and the REST /fills backfill looks back only 7 days. The btnl-* checks in recon-engine find the difference but do not correct it.

Account BC000004cf shows the effect. It has cash and a position at Bitnomial and zeros in AX. Its two fills, on 2026-08-01 and 2026-08-04, came before the account was linked on 2026-08-28. AX dropped them, and they are older than the backfill window. A repair from Bitnomial's records is the only fix. This account is the acceptance test for this SoW.

Current behavior

• Our ledger. btnl-trade-engine writes fills from drop copy to ClickHouse trades, positions and transactions, and to Postgres current_balances.
• Checks. recon-engine runs read-only checks that compare AX with Bitnomial and send alerts: btnl-positions-match, btnl-cash-transfers-match, btnl-execution-fees-match, btnl-funding-payments-match, btnl-orders-match, btnl-account-state-matches-risk and btnl-instruments-match. They do not write.
• bitnomial-reconciler. One instance runs in each environment. It runs the IB accounts sweep and the registration status sweep (docs/internal/overview/engines/bitnomial-reconciler.mdx).
• Statement delivery. Bitnomial uploads to an SFTP server that AX hosts (terraform/aiex/sftp/). SFTPGo writes each upload to s3://8dc8620c-aiex-ext-documents/incoming/bitnomial/. The bucket is versioned, uses Object Lock, and expires nothing. SFTPGo uses multipart upload, so an object is visible only when it is complete. The aiex-prod instance role has a read-only policy on incoming/* in terraform/aiex/prod/main.tf (read-clearing-docs); see Ops items.
• Statement files. Files arrive on business days at about 20:00 CT, four hours after the bracket ends, at incoming/bitnomial/statements/<externalAccountId>/<date>/statement_<date>.jsonl. Each file holds one JSON object for one account. On 2026-08-28, 67 accounts had statements. A statement has:
  • statementStartTime and statementEndTime;
  • openPositions[]: symbol, market price, signed netPosition as {tag: Long|Short, contents}, vwaPrice, openTradeEquity;
  • accountSummaries[]: cash by origin (Seg, NonReg) with beginning and ending balances, fees by type (ClearingFee, Commission, ExchangeFee, NFAFee), net liquidating value, initial and maintenance margin, and funding for the bracket;
  • perpsFundingPayments: one row per 8-hour funding interval;
  • accountCollateral: BTC and ETH with the haircut conversionFactor;
  • trades[], cashTransactions[] and cashAdjustments[]. Every file so far has these empty, so their schemas are unknown.
• ARC files. Bitnomial also uploads daily CSV and XLSX reports to incoming/bitnomial/: new accounts, trade activity, cash transactions and open positions (ARC_*_<date>). AX does not read them.
• IB API. GET /introducing-broker/account-states returns balance, equity and margin for each account, cached for up to 2 minutes. /cash-activity (also as an SSE stream) returns deposits and withdrawals. /trade-activity returns fills with fees by type and feeTotal. Prices on /trade-activity are in cents. The client in rs/bitnomial/src/clearing/ models all three.
• Account identity. Every source uses the external account id, for example BC000004cf. It is FIX tag 1 and trading_accounts.btnl_account_id. The clearing firm code is DCZ.

The model

S3 statement ──ingest──> ClickHouse statement tables ─┐
                                                      ├─planner─> plan + actions (Postgres)
our ledger (ClickHouse, Postgres) ────────────────────┘                │
                                                                       ▼
                               #ax-support <── notifier <── support task
                                                                       │
                                                             admin accepts
                                                                       ▼
                                        executor: freeze, drain, check, apply, unfreeze

bitnomial-reconciler runs the statement ingest, the planner and the ad-hoc mode. It runs as one instance, so no loop needs a leader. The planner runs after each ingest pass, so a new statement starts planning.

recon-engine stays read-only. Its checks read what bitnomial-reconciler writes, and its notifier sends the alerts for support tasks.

The executor applies an approved plan. It runs in bitnomial-reconciler (decision 10).

Decisions

1. Bitnomial is the source of truth for settled state. Between statements, AX tracks each account from drop copy. Each statement corrects it.
2. Nothing changes the ledger until an admin approves the plan. No action is applied automatically. This is propose mode in the price-band SoW.
3. The executor does not apply adjustments. A plan with a position_adjustment or balance_adjustment gets no proposal, and its task says why. A person repairs the account by hand. An adjustment has no fill that explains it, and no known case needs one. Most differences are fills that the planner does not see until Stage 4. balance_adjustment depends on ledger corrections (#3509).
4. The freeze is needed only to apply. The daily comparison uses the ledger at the end of the bracket, which is hours in the past when the planner runs, so it needs no freeze. The executor freezes the account while it applies a plan, so the customer cannot trade on a position that the executor is changing.
5. Statements are read with the S3 SDK. The IAM read policy exists, and the SDK is one dependency with no host setup. A filesystem mount would add a daemon and a failure mode.
6. btnl-trade-engine books the fills. The executor queues each fill in Postgres, and btnl-trade-engine books it through the same state machine as drop copy. btnl-trade-engine holds each account's position in memory and writes it on every fill, so it must book every fill on its accounts. The state machine books queued fills and drop-copy fills one at a time, in the order it receives them.
7. Every applied action is tagged. Each action is append-only and carries reference_id = recon://<date>/<plan-id>, like the fee RFC's clearing-report://, so the ledger records why it changed.
8. Plans are support tasks. The planner is a support-queue producer. The Support Queue page shows the plan on web and mobile, and the notifier sends the alert. There is no reconciliation page.
9. One task per account. The subject is btnl_recon:{external_account_id}, and it stays the same while the account has unapplied plans. A subject per bracket would open one task and one alert for each bracket after an outage.
10. bitnomial-reconciler runs the executor. It runs one instance for each environment and holds the plan store, so the executor needs no leader. btnl-trade-engine books the executor's fills (decision 6), and recon-engine is read-only. The accept route in api-gateway records the approval and returns, because freeze and apply can take longer than one HTTP request.
11. The newest plan covers the older ones. The proposal names the newest unapplied plan, and applying it closes the older ones. Each plan compares the whole account at the end of its bracket, not the change since the plan before it. A difference that lasts three brackets is in all three plans, so the executor applies only one of them.
12. A recon task closes only when the account has no unapplied plan. This happens when the executor applies a plan or a replan finds no difference. An admin cannot dismiss the task, because sync_conditions opens it again on the next pass.

Schema

ClickHouse (db/clickhouse/init.sql). All three tables are ReplacingMergeTree(ingested_at), so when a file is ingested again, the newer rows replace the older ones.

Postgres (db/postgres/1.sql, schema bitnomial_reconciler).

A plan's status is one of these:

The support task events record who approved a plan, so plans needs no approver columns.

Stage 1: statement ingest

A loop in bitnomial-reconciler, turned on by BTNL_STATEMENTS_S3_BUCKET. bitnomial-reconciler clearing-statements runs one pass and exits.

• Each pass lists incoming/bitnomial/statements/, skips the keys already in btnl_statement_files, downloads the rest, and writes their raw and parsed content to ClickHouse. There is no cursor. A rerun, or a restart after a crash, repeats only the missing work.
• A file that fails to parse is stored raw with parse_error set. Each pass parses these files again from raw, so a parser fix applies without reading S3.
• The recon-engine check btnl-statements-current fails when the newest parsed bracket is older than expected, or when a file has a parse error. It expects a business day's statement by 23:00 CT. It does not know exchange holidays, so a holiday gives a false alert.
• A failed pass is retried with backoff for up to 30 minutes. After that, the service exits and its health check fails.
• The ARC files and the trades[] and cashTransactions[] contents are not parsed. They wait for a populated sample (Ops items).

Stage 2: planner

The planner runs for each account after each ingest pass (daily mode), or for one account on request (Stage 4).

• Our side. Our ledger at the end of the bracket, from ClickHouse timestamps and sequence numbers: cash, open positions, and the fills in the bracket.

• Bitnomial's side. The statement in daily mode, or the IB API in ad-hoc mode.

• Actions.

  • insert_fill: a Bitnomial fill with no matching trade in AX, as for BC000004cf. The executor books it with the venue execution id, price, quantity and time.
  • insert_transaction: a missing deposit, withdrawal, fee or funding payment.
  • position_adjustment: a position difference that no fill explains, such as a give-up, a transfer or a correction after settlement. Always parked.
  • balance_adjustment: a cash difference left after all other actions. Always parked.

  Until a statement with a populated trades[] exists, the planner writes only parked adjustments.

• Support task. When BTNL_RECON_SUPPORT_TASKS is true, each run calls DbSupportTask::sync_conditions with one condition for each account that has an unapplied non-empty plan.

  • detail lists each open bracket and its difference.
  • The proposal is {action: "apply_btnl_recon_plan", plan_id, rev, expires_at, needs_human_reasons}. plan_id is the newest unapplied plan (decision 11), and rev is its creation time in microseconds, so a replan changes it. There is no proposal when the plan has an adjustment (decision 3) or no AX trading account maps to the Bitnomial account. detail.unfixable_reasons gives the reason.
  • Severity is action for a non-empty plan. It is info for a difference that the planner expects, such as the Friday drop-copy reset (Stage 5), so that it sends no alert.
  • When the account has no unapplied plan, the planner leaves it out of the call and the task resolves as self_resolved. An admin cannot dismiss the task (decision 12).

• Check. btnl-recon-plans-clean fails while a plan is pending. It appears on the Invariants page and sends no alert of its own; the task sends the alert.

• Routes. Read-only admin routes return a plan and its actions for the task card.

Stage 3: freeze and executor

• Freeze. A new flag on trading_accounts that blocks new orders. The order gateway checks it with is_onboarded and the user's is_frozen. It differs from is_close_only, which allows orders that reduce a position. An admin or the executor sets it, and each change writes an audit row.
• Approval. An admin accepts the task on the Support Queue page. The accept route (POST /admin/support-tasks/{id}/accept, price-band stage U1) accepts only a signed-in person (require_human_admin_actor). It returns 409 when rev has changed or the proposal has expired.
• Apply. For apply_btnl_recon_plan, the accept route records accepted with the rev, returns 202 and does not resolve the task. It returns 409 while an earlier accept waits for the executor. The executor in bitnomial-reconciler polls for open recon tasks whose newest event is accepted. When the rev no longer names the newest plan, it records execution_failed. Otherwise it marks the plan applying, and then:
  1. freezes the account;
  2. checks the plan's inputs against the ledger: AX's cash and positions at the end of the bracket must still equal the plan's details;
  3. queues each insert_fill in bitnomial_reconciler.fill_bookings. A fill whose execution id the ledger has on this account counts as applied. A fill whose execution id the ledger has on another AX account fails the plan, because the executor books one side of a trade and that trade is between two AX accounts;
  4. waits until each fill is in ClickHouse trades, with a time limit;
  5. in one Postgres transaction, marks the plan applied, marks the account's older unapplied plans superseded, records executed and resolves the task as accepted. The producer cannot resolve the task as self_resolved before this transaction commits;
  6. removes the freeze.
• Booking. btnl-trade-engine polls fill_bookings for rows that are not booked and sends each one to its state machine. The state machine skips an execution id it has booked, and books the rest one side at a time, as for a fill whose counterparty is another firm. The trade row keeps the fill's time. The position row and its PnL carry the booking time, so as-of reads before the booking do not change. The order id is recon://<date>/<plan-id> (decision 7). Booking does not include fees. Stage 4 books them as Fee transactions.
• Changes during apply. When step 2 finds a change, the executor marks the plan aborted, replans the account for that bracket, and records execution_failed. The new plan gets a new proposal and rev. Any other failure sets the plan to pending, records execution_failed and keeps the task open.
• Restart. At startup the executor sets each applying plan to pending, records execution_failed and removes its freeze. An admin must accept the plan again. The rerun skips fills that are already booked.

Stage 4: ad-hoc mode

An admin starts a comparison for one account now. The planner uses the IB API in place of a statement, and the executor is the same. This is the tool for an account that looks wrong now, and for repairs such as BC000004cf.

• As-of time. The executor freezes the account before the planner reads the IB API, so the comparison has a fixed time. account-states can be up to 2 minutes old, so the planner reads it until two reads agree, or parks the difference.
• Freeze length. The freeze lasts while an admin reads the plan. The proposal's expires_at limits this time. When the proposal expires, the executor removes the freeze and the planner writes a new proposal.
• Producer name. sync_conditions resolves every open task of a producer whose subject is not in the call. An ad-hoc run reports one account, so it must not use the daily producer's name. It adds its account to the daily producer's next call, or it uses a producer name for each account (bitnomial-reconciler/adhoc:{external_account_id}).

Stage 5: acceptance and hardening

• Repair BC000004cf with Stage 4. The task shows the two missing fills (execution ids 7668829274270413697 and 7669932909071499348, $0.14 in fees each) and the $1,000 deposit. An admin accepts it. After the apply, AX matches the statement. This is the definition of done.
• Gating. btnl-positions-match and btnl-cash-transfers-match change from report-only to gating, using the bracket-close state.
• Schedule. The daily planner runs after the statements arrive. The Friday maintenance window from 5PM to 6PM CT resets the drop-copy sequence number, and sometimes ends late. The planner expects this gap and gives it info severity.
• Tests. Client disconnect, server disconnect, a restart or crash during apply, and reconnect and recovery (CLAUDE.md). Also the support-queue cases: two admins accept the same task; api-gateway or the executor restarts between accepted and executed; a proposal expires while the confirm dialog is open; an incident.io alert differs from its task.

Delivery

• Each stage needs the stage before it.
• Stage 3 needs the accept route from price-band stage U1.
• Recon tasks turn on after the price-band U2 producer has run in demo for a week. U2 is the first producer outside recon-engine and the first on the AIEX edition.
• Recon tasks send alerts only after the AIEX alert route exists (Ops items). Until then they appear on the Support Queue page only.

Failure and restart

bitnomial-reconciler keeps no state in memory. S3 holds the statements, ClickHouse holds what was ingested, and Postgres holds the plans and tasks. After a restart, the next pass reads all of them again.

Out of scope

• Automatic apply. Every plan needs an admin (decision 2). A later auto mode uses the price-band SoW's rule-accept route and rules.
• ARC file ingest. The statements hold the same data.
• The EP3 edition.

Ops items (no PRs)

• Run terraform -chdir=terraform/aiex/prod apply for the read-clearing-docs policy. On 2026-08-31 the aiex-prod role got AccessDenied on the bucket.
• Place one small trade on BC000004cf. The next statement then has a populated trades[], which gives its schema.
• Add an incident.io alert route for the AIEX environments, because the support-queue route in terraform/incident-io/prod.tf reads only the ax_prod source. Set RECON_ENGINE_INCIDENT_IO_* in the AIEX environments.

Open questions

1. When the statement and account-states differ, which one wins? The proposal is that the statement wins for its bracket, and account-states only raises an alert. It is never a source for a cash action.
2. Does a freeze need a message to the customer, such as a banner in the GUI? A daily freeze lasts seconds. An ad-hoc freeze lasts until the admin decides or the proposal expires.
3. When AX and Bitnomial disagree on a fee for a long time, which number does the customer pay? This is a business decision.