← SoWs · afintech.dev

SoW: Price-band automation and the support queue

Tracking: A-3254 (parent). Includes A-3257 (automatic recenter), A-3663 (band alerts to incident.io) and A-4540 (dated futures).

Replaces: docs/rfc/auto-band-recenter.md. Uses: POST /admin/trading-reference-price, POST /admin/anchorage/deposits/{id}/credit, the Databento and Pyth sources in underlying-publisher, and the recon-engine check framework. Does not apply to: the Bitnomial edition, where the venue sets the bands.

Objective. When a price band is close to rejecting fairly priced orders, AX opens a task in a support queue and sends an alert to #ax-support. An instrument runs in one of three modes. The modes ship in this order:

  1. alert: the task shows the detail. An operator moves the band by hand.
  2. propose: the task also carries a proposed new anchor price. An operator accepts or rejects it.
  3. auto: a rule accepts the proposal when the move passes every safety rule. When a rule fails, the task waits for an operator.

The support queue is a new page in the admin GUI, below Dashboard. Two conditions that exist today, uncredited deposits and Bitnomial account sync errors, use it first (Validation producers). Band tasks follow.

Terms

The problem

When the underlying price moves and the anchor does not, the market price reaches a band limit and EP3 rejects orders. No alert is sent. An operator learns of it from a customer, or from a red card on the admin dashboard. The operator then finds a price on a public website and types it into the settlement dialog.

Evidence, read on 2026-09-21:

Current behavior

The model

producer ──opens──> support task ──notifier──> incident.io ──> #ax-support
                        │
                        ├── proposal (optional) ──accept──> executor ──> EP3
                        └── events (append-only)
Record Meaning Table
Support task One condition on one subject that needs an operator. support_tasks (new)
Proposal The action that the producer recommends, with the inputs it used, its revision, its expiry and the reasons it waits for a person. It is part of the task row. A task in alert mode has none. support_tasks.proposal
Event One thing that happened to a task, and who did it. support_task_events (new)

A producer is a service that detects a condition. It runs as one instance. On every run it reports each condition that holds. It opens a task when a condition starts. It updates last_seen_at while the condition holds. It resolves the task as self_resolved when the condition ends. When a producer stops, its tasks stay open and last_seen_at shows their age. A producer writes tasks through ax_db. recon-engine already writes to Postgres this way (the leaderboard upsert), so it needs no new role. The band producer is a recon-engine check, because recon-engine runs as one instance and already has scheduling, timeouts, and Postgres and ClickHouse pools.

The notifier is a loop in recon-engine. It sends one incident.io alert for each open urgent or action task. It resolves the alert when the task resolves or is deferred. It uses the fire, re-send and resolve logic that recon-engine checks use today (reconcile_alert in rs/recon-engine/src/lib.rs). The deduplication key is support-task/{env}/{subject}, so each symbol has its own alert. The alert title is support-queue: {subject}, and the description is detail rendered as text, as format_details does for a check today. A new producer needs no alert code.

The executor is the code in api-gateway that carries out a proposal. It is the only code that writes a band change to EP3 for a task. api-gateway already has the EP3 admin client and the trading-reference-price write. recon-engine needs only read access to EP3.

Decisions

  1. A task is one condition that one decision closes. A producer detects it, and it is urgent. Onboarding review and MMLP accrual approval have many steps and keep their own pages. Such a page can open a task that links to it.
  2. One open task per subject. The subject is {kind}:{key}, for example price_band_drift:AAPL-PERP. The kind is the text before the first colon, so a kind name cannot contain one. The key is stable for the life of the condition. A partial unique index enforces this, as it does for btnl_ib_account_sync_errors. A condition that continues updates its task.
  3. Reject defers the task. To reject a proposal, the operator gives a note and a duration (default 1 hour). The task stays open and its alert resolves. When the deferral ends and the condition still holds, the producer makes a new proposal and the alert fires again. Reject does not close the task, because the producer opens a new task on its next run while the condition holds.
  4. Accept names the proposal revision. The executor refuses with 409 when proposal.rev has changed, when the proposal is more than 5 minutes old, or when the EP3 anchor differs from the anchor in the proposal.
  5. Accept can set a different price. The operator can enter a price within the 15% ceiling. The event records the proposed price and the entered price. The difference measures how good the proposals are.
  6. Only a person can use the accept route. The route uses require_human_admin_actor (rs/api-gateway/src/admin_routes.rs), which refuses service tokens and API keys. The rule uses a second route, rule-accept, which takes only the service token.
  7. The executor checks the rules itself. On rule-accept the executor reads the mode and the caps and applies every rule in Auto mode. A caller with the service token cannot move a band further than the instrument's settings allow.
  8. The band check sends no alert of its own. It is an InvariantCheck, so it appears on the Invariants page and in invariants_log. Its paging flag is off (the flag is from the continuous-recon-checks RFC). Its tasks send the alerts.
  9. A recenter writes only the trading reference price. It does not write the settlement price. It does not clear the trading reference price. The band percentages change only for a dated future (Dated futures).
  10. A proposal comes from an external price series. The producer does not use the AX last trade, mark price, or best bid and offer. Trading on AX then cannot move the AX band.
  11. An automatic move sends no alert. The task opens and resolves in the same run. It appears in the History tab and in a daily digest.
  12. The mobile screen ships with the web page. Many band moves happen overnight, when the operator has only a phone. gui/packages/admin/src/editionParity.test.ts fails when a web section has no mobile screen.
  13. The page does not depend on the edition. Band tasks exist only on the EP3 edition. The page, the tables and the notifier work on both.
  14. Rejects from one account send no alert. The reject trigger needs two or more accounts. Rejects from one account open an info task, which appears in the queue and sends no alert.
  15. Alerts are on for every EP3 instrument. An instrument with no settings runs in alert mode. An operator turns on propose or auto for each instrument.

Schema

The tables go in db/postgres/1.sql. Atlas applies them.

CREATE TABLE support_tasks (
    id            UUID PRIMARY KEY,
    subject       TEXT NOT NULL,        -- '{kind}:{key}', see decision 2
    severity      TEXT NOT NULL
        CHECK (severity IN ('info', 'action', 'urgent')),
    detail        JSONB NOT NULL,       -- rewritten on every producer run
    proposal      JSONB,                -- NULL when there is none
    producer      TEXT NOT NULL,
    opened_at     TIMESTAMPTZ NOT NULL,
    last_seen_at  TIMESTAMPTZ NOT NULL,
    deferred_until TIMESTAMPTZ,
    resolved_at   TIMESTAMPTZ,
    resolution    TEXT CHECK (resolution IN
        ('accepted', 'auto_applied', 'self_resolved', 'dismissed')),
    resolved_by   TEXT,
    CHECK ((resolved_at IS NULL) = (resolution IS NULL))
);
CREATE UNIQUE INDEX support_tasks_one_open
    ON support_tasks (subject) WHERE resolved_at IS NULL;

CREATE TABLE support_task_events (
    id         BIGSERIAL PRIMARY KEY,
    task_id    UUID NOT NULL REFERENCES support_tasks (id),
    at         TIMESTAMPTZ NOT NULL DEFAULT now(),
    actor_kind TEXT NOT NULL CHECK (actor_kind IN ('admin', 'rule', 'producer')),
    actor      TEXT NOT NULL,
    event      TEXT NOT NULL CHECK (event IN (
        'opened',            -- producer
        'severity_changed',  -- producer; the alert re-sends at the new priority
        'proposed',          -- producer; payload holds the proposal
        'deferred',          -- admin; note required
        'rejected',          -- admin; note required, sets a deferral
        'accepted',          -- admin; payload holds rev and both prices
        'executed',          -- admin or rule
        'execution_failed',  -- admin or rule; payload holds the error
        'rule_declined',     -- rule; payload holds the failed rule names
        'self_resolved',     -- producer
        'dismissed')),       -- admin
    note       TEXT,
    payload    JSONB
);

A trigger makes support_task_events append-only. application_decisions has the same kind of trigger. actor_kind has the same meaning as application_decisions.decided_by_kind. Stage A emits only opened, severity_changed, deferred, self_resolved and dismissed; the CHECK lists every event now to save a later schema change. A detail update is not an event, or the table would gain one row per subject per run.

The three JSON fields answer three questions from the operator. detail is what the producer saw. proposal is what it suggests, and it carries rev, expires_at and needs_human_reasons. needs_human_reasons is why the proposal waits for a person instead of the rule; it is empty when the proposal is empty and resets when the proposal is replaced. The queue has no claim: the team is small, and concurrent accepts are handled by the row lock and rev.

Severity sets the alert. An urgent or action task sends an alert and counts in the badge. An info task does neither.

Each instrument has its settings in one new column:

instruments.band_recenter_config JSONB NULL

BandRecenterConfig {
    mode: Off | Alert | Propose | Auto,   // column NULL means Alert
    max_up_pct: Decimal,                  // default 10, ceiling 15
    max_down_pct: Decimal,                // default 10, ceiling 15
    max_cumulative_pct: Decimal,          // default 20
    alert_fraction: Decimal,              // default 0.5
    sources: [Source],                    // default [Databento, Pyth, Ornn]
    width: Option<{ base_pct, carry_pct_per_year }>,   // dated futures only
}

The band producer

The band producer runs every 60 seconds for each EP3 instrument:

series   = the first source in `sources` with a fresh newest row
           (fresh = newer than 3 × the source's write interval)
check    = the next fresh source in `sources`, if there is one
twap     = mean of `series` over the last 5 minutes
anchor   = trading reference price, else settlement price   [EP3 book stats]
drift    = (twap − anchor) / anchor
consumed = |drift| / band percentage on that side

underlying trigger : consumed ≥ alert_fraction              (default 0.5)
reject trigger     : ≥ 50 PRICE_OUT_OF_BOUNDS rejects on the symbol in the
                     last 5 minutes, from ≥ 2 accounts       [order_log]

Dated futures

A dated future uses the same anchor as a perp: the TWAP of the underlying. The anchor has no basis term. The band allows for the basis instead. It is wider when expiry is far, and it equals base_pct at expiry:

band pct = base_pct + carry_pct_per_year × years to `instruments.expiration`

carry_pct_per_year is the largest annual basis (cost of carry) that the band allows on each side. With base_pct 10 and carry_pct_per_year 12, a contract three months from expiry has a ±13% band. One week from expiry it has ±10.2%.

The producer computes the width once a day, after settlement. When the width differs from the EP3 percentages by 0.25 points or more, the producer opens a price_band_width task with a set_price_bands proposal and calls rule-accept. The executor writes through the existing set_price_bands code. This runs in every mode except Off, because the width depends only on the date and on two settings that an operator chose. The task appears in the History tab and sends no alert.

Alert mode

Stages A, C and D deliver this mode.

A task sends an alert to #ax-support with a link to the task. Each symbol has its own alert. The task has no proposal. The operator recenters with the trading-reference dialog from stage 0. The producer resolves the task on its next run.

incident.io and the queue each own one part of the work. incident.io owns the page: notify, re-notify, acknowledge and snooze. These say only that a person has seen the alert. They do not change the task, and the queue does not read them back. incident.io has no webhook and no API for an acknowledgement or a snooze. The queue owns the task: defer, dismiss, accept and reject. These resolve the alert, and the route cancels the escalation with it.

An operator Result
Acknowledges or snoozes in incident.io The re-notifications stop or pause. The task stays open and counts on the badge.
Defers in the queue The alert resolves. It fires again when the deferral ends, if the condition holds.
Dismisses in the queue, or the producer self-resolves The alert resolves.
Resolves the alert in incident.io The notifier fires it again within a minute, because the task is open.

An incident.io snooze ends on a timer and does not know the condition. A deferral ends the same way but fires again only if the condition holds, and it is in the task's history. So an operator who wants quiet for longer than the re-notification cadence defers the task. The alert's resolve text names the deferral, so incident.io shows why the alert closed.

Propose mode

Stage E delivers this mode.

When the mode is Propose and an allowed source is fresh, the task carries a proposal:

{ "action": "set_trading_reference_price",
  "rev": 3, "expires_at": "…", "needs_human_reasons": [],
  "symbol": "SNDK-PERP", "price": "1738.90",
  "basis": { "anchor": "1655.20", "anchor_kind": "settlement",
             "anchor_set_time": "…", "twap": "1738.94",
             "source": "DATABENTO", "window_secs": 300, "points": 5,
             "new_lower": "1565.05", "new_upper": "1912.75" } }

The producer replaces the proposal and increments rev in two cases: the TWAP differs from the proposed price by more than one tenth of the band width, or the proposal has expired. The proposed price therefore stays the same while the operator reads it.

The routes are in a new file, rs/api-gateway/src/admin_support_routes.rs, under /admin/support-tasks:

Route Who can call it Effect
GET /, GET /{id} admin Lists tasks with the open count; returns one task with its events
POST /{id}/defer admin Sets deferred_until; needs a note
POST /{id}/reject a person Decision 3
POST /{id}/accept a person Decisions 4 to 6; body {rev, price_override?, note?}; price_override applies to band proposals only
POST /{id}/dismiss a person Closes a task of a kind that does not resolve itself
POST /{id}/rule-accept service token Auto mode

Accept does these steps in order. The executor dispatches on proposal.action; stage U1 delivers the route with the credit_deposit action, and stage E adds set_trading_reference_price.

  1. Lock the task row.
  2. Check the revision, the expiry and the anchor (decision 4).
  3. Write to EP3 through the existing set_trading_reference_price code.
  4. Record executed and resolve the task as accepted.

When the EP3 write fails, the executor records execution_failed and the task stays open.

Auto mode

Stage F delivers this mode.

When the mode is Auto, the producer calls rule-accept on a proposal in the run that creates it. The executor applies the move when every rule in the table passes. When a rule fails, the executor records rule_declined and adds the reason to the proposal's needs_human_reasons. The task then stays open, sends an alert, and waits for an operator, as in propose mode.

Rule Default Reason when it fails
The move from the current anchor is within the cap for its direction 10%, ceiling 15% extreme_move
The move from the trusted anchor is within the cumulative cap 20% cumulative_limit
The window has at least 4 of the 5 expected points thin_series
The points in the window differ by less than 3% unstable_series
The AX mid price, when both sides have quotes, is within half a band of the TWAP market_disagrees
A second fresh source, when there is one, is within 1% sources_disagree
The time is not within 15 minutes of the instrument's settlement time near_settlement
The anchor was last set at least 30 seconds ago none; the producer tries again on its next run

The trusted anchor is the later of two prices: the last settlement price and the last price that an operator accepted. The directional cap limits one move. The cumulative cap limits the total of the automatic moves since the trusted anchor. A faulty series can drift in steps that each pass the directional cap, and the cumulative cap stops the total.

The executor applies a move in full or not at all. It does not apply part of a move that exceeds a cap.

The Support Queue page

Validation producers

Stages U1 and U2 put two conditions that exist today into the queue before the band producer. They exercise the tables, the routes, the notifier and the page on real rows, on both editions, with no EP3 write in the loop. The two were chosen from a survey of the admin pages and the service loops on 2026-09-22 (Other task kinds has the rest).

U1: uncredited deposits (EP3 edition)

Every Anchorage deposit stays at INGESTED until a person calls POST /admin/anchorage/deposits/{id}/credit. No GUI calls that route, and the Deposits page is read-only. No alert exists for a deposit that waits.

U2: Bitnomial account sync errors (AIEX edition)

onboarding_gateway.btnl_ib_account_sync_errors is already the shape of a task: one open row per (email, kind), first_seen_at, last_seen_at, and resolved_by = 'sweep' for self-resolution. Its page has the only sidebar badge. The hourly sweep in rs/bitnomial-reconciler/src/ib_accounts/task.rs is the producer.

Other task kinds

The survey on 2026-09-22 found these further conditions. None is in scope.

Condition today Fit
Failing invariants Each check is a task keyed by check_id. The Invariants snooze is keyed by check, and a per-task deferral would lose it when a check flaps. Convert after the notifier has run on U1 for a month.
Monthly loss-limit breaches Latched per account in monthly_loss_limits; the cure is one PATCH; no alert today. A good producer, but breaches are rare, so it validates little.
TRM screening in OUTCOME_UNKNOWN No route and no GUI reopen it. Needs the reopen route before it can be a task.
Parked liquidations liquidation_engine.account_state has the task shape, but no service runs the engine yet.
EP3 surveillance alerts Stays as it is. EP3 owns the status.
Onboarding review, MMLP accruals, special settlements Stay as they are (decision 1).

Delivery

Stage Scope Reviewers
0 Trading-reference dialog on web and mobile; anchor kind and set time in MarketsTable; settlement dialog renamed "Set EP3 Settlement Price (EOD)"; runbook for a band alert frontend
A support_tasks, support_task_events, ax_db entities; the read, defer and dismiss routes backend
B Support Queue page, badge, mobile screen, dashboard card frontend
C Notifier loop in recon-engine; terraform route and escalation path backend, infra
U1 anchorage-deposits-uncredited producer; the accept route and the executor with the credit_deposit action; the deposit card backend, frontend
U2 Bitnomial sync shadow producer in bitnomial-reconciler; dismiss wired to resolve_by_id; the sync-error card backend, frontend
D Band producer in alert mode, starting from the check on origin/alee/auto-band-recenter-impl: both triggers, the cross-check, band_recenter_config, the dated-future width, an EP3 read client in recon-engine, the paging flag backend
E Band proposals; the reject route; the set_trading_reference_price action in the executor; the band card actions backend, frontend, high-risk
F rule-accept; the rules in Auto mode; the daily digest backend, high-risk
G Pyth rows in prod underlying_prices for overnight equities, futures and crypto; pyth_config on every instrument that has a Pyth feed backend 
0
A ── B ─────────────────────────────┐
└─── C ── U1 ── U2 ── D ── [alert] ── E ── [propose] ── F ── [auto]
G

Conditions to move a symbol from propose to auto. The symbol has been in propose for two weeks or more. Operators have decided ten or more proposals. They accepted 90% or more with no price change. They rejected none as a wrong price. The numbers come from support_task_events. Use demo first, with one symbol for a week. Then use prod, one symbol at a time.

Tests. These cases need a test, because the feature is a risk control: EP3 disconnects during accept; api-gateway restarts between accepted and executed; recon-engine restarts with a task open; two operators accept at the same time; a proposal expires while the confirm dialog is open; an incident.io alert differs from its task. ep3-mock already stores the trading reference price. For U1: two operators accept the same deposit; a deposit is credited through the route while its task is open; api-gateway restarts between the credit and executed.

Failure and restart

The producer keeps no state in memory. EP3 holds the anchor and the time it was set. Postgres holds the settings and the tasks. ClickHouse holds the series. After a restart, the next run reads all of them again. The 30-second rule reads the set time from EP3, so a producer that restarts in a loop cannot recenter more often than every 30 seconds.

Failure Result
recon-engine is down Bands do not move. Open tasks stay open and last_seen_at stops. The service heartbeat sends an alert.
ClickHouse is down No source is fresh, so the underlying trigger does not run.
api-gateway is down Accept fails. The task stays open.
The EP3 write fails The executor records execution_failed. The task stays open.

Until this ships

  1. Recenter with POST /admin/trading-reference-price. Do not use the settlement dialog. The route has no dialog until stage 0.
  2. At the start of a shift, run the reject query from The problem. It is one ClickHouse query on order_log, grouped by symbol.

Out of scope

Ops items (no PRs)

Open questions

  1. What does EP3 do to a resting order that is outside the limits after a recenter? It can cancel the order, keep it and block matching, or reject it when another order matches it. The protos do not say. The answer decides if the count of such orders is evidence only, or a rule in Auto mode. Ask Connamara, or test on the sandbox (docs/internal/operations/ep3/sandbox-reset.mdx), before stage E is in prod.
  2. Answered: recon-engine writes to Postgres as the shared postgres user today, so producers write tasks through ax_db (The model).
  3. Does the daily settlement clear the trading reference price? The proposal is no. The trading reference price stays the anchor until an operator clears it. A settlement still becomes the trusted anchor.
  4. Answered: the queue does not show an incident.io acknowledgement. incident.io owns the page and the queue owns the task (Alert mode).
  5. Is the width of a dated future linear in the time to expiry? Cost of carry is linear. A square-root term fits if the band must also allow for volatility over the remaining life. The proposal is linear.