Runbook: Fee Program

Last updated: 2026-08-20

This runbook covers operating the dynamic maker/taker fee program: publishing a new ladder, how accounts are re-tiered from trailing volume, the engine's failure and safety behavior, and how to grant a promotional rate or roll back a ladder. The engine shipped under A-4094 and its SoW has been retired (see git history for the design); this document is the operational source of truth. The re-tiering task was extracted from recon-engine into its own accrual-engine service under A-4647 — same logic, new host and env-only config.

How it works

A single uniform ladder applies to every user-owned trading account — there is no per-account published program. Each account is placed on a tier by its own trailing-30-day traded notional; the tier sets its maker/taker rate.

Ladder = fee_schedules rows. The active ladder is the row with the greatest effective_at <= now (db/postgres/1.sql, fee_schedules). Rows are immutable — a DB trigger rejects UPDATE/DELETE/TRUNCATE. You publish a change by inserting a superseding row, so the row sequence is the full rate history. Tiers are inline JSONB ({min_notional_usd, maker_fee, taker_fee}, numbers as strings for NUMERIC precision).
Authoritative source is Git, not the DB. The canonical ladder is rs/admin-cli/examples/fee_program.yaml, typed by rs/sdk-internal/src/fee_program.rs (FeeProgram) with a validate() gate (non-empty, monotonic tiers, no duplicate thresholds). The DB row is a materialized cache of that file. A unit test pins the example to the published ladder (rs/admin-cli/src/fee_program.rs).
Engine re-tiers accounts. refresh_fee_rates_task in accrual-engine (rs/accrual-engine/src/fee.rs) runs on a periodic tick. Each tick:
- loads the current schedule (no-op if none, or if the ladder fails validate() — it will not tier anyone on a bad ladder);
- computes each account's hysteretic trailing volume from ClickHouse trades (multiplier-scaled notional);
- maps volume → tier → (maker_fee, taker_fee);
- writes only the accounts whose rate changed, in one bulk UPDATE guarded on the schedule still being current.
Rate propagates unchanged. The engine only writes trading_accounts.maker_fee / taker_fee. Logical replication carries the change into trade-engine2, which applies fee = rate × price × quantity at fill time. recon-engine's trade_fees_square invariant still holds — it checks the rate recorded on each trade, not the schedule.

Hysteresis (promote fast, demote slow)

Tiering volume is the max over the last hysteresis_days daily trailing-30d windows (hysteretic_volumes in fee.rs). An account promotes the first day it clears a threshold and only demotes once every window in the lookback has rolled below it. hysteresis_days = 1 is plain trailing-30d. Default is 7; valid range 1..=30.

Publishing a new ladder

Edit the canonical file. Change rs/admin-cli/examples/fee_program.yaml. Rates are decimal fractions of notional (0.0005 = 5 bps); a negative maker_fee is a rebate. Keep min_notional_usd thresholds distinct and ascending.
Update the pinning test in rs/admin-cli/src/fee_program.rs so the example-matches-published-ladder assertions reflect the new numbers, and run:
```
cargo test -p ax-admin-cli --manifest-path rs/Cargo.toml
```

Dry-run the materializer to see the plan without writing:

[ec2-user@ax-demo] $ run admin_cli fee materialize \
  --file rs/admin-cli/examples/fee_program.yaml --dry-run

Materialize (inserts one new fee_schedules row, effective now):
```
[ec2-user@ax-demo] $ run admin_cli fee materialize \
  --file rs/admin-cli/examples/fee_program.yaml
```
Runs against the environment's Postgres (standard PostgresDbConfig env vars). The command prints the new schedule UUID.
Verify the row is current and the engine picks it up on the next tick:
```
GET /admin/fee-schedules   -- newest effective first
```
Watch the accrual-engine log for fee schedule <id>: N of M accounts re-tiered.

Warning: Publishing takes effect immediately (effective_at = now) and is irreversible in place — the row cannot be edited or deleted. To back out, publish another superseding row (see Rolling back below).

Configuration

accrual-engine is configured by environment variables only — there is no config file section. All four are optional and fall back to the defaults below.

Env var	Meaning	Default
`FEE_RATES_ENABLED`	Run the re-tiering task at all	`true`
`FEE_RATES_REFRESH_SECS`	Tick cadence (must be nonzero)	`86400` (daily)
`FEE_RATES_STARTUP_DELAY_SECS`	Delay before first tick	`60`
`FEE_RATES_HYSTERESIS_DAYS`	Lookback windows for max-volume hysteresis (1..=30)	`7`

The service also needs the standard PostgresDbConfig / ClickhouseConfig env vars and REDIS_CONNECTION_STRING (health publishing under the accrual-engine key).

A bad value fails startup rather than the first tick: a zero refresh or a hysteresis_days outside 1..=30 is rejected in from_env, so the pod crash-loops instead of tiering on a bad knob. The resolved values are logged at boot (fee-rates: enabled=… refresh=… startup_delay=… hysteresis_days=…) — check that line first when the engine is not behaving as configured.

Stopping the engine

Set FEE_RATES_ENABLED=false and restart the service. The process stays up and healthy but tiers no one; rates freeze at their current values until re-enabled. Config is read once at boot, so a change needs a restart.

Promotional rates for a single account

A fee promotion sets one account's maker/taker rates to fixed values until an expiry date. Grant one with POST /admin/fee-promotions or from the Fee Programs page in the admin GUI.

While the promotion is live, the engine sets the account to the promotion's rates and ignores the volume-earned tier. The account pays the granted rates, whether they are higher or lower than its tier. Volume traded during the promotion still accrues in the trailing windows.

The promotion expires on its own. The first tick after expires_at re-tiers the account from its earned volume. A revoke (DELETE /admin/fee-promotions/{account_id}) also takes effect at the next daily tick (≤24h). A grant applies its rates immediately.

A new grant supersedes the account's existing promotion. Retired rows remain in fee_promotions as audit history.

The promotion is the only per-account rate mechanism. The engine owns maker_fee/taker_fee and writes them each tick.

Rolling back

Schedules are immutable, so a rollback is a forward publish: re-materialize the previous ladder (or edit the YAML back and materialize), creating a new current row. The engine converges every account to it on the next tick. The prior rows remain as history.

Failure and safety behavior

Idempotent ticks. Each tick recomputes from scratch and writes only on a diff; a crashed or duplicated tick converges to the same rates. No resume token.
Missing contract multiplier → whole tick refused. A symbol with no multiplier would scale at 1x, understate notional, and over-tier the account. Because there is no dry-run/audit ledger backing the write, the tick ensure!s and aborts rather than mis-bill. Fix the multiplier (instruments) and the next tick recovers.
No / invalid current schedule → no-op. If no schedule is effective yet, or the current one fails validate(), the engine logs and tiers no one.
Mid-tick schedule swap. If a publish lands during a tick, the engine recomputes against the newest schedule (up to a bounded retry) or defers to the next run; the write is guarded so a swap in the read-write gap applies zero rows and the next tick reconciles.
ClickHouse lag only delays a promotion by one tick — acceptable.
Failure isolation. A tick error (bad CH conn, etc.) is logged as periodic task 'fee_rates' failed: … and the loop continues to the next tick; it never wedges the rate path. Since the extraction it also runs in its own process, so it cannot affect recon checks.
A dead task kills the service. The host supervises its tasks: if the fee-rates task panics or returns before shutdown, watch errors and the process exits nonzero so the orchestrator restarts it. It will not sit healthy and idle with rates silently frozen.

Interactions

Delisting a product. A forced closeout prints large fee-free settlement trades but also injects real traded notional that can promote large holders' tiers for up to 30 days. Check this before any delist — see Delisting a Product.
Client-facing docs. The published ladder is described in the public fee structure guide; keep it in sync when the ladder changes.
Liquidity program. Maker rebates here are separate from market-maker liquidity rewards, which are published and accrued independently — see Liquidity Program.