Runbook: Liquidity Program

Last updated: 2026-09-04

This runbook covers operating the market-maker liquidity program (MMLP): publishing a per-symbol program, how closed epochs accrue into the accrual ledger, the accrual run's failure and safety behavior, and how to roll a program back. The pure accrual core is ax-mmlp2 (rs/sdk-internal/mmlp2/); the accrual-engine accrual task is its host. The accrual task itself moves no money — it writes an accrual ledger only, and a separate payout task pays approved rows (see Payout below).

How it works

Unlike the uniform fee ladder, a liquidity program is per symbol. Each symbol has its own mmlp.liquidity_programs row, versioned by effective_at; the active program for a symbol is the row with the greatest effective_at <= now.

Program = mmlp.liquidity_programs rows. (db/postgres/1.sql, mmlp.liquidity_programs.) Rows are immutable — a DB trigger (liquidity_programs_freeze) rejects UPDATE/DELETE/TRUNCATE, and (symbol, effective_at) is unique. You publish a change by inserting a superseding row, so the row sequence is the full program history. params are stored as columns (pool_usd, v_bps, q_min) plus bonus_windows JSONB ([start_minute, end_minute, multiplier] tuples). CHECK constraints mirror the ProgramParams::validate() gate.
Authoritative source is Git, not the DB. The canonical programs live in rs/admin-cli/examples/mmlp.yaml, typed by ax_mmlp2::ProgramParams (rs/sdk-internal/mmlp2/src/params.rs) with a validate() gate (pool_usd > 0, v_bps > 0, q_min >= 0; each bonus window start_minute < end_minute and multiplier > 0). The DB rows are a materialized cache of that file. A unit test pins the example to what parses and validates (rs/admin-cli/src/mmlp.rs).

params block.

Field	Meaning
`pool_usd`	Monthly pool `P` in USD for the symbol
`v_bps`	Score-distance scale `v` in basis points
`q_min`	Minimum resting size to score
`bonus_windows`	Time-of-epoch bonus multipliers `[start_minute, end_minute, multiplier]`; empty ⇒ `B(t) = 1` throughout

The daily pool is P_daily = P / n, where n is the number of days in the epoch's calendar month in US/Central (days_in_month_central in epoch.rs) — daily pools sum to the monthly pool over the month.

Accrual run accrues closed epochs. accrue_task (mmlp_accrue) in accrual-engine (rs/accrual-engine/src/mmlp/accrue.rs) runs on a periodic tick. Each tick (accrue_once):
- scans ClickHouse back MMLP_ACCRUE_LOOKBACK_DAYS for closed epochs;
- skips any epoch already in the ledger (idempotent);
- for each remaining epoch, loads the program in force at epoch_start (point-in-time query_effective_at, so a mid-flight publish never re-prices a scored epoch), reads that epoch's scored snapshots from ClickHouse, drops any row scored inside the daily 15:00–16:00 US/Central maintenance hour (MAINTENANCE_WINDOW in epoch.rs; logged at info), and runs the pure accrue_day core on the full daily pool. Coverage never scales the pool: weights normalize over the minutes that actually scored, so each day's pool is distributed entirely across the minutes where someone provided two-sided liquidity, and a day with no scoring snapshots pays nothing at all (doc §3.3). A thin epoch therefore concentrates a whole day's money on whoever was quoting — the accrual run logs a warn naming the coverage when it does;
- writes the resulting accruals with insert_ignore_batch (ON CONFLICT DO NOTHING).
Accrual math (pure, spec-fixed). accrue_day (rs/sdk-internal/mmlp2/src/accrue.rs) applies, in order: the per-minute bonus multiplier B(t), an anti-concentration cap (no trader takes more than cap_fraction of the daily pool, default 40%), a minimum-payout gate (min($10, 1% of pool)), and floor-to-cent rounding. Every dollar is either accrued to a trader or retained by the exchange.
Accrual ledger. Rows land in accrual_engine.liquidity_accruals (db/postgres/1.sql), keyed (epoch_start_ts, symbol, account_id). Each row records pre_cap_usd (uncapped pro-rata share, for transparency), accrued_usd (final payable), and status — ACCRUED if at/above the min-payout threshold, else FORFEITED. The ledger is append-only: a trigger blocks DELETE/TRUNCATE and every column except status, and permits only ACCRUED → APPROVED → PAID or ACCRUED → FORFEITED.

Note: The accrual run is disabled by default (MMLP_ACCRUE_ENABLED = false). Accruing on epoch_end <= now alone can freeze a partial snapshot generation into the immutable ledger. Keep it off until the snapshot producer signals a completed generation per epoch; flip it on only once that contract lands. With no snapshots for an enrolled program, each tick is a no-op.

Publishing a new program

Edit the canonical file. Change rs/admin-cli/examples/mmlp.yaml. symbol/symbols accepts one symbol, a comma-separated string, or a list; every symbol in an entry shares its params, effective_at and optional category (the published tier — Core, Active, Emerging, Bootstrap, ... — that the rewards page groups by; it is a label, not a scoring input). Omit effective_at only if you intend it to take effect immediately.
Update the pinning test in rs/admin-cli/src/mmlp.rs so the example-parses-and-validates assertions reflect the new programs, and run:
```
cargo test -p ax-admin-cli --manifest-path rs/Cargo.toml
```

Dry-run the materializer to see the plan (parse + validate, no write):

[ec2-user@ax-demo] $ run admin_cli mmlp materialize \
  --file rs/admin-cli/examples/mmlp.yaml --dry-run

Materialize (inserts one liquidity_programs row per symbol, in a single statement so a rejected row publishes nothing):
```
[ec2-user@ax-demo] $ run admin_cli mmlp materialize \
  --file rs/admin-cli/examples/mmlp.yaml
```
Runs against the environment's Postgres (standard PostgresDbConfig env vars). The command prints each new program's UUID, symbol, effective time, and pool.
Verify the published rows, newest effective first:
```
[ec2-user@ax-demo] $ run admin_cli mmlp list
```

Warning: A program row is immutable — it cannot be edited or deleted. An effective_at in the past takes effect for every epoch from that instant once the accrual run happens. To back out, publish another superseding row (see Rolling back below).

Configuration

accrual-engine, MMLP_* env. The snapshotter samples the minute that just closed; the backfill pass drains what it missed by replaying order_log on its own cadence; the accrual run prices closed epochs.

Env	Meaning	Default
`MMLP_EP3_MAX_CONCURRENCY`	Live-pass books fetched at once — the in-flight EP3 cap. Must be `1..=64`	`16`
`MMLP_SNAPSHOT_ENABLED`	Whether the live snapshot pass runs (needs the EP3 admin API)	`false`
`MMLP_SNAPSHOT_STARTUP_DELAY_SECS`	Delay before first tick	`15`
`MMLP_SNAPSHOT_DISPATCH_RATE`	Dispatch pacing, e.g. `30/1s`	`30/1s`
`MMLP_BACKFILL_ENABLED`	Whether the backlog drain runs	`false`
`MMLP_BACKFILL_STARTUP_DELAY_SECS`	Delay before first tick	`45`
`MMLP_BACKFILL_INTERVAL_SECS`	Tick cadence; the pass deadline is a share of it	`60`
`MMLP_BACKFILL_LOOKBACK_SECS`	How far back a pass reaches for epochs to drain	`10800` (3h)
`MMLP_BACKFILL_REPLAY_LOOKBACK_SECS`	How far before a symbol's first owed minute to replay `order_log`	`172800` (48h)
`MMLP_BACKFILL_REPLAY_CONCURRENCY`	Symbols the backfill pass replays at once (not an EP3 cap)	`8`
`MMLP_ACCRUE_ENABLED`	Whether the periodic accrual run runs	`false`
`MMLP_ACCRUE_REFRESH_SECS`	Tick cadence	`3600` (hourly)
`MMLP_ACCRUE_STARTUP_DELAY_SECS`	Delay before first tick	`60`
`MMLP_ACCRUE_LOOKBACK_DAYS`	How far back a tick scans for accruable epochs	`30`
`MMLP_ACCRUE_GRACE_SECS`	How long past close an epoch is left alone, so the backfill can still raise its coverage	`10800` (3h)
`MMLP_ACCRUE_MIN_COVERAGE_PCT`	Coverage floor under which an epoch is left unaccrued; `0` disables the gate	`0`
`MMLP_PAYOUT_ENABLED`	Whether the periodic payout run runs — the only MMLP task that moves money	`false`
`MMLP_PAYOUT_REFRESH_SECS`	Tick cadence	`86400` (daily)
`MMLP_PAYOUT_STARTUP_DELAY_SECS`	Delay before first tick	`120`
`MMLP_PAYOUT_AUTO_APPROVE_UNDER_USD`	Auto-approve accruals under this amount; unset means every row waits on an admin approval	unset

The coverage floor is an operational hold, not a payout rule. Above it an epoch pays the full daily pool; below it the tick fails and writes nothing, for an operator to unenroll the symbol or lower the floor. It never pays a reduced amount — the program terms define no coverage discount, so any setting that changed a payout would be off-terms. Coverage is displayed on the admin page and alerted on (#3889); it is not an input to the money math.

Note: MMLP_EP3_MAX_CONCURRENCY caps only the live pass: backfill rebuilds books from ClickHouse order_log and never hits EP3. Size it to EP3's own admin-API limit minus headroom for other consumers. MMLP_SNAPSHOT_DISPATCH_RATE paces dispatch inside that fan-out.

Note: MMLP_ACCRUE_GRACE_SECS must be at or above MMLP_BACKFILL_LOOKBACK_SECS and strictly under MMLP_ACCRUE_LOOKBACK_DAYS, or the service refuses to start. An epoch accrues once, on the coverage it has then, so the grace has to outlast the backfill's reach into it.

Warning: The economic rules — cap_fraction (40%), min-payout, and payout precision — are spec-fixed in ax_mmlp2::AccrualConfig and are deliberately not config-overridable. A runtime override would change already-written immutable customer accruals. Only the operational knobs above are tunable.

Rolling back

Programs are immutable, so a rollback is a forward publish: re-materialize the previous params for the symbol (or edit the YAML back and materialize), creating a new current row. The prior rows remain as history.

Note this only changes accruals for epochs accrued after the new row's effective_at. Epochs already accrued against an earlier program are frozen in the ledger and are not recomputed — the point-in-time read binds each epoch to the program that scored it.

Payout

The accrual run is scores-only: it writes accrued_usd and a status, and moves no money. Disbursement is the separate payout task (MMLP_PAYOUT_*, also off by default), which credits maker balances for accruals that reached APPROVED and flips them to PAID. With the payout task disabled, status can still be advanced in the ledger while nothing credits the trader — so do not tell a customer an accrual has been paid on the basis of a PAID row alone unless the payout task is running in that environment.

Failure and safety behavior

Idempotent ticks. insert_ignore_batch uses ON CONFLICT DO NOTHING, and each tick re-skips already-accrued epochs, so a crashed or duplicated tick converges to the same ledger. The sweep re-scans the lookback window, so it self-heals across restarts.
No program in force for an epoch → skip. Expected when an epoch was scored before its program's effective_at; logged at debug (the accrual run re-checks each tick) and no accrual is written.
Zero-total scored row → per-epoch error, not a silent skip. A zero-total snapshot is a scorer bug; SnapshotRow::try_from fails loudly rather than divide by zero.
One bad epoch doesn't strand the rest. A failing epoch is logged and skipped; the tick still returns Err if any epoch failed, surfacing the failure without blocking the others.
Mid-tick program swap. Each epoch is accrued against the program in force at its epoch_start (point-in-time read), so a publish landing during a tick never re-prices an in-flight epoch; the next tick reconciles.
ClickHouse lag only delays an accrual by one tick — acceptable.
Partial snapshot hazard. See the Note above — the accrual run stays disabled until a completed-generation signal exists, because accruing a still-writing epoch would freeze a partial result into the immutable ledger.

Interactions

Fee program. Liquidity rewards are independent of the maker/taker fee ladder; both read the same ClickHouse trade/snapshot data but write disjoint state. See Fee Program.
Delisting a product. A per-symbol program has no meaning once the symbol is delisted; there is no auto-teardown, but the accrual run simply finds no new closed epochs. Check open accruals before a delist — see Delisting a Product.