← RFCs · afintech.dev

RFC: Unbounded History Pagination

Date: 2026-06-09

Status: Draft. The transactions reform (§Rollout stage 1–2) is being implemented; the rest of this document is direction we may adopt incrementally, not a commitment.

Background

In early June we shipped a hard 7-day cap on all history endpoints (MAX_HISTORICAL_QUERY_WINDOW_NS, rs/sdk/src/protocol/time_range.rs): both time bounds are required, and any range wider than 7 days is rejected with a 400. This was a deliberate hotfix lineage:

The cap stopped the bleeding, but on 2026-06-09 it produced its first customer-facing incident: a Flowdesk customer's withdrawals "disappeared" from the Admin UI because they were older than the UI's largest selectable window. "Show me all withdrawals for this customer" is now impossible to express against the API — by design.

Meanwhile the GUI has independently grown three client-side workarounds:

Every external API consumer (e.g. Flowdesk's integration, ops curl scripts) must rebuild the same loop. When a client-side loop must be reimplemented by every caller of an API, the loop belongs on the server.

Problem statement

The 7-day cap bounds the wrong variable. Query cost in ClickHouse is not driven by the width of the requested time range; it is driven by how many granules the query reads, which is a function of how well the query's predicates align with the table's primary key, partitioning, and skip indexes. Concretely:

The cap therefore simultaneously over- and under-protects, while exporting a window-walking loop to every client and silently hiding data in UIs that don't walk.

Industry survey (condensed)

The consensus mechanism for "unbounded history without unbounded scans" is: cursor = bounded per-page work, server-side slicing for non-indexed predicates, scan budgets as the backstop.

Target semantic model

Four commitments in the current public contract (docs/internal/overview/pagination.mdx) should be walked back:

  1. The cursor is the sole, opaque carrier of position. Today the cursor format ({timestamp_ns}:{id}) is documented and transparent; clients will couple to it, foreclosing evolution (window-walking progress, snapshot anchoring). The public contract becomes: pass it back unchanged. The encoding lives in internal docs only.
  2. The time range is an optional filter, subordinate to the cursor. Today the client owns a mandatory bounded range and re-sends it with every page; the cursor is position within that fixed window. This inverts: progress state (including which window the server is scanning, and the pinned "as-of" upper bound for open-ended queries) lives in the cursor.
  3. Explicit completeness signaling. Responses gain complete: bool and a searched_until_ns watermark. An empty page with a next_cursor is valid — it means "scanned my budget, found nothing yet, resume here." Termination is absence of next_cursor, never an empty page. (The current docs teach the opposite invariant; this is a contract change to state before third parties couple to the wording.)
  4. Counts are lower bounds or absent. total_count on cursor pages is semantically ambiguous (count of which scope?) and costs a COUNT(*) ... FINAL per page. Omit it for wide/unbounded ranges; where provided, it means "at least N."

Design

DB layer (ClickHouse)

Align the index with the dominant query. History queries are overwhelmingly "one entity, ordered by time."

API layer (api-gateway / order-gateway)

Accept unbounded ranges; walk windows server-side. The handler chunks the requested (or open-ended) range into slices — MAX_HISTORICAL_QUERY_WINDOW_NS becomes the slice size instead of a rejection threshold — and queries slice-by-slice in sort order until the page fills or the per-request slice budget is exhausted:

resolve range:  start = requested or 0; end = requested or now (pinned)
position     =  cursor position, else range edge per sort direction
loop (≤ MAX_SLICES_PER_REQUEST, e.g. 13 ≈ 90 days of scanning):
    query slice [max(start, pos − slice), pos) with remaining limit + 1
    accumulate rows; advance pos to slice boundary
    stop when limit + 1 rows collected or range exhausted
respond:
    next_cursor = row position (page full) | slice boundary (budget hit)
                | absent (range exhausted)
    complete    = range exhausted within this request chain
    searched_until_ns = pos

The GUI's stepWindowedQuery is a working prototype of exactly this algorithm, written in the wrong tier; the reform ports it down the stack.

Cursor v2: a versioned, opaque token (base64) encoding {position: row(ts, id) | boundary(ts), as_of_end_ns, …}. Servers accept legacy {ts}:{id} cursors indefinitely (they decode to a row position with no pinned end). Pinning as_of_end_ns at first request fixes the live-drift problem historyWindow.ts currently works around with its 'live' sentinel.

Backstops, not gates: per-request slice budget + per-slice max_execution_time. A whole-ledger admin scan with no entity filter remains the one genuinely unprunable query; option: require an entity, symbol, or type filter for open-ended ranges on admin endpoints. Per-query cost protection is strictly stronger than the range cap (a 7-day window over a hot month is still a heavy query today), which also addresses the admin-vs-trading-traffic isolation concern.

GUI layer

Docs / public contract

Rewrite docs/internal/overview/pagination.mdx (and any public derivative) to state the target model: opaque cursor, optional range, completeness signaling, lower-bound counts, empty-page-with-cursor contract. The public version must describe look-ahead and budgets behaviorally, without naming the datastore.

Rollout

  1. Transactions endpoints first (the incident surface): cursor v2 accepted+emitted, response gains complete/searched_until_ns, validation relaxed from "reject >7d" to "walk", per-page COUNT(*) dropped for wide ranges, bloom_filter(account_id) skip index + materialization migration.
  2. Fills / trades / funding-transactions / orders follow the same shape once transactions soaks. (historical_orders already has its projection; trades is symbol-led and may want a per-account projection or index of its own.)
  3. GUI simplification per above, including reverting the load-bearing parts of PR #2331's filter clamping to optional UX.
  4. Docs rewrite lands with stage 1, before third parties couple to the current cursor format and termination semantics.

What we are explicitly not committing to

Open questions