RFC: Admin Interface Hardening — Perimeter and RBAC

Date: 2026-07-28

Status: Draft

Related: The Cloudflare Access for Admin GUI RFC was superseded by this one and has been retired; its perimeter design is carried forward in §Design part A. The Clerk authN migration has shipped (its RFC is retired) — Clerk is now the actor identity and step-up factor for staff, and this RFC builds on it rather than replacing it. Notifications §Admin GUI — already assumes an Access perimeter exists and argues that "someone behind Cloudflare Access" is not an acceptable audit answer.

Background

The admin surface is large and dangerous, and the authorization model behind it is a single boolean.

The surface. The admin GUI (gui/apps/admin, plus the Expo app gui/apps/admin-mobile) is a Cloudflare Pages SPA talking to roughly seventy /admin/* endpoints spread across four services:

api-gateway (rs/api-gateway/src/admin_routes.rs and siblings): user creation, is_admin promotion, deposits, withdrawals, lending credit/debit, price bands, settlement prices, fee schedules, account permission grants, monthly loss limits, MMLP, settlement triggering, PII reads.
order-gateway / btnl-order-gateway: cancel any order, cancel all orders, place liquidation orders, set instrument state, set trading schedules, admin firehose WS.
onboarding-gateway: KYC documents, beneficial owners, Plaid identity verification, onboarding reset, risk assessments — the heaviest PII in the system.
settlement-engine: /trigger/{runner}, gated only by the shared service token, with no user identity at all (rs/settlement-engine/src/server.rs:217-222).

The authorization model. Every one of those routes is gated by exactly one check, authorize_admin (rs/sdk-internal/web-auth/src/lib.rs:196-220):

let auth = authenticate_user(&*state, &mut req, &cookies).await?;
if !auth.user.is_admin {
    return Err(AuthError::InsufficientPermissions);
}

users.is_admin is a boolean (db/postgres/1.sql:8), and the schema comment above it already acknowledges the debt:

-- NB alee: shorthand admin permissions boolean; more granular
-- ACL system using roles=set(permissions...) coming later
is_admin BOOLEAN NOT NULL DEFAULT FALSE,

Concretely, any staff member with the bit set can mint users, flip anyone else's is_admin (rs/api-gateway/src/admin_routes.rs:335-418), move funds, grant themselves arbitrary account_permissions over customer accounts (rs/api-gateway/src/admin_trading_account_routes.rs), cancel any order, and read all KYC PII. The code says so itself: "Admin holds god-mode" (rs/api-gateway/src/utils.rs:263-268).

Known weaknesses beyond the boolean:

authorize_admin has a service-token fast path (rs/sdk-internal/web-auth/src/lib.rs:202-209): any caller presenting the single shared SERVICE_AUTH_TOKEN bearer gets the entire admin surface with no identity, no Auth extension, and no reverification.
There is no admin-action audit log. initiated_by_user_id is stamped on transaction legs, but the admin deposit/withdraw handlers take no Extension<Auth> at all, and there is no generic admin_actions table.
The admin GUI bundle is world-downloadable; the April RFC to put Cloudflare Access in front of it was never implemented. There is no Access, service token, or Cf-Access-* verification anywhere in rs/ or gui/.
The prod origin accepts 80/443 (and SSH) from 0.0.0.0/0 (terraform/prod/ec2.tf:35-59), and admin API traffic shares the customer hostname and nginx (nginx.conf:16-37) — admin and customer traffic are indistinguishable at the edge.
Edition gating in the admin GUI (gui/apps/admin/src/edition.ts) hides sections cosmetically; the backend routes stay live.

What is already good. The Clerk migration landed: human staff login is Clerk-only with MFA, and PII routes enforce Clerk fva (factor verification age) reverification — MultiFactor at 1 minute in api-gateway (rs/api-gateway/src/admin_routes.rs:2733-2787), SecondFactor at 10 minutes in onboarding-gateway (rs/onboarding-gateway/src/auth.rs:76-115), failing closed when JWKS is unreachable. This is the template to generalize, not something to replace.

Goals

No unauthenticated party can load the admin GUI bundle or reach the admin API path at all (perimeter).
Admin authorization is per-action, not all-or-nothing: a support agent reviewing KYC cannot move funds; a risk operator cancelling orders cannot promote users.
Every admin mutation is attributable to a named human and recorded in an audit log.
The shared service token no longer grants the full admin surface.
The design composes with Clerk (actor identity, MFA, step-up) rather than introducing a competing identity system.

Non-Goals

Fine-grained customer authorization — account_permissions already covers that and is deliberately a different model.
Runtime-editable permission matrices or a role-editor UI. The matrix is security policy; it should change via code review.
Hardening of admin-cli beyond noting it (it writes is_admin directly against the DB and is gated by infra access, not this stack).
Internal control panels outside the trading stack (NATE, Grafana, BARD) — NATE already sits behind Access and has its own hardening runbook.

Decision: where does RBAC live?

The prompt for this RFC was: use Cloudflare Zero Trust, or put admin RBAC in a DB table — preferring Cloudflare if it has a good RBAC story. We evaluated Cloudflare as the RBAC source of record and concluded it is not one, for four reasons:

Access policies gate URLs, not actions. An Access application is a hostname + path prefix with an allow policy. Our admin surface is ~70 endpoints multiplexed behind /api/admin/* across four services, plus a WebSocket firehose. Expressing "may trigger settlement but not withdraw" in Access means one Access app per endpoint family, maintained by hand in the Cloudflare dashboard/terraform, drifting silently from the actual router definitions. Path-scoped apps are the right tool for one coarse perimeter, not for a permission matrix.
Role data in the Access JWT is unreliable. IdP groups are only included in the token if explicitly configured as a custom claim, and Cloudflare trims the serialized custom claim on a best-effort basis at roughly 1 KB, dropping values from the end of the list. A staff member in many Google groups can silently lose the claim that carries their role. Cloudflare's documented workaround is calling the get-identity endpoint per request — which makes every authorization decision depend on a Cloudflare API call.
Authorization must survive paths Cloudflare never sees. Service-token calls, internal callbacks (POST /admin/onboarding/complete), the settlement-engine proxy, and admin-cli all bypass the edge. Enforcement therefore has to live in the Rust services regardless; Cloudflare-resident roles would just be a second, divergent source of truth.
Audit needs an in-app actor. As the notifications RFC already put it, "someone behind Cloudflare Access" is not an acceptable answer to "who moved these funds." Attribution has to be recorded where the action executes.

So the design is both layers, with a clear division of labor:

Layer	Technology	Answers
Perimeter	Cloudflare Access (Zero Trust)	"Is this request from an authenticated staff member's browser at all?"
Actor identity + step-up	Clerk (already landed)	"Which human is this, and did they recently re-prove MFA?"
Authorization	Postgres `staff_role_grants` + code-defined matrix	"Is this human allowed to perform this action?"
Attribution	Postgres `staff_audit_log`	"Who did what, when?"

Cloudflare gets the job it is genuinely good at — nobody outside the Workspace org can even fetch the JS bundle or handshake with the admin API — and the per-action decisions stay in code and Postgres where they can be tested, reviewed, and audited.

Design part A — Cloudflare Access perimeter

This absorbs and updates the April RFC. Two changes of circumstance since then: Clerk landed (so the "avoid double login by minting the AX session from the Access JWT" idea is retired — see below), and NATE established the org's Access + cloudflared precedent (configs/bard/compose.yml:462-463, docs/runbooks/nate-cloudflare-access.md).

A1. Access over the Pages projects

Create a self-hosted Access application per admin GUI deployment (ax-admin-gui, ax-admin-gui-demo, ai-exchange-admin-gui, ai-exchange-admin-gui-demo), IdP = Google Workspace, policy = allow @architect.co (tighten to a staff group later). Cloudflare Pages has native Access integration; preview deployments get covered too. This is pure configuration — no code change — and immediately stops bundle enumeration.

A2. Access over the admin API path

The admin API currently shares the customer hostname. Proxy the API zone through Cloudflare (orange-cloud) and add a second self-hosted Access application scoped to <api-host>/api/admin with the same policy. Customer traffic on other paths is unaffected — Access only intercepts the scoped prefix.

This is only meaningful if the origin stops being directly reachable: restrict the prod security group's 80/443 ingress from 0.0.0.0/0 (terraform/prod/ec2.tf:44-59) to Cloudflare's published IP ranges, or front the origin with a cloudflared tunnel as NATE does. This has to happen in the same phase as A2, or Access is decorative.

A3. Origin verification of the Access JWT

Defense in depth: the services must not assume the edge did its job. Add a middleware in sdk-internal/web-auth that, when CLOUDFLARE_ACCESS_AUD is configured, requires a valid Cf-Access-Jwt-Assertion on /api/admin/*:

signature against https://<team>.cloudflareaccess.com/cdn-cgi/access/certs (JWKS cached, same pattern as the Clerk verifier in rs/sdk-internal/web-auth/src/reverification.rs)
aud pinned to the per-application AUD tag — without this, a JWT minted for any other Access app in the tenant validates
exp in the future

Never trust the plaintext Cf-Access-Authenticated-User-Email header alone; that is the exact pattern NATE's own nginx config flags as its outstanding hardening step. Requests bearing a valid SERVICE_AUTH_TOKEN (internal callers that never traverse the edge) skip this check — their scoping is tightened separately in B4. Environments without CLOUDFLARE_ACCESS_AUD set (local dev) skip the check; deployed environments should fail startup if Access is expected but the AUD is unset.

The April RFC proposed exchanging the Access JWT for an AX session to avoid two logins. That is retired. Clerk is now the actor identity, MFA factor store, and fva step-up source — minting sessions from the Access email would discard all of that. In practice the friction is minimal: Access with Google Workspace is a silent redirect for anyone already signed into their work Google account; the one interactive login staff see remains the Clerk screen. The two layers are doing different jobs, and both stay.

A5. Admin mobile app

gui/apps/admin-mobile cannot ride the browser Access flow. Options: require WARP enrollment on staff devices (Access honors WARP as an identity source), issue an Access service token to the app (weak — a shared secret on mobile devices), or exclude the mobile app's traffic from the Access path and lean on Clerk + RBAC only. Recommendation: WARP if mobile usage is load-bearing, otherwise scope the mobile app down to read-only permissions (part B makes that expressible) and revisit. Flagged as an open question.

Design part B — staff RBAC in Postgres

B1. Permissions and roles are code; grants are data

The permission matrix is security policy and changes rarely; it belongs in code where it is reviewed, typed, and exhaustively matched. Assignments change with staffing; they belong in the DB. Concretely, in sdk-internal:

pub enum StaffPermission {
    Read,              // dashboards, positions, orders, stats, firehose
    ReadPii,           // identity cards, KYC docs, risk assessments
    ManageOnboarding,  // KYC review, documents, beneficial owners, resets
    ManageUsers,       // create users, user status, account permissions
    GrantRoles,        // assign/revoke staff roles
    MoveFunds,         // deposit, withdraw, lending credit/debit, treasury
    ManageMarkets,     // price bands, settlement/reference prices, fees,
                       // funding schedules, instrument closing/staff-only
    ManageRisk,        // loss limits, risk profiles, invariant snooze, LTV
    TradeOps,          // cancel order(s), liquidation orders, instrument
                       // state, trading schedules
    TriggerSettlement, // settlement runners, special settlements
}

pub enum StaffRole {
    ReadOnly,   // {Read}
    Support,    // {Read, ReadPii, ManageOnboarding}
    RiskOps,    // {Read, ManageRisk, TradeOps}
    MarketOps,  // {Read, ManageMarkets, TriggerSettlement}
    Treasury,   // {Read, MoveFunds}
    Superuser,  // everything, including GrantRoles
}

The exact partition of the ~70 endpoints into these ten permissions is an implementation detail to be settled in the PR; the shape — roughly ten permissions, six roles, fn permissions(role: StaffRole) -> &'static [StaffPermission] as a pure function — is the decision. This mirrors the house pattern already used for customer authority (Action::allows() in rs/sdk-internal/db/src/entities.rs).

B2. Schema

One new table in db/postgres/1.sql, plus the audit log (B5):

CREATE TABLE staff_role_grants (
    user_id CHAR(16) NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    role TEXT NOT NULL,
    granted_by_user_id CHAR(16) NOT NULL REFERENCES users(id),
    granted_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, role)
);

A user is staff iff they have at least one grant. users.is_admin is retained through the migration as "has the Superuser role" and then deprecated (B6). No roles/permissions lookup tables — the matrix lives in code by design (see Alternatives).

B3. Enforcement

authorize_admin changes from a boolean check to loading the caller's grants into Auth (cached alongside the existing per-request user load). Route groups then declare their requirement:

.layer(middleware::from_fn_with_state(
    state.clone(),
    require_staff_permission::<_, { StaffPermission::MoveFunds }>,
))

applied per nested router in admin_routes.rs, admin_trading_account_routes.rs, admin_settlement_routes.rs, admin_mmlp_routes.rs, order-gateway, btnl-order-gateway, and onboarding-gateway. require_owner_or_admin! (rs/api-gateway/src/utils.rs:117-138) becomes permission-aware: the admin short-circuit on customer routes requires an explicit permission (likely Read for reads, the relevant mutation permission otherwise) instead of the blanket bit. Staff-only instrument visibility keys off "is staff" (any grant) rather than is_admin.

Grant/revoke endpoints (POST/DELETE /admin/staff-roles) require GrantRoles, and a caller can never grant or revoke roles on themselves. PUT /admin/user/status loses the ability to set is_admin — role grants become the only promotion path, which closes today's any-admin-promotes-anyone hole.

B4. Scope the service token

The authorize_admin fast path is the largest single hole: one shared secret grants everything with no identity. Change it to:

carry a synthetic Auth identifying the calling service (so downstream handlers and the audit log always have an actor), and
be accepted only on an explicit allowlist of internal routes (the onboarding-complete callback, the settlement-engine proxy hop) rather than the entire admin surface.

Per-service tokens (or mTLS) are the natural next step but are deliberately out of scope here; the allowlist alone removes the "any service secret = full god-mode" property.

B5. Audit log

CREATE TABLE staff_audit_log (
    id BIGSERIAL PRIMARY KEY,
    occurred_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
    user_id CHAR(16) NOT NULL,          -- or synthetic service actor
    permission TEXT NOT NULL,
    method TEXT NOT NULL,
    path TEXT NOT NULL,
    request_summary JSONB,
    status_code SMALLINT
);

Written from the permission middleware for every admin mutation (writes only; reads would be noise, with the possible exception of ReadPii). This also forces the fix for the handlers that currently take no Extension<Auth> at all (admin deposit/withdraw, rs/api-gateway/src/admin_routes.rs:1806,1874).

B6. Extend step-up, retire the boolean

Generalize the existing Clerk fva reverification from PII-reveal to the dangerous permission classes: MoveFunds, GrantRoles, and ManageUsers mutations require MultiFactor within a short window (the 1-minute precedent from /admin/verify-identity may be relaxed to ~10 minutes for usability; to be settled in review). Once all callers key off grants, users.is_admin is dropped, resolving the NB alee schema comment.

Alternatives considered

Cloudflare Access as the RBAC source of record. Rejected for the four reasons in the Decision section: URL-not-action policy granularity, the ~1 KB best-effort trim on group claims, no coverage of non-edge paths, and audit attribution. Access is adopted for the perimeter only.
Clerk organizations/roles as the source of record. Clerk supports org roles surfaced in session claims, and we already run Clerk. Rejected: role changes would lag live sessions, API-key/service/CLI paths never touch Clerk, and it puts authorization state for a derivatives exchange in a third party's data model. Clerk remains identity + step-up.
Fully normalized roles/permissions/role_permissions tables with a runtime-editable matrix. Rejected for now: the matrix is small, changes rarely, and a code-defined matrix gets exhaustive match coverage, type-checked handler requirements, and PR review. If matrix churn proves real, promoting it to data later is a small migration; the grants table and enforcement points don't change.
Per-endpoint Access applications (pure Cloudflare, no code). Rejected: unreviewable configuration sprawl that drifts from the routers, and useless for the WS firehose, service paths, and audit.

Rollout

Each phase is independently shippable and independently valuable.

Access on Pages (A1): demo first, then prod. Config only; the riskiest failure mode is locking staff out, which demo exercises.
Origin JWT verification + Access on the admin API path + origin lockdown (A2, A3): ship the middleware dark (log-only) first, confirm every legitimate admin request carries a valid assertion, then enforce. Tighten the prod security group in the same change window.
RBAC schema + shadow enforcement (B1–B3): create tables, backfill every is_admin = true user to Superuser (grants preserved behavior exactly), deploy require_staff_permission in log-would-deny mode, review the logs, then enforce. Assign real (narrower) roles after enforcement is proven.
Service-token scoping, audit log, step-up tiers (B4–B6a).
Drop users.is_admin (B6b) once nothing reads it.

Per the house rule for auth/session features: before each enforcement flip, test client disconnect, server-initiated disconnect, restart, and reconnect/recovery paths — in particular that a mid-session role revocation takes effect on the next request (grants are read per-request, not baked into the session token, precisely so revocation is immediate).

Security considerations

Always verify the Access JWT server-side; never trust plaintext headers. If the origin is reachable by any path that bypasses Cloudflare, Cf-Access-Authenticated-User-Email is trivially spoofable.
Pin the AUD per application and fail startup if unset in environments that expect Access.
Perimeter and authz must fail independently. The design assumes either layer can be misconfigured; neither alone is sufficient, which is why origin JWT verification and in-service RBAC both exist.
Roles are read per-request from Postgres, not embedded in session tokens, so revocation is immediate and there is no stale-claim window.
No self-service on GrantRoles. A superuser cannot modify their own grants; whether promotion to Superuser should require a second superuser (two-person rule) is an open question.
The service token stops being an identity-free god key (B4). Until phase 4 lands, treat SERVICE_AUTH_TOKEN with production-secret discipline: it is currently equivalent to a superuser session.
Client-side code remains visible to authorized staff. Access stops unauthenticated bundle enumeration only; anything truly sensitive belongs server-side.

Open questions

Do we cover demo with Access from day one? (Recommendation: yes — demo is where lockout mistakes are cheap, and demo admin can still move demo funds.)
What do we do with admin-mobile — WARP enrollment, read-only role scoping, or retirement? (A5.)
Two-person rule for Superuser grants: worth the ceremony now, or after the base RBAC lands?
Should the aiex edition prune admin routes server-side using the same permission layer (edition → permission subset), replacing the cosmetic GUI gating?
Per-service auth tokens or mTLS to replace the single shared SERVICE_AUTH_TOKEN — follow-up RFC?
Does staff_audit_log also ship to ClickHouse for retention/analytics, or is Postgres sufficient?

Appendix: source index

Concern	Where
The one admin check + service-token fast path	`rs/sdk-internal/web-auth/src/lib.rs:196-220`
`is_admin` column + ACL-owed comment	`db/postgres/1.sql:2-16`
Admin promotes anyone via user status	`rs/api-gateway/src/admin_routes.rs:335-418`
Admin deposit/withdraw, no `Extension<Auth>`	`rs/api-gateway/src/admin_routes.rs:1798-1874`
Account-permission grants by admins	`rs/api-gateway/src/admin_trading_account_routes.rs:441-450`
"Admin holds god-mode"	`rs/api-gateway/src/utils.rs:263-268`
`require_owner_or_admin!` short-circuit	`rs/api-gateway/src/utils.rs:117-138`
Order-gateway admin routes	`rs/order-gateway/src/rest_service.rs:282-286`
Btnl-order-gateway admin routes	`rs/btnl-order-gateway/src/rest_service.rs:78-102`
Onboarding admin routes (PII)	`rs/onboarding-gateway/src/admin_onboarding_routes.rs:1626-1648`
Settlement-engine service-token-only trigger	`rs/settlement-engine/src/server.rs:217-222`
Clerk `fva` reverification (template)	`rs/sdk-internal/web-auth/src/reverification.rs:31-75`
PII step-up endpoints	`rs/api-gateway/src/admin_routes.rs:2733-2787`, `rs/onboarding-gateway/src/auth.rs:76-115`
Customer `Action` permission pattern	`rs/sdk-internal/db/src/entities.rs:3596-3662`
Admin GUI Pages deploys	`.github/workflows/build-and-release-gui*.yml`
Shared customer/admin nginx	`nginx.conf:16-37`
Prod origin open to the world	`terraform/prod/ec2.tf:35-59`
Org Access precedent (NATE)	`configs/bard/nginx-nate.conf`, `configs/bard/compose.yml:462-463`, `docs/runbooks/nate-cloudflare-access.md`
Cosmetic edition gating in admin GUI	`gui/apps/admin/src/edition.ts`