RFC: Factor Cron Schedulers into `sdk-internal`

Date: 2026-04-15

Background

The workspace has three near-duplicate cron/schedule implementations, each with slightly different semantics and spec formats:

Crate	File	Shape
`settlement-engine`	`src/daemon.rs`	`cron::Schedule` + tz, spec format `"<cron> (tz)"`, generic `run_scheduled_loop<F, Fut>` driver with skew detection
`recon-engine`	`src/schedule.rs`	`Schedule` enum (`Cron` / `CronTz` / `Disabled`), spec format `"<cron> <tz>"` (bare trailing token), `next_occurrence(now)` math only, no driver
`recon-engine/mm_reports`	`src/mm_reports.rs`	Hand-rolled `loop { sleep_until_hour() }`, env var `MMLP_CRON_HOUR_UTC=<int>`, UTC only, no tz support

Each call site has reinvented the same thing. When the next cron-shaped feature lands, we will be tempted to reinvent it a fourth time.

Goal

A single ax_sdk_internal::scheduler module that owns:

A Schedule type that parses cron specs with optional timezone and a "disabled" sentinel
A generic run_scheduled_loop<F, Fut> driver that sleeps until the next fire time, detects clock skew, invokes the job, and cooperates with shutdown

Proposed API sketch

// ax-sdk-internal/src/scheduler.rs

pub enum Schedule {
    /// Cron expression in UTC.
    Cron(Box<cron::Schedule>),
    /// Cron expression evaluated in a specific timezone (DST-aware).
    CronTz(Box<cron::Schedule>, chrono_tz::Tz),
    /// Skip scheduling (for feature-flagged jobs).
    Disabled,
}

impl Schedule {
    pub fn next_occurrence(&self, now: DateTime<Utc>) -> Option<DateTime<Utc>>;
}

impl FromStr for Schedule { /* see format below */ }

pub async fn run_scheduled_loop<F, Fut>(
    name: &str,
    schedule: Schedule,
    shutdown: CancellationToken,
    job: F,
) where
    F: Fn() -> Fut,
    Fut: Future<Output = anyhow::Result<()>>;

Spec format

Settle on settlement-engine's format with tz optional:

<sec> <min> <hour> <dom> <mon> <dow>                 # UTC, 6 fields
<sec> <min> <hour> <dom> <mon> <dow> (<timezone>)    # explicit tz
disabled                                             # no-op sentinel

Examples:

0 0 6 * * * — daily at 06:00 UTC
0 0 16 * * Mon-Fri (Europe/London) — FX settlement pattern
disabled — skip

This format is unambiguous (parens can't appear inside a valid cron expression), readable (the tz is clearly set apart from the cron), and a superset of what today's call sites need.

Skew detection & shutdown

Port settlement-engine's existing behavior:

If now - expected_fire_time > 1s, log warn!("time skew detected")
Take a tokio_util::sync::CancellationToken and wake early on cancel instead of only when the timer fires

Job errors are logged (error!) but do not terminate the loop — same as settlement today. If a caller wants hard-fail semantics they can shut down the token from inside their job.

Migration plan

Add ax_sdk_internal::scheduler with the API above plus tests (settlement and recon both have solid test suites to merge).
Migrate settlement-engine/src/daemon.rs to call run_scheduled_loop instead of its local copy. Keep its notifier/JoinSet orchestration in daemon.rs — that's settlement-specific glue.
Migrate recon-engine/src/schedule.rs consumers (check loop in lib.rs) to re-export from sdk_internal. Delete the local module.
Migrate recon-engine/src/mm_reports.rs from the hand-rolled hourly loop to run_scheduled_loop. The env var MMLP_CRON_SCHEDULE already accepts the target spec format (see "Forward-compatibility" below), so the migration is purely internal.

Each step is independently shippable and reversible.

Backwards-compatibility with the current mm_reports config

PR #1637 ships mm_reports with a simpler config shape: MMLP_CRON_HOUR_UTC=<int> (default 6), representing "daily at HH:00 UTC". This RFC's format is a strict superset — any integer hour H maps unambiguously to the cron spec "0 0 H * * *".

The mm_reports migration step must preserve deployed environments without a config change. Two options:

Accept both env vars at the callsite. MMLP_CRON_SCHEDULE takes precedence; if absent, fall back to translating MMLP_CRON_HOUR_UTC into "0 0 H * * *" and feed that into the shared Schedule. Log a deprecation warning when the legacy var is used. Drop the legacy var in a follow-up once deployments have migrated.
Migrate env vars as part of the rollout. Update each environment's config to MMLP_CRON_SCHEDULE="0 0 6 * * *" in the same PR that flips the code. Simpler code, but couples the deploy to the code change.

Option 1 is safer for a shared infra component. Spec format in the RFC is unchanged either way — this is purely about the mm_reports migration shim.

Open questions

Do we keep both Schedule::Cron and Schedule::CronTz as separate enum variants, or collapse into Schedule { expr, tz: Option<Tz> }? The enum split is slightly nicer for pattern-matching; the struct form is simpler to extend (e.g. adding retry policy). Probably the struct form.
Retry / backoff on job failure. Settlement and recon today both log and loop. Do we want an optional RetryPolicy knob now, or defer until a caller wants it?
Jitter. Not needed for current call sites (all daily, well-spaced), but worth thinking about if we ever schedule high-frequency jobs.
Per-job timeout. A stuck job today blocks its own loop forever. Wrapping job() in tokio::time::timeout behind an optional knob is cheap insurance.

Non-goals

Replacing the existing cron crate. Both cron::Schedule::from_str and schedule.upcoming(tz) stay as the primitive.
Introducing a persistent scheduler (durable jobs, distributed locking). That's a separate problem; this RFC is about in-process cron loops only.

Size estimate

~half a day:

Module + tests: 2-3 hours (most tests already exist, just need to be merged).
Three migrations: ~1 hour each, mostly mechanical.

RFC: Factor Cron Schedulers into sdk-internal