Job Scheduler

The Job Scheduler decides when batch work runs and keeps an authoritative record of every job’s lifecycle. It backs the Batch Jobs API and hands accepted work to the Spark Runner.

Idempotent by date

Each daily job is keyed by its target date. Re-submitting the same date is a no-op if a successful run already exists, which makes retries and backfills safe to trigger repeatedly.

How a job runs

Trigger

A cron tick (or an operator via the API) enqueues a job for a target date.

Dispatch

The scheduler respects a concurrency limit, dispatching queued jobs to the Spark Runner as slots free up.

Track

Status transitions — running, succeeded, failed, cancelled — are recorded and exposed through the API for monitoring.

Notes

Failed jobs are re-queued with bounded retries before being surfaced as a hard failure.
Cancellation is cooperative: the scheduler marks intent and the Spark Runner tears down the application.

Job Scheduler

The Job Scheduler decides when batch work runs and keeps an authoritative record of every job’s lifecycle. It backs the Batch Jobs API and hands accepted work to the Spark Runner.

Idempotent by date

Each daily job is keyed by its target date. Re-submitting the same date is a no-op if a successful run already exists, which makes retries and backfills safe to trigger repeatedly.

Failed jobs are re-queued with bounded retries before being surfaced as a hard failure.
Cancellation is cooperative: the scheduler marks intent and the Spark Runner tears down the application.