Designing a Distributed Job Scheduler

A job scheduler runs tasks at specified times or intervals: nightly reports, email campaigns, billing runs and retries. This design targets cron-like recurring jobs and one-off delayed jobs at scale, similar to Airflow, Quartz clusters or an internal platform scheduler.

Requirements

Functional

Create, update, pause and delete jobs: one-off at a time T, or recurring by cron expression.
Execute a job as an HTTP call, queue message or container.
Retries with backoff, timeouts, concurrency policy (allow, skip or replace overlapping runs).
Execution history and status queries.

Non-functional

100 million scheduled jobs, 10,000 job triggers per second at peak (top of the hour spikes).
Trigger delay under 1 second at p99.
At-least-once execution; no job silently skipped even if nodes crash.
99.99 percent availability; no single point of failure.

Capacity estimation

If 100 million jobs each run once daily on average, that is about 1,150 executions per second average, with top-of-the-hour bursts reaching 10,000 or more. A job record of 1 KB gives 100 GB for definitions; execution history at 500 bytes per run and 100 million runs per day yields 50 GB per day, so a 30-day retention is 1.5 TB, partitioned by day.

High-level design

 API service --> Job store (sharded SQL / KV)
                       |
        Scheduler workers (own shards via leases)
        scan: next_run_time <= now + window
                       |  enqueue
                       v
                 Dispatch queue
                       |
              Executor workers (pull)
                       |
        run -> report status -> history store
 Coordinator (etcd/ZooKeeper): leader election, shard leases

Data model

Job table: job id, owner, schedule (cron or timestamp), payload or target, next_run_time, status, retry policy, timeout, concurrency policy, version. An index on (shard, next_run_time) powers the scan. Execution table: execution id (job id plus scheduled time), state (queued, running, succeeded, failed), attempt count, worker id, lease expiry, timestamps. The pair job id and scheduled time is a natural idempotency key.

Key deep-dives

Sharding the timeline. Hash job ids into, say, 1,024 logical shards. A coordinator assigns shards to scheduler nodes using leases that expire if a node stops heartbeating; another node picks up orphaned shards. Each scheduler scans its shards every second for jobs due in the next few seconds, using an index range query, and hands them to a delay structure.

Timers. Near-term jobs sit in an in-memory priority queue or hierarchical timing wheel in the scheduler, which gives O(1) insertion and fires on time. Far-future jobs remain only in the database, so memory holds a short window. Redis sorted sets keyed by trigger time are a common lighter alternative.

Exactly one trigger per slot. Two schedulers must not both fire the same run. Create the execution row with a unique constraint on job id plus scheduled time; only the writer that inserts successfully enqueues the run. Then advance next_run_time using a conditional update (compare version) so a retry is safe.

Execution and leases. Executors pull from the queue and take a lease with heartbeat. If a worker dies, the lease expires and the run is re-queued. This yields at-least-once semantics, so job handlers should be idempotent; pass the execution id so downstream systems can dedupe.

Retries and failure handling. Exponential backoff with jitter (1, 2, 4, 8 minutes, capped), max attempts, then a dead-letter state with an alert. Separate transient failures from permanent ones such as a 400 response.

Thundering herds. Many jobs set to midnight or the top of the hour create spikes. Add per-job random jitter where allowed, rate-limit dispatch per tenant, and autoscale executors from queue depth.

Misfires. If the scheduler was down when a job was due, policy decides: run once immediately, run all missed, or skip. Default to run-once for recurring and run for one-off.

Workflows. For dependencies, model a DAG: a run starts when all upstream runs succeed, with state kept in the database and transitions driven by completion events.

Failure modes and trade-offs

Scheduler crash: shard leases lapse in 10 to 30 seconds; another node resumes. Jobs due inside that gap fire late, not never.
Clock skew: schedulers must use the database or coordinator time as the source of truth for due checks, or keep NTP drift under tens of milliseconds.
Long-running jobs: use heartbeats rather than long timeouts so crashes are detected quickly.
Duplicate execution: possible after lease expiry on a slow but alive worker; fence with execution attempt numbers and reject stale completions.
Polling versus event-driven: polling is simple and robust but adds database load; reduce it with batching and a lookahead window.
Precision versus cost: one-second tick granularity is cheap; millisecond precision needs in-memory timers and more complex recovery.

Key takeaways

Persist every job in a sharded store; hold only a short lookahead window in memory timers.
Lease-based shard ownership gives automatic failover without a single scheduler.
A unique key on job id plus scheduled time makes triggering idempotent across competing schedulers.
At-least-once with idempotent handlers and fencing tokens is the practical guarantee.
Plan for bursts at round times with jitter, rate limits and queue-depth autoscaling.