Why once cannot be promised
A worker sends an invoice email and dies before recording success. The scheduler cannot tell whether the email went. It can retry, risking a second email, or give up, risking none. No protocol between the scheduler and the worker removes that gap, because the side effect lives in a third system.
The worse case is a worker that is not dead. A long garbage-collection pause or a network partition stops its heartbeats, its lease expires, a second worker claims the run, and then the first worker wakes up and carries on. Two executions are now running at once. This is the same detection problem covered in crashed or merely slow: a timeout is a guess, so the design must make a wrong guess survivable.
Definitions and runs are different rows
The job scheduler keeps two tables. A job definition holds the schedule, the handler, the payload and the missed-run policy. A run is one firing: job id, scheduled time, state, lease owner, lease expiry and a fencing token. (The single-process version of this design, with a delay queue and a worker pool, is the in-memory job scheduler machine-coding question.)
Materialise runs ahead of time, a few minutes at most, with a unique constraint on job id plus scheduled time. That constraint is what stops two scheduler nodes creating the same firing twice. When a recurring run is claimed, insert its successor in the same transaction, computing the next time from the scheduled time rather than from the moment the job finished. Computing from finish time makes an hourly job drift later every hour.
Claiming with leases
Dispatchers poll for due runs and claim them. On PostgreSQL, FOR UPDATE SKIP LOCKED lets many dispatchers poll one table without blocking on each other's rows.
WITH claimable AS (
SELECT id FROM job_runs
WHERE (state = 'pending' AND due_at <= now())
OR (state = 'leased' AND lease_expires_at < now())
ORDER BY due_at
LIMIT 100
FOR UPDATE SKIP LOCKED
)
UPDATE job_runs r
SET state = 'leased',
lease_owner = $1,
lease_expires_at = now() + interval '30 seconds',
fencing_token = r.fencing_token + 1
FROM claimable
WHERE r.id = claimable.id
RETURNING r.id, r.job_id, r.due_at, r.fencing_token;
The worker renews the lease every 10 seconds. A 30-second lease therefore survives two missed heartbeats before the run is reclaimed. Completion sets the state to done only if the worker still holds the current fencing token, so a worker whose lease was taken over cannot overwrite the newer attempt's result.
Making the second execution harmless
The lease decides who should be running. It cannot stop a paused worker from acting. Two mechanisms close that gap.
The first is idempotency. The job passes job id plus scheduled time as an idempotency key to whatever it changes: a unique constraint on the invoice table, a key on the payment call. A repeat then finds the work already done.
The second is a fencing token. The job sends its token with each write, and the downstream store rejects a token lower than the highest it has seen. This only works where the target can check tokens, so in practice idempotency carries most of the weight. Jobs that can be neither idempotent nor fenced need a human-visible warning in the API: this job may run twice.
One clock decides
Every comparison above uses the database's now(), not the worker's clock. If each dispatcher judged due-ness by its own clock, a node running 20 seconds fast would fire jobs 20 seconds early, and a node judging lease expiry by a fast clock would steal live leases.
Workers still need local time for one thing: knowing when to stop. A worker should treat its lease as expired at the lease duration minus a safety margin, measured on its monotonic clock from when it sent the renewal, and stop committing side effects past that point. The lease duration must exceed the longest pause you expect plus that margin, which is why leases of a few seconds are a false economy.
Partitioning and the top of the hour
Derive the peak. Assume 2 million recurring jobs, and that 20% of them are hourly jobs written as "minute zero". That is 400,000 runs due at the same instant every hour. If the promise is to start within 60 seconds, dispatch must sustain 400,000 / 60, about 6,700 claims a second, for that minute, and almost nothing for the other fifty-nine.
A job scheduler built on one table polled by skip-locked dispatchers handles a modest load. Past that, hash job ids into partitions and give each scheduler node a lease on a set of partitions, so each node polls only its own runs. Two cheaper levers matter as much: add deterministic jitter to jobs that do not need exact minute zero, and state the lateness bound in the API. A 5-second poll interval alone makes runs up to 5 seconds late, and a dead worker adds up to the 30-second lease.
Missed runs are a product decision
After a 40-minute outage, a job scheduled every 5 minutes has eight missed firings. Running all eight floods downstream systems; running none silently skips a billing cycle. Make it a per-job policy: run once to coalesce, run all to catch up, or skip. Record which one happened on the run rows, so the job's owner can see it.
The job scheduler can only promise that a run will be attempted until it succeeds. Running once is the job's responsibility, and the scheduler's job is to hand it the key that makes a repeat harmless.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).