Background Jobs Rules

Almost every queue delivers at-least-once, not exactly-once — design every job handler to be safely re-runnable with the same input, because it eventually will be, whether from a retry, a redeploy mid-processing, or a duplicate enqueue.

Idempotency

• Make every job handler idempotent: running it twice with the same payload must produce the same end state as running it once — this is the single property that makes retries, redeploys, and duplicate deliveries safe instead of dangerous.
• Use a natural dedup key (an external event ID, an order ID) to detect and skip a job that already ran, rather than assuming the queue itself guarantees single delivery — it doesn't, even for queues that claim exactly-once, under real failure conditions.
• Design side effects (charging a card, sending an email) to check current state before acting ("has this order already been charged?") rather than blindly re-applying — dedup-by-ID catches exact duplicates, but state-checking catches logical duplicates too.

Retries and dead letters

• Retry transient failures (timeouts, 5xx from a downstream service) with exponential backoff and jitter, never a fixed short interval — a fixed interval retry storm amplifies exactly the outage it's reacting to.
• Cap retry attempts and route exhausted jobs to a dead-letter queue instead of retrying forever or silently dropping them — a dead letter queue is what turns "this job vanished" into "this job is sitting here for someone to look at."
• Do not retry failures that are permanent given the same input (a validation error, a malformed payload) — retrying a job that can never succeed just wastes capacity; fail it immediately and alert.

Payload design

• Put IDs in the job payload, not full objects or blobs — pass { orderId: "123" } and have the handler fetch current data, not a snapshot of the order at enqueue time; the snapshot can be stale by the time the job actually runs.
• Keep payloads small (well under the queue's size limit) — large payloads slow down every stage of the queue (serialization, network, storage) and most queues have a hard size cap that a growing payload will eventually hit.

Timeouts and monitoring

• Set an explicit timeout on every job, shorter than the queue's own visibility-timeout/redelivery window — a job that hangs without an internal timeout gets redelivered while the original is still running, and now two copies are racing.
• Monitor queue depth and oldest-message age, not just job success/failure rate — a queue that's growing unboundedly is a worker capacity problem invisible in per-job metrics alone.
• Alert on dead-letter queue growth specifically — a slow trickle into the DLQ is easy to miss in aggregate error rates but represents real, unrecovered failures.

Worker lifecycle

• Handle shutdown signals (SIGTERM) by stopping the acceptance of new jobs, finishing in-flight jobs (or safely checkpointing them), and only then exiting — a worker that dies mid-job on every deploy turns routine deploys into a steady source of failed/duplicated jobs.
• Set the queue's visibility timeout longer than the worker's shutdown grace period, so a job in flight during a deploy isn't redelivered to another worker while the first is still finishing it.

General mindset

• Assume at-least-once delivery as the default mental model for any queue, unless you have specifically verified and configured exactly-once semantics (and understood their cost) — building on an assumed guarantee the queue doesn't actually provide is the root cause of most background-job data bugs.