Design a Background Job Queue System
Fast recall
Quick revision
Core answer
ZADD jobs:scheduled ) with a poller that moves due jobs to the ready queue, or Kafka with a deferred-topic + scheduler pattern. For retries, classify failures (transient → retry with exponential backoff + jitter, fatal → dead-letter queue), cap retry count, and persist attempt history. Add priority queues (per-tier broker pools), idempotency keys (so safe re-execution), visibility timeouts (job leased to one worker only), and observability (queue depth, processing time, failure rate). Real-world: Sidekiq (Redis), Celery (RabbitMQ), AWS SQS + Lambda, Temporal for complex workflows.Broker choice, worker pool model, delayed-job mechanics, retry+DLQ design, idempotency, and visibility/lease semantics.
Treating job queue as a simple FIFO (real systems need delays, priorities, retries, dead-lettering), or assuming exactly-once is achievable broker-side (it isn't — idempotency at consumer is mandatory).
Walk through Redis sorted-set delayed jobs vs Kafka deferred topic; discuss Temporal/Cadence as durable workflow alternatives; explain leased-vs-acked semantics and why that's different from at-most-once.
Interview-ready guide
Interview answer
Architecture
Broker holds jobs; stateless workers lease, execute, ack. Producer hits enqueue API — immediate jobs go to ready queue, future jobs to scheduled sorted set. Scheduler poller every second moves due jobs from scheduled to ready.
Workers BLPOP from ready, lease the job (visibility timeout hides from other workers), execute, ack. Failure → retry with backoff or DLQ. Per-tenant or per-priority queues for fairness.
Storage choice
- Redis (Sidekiq, RQ, Bull): lists for ready, sorted sets for scheduled. Sub-ms enqueue, atomic Lua, AOF for durability. Bounded by RAM.
- Kafka: per-priority topics, partitions for parallelism. Massive throughput, replay support, durable. No native delay; harder per-job lifecycle.
- SQS: native visibility timeout, delay, DLQ. Zero ops. Cost at scale.
- Postgres SELECT FOR UPDATE SKIP LOCKED: transactional with rest of data. Hard ceiling at 5-10K jobs/sec.
Delayed jobs
Redis ZADD jobs:scheduled with score = runAt epoch. Scheduler process every second runs Lua: ZRANGEBYSCORE up to now, ZREM, LPUSH to ready. O(log N + K) extraction; handles millions of pending.
Kafka deferred topic: enqueue to 'deferred' with deliver_at header; scheduler service holds in memory until due, re-publishes to ready.
For sub-second precision: hashed timing wheel (Netty pattern).
Recurring jobs: cron scheduler emits fresh job at each schedule (Quartz, Sidekiq-cron, Celery-beat).
Workers and leases
Lease = visibility timeout. When worker pulls a job, broker hides it from other workers for leaseDuration. If worker doesn't ack in time (crash), broker redelivers. Lease must exceed expected job runtime.
Long-running jobs: worker heartbeats to extend lease (broker.extend +5min).
Leased ≠ acked. Worker crash between lease and ack → redelivery. This is the source of at-least-once delivery — consumers must be idempotent.
Worker sizing: threads per process bounded by runtime; processes per machine 1-2 per core for CPU-heavy, more for IO-bound.
Retries and DLQ
Classify failures:
- Transient (timeout, 5xx, network) → retry
- Fatal (validation, bad payload) → DLQ immediately
Exponential backoff + jitter, capped attempts (5 typical):
- nextRetryAt = now + min(maxDelay, base × 2^attempt) + random(0, base)
- Schedule via the same scheduled sorted set as delayed jobs
DLQ semantics: failed jobs to jobs:dead with full error chain. Operator inspects, fixes, replays. Alert on DLQ depth.
Idempotency: job carries idempotency_key; worker SETNX in Redis to claim; skip if already done. Without it, retries cause duplicate emails, double charges.
Poison pill: job that crashes worker every time. After N consecutive crashes on same job, force-DLQ.
Priorities
Separate queues: jobs:ready:critical / high / default / low. Workers consume in priority order via BRPOP across multiple lists.
Strict priority risks starving low queues. Weighted fair queueing (10:5:3:1 ratio) gives every priority some throughput.
Multi-tenant: queue per tenant; workers fairly distribute. Cap workers per tenant for hard fairness.
LLD
- JobQueue facade with enqueue / enqueueAt
- Broker interface (RedisBroker, KafkaBroker impls) — push, lease, extend, ack, fail
- Scheduler polls scheduled set
- Worker loops: lease, dispatch via HandlerRegistry, ack/fail
- JobHandler interface per job type
- RetryPolicy: nextDelay, shouldRetry
- IdempotencyStore: tryClaim, markComplete
Failure handling
- Worker crash → lease expires, redelivery, idempotency makes safe
- Broker outage → enqueue fails (return 503 + Retry-After), workers pause; for critical events use durable enqueue via DB outbox
- Backlog grows → autoscale workers on queue depth; alert on > 5-10 min backlog at current rate
- Stuck job → per-job time limit overrides lease extension
- DLQ growth → alert; bug or upstream change
- Hot job type → per-type concurrency cap
- Broker memory blow → AOF + replication, size for peak + headroom
When to upgrade beyond a queue
Multi-step long-running workflows (saga with compensations, complex retry trees) get unwieldy in raw queues. Use Temporal or Cadence — durable workflow runtime with native saga, retries, and history.
Recommendation
Redis-backed (Sidekiq-style) for most teams: list for ready, sorted set for scheduled and retry, Lua scripts for atomic operations, stateless worker fleet with lease + heartbeat, exponential backoff retry with DLQ, idempotency keys at the consumer. Scale with multiple priority queues and per-tenant isolation. Move to Kafka when throughput exceeds ~100K jobs/sec or when you need replay. Use SQS for managed simplicity. Reach for Temporal/Cadence when workflows become long-lived multi-step with compensations — a job queue alone won't carry that weight cleanly.
Design a Background Job Queue System
Direct Answer
Use a broker (Redis or Kafka) holding job records + a fleet of stateless worker processes that lease jobs, execute, and ack. For delayed jobs, use a Redis sorted set (ZADD jobs:scheduled <runAt> <jobId>) with a poller that moves due jobs to the ready queue, or Kafka with a deferred-topic + scheduler pattern. For retries, classify failures (transient → retry with exponential backoff + jitter, fatal → dead-letter queue), cap retry count, and persist attempt history. Add priority queues (per-tier broker pools), idempotency keys (so safe re-execution), visibility timeouts (job leased to one worker only), and observability (queue depth, processing time, failure rate). Real-world: Sidekiq (Redis), Celery (RabbitMQ), AWS SQS + Lambda, Temporal for complex workflows.
Common Mistakes & Pitfalls
Treating job queue as a simple FIFO (real systems need delays, priorities, retries, dead-lettering), or assuming exactly-once is achievable broker-side (it isn't — idempotency at consumer is mandatory).
Check your understanding
Answer these without looking back at the article.
- Walk through the failure scenario where a worker crashes after sending an email but before acking the job — how do you prevent the duplicate email?
- Compare Redis-backed (Sidekiq) vs Kafka-backed job queues for a 100K jobs/sec workload.
- Design rate-limiting per tenant on job processing.
- When would you use Temporal/Cadence instead of a raw job queue? Walk through the saga pattern.
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.