Job Retry and Failure
Fast recall
Quick revision
Core answer
Whether you know Sidekiq retries failed jobs with exponential backoff (default 25x) then moves them to the dead set.
Retrying non-idempotent jobs (double charges/emails) — make jobs idempotent; or retry: 0 for bad-arg errors.
Sidekiq retries failed jobs (default 25x, exponential backoff) then dead set. sidekiq_options retry: N. Make jobs idempotent. Web UI: /sidekiq/retries, /sidekiq/morgue.
Interview-ready guide
Interview answer
Sidekiq treats a raised exception as a failed execution. It records error details and, when retry is enabled, places the job in the retry schedule with increasing delay. This gives transient failures such as a provider outage time to recover without blocking a web request.
Current Sidekiq defaults retry a job a finite number of times over an extended period, then move exhausted work into the Dead set where operators can inspect or retry it before retention expires. Those numbers and retention can change by version or configuration, so operational policy should be read from the deployed setup rather than memorised as a universal guarantee.
For example, a charge request may reach the payment provider, then time out before the worker records success. Retrying the same payload without a provider idempotency key can charge twice. Reusing one business key lets the provider return the original result for the repeated logical operation.
Retries suit unexpected, potentially temporary failures. An invalid address, cancelled order, or rejected business rule will not become valid through blind repetition; the job should record a permanent outcome, discard according to policy, or move the workflow to human review. Broad rescue that only logs an error creates false success.
Reliable operation combines idempotent effects, bounded timeouts, error-service reporting, retry and dead counts, oldest-job age, and a named owner for recovery. Retrying is one control within that system, not proof that work cannot be lost or duplicated.
Job Retry and Failure
Direct Answer
When a Sidekiq job raises an unexpected exception, Sidekiq records the failure and normally schedules retries with increasing delay. Retry count and final handling are configurable; under current defaults, exhausted jobs move to the Dead set for later inspection before eventual removal. Retrying creates at-least-once execution possibilities, so jobs must be idempotent. Expected business failures should be modelled explicitly rather than retried, and persistent failures need error reporting, queue metrics, ownership, and a safe manual recovery path.
Deep Dive & Architecture
A failure can occur before an effect, during it, or after the external system completed it but before the worker recorded completion. The last window is why exception type alone cannot make retry safe. The job needs a durable key or state transition that survives process loss and distinguishes a repeated attempt from a new business request.
Backoff protects a struggling dependency and gives transient conditions time to clear. Immediate tight retries amplify an outage. Randomness and increasing delay reduce synchronised retry spikes, but long delays also mean the product must expose pending or failed state rather than silently promising instant completion.
Permanent and expected outcomes belong outside automatic exception retry. A payment decline is a business result; invalid serialized data may require dead-letter inspection; a missing record may be harmless or a data-loss signal depending on the operation. Classifying each path prevents wasted attempts and hidden abandonment.
Manual retry is another execution and needs authorization, audit, and current-state checks. After deploying a fix, an operator should know which payloads are safe to replay, which external effects already occurred, and how success will be verified. Deleting failed work without reconciliation changes queue cleanliness, not business correctness.
Code Example
class CapturePaymentJob
include Sidekiq::Job
sidekiq_options retry: 10
def perform(payment_id)
payment = Payment.find(payment_id)
return if payment.captured?
result = Gateway.capture(
payment.amount_cents,
idempotency_key: "payment-#{payment.id}"
)
payment.update!(provider_reference: result.id, captured_at: Time.current)
end
end
# A timeout raises; the next attempt reuses the same provider key.
Common Mistakes & Pitfalls
Retrying non-idempotent jobs (double charges/emails) — make jobs idempotent; or retry: 0 for bad-arg errors.
Check your understanding
Answer these without looking back at the article.
- Which failure window can cause an external effect to succeed before a retry?
- Why should a payment decline usually not use automatic exception retries?
- What evidence is needed before manually retrying a dead job?
Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.