Reliable Webhook Delivery: Retries, Idempotency, and DLQ
Fast recall
Quick revision
Core answer
webhook_deliveries table, retry up to ~24 hours with backoff like 1m / 5m / 30m / 2h / 6h / 12h, send a unique Idempotency-Key (or X-Event-Id) header, and after final failure mark the delivery dead and surface it on a dashboard for replay.Whether you can design a production webhook publisher that handles network flakiness, receiver outages, and duplicate delivery without losing events or hammering the receiver
Relying on synchronous in-process retries with `Thread.sleep` — this blocks the request thread and loses webhooks if the publisher restarts mid-retry
Discuss jitter to prevent thundering herd, distinguishing 4xx (don't retry) vs 5xx (retry) vs 429 (respect Retry-After), and why receivers need an idempotency key even with at-least-once delivery
Interview-ready guide
Interview answer
Webhooks are at-least-once HTTP, so you build the publisher like a message queue rather than a normal HTTP call.
Persistence:
- Every webhook gets written to a
webhook_deliveriestable in the same DB transaction as the business event that produced it — exactly the Outbox pattern - A separate worker polls the table for due deliveries and POSTs them, so the publisher can restart safely without losing webhooks mid-retry
Retry policy:
- Exponential backoff with explicit schedule — Stripe-style 1m, 5m, 30m, 2h, 6h, 12h, 24h
- Jitter on every scheduled delay so a recovering receiver isn't crushed by every publisher synchronizing on the same retry time — full-jitter, uniform between 0 and the computed delay, is the simplest correct algorithm
- Status-code policy: 2xx is success, 5xx and 408 and 429 are retryable, every other 4xx goes straight to dead-letter because retrying a 401 or 422 produces the same error
- For 429 responses honor the
Retry-Afterheader instead of the default schedule - An age cap of 24 hours from the event time so a stuck delivery doesn't retry forever
Idempotency:
- Send an
Idempotency-Keyheader equal to the delivery row's UUID — every retry of the same delivery sends the same key, so the receiver can dedupe - This is essential because the receiver has no other way to tell a genuine retry from a duplicate event
Dead-letter handling:
- Deliveries that exhaust retries or hit the age cap land in a
deadstate - Expose them on an operator dashboard with a replay button — same UX Stripe and GitHub have
- Auto-disable endpoints that produce 50+ consecutive failures so a permanently broken receiver doesn't saturate the worker pool
- Track per-endpoint failure rate and per-attempt latency so degrading receivers can be detected before they go fully dark
In Spring Boot the worker is a @Scheduled job using SELECT ... FOR UPDATE SKIP LOCKED for safe horizontal scaling, with RestClient configured for sane connect and read timeouts.
Reliable Webhook Delivery: Retries, Idempotency, and DLQ
Direct Answer
Reliable outbound webhooks need three things: retries with exponential backoff and jitter for transient failures, an idempotency key header so the receiver can dedupe redeliveries, and a dead-letter queue for permanently failed deliveries that human operators can inspect and replay. Stripe, GitHub, and Slack all converge on this pattern: persist every webhook in a webhook_deliveries table, retry up to ~24 hours with backoff like 1m / 5m / 30m / 2h / 6h / 12h, send a unique Idempotency-Key (or X-Event-Id) header, and after final failure mark the delivery dead and surface it on a dashboard for replay.
Code Example
@Entity @Table(name = "webhook_deliveries")
public class WebhookDelivery {
@Id @GeneratedValue UUID id;
String endpointUrl;
String eventType;
@Column(columnDefinition = "jsonb") String payload;
@Enumerated(EnumType.STRING) Status status;
int attempts;
Instant nextAttemptAt;
Instant createdAt;
Instant lastAttemptAt;
Integer lastResponseStatus;
String lastError;
enum Status { PENDING, DELIVERING, DELIVERED, DEAD }
}
@Component
public class WebhookDeliveryWorker {
private static final List<Duration> SCHEDULE = List.of(
Duration.ZERO, Duration.ofMinutes(1), Duration.ofMinutes(5),
Duration.ofMinutes(30), Duration.ofHours(2), Duration.ofHours(6),
Duration.ofHours(12));
private static final Duration MAX_AGE = Duration.ofHours(24);
private static final Random JITTER = new Random();
private final WebhookDeliveryRepository repo;
private final RestClient http; // configured with sane connect/read timeouts
@Scheduled(fixedDelay = 1_000) // poll every 1s for due deliveries
public void drain() {
List<WebhookDelivery> due = repo.findTop100ByStatusAndNextAttemptAtBefore(
Status.PENDING, Instant.now());
due.forEach(this::attempt);
}
private void attempt(WebhookDelivery d) {
d.status = Status.DELIVERING;
d.attempts += 1;
d.lastAttemptAt = Instant.now();
repo.save(d);
try {
ResponseEntity<Void> resp = http.post()
.uri(d.endpointUrl)
.header("Content-Type", "application/json")
.header("Idempotency-Key", d.id.toString()) // stable per delivery
.header("X-Event-Id", d.id.toString())
.header("X-Event-Type", d.eventType)
.header("X-Attempt", String.valueOf(d.attempts))
.body(d.payload)
.retrieve()
.toBodilessEntity();
handleResponse(d, resp.getStatusCode().value(), null);
} catch (HttpClientErrorException e) {
handleResponse(d, e.getStatusCode().value(), e.getMessage());
} catch (HttpServerErrorException e) {
handleResponse(d, e.getStatusCode().value(), e.getMessage());
} catch (ResourceAccessException e) {
handleResponse(d, 0, "network: " + e.getMessage());
}
}
private void handleResponse(WebhookDelivery d, int status, String err) {
d.lastResponseStatus = status == 0 ? null : status;
d.lastError = err;
boolean ok = status >= 200 && status < 300;
boolean retryable = status == 0 || status == 408 || status == 429 || status >= 500;
boolean ageExceeded = Duration.between(d.createdAt, Instant.now()).compareTo(MAX_AGE) > 0;
if (ok) {
d.status = Status.DELIVERED;
} else if (!retryable || ageExceeded || d.attempts >= SCHEDULE.size()) {
d.status = Status.DEAD; // permanent 4xx, too old, or out of attempts
} else {
Duration base = SCHEDULE.get(d.attempts);
// Full-jitter: sleep is uniform between 0 and base.
long jitterMs = (long) (JITTER.nextDouble() * base.toMillis());
d.nextAttemptAt = Instant.now().plusMillis(jitterMs);
d.status = Status.PENDING;
}
repo.save(d);
}
}
Notes on this implementation:
- The
Idempotency-Keyis the delivery row's UUID, so every retry of the same delivery sends the same key and the receiver dedupes naturally. - The 1-second polling loop reads up to 100 due rows per cycle; for higher throughput, partition the work across N workers using
SELECT ... FOR UPDATE SKIP LOCKED. - Network exceptions are treated as status
0and retried — these dominate transient failures in practice. - The age cap is checked alongside the attempt cap so a slow receiver that returns 5xx for 24 hours is dead-lettered even if it hasn't burned all attempts.
- Producing the row should happen in the same DB transaction as the business event that triggered it — same Outbox pattern as Kafka.
Common Mistakes & Pitfalls
Relying on synchronous in-process retries with Thread.sleep — this blocks the request thread and loses webhooks if the publisher restarts mid-retry
Check your understanding
Answer these without looking back at the article.
- Why is jitter required for backoff to work correctly?
- How do you distinguish a duplicate webhook from a genuine new event on the receiver side?
- What's the difference between an idempotency key per request vs per logical operation?
- When would you auto-disable a webhook endpoint, and how do you re-enable it?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.