Webhooks: Design, Reliability, and Security

Webhooks are HTTP callbacks from your service to third-party URLs when an event occurs. Stripe calls your webhook URL when a payment succeeds; GitHub calls your CI server on code push.

They're the industry-standard way to notify third-party systems of events in real time, winning over polling (wasteful) and message queues (can't cross trust boundaries).

Design concerns

Delivery reliability

• Receivers go down, become slow, get deployed
• Sender must retry; exponential backoff with jitter
• Stripe's schedule: 1s, 5s, 30s, 1m, 5m, 30m, 1h, 6h, 12h, daily for 3 days
• 3 to 7 day retry window is standard
• Dead-letter queue after max retries; alert customer

Security

• Receiver URL is public; anyone can POST
• HMAC-SHA256 signature with shared secret
• Include timestamp in signing string to prevent replay attacks
• 5-minute tolerance for clock skew
• Constant-time signature comparison to prevent timing attacks
• Optionally IP allowlist (Stripe, GitHub publish ranges)

Idempotency

• Retries cause duplicates
• Receivers must dedupe on event ID
• Store processed event IDs with TTL (72 hours typical)
• Return 2xx for duplicates; not an error

Ordering

• HTTP delivers in arbitrary order, especially after retries
• Don't assume order; include sequence numbers if order matters
• Or use single retryable queue per subscription (no parallel workers)

Consumer isolation

• One slow receiver must not block others
• Per-subscription or per-tenant queues
• Separate worker pools
• Rate limit outgoing per URL (10 concurrent max typical)

Sender architecture

• Event fires; fan out to all active subscriptions
• Durable queue per subscription
• Worker pool signs and POSTs; 10 second timeout
• On 2xx: mark delivered
• On 4xx non-retryable: mark failed, alert customer
• On 5xx or timeout: increment retry, exponential backoff + jitter
• After max retries: DLQ, alert, optionally auto-pause subscription
• Customer dashboard with delivery history, replay button, secret rotation

Receiver pattern

• Verify signature and timestamp; reject if invalid
• Dedupe on event ID; return 2xx if already processed
• Enqueue for async processing
• Return 200 immediately; ack within 5 seconds
• Never process inline; senders timeout at 10 to 30 seconds
• Separate background worker does actual work with idempotent logic
• Be tolerant of unknown event types (ack and log, don't reject)
• Be tolerant of schema evolution (ignore unknown JSON fields)

Common mistakes

• Synchronous processing in handler (times out; sender retries; duplicates everywhere)
• No signature verification (security hole)
• No timestamp in signature (replay attacks)
• Using equals() to compare signatures (timing attack)
• No dedupe (duplicate side effects)
• Shared worker pool across customers (one slow customer blocks all)
• No retry window (transient failures become permanent)
• No DLQ visibility (customer never knows events were lost)

Signature schemes

• HMAC-SHA256 is industry standard (Stripe)
• HMAC-SHA1 older (GitHub historical); still secure but SHA256 preferred
• Ed25519 asymmetric for high-security (GitHub apps): sender signs with private key, receiver verifies with public
• mTLS: both sides present certs; operationally complex but very strong

Secret rotation

• Receivers should accept two active secrets during transition
• Sender rotates; waits for old-signed events to drain; deactivates old

Observability

• Sender: per-subscription success rate, p95 latency, retry count, DLQ count
• Receiver: signature failure rate (should be near zero), dedupe hit rate (expected with retries), processing lag
• Dashboards and alerts on both sides

Webhook alternatives

• Polling: pull; wasteful but no endpoint required; works behind NAT
• SSE or WebSockets: persistent; best for browsers, not server-server
• Message queues: best for internal; not usable across trust boundaries
• Webhook platforms (Svix, Hookdeck, Ngrok): SaaS handles signing, retries, DLQ; build-vs-buy

Replay capability

• Offer customer a 'replay this event' button
• Re-enqueues; receiver dedupes or not depending on customer preference
• Critical for customer trust

The strategic answer

• Treat webhooks as async message delivery over HTTP
• Same patterns as Kafka consumers: durable queue, idempotent consumer, at-least-once, DLQ
• Sender: queue, worker pool, retry with backoff, DLQ, customer dashboard
• Receiver: verify signature, dedupe, ack fast, process async
• HMAC-SHA256 + timestamp; constant-time compare; idempotent processing
• Observability on both sides; SLAs documented and monitored