Dead Letter Queue: Handling Failed Events in Spring

A Dead Letter Queue is a safety net for event processing failures, but the real problem it solves is subtler than handling failures gracefully.

• The genuinely dangerous scenario is a poison pill — a single malformed message that will never parse successfully.
• The consumer throws, retries, throws again, and the Kafka offset never advances.
• Every healthy message behind it on that partition is silently blocked.
• The consumer appears alive on health checks, but one-sixth of your processing has stopped.

The DLQ pattern in Spring Kafka:

• Attempt processing, retry with exponential backoff on failure, and after exhausting retries, route to a dead letter topic
• @RetryableTopic handles this declaratively — configure attempts, backoff multiplier, and max delay; it creates intermediate retry topics and a final DLT automatically
• The @DltHandler receives exhausted messages where you alert the team and persist them for investigation and replay

RabbitMQ dead lettering:

• Broker-native via x-dead-letter-exchange on the queue — when a message is rejected or exceeds TTL, the broker routes it to the dead letter queue without application-level retry logic
• The mechanism is transparent to the consumer; the broker handles routing automatically on nack or expiry

The critical follow-through:

• A DLQ that silently fills up is worse than no DLQ — you're hiding production failures while the system appears healthy
• Set alerts for DLT depth exceeding zero for more than five minutes — this is a production incident, not a backlog item
• Replay must be safe: consumers need deduplication by event ID because replaying means processing messages a second time

I recommend wiring dead-letter exchanges and @DltHandler alerts from day one — discovering that a poison pill has silently blocked partition processing after hours of missed processing is far more expensive to recover from.