Idempotent Consumers: Safe Duplicate Event Processing
Fast recall
Quick revision
Core answer
Whether you handle duplicate message delivery (guaranteed in at-least-once systems) without causing double-charges, double-inventory-decrements, or other data corruption
Building non-idempotent consumers and relying on 'exactly-once delivery' which is either unavailable or expensive — at-least-once is the practical default
Discuss the deduplication window (how long to retain processed event IDs), show the check-then-act pattern within a transaction, and mention natural idempotency (SET status = 'SHIPPED' is naturally idempotent)
Interview-ready guide
Interview answer
Idempotent consumers are non-negotiable in at-least-once messaging systems. Kafka and RabbitMQ both deliver duplicates during consumer restarts, rebalances, and network issues. If your consumer charges a credit card and then crashes before committing the offset, the broker redelivers and the card gets charged twice.
The explicit deduplication approach:
- Before processing, check if the event ID exists in a processed-events table; if it does, skip; if not, process and record the ID — both within the same database transaction
- If the consumer crashes after processing but before committing, the insert also rolls back so redelivery correctly reprocesses the event from scratch
- Set a 7-day retention window for the deduplication table — this covers consumer restarts, rebalances, and outbox replays while keeping the table bounded
Natural idempotency — the cleaner path when applicable:
- State-setting operations like setting an order status to SHIPPED are naturally idempotent — doing it twice leaves it shipped; use upserts with a version guard so stale events arriving out of order cannot overwrite newer state
- The
WHERE version < EXCLUDED.versionclause in a PostgreSQL upsert makes out-of-order delivery harmless without a deduplication table lookup
External side effects — the hardest case:
- Charging a credit card, sending an SMS, calling an external API — these require the event ID as an idempotency key in the API call
- Stripe and most payment processors deduplicate by this key — same key produces the same result, no second charge
I recommend natural idempotency (upserts and state-setting operations) wherever possible, and explicit deduplication tables only for operations that increment counters, charge cards, or trigger irreversible external side effects.
Idempotent Consumers: Safe Duplicate Event Processing
Direct Answer
An idempotent consumer produces the same result regardless of how many times the same event is processed. Implement idempotency by storing processed event IDs and checking before processing. For database operations, use INSERT ON CONFLICT DO NOTHING or upsert patterns. For external calls, use idempotency keys in API requests.
Common Mistakes & Pitfalls
Building non-idempotent consumers and relying on 'exactly-once delivery' which is either unavailable or expensive — at-least-once is the practical default
Check your understanding
Answer these without looking back at the article.
- How do you handle the case where the event processing succeeds but recording the event ID fails?
- What is the deduplication window, and how do you decide how long to keep processed event IDs?
- How does optimistic locking with @Version relate to idempotent processing?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.