Delivery Semantics: Exactly-Once vs At-Least-Once vs At-Most-Once

These three delivery semantics describe what happens when failures occur in the messaging pipeline.

• At-most-once acknowledges before processing — fast, but if the consumer crashes, the message is gone.
• At-least-once acknowledges after processing — no message loss, but the producer might retry or the consumer might reprocess after a crash, creating duplicates.

Exactly-once — the expensive guarantee:

• Requires coordination between producer, broker, and consumer that adds roughly 2–3× latency per batch
• Kafka achieves it with idempotent producers — which deduplicate retries using a sequence number, enabled by default since Kafka 3.0 — plus transactional producers that atomically commit consumed offsets and produced messages
• The important boundary: Kafka's exactly-once guarantee stops at Kafka; writing to PostgreSQL or calling a payment API is outside the transaction, so external effects still need application-level idempotency

The practical alternative — at-least-once with idempotent consumers:

• Most systems don't need true exactly-once; at-least-once with idempotent consumers gives effectively the same outcome at lower cost
• If your consumer checks a processed-events table before doing work, or uses database upserts, or relies on naturally idempotent operations like setting a status field, then duplicates are harmless
• You end up writing deduplication logic anyway even with Kafka transactions, since external effects are outside the transaction boundary

When to use each:

• At-least-once with idempotent consumers for the vast majority of event processing
• Reserve Kafka transactions for financial ledgers and billing where a duplicate line item is a compliance violation

I recommend at-least-once with idempotent consumers as the default — it gives you safety without the Kafka transaction latency overhead, and you control the deduplication logic explicitly.