Kafka Replication and Producer Durability: acks, Idempotence, Transactions
Fast recall
Quick revision
Core answer
replication.factor times across brokers; the leader handles writes and followers fetch to stay in the ISR. Producer `acks` controls what counts as a successful write: acks=0 is fire-and-forget, acks=1 waits for the leader, acks=all waits for every ISR member. `enable.idempotence=true` (default since Kafka 3.0) gives exactly-once per partition by tagging records with a producer ID and sequence number so the broker rejects duplicates from retries. Transactional producers (transactional.id) extend exactly-once across multiple partitions and topics by wrapping produces and offset commits in an atomic broker-coordinated transaction.Whether you understand the layered guarantees Kafka offers: replication for durability, acks for visibility, idempotence for per-partition exactly-once, and transactions for cross-partition atomicity
Believing `acks=all` alone gives exactly-once — it gives at-least-once; without idempotence, retries still produce duplicates
Explain that idempotence is per-partition only, transactions are required for atomic multi-partition writes, and that read_committed isolation is mandatory for downstream exactly-once
Interview-ready guide
Interview answer
Kafka durability is built in layers. Replication is the foundation: each partition has a leader and N followers, and followers fetch continuously from the leader to stay in the ISR. Producer settings then decide what counts as a successful write.
The acks setting:
acks=0is fire-and-forget — fast but unsafeacks=1waits for the leader only — leader failure before replication loses dataacks=allwaits for every current ISR member — but it silently degrades toacks=1if the ISR shrinks to just the leader, which is why you needmin.insync.replicas=2on the topic
Idempotence:
enable.idempotence=true, default since Kafka 3.0, gives exactly-once per partition- The broker assigns each producer a unique producer ID, the producer tags every record with a sequence number per partition, and the broker dedupes retries by checking the highest sequence it accepted
- The broker also rejects out-of-order writes, so ordering is preserved even with multiple in-flight requests
- Idempotence requires
acks=alland the producer enforces it
Transactions:
- Idempotence is per partition only; for atomic writes across multiple partitions or topics — including the consumer offset commit — use a transactional producer with
transactional.id - A broker-side transaction coordinator tracks the transaction and writes commit or abort markers to every involved partition
- Downstream consumers must set
isolation.level=read_committedto skip records from aborted or in-flight transactions - Throughput typically drops 10-30% versus pure idempotence, so use transactions only when you actually need cross-partition atomicity
Important boundary:
- Kafka transactions don't cover Kafka + a relational database — for that use the outbox pattern
My default for any new producer is acks=all, enable.idempotence=true, min.insync.replicas=2 on the topic, and add transactions only when the workload genuinely needs atomic multi-partition writes.
Kafka Replication and Producer Durability: acks, Idempotence, Transactions
Direct Answer
Each partition is replicated replication.factor times across brokers; the leader handles writes and followers fetch to stay in the ISR. Producer acks controls what counts as a successful write: acks=0 is fire-and-forget, acks=1 waits for the leader, acks=all waits for every ISR member. enable.idempotence=true (default since Kafka 3.0) gives exactly-once per partition by tagging records with a producer ID and sequence number so the broker rejects duplicates from retries. Transactional producers (transactional.id) extend exactly-once across multiple partitions and topics by wrapping produces and offset commits in an atomic broker-coordinated transaction.
Code Example
Idempotent producer (covers most use cases):
@Configuration
public class KafkaProducerConfig {
@Bean
public ProducerFactory<String, OrderEvent> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092,kafka2:9092,kafka3:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
// Durability and idempotence (default true on 3.0+, set explicitly for clarity)
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);
return new DefaultKafkaProducerFactory<>(props);
}
@Bean
public KafkaTemplate<String, OrderEvent> kafkaTemplate(ProducerFactory<String, OrderEvent> pf) {
return new KafkaTemplate<>(pf);
}
}
Transactional producer (atomic multi-partition writes + offset commit):
@Configuration
public class TransactionalKafkaConfig {
@Bean
public ProducerFactory<String, OrderEvent> txnProducerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092,kafka2:9092,kafka3:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
DefaultKafkaProducerFactory<String, OrderEvent> pf = new DefaultKafkaProducerFactory<>(props);
// The transactional.id prefix MUST be stable per logical producer instance.
// Spring appends a per-instance suffix.
pf.setTransactionIdPrefix("order-svc-tx-");
return pf;
}
@Bean
public KafkaTransactionManager<String, OrderEvent> kafkaTxManager(
ProducerFactory<String, OrderEvent> pf) {
return new KafkaTransactionManager<>(pf);
}
@Bean
public KafkaTemplate<String, OrderEvent> txnTemplate(ProducerFactory<String, OrderEvent> pf) {
return new KafkaTemplate<>(pf);
}
}
@Service
public class OrderProcessor {
private final KafkaTemplate<String, OrderEvent> kafka;
public OrderProcessor(KafkaTemplate<String, OrderEvent> kafka) { this.kafka = kafka; }
// Both sends commit atomically with the consumer offset — or all roll back.
@KafkaListener(topics = "orders", groupId = "order-svc")
@Transactional("kafkaTxManager")
public void onOrder(OrderEvent in) {
kafka.send("inventory-updates", in.orderId(), new InventoryAdjust(in));
kafka.send("email-queue", in.orderId(), new EmailRequest(in));
// No explicit commit — Spring commits offset + sends inside one transaction.
}
}
Downstream consumer must use read_committed to skip aborted or in-flight transactional records:
spring:
kafka:
consumer:
isolation-level: read_committed
enable-auto-commit: false
Common Mistakes & Pitfalls
Believing acks=all alone gives exactly-once — it gives at-least-once; without idempotence, retries still produce duplicates
Check your understanding
Answer these without looking back at the article.
- Why does idempotence require `acks=all`?
- What's the practical throughput cost of enabling Kafka transactions?
- How does `isolation.level=read_committed` change consumer behavior?
- When would you choose the outbox pattern over Kafka transactions?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.