Kafka Log Retention vs Log Compaction
Fast recall
Quick revision
Core answer
retention.ms or once total size exceeds retention.bytes. The compact policy keeps only the latest value per key and uses null-valued tombstones to delete keys; older versions get garbage collected. Use delete retention for event streams and audit logs; use compaction for stateful topics like __consumer_offsets, CDC change logs, and config topics where consumers only care about the current state per key. The two policies can be combined as cleanup.policy=compact,delete.Whether you understand Kafka's segment-based log layout and the two retention strategies, including when to use compaction vs time-based deletion
Assuming compaction immediately removes old values — it only runs on inactive segments, so the active segment always has duplicates and tombstones aren't deleted until `delete.retention.ms` later
Explain the role of the active segment, why `__consumer_offsets` is a compacted topic, and the `cleanup.policy=compact,delete` combination for time-bounded compacted topics
Interview-ready guide
Interview answer
Kafka stores each partition as a series of segment files — one active segment that producers append to, plus a chain of immutable closed segments. Cleanup policies operate on closed segments by deleting or rewriting whole files.
Two policies: - cleanup.policy=delete is the default — once a closed segment is older than retention.ms or the partition exceeds retention.bytes, the broker drops the whole segment file.
- Use this for event streams and audit logs.
cleanup.policy=compactkeeps only the most recent record per key.- To delete a key you publish a record with the same key and a null value — that's a tombstone, and after
delete.retention.msit gets garbage collected too.
Important operational details:
- The active segment is never compacted, so a low-volume compacted topic needs
segment.msset to a few hours so segments roll on time even without producer traffic — otherwise duplicates pile up indefinitely - Compaction runs when the dirty ratio crosses
min.cleanable.dirty.ratio, default 0.5; the cleaner thread reads closed segments, builds a key-to-latest-offset map, and rewrites them keeping only the latest value per key - For compacted topics, producers must always send a real key — null keys can't be deduplicated and the topic will grow without bound
Where each fits:
- Delete for order events, page views, and audit logs where history matters
- Compact for
__consumer_offsets, Kafka Streams state stores, CDC change logs, and configuration topics where consumers only care about current state per key cleanup.policy=compact,deletecombines the two for compacted state that also needs a hard time bound — useful for GDPR or just controlling storage cost
A single compacted partition can replace what would otherwise need a separate database for storing per-key state, which is why it underpins so much of the Kafka Streams ecosystem.
Kafka Log Retention vs Log Compaction
Direct Answer
Kafka stores each partition as an ordered set of segment files. The delete retention policy (default) drops whole segments older than retention.ms or once total size exceeds retention.bytes. The compact policy keeps only the latest value per key and uses null-valued tombstones to delete keys; older versions get garbage collected. Use delete retention for event streams and audit logs; use compaction for stateful topics like __consumer_offsets, CDC change logs, and config topics where consumers only care about the current state per key. The two policies can be combined as cleanup.policy=compact,delete.
Code Example
Topic configs are normally set with kafka-configs.sh or via the kafka-topics.sh --create command. The three common shapes:
# 1) Pure delete retention — event log, 7 days, max 100 GiB
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
--topic order-events \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=delete \
--config retention.ms=604800000 \
--config retention.bytes=107374182400
# 2) Pure compaction — keep latest value per key forever
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
--topic user-profile-changelog \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.5 \
--config segment.ms=3600000 \
--config delete.retention.ms=86400000
# 3) Hybrid: compact AND delete after 30 days
# Useful when you only care about latest value per key
# BUT you also want a hard time bound (GDPR, storage cost)
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
--topic device-state-changelog \
--partitions 12 --replication-factor 3 \
--config cleanup.policy=compact,delete \
--config retention.ms=2592000000
For compacted topics in Spring producer code, always send a real key (not null) — compaction is keyed and a null key cannot be deduplicated:
ProducerRecord<String, String> rec = new ProducerRecord<>(
"user-profile-changelog",
user.getId(), // KEY required for compaction
objectMapper.writeValueAsString(user)
);
kafkaTemplate.send(rec);
// To DELETE a key, send a tombstone (null value, same key):
kafkaTemplate.send(new ProducerRecord<>(
"user-profile-changelog", deletedUserId, null));
Common Mistakes & Pitfalls
Assuming compaction immediately removes old values — it only runs on inactive segments, so the active segment always has duplicates and tombstones aren't deleted until delete.retention.ms later
Check your understanding
Answer these without looking back at the article.
- Why is the `__consumer_offsets` topic compacted instead of using delete retention?
- Why must compacted topics always have a non-null key?
- What is `min.cleanable.dirty.ratio` and when would you tune it?
- How does Kafka Streams use compacted topics to back state stores?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.