Retention and Log Compaction

Key points

• Default cleanup.policy=delete: segments older than retention.ms (7 days) or past retention.bytes are dropped.
• Retention is per topic and has nothing to do with consumption. Data is never deleted because someone read it.
• cleanup.policy=compact keeps the latest value for every key indefinitely. The topic becomes a changelog you can rebuild state from.
• To delete a key from a compacted topic, send a tombstone: the same key with a null value.
• Compaction is lazy and runs in the background. The head of the log always holds duplicates, so never assume one record per key.
• Use compact,delete when you want the latest value per key but only within a bounded time window.

Example

# Raw events: keep 7 days, then drop
kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name order-events \
  --add-config retention.ms=604800000,cleanup.policy=delete

# State topic: latest value per customer, kept forever
kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name customer-profiles \
  --add-config cleanup.policy=compact,min.cleanable.dirty.ratio=0.1

# A tombstone is simply a record with a null value:
#   producer.send(new ProducerRecord<>("customer-profiles", "customer-7", null));

delete is a time window. compact is a keyed snapshot. Choose by what the topic means.