Tiered storage and long retention

Key points

• Traditionally, retention is bounded by broker disk. Keeping ninety days means every broker carries ninety days of local storage, and the cluster is sized for storage rather than for throughput.
• That coupling has a second cost: a broker holding terabytes takes a long time to replace, because a replacement must replicate all of it before it is useful.
• Tiered storage (KIP-405) moves closed segments to object storage. The broker keeps recent data locally and fetches older data from the remote tier on demand.
• This decouples the two dimensions. Local disk is sized for the working set - what consumers actually read - and retention is limited only by what you are willing to pay object storage for.
• Reads of tiered data are slower and go over the network, so it suits backfill and replay rather than a live consumer. A consumer that has lagged into the remote tier will be noticeably slower until it catches up.
• It also makes broker replacement fast again, since only the local working set has to be replicated rather than the entire history.

Example

# Broker: enable the remote tier.
remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.\
  RemoteLogSegmentManager

# Per topic: what stays local, and what the total retention is.
$ kafka-configs.sh --bootstrap-server localhost:9092 --alter \
    --entity-type topics --entity-name events --add-config \
    'remote.storage.enable=true,\
     local.retention.ms=86400000,\
     retention.ms=7776000000'

#   local.retention.ms   1 day    <- on broker disk, served at full speed
#   retention.ms        90 days   <- total; days 2-90 live in object storage

---
# The sizing argument, made concrete. 1 TB/day ingest, replication factor 3:
#
#   Without tiering, 90 days:  1 TB x 90 x 3 = 270 TB of broker disk
#   With tiering,    1 day local: 1 TB x 1 x 3 =   3 TB of broker disk
#                    + 90 TB in object storage (unreplicated, far cheaper)
#
# And broker replacement drops from "replicate 90 TB" to "replicate 1 TB".
#
# The catch: a consumer that falls more than a day behind starts reading from
# object storage and slows down exactly when it most needs to catch up.

Tiered storage separates working-set disk from retention, so long history stops dictating broker size and replacement time.