How a partition is stored on disk

Key points

• A partition is a directory of segment files, not one file. Only the newest segment is open for writing; the rest are immutable.
• Retention deletes whole segments, never individual records. That is why data can outlive its retention window: a low-traffic partition takes a long time to fill segment.bytes, and segment.ms is what actually bounds it.
• Each segment has two companion index files - offset to file position, and timestamp to offset - which are sparse, so a lookup finds the nearest entry and scans forward from there.
• Kafka does not maintain its own cache. Writes go to the OS page cache and are flushed by the kernel, which is why a broker runs a small JVM heap and leaves most of the machine to the operating system.
• A consumer reading recent data usually never touches disk - the pages are still cached from the write. This is why consumer lag matters for throughput and not only for freshness.
• Zero-copy: sendfile moves bytes from page cache straight to the socket without passing through the JVM. Anything that forces the broker to re-encode a batch - TLS, recompression, format down-conversion - loses it.

Example

# One partition on disk. Base filename is the first offset in the segment.
$ ls -la /var/lib/kafka/data/orders-0/

  00000000000000000000.log     1.0G   # closed, immutable
  00000000000000000000.index   10M    # sparse offset -> position
  00000000000000000000.timeindex 10M  # sparse timestamp -> offset
  00000000000002418533.log     412M   # ACTIVE - being appended to
  00000000000002418533.index   10M
  leader-epoch-checkpoint

# Read the log directly - this is how you confirm what is really stored.
$ kafka-dump-log.sh --files 00000000000002418533.log --print-data-log | head

  baseOffset: 2418533 lastOffset: 2418591 count: 59 compresscodec: LZ4
  producerId: 4001 producerEpoch: 3 isTransactional: true
  | offset: 2418533 keySize: 8 valueSize: 214 payload: {...}

# The settings that decide when a segment closes - and therefore when
# retention can act on it at all:
#
#   segment.bytes = 1073741824   (1 GiB)  roll when full
#   segment.ms    = 604800000    (7 days) roll on age, whichever comes first
#   retention.ms  = 604800000    (7 days) delete CLOSED segments older than this
#
# A topic doing 1 MB/day never fills a 1 GiB segment, so without segment.ms
# it would keep everything for years while claiming a 7-day retention.

Retention operates on closed segments, so segment.ms - not retention.ms alone - is what really bounds how long data lives.