Topics, Partitions, Offsets

Key points

• A topic is a named stream, like orders. Every topic is split into partitions.
• Each partition is an ordered, append-only log. Every record gets an offset, its position in that log, starting at 0.
• Partitions are the unit of parallelism. Six partitions means at most six consumers in one group can work at the same time.
• You can add partitions later but never remove them. Adding them changes which partition a key hashes to.
• Size it up front: a few thousand partitions per broker is comfortable, hundreds of thousands is not.

Example

# Create a topic with 6 partitions, replicated 3x
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders \
  --partitions 6 --replication-factor 3

# Inspect it: leader, replicas, in-sync replicas per partition
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

# Grow it. This is one-way -- partitions never shrink.
kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 12

Partition count sets the ceiling on your parallelism. Pick it deliberately.