Brokers, Controllers and KRaft

Key points

• A cluster is a set of brokers. Each broker stores partition logs on local disk and serves the partitions it currently leads.
• Clients connect to any host in bootstrap.servers, fetch metadata once, then talk directly to each partition leader.
• KRaft keeps cluster metadata in its own Raft-replicated log, managed by a quorum of controllers. ZooKeeper is removed in Kafka 4.0.
• One controller is active and the rest are hot standbys. Losing it triggers a Raft election measured in seconds, not an outage.
• Dev boxes run combined nodes (process.roles=broker,controller). Production runs three dedicated controllers.
• Nothing about the client API changed with KRaft. It removed an operational dependency, not a concept you have to learn.

Example

# Single node acting as both broker and controller -- fine for dev
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093

listeners=PLAINTEXT://:9092,CONTROLLER://:9093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka/data

# Production: controllers are their own nodes, brokers are their own nodes
# process.roles=controller
# controller.quorum.voters=1@c1:9093,2@c2:9093,3@c3:9093

Brokers hold the data, the controller quorum holds the metadata, and nothing holds ZooKeeper.