Rebalancing Without the Pain

Kafka · lesson 5 of 34 · 4 min read

Know what triggers a rebalance, what it costs you, and the settings that stop it from hurting.

Open this lesson in the learning hub

Key points

  • A rebalance reassigns partitions when a member joins, leaves or dies. Processing pauses for the partitions that move.
  • Most surprise rebalances are self-inflicted: a poll loop that took longer than max.poll.interval.ms (5 minutes by default).
  • Fix slow loops with a smaller max.poll.records, not a bigger timeout. Small batches, frequent polls.
  • CooperativeStickyAssignor revokes only the partitions that actually have to move, instead of everyone dropping everything.
  • For rolling restarts set group.instance.id. That is static membership: a quick restart triggers no rebalance at all.
  • Kafka 4.x adds a broker-side protocol. Set group.protocol=consumer for incremental, non stop-the-world rebalances.

Example

group.id=billing-service

# Cooperative: only the partitions that move get revoked
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Keep every poll loop short and predictable
max.poll.records=100
max.poll.interval.ms=300000
session.timeout.ms=45000
heartbeat.interval.ms=3000

# Stable identity across restarts -> no rebalance on a rolling deploy
group.instance.id=billing-service-pod-0

# Kafka 4.x broker-side protocol (opt in)
# group.protocol=consumer

Rebalances are normal. Long poll loops are not.

Consumers and Consumer Groups What Kafka Guarantees About Order

This is a reading copy. The full lesson — with the visual explainer, the interactive lab and a Run button for the code — lives in the Kafka course, and every lesson in it is listed on the Kafka contents page.