Partition reassignment without hurting the cluster

Key points

• Adding a broker does not move any data. New partitions may land on it, but existing ones stay where they are until you reassign them explicitly.
• Reassignment copies whole partitions between brokers. On a large topic that is hundreds of gigabytes of replication traffic competing with live produce and fetch.
• Unthrottled, it saturates network and disk, pushes followers out of the ISR, and can cause the instability it was meant to fix. Always set a replication throttle.
• Leadership skew is a separate problem from data skew. After failures, leadership drifts and concentrates - and since only leaders serve traffic, a broker can be idle on disk while overloaded on network.
• Preferred leader election restores leadership to the first replica in each assignment list, which is the cheap fix for that skew - no data moves at all.
• Automated balancers exist because the input to a good plan is more than partition count: disk usage, leader distribution, rack placement and current throughput all matter, and hand-written plans rarely account for all of them.

Example

# 1. Describe what you have before changing anything.
$ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

  Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Topic: orders  Partition: 1  Leader: 1  Replicas: 1,3,2  Isr: 1,3,2
  #                            ^^^^^^^^^ every leader on broker 1 - skewed

# 2. Generate a plan, and KEEP the rollback file it prints.
$ kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
    --topics-to-move-json-file topics.json \
    --broker-list "1,2,3,4" --generate

# 3. Execute WITH a throttle. This is the step people skip.
$ kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
    --reassignment-json-file plan.json --execute \
    --throttle 50000000            # 50 MB/s, leaving headroom for live traffic

# 4. Verify - and note that this is also what REMOVES the throttle.
#    Forgetting it leaves replication permanently capped, which shows up
#    weeks later as unexplained under-replicated partitions.
$ kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \
    --reassignment-json-file plan.json --verify

# Leadership skew only - no data movement, seconds not hours:
$ kafka-leader-election.sh --bootstrap-server localhost:9092 \
    --election-type PREFERRED --all-topic-partitions

Throttle every reassignment and remember that --verify is what removes the throttle - a forgotten one caps replication indefinitely.