Kafka Partition Reassignment

When a new Kafka node is added, Kafka automatically does not do the partition reassignment. The following steps are used to perform manual reassignment of Kafka partitions for specified topic(s).

1. Enter shell of Kafka Docker container:

   docker exec -it kafka-1 bash
   

2. Create /tmp/topics.json with topics whose partitions should be reassigned in the following format:

   cat << EOF | tee /tmp/topics.json
   {
   "topics": [
   {"topic": "events.tenant.stream"},
   ],
   "version": 1
   } 
   EOF
   

3. Generate reassignment JSON output from list of topics to be migrated. Specify the broker IDs in the broker list:

   /usr/bin/kafka-reassign-partitions --zookeeper localhost:2181 --broker-list "121,122,221,222" --generate --topics-to-move-json-file /tmp/topics.json
   

   The result should be stored in /tmp/reassign.json and look as follows, with all topics and partitions having their new assignment specified:

   [appuser@lm11 data]$ cat /tmp/reassign.json
   {"version":1,"partitions":[{"topic":"events.tenant.stream","partition":0,"replicas":[122],"log_dirs":["any"]},{"topic":"events.tenant.stream","partition":1,"replicas":[221],"log_dirs":["any"]},{"topic":"events.tenant.stream","partition":2,"replicas":[222],"log_dirs":["any"]},{"topic":"events.tenant.stream","partition":3,"replicas":[121],"log_dirs":["any"]},{"topic":"events.tenant.stream","partition":4,"replicas":[122],"log_dirs":["any"]},{"topic":"events.tenant.stream","partition":5,"replicas":[221],"log_dirs":["any"]}]}
   

4. Use the output from the previous command as input to the execution of the reassignment/rebalance:

   /usr/bin/kafka-reassign-partitions --zookeeper localhost:2181  --execute --reassignment-json-file /tmp/reassign.json --additional --bootstrap-server localhost:9092
   

That's it! Now Kafka should perform the partitions reassignment within the following hours.

For more information, see Reassigning partitions in Apache Kafka Cluster.