"Load balancing is a key factor in achieving high performance and cost efficiency for Kafka clusters. It helps on saving over-provisioned resources caused by skewed brokers, either CPU, memory, or disk storage. However, it’s not easy to maintain load balancing for a cluster due to the variable topic traffic pattern and capacity expansion. Data ingestion use cases have several unique characteristics: 1) There is no order requirement for data events; 2) All partitions from a topic are produced and consumed evenly; 3) Kafka producers and consumers are transparent about the partition count increase. Therefore, each partition from the same topic has the same requirements of Kafka broker resources. With these observations, we proposed a new partition placement strategy that is able to achieve the optimal cluster load balancing with Zero Data movement. The strategy will place partitions and leaders from the same topic evenly among all brokers for cluster operations, such as new topic creation, topic parallelism change, and Kafka cluster expansion. It does not need costly data partitions movement among the brokers in the runtime. Leveraging this approach, all of our Kafka clusters are balanced very well, which has resulted in measurable storage cost savings without any performance degradation."

1. Balance Kafka Cluster with Zero Data Movement
Yaodong Yang (Apple), Haochen Li (Apple)

2. Yaodong Yang, Apple Inc. May, 2023
Haochen Li, Apple Inc. NOT A CONTRIBUTION
Balance Kafka Cluster with Zero
Data Movement

3. Kafka Cluster Load Balancing
• Bene
fi
ts
• High Performance
• Cost E
ffi
ciency
• Determining Factors
• Kafka Partition Placement
• Kafka Partition Access Pattern
• Challenges
• Kafka Partitions are Heterogenous
• Storage Retention Requirement
• Produce & Consume Tra
ffi
c Pattern

4. Current Solution
• Continuously rebalance Kafka cluster based on Load Metrics
• collect the load metrics from Kafka
• generate the cluster load model
• compute the optimization proposal
• execute the proposal
• Overhead
• data movement between di
ff
erent brokers
• negative impact for producers and consumers
• long time to
fi
nish (hours or even days)
• infra cost

5. Data Ingestion Use Case
• Workload Pattern
• Data events are randomly assigned
to partitions from the kafka topic
• All partitions from one topic are
consumed evenly
• Kafka producers and consumers
don’t have strict requirement for
Kafka Partition Count
• Kafka Partitions from the same topic
• Same data volumes produced,
consumed and retained

6. Kafka Partition Replica Placement
• Partition Replica Placement Strategy
• Partition Count
• scale_number: Number of Leader Replica per broker
for a topic
• partition_count = scale_number * broker_count
• Partition Replica Placement
• For every Kafka Topic, the number of Replicas in each
broker should be the same.
• For every Kafka Topic, the number of Leader Replicas in
each broker should be the same.
• Same load on individual Kafka Brokers.
• Same hardware utilization on individual Kafka brokers
• CPU
• Storage Volume
• Network

7. Scenarios
• New Topic Creation
• Generate the Replica Assignment for the new topic
• Create the topic in the Kafka cluster with the above Replica Assignment

8. Scenarios
• Increase Partition Count: scale_number increase
• Generate the Replica Assignment for the new partitions
• Create partitions in the Kafka cluster with the above Replica Assignment

9. Scenarios
• Add more brokers
• Generate the Replica Assignment for partitions in new brokers
• Create partitions in the Kafka cluster with the above Replica Assignment

10. Scenarios
• Ingestion tra
ffi
c volume and retention changes
• no impact on the load balance of Kafka Cluster
• Remove some brokers
• data movement is unavoidable
• avoid it with cluster migration if possible
• Cluster Migration & Merge
• rebalance the cluster:
• partition reassignment
• scale_number increase

11. Implementation
• Current
• Implemented as a Topic Operator
• Deployed in production
• Plan
• Open a KIP in Apache Kafka Project
• Contribute back to upstream

12. Take Away
• Partition Placement Strategy can greatly improve the Load Balance of Kafka
Clusters

13. Thank you!