Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
confluent, profile picture
Uploaded byconfluent
11,500 views

Everything You Always Wanted to Know About Kafka's Rebalance Protocol but Were Afraid to Ask

The document provides a comprehensive overview of Kafka's rebalance protocol, detailing concepts such as group membership, resource assignment, and the behavior of consumer groups during rebalancing events. It discusses design decisions on both the broker and client sides, highlights improvements introduced in Kafka 2.3 related to static group membership and incremental rebalancing, and outlines future work for upcoming versions. Additionally, it includes references to key proposals and resources for further learning about Kafka's rebalancing processes.

Technology◦
Related topics:
Apache Kafka•
1 1 Everything you always wanted to know about Kafka’s rebalance protocol but you were afraid to ask Matthias J. Sax | Software Engineer @MatthiasJSax
What is rebalancing about? ● Group membership ● Resource assignment ● Example: KafkaConsumer ○ Consumer group ○ Partition ownership @MatthiasJSax 2
3 3 3 @MatthiasJSax Design Decisions ● Broker side: membership ○ JoinGroup ○ Heartbeat ○ LeaveGroup ● Client side: assignment ○ SyncGroup ○ Leader
4 4 4 Rebalancing happens if (a) a member joins/leaves the group (b) resources need to be reassigned @MatthiasJSax
5 Let’s Rebalance GroupCoordinator (broker side) heartbeat ok heartbeat ok session.timeout.ms heartbeat.interval.ms C1 C2 C3 @MatthiasJSax 5
6 Let’s Rebalance GroupCoordinator (broker side) C1 C2 C3 C4 JoinGroup (subscription) rebalanceheartbeat @MatthiasJSax synchronization barrier JoinGroup (subscription) JoinResponse max.poll.interval.ms (consumer) rebalance.timeout.ms (connect) C1 C2 C3 C4 6
7 Let’s Rebalance @MatthiasJSax rebalance synchronization barrier JoinGroup (subscription) GroupCoordinator (broker side) Group Leader (receives all subscriptions) Leader Selection C1 C2 C3 C4 SyncGroup JoinResponse SyncResponse (assignment) 6
8 Group- Coordinator Group- Coordinator Group- Coordinator Group- Coordinator Brokers: - Maintain/monitor groups - Store group metadata AbstractCoordinator WorkerCoordinator ConsumerCoordinator Connect API Consumer / Streams API Range RR Sticky custom StreamsAssignor Clients: - Assign resources __consumer_offsets @MatthiasJSax 7
Kafka Connect ● Cluster of workers ● Single group ● Resources: ○ Tasks ○ Configuration @MatthiasJSax 8
10 10 Kafka Streams ● Application instances / threads ● Tasks plus Standbys ○ Stateful ○ Co-partitioning ● Interactive Queries ○ Endpoint metadata @MatthiasJSax 9
11 11 11 Issues @MatthiasJSax 10
12 12 Unnecessary Rebalance group.id=“grp” POD Application member.id=1 group.id=“grp” POD Application member.id=2 group.id=“grp” POD Application member.id=3 GroupCoordinator (broker side) group.id List of member.id “grp” 1,2,3 @MatthiasJSax 11
13 13 Unnecessary Rebalance group.id=“grp” POD Application member.id=2 group.id=“grp” POD Application member.id=3 GroupCoordinator (broker side) group.id List of member.id “grp” 2,3 @MatthiasJSax 11
14 14 Unnecessary Rebalance group.id=“grp” POD Application member.id=4 group.id=“grp” POD Application member.id=2 group.id=“grp” POD Application member.id=3 GroupCoordinator (broker side) group.id List of member.id “grp” 2,3,4 group.id List of member.id “grp” 2,3,4 @MatthiasJSax 11
15 15 Unnecessary Rebalance group.id=“grp” POD Application member.id=4 group.id=“grp” POD Application member.id=2 group.id=“grp” POD Application member.id=3 Why rebalance if we know that the application is restarted anyway? @MatthiasJSax 11
16 16 Stop-the-World Effect Application Application Application Why stop processing if partitions are reassigned anyway? @MatthiasJSax 12
Latest Improvements ● Apache Kafka 2.3 / Confluent Platform 5.3 ○ static group membership ○ incremental rebalancing in Kafka Connect @MatthiasJSax 13
18 18 Static Group Membership group.id=“grp” group.instance.id=“A” POD Application member.id=1 group.id=“grp” group.instance.id=“C” POD Application member.id=3 group.id=“grp” group.instance.id=“B” POD Application member.id=2 group.instance.id member.id A 1 B 2 C 3 GroupCoordinator (broker side) @MatthiasJSax 14
19 19 Static Group Membership group.instance.id member.id A 1 B 2 C 3 GroupCoordinator (broker side) group.id=“grp” group.instance.id=“A” POD Application member.id=1 group.id=“grp” group.instance.id=“C” POD Application member.id=3 group.id=“grp” group.instance.id=“B” POD Application member.id=2 @MatthiasJSax 14
20 20 Static Group Membership group.instance.id member.id A 1 B 2 C 3 GroupCoordinator (broker side) group.id=“grp” group.instance.id=“C” POD Application member.id=3 group.id=“grp” group.instance.id=“B” POD Application member.id=2 @MatthiasJSax 14
21 21 Static Group Membership group.instance.id member.id A 1 B 2 C 3 GroupCoordinator (broker side) group.id=“grp” group.instance.id=“A” POD Application member.id=1 group.id=“grp” group.instance.id=“C” POD Application member.id=3 group.id=“grp” group.instance.id=“B” POD Application member.id=2 @MatthiasJSax 14
22 22 22 Looking into the Future ● Work in progress ○ Incremental rebalancing for Kafka Consumers and Kafka Streams ● Future work ○ Smooth scale-out for Kafka Streams @MatthiasJSax 15
23 23 Incremental Rebalancing GroupCoordinator (broker side) C1 C2 C3 @MatthiasJSax C1 C2 C3 JoinGroup (subscription) rebalanceheartbeat JoinGroup (subscription) JoinResponse 16
24 24 Incremental Rebalancing @MatthiasJSax C1 C2 C3 GroupCoordinator (broker side) Group Leader (received all subscriptions) JoinResponse SyncGroup (intended assignment) SyncResponse (enforce revoke) 16
25 25 Incremental Rebalancing @MatthiasJSax C1 C2 C3 GroupCoordinator (broker side) Group Leader (received all subscriptions) synchronization barrier JoinResponse SyncGroup (intended assignment) SyncResponse (enforce revoke) JoinGroup JoinResponse 16
26 26 Incremental Rebalancing @MatthiasJSax C1 C2 C3 Group Leader eived all subscriptions) synchronization barrier GroupCoordinator (broker side) SyncGroup (intended assignment) SyncResponse (enforce revoke) JoinGroup Group Leader (received all subscriptions) JoinResponse SyncGroup SyncResponse 16
27 27 27 @MatthiasJSax Summary ● Deep dive into rebalance protocol ○ Powerful and flexible ○ Stop-the-world property ● Since AK 2.3 / CP 5.3 ○ Static group membership ○ Incremental rebalancing (Connect) ● Work in progress for AK 2.4 / CP 5.4 17
28 28 28 ● heartbeat.interval.ms ● session.timeout.ms vs max.poll.interval.ms ● rebalance.timeout.ms ● Static group membership ○ group.instance.id ● Quick vs “considered” rebalance Lessons Learned @MatthiasJSax 18
29@MatthiasJSax Resources • KIP-345: Introduce static membership protocol to reduce consumer rebalances (Apache Kafka 2.3 / Confluent Platform 5.3) https://cwiki.apache.org/confluence/display/KAFKA/KIP- 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances • Design doc: Incremental Cooperative Rebalancing https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies • KIP-415: Incremental Cooperator Rebalancing in Kafka Connect (Apache Kafka 2.3 / Confluent Platform 5.3) https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect • KIP-429: Kafka Consumer Incremental Rebalance Protocol (accepted) https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol • KIP-441: Smooth Scaling Out for Kafka Streams (under discussion) https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams • "The Magical Rebalance Protocol of Apache Kafka" by Gwen Shapira (Strange Loop Talk, Sep 2018) https://www.youtube.com/watch?v=MmLezWRI3Ys&t=8s • Thanks to Jason Gustafson and Guozhang Wang 19
30 30 We are hiring! @MatthiasJSax matthias@confluent.io | mjsax@apache.org