The document proposes an incremental cooperative rebalancing protocol to address issues with Kafka's current eager rebalancing algorithm. The current algorithm revokes all partitions from consumers and immediately triggers a full reassignment, causing stop-the-world behavior. The new protocol would only reassign partitions that need to change ownership after membership changes, reducing unnecessary rebalances and revocations. It aims to rebalance incrementally and cooperatively rather than eagerly.

1. A. Sophie Blee-Goldman, Guozhang Wang
Bay Area Kafka Meetup, Dec. 5, 2019
The Silver Bullet for Endless Rebalances
Introduction to the Incremental Cooperative Protocol

2. Outline
• Review of the current eager rebalance algorithm
• Identify the known issues with common scenarios
• A new proposal: incremental cooperative rebalancing
2

3. 3
A Short History of Consumer Groups
Topic 1
Topic 2
Partitions
Producers
Producers
Consumers
Consumers
Brokers

4. 4
A Short History of Consumer Groups
Consumers
Consumers
fetch
fetch
1) assignment (who owns what)
2) offset (consumed up to where)
Kafka 0.8.2-

5. 5
A Short History of Consumer Groups
Consumers
Consumers
fetch
fetch
1) assignment (who owns what)
2) offset (consumed up to where)
Kafka 0.9.0+
Group Coordinator

6. 5
A Short History of Consumer Groups
Consumers
Consumers
fetch
fetch
1) assignment (who owns what)
2) offset (consumed up to where)
Kafka 0.9.0+
Group Coordinator

7. 6
Consumer Rebalance Protocol
• A rebalance happens when:
• Membership change
• Member crash: failure of a consumer
• Scaling in: member leaves the group
• Scaling out: new member joins
• Partition resources change
• Topics are created or deleted
• More partitions added to topics

8. 7
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)

9. 7
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

10. 7
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat heartbeat
ok ok

11. 8
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

12. 8
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

13. 8
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

14. 8
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

15. 8
Member Crash: Failure Detection (heartbeat)
C1
C2
Group Coordinator (broker side)
session.timeout.ms
heartbeat
ok

16. 9
Scaling In: Consumer Shutdown (leave-group)
C1
C2
Group Coordinator (broker side)
heartbeat
ok

17. 9
Scaling In: Consumer Shutdown (leave-group)
C1
C2
Group Coordinator (broker side)
heartbeat leave-group
ok

18. 9
Scaling In: Consumer Shutdown (leave-group)
C1
C2
Group Coordinator (broker side)
heartbeat leave-group
ok

19. 10
Scaling Out: Consumer Startup (join-group)
C1
C2
Group Coordinator (broker side)

20. 10
Scaling Out: Consumer Startup (join-group)
C1
C2
Group Coordinator (broker side)
C3

21. 10
Scaling Out: Consumer Startup (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group

22. 10
Scaling Out: Consumer Startup (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group

23. 11
Resources Change: Re-Subscribe (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group

24. 11
Resources Change: Re-Subscribe (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group
consumer resubscribe

25. 11
Resources Change: Re-Subscribe (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group join-group
consumer resubscribe

26. 11
Resources Change: Re-Subscribe (join-group)
C1
C2
Group Coordinator (broker side)
C3
join-group join-group
consumer resubscribe

27. 12
Consumer Rebalance Protocol
• During the rebalance:
• Existing consumers re-join the group
• A single member is chosen as group leader
• leader determines partition assignment (user customizable)

28. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
1 2 3
4 5 6

29. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
1 2 3
4 5 6

30. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
1 2 3
4 5 6

31. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
1 2 3
4 5 6

32. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
1 2 3
4 5 6

33. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
1 2 3
4 5 6

34. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)1 2 3
4 5 6

35. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

36. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

37. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
sync. barrier
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

38. 13
Consumers Re-join Group
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
rebalance.timeout.ms
sync. barrier
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

39. 14
Partition Reassignment (sync-group)
re-join
join-group
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

40. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

41. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

42. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
sync-group
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

43. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

44. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

45. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

46. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

47. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

48. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
#onPartitionsAssigned(1,2)
#onPartitionsAssigned(4,5)
#onPartitionsAssigned(3,6)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

49. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
#onPartitionsAssigned(1,2)
#onPartitionsAssigned(4,5)
#onPartitionsAssigned(3,6)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
1 2
4 5
3 6
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

50. 14
Partition Reassignment (sync-group)
re-join
join-group
select C1 as leader
#assign(…)
sync-group
#onPartitionsAssigned(1,2)
#onPartitionsAssigned(4,5)
#onPartitionsAssigned(3,6)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
1 2
4 5
3 6
#onPartitionsRevoked(1,2,3)
#onPartitionsRevoked(4,5,6)

51. 15
Summary of Rebalance Protocol
• ConsumerRebalanceListener
• #onPartitionsRevoked (before sending join-group)
• #onPartitionsAssigned (after receiving sync-group)
• ConsumerPartitionAssignor
• #assign (only triggered by the leader)
Built-in: {range, round-robin, sticky}; Custom: {streams, …}

52. 16
Known Issue #1: Stop-the-world Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions) #assign(…)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
1 2 3
4 5 6
1 2
4 5
3 6

53. 16
Known Issue #1: Stop-the-world Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions) #assign(…)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
revoked all
1 2 3
4 5 6
1 2
4 5
3 6

54. 16
Known Issue #1: Stop-the-world Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions) #assign(…)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
revoked all re-assigned most
1 2 3
4 5 6
1 2
4 5
3 6

55. 16
Known Issue #1: Stop-the-world Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions) #assign(…)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
revoked all re-assigned most
eager rebalance:
before rebalance revoked all the partitions,
after rebalance most of the partitions are reassigned back
1 2 3
4 5 6
1 2
4 5
3 6

56. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
1 2
4 5
3 6
1 2
4 5
3 6

57. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
1 2
4 5
3 6
1 2
4 5
3 6

58. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
1 2
4 5
3 6
1 2
4 5
3 6

59. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
bounce a consumer
1 2
4 5
3 6
1 2
4 5
3 6

60. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
bounce a consumer
1 2
4 5
3 6
1 2
4 5
3 6

61. 17
Known Issue #2: Back-and-forth Rebalance
join-group
re-
join-group
#onPartitionsRevoked(all partitions)
sync-
#onPartitionsAssigned(given partitions)
re-join
sync-group
C1
C2
Group Coordinator(broker side)
C3
leave-group
#assign(…) #onPartitionsRevoked(all partitions) #assign(…)
unnecessary rebalances:
first one to move partitions from C3 to C1/C2,
second one to move them back to C3 from C1/C2
bounce a consumer
1 2
4 5
3 6
1 2
4 5
3 6

62. 18
Let’s Revisit:
When to trigger a rebalance,
Who to participate in a rebalance,
What to reassign during rebalance

63. 19
Rebalance Protocols
When Who What
Current Protocol
(Eager)
Immediately Everyone Everything

64. 20
When Who What
Current Protocol
(Eager)
Immediately Everyone Everything
Proposed Protocol
(Cooperative)
After determining what
needs to be reassigned
Only those whose
assignment will be
changed
Only those partitions
who change ownership
Rebalance Protocols

65. 21
Rebalance Protocols
• [KIP-415] : incremental rebalance in Connect(2.3+)
• [KIP-345] : static membership in Consumer / Streams(2.3+)
• [KIP-429] : incremental rebalance in Consumer / Streams(2.4+)

66. 21
Rebalance Protocols
• [KIP-415] : incremental rebalance in Connect(2.3+)
• [KIP-345] : static membership in Consumer / Streams(2.3+)
• [KIP-429] : incremental rebalance in Consumer / Streams(2.4+)

67. 22
Incremental Assignment in the Consumer

68. 22
Incremental Assignment in the Consumer
owned-partitions

69. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions

70. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions
unchanged-partitions

71. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions
partitions-to-be-revoked
unchanged-partitions

72. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions
partitions-to-be-revoked
unchanged-partitions
#onPartitionsRevoked

73. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions
partitions-to-be-revoked partitions-to-be-added
unchanged-partitions
#onPartitionsRevoked

74. 22
Incremental Assignment in the Consumer
owned-partitions assigned-partitions
partitions-to-be-revoked partitions-to-be-added
unchanged-partitions
#onPartitionsRevoked #onPartitionsAssigned

75. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
1 2 3
4 5 6

76. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6

77. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }

78. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }

79. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }

80. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }

81. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }

82. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2 3
4 5 6
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)

83. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2
4 5
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }

84. 23
Cooperative Protocol
C1
C2
Group Coordinator (broker side)
C3
join-group
re-join
join-group
1 2
4 5
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: { }
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> { }
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }

85. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
1 2
4 5
C1
C2
C3

86. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
1 2
4 5
C1
C2
C3

87. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
1 2
4 5
C1
C2
C3

88. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
#assign(…)
1 2
4 5
C1
C2
C3

89. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}1 2
4 5
C1
C2
C3

90. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
sync-group
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}1 2
4 5
C1
C2
C3

91. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
sync-group
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}1 2
4 5
C1
C2
C3

92. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
sync-group
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsAssigned(3,6)
1 2
4 5
3 6
C1
C2
C3

93. 24
Cooperative Protocol
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2}
C2: {4, 5}
C3: { }
sync-group
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsAssigned(3,6)
1 2
4 5
3 6
C1
C2
C3

94. 25
Cooperative Rebalance
C1
C2
C3
join-group
re-
join-group
1 2 3
4 5 6
C1: {1, 2, 3} C2: {4, 5, 6}
sync-group
#assign(…)
C1: {1, 2, 3}
C2: {4, 5, 6}
C3: {3*, 6*} -> {}
#onPartitionsRevoked(3)
#onPartitionsRevoked(6)
join-group
C1: {1, 2} C2: {4, 5}
sync-group
#assign(…)
C1: {1, 2}
C2: {4, 5}
C3: {3, 6}
#onPartitionsAssigned(3,6)
1 2
4 5
3 6
Group Coordinator (broker side)
• Trade-off: more rebalances, but way cheaper
• Works better with a “sticky” assignor: fewer partitions to migrate
• Consumers can continue to fetch during a rebalance event (2.5+)

95. 26
Benchmark Results
• 10 streams instances rolling bounce, measuring process rate
• …and pause time: 3522 ms v.s. 37138 ms

96. 26
Benchmark Results
• 10 streams instances rolling bounce, measuring process rate
• …and pause time: 3522 ms v.s. 37138 ms

97. 26
Benchmark Results
• 10 streams instances rolling bounce, measuring process rate
• …and pause time: 3522 ms v.s. 37138 ms

98. 27
Augmented Assignor Interface
ConsumerPartitionAssignor
• #assign (subscription now includes “owned-partitions”)
• #supportedProtocols (eager and/or cooperative)
Built-in: {range, round-robin, sticky : eager};
{sticky-cooperative : cooperative}
Custom: {streams, … : eager and cooperative}

99. 28
Augmented Listener Interface
ConsumerRebalanceListener
• #onPartitionsRevoked (will not be triggered if there is nothing to revoke)
• #onPartitionsAssigned (triggered at completion of rebalance, regardless of newly added partitions)
• # #onPartitionsLost (triggered instead of onPartitionRevoked when a member falls out of group)

100. 29
Switch to Cooperative Rebalancing
In Consumer
• first rolling bounce: add “sticky-cooperative” / “my-cooperative” to [partition.assignment.strategy]
• second rolling bounce: remove old assignor (e.g.,“range”) from the config
In Streams
• first rolling bounce: set [upgrade.from = old version (“2.3”)]
• second rolling bounce: remove [upgrade.from] config

101. Take-aways
• We have extended the rebalance protocol to enable
smarter assignment (when, who, and what)
30

102. Take-aways
• We have extended the rebalance protocol to enable
smarter assignment (when, who, and what)
• No more stop-the-world rebalances with the incremental
cooperative protocol!
31

103. THANKS!
Guozhang Wang | guozhang@confluent.io | @guozhangwang
32
A. Sophie Blee-Goldman | sophie@confluent.io | @ableegoldman