At Current 2022, we introduced KIP-848: The Next Generation of the Consumer Rebalance Protocol. Almost a year after, its development is well under way! In this talk, we will dive into all aspects of the new protocol, look into the architecture of Apache Kafka's brand new group coordinator, discuss the upgrade path for both the consumers and the brokers, and finally update the community about where we stand in the development.

1. Apache Kafka's Next-Gen Rebalance Protocol
Towards More Stable and Scalable Consumer Groups
David Jacot
Apache Kafka PMC
Staff Software Engineer II, Confluent

2. Consumer Group 101
01
New Consumer Rebalance Protocol
02
New Group Coordinator
03
New Consumer
04
Conclusion
05
2
Agenda

3. Consumer Group 101
3

4. 4
Broker 2
bar-2
bar-3
Broker 1
bar-0
bar-1
Consumer Group
(group.id = foo)
Consumer A
Consumer B
Consumer C

5. 5
Broker 2
bar-2
bar-3
Broker 1
bar-0
bar-1
Consumer Group
(group.id = foo)
Consumer A
Consumer B
Consumer C

6. 6
Broker 2
bar-2
bar-3
Broker 1
bar-0
bar-1
Consumer Group
(group.id = foo)
Consumer A
Consumer B
Consumer C
��

7. 7
Broker 2
bar-2
bar-3
Broker 1
bar-0
bar-1
Consumer Group
(group.id = foo)
Consumer A
Consumer B
Consumer C
��

8. New Consumer Rebalance Protocol
8

9. Motivation for a new Group Rebalance Protocol
9
2
1 Remove group wide synchronization barrier
Simplify the consumer

10. Let’s dive into the new protocol by
walking through an example.
10

11. 11
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

12. ConsumerGroupHeartbeat API
Liveness, Subscriptions & Assignment
12

13. 13
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
Heartbeat Req (A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

14. 14
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
Heartbeat Req (A, 5, -)
Heartbeat Rsp (A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

15. 15
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
(B, 5, -)
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

16. 16
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
(B, 5, -)
(B, 5, -)
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

17. 17
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 0
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

18. 18
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 0
(-, 0, [])
P1 P2 P3
5 P1 P2 P3
5
P4 P5 P6
5
P4 P5 P6
5

19. 19
A
B
C
P1 P2 P3
P4 P5 P6
Target Assignment (5)
P
P
P
Subscriptions
P1 P2
P4 P5
P3 P6
Target Assignment (6)

20. 20
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 0
(-, 0, [])
P1 P2 P3
5 P1 P2
6
P4 P5
6
P4 P5 P6
5
P3 P6
6

21. 21
Transition to
Next Epoch
2
Revoke
Partitions
1
Assign
Partitions
3

22. 22
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 0
(-, 0, [])
P1 P2 P3
5 P1 P2
6
P4 P5
6
P4 P5 P6
5
P3 P6
6
6

23. 23
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 0
(-, 0, [])
P1 P2 P3
5 P1 P2
6
P4 P5
6
P4 P5 P6
5
P3 P6
6
6
(C, 6, [])

24. 24
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
(-, 0, [])
P1 P2 P3
5 P1 P2
6
P4 P5
6
P4 P5 P6
5
P3 P6
6
6
(C, 6, [])

25. 25
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
(A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6

26. 26
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
(A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6

27. 27
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
(A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(A, 5, [P1, P2])

28. 28
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
(A, 5, -)
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(A, 5, [P1, P2])

29. 29
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)

30. 30
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)

31. 31
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)
(B, 5, [P4, P5])

32. 32
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)
(B, 5, [P4, P5])

33. 33
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)

34. 34
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)
(C, 6, -)

35. 35
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(A, 5, -)

36. 36
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(A, 5, -)
(A, 5, -)

37. 37
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)

38. 38
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, -)
(B, 5, -)

39. 39
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)

40. 40
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)
(C, 6, -)

41. 41
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5 P6
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6

42. 42
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6

43. 43
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
5
C 6
P1 P2 P3
5
P4 P5 P6
5
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, [P4, P5])

44. 44
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
5
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, [P4, P5])

45. 45
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
5
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, [P4, P5])
(B, 6, [P4, P5])

46. 46
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(B, 5, [P4, P5])
(B, 6, [P4, P5])

47. 47
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)

48. 48
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)
P6

49. 49
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)
(C, 6, [P6])
P6

50. 50
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
(C, 6, -)
(C, 6, [P6])
P6
P6

51. 51
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P1 P2 P3
5
P4 P5
6
6
P1 P2
6
P4 P5
6
P3 P6
6
P6
P6
💥

52. 52
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P4 P5
6
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2

53. 53
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P4 P5
6
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
(B, 6, -)

54. 54
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P4 P5
7
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
(B, 6, -)
P1

55. 55
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
6
C 6
P4 P5
7
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
(B, 6, -)
P1
(B, 7, [P4, P5, P1])

56. 56
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
6
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
(B, 6, -)
P1
(B, 7, [P4, P5, P1])
P1

57. 57
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
7
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1

58. 58
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
7
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 6, [P6])

59. 59
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
7
6
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 6, [P6])

60. 60
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 6, [P6])
P3 P2

61. 61
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 6
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 6, [P6])
P3 P2
(C, 7, [P2, P3, P6])

62. 62
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 6, [P6])
P3 P2
(C, 7, [P3, P2])
P3 P2

63. 63
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2

64. 64
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2
(B, 7, [P4, P5, P1])

65. 65
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2
(B, 7, [P4, P5, P1])

66. 66
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2
(B, 7, [P4, P5, P1])
(B, 7, -)

67. 67
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2

68. 68
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2

69. 69
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 7, [P6, P3, P2])
P3 P2
P3 P2

70. 70
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 7, [P6, P3, P2])
P3 P2
P3 P2

71. 71
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 7, [P6, P3, P2])
P3 P2
(C, 7, -)
P3 P2

72. 72
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2
(B, 7, -)

73. 73
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
P3 P2
P3 P2
(B, 7, -)
(B, 7, -)

74. 74
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 7, -)
P3 P2
P3 P2

75. 75
Consumers Group Coordinator
Target Assignment
Current Assignment
A
B
P1 P2 P3
5
P4 P5
7
C 7
P4 P5
7
7
P4 P5
7
P3 P6
7
P6
P6
💥
P1
P2
P1
P1
(C, 7, -)
P3 P2
(C, 7, -)
P3 P2

76. New Group Coordinator
76

77. Motivation for a new Group Coordinator
77
2
1 State Management
Concurrency / Threading Model
3 Scala to Java

78. New Group Coordinator Architecture (I)
78
Group Coordinator
__consumer_offsets-0
Request
Handlers
Shard #0
Shard #1
Shard #N
Shard #2
Event
Processors
__consumer_offsets-1
__consumer_offsets-2
__consumer_offsets-N

79. Shard
New Group Coordinator Architecture (II)
79
0 1 2 3 4 5 6 7 8 9
Replicated
State Machine
Request
Response
Records
Purgatory
6 Resp
7 Resp
9 Resp
Business
Logic
Response 6
Response 7

80. New Configurations
● group.coordinator.threads (default: 1)
● group.consumer.assignors (default: Uniform, Range)
● group.consumer.session.timeout.ms (default: 45s)
● group.consumer.heartbeat.interval.ms (default: 5s)
80

81. Server Side Assignors
81
Your Own
● Pluggable
● PartitionAssignor
interface
Range
1 1 2 2
5 5
3 3 4 4
6 6
A
B
C
D
Uniform
1 3
5
5
3 1
2 4
6
4
4 2
A
B
C
D 6

82. Upgrade Path
● The new Group Coordinator is fully backward compatible.
● The new Group Coordinator will become the default in the
future release of Apache Kafka (version TBD).
82

83. New Consumer
83

84. Motivation for a new Consumer
84
1 Threading Model (Heartbeat Thread)

85. Background Thread
New Consumer Architecture
85
Application Thread
Consumer
Network
Client
State
Event
Loop

86. New Configurations
● group.protocol (generic, consumer or auto)
○ generic - old consumer group protocol and threading model
○ consumer - new consumer group protocol and threading model
● group.local.assignors (default: null)
○ null means that the server uses the default one
86

87. New Wildcard Subscriptions
Consumer#subscribe(Pattern pattern)
Consumer#subscribe(SubscriptionPattern pattern)
87

88. Live Consumer Upgrade, How?
88
Group Coordinator
Group Foo
Consumer A
(group.protocol=generic)
Consumer B
(group.protocol=generic)
Consumer C
(group.protocol=generic)
Old Protocol
Old Protocol
Old Protocol

89. New Protocol
Live Consumer Upgrade, How?
89
Group Coordinator
Consumer A
(group.protocol=consumer)
Consumer B
(group.protocol=generic)
Consumer C
(group.protocol=generic)
Old Protocol
Old Protocol
Group Foo

90. New Protocol
Live Consumer Upgrade, How?
90
Group Coordinator
Consumer A
(group.protocol=consumer)
Consumer B
(group.protocol=generic)
Consumer C
(group.protocol=generic)
Old Protocol
Old Protocol
Group Foo

91. New Protocol
New Protocol
Live Consumer Upgrade, How?
91
Group Coordinator
Group Foo
Consumer A
(group.protocol=consumer)
Consumer B
(group.protocol=consumer)
Consumer C
(group.protocol=generic)
Old Protocol

92. New Protocol
New Protocol
New Protocol
Live Consumer Upgrade, How?
92
Group Coordinator
Group Foo
Consumer A
(group.protocol=consumer)
Consumer B
(group.protocol=consumer)
Consumer C
(group.protocol=consumer)

93. Conclusion
93

94. Where are we?
● ~120 pull requests merged so far!
● The new Group Coordinator is almost fully implemented.
● The new Consumer’s threading model is being merged to trunk.
We are working on implementing the new consumer group
protocol.
● Our aim is to release a preview of both soon!
94

95. References
● https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next
+Generation+of+the+Consumer+Rebalance+Protocol
● https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+
refactor+project+overview
95

96. Takeaways
96
2
1
3
Incremental protocol for and more scalable and stable consumer
group
Simpler and thinner consumer
Online consumer group upgrade

97. Thanks!
97
David Jacot
@davidjacot
dajac@apache.org