1. The Next Generation of the
Consumer Rebalance Protocol
David Jacot

2. Agenda
● Consumer Groups
● Current Rebalance Protocol and its Pitfalls
● Next-Gen Consumer Rebalance Protocol
● Upgrade Path
● Takeaways
2

3. Consumer Groups
3

4. Kafka Cluster
What do we need them for?
4
Consumer Group
(group.id = foo)
Foo 1
Foo 2
Foo 3
Foo 4
Consumer A
Consumer B
Consumer C

5. Kafka Cluster
What do we need them for?
5
Consumer Group
(group.id = foo)
Foo 1
Foo 2
Foo 3
Foo 4
Consumer A
Consumer B
Consumer C

6. Kafka Cluster
What do we need them for?
6
Consumer Group
(group.id = foo)
Foo 1
Foo 2
Foo 3
Foo 4
Consumer A
Consumer B
Consumer C
Consumer D

7. Kafka Cluster
What do we need them for?
7
Consumer Group
(group.id = foo)
Foo 1
Foo 2
Foo 3
Foo 4
Consumer A
Consumer B
Consumer C
Consumer D
��

8. Current Rebalance Protocol
and its Pitfalls
8

9. Rebalance Protocol: A Layered Protocol
9
Broker
Rebalance Protocol
Consumer
Consumer Protocol
Connect
Connect Protocol
Custom
Protocol
Membership
Assignment
Custom
Metadata
Custom
Assignor
Streams
Metadata

10. Pitfall #1
Layered
Protocol
● Fat Client
● Interoperability
● Visibility (on the server side)

11. Eager Rebalance Protocol (e.g. range)
11
Member A
Member B
Group
Coordinator
P1 P2 P3
P4 P5 P6
H
e
a
r
t
b
e
a
t
H
e
a
r
t
b
e
a
t
H
e
a
r
t
b
e
a
t
H
e
a
r
t
b
e
a
t

12. Pitfall #2
Session
Timeout
● Session timeout and interval are defined
on the client side
● KIP-735: Increase default consumer
session timeout

13. Eager Rebalance Protocol (e.g. range)
13
A
B
Group
Coordinator
P1 P2 P3
H
e
a
r
t
b
e
a
t
H
e
a
r
t
b
e
a
t
C joins
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
P4 P5 P6
A revokes its partitions
and commits offsets
B revokes its partitions
and commits offsets
The coordinator tells A
to rejoin

14. Eager Rebalance Protocol (e.g. range)
14
A
B
Group
Coordinator
C
J
o
i
n
G
r
o
u
p
P1 P2
P3
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
P4 P5
P6
A is elected as leader and
computes the assignment
Members resume
processing

15. Pitfall #3
Group Wide
Synchronization
Barrier
● Stop-the-World
● Impact of faulty members

16. Cooperative Rebalance Protocol (e.g. cooperative-sticky)
16
A
B
Group
Coordinator
P1 P2 P3
H
e
a
r
t
b
e
a
t
H
e
a
r
t
b
e
a
t
C joins
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
P4 P5 P6
A keeps its partitions
B keeps it partitions

17. Cooperative Rebalance Protocol (e.g. cooperative-sticky)
17
A
B
Group
Coordinator
C
J
o
i
n
G
r
o
u
p
P1 P2
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
P4 P5
A is elected as leader and
computes the assignment
A and B revokes
P3 and P4

18. Cooperative Rebalance Protocol (e.g. cooperative-sticky)
18
A
B
Group
Coordinator
C
P1 P2
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
S
y
n
c
G
r
o
u
p
P4 P5
A is elected as leader and
computes the assignment
A and B revokes
P3 and P4
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
J
o
i
n
G
r
o
u
p
P1 P2
P4 P5
P3 P6
H
e
a
r
t
b
e
a
t

19. Pitfall #4
Incremental
but…
● Offset Commits and Heartbeats are
blocked by the rebalance
● Multiple (possibly slow) rebalances
required to converge; a slow members
hold back the entire group
● Increased complexity on the client side
● Doesn’t scale well to 1000s of members

20. Next-Gen Consumer Rebalance Protocol
20

21. Design Goals
21
Client Side Server Side
Generic Protocol Consumer Protocol
Stop the World Incremental

22. Two Protocols
22
Broker
Consumer Group Protocol
Consumer
Core Protocol
(server side assignors)
Broker
Consumer Group Protocol
Consumer
KStreams
Custom
Assignor
Metadata
Extended Protocol
(client side assignors)

23. Core Protocol
(server side assignor)
23

24. (ConsumerGroup)
Heartbeat API
24
Membership / Liveness
Assignment
1
2

25. Request
Heartbeat API
25
- Group Id
- Member Id
- Member Epoch
- Instance Id
- Rack Id
- Rebalance Timeout
- Server Assignors
- Subscribed Topics
- Subscribed Regex
- Owned Partitions
- Error Code
- Member Id
- Member Epoch
- Heartbeat Interval
- Assignment
Response

26. Session Timeout
● A member is kicked out from the group
if it does not heartbeat within the
group.consumer.session.timeout.ms.
● group.consumer.session.timeout.ms
is defined on the server side.
● It can be dynamically defined per group.

27. Wildcard
Subscriptions
● Wildcard subscriptions are managed by
the group coordinator
● Google RE2/J

28. 28
Declarative
Assignment
Coordinator reconciles the
group from the current
state to the desired state
2
Assignor defines the
desired state
1

29. Declarative Assignment
A
B
A
B
C
B
C
Assign. 1
💥
C joins the group.
A and B join the
group.
A fails
foo-1
foo-2
foo-3
foo-4
foo-5
foo-6
Assign. 2
foo-1
foo-2
foo-4
foo-5
foo-3
foo-6
Assign. 3
foo-1
foo-4
foo-5
foo-2
foo-3
foo-6

30. Declarative Assignment
The assignors are on the server-side:
● group.consumer.assignors = [uniform, range]
● The first one is the default assignor used.
Kafka will ship with two implementations:
● Uniform: Partitions are assigned uniformly in the group (round-robin)
● Range: Partitions are co-partitioned
Customer assignors can be implemented by implementing the
PartitionAssignor interface.
30

31. 31
Reconciliation Members can converge
independently
Coordinator resolves
dependencies between the
members
2
3
Coordinator drives
members to their next
desired assignment
1

32. Reconciliation has Three Phases
32
Revoke
Partitions
1
Transition to
Next Epoch
2
Assign
Partitions
3

33. Revocation
Timeout
● The consumer has up to the revocation
timeout to revoke partitions.
● It is kicked out from the group
otherwise.
● The revocation timeout is defined by the
max.poll.interval.ms.

34. Example
34

35. 35
P1 P2 P3
1
P4 P5 P6
1
P1 P2 P3
1
P4 P5 P6
1
P1 P2 P3
1
P4 P5 P6
1
Member A Group Coordinator
Member B
(1,-)
(1,-)
(1,-)
(1,-)
0
Member C
(0,-)
(2,-)
P1 P2
2
P4 P5
2
P3 P6
2
P3 P6
2
2
Desired
Assignment
Members
State

36. 36
P1 P2 P3
1
P4 P5 P6
1
Member A Group Coordinator
Member B
(1,-)
(1,[p1,p2])
(1,-)
(1,[p4,p5])
2
Member C
P1 P2 P3
1
P4 P5 P6
1
P1 P2
2
P4 P5
2
P3 P6
2
P3 P6
2
P1 P2 P3
1
P1 P2 P3
1
P4 P5 P6
1
P4 P5 P6
1

37. 37
P1 P2 P3
1
P4 P5 P6
1
Member A Group Coordinator
Member B
(1,-)
(1,-)
(1,-)
(1,-)
2
Member C
P1 P2 P3
1
P4 P5 P6
1
P1 P2
2
P4 P5
2
P3 P6
2
P3 P6
2
(2,-)
(2,-)

38. 38
P1 P2 P3
1
P4 P5 P6
1
Member A Group Coordinator
Member B
(1,[p4,p5])
(2,[p4,p5])
2
Member C
P1 P2 P3
1
P4 P5 P6
1
P1 P2
2
P4 P5
2
P3 P6
2
P3 P6
2
(2,-)
(2,[p6])
P4 P5
1
P4 P5
2
P4 P5
2
P3 P6
2
2
P6

39. 39
P1 P2 P3
1
P4 P5
2
Member A Group Coordinator
Member B
2
Member C
P1 P2 P3
1
P4 P5
2
P1 P2
2
P4 P5
2
P3 P6
2
P3 P6
2
(2,-)
(2,[p3,p6])
P6
(1,[p1,p2])
(2,[p1,p2])
P1 P2
1
P1 P2
2
P1 P2
2
P3 P6
2
2
P6
P3

40. Extended Protocol
(client side assignor)
40

41. 41
Declarative
Client-Side
Assignment
Coordinator reconciles the
group from the current
state to the desired state
2
Assignor defines the
desired state
1
Coordinator delegates the
assignment to a member
3

42. Request
Heartbeat API
42
- Group Id
- Member Id
- Member Epoch
- Instance Id
- Rack Id
- Rebalance Timeout
- Client Assignors
- Subscribed Topics
- Subscribed Regex
- Owned Partitions
- Error Code
- Member Id
- Member Epoch
- Heartbeat Interval
- Should Compute Assignment
- Assignment
- Error
- Version
- Metadata
- Partitions
Response

43. Request
PrepareAssignment API
43
- Group Id
- Member Id
- Member Epoch
- Error Code
- Group Epoch
- Assignor Name
- Members
- Member Id
- Member Epoch
- Instance Id
- Rack Id
- Subscribed Topics
- Assignor Metadata
- Assigned Partitions
Response

44. Request
InstallAssignment API
44
- Group Id
- Member Id
- Member Epoch
- Group Epoch
- Error
- Members
- Member Id
- Assigned Partitions
- Metadata
- Error Code
Response

45. 45
Member A Coordinator
Heartbeat(0,[])
Heartbeat(0,[],true)
P1 P2
1
P1 P2
1
P1 P2
1
Member B
0
PrepareAssignment(0)
2 2
Undef Undef
Assignment
timeout
starts

46. 46
eAssignment(0)
PrepareAssignment(members)
InstallAssignment(members)
InstallAssignment()
2 2
P1 P2

47. 47
InstallAss
Heartbeat(0,[])
Heartbeat(2,[p2])
P2
2
Heartbeat(1,-)
Heartbeat(2,[p1]) P1 P2
1

48. Upgrade Path
48

49. How to upgrade brokers? (KRaft only)
49
Upgrade brokers to the
required version
1
Bump metadata.version
to enable the new protocol
2

50. New Broker’s Configurations
● group.coordinator.threads (default: 1)
● group.consumer.assignors (default: Uniform, Range)
● group.consumer.session.timeout.ms (default: 45s)
● group.consumer.heartbeat.interval.ms (default: 5s)
50

51. How to upgrade consumers? Manually
51
Upgrade consumers with
group.protocol=generic
1
Roll consumers with
group.protocol=consumer
2

52. How to upgrade consumers? Automatically
52
Upgrade consumers with
group.protocol=auto
1

53. Live Protocol Upgrade, How?
53
Group Coordinator
Group Foo
Consumer A
Consumer B
Consumer C
Old Protocol
Old Protocol
Old Protocol

54. New Protocol
Live Protocol Upgrade, How?
54
Group Coordinator
Consumer A
Consumer B
Consumer C
Old Protocol
Old Protocol
Group Foo
Check if Foo can be
upgraded

55. Upgrade
Requirements
● All members have the minimum
required protocol version
● All members use compatible regex
based subscriptions
● All members use compatible client
assignors if client side assignors are
used

56. New Protocol
Live Protocol Upgrade, How?
56
Group Coordinator
Group Foo
(Assignment)
Consumer A
Consumer B
Consumer C
Old Protocol
Old Protocol
Old Protocol is proxied to
the new protocol.
Foo is upgraded to the
new protocol

57. New Protocol
New Protocol
Live Protocol Upgrade, How?
57
Group Coordinator
Group Foo
(Assignment)
Consumer A
Consumer B
Consumer C
Old Protocol

58. New Protocol
New Protocol
New Protocol
Live Protocol Upgrade, How?
58
Group Coordinator
Group Foo
(Assignment)
Consumer A
Consumer B
Consumer C

59. New Consumer’s Configurations
● group.protocol (generic, consumer or auto)
● group.remote.assignors (default: null)
● group.local.assignors (default: null)
59

60. New Consumer’s Apis
Move from java.util.Regex to SubscriptionPattern:
Consumer#subscribe(
SubscriptionPattern pattern,
ConsumerRebalanceListener callback
)
Move to new assignor interface:
org.apache.kafka.clients.consumer.PartitionAssignor
60

61. Takeaways
61

62. Protocol tailored to the Consumer.
1
Incremental protocol that improves the stability
and the scalability of consumers groups.
2
Server side assignors by default; client side
assignors for power users.
3
Live protocol upgrade.
4

63. 63
Thank you!
David Jacot
@davidjacot
dajac@apache.org