The Next Generation of the Consumer Rebalance Protocol With David J...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

The Next Generation of the
Consumer Rebalance Protocol
in Apache Kafka
David Jacot (@davidjacot)
Apache Kafka PMC
Staff So...

Agenda
● Current Consumer Rebalance Protocol
● Shortcomings
● Next-Gen Consumer Rebalance Protocol
○ Core Protocol
○ Exten...

Current Consumer Rebalance Protocol
3

Consumer Group (group.id = foo)
What is the Rebalance Protocol used for?
4
Partition 1 Partition 2 Partition 3 Partition 4...

Current Consumer Rebalance Protocol
5
JoinGoup, SyncGroup, Heartbeat APIs
Group Coordinator
Consumer A
Membership
Resource...

A Rebalance is a Synchronization Barrier
6
Member
A
Member
B
Member
A
Member
B
Member
C
Member
B
Member
C
Member
B
Member
...

Rebalance’s 2-Phases: JoinGoup & SyncGroup
7
Member A
Member B
Group Coordinator
J
o
i
n
G
r
o
u
p
(
m
e
t
a
d
a
t
a
)
J
o...

Rebalance’s 2-Phases: JoinGoup & SyncGroup
8
Member A
Member B
Group Coordinator
J
o
i
n
G
r
o
u
p
(
m
e
t
a
d
a
t
a
)
J
o...

Rebalance’s 2-Phases: JoinGoup & SyncGroup
9
Member A
Member B
Group Coordinator
J
o
i
n
G
r
o
u
p
(
m
e
t
a
d
a
t
a
)
J
o...

Rebalance’s 2-Phases: JoinGoup & SyncGroup
10
Member A
Member B
Group Coordinator
J
o
i
n
G
r
o
u
p
(
m
e
t
a
d
a
t
a
)
J
...

Failure Detection - Liveness
11
Member A
Member B
Group Coordinator
Session Timeout*
Heartbeat Interval*
Heartbeat
Request...

Eager Assignment (e.g. range)
12
Member A
Member B
Group Coordinator
Member C joins
P1 P2 P3
P4 P5 P6
P1 P2
P3
P4 P5
P6
As...

Cooperative Assignment (e.g. cooperative-sticky)
13
Member A
Member B
Group Coordinator
Member C joins
P1 P2 P3
P4 P5 P6
J...

Cooperative Assignment (e.g. cooperative-sticky)
14
Member A
Member B
Group Coordinator
Member C joins
P1 P2 P3
P4 P5 P6
P...

Shortcomings
15

Group-Wide Synchronization Barrier
● “Stop the World” is a fundamental issue.
● Cooperative assignors improved it but Cons...

With Great Power Comes Great Responsibility
● Consumer drives key conﬁgurations (e.g. session.timeout.ms).
● Client side a...

Embedded Consumer Protocol
● Interoperability between consumer libraries, toolings, etc. relies on a
correct deﬁnition of ...

Next-Gen Consumer Rebalance Protocol
The Core Protocol
19

Consumer A
Empower the Group Coordinator, and
Simplify the Consumer
20
Group Coordinator
Declarative
Assignment
(Subscript...

Declarative Assignment
21
Member
A
Member
B
Member
A
Member
B
Member
C
Member
B
Member
C
Member
B
Member
C
Assignment
1
As...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignment
Group Coordinato...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process
Member A
Member B
Member C
P1 P2
P4 P5
P1 P2
P4 P5
P3 P6
Member A
Member B
Member C
Assignment
Grou...

Reconciliation Process’ Key Principles
● Members are not globally synchronized anymore.
● The group coordinator reconciles...

Declarative Server-Side Assignor
● The assignor is declarative! It tells the consumer what the end state
should be.
● Rang...

Rebalance Timeout (or Revocation Timeout)
● When revoking partitions, the group coordinator gives the
rebalance timeout to...

Session Timeout
Member A
Member B
Member C
Group Coordinator
Member D
Session Timeout
Heartbeat Interval

Dynamic Group Configurations
● All configurations are defined on the server side (except the
rebalance timeout):
● They are d...

Wildcard Subscriptions & Metadata Monitoring
● Wildcard subscriptions are managed centrally by the group
coordinator.
● Th...

Topic Identiﬁers (KIP-516)
● Fetch & Metadata APIs use Topic IDs since KIP-516.
● Current Rebalance Protocol, OffsetCommit...

Next-Gen Consumer Rebalance Protocol
The Extended Protocol
48

PrepareAssignment API
InstallAssignment API
Heartbeat API
Declarative Client-Side Assignor
49
Group Coordinator
Consumer A...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Client-Side Assignment Delegation Process
Member A
Member B
P1 P2 P3
P4 P5 P6
P1 P2 P3
P4 P5 P6
Member A
Member B
Assignme...

Declarative Client-Side Assignor
● Assignors are still declarative!
● The assignment is delegated to one member of the gro...

Next-Gen Consumer Rebalance Protocol
Upgrade Path
60

How to upgrade to the new protocol?
1. Upgrade the cluster to a software version that supports the new
protocol.
2. Roll t...

Current Protocol Next-Gen Protocol
Live Consumer Group Upgrade, How?
62
ConsumerGroupHeartbeat API
● Updates Subscriptions...

Live Consumer Group Upgrade, How?
63
Group Coordinator
Group Foo
Consumer A
Consumer B
Consumer C
Old Protocol
Old Protoco...

New Protocol
Live Consumer Group Upgrade, How?
64
Group Coordinator
Group Foo
(Assignment)
Consumer A
Consumer B
Consumer ...

New Protocol
New Protocol
Live Consumer Group Upgrade, How?
65
Group Coordinator
Group Foo
(Assignment)
Consumer A
Consume...

New Protocol
New Protocol
New Protocol
Live Consumer Group Upgrade, How?
66
Group Coordinator
Group Foo
(Assignment)
Consu...

Takeaways
67

Takeaways
● The new rebalance protocol is tailored to the Consumer.
● The group-wide synchronization barrier is gone.
● Th...

KIP-848: The Next Generation of the
Consumer Rebalance Protocol
69
Thank you!
David Jacot
@davidjacot
dajac@apache.org

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 69 Ad

1 of 69 Ad