SlideShare a Scribd company logo

Server Side Assignors for Consumer Groups : KIP-848

•
•
HostedbyConfluent
HostedbyConfluent

"In this talk, we delve into a specific detail of the latest evolution of Kafka's consumer group protocol as introduced in KIP-848 - Server-Side Assignors. This transformation not only streamlines client operations but also introduces a new dimension of efficiency and flexibility in message processing. Before exploring the features of server-side assignors, it's essential to understand the fundamental concept of Kafka consumer groups. In Kafka, a consumer group is a collection of consumers working together to process data from one or more topics. The assignors are responsible for dividing the partitions of topics subscribed to by the group - among the consumers. The server side assignor is pluggable and the client can choose the one that it wants to use by providing its name in the heartbeat request. There are two types of assignors configured in the coordinator: Range and Uniform. The Range Assignor, known for its ability to distribute partitions across consumers evenly, ensures that each consumer handles at least one partition per topic, making it an ideal choice for scenarios with fluctuating topic demands. Conversely, the Uniform Assignor, which comes in two flavors – Optimized and General – adopts a more intelligent approach, selecting the most suitable assignment strategy based on the group's subscription pattern. By the end of this session, attendees will gain a deeper understanding of Kafka's new server-side assignors, equipped with the knowledge to make informed decisions for optimizing their consumer group configurations."

1 of 92
Download to read offline
Server Side Assignors : KIP 848 Ritika Reddy Software Engineer
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples How do we know which assignor to use when? Any questions? 2 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 3 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 4 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 5 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 6 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
Introduction What are Consumer Groups? 7
Consumers in Kafka Consumer Topic 1 P0 P1 P2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Types of subscriptions M1 M2 M3 Consumer Group T1 T2 T3 Topics Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Types of subscriptions Homogenous Subscriptions: All members of the consumer group are subscribed to the same set of topics. M1 M2 M3 Consumer Group Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions T1 T2 T3 Topics
Types of subscriptions Heterogenous Subscriptions: Different members are subscribed to different topics. M1 M2 M3 Consumer Group Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions T1 T2 T3 Topics
Introduction Why are we changing the Current Protocol? 15
Drawbacks of Current Protocol Thick Client: Debugging and updates rely heavily on clients Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Global Synchronization Barrier: A single misbehaving consumer can disrupt the entire group
Introduction The New Consumer Group Protocol : KIP-848 17
Improvements in the new protocol: KIP-848 ConsumerGroupHeartbeat API Integrate partition assignments within heartbeats streamlining state management. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Complexity Shift from Client to Server Easier debugging and update adoption Incremental & Cooperative Rebalance Eliminates existing global sync barriers
The Current Protocol: A Layered Protocol The Next-Gen Protocol: Server Side Assignors Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
20 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
Assignors Roles & Properties 21
Role of an Assignor The assignment is not set in stone. Assign partitions to members when the group coordinator says so Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
How does the assignor work in the new protocol? Assignor Metadata Group Coordinator Target Assignment Input Output 1. Member Info 2. Current Assignment 3. Topic and Partition Info Rebalance Triggers 4. Assignor Changes 1. Member Joins/Leaves 2. Topic Metadata Changes 3. Broker rack changes Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Stickiness Minimizes partition movements between members by retaining as many partitions from the existing assignment. Rack Awareness Aligns partition replicas with members in the same rack to minimize cross-zone traffic and costs.
General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack Awareness Aligns partition replicas with members in the same rack to minimize cross-zone traffic and costs. Stickiness Minimize partition movements between members by retaining as many partitions from the existing assignment as possible.
General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Stickiness Minimizes partition movements between members by retaining as many partitions from the existing assignment. Rack Awareness Align partition replicas with members in the same rack to minimize cross-zone traffic and costs.
Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Topic Name = Gold Partition = Coin Members of Consumer Group Analogy for Rack Matching
28 Gold Coins 1-10 Princess 1 Rack 1 Rack 2 Gold Coins 10-20 Princess 2 Availability Zones (AZ) Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Priority Order of Properties 29 Enhances group efficiency and reduces latency Balance Optimizes processing speed and reduces cross-AZ costs Rack Awareness Minimizes consumer downtime and operational impact Stickiness Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
30 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
Types of Assignors What are the different types of assignors? 31
Types of Assignors 32 Range Assignor Uniform Assignor Assigns contiguous partition ranges to consumers. Ensures ordered processing within topics for balanced workload distribution. Distributes topic partitions among group members in a balanced fashion. Two implementations: ★ Optimized ★ General Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Types of Assignors The Range Assignor 33
M0 M1 M2 M3 TOPIC 1 Consumer Group 1 Range Assignor Partition Distribution: • Each subscribed member receives at least one partition from that topic. P0 P1 P2 * Except when number of members outnumber partitions. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Partition Co-Allocation: • Each member receives the same partition number from every subscribed topic. M0 M1 M2 M3 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Range Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Per-topic co-allocation for joins: 36 Source: medium.com Credit: Gavin Fong Range Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
37 Map members per Topic Calculate quotas Assign sticky partitions Assign the leftover partitions using ranges *Rack Awareness is not yet implemented for the Range Assignor Range Assignor Algorithm Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Topic 1 Topic 2 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input Range Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Conditions: 1. Each princess gets an equal share of the gold. 2. Once split equally, the extra coins will also be split uniformly starting from the eldest ones first.. 3. Each princess can only get one extra gold coin. Partition = Coin Members of Consumer Group Topic Name = Gold Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Min Quota : Minimum partitions that each member should get Num of partitions (Coins) = 3 Number of members subscribed (Princesses) = 2 3 / 2 = 1 Each princess gets at least 1 coin Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Number of members with extra partitions Num of partitions (Coins) = 3 Number of members subscribed (Princesses) = 2 1 extra coin given to the eldest first Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions 3 % 2 = 1
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 ➢ Partitions are sorted in ascending order ➢ Members are implicitly sorted in the same order per topic Topic 1 Consumer Group Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 extra Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 Topic 2 -> same process as topic 1 extra Topic Name = Silver Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Map Members Assign Leftover Partitions Range Assignor Example 1 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Range Assignor Example 1 - Trigger rebalance by adding a new member Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions M3
Topic 1 Topic 2 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input M3 NOTE: Subscriptions of members can be different Joins won’t be feasible in this case Range Assignor Example 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Number of members subscribed = 3 Number of partitions = 3 Min Quota = numPartitionsForTopic / number of subscribed members Extra Partitions = numPartitionsForTopic % number of subscribed members Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2, M3 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 3 / 3 = 1 Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions 3 % 3 = 0
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 Prev Assignment Ref M1 -> T1P0, T1P1 T2P0, T2P1 M2 -> T1P2, T2P2 T1P0 T1P1 T1P2 M1 M2 Available for assignment Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Assign Leftover Partitions Assign Sticky Partitions Calculate Quotas Quotas Min Quota = 1 Members with Extras = 0
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 M3 T1P1 Unassigned Partitions Map Members Assign Leftover Partitions Calculate Quotas Assign Sticky Partitions Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 Prev Assignment Ref M1 -> T1P0, T1P1 T2P0, T2P1 M2 -> T1P2, T2P2 T2P0 T2P1 T2P2 M1 M2 Quotas Min Quota = 1 Members with Extras = 1 extra Range Assignor Example 2 - Topic 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Assign Leftover Partitions Assign Sticky Partitions Calculate Quotas
M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 M3 1 revocation 1 allocation Range Assignor Example 2 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Highly unbalanced distribution is possible since every extra partition always goes to the first few members. 54 M1 M2 T1P0 T1P1 T2P0 T2P1 T3P0 T3P1 T1P2 T2P2 T3P2 extras M1 has twice as many partitions as M2 Range Assignor Drawbacks Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Types of Assignors The Uniform Assignor 55
The uniform assignor is implemented in two different ways: ● Optimized Assignment Builder ● General Assignment Builder The right implementation is automatically chosen based on the topic subscriptions. This is abstracted away from the user. Uniform General Optimized Homogeneous Topic Subscriptions Heterogeneous Topic Subscriptions Uniform Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
❏ Homogenous Subscriptions: ❏ All partitions have the same weight. ❏ Rebalance times can be improved with quotas. ❏ Heterogenous Subscriptions: ❏ More complexity involved to achieve a balanced assignment. ❏ Less frequent. Why do we need two implementations? Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
58 Calculate Quotas Assign Sticky Partitions Rack Aware Round Robin Assignment* Unassigned Partitions Round Robin *Rack aware matching is possible only when both member and broker racks are configured Optimized Uniform Assignment Builder Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Topic 2 Topic 1 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input Rack 1 Rack 2 P0 P1 P2 Rack 0, Rack 1 Rack 1, Rack 2 Rack 2, Rack 3 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Replica Racks
Min Quota = total partitions / total number of members Num Members with Extra Partitions = total partitions % total number of members Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Subscriptions - M1 -> T1, T2 M2 -> T1, T2 (3 * 2) / 2 = 3 (3 * 2) % 3 = 0 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
M1 Rack 1 Rack 2 M2 Members Racks M1 -> rack1 M2 -> rack2 Partition Racks P0 = [rack0, rack1] P1 = [rack1, rack2] P2 = [rack3, rack2] P0 P1 P2 P1 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
M1 Rack 1 M2 Rack 2 Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1) T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Sort partitions in ascending order of number of members in the same rack Sorting is done to maximize rack matching Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
M1 M2 Rack 1 Rack 2 T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Quota is already met No other member in the same rack Later partition will go to M2 with no rack matching Quotas Ref MinQuota = 3 Life Without Sorting Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1)
M1 M2 Rack 1 Rack 2 Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1) T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Quotas Ref MinQuota = 2 Members with extra Partition = 0 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
Consumer Group 1 M1 M2 T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Optimized Uniform Assignor Example 1 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
66 Assign Sticky Partitions Rack Aware Round Robin Unassigned Round Robin Balance Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions General Uniform Assignment Builder
Key Differences from prev assignors: ❏ No concept of Quotas ❏ No good way to pre-determine quotas ❏ Tried a greedy algorithm and failed at a few edge cases ❏ Sorting members and topics based on various criteria ❏ Rebalance Loops ❏ Iterate over the assignment until a balance is reached General Uniform Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
Topic 2 P2 P0 P1 M1 Metadata Input Rack 0 Rack 1 Topic 3 P2 P1 P0 M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 Current Assignment M2 Rack 2 M3 P3 P3 P2 P1 P0 Topic 1 P3 P4 P5 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions REBALANCE TRIGGER: ASSIGNOR CHANGE
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1 T1P2 M1
M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 T1 -> P0 M2 T1 -> T2 -> P0 M3 T1 -> P5 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T1P1 T1P2 T1P3 T1P4 T2P3 T3P0 T3P3 M1 M1 M2 M3 M3 M3 M3 If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future General Uniform Assignor Example 1
M1 P3 P0 P4 M2 P1 P0 P4 M3 P1 P2 P5 P5 RACK 0 RACK 1 RACK 2 Assign partition if member in matching rack General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Unassigned Round Robin Balance Assign Sticky Partitions Rack aware round robin
M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T1P3 T1P2 T3P3 T2P3 T1P4 T3P0 T1P1 If partition is in the same rack as member: Assign SORTED UNASSIGNED PARTITIONS General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Unassigned Round Robin Balance Assign Sticky Partitions Rack aware round robin
M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T3P3 T2P3 T3P0 UNASSIGNED PARTITIONS General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin SORT SUBSCRIBED MEMBERS IN ASCENDING ORDER BY ASSIGNMENT SIZE AND ASSIGN FOR BALANCE
P3 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin T3P3 T2P3 T3P0 TARGET ASSIGNMENT UNASSIGNED PARTITIONS Subscriptions - T1 -> M1, M2, M3 T2 -> M2, M3 T3 -> M3 M3 M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2
M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3 P3 SORT SUBSCRIBED MEMBERS IN ASCENDING ORDER BY ASSIGNMENT SIZE General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin T3P3 T2P3 T3P0 UNASSIGNED PARTITIONS Subscriptions - T1 -> M1, M2, M3 T2 -> M2, M3 T3 -> M3 M3 M2 TARGET ASSIGNMENT
3 3 8 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3, P0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
Reassignments are made based on the comparative load of members. ○ Try to maintain rack matching and stickiness Difference between two member’s assignment sizes is one. OR No member can receive additional partitions without disrupting the balance. What is considered a balanced assignment? General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin Balancing the Assignment
3 3 8 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3, P0 T1P2 can be moved from M3 to member M1 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
4 3 7 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4, P2 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5 T2 -> P1, P2 T3 -> P1, P2, P3, P0 T1P2 can be moved from M3 to member M1 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
5 4 5 M1 T1 -> P0, P1, P2, P3, P4 M2 T2 -> P0, P1, P2, P3 M3 T1 -> P5 T3 -> P0, P1, P2, P3 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin Final Assignment
86 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
Assignor Selection Process User Assignor Selection 87
When should you use the Range Assignor? • To balance the load of a single topic evenly among members • Perform joins on co-partitioned streams • Particularly beneficial for applications where the processing sequence of partitions is critical When should you use the Uniform Assignor? • To uniformly distribute the overall load among ALL members • No use cases for joins or partition ordering Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Choosing the right assignor
Assignor Selection Process Server Side Assignor Selection 89
Pluggable Assignor: Clients specify preferred server-side assignor in heartbeat requests. Multiple Assignors Specified: The group coordinator uses the most common one. No Assignor Specified: The group coordinator will default to the first one in group.consumer.assignors. Error for Unsupported Assignors: Heartbeats with invalid assignors receive an UNSUPPORTED_ASSIGNOR error. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assignor Selection: Server Side Mode
What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? What are the different types of assignors? Examples of Range Assignors Examples of Uniform Assignors How do we know which assignor to use when? Any questions? 91 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
Questions? 92

Recommended

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
HostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
HostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
HostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
HostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
HostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
HostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
HostedbyConfluent
 

Recommended

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
 
Renaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit LondonRenaming a Kafka Topic | Kafka Summit London
Renaming a Kafka Topic | Kafka Summit London
HostedbyConfluent
 
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at TrendyolEvolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
HostedbyConfluent
 
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesEnsuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
HostedbyConfluent
 
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and KafkaExactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
HostedbyConfluent
 
Fish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit LondonFish Plays Pokemon | Kafka Summit London
Fish Plays Pokemon | Kafka Summit London
HostedbyConfluent
 
Tiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit LondonTiered Storage 101 | Kafla Summit London
Tiered Storage 101 | Kafla Summit London
HostedbyConfluent
 
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And WhyBuilding a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
HostedbyConfluent
 
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
HostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
HostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
HostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
HostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
HostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
HostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
HostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
HostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
HostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
HostedbyConfluent
 
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
HostedbyConfluent
 
How to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical GridHow to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical Grid
HostedbyConfluent
 
Keep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with ShowbacksKeep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with Showbacks
HostedbyConfluent
 
When Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and DeathWhen Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and Death
HostedbyConfluent
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
HostedbyConfluent
 
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
HostedbyConfluent
 
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch ProcessingFlink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
HostedbyConfluent
 
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
HostedbyConfluent
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 

More Related Content

More from HostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
HostedbyConfluent
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
HostedbyConfluent
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
HostedbyConfluent
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
HostedbyConfluent
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
HostedbyConfluent
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
HostedbyConfluent
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
HostedbyConfluent
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
HostedbyConfluent
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
HostedbyConfluent
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
HostedbyConfluent
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
HostedbyConfluent
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
HostedbyConfluent
 
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
HostedbyConfluent
 
How to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical GridHow to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical Grid
HostedbyConfluent
 
Keep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with ShowbacksKeep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with Showbacks
HostedbyConfluent
 
When Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and DeathWhen Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and Death
HostedbyConfluent
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
HostedbyConfluent
 
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
HostedbyConfluent
 
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch ProcessingFlink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
HostedbyConfluent
 
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
HostedbyConfluent
 

More from HostedbyConfluent (20)

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
 
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
 
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka ClustersNavigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
 
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data PlatformApache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
 
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy PubExplaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
 
TL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit LondonTL;DR Kafka Metrics | Kafka Summit London
TL;DR Kafka Metrics | Kafka Summit London
 
A Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSLA Window Into Your Kafka Streams Tasks | KSL
A Window Into Your Kafka Streams Tasks | KSL
 
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceMastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
 
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and BeyondData Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
 
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink AppsCode-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
 
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC EcosystemDebezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
 
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local DisksBeyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
 
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
Automating Speed: A Proven Approach to Preventing Performance Regressions in ...
 
How to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical GridHow to Build an Event-based Control Center for the Electrical Grid
How to Build an Event-based Control Center for the Electrical Grid
 
Keep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with ShowbacksKeep Your Kafka Cloud Costs in Check with Showbacks
Keep Your Kafka Cloud Costs in Check with Showbacks
 
When Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and DeathWhen Securing Access to Data is About Life and Death
When Securing Access to Data is About Life and Death
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
 
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...
 
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch ProcessingFlink 2.0: Navigating the Future of Unified Stream and Batch Processing
Flink 2.0: Navigating the Future of Unified Stream and Batch Processing
 
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...
 

Recently uploaded

Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
nkrafacyberclub
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Nexer Digital
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
名前 です男
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 

Recently uploaded (20)

Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptxSecstrike : Reverse Engineering & Pwnable tools for CTF.pptx
Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?Elizabeth Buie - Older adults: Are we really designing for our future selves?
Elizabeth Buie - Older adults: Are we really designing for our future selves?
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 

Server Side Assignors for Consumer Groups : KIP-848

  • 1. Server Side Assignors : KIP 848 Ritika Reddy Software Engineer
  • 2. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples How do we know which assignor to use when? Any questions? 2 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
  • 3. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 3 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
  • 4. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 4 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
  • 5. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 5 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
  • 6. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? How do we know which assignor to use when? Any questions? 6 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda What are the different types of assignors? Range Assignor with Examples Uniform Assignor with Examples
  • 8. Consumers in Kafka Consumer Topic 1 P0 P1 P2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 9. What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 10. What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 11. What is a consumer group? Topic 1 P0 P1 P2 Consumer Group C1 C2 C3 group.id = cg1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 12. Types of subscriptions M1 M2 M3 Consumer Group T1 T2 T3 Topics Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 13. Types of subscriptions Homogenous Subscriptions: All members of the consumer group are subscribed to the same set of topics. M1 M2 M3 Consumer Group Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions T1 T2 T3 Topics
  • 14. Types of subscriptions Heterogenous Subscriptions: Different members are subscribed to different topics. M1 M2 M3 Consumer Group Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions T1 T2 T3 Topics
  • 15. Introduction Why are we changing the Current Protocol? 15
  • 16. Drawbacks of Current Protocol Thick Client: Debugging and updates rely heavily on clients Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Global Synchronization Barrier: A single misbehaving consumer can disrupt the entire group
  • 17. Introduction The New Consumer Group Protocol : KIP-848 17
  • 18. Improvements in the new protocol: KIP-848 ConsumerGroupHeartbeat API Integrate partition assignments within heartbeats streamlining state management. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Complexity Shift from Client to Server Easier debugging and update adoption Incremental & Cooperative Rebalance Eliminates existing global sync barriers
  • 19. The Current Protocol: A Layered Protocol The Next-Gen Protocol: Server Side Assignors Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 20. 20 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
  • 22. Role of an Assignor The assignment is not set in stone. Assign partitions to members when the group coordinator says so Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 23. How does the assignor work in the new protocol? Assignor Metadata Group Coordinator Target Assignment Input Output 1. Member Info 2. Current Assignment 3. Topic and Partition Info Rebalance Triggers 4. Assignor Changes 1. Member Joins/Leaves 2. Topic Metadata Changes 3. Broker rack changes Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 24. General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Stickiness Minimizes partition movements between members by retaining as many partitions from the existing assignment. Rack Awareness Aligns partition replicas with members in the same rack to minimize cross-zone traffic and costs.
  • 25. General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack Awareness Aligns partition replicas with members in the same rack to minimize cross-zone traffic and costs. Stickiness Minimize partition movements between members by retaining as many partitions from the existing assignment as possible.
  • 26. General Properties of Assignors Balance Even distribution of subscribed topic partitions across all group members to prevent overloading. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Stickiness Minimizes partition movements between members by retaining as many partitions from the existing assignment. Rack Awareness Align partition replicas with members in the same rack to minimize cross-zone traffic and costs.
  • 27. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Topic Name = Gold Partition = Coin Members of Consumer Group Analogy for Rack Matching
  • 28. 28 Gold Coins 1-10 Princess 1 Rack 1 Rack 2 Gold Coins 10-20 Princess 2 Availability Zones (AZ) Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 29. Priority Order of Properties 29 Enhances group efficiency and reduces latency Balance Optimizes processing speed and reduces cross-AZ costs Rack Awareness Minimizes consumer downtime and operational impact Stickiness Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 30. 30 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
  • 31. Types of Assignors What are the different types of assignors? 31
  • 32. Types of Assignors 32 Range Assignor Uniform Assignor Assigns contiguous partition ranges to consumers. Ensures ordered processing within topics for balanced workload distribution. Distributes topic partitions among group members in a balanced fashion. Two implementations: ★ Optimized ★ General Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 33. Types of Assignors The Range Assignor 33
  • 34. M0 M1 M2 M3 TOPIC 1 Consumer Group 1 Range Assignor Partition Distribution: • Each subscribed member receives at least one partition from that topic. P0 P1 P2 * Except when number of members outnumber partitions. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 35. Partition Co-Allocation: • Each member receives the same partition number from every subscribed topic. M0 M1 M2 M3 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Range Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 36. Per-topic co-allocation for joins: 36 Source: medium.com Credit: Gavin Fong Range Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 37. 37 Map members per Topic Calculate quotas Assign sticky partitions Assign the leftover partitions using ranges *Rack Awareness is not yet implemented for the Range Assignor Range Assignor Algorithm Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 38. Topic 1 Topic 2 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input Range Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 39. Conditions: 1. Each princess gets an equal share of the gold. 2. Once split equally, the extra coins will also be split uniformly starting from the eldest ones first.. 3. Each princess can only get one extra gold coin. Partition = Coin Members of Consumer Group Topic Name = Gold Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 40. Min Quota : Minimum partitions that each member should get Num of partitions (Coins) = 3 Number of members subscribed (Princesses) = 2 3 / 2 = 1 Each princess gets at least 1 coin Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 41. Number of members with extra partitions Num of partitions (Coins) = 3 Number of members subscribed (Princesses) = 2 1 extra coin given to the eldest first Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions 3 % 2 = 1
  • 42. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 ➢ Partitions are sorted in ascending order ➢ Members are implicitly sorted in the same order per topic Topic 1 Consumer Group Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
  • 43. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 extra Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
  • 44. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
  • 45. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2 T2 -> M1, M2 M1 M2 T1P0 T1P1 T1P2 Quotas Ref MinQuota = 1 Members with extra Partition = 1 Topic 2 -> same process as topic 1 extra Topic Name = Silver Map Members Assign Leftover Partitions Range Assignor Example 1 - Topic 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
  • 46. M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Map Members Assign Leftover Partitions Range Assignor Example 1 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate Quotas Assign Sticky Partitions
  • 47. M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 Range Assignor Example 1 - Trigger rebalance by adding a new member Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions M3
  • 48. Topic 1 Topic 2 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input M3 NOTE: Subscriptions of members can be different Joins won’t be feasible in this case Range Assignor Example 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 49. Number of members subscribed = 3 Number of partitions = 3 Min Quota = numPartitionsForTopic / number of subscribed members Extra Partitions = numPartitionsForTopic % number of subscribed members Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2, M3 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 3 / 3 = 1 Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Calculate Quotas Assign Leftover Partitions Assign Sticky Partitions 3 % 3 = 0
  • 50. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 Prev Assignment Ref M1 -> T1P0, T1P1 T2P0, T2P1 M2 -> T1P2, T2P2 T1P0 T1P1 T1P2 M1 M2 Available for assignment Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Assign Leftover Partitions Assign Sticky Partitions Calculate Quotas Quotas Min Quota = 1 Members with Extras = 0
  • 51. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 M3 T1P1 Unassigned Partitions Map Members Assign Leftover Partitions Calculate Quotas Assign Sticky Partitions Range Assignor Example 2 - Topic 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 52. Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Members Per Topic - T1 -> M1, M2, M3 T2 -> M1, M2 Prev Assignment Ref M1 -> T1P0, T1P1 T2P0, T2P1 M2 -> T1P2, T2P2 T2P0 T2P1 T2P2 M1 M2 Quotas Min Quota = 1 Members with Extras = 1 extra Range Assignor Example 2 - Topic 2 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Map Members Assign Leftover Partitions Assign Sticky Partitions Calculate Quotas
  • 53. M1 M2 P0 P1 P2 TOPIC 2 P0 P1 P2 TOPIC 1 Consumer Group 1 M3 1 revocation 1 allocation Range Assignor Example 2 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 54. Highly unbalanced distribution is possible since every extra partition always goes to the first few members. 54 M1 M2 T1P0 T1P1 T2P0 T2P1 T3P0 T3P1 T1P2 T2P2 T3P2 extras M1 has twice as many partitions as M2 Range Assignor Drawbacks Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 55. Types of Assignors The Uniform Assignor 55
  • 56. The uniform assignor is implemented in two different ways: ● Optimized Assignment Builder ● General Assignment Builder The right implementation is automatically chosen based on the topic subscriptions. This is abstracted away from the user. Uniform General Optimized Homogeneous Topic Subscriptions Heterogeneous Topic Subscriptions Uniform Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 57. ❏ Homogenous Subscriptions: ❏ All partitions have the same weight. ❏ Rebalance times can be improved with quotas. ❏ Heterogenous Subscriptions: ❏ More complexity involved to achieve a balanced assignment. ❏ Less frequent. Why do we need two implementations? Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 58. 58 Calculate Quotas Assign Sticky Partitions Rack Aware Round Robin Assignment* Unassigned Partitions Round Robin *Rack aware matching is possible only when both member and broker racks are configured Optimized Uniform Assignment Builder Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 59. Topic 2 Topic 1 P2 P0 P1 P2 P1 P0 M1 M2 Metadata Input Rack 1 Rack 2 P0 P1 P2 Rack 0, Rack 1 Rack 1, Rack 2 Rack 2, Rack 3 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Replica Racks
  • 60. Min Quota = total partitions / total number of members Num Members with Extra Partitions = total partitions % total number of members Metadata Ref Topics - T1, T2 Num of Partitions per topic - 3 Members - M1, M2 Subscriptions - M1 -> T1, T2 M2 -> T1, T2 (3 * 2) / 2 = 3 (3 * 2) % 3 = 0 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
  • 61. M1 Rack 1 Rack 2 M2 Members Racks M1 -> rack1 M2 -> rack2 Partition Racks P0 = [rack0, rack1] P1 = [rack1, rack2] P2 = [rack3, rack2] P0 P1 P2 P1 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
  • 62. M1 Rack 1 M2 Rack 2 Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1) T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Sort partitions in ascending order of number of members in the same rack Sorting is done to maximize rack matching Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
  • 63. M1 M2 Rack 1 Rack 2 T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Quota is already met No other member in the same rack Later partition will go to M2 with no rack matching Quotas Ref MinQuota = 3 Life Without Sorting Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1)
  • 64. M1 M2 Rack 1 Rack 2 Number of members in same rack as Partition P0 = M1 (1) P1 = M1, M2 (2) P2 = M2 (1) T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Quotas Ref MinQuota = 2 Members with extra Partition = 0 Optimized Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
  • 65. Consumer Group 1 M1 M2 T1P0 T1P1 T1P2 T2P0 T2P1 T2P2 Optimized Uniform Assignor Example 1 - Final Assignment Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Calculate quotas Assign Sticky Partitions Rack aware round robin assignment* Unassigned Round Robin
  • 66. 66 Assign Sticky Partitions Rack Aware Round Robin Unassigned Round Robin Balance Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions General Uniform Assignment Builder
  • 67. Key Differences from prev assignors: ❏ No concept of Quotas ❏ No good way to pre-determine quotas ❏ Tried a greedy algorithm and failed at a few edge cases ❏ Sorting members and topics based on various criteria ❏ Rebalance Loops ❏ Iterate over the assignment until a balance is reached General Uniform Assignor Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions
  • 68. Topic 2 P2 P0 P1 M1 Metadata Input Rack 0 Rack 1 Topic 3 P2 P1 P0 M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 Current Assignment M2 Rack 2 M3 P3 P3 P2 P1 P0 Topic 1 P3 P4 P5 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions REBALANCE TRIGGER: ASSIGNOR CHANGE
  • 69. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions
  • 70. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions
  • 71. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0
  • 72. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1
  • 73. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1
  • 74. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 M2 M3 TARGET ASSIGNMENT If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future M1 P3 P0 P4 RACK 0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Rack aware round robin Unassigned Round Robin Balance Assign Sticky Partitions T1 -> P0 T1P1 M1 T1P2 M1
  • 75. M1 T1 -> P0, P1, P2 M2 T1 -> P3 T2 -> P0 M3 T1 -> P4, P5 T2 -> P1, P2, P3 T3 -> P0, P1, P2, P3 CURRENT ASSIGNMENT M1 T1 -> P0 M2 T1 -> T2 -> P0 M3 T1 -> P5 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T1P1 T1P2 T1P3 T1P4 T2P3 T3P0 T3P3 M1 M1 M2 M3 M3 M3 M3 If partition is in the same rack as member: Retain CURRENT PARTITION OWNERS If partition is in a different rack: Track for future General Uniform Assignor Example 1
  • 76. M1 P3 P0 P4 M2 P1 P0 P4 M3 P1 P2 P5 P5 RACK 0 RACK 1 RACK 2 Assign partition if member in matching rack General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Unassigned Round Robin Balance Assign Sticky Partitions Rack aware round robin
  • 77. M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T1P3 T1P2 T3P3 T2P3 T1P4 T3P0 T1P1 If partition is in the same rack as member: Assign SORTED UNASSIGNED PARTITIONS General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Unassigned Round Robin Balance Assign Sticky Partitions Rack aware round robin
  • 78. M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2 TARGET ASSIGNMENT T3P3 T2P3 T3P0 UNASSIGNED PARTITIONS General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin SORT SUBSCRIBED MEMBERS IN ASCENDING ORDER BY ASSIGNMENT SIZE AND ASSIGN FOR BALANCE
  • 79. P3 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin T3P3 T2P3 T3P0 TARGET ASSIGNMENT UNASSIGNED PARTITIONS Subscriptions - T1 -> M1, M2, M3 T2 -> M2, M3 T3 -> M3 M3 M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2
  • 80. M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3 P3 SORT SUBSCRIBED MEMBERS IN ASCENDING ORDER BY ASSIGNMENT SIZE General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Balance Assign Sticky Partitions Unassigned Round Robin Rack aware round robin T3P3 T2P3 T3P0 UNASSIGNED PARTITIONS Subscriptions - T1 -> M1, M2, M3 T2 -> M2, M3 T3 -> M3 M3 M2 TARGET ASSIGNMENT
  • 81. 3 3 8 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3, P0 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
  • 82. Reassignments are made based on the comparative load of members. ○ Try to maintain rack matching and stickiness Difference between two member’s assignment sizes is one. OR No member can receive additional partitions without disrupting the balance. What is considered a balanced assignment? General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin Balancing the Assignment
  • 83. 3 3 8 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5, P2 T2 -> P1, P2 T3 -> P1, P2, P3, P0 T1P2 can be moved from M3 to member M1 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
  • 84. 4 3 7 TARGET ASSIGNMENT M1 T1 -> P0, P3, P4, P2 M2 T1 -> P1 T2 -> P0, P3 M3 T1 -> P5 T2 -> P1, P2 T3 -> P1, P2, P3, P0 T1P2 can be moved from M3 to member M1 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin
  • 85. 5 4 5 M1 T1 -> P0, P1, P2, P3, P4 M2 T2 -> P0, P1, P2, P3 M3 T1 -> P5 T3 -> P0, P1, P2, P3 General Uniform Assignor Example 1 Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assign Sticky Partitions Rack aware round robin Balance Unassigned Round Robin Final Assignment
  • 86. 86 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda
  • 87. Assignor Selection Process User Assignor Selection 87
  • 88. When should you use the Range Assignor? • To balance the load of a single topic evenly among members • Perform joins on co-partitioned streams • Particularly beneficial for applications where the processing sequence of partitions is critical When should you use the Uniform Assignor? • To uniformly distribute the overall load among ALL members • No use cases for joins or partition ordering Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Choosing the right assignor
  • 89. Assignor Selection Process Server Side Assignor Selection 89
  • 90. Pluggable Assignor: Clients specify preferred server-side assignor in heartbeat requests. Multiple Assignors Specified: The group coordinator uses the most common one. No Assignor Specified: The group coordinator will default to the first one in group.consumer.assignors. Error for Unsupported Assignors: Heartbeats with invalid assignors receive an UNSUPPORTED_ASSIGNOR error. Introduction > Assignors > Types of Assignors > Assignor Selection Process > Questions Assignor Selection: Server Side Mode
  • 91. What are consumer groups? Drawbacks of the current protocol How is the new protocol different? What are assignors? What are their roles and properties? What are the different types of assignors? Examples of Range Assignors Examples of Uniform Assignors How do we know which assignor to use when? Any questions? 91 Assignors Types of Assignors Assignor Selection Process Questions Introduction Agenda