Design and Implementation of Incremental Cooperative Rebalancing

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

1
Design and Implementation of
Incremental Cooperative Rebalancing
Konstantine Karantasis, PhD
Software Engineer, Confluen...

2
Nice to meet you
Contributor:
● Apache Kafka®
● Confluent Platform
● Confluent Community
Components
kkonstantine
@karant...

3
Agenda
● Load balancing in Kafka Clients
● Challenges
● A new protocol
● Its first implementation
● Is it working?
● Pat...

44
Distributed Systems
are intriguing
An abundance of holy grails
● Consistency
● Consensus
● Load balancing

5
Load balancing in Kafka clients
a.k.a rebalancing
● Built on top of Group Membership
● Broker coordinator assigns the le...

6
Load balancing as an embedded protocol
Two important pairs or Request/Response
● Join group request/response
● Sync grou...

7
Kafka clients and resources to be balanced
● Consumer, Streams: Topic-partitions
● Connect: Tasks
● Schema Registry: Lea...

8
Stop-the-world rebalancing
● In every rebalance clients release their resources
● Resources ➡ global pool
● Releasing re...

99
Challenges at Large
Scale
“There is a coming and a going
A parting and often no--meeting again”
Both in the Cloud and O...

10
Load balancing challenges
1. Scaling up and down
2. Multitenancy
3. Kubernetes process death
4. Rolling bounce

1111
1. Scaling up and down
Client 1 (Leader) Client 2 (Member)

1212
1. Scaling up and down
Client 1 (Leader) Client 2 (Member)

1313
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

1414
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

1515
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

16
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

17
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

18
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

19
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

20
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

21
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

22
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

2323
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

2424
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

2525
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

2626
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

2727
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

2828
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

2929
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

3030
4. Rolling restarts
Client 1 (Leader) Client 2 (Member)

3131
4. Rolling restarts
Client 1 (Leader) Client 2 (Member)

3232
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

3333
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

3434
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

3535
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

3636
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

3737
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)Client 2 (Member)

3838
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

3939
Kafka clients load balancing revisited
Do not stop-the-world
It’s expensive

40
Incremental Cooperative Rebalance
Goal: Address the challenges at large scale
Why incremental:
● No need to reach final...

41
KIP-415
● Adopted in Apache Kafka 2.3
● Does not require broker coordinator upgrade
● Easy upgrades and extensions
● Th...

4242
A new worker joins the group

43
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

44
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)
Revoked:
1st Rebalance

45
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)
2nd Rebalance

4646
But what about failures or restarts?

4747
Rebalance with a delay
● Start and stop can be expensive
● Avoid unnecessary task shuffling
● Don’t interfere with ru...

4848
An existing worker leaves and bounces back

49
An existing worker bounces
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

50
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance

51
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance
Unassigned:
scheduled.reb...

52
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)
2nd Rebalance
Unassigned:
Worker 2 returns within the in...

53
scheduled.rebalance.max.delay.ms has expired
Upon expiration all the workers rejoin the group triggering a 3rd rebalanc...

5454
An existing worker leaves permanently

55
An existing worker leaves permanently
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

56
An existing worker leaves permanently
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance

57
Unassigned:
scheduled.rebalance.max.delay.ms goes in effect. Starts with a default value of 5 min
An existing worker le...

58
scheduled.rebalance.max.delay.ms has expired
Upon expiration, workers rejoin the group triggering a 3rd rebalance
Unass...

59
Implementation in Kafka Connect
● Behind the scenes, a state machine with:
○ Three base sets acting as sources of truth...

6060
New Rebalancing in Action

6161
90 connectors
900 tasks
3 workers

62
Rebalance independently of the current load
With Incremental Cooperative Rebalancing the cost does not scale with the
n...

63
Rebalances become lightweight
More rebalances but less expensive
Time (hour : minute)
Total number of rebalances Eager ...

64
The impact of rebalancing on throughput
90 S3 Connectors/900 Tasks
Eager
Rebalancing
Incremental Cooperative
Rebalancin...

6565
Patterns and Predictions

66
Freedom from
workarounds
● Fragmented deployments
● Increased timeouts to avoid
rebalance storms

67
Smaller clusters, more of them?
Takeaways
● Running Kafka clients at scale is becoming a reality
● Bigger clusters of c...

68
Predictions for 2020 and beyond
● Increased number of large scale
Connector deployments of thousands
of tasks in produc...

6969
cnfl.io/slack
Stay in touch!
cnfl.io/download cnfl.io/blog

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1056584 views
Storytelling For The Web: Integrate... by Chiara Aliotta 973080 views
Artificial Intelligence, Data and C... by OECD Directorate ... 908885 views
How to Leverage AI to Boost Employe... by SocialHRCamp 384853 views
2024 State of Marketing Report – by... by Marius Sescu 217959 views
Everything You Need To Know About C... by Expeed Software 235285 views

View on Slideshare

1 of 70 Ad

View on Slideshare

1 of 70 Ad