Watch this talk here: https://www.confluent.io/online-talks/design-and-implementation-of-incremental-cooperative-rebalancing-on-demand Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself. Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol. However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs. This talk will cover: -The internals of Incremental Cooperative Rebalancing -Uses cases that benefit from Incremental Cooperative Rebalancing -Implementation in Kafka Connect -Performance results in Kafka Connect clusters

1. 1
Design and Implementation of
Incremental Cooperative Rebalancing
Konstantine Karantasis, PhD
Software Engineer, Confluent, Inc.

2. 2
Nice to meet you
Contributor:
● Apache Kafka®
● Confluent Platform
● Confluent Community
Components
kkonstantine
@karantasis

3. 3
Agenda
● Load balancing in Kafka Clients
● Challenges
● A new protocol
● Its first implementation
● Is it working?
● Patterns and predictions

4. 44
Distributed Systems
are intriguing
An abundance of holy grails
● Consistency
● Consensus
● Load balancing

5. 5
Load balancing in Kafka clients
a.k.a rebalancing
● Built on top of Group Membership
● Broker coordinator assigns the leader
● Load balancing is piggybacked to the Group Membership API calls

6. 6
Load balancing as an embedded protocol
Two important pairs or Request/Response
● Join group request/response
● Sync group request/response

7. 7
Kafka clients and resources to be balanced
● Consumer, Streams: Topic-partitions
● Connect: Tasks
● Schema Registry: Leadership

8. 8
Stop-the-world rebalancing
● In every rebalance clients release their resources
● Resources ➡ global pool
● Releasing resources is expensive
● Reacquiring resources is expensive

9. 99
Challenges at Large
Scale
“There is a coming and a going
A parting and often no--meeting again”
Both in the Cloud and On-Prem
- Franz Kafka, 1897

10. 10
Load balancing challenges
1. Scaling up and down
2. Multitenancy
3. Kubernetes process death
4. Rolling bounce

11. 1111
1. Scaling up and down
Client 1 (Leader) Client 2 (Member)

12. 1212
1. Scaling up and down
Client 1 (Leader) Client 2 (Member)

13. 1313
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

14. 1414
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

15. 1515
1. Scaling up and down
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

16. 16
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

17. 17
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

18. 18
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

19. 19
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

20. 20
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

21. 21
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

22. 22
2. Multitenancy
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

23. 2323
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

24. 2424
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

25. 2525
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

26. 2626
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member)

27. 2727
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

28. 2828
3. Kubernetes process death
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

29. 2929
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

30. 3030
4. Rolling restarts
Client 1 (Leader) Client 2 (Member)

31. 3131
4. Rolling restarts
Client 1 (Leader) Client 2 (Member)

32. 3232
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

33. 3333
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

34. 3434
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

35. 3535
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

36. 3636
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)

37. 3737
4. Rolling restarts
Client 1 (Leader) Client 3 (Member)Client 2 (Member)

38. 3838
4. Rolling restarts
Client 1 (Leader) Client 2 (Member) Client 3 (Member)

39. 3939
Kafka clients load balancing revisited
Do not stop-the-world
It’s expensive

40. 40
Incremental Cooperative Rebalance
Goal: Address the challenges at large scale
Why incremental:
● No need to reach final state within a single rebalance round
● A grace period is configurable
● The protocol converges smoothly to a state of balanced load
Why cooperative:
● Resource revocation and release is graceful

41. 41
KIP-415
● Adopted in Apache Kafka 2.3
● Does not require broker coordinator upgrade
● Easy upgrades and extensions
● The Connect protocol was extended to include:
a) assigned tasks
b) revoked tasks
c) a scheduled delay to perform another rebalance
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect

42. 4242
A new worker joins the group

43. 43
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

44. 44
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)
Revoked:
1st Rebalance

45. 45
A new worker joins
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)
2nd Rebalance

46. 4646
But what about failures or restarts?

47. 4747
Rebalance with a delay
● Start and stop can be expensive
● Avoid unnecessary task shuffling
● Don’t interfere with running tasks
Postpone task shuffling with:
scheduled.rebalance.max.delay.ms
Affects only “lost” tasks

48. 4848
An existing worker leaves and bounces back

49. 49
An existing worker bounces
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

50. 50
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance

51. 51
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance
Unassigned:
scheduled.rebalance.max.delay.ms goes in effect. Starts with a default value of 5 min

52. 52
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)
2nd Rebalance
Unassigned:
Worker 2 returns within the initial delay of 5 min
scheduled.rebalance.max.delay.ms is still in effect. Value is between 0 and 5 min
Worker 2 (Member)

53. 53
scheduled.rebalance.max.delay.ms has expired
Upon expiration all the workers rejoin the group triggering a 3rd rebalance
An existing worker bounces
Worker 1 (Leader) Worker 3 (Member)
3rd Rebalance
Worker 2 (Member)

54. 5454
An existing worker leaves permanently

55. 55
An existing worker leaves permanently
Worker 2 (Member)Worker 1 (Leader) Worker 3 (Member)

56. 56
An existing worker leaves permanently
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance

57. 57
Unassigned:
scheduled.rebalance.max.delay.ms goes in effect. Starts with a default value of 5 min
An existing worker leaves permanently
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
1st Rebalance

58. 58
scheduled.rebalance.max.delay.ms has expired
Upon expiration, workers rejoin the group triggering a 3rd rebalance
Unassigned tasks are distributed to the existing workers
An existing worker leaves permanently
Worker 1 (Leader) Worker 3 (Member)Worker 2 (Member)
2nd Rebalance

59. 59
Implementation in Kafka Connect
● Behind the scenes, a state machine with:
○ Three base sets acting as sources of truth for the leader
○ Several derived sets extracted from the base sets
○ Logic to handle the delays
○ Weighted round-robin for task assignment

60. 6060
New Rebalancing in Action

61. 6161
90 connectors
900 tasks
3 workers

62. 62
Rebalance independently of the current load
With Incremental Cooperative Rebalancing the cost does not scale with the
number of tasks
Time (hour : minute)
Maximum Rebalance Time
(milliseconds)
Eager
Worker Startup
Startup 900
tasks Shutdown 900
tasks Startup 900
tasks
Shutdown 900
tasks
Worker Startup
Incremental Cooperative

63. 63
Rebalances become lightweight
More rebalances but less expensive
Time (hour : minute)
Total number of rebalances Eager Incremental Cooperative

64. 64
The impact of rebalancing on throughput
90 S3 Connectors/900 Tasks
Eager
Rebalancing
Incremental Cooperative
Rebalancing
Aggregate throughput (MB/s) 252.68 537.81 2x improvement
Maximum throughput (MB/s) 0.41 3.82 9x improvement
For details, read:
https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka

65. 6565
Patterns and Predictions

66. 66
Freedom from
workarounds
● Fragmented deployments
● Increased timeouts to avoid
rebalance storms

67. 67
Smaller clusters, more of them?
Takeaways
● Running Kafka clients at scale is becoming a reality
● Bigger clusters of clients are now more manageable and can be diverse
● Get it for free! Just upgrade your Connect cluster to version 2.3 and beyond
● Fall back to Eager Rebalancing by setting connect.protocol config

68. 68
Predictions for 2020 and beyond
● Increased number of large scale
Connector deployments of thousands
of tasks in production
● Rebalancing improvements are coming
to the Kafka consumer and Kafka
Streams
(KIP-429 and KIP-441)

69. 6969
cnfl.io/slack
Stay in touch!
cnfl.io/download cnfl.io/blog

70. 70