Uploaded byC4Media
Kafka Needs No Keeper
Kafka is evolving to remove its dependency on Zookeeper. The Kafka Improvement Proposal 500 (KIP-500) aims to manage Kafka's metadata log with a self-managed Raft consensus algorithm and controller quorum rather than relying on Zookeeper. This will improve scalability, robustness, and make deployment easier. It will take multiple releases to fully implement KIP-500, beginning with removing Zookeeper from clients and ending with a release where Zookeeper is no longer required.
More Related Content
Kafka Needs No Keeper
- 1. 1 Kafka Needs NoKeeper Colin McCabe
- 2. InfoQ.com: News &Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ kafka-zookeeper/
- 3. Purpose of QCon -to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. 2 ● Kafka hasgotten its mileage out of Zookeeper ● But it is still a second system ● KIP-500 has been adopted by the community ● This is not a 1-1 replacement ● We’ve been headed this direction for years Introduction
- 5. 3 Evolution of Apache KafkaClients
- 6.
- 7.
- 8.
- 9. 7 write to topics read from topics offset fetch/commit grouppartition assignment Producer Consumer Admin Tools
- 10. 8 write to topics read from topics offset fetch/commit grouppartition assignment topic create/delete Producer Consumer Admin Tools
- 11.
- 12.
- 13.
- 14.
- 15. 13 Consumer Consumer APIs ● Fetchoffset fetch/commit grouppartition assignment read from topics __offsets
- 16. 14 offset fetch/commit Consumer group partition assignment read from topics ConsumerAPIs ● Fetch ● OffsetCommit ● OffsetFetch __offsets
- 17. 15 Consumer group partition assignment read from topics offset fetch/commitConsumer APIs ● Fetch ● OffsetCommit ● OffsetFetch __offsets
- 18. 16 Consumer group partition assignment read from topics offset fetch/commitConsumer APIs ● Fetch ● OffsetCommit ● OffsetFetch __offsets
- 19. 17 group partition assignment Consumer read from topics offset fetch/commitConsumer APIs ● Fetch ● OffsetCommit ● OffsetFetch ● JoinGroup ● SyncGroup ● Heartbeat __offsets
- 20. 18 Consumer read from topics offset fetch/commit group partition assignment ConsumerAPIs ● Fetch ● OffsetCommit ● OffsetFetch ● JoinGroup ● SyncGroup ● Heartbeat __offsets
- 21. 19 Consumer read from topics offset fetch/commit group partition assignment ConsumerAPIs ● Fetch ● OffsetCommit ● OffsetFetch ● JoinGroup ● SyncGroup ● Heartbeat __offsets
- 22. 20 read from topics offset fetch/commit group partition assignment Consumer ConsumerAPIs ● Fetch ● OffsetCommit ● OffsetFetch ● JoinGroup ● SyncGroup ● Heartbeat __offsets
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. 28 AdminClient Admin Tools ACL Enforcement create/delete topics AdminAPIs: ● CreateTopics ● DeleteTopics ● AlterConfigs ● ...
- 31. 29 Admin APIs: ● CreateTopics ●DeleteTopics ● AlterConfigs ● ... AdminClient Admin Tools ACL Enforcement
- 32. 30 Producer Consumer AdminClient Client APIs: ● Produce ●Fetch ● Metadata ● CreateTopics ● DeleteTopics ● ...
- 33. 31 Producer Consumer AdminClient Client APIs: ● Produce ●Fetch ● Metadata ● CreateTopics ● DeleteTopics ● ... ● Encapsulation ● Security ● Validation ● Compatibility
- 34.
- 35.
- 36. 34 Broker Registration ACL Management DynamicConfiguration ISR Management
- 37. 35 Controller Broker Registration ACL Management DynamicConfiguration ISR Management
- 38. 36 Controller Broker Registration ACL Management DynamicConfiguration ISR Management Controller Election
- 39. 37 Controller Broker Registration ACL Management DynamicConfiguration ISR Management Controller Election
- 40. 38 Controller Broker Registration ACL Management DynamicConfiguration ISR Management Controller Election
- 41. 39 Controller Controller APIs: ●LeaderAndIsr ● UpdateMetadata ● StopReplica Leader/ISR Push Update Metadata Stop/Delete Replica Broker Registration ACL Management Dynamic Configuration ISR Management Controller Election
- 42. 40 Controller Controller APIs: ●LeaderAndIsr ● UpdateMetadata ● StopReplica Leader/ISR Push Update Metadata Stop/Delete Replica Broker Registration ACL Management Dynamic Configuration ISR Management Controller Election
- 43. 41 Controller Controller APIs: ●LeaderAndIsr ● UpdateMetadata ● StopReplica Leader/ISR Push Update Metadata Stop/Delete Replica Broker Registration ACL Management Dynamic Configuration ISR Management Controller Election
- 44. 42 Controller Controller APIs: ●LeaderAndIsr ● UpdateMetadata ● StopReplica ● AlterIsr Broker Registration ACL Management Dynamic Configuration ISR Management Controller Election Leader/ISR Push Update Metadata Stop/Delete Replica
- 45. 43 Controller Leader/ISR Push Update Metadata Stop/DeleteReplica ISR Management Controller APIs: ● LeaderAndIsr ● UpdateMetadata ● StopReplica ● AlterIsr Broker Registration ACL Management Dynamic Configuration ISR Management Controller Election
- 46.
- 47. 45 ● Encapsulation ● Compatibility ●Ownership
- 48.
- 49.
- 50. 48 /brokers/1 -> { host:10.10.10.1:9092 rack: rack-1 }
- 51. 49 /brokers/1 -> { host:10.10.10.1:9092 rack: rack-1 }
- 52.
- 53.
- 54. 52 Watch trigger Broker 1is offline
- 55.
- 56.
- 57. 55 Case 1: Totalpartition
- 58. 56 Case 2: Brokerpartition
- 59. 57 Case 3: ZkPartition
- 60. 58 Case 4: Controllerpartition
- 61.
- 62.
- 63.
- 64. 62 Metadata Source of Truth MetadataCache - sync writes - async updates
- 65. 63 Metadata Source of Truth MetadataCache - async update Metadata Cache - sync writes - async updates Metadata Cache - async update
- 66.
- 67.
- 68.
- 69. 67 Last Resort: > rmr/controller
- 70. 68 Last Resort: > rmr/controller New controller!
- 71. 69 Last Resort: > rmr/controller Load ALL Metadata
- 72. 70 Last Resort: > rmr/controller Load ALL Metadata
- 73. 71 Last Resort: > rmr/controller Push ALL Metadata
- 74. 72 Last Resort: > rmr/controller Push ALL Metadata
- 75. 73 Last Resort: > rmr/controller Push ALL Metadata How do you know the metadata has diverged?
- 76.
- 77.
- 78.
- 79.
- 80.
- 81. 79 Load ALL Metadata Complexity: O(N) N= number of partitions
- 82.
- 83.
- 84. 82 Push ALL Metadata Complexity: O(N*M) N= number of partitions M = number of brokers
- 85.
- 86. 8484 Metadata as an EventLog - Each change becomes a message - Changes are propagated to all brokers ... 924 Create topic ”foo” 925 Delete topic “bar” 926 Add node 4 to the cluster 927 Create topic “baz” 928 Alter ISR for “foo-0” 929 Add node 5 to the cluster
- 87. 8585 Metadata as an EventLog - Clear ordering - Can send deltas - Offset tracks consumer position - Easy to measure lag ... 924 Create topic ”foo” 925 Delete topic “bar” 926 Add node 4 to the cluster 927 Create topic “baz” 928 Alter ISR for “foo-0” 929 Add node 5 to the cluster
- 88.
- 89.
- 90.
- 91. 89 Can we usethe existing Kafka log replication protocol? - How do we elect the leader? We need a self-managed quorum. Implementing the Controller Log
- 92. 90 Can we usethe existing Kafka log replication protocol? - How do we elect the leader? We need a self-managed quorum. Implementing the Controller Log Enter Raft. Leader election is by simple majority.
- 93. 91 Kafka Raft Writes SingleLeader Single Leader Fencing Monotonically increasing epoch Monotonically increasing term Log reconciliation Offset and epoch Term and index Push/Pull Pull Push Commit Semantics ISR Majority Leader Election From ISR through Zookeeper Majority
- 94.
- 95. 93 offset=1 offset=2 Broker Broker Controller Controller Controller The Controller RaftQuorum - The leader is the active controller - Controls reads / writes to the log - Typically 3 or 5 nodes, like ZK
- 96. 94 offset=1 offset=2 Broker Broker Controller Controller Controller Instant Failover - Low-latencyfailover via Raft election - Standbys contain all data in memory - Brokers do not need to re-fetch
- 97. 95 /mnt/logs/kafka/metadata offset=1 Broker Broker Controller Controller Controller Metadata Caching - Brokerscan persist metadata to disk - Only fetch what they need - Use snapshots if we’re too far behind /mnt/logs/kafka/metadata offset=2
- 98. 96 Broker Registration - Buildinga map of the cluster - What brokers exist in the cluster? - How can they be reached? Controller
- 99. 97 Broker Registration - Brokerssend heartbeats to the active controller - The controller uses this to build a map of the cluster Controller
- 100. 98 Controller Broker Registration - Brokerssend heartbeats to the active controller - The controller uses this to build a map of the cluster - The controller also tells brokers if they should be fenced or shut down
- 101. 99 Controller Fencing - Brokers needto be fenced if they’re partitioned from the controller, or can’t keep up - Brokers self-fence if they can’t talk to the controller
- 102.
- 103. 101 Case 1: Totalpartition
- 104. 102 Case 1: Totalpartition
- 105.
- 106.
- 107.
- 108. 106 Deployment Current KIP-500 Configuration FileKafka and ZooKeeper Kafka Metrics Kafka and ZK Kafka Administrative Tools ZK Shell, Four letter words, Kafka tools Kafka tools Security Kafka and ZK Kafka
- 109. 107 Shared Controller Nodes - Fewer resources used -Single node clusters (eventually)
- 110. 108 Separate Controller Nodes - Better resource isolation -Good for big clusters
- 111.
- 112.
- 113. 111 Remove Client-side ZK dependencies Remove Broker-side ZK dependencies Controller Quorum IncrementalKIP-4 Improvements - Create new APIs - Deprecate direct ZK access
- 114. 112 Remove Client-side ZK dependencies Remove Broker-side ZK dependencies Controller Quorum Broker-SideFixes - Remove deprecated direct ZK access for tools - Create broker-side APIs - Centralize ZK access in the controller
- 115. 113 Remove Client-side ZK dependencies Remove Broker-side ZK dependencies Controller Quorum FirstRelease without ZooKeeper - Raft - Controller quorum
- 116. 114 Upgrade Issues - Tools usingZK - Brokers accessing ZK - State in ZK KIP-500 Release Older Kafka Release
- 117. 115 Bridge Release KIP-500 Release OlderKafka Release Bridge Release - No ZK access from tools, brokers (except controller)
- 118. 116 Upgrading - Starting fromthe bridge release
- 119. 117 Upgrading - Start newcontroller nodes (possibly combined) - Quorum elects leader - Claims leadership in ZK
- 120. 118 Upgrading - Roll nodesone by one as usual - Controller continues sending LeaderAndIsr, etc. to old nodes
- 121. 119 Upgrading - When allbrokers have been rolled, decommission ZK nodes
- 122.
- 123. 121 Apache ZooKeeper hasserved us well - KIP-500 is not a 1:1 replacement, but a different paradigm We have already started removing ZK from clients - Consumer, AdminClient - Improved encapsulation, security, upgradability
- 124. 122 Metadata should bemanaged as a log - Deltas, ordering, caching - Controller Failover, Fencing - Improved scalability, robustness, easier deployment The metadata log must be self-managed - Raft - Controller quorum
- 125. 123 It will takea few releases to implement KIP-500 - Additional KIPs for APIs, Raft, Metadata, etc. Rolling upgrades will be supported - Bridge release - Post-ZK release Kafka needs no Keeper
- 126. 124 cnfl.io/meetups cnfl.io/blog cnfl.io/slack THANKYOU Colin McCabe cmccabe@confluent.io
- 127. Watch the videowith slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ kafka-zookeeper/