A Deep Dive into Kafka Controller

•

3 likes•338 views

Speaker: Jun Rao, VP of Apache Kafka and Co-founder of Confluent The controller is the brain of Apache Kafka®. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. In this talk, Jun will outline the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun will then describe recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster. Jun Rao is the co-founder of Confluent, a company that provides a streaming data platform on top of Apache Kafka. Previously, Jun was a senior staff engineer at LinkedIn, where he led the development of Kafka, and a researcher at IBM's Almaden research datacenter, where he conducted research on database and distributed systems. Jun is the PMC chair of Apache Kafka and a committer of Cassandra. He writes at https://cnfl.io/blog-jun-rao.

Technology

A Deep Dive into Kafka
Controller
Jun Rao
VP of Apache Kafka
Co-founder of Confluent

Kafka Replication
• Configurable replication factor
• Tolerating f – 1 failures with f replicas
• Automated failover
topic1-part1
logs
broker 1
topic1-part2
logs
broker 2
topic2-part2
topic2-part1
logs
broker 3
topic1-part1
logs
broker 4
topic1-part2
topic2-part2 topic1-part1 topic1-part2
topic2-part1
topic2-part2
topic2-part1

High Level Data Flow in Replication
broker 1
producer
leader
broker 2
follower
broker 3
follower
4
2
2
3
commit
ack
topic1-part1 topic1-part1 topic1-part1
consumer
1

What’s controller
4
• One broker in a cluster acts as controller
• Monitor the liveness of brokers
• Elect new leaders on broker failure
• Communicate new leaders to brokers

Controller election
Zookeeper
/controller  broker 0
Controller
broker 0 broker 3broker 2broker 1

Partition state: stored in ZK, cached in
controller
Zookeeper
/topic/t1/0  leader:1
/topic/t1/1  leader:3
…
/topic/t1/9  leader:2
Controller
broker 0 broker 3broker 2broker 1

Controlled shutdown
SIG_TERM
Zookeeper
Controller
1
2
broker 2
part t-0: follower
part t-1: follower
broker 1
part t-0: leader
part t-1: leader
broker 0
Zookeeper
Controller
3
5
broker 2
part t-0: leader
part t-1: leader
broker 1
part t-0:
part t-1:
broker 0
4
/topics/t/0  2
/topics/t/1  2

Issues with controlled shutdown (pre 1.1)
Zookeeper
Controller
3
5
broker 0
4
Writes to ZK
are serial
Impact:
longer
shutdown
time
Communication of new
leaders not batched
Impact: client timeout
broker 2
part t-0: leader
part t-1: leader
broker 1
part t-0:
part t-1:
/topics/t/0  2
/topics/t/1  2

Controller failover
Zookeeper
/controller  broker 0
Controller
broker 0 broker 3broker 2broker 1
1

Controller failover
Controller
broker 0 broker 3broker 2broker 1
1 2
Controller
Zookeeper
/controller  broker 2

Controller failover
Zookeeper
/controller  broker 2
/topic/t1/0  leader:1
/topic/t1/1  leader:3
…
/topic/t1/9  leader:2
Controller
broker 0 broker 3broker 2broker 1
1 2
3
Controller

Issues with controller failover (pre 1.1)
Controller
broker 0 broker 3broker 2broker 1
1 2
3
Controller
Reads from ZK are serial
Impact: availability
Zombie old controller
Impact: inconsistency
Zookeeper
/controller  broker 2
/topic/t1/0  leader:1
/topic/t1/1  leader:3
…
/topic/t1/9  leader:2

Performance improvements in 1.1
13
• Controller uses async ZK api for reads/writes
• Controller communicates new leaders to brokers in batches
part 1 part 2 part 3 part 4
part 1
part 2
part 3
part 4
Old (serial):
New (pipelined):

/topics/t/0  2
/topics/t/1  2
Controlled shutdown (post 1.1)
Zookeeper
Controller
3
5
broker 0
4
Writes to ZK
pipelined
Communication of new
leaders batched
broker 2
part t-0: leader
part t-1: leader
broker 1
part t-0:
part t-1:

Controller failover (post 1.1)
Controller
broker 0 broker 3broker 2broker 1
1 2
3
Controller
Reads from ZK pipelined
Zookeeper
/controller  broker 2
/topic/t1/0  leader:1
/topic/t1/1  leader:3
…
/topic/t1/9  leader:2

Results for controlled shutdown
16
• 5 ZK nodes and 5 brokers on different racks
• 25K topics, 1 partition, 2 replicas
• 10K partitions per broker
Kafka 1.0.0 Kafka 1.1.0
Controlled shutdown time 6.5 minutes 3 seconds

Results for controller failover
17
• 5 ZK nodes and 5 brokers on different racks
• 2K topics, 50 partitions, 1 replica
• Controller failover: reload100K partitions from ZK
Kafka 1.0.0 Kafka 1.1.0
State reload time 28 seconds 14 seconds

Fencing zombie requests from controller
18
• Zombie controller
• ZK session expiration: better handling in the controller (1.1)
• Controller path deletion: writes to ZK conditioned on controller
epoch (2.1)
• Zombie request from broker restart
• Broker epoch (KIP-380 in 2.2)
• Also fixed the missing ZK watcher issue

Summary
• Significant performance improvement in controller in 1.1
• Allow 10X more partitions in a Kafka cluster
• Better fencing of zombie requests from controller (1.1, 2.1,
2.2)
• More details and remaining work in KAFKA-5027

Q/A
• Acknowledgment: Onur Karaman, Manikumar Reddy,
Prasanna Gautam, Ismael Juma, Mickael Maison, Sandor
Murakozi, Rajini Sivaram,Ted Yu, Zhanxiang Huang

Presentation at Strata Data Conference 2018, New York The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.

Data Loss and Duplication in Kafka

Jayesh Thakrar

The document discusses data loss and duplication in Apache Kafka. It begins with an overview of Kafka and how it works as a distributed commit log. It then covers sources of data loss, such as failures at the producer or cluster level. Data duplication can occur when producers retry messages or consumers restart from an unclean shutdown. The document provides configurations and techniques to minimize data loss and duplication, such as using producer acknowledgments and storing metadata to validate messages. It also discusses monitoring Kafka using JMX metrics to detect issues.

Introduction to Apache Kafka

Shiao-An Yuan

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

Deep Dive into Apache Kafka

confluent

In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular. - Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput. - Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism. - One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.

A Kafka Client’s Request: There and Back Again with Danica Fine

HostedbyConfluent

"Do you know how your data moves into and out of your Apache Kafka® instance? From the programmer’s point of view, it’s relatively simple. But under the hood, writing to and reading from Kafka is a complex process with a fascinating life cycle that’s worth understanding. When you call producer.send() or consumer.poll(), those calls are translated into low-level requests which are sent along to the brokers for processing. In this session, we’ll dive into the world of Kafka producers and consumers to follow a request from an initial call to send() or poll(), all the way to disk, and back to the client via the broker’s final response. Along the way, we’ll explore a number of client and broker configurations that affect how these requests are handled and discuss the metrics that you can monitor to help you to keep track of every stage of the request life cycle. By the end of this session, you’ll know the ins and outs of the read and write requests that your Kafka clients make, making your next debugging or performance analysis session a breeze."

Kafka internals

David Groozman

Espresso Database Replication with Kafka, Tom Quiggle

confluent

This document discusses using Apache Kafka for database replication in LinkedIn's ESPRESSO database system. It provides an overview of ESPRESSO's architecture and transition from per-instance to per-partition replication using Kafka. Key aspects covered include Kafka configuration, the message protocol for ensuring in-order delivery, and checkpointing by the Kafka producer to allow resuming replication from the last committed transaction after failures.

Introduction to Apache Kafka

Jeff Holoman

The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.

The document introduces Apache Kafka's Streams API for stream processing. Some key points covered include: - The Streams API allows building stream processing applications without needing a separate cluster, providing an elastic, scalable, and fault-tolerant processing engine. - It integrates with existing Kafka deployments and supports both stateful and stateless computations on data in Kafka topics. - Applications built with the Streams API are standard Java applications that run on client machines and leverage Kafka for distributed, parallel processing and fault tolerance via state stores in Kafka.

Cruise Control: Effortless management of Kafka clusters

Prateek Maheshwari

Kafka has become the de facto standard for streaming data with high-throughput, low-latency, and fault-tolerance. However, its rising adoption raises new challenges. In particular, the growing cluster sizes, increasing volume and diversity of user traffic, and aging network and server components induce an overhead in managing the system. This overhead makes it infeasible for human operators to constantly monitor, identify, and mitigate issues. The resulting utilization imbalance across brokers leads to unpredictable client performance due to the high variation in their throughput and latency. Finally, properly expanding, shrinking, or upgrading clusters also incurs a management overhead. Hence, adopting a principled approach to manage Kafka clusters is integral to the sustainability of the infrastructure. This talk will describe how LinkedIn alleviates the management overhead of large-scale Kafka clusters using Cruise Control. To this end, first, we will discuss the reactive and proactive techniques that Cruise Control uses to support admin operations for cluster maintenance, enable anomaly detection with self-healing, and provide real-time monitoring for Kafka clusters. Next, we will examine how Cruise Control performs in production. Finally, we will conclude with questions and further discussion.

Everything You Always Wanted to Know About Kafka's Rebalance Protocol but Wer...

confluent

Watch this talk here: https://www.confluent.io/online-talks/everything-you-always-wanted-to-know-about-kafkas-rebalance-protocol-but-were-afraid-to-ask-on-demand Apache Kafka® is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcomings lead to two major recent improvement proposals, namely static group membership and incremental rebalancing. This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work. We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.

A visual introduction to Apache Kafka

Paul Brebner

Kafka 101 and Developer Best Practices

confluent

Deploying Confluent Platform for Production

confluent

Kafka presentation

Mohammed Fazuluddin

Kafka is an open source messaging system that can handle massive streams of data in real-time. It is fast, scalable, durable, and fault-tolerant. Kafka is commonly used for stream processing, website activity tracking, metrics collection, and log aggregation. It supports high throughput, reliable delivery, and horizontal scalability. Some examples of real-time use cases for Kafka include website monitoring, network monitoring, fraud detection, and IoT applications.

Introduction to Kafka

Akash Vacher

Disaster Recovery with MirrorMaker 2.0 (Ryanne Dolan, Cloudera) Kafka Summit ...

confluent

Increasingly, organizations are relying on Kafka for mission critical use-cases where high availability and fast recovery times are essential. In particular, enterprise operators need the ability to quickly migrate applications between clusters in order to maintain business continuity during outages. In many cases, out-of-order or missing records are entirely unacceptable. MirrorMaker is a popular tool for replicating topics between clusters, but it has proven inadequate for these enterprise multi-cluster environments. Here we present MirrorMaker 2.0, an upcoming all-new replication engine designed specifically to provide disaster recovery and high availability for Kafka. We describe various replication topologies and recovery strategies using MirrorMaker 2.0 and associated tooling.

Securing Kafka

confluent

Disaster Recovery and High Availability with Kafka, SRM and MM2

Abdelkrim Hadjidj

In this talk, we will present Streams Replication Manager, a new open source Kafka mirroring solution designed specifically to provide disaster recovery and high availability for Kafka. We will describe and demo various replication topologies and recovery strategies using SRM and associated tooling. Finally, we will provide an update on the ongoing work to make this engine available for the Apache Kafka community as MirrorMaker2 (KIP-382).

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...

confluent

In order to maximize Kafka accessibility within an organization, Kafka operators must choose an authentication option that balances security with ease of use. Kafka has been historically limited to a small number of authentication options that are difficult to integrate with a Single Signon (SSO) strategy, such as mutual TLS, basic auth, and Kerberos. The arrival of SASL/OAUTHBEARER in Kafka 2.0.0 affords system operators a flexible framework for integrating Kafka with their existing authentication infrastructure. Ron Dagostino (State Street Corporation) and Mike Kaminski (The New York Times) team up to discuss SASL/OAUTHBEARER and it’s real-world applications. Ron, who contributed the feature to core Kafka, explains the origins and intricacies of its development along with additional, related security changes, including client re-authentication (merged and scheduled for release in v2.2.0) and the plans for support of SASL/OAUTHBEARER in librdkafka-based clients. Mike Kaminski, a developer on The Publishing Pipeline team at The New York Times, talks about how his team leverages SASL/OAUTHBEARER to break down silos between teams by making it easy for product owners to get connected to the Publishing Pipeline’s Kafka cluster.

Hardening Kafka Replication

confluent

(Jason Gustafson, Confluent) Kafka Summit SF 2018 Kafka has a well-designed replication protocol, but over the years, we have found some extremely subtle edge cases which can, in the worst case, lead to data loss. We fixed the cases we were aware of in version 0.11.0.0, but shortly after that, another edge case popped up and then another. Clearly we needed a better approach to verify the correctness of the protocol. What we found is Leslie Lamport’s specification language TLA+. In this talk I will discuss how we have stepped up our testing methodology in Apache Kafka to include formal specification and model checking using TLA+. I will cover the following: 1. How Kafka replication works 2. What weaknesses we have found over the years 3. How these problems have been fixed 4. How we have used TLA+ to verify the fixed protocol. This talk will give you a deeper understanding of Kafka replication internals and its semantics. The replication protocol is a great case study in the complex behavior of distributed systems. By studying the faults and how they were fixed, you will have more insight into the kinds of problems that may lurk in your own designs. You will also learn a little bit of TLA+ and how it can be used to verify distributed algorithms.

APACHE KAFKA / Kafka Connect / Kafka Streams

Ketan Gote

No data loss pipeline with apache kafka

Jiangjie Qin

The document discusses how to configure Apache Kafka to prevent data loss and message reordering in a data pipeline. It recommends settings like enabling block on buffer full, using acks=all for synchronous message acknowledgment, limiting in-flight requests, and committing offsets only after messages are processed. It also suggests replicating topics across at least 3 brokers and using a minimum in-sync replica factor of 2. Mirror makers can further ensure no data loss or reordering by consuming from one cluster and producing to another in order while committing offsets. Custom consumer listeners and message handlers allow for mirroring optimizations.

From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning

confluent

Apache Kafka - Martin Podval

Martin Podval

Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.

An Introduction to Apache Kafka

Amir Sedighi

Fundamentals of Apache Kafka

Chhavi Parasher

Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka

confluent

- Apache Kafka is a streaming platform that provides high availability, durability, and the ability to retain database-like data through features like log compaction. - It ensures reliability through configurable replication, automatic failover, and an in-sync replica process. - Log compaction allows Kafka to retain only the latest value for each message key in the log, useful for building indexes and retaining only updated records.

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

confluent

What's hot

Introducing Kafka's Streams API

confluent

Cruise Control: Effortless management of Kafka clusters

Prateek Maheshwari

Everything You Always Wanted to Know About Kafka's Rebalance Protocol but Wer...

confluent

A visual introduction to Apache Kafka

Paul Brebner

Kafka 101 and Developer Best Practices

confluent

Deploying Confluent Platform for Production

confluent

Kafka presentation

Mohammed Fazuluddin

Introduction to Kafka

Akash Vacher

Disaster Recovery with MirrorMaker 2.0 (Ryanne Dolan, Cloudera) Kafka Summit ...

confluent

Securing Kafka

confluent

Disaster Recovery and High Availability with Kafka, SRM and MM2

Abdelkrim Hadjidj

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...

confluent

Hardening Kafka Replication

confluent

APACHE KAFKA / Kafka Connect / Kafka Streams

Ketan Gote

No data loss pipeline with apache kafka

Jiangjie Qin

From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning

confluent

Apache Kafka - Martin Podval

Martin Podval

An Introduction to Apache Kafka

Amir Sedighi

Fundamentals of Apache Kafka

Chhavi Parasher

What's hot (20)

Introducing Kafka's Streams API

Cruise Control: Effortless management of Kafka clusters

Everything You Always Wanted to Know About Kafka's Rebalance Protocol but Wer...

A visual introduction to Apache Kafka

Kafka 101 and Developer Best Practices

Deploying Confluent Platform for Production

Kafka presentation

Introduction to Kafka

Disaster Recovery with MirrorMaker 2.0 (Ryanne Dolan, Cloudera) Kafka Summit ...

Securing Kafka

Disaster Recovery and High Availability with Kafka, SRM and MM2

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...

Hardening Kafka Replication

APACHE KAFKA / Kafka Connect / Kafka Streams

No data loss pipeline with apache kafka

From Message to Cluster: A Realworld Introduction to Kafka Capacity Planning

Apache Kafka - Martin Podval

An Introduction to Apache Kafka

Fundamentals of Apache Kafka

Similar to A Deep Dive into Kafka Controller

Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka

confluent

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

confluent

Kafka replication apachecon_2013

Jun Rao

The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.

The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...

HostedbyConfluent

KRaft is Kafka's implementation of the Raft consensus protocol that replaces Zookeeper for managing Kafka cluster metadata and electing controllers. It stores metadata in a replicated log with a single writer and allows fast controller failovers. The Quorum Controller runs on top of KRaft, caching metadata from the log. It provides ordered, consistent metadata updates and faster broker shutdowns and startups compared to the previous Zookeeper-based controller. KRaft will be available in early access in the next Kafka release while the Zookeeper dependency is deprecated and eventually removed.

Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...

confluent

We have been served well by Zookeeper over the years, but it is time for Kafka to stand on its own. This is a talk on the ongoing effort to replace the use of Zookeeper in Kafka: why we want to do it and how it will work. We will discuss the limitations we have found and how Kafka benefits both in terms of stability and scalability by bringing consensus in house. This effort will not be completed over night, but we will discuss our progress, what work is remaining, and how contributors can help. (Note that I am proposing this as a joint talk with Colin McCabe, who is also a committer on the Apache Kafka project.)

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Microservices interaction at scale using Apache Kafka

Ivan Ursul

This document discusses using Apache Kafka to enable communication between microservices at scale. It begins by describing how monolithic applications can be broken into independent microservices. Next, it covers common communication patterns for microservices, including shared databases, separate databases per service, and asynchronous messaging. The bulk of the document then focuses on Apache Kafka, describing it as a distributed publish-subscribe messaging system that is fast, scalable and durable. It covers how Kafka works, including its use of a commit log distributed across brokers, and common usage patterns such as event sourcing, change data capture, and Kafka Connect. Overall, the document promotes using Kafka as the backbone for event-driven communication between microservices.

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

HostedbyConfluent

Velocity 2019 - Kafka Operations Deep Dive

Gwen (Chen) Shapira

The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...

confluent

1. The document discusses various architectures for running Kafka in a multi-datacenter environment including running Kafka natively in multiple datacenters, mirroring data between datacenters, and using hierarchical Zookeeper quorums. 2. Key considerations for multi-DC Kafka include replication settings, consumer reconfiguration needs during outages, and handling consumer offsets and processing state across datacenters. 3. Native multi-DC Kafka is preferred but mirroring can be an alternative approach for inter-region traffic when latency is over 30ms or datacenters cannot be combined into a single cluster. Asynchronous mirroring acts differently than a single Kafka cluster and impacts operations.

Static Membership: Rebalance Strategy Designed for the Cloud (Boyang Chen,Con...

confluent

In this presentation, we introduce static membership (KIP-345) and share the story of adopting it at Pinterest. The static membership aims to improve the availability of stream applications, consumer groups and other applications built on top of it. The original rebalance protocol relies on the group coordinator to allocate entity ids to group members. These generated ids are ephemeral and will change when members restart and rejoin. For consumer based apps, this "dynamic membership" can cause a large percentage of tasks re-assigned to different instances during administrative operations such as code deploys, configuration updates and periodic restarts. For large state applications, shuffled tasks need a long time to recover their local states before processing and cause applications to be partially or entirely unavailable. At Pinterest, the group membership is stable between administrative operations. Motivated by this observation, we modified the Kafka's group management protocol allowing group members to provide persistent entity ids. Group membership remains unchanged based on those ids, thus no rebalance will be triggered. We can conveniently leverage Kubernetes or other cloud management frameworks to provide entity ids. By adopting static membership to the realtime infrastructure at Pinterest, applications resume processing only a few seconds after administrative operations finish. Previously with dynamic membership, it can take more than 30 minutes before applications resume. The talk is organized as follows: we first review Kafka's group management protocol and demonstrate high availability use cases that dynamic membership is unable to support. Then we share the design and adoption story of static membership. At the end, we do a live demo to show the impact of static membership.

Kafka Needs No Keeper

C4Media

Kafka is evolving to remove its dependency on Zookeeper. The Kafka Improvement Proposal 500 (KIP-500) aims to manage Kafka's metadata log with a self-managed Raft consensus algorithm and controller quorum rather than relying on Zookeeper. This will improve scalability, robustness, and make deployment easier. It will take multiple releases to fully implement KIP-500, beginning with removing Zookeeper from clients and ending with a release where Zookeeper is no longer required.

Grokking TechTalk #24: Kafka's principles and protocols

Grokking VN

Bài talk sẽ giới thiệu về Kafka, và đào sâu về các principles của Kafka, các thiết kế của Kafka để làm Kafka nhanh, scalable và độ ổn định cao. Bài talk cũng chia sẻ về cách Kafka servers tương tác với Kafka clients. Bài talk đào sâu vào internals của Kafka và phân tích tại sao các design decisions được thiết kế như vậy. Bài talk phù hợp cho các bạn software engineer đã, đang muốn tìm hiểu về các job queue, message queue khác nhau. Speaker: Nguyen Quang Minh - Software Engineer, Technical Lead @ Employment Hero - Contributor of `ruby-kafka` (the most popular Kafka client for Ruby)

SFBigAnalytics_20190724: Monitor kafka like a Pro

Chester Chen

Kafka operators need to provide guarantees to the business that Kafka is working properly and delivering data in real time, and they need to identify and triage problems so they can solve them before end users notice them. This elevates the importance of Kafka monitoring from a nice-to-have to an operational necessity. In this talk, Kafka operations experts Xavier Léauté and Gwen Shapira share their best practices for monitoring Kafka and the streams of events flowing through it. How to detect duplicates, catch buggy clients, and triage performance issues – in short, how to keep the business’s central nervous system healthy and humming along, all like a Kafka pro. Speakers: Gwen Shapira, Xavier Leaute (Confluence) Gwen is a software engineer at Confluent working on core Apache Kafka. She has 15 years of experience working with code and customers to build scalable data architectures. She currently specializes in building real-time reliable data processing pipelines using Apache Kafka. Gwen is an author of “Kafka - the Definitive Guide”, "Hadoop Application Architectures", and a frequent presenter at industry conferences. Gwen is also a committer on the Apache Kafka and Apache Sqoop projects. Xavier Leaute is One of the first engineers to Confluent team, Xavier is responsible for analytics infrastructure, including real-time analytics in KafkaStreams. He was previously a quantitative researcher at BlackRock. Prior to that, he held various research and analytics roles at Barclays Global Investors and MSCI.

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning

Guido Schmutz

Apache Kafka is a distributed streaming platform. It provides a high-throughput distributed messaging system with publish-subscribe capabilities. The document discusses Kafka producers and consumers, Kafka clients in different programming languages, and important configuration settings for Kafka brokers and topics. It also demonstrates sending messages to Kafka topics from a Java producer and consuming messages from the console consumer.

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

confluent

This document provides guidance on upgrading Kafka. It emphasizes the importance of upgrading early and often to the latest bugfix release in order to address security vulnerabilities and other bugs. It recommends using automated rolling upgrades to upgrade brokers one by one with zero downtime. It also outlines best practices like backing up configurations, reading release notes, and ensuring protocol compatibility when upgrading.

Kafka Technical Overview

Sylvester John

The document provides an overview of Kafka including its problem statement, use cases, key terminologies, architecture, and components. It defines topics as streams of data that can be split into partitions with a unique offset. Producers write data to brokers which replicate across partitions for fault tolerance. Consumers read data from partitions in a consumer group. Zookeeper manages the metadata and brokers act as the developers while topics are analogous to modules with partitions as tasks.

Kafka High Availability in multi data center setup with floating Observers wi...

HostedbyConfluent

The document discusses how to set up Kafka high availability in a multi-datacenter configuration using floating observers. Observers are regular brokers that are never part of the in-sync replica set for a topic. This allows forced replication to another data center for high availability. The document explains how to create topics that use observers, what happens to existing topics when observers are introduced, and how to distribute load evenly between data centers by preparing observers to float and using rack awareness.

Kafka Evaluation - High Throughout Message Queue

Shafaq Abdullah

This document summarizes Kafka's performance in handling data pipelines and ETL workloads. It discusses Kafka's high-level architecture, scalability, fault tolerance, and monitoring capabilities. The document also includes results from benchmark tests showing Kafka can process over 47 million transactions in under 6 minutes with latency under 2 milliseconds. It proposes using Kafka to integrate data pipelines between various systems and services at a company.

Similar to A Deep Dive into Kafka Controller (20)

Kafka Summit NYC 2017 - Deep Dive Into Apache Kafka

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

Kafka replication apachecon_2013

The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...

Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Microservices interaction at scale using Apache Kafka

Citi TechTalk Session 2: Kafka Deep Dive

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

Velocity 2019 - Kafka Operations Deep Dive

The Foundations of Multi-DC Kafka (Jakub Korab, Solutions Architect, Confluen...

Static Membership: Rebalance Strategy Designed for the Cloud (Boyang Chen,Con...

Kafka Needs No Keeper

Grokking TechTalk #24: Kafka's principles and protocols

SFBigAnalytics_20190724: Monitor kafka like a Pro

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

Kafka Technical Overview

Kafka High Availability in multi data center setup with floating Observers wi...

Kafka Evaluation - High Throughout Message Queue

More from confluent

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

Workshop híbrido: Stream Processing con Flink

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.

Build real-time streaming data pipelines to AWS with Confluent

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

Confluent & GSI Webinars series - Session 3

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

confluent

This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Data In Motion Paris 2023

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Confluent Partner Tech Talk with Synthesis

confluent

The Future of Application Development - API Days - Melbourne 2023

confluent

This document discusses the future of application development and key topics in streaming data and AI. It begins with an overview of streaming concepts like topics, streams, and tables. It then covers the Kappa architecture for stream processing using tools like Kafka Streams, ksqlDB, and Flink. The document also discusses challenges with generative AI models like handling private data, long-term context and memory, and integration into businesses. It concludes with recommendations to simplify architectures and use streaming as smart pipes to process raw and enriched data.

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

The Future of Application Development - API Days - Melbourne 2023

Recently uploaded

Nordic Marketo Engage User Group_June 13_ 2024.pptx

MichaelKnudsen27

Mariano G Tinti - Decoding SpaceX

Mariano Tinti

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

How to use Firebase Data Connect For Flutter

Daiki Mogmet Ito

5th LF Energy Power Grid Model Meet-up Slides

DanBrown980551

5th Power Grid Model Meet-up It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology. Power Grid Model The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services. Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability. Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization. What to expect For the upcoming meetup we are organizing, we have an exciting lineup of activities planned: -Insightful presentations covering two practical applications of the Power Grid Model. -An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024. -An interactive brainstorming session to discuss and propose new feature requests. -An opportunity to connect with fellow Power Grid Model enthusiasts and users.

Digital Marketing Trends in 2024 | Guide for Staying Ahead

Wask

https://www.wask.co/ebooks/digital-marketing-trends-in-2024 Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

Skybuffer SAM4U tool for SAP license adoption

Tatiana Kojar

Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool. SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.

GraphRAG for Life Science to increase LLM accuracy

Tomaz Bratanic

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

Building Production Ready Search Pipelines with Spark and Milvus

Zilliz

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on integration of Salesforce with Bonterra Impact Management. Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

June Patch Tuesday

Ivanti

Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

Alpen-Adria-Universität

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

Recently uploaded (20)

Nordic Marketo Engage User Group_June 13_ 2024.pptx

Mariano G Tinti - Decoding SpaceX

Introduction of Cybersecurity with OSS at Code Europe 2024

20240609 QFM020 Irresponsible AI Reading List May 2024

Artificial Intelligence for XMLDevelopment

How to use Firebase Data Connect For Flutter

5th LF Energy Power Grid Model Meet-up Slides

Digital Marketing Trends in 2024 | Guide for Staying Ahead

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

Skybuffer SAM4U tool for SAP license adoption

GraphRAG for Life Science to increase LLM accuracy

Presentation of the OECD Artificial Intelligence Review of Germany

Building Production Ready Search Pipelines with Spark and Milvus

20240607 QFM018 Elixir Reading List May 2024

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

June Patch Tuesday

Energy Efficient Video Encoding for Cloud and Edge Computing Instances

UiPath Test Automation using UiPath Test Suite series, part 6

How to Get CNIC Information System with Paksim Ga.pptx

A Deep Dive into Kafka Controller

1. A Deep Dive into Kafka Controller Jun Rao VP of Apache Kafka Co-founder of Confluent

2. Kafka Replication • Configurable replication factor • Tolerating f – 1 failures with f replicas • Automated failover topic1-part1 logs broker 1 topic1-part2 logs broker 2 topic2-part2 topic2-part1 logs broker 3 topic1-part1 logs broker 4 topic1-part2 topic2-part2 topic1-part1 topic1-part2 topic2-part1 topic2-part2 topic2-part1

3. High Level Data Flow in Replication broker 1 producer leader broker 2 follower broker 3 follower 4 2 2 3 commit ack topic1-part1 topic1-part1 topic1-part1 consumer 1

4. What’s controller 4 • One broker in a cluster acts as controller • Monitor the liveness of brokers • Elect new leaders on broker failure • Communicate new leaders to brokers

5. Controller election Zookeeper /controller  broker 0 Controller broker 0 broker 3broker 2broker 1

6. Partition state: stored in ZK, cached in controller Zookeeper /topic/t1/0  leader:1 /topic/t1/1  leader:3 … /topic/t1/9  leader:2 Controller broker 0 broker 3broker 2broker 1

7. Controlled shutdown SIG_TERM Zookeeper Controller 1 2 broker 2 part t-0: follower part t-1: follower broker 1 part t-0: leader part t-1: leader broker 0 Zookeeper Controller 3 5 broker 2 part t-0: leader part t-1: leader broker 1 part t-0: part t-1: broker 0 4 /topics/t/0  2 /topics/t/1  2

8. Issues with controlled shutdown (pre 1.1) Zookeeper Controller 3 5 broker 0 4 Writes to ZK are serial Impact: longer shutdown time Communication of new leaders not batched Impact: client timeout broker 2 part t-0: leader part t-1: leader broker 1 part t-0: part t-1: /topics/t/0  2 /topics/t/1  2

9. Controller failover Zookeeper /controller  broker 0 Controller broker 0 broker 3broker 2broker 1 1

10. Controller failover Controller broker 0 broker 3broker 2broker 1 1 2 Controller Zookeeper /controller  broker 2

11. Controller failover Zookeeper /controller  broker 2 /topic/t1/0  leader:1 /topic/t1/1  leader:3 … /topic/t1/9  leader:2 Controller broker 0 broker 3broker 2broker 1 1 2 3 Controller

12. Issues with controller failover (pre 1.1) Controller broker 0 broker 3broker 2broker 1 1 2 3 Controller Reads from ZK are serial Impact: availability Zombie old controller Impact: inconsistency Zookeeper /controller  broker 2 /topic/t1/0  leader:1 /topic/t1/1  leader:3 … /topic/t1/9  leader:2

13. Performance improvements in 1.1 13 • Controller uses async ZK api for reads/writes • Controller communicates new leaders to brokers in batches part 1 part 2 part 3 part 4 part 1 part 2 part 3 part 4 Old (serial): New (pipelined):

14. /topics/t/0  2 /topics/t/1  2 Controlled shutdown (post 1.1) Zookeeper Controller 3 5 broker 0 4 Writes to ZK pipelined Communication of new leaders batched broker 2 part t-0: leader part t-1: leader broker 1 part t-0: part t-1:

15. Controller failover (post 1.1) Controller broker 0 broker 3broker 2broker 1 1 2 3 Controller Reads from ZK pipelined Zookeeper /controller  broker 2 /topic/t1/0  leader:1 /topic/t1/1  leader:3 … /topic/t1/9  leader:2

16. Results for controlled shutdown 16 • 5 ZK nodes and 5 brokers on different racks • 25K topics, 1 partition, 2 replicas • 10K partitions per broker Kafka 1.0.0 Kafka 1.1.0 Controlled shutdown time 6.5 minutes 3 seconds

17. Results for controller failover 17 • 5 ZK nodes and 5 brokers on different racks • 2K topics, 50 partitions, 1 replica • Controller failover: reload100K partitions from ZK Kafka 1.0.0 Kafka 1.1.0 State reload time 28 seconds 14 seconds

18. Fencing zombie requests from controller 18 • Zombie controller • ZK session expiration: better handling in the controller (1.1) • Controller path deletion: writes to ZK conditioned on controller epoch (2.1) • Zombie request from broker restart • Broker epoch (KIP-380 in 2.2) • Also fixed the missing ZK watcher issue

19. Summary • Significant performance improvement in controller in 1.1 • Allow 10X more partitions in a Kafka cluster • Better fencing of zombie requests from controller (1.1, 2.1, 2.2) • More details and remaining work in KAFKA-5027

20. Q/A • Acknowledgment: Onur Karaman, Manikumar Reddy, Prasanna Gautam, Ismael Juma, Mickael Maison, Sandor Murakozi, Rajini Sivaram,Ted Yu, Zhanxiang Huang

Editor's Notes

Leaders spread over brokers. Another picture. Animated transitioin.

A Deep Dive into Kafka Controller

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to A Deep Dive into Kafka Controller

Similar to A Deep Dive into Kafka Controller (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

A Deep Dive into Kafka Controller

Editor's Notes