Kafka overview and use cases

•Download as PPTX, PDF•

3 likes•958 views

Overview of kafka, how it works, components of kafka, use cases. Kafka at LinkedIn. Download the slides to see animations explaining how the components fit. This was presented on kafka meetup help on June 11, 2016 @LinkedinBangalore ofc

Technology

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka - Overview
Indrajeet Kumar
Site Reliability Engineer at LinkedIn

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved. 2
So what is it?
It is a high-throughput, low-latency messaging system

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
And who uses it?
3

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
What for?
•Messaging
•Website Activity Tracking
•Metrics
•Log Aggregation
•Stream Processing
•For fun ;)
4

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
So how does it work?
▪ Components
– Producer
– Broker
▪ Topic
▪ Partition
– Consumer
5

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Broker
producer producer producer
B2B1P1 P2P1
R P2
R
6
consumer consumer consumer

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The consumer
7
consumer
B2B1P1 P2
B3P3
C1 C2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
The Producer
8
Producer
B2B1P1 P2
B3P3
P1 P2
P1
R

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Attributes of a Kafka Cluster
▪ Durable
▪ Scalable
▪ Low Latency
▪ Finite Retention
▪ No single point of failure
9

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Kafka At LinkedIn
▪ Multiple Datacenters, Multiple Clusters
▪ Mirroring between clusters
▪ Message Types
– Metrics
– Tracking
– Queuing
▪ Data transport from applications to Hadoop, and back
10

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Some numbers!
▪ 1800+ Broker machines
▪ 79K+ Topics
▪ 1.1M+ Partitions
▪ 1.3 Trillion messages per day
▪ 330 Terabytes in/day
▪ 1.2 Petabytes out/day
▪ Peak load for a single cluster
– 2 million messages/sec
– 4.7 Gigabits/sec inbound
– 15 Gigabits/sec outbound
11

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.
Questions
12

SITE RELIABILITY ENGINEERING©2016 LinkedIn Corporation. All Rights Reserved.

What's hot

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Watch this talk here: https://www.confluent.io/online-talks/best-practices-for-streaming-iot-data-with-MQTT-and-apache-kafka-on-demand Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges. In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.

Best Practices for Streaming IoT Data with MQTT and Apache Kafka®

confluent

Presented at Kafka Summit 2016 Operating out of multiple datacenters is a large part of most disaster recovery plans, but it brings extra complications to our data pipelines. Instead of having a straight path from front to back, it now has forks and dead ends and odd little use cases that don’t match up with a perfect view of the world. This talk will focus on how to best utilize Apache Kafka in this world, including basic architectures for multi-datacenter and multi-tier clusters. We will also touch on how to assure messages make it from producer to consumer, and how to monitor the entire ecosystem.

More Datacenters, More Problems

Todd Palino

While Apache Kafka is designed to be fault-tolerant, there will be times when your Kafka environment just isn’t working as expected. Whether it’s a newly configured application not processing messages, or an outage in a high-load, mission-critical production environment, it’s crucial to get up and running as quickly and safely as possible. IBM has hosted production Kafka environments for several years and has in-depth knowledge of how to diagnose and resolve problems rapidly and accurately to ensure minimal impact to end users. This session will discuss our experiences of how to most effectively collect and understand Kafka diagnostics. We’ll talk through using these diagnostics to work out what’s gone wrong, and how to recover from a system outage. Using this new-found knowledge, you will be equipped to handle any problem your cluster throws at you.

Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...

HostedbyConfluent

Describing the convenience of building an event driven application using stream processing and leveraging the power of KSQL. Events model our lives and actions be it machine or human generated. Apache Kafka used with KSQL help us transform and process events, using KSQL to express what we want to do with the respective data. In this talk, we will be doing a live demo with a custom made application relaying any sort of data to an internal system/application/rest proxy, this will publish events which in turn will be processed and transformed via KSQL queries and analysed using a visual tool. All the while describing how just simple queries (continuous queries in terms of KSQL) can work the magic of complete applications. We will be discussing broadly what streams and stream processing is and how does KSQL come into the picture to help us make our lives easier. This can be leveraged in any kind or nature of industry.

Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter

HostedbyConfluent

Eine weitere neue sicherheitsrelevante Funktion in Confluent Platform 5.4 sind Structured Audit Logs. Jetzt ist natürlich alles in Kafka ein Log, aber Kafka protokolliert nicht, was Kafka mit Kafka macht - nur das, was in einen Topics geschrieben wird. Im dritten Teil der Deep Dive Sessions besprechen wir neben den Structured Audit Logs außerdem die "Weiterentwicklung" der bereits bekannten Schema Registry: Die Schema Validation agiert auf dem Topic-Level und stellt sicher, dass jede einzelne Message, die zu einem bestimmten Topic erstellt wird in der Schema Registry überprüft wird. Mehr dazu erklären wir in unserem Deep Dive #3.

Deep Dive Series #3: Schema Validation + Structured Audit Logs

confluent

In this session we share our experience of building a real-time data pipelines at Tencent PCG - one that handles 20 trillion daily messages with 700 clusters and 100Gb/s bursting traffic from a single app. We discuss our roadmap of enhancing Kafka to break its limits in terms of scalability, robustness and cost of operation. We first built a proxy layer that aggregates physical clusters in a way agnostic to the clients. While this architecture solves many operational problems, it requires significant development to stay future-proof. With retrospection with our customer and careful study of the ongoing work from the community, we then designed a region federation solution in the broker layer, which allows us to deploy clusters at a much larger scale than previously possible, while at the same time providing better failure recovery and operability. We discuss how we make this development compatible with KIP-500 and KIP-405, and the two KIP (693, 694) that we submitted for discussion.

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

HostedbyConfluent

Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services. At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure. This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

HostedbyConfluent

(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018 When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project. The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge. After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform. Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.

Creating an Elastic Platform Using Kafka and Microservices in OpenShift

confluent

Connected and Automated Vehicle education (CAVe)-in-a-box is an educational tool developed under the FHWA Workforce Development project. CAVe-in-a-box is an interconnected set of Intelligent transportation system equipment that was developed to serve as training and educational resources for the emerging Intelligent transportation system workforce. (CAVe)-in-a-box uses Kafka for real-time data streaming, data processing, visualization, and data collection. In this talk, we explore how Kafka is being used in cutting-edge connected and automated vehicle research.

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...

HostedbyConfluent

Quality Matters … and as event-driven architectures (EDA) become increasingly popular in the microservices space, ensuring the delivery and performance of your EDA increases in importance. But while it’s powerful architecture, it does come with its challenges, especially from a testing perspective. For example, most organizations are not reliant on Kafka alone, but a multitude of interconnected APIs like REST, GraphQL and gRPC. One of the questions that arise from this challenge: How do you build end-to-end tests when the APIs are completely different technologies—without relying on fragile scripts? In our talk, we’ll tackle this question and many more when it comes to the testing of Apache Kafka endpoints and your services architecture. We’ll cover what makes testing in EDA difficult; technologies that can help you; and how we at SmartBear are thinking about these testing problems and, most importantly, how we are trying to solve for them.

Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...

HostedbyConfluent

As an AWS shop, Zillow engineering teams have been using various messaging and streaming services for years. As Zillow 2.0 piled through, new requirements and pain points made us rethink our streaming stack. The need for high data quality, decoupling producers & consumers and real time homes data called for a new platform which would empower developers, enable data governance and reduce incidents caused by bad data. In this session, you will learn why Zillow decided to go with Kafka for that platform, what tools we built to meet developers where they are and what common challenges you could face as you migrate other streaming solutions to Kafka.

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...

HostedbyConfluent

Elastically Scaling Kafka Using Confluent

confluent

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

What is Apache Kafka and What is an Event Streaming Platform?

confluent

How many connects can you run in a single cluster? Disney + Hotstar runs over 10 different connect clusters with over 2000+ connectors. In this talk, we share our experience of running Kafka connect at scale. We will walk through our decisions of using one cluster vs many and how the improvements in the connect ecosystem like incremental rebalancing have allowed us to scale to thousands of connects. We will also discuss challenges with scaling up & down connect workers while keeping the ecosystem stable & present a wishlist of the missing features in this distributed task framework.

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

HostedbyConfluent

Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes. Legacy systems however pose severe limits in addressing critical business needs, as well as seizing opportunities for future growth. Digital Integration Hub (DIH) is a quite popular architecture to modernize legacy systems allowing the IT to evolve faster and better integrate modern technologies. In this talk we’ll: - Review DIH general architecture - Illustrate different patterns to offload data from legacy systems to Kafka with pros and cons in term of consistency and latency - Present different ways to consume data offloaded from legacy system to Kafka and their fit with different use cases - Show how to close the loop and use Kafka and CQRS pattern to handle all the data moving backward from applications to legacy systems

From legacy systems to microservices and back | Andera Gioia, Quantyca

HostedbyConfluent

Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model. Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance. Speakers: Justin Cunningham Technical Lead, Software Engineer, Yelp Gehrig Kunz Technical Product Marketing Manager, Confluent

How Yelp Leapt to Microservices with More than a Message Queue

confluent

At LinkedIn, the Kafka infrastructure is run as a service: the Streaming team develops and deploys Kafka, but is not the producer or consumer of the data that flows through it. With multiple datacenters, and numerous applications sharing these clusters, we have developed an architecture with multiple pipelines and multiple tiers. Most days, this works out well, but it has led to many interesting problems. Over the years we have worked to develop a number of solutions, most of them open source, to make it possible for us to reliably handle over a trillion messages a day.

Multi tier, multi-tenant, multi-problem kafka

Todd Palino

AWS Api Gateway by Łukasz Marchewka Scalacc

Scalac

Kafka has fast become the center of streaming analytics applications in the modern digital enterprise. Kafka operates in the context of a broad ecosystem of data lifecycle components which need a consistent platform of security, monitoring, management and governance. This problem becomes paramount when your streaming architectures go hybrid by spanning from on-premises to the cloud. Throw in the reality of a multi-cloud setup that a lot of enterprises are facing and now, you have a complex streaming architecture that is difficult to operationally manage, monitor, secure or govern. Cloudera remains committed to an open community driven approach and increasing the ease of use and visibility for Kafka based solutions. Attend this session to understand more about how streaming architectures can be extended easily to the hybrid cloud and multi-cloud. Also, learn about our plans for further community contributions.

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

HostedbyConfluent

What's hot (20)