Apache Kafka: Next Generation Distributed Messaging System

www.edureka.co/apache-kafka
Apache Kafka : Next Generation Distributed Messaging System

What will you learn today ?
 What is Apache Kafka ?
 Architecture of Kafka
 Multiple ways of setting Kafka cluster
 Comparing Kafka with other messaging systems
 Hands-On : Getting started with Kafka

Data : The Ingredient
Data is the main ingredient of Internet applications and typically includes the following :
 Page visits and clicks
 User activities
 Events corresponding to logins
 Social networking activities such as likes, shares, and comments
 Application specific metrics (e.g. logs, page load time, performance etc.)

Need : Real Time Analytics
In todays applications, activity data has become a part of production data and is used to run
analytics in real time. These analytics can be:
 Delivering advertisements to the masses
 Any abnormal user behavior or application hacking
 Search-based on relevance
 Recommendations based on popularity

Messaging Systems
Messaging systems provide seamless integration among distributed applications with the
help of messages, that are shared between them
In the present big-data era, the very first challenge is to collect the data as it is a huge and the
second challenge is to analyze it, one way to solve this problem is by using messaging systems
Problem :
Solution :

Apache Kafka
 Apache Kafka is a distributed publish-subscribe messaging system
 Originally developed at LinkedIn and later on became a part of Apache project
 Kafka is fast, scalable, durable and distributed by design

Kafka Architecture
Producer
ConsumerConsumerConsumer
Producer Producer
Kafka Cluster
 A stream of messages of particular category is called a topic. Producers publish messages to a topic
 A Producer can be any application who can publish messages to a topic
 Consumers subscribe to topics and consume the messages
 Kafka cluster is a set of servers, each of which is called a broker
Kafka Architecture

ZooKeeper and Kafka
 Each Kafka broker coordinates with other Kafka brokers using ZooKeeper
 Producers and Consumers are notified by ZooKeeper service about the presence
of new broker in Kafka system or failure of the broker in Kafka system

Kafka Clusters
With Kafka we can create multiple types of clusters, such as the following :
 Single node single broker cluster
 Single node multiple broker cluster
 Multiple nodes multiple broker cluster

Single Node Single Broker Cluster
Producer
Producer
Producer
Consumer
Consumer
Consumer
Kafka Broker
ZooKeeper
Single Node Single Broker Cluster

Single Node Multiple Broker Cluster
Producer
Producer
Producer
Consumer
Consumer
Consumer
ZooKeeper
Single Node Multiple Broker Cluster
Broker 1
Broker 2
Broker 3

Multiple Node Multiple Broker Cluster
Producer
Producer
Producer
Consumer
Consumer
Consumer
ZooKeeper
Multiple Node Multiple Broker Cluster
Broker 1
Broker 2
Broker 1
Broker 2
Node 1
Node 2

Kafka @ LinkedIn
LinkedIn Newsfeed is powered by Kafka
LinkedIn recommendations are powered by Kafka

Kafka @ LinkedIn
LinkedIn notifications are powered by Kafka
Apart from this LinkedIn uses Kafka for many
other purposes like log monitoring, performance
metrics, search improvement etc.

Who else uses Kafka ?
DataSift uses Kafka as a collector of monitoring events and to track user’s
consumption of data streams in real time
Wooga uses Kafka to aggregate and process tracking data from all their
facebook games (hosted at various providers) in a central location
Spongecell uses Kafka to run their entire analytics and monitoring pipeline
driving both real-time and ETL applications
Loggly is the world's most popular cloud-based log management. It uses
Kafka for log collection
An exhaustive list of companies using Kafka can be found here : https://cwiki.apache.org/confluence/display/KAFKA/Powered+By

But what about
other messaging
systems e.g.
ActiveMQ,
RabbitMQ etc.

Comparing Messaging Systems
Kafka has a more efficient storage
format.
On average, each message had an
overhead of 9 bytes in Kafka,
versus 144 bytes in ActiveMQ

Comparing Messaging Systems
In both ActiveMQ and RabbitMQ
brokers maintain delivery state of
every message by writing to disk
but in case of Kafka there is no
disk write which makes it fast

Hands-on
Getting Started with Kafka

References
Apache Kafka :
http://kafka.apache.org/
Kafka Papers :
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations
Powered by Kafka :
https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
LinkedIn Performance Insights :
https://engineering.linkedin.com/samza/real-time-insights-linkedins-performance-using-apache-samza

Survey
Your feedback is vital for us, be it a compliment, a suggestion or a complaint. It helps us to make your
experience better!
Please spare few minutes to take the survey after the webinar.

Course Details
Edureka's Apache Kafka course:
• Introduction of course
• Online Live Classes: 15 hours
• Assignments: 25 hours
• Project: 20 hours
• Lifetime Access + 24 X 7 Support
Go to www.edureka.co/apache-kafka
Batch starts from 07 November (Weekend Batch)

Thank You …
Questions/Queries/Feedback
Recording and presentation will be made available to you within 24 hours

Apache Kafka: Next Generation Distributed Messaging System

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Apache Kafka: Next Generation Distributed Messaging System

Similar to Apache Kafka: Next Generation Distributed Messaging System (20)

More from Edureka!

More from Edureka! (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka: Next Generation Distributed Messaging System