This document provides an overview of Apache Kafka. It begins with background on messaging systems and defines the key types as point-to-point and publish-subscribe. It then defines Kafka as a distributed commit log that evolved from a messaging queue to a full streaming platform. The benefits of Kafka are outlined as reliability, scalability, durability, and performance. Core concepts are explained including topics, partitions, producers, consumers, brokers, clusters, and Zookeeper. An example Kafka deployment is described with brokers, topics, partitions, producers, and consumer groups. Finally, steps to produce the example are listed.

1. Kafka
10000 Feet View
BOUTKHOURST Youness
IBM A&T
07/11/2018

2. Agenda
- Some Background
- What is Kafka
- Kafka Benefits
- Kafka Core Concepts
- Example
- Steps to Produce the Example

3. Some Background. Messaging Systems
An enterprise messaging system (EMS) or messaging system in brief
[1]
is a set of published enterprise-wide
standards that allows organizations to send semantically precise messages between computer systems.
EMS systems promote loosely coupled architectures that allow changes in the formats of messages to have
minimum impact on message subscribers.
EMS systems are facilitated by the use of structured messages (such as using XML or JSON), and appropriate
protocols, such as DDS, MSMQ, AMQP or SOAP with web services.
Source : Wikipedia

4. Some Background. Types Of Messaging
Systems
Point-To-Point :
In this type of Messaging Systems a message can be consumed ONLY ONCE.
Producer Consumer 2
Consumer 3
Consumer 1
Queue X
X

5. Some Background. Types Of Messaging
Systems
Publish-Subscribe :
Messages are persisted in a topic and consumers can consume all the messages in that topic.
Producer Consumer 2
Consumer 3
Consumer 1
Queue

6. What is Kafka?
Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log.
Kafka has evolved from messaging queue to a full-fledged streaming platform.
Kafka has 3 main characteristics :
● Publish + Subscribe : highly scalable, fault tolerant distributed system
● Store : Can be used as a source of truth like any other database
● Process : Manipulate that data as it arrives. This is what makes it a powerful streaming platform.

7. Kafka Benefits
● Reliability − Distributed, partitioned, replicated and fault tolerance.
● Scalability − Scales easily without down time..
● Durability − Uses Distributed commit log which means messages persists on disk as fast
as possible, hence it is durable..
● Performance − High throughput for both publishing and subscribing messages. It
maintains stable performance even many TB of messages are stored.

8. Kafka Core Concepts
- Topics are to Kafka what Tables are to Relational Databases
- One Topic holds a set of messages of the same category
- A Topic can be partitioned into multiple partitions
Topics :

9. Kafka Core Concepts
- A Partition is a single part of a Topic
- Helps scale horizontally across multiple Servers
- Each partition can be hosted on a different server
- Partition Strategies : Round Robin, Static, Hash
Partitions :
9 8 7 6 5 4 3 2 1 0
8 7 6 5 4 3 2 1 0
10 9 8 7 6 5 4 3 2 1 0
Partition 0
Partition 1
Partition 2
Messages
Topic

10. Kafka Core Concepts
- Producers are the publishers of messages
- Messages can be redirected to a specific partition using:
- The message key and a partitioner that will generate a hash of the key and map it to a specific
partition
- Producers send messages to a Leader Partition
Producers :

11. Kafka Core Concepts
- A consumer subscribes to one or more topics
- Consumers read messages sequentially from a Leader Partition
- Keeps track of the messages it has already consumed (via offsets)
- Work as part of a consumer group to consume a topic
- The group ensures that each partition is only consumed by one member of the group
Consumers :
9 8 7 6 5 4 3 2 1 0
8 7 6 5 4 3 2 1 0
10 9 8 7 6 5 4 3 2 1 0
Consumer 1
Consumer 2
Messages
Consumer
Group

12. - A single Kafka Server/Node
- Assigns offsets to messages received from producers
- A single broker can handle thousands of partitions and millions of messages/s
Kafka Core Concepts
Brokers :

13. - A set of kafka brokers
- One broker in the cluster acts as the Controller
Kafka Core Concepts
Clusters :
- Manages partition assignments
- Monitors Brokers
- Maintaining the leader/follower relationship for all the partitions
Controllers :

14. - Electing a controller
- Cluster membership
- Topic configuration
Kafka Core Concepts
Zookeeper :

15. Example

16. Example
In this example we have:
- 3 Brokers : Broker 1, Broker 2, Broker 3
- 1 Topic with 3 partitions and a replication factor of 3:
- The main partitions are : Partition 0 Leader, Partition 1 Leader, Partition 2 Leader
- Each Partition that is a Leader has 2 replicas (factor of replication is 3)
- The replicas never reside on the same server as the Leader Partition
- 1 Producer
- 2 Consumer Groups: Group 1, Group 2
- Group 1 has 3 consumers : So Each Consumer is assigned a Partition
- Group 2 has 2 consumers : One consumer takes charge of consuming messages from 2 partitions
- Producer and Consumers always write/read from Leader Partitions

17. Steps to Produce the Example
Read this File : https://github.com/youbout/kafka-10000-feet