Apache Kafka is an open source distributed streaming platform developed by LinkedIn and donated to the Apache Software Foundation. It allows for real-time streaming and processing of data and is commonly used for building real-time data pipelines and streaming apps. Key components include the Kafka broker for hosting streams of records called topics, which can be produced to and consumed from by clients using the Kafka producer and consumer.

1. Apache Kafka
Degendra Sivakoti
MSc IT | Islington College

2. Apache Kafka
Apache Kafka® is an open sourced distributed streaming platform.

3. History
- Developed by LinkedIn / Microsoft
- Open sourced in early 2011
- Donated to Apache Software Foundation on 2012
- Latest release and stable version 2.5.0
- Written in Scala/ Java

4. Why Kafka?

8. Major Features
- Kafka as Messaging System
- Kafka for Stream Processing

9. Core Components
- Zookeeper
- Core dependency as of version 2.5.0
- Broker
- Kafka server
- Producer
- Produces message to topic partition
- Consumer
- Listens to topic partition
- Topic
- Partition

10. Topic
- A topic can have one or more partitions
- Topics are always multi-subscriber
- Partitions (logs):
- One leader and zero or more followers
- Distributed

11. Producer
- Produce message to a topic
- Select topic partition
- Normally, uses round-robin technique to balance load

12. Consumer
- Consumer listens to one or more topics
- Consumer belongs to a consumer group
- Only on consumer from a consumer group
can consume message from a topic
partition.
- Rebalancing
- More consumers in a group than partition
means idle consumers

13. Example

14. Kafka and BigData
- Process existing big data
- Sitting on HDFS
- Sitting in a database
- How does new data get into your cluster?
- New log from web servers
- New sensor data from IoT systems
- New stock trades
- Use for data ingestion in Hadoop system

15. Kafka and Hadoop Ecosystem

16. Questions?

17. Thank you!