Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

1. Yogesh..BG
27th Nov 2015

2. • Introduction
• Features
• Architecture
• Storage
• Topic
• Producer and consumer
• Broker
• Zookeeper
• Use Cases
• Comparison with RMQ & AMQ
10/25/2016 Confidential 2
Agenda

3. Introduction
10/25/2016 Confidential 3
Open Source messaging Q written in scala.
LinkedIn Corp -> Apache
Producer1
Producer2
Queue
Consumer1
Consumer2
Consumer3

4. Features
10/25/2016 Confidential 4
• Brokered
• Distributed
• high throughput for publish and subscribe
• easy scalable
• fast
• replicated commit log service
• partitioned
• stores messages on disk
• In order delivery, per partition.

5. Architecture
10/25/2016 Confidential 5
Elements are:
• Messages
• Producer
• Consumer
• Topic
• Broker [Cluster]
• Zookeeper

6. Broker
10/25/2016 Confidential 6
• Stateless brokers.
• Transient System – deletes data
• Time-based SLA for the retention policy.
• Zero copy

7. Storage
10/25/2016 Confidential 7

8. Topic
10/25/2016 Confidential 8
• Unique identification for messages – offset
• Consumer can change the offset to re consume or skip the message
• Replicated among the configurable number of servers.
• Retention policy
• Parallelism at partitions level

9. Producers and Consumers
10/25/2016 Confidential 9
Producers:
• Decides which message to which partition - LB.
• Batch the messages
• Async Send
Consumers:
• Pull
• Queuing and pub-sub
• Consumer groups - cluster of consumers
• Ordered per partition

10. Zookeeper
10/25/2016 Confidential 10

11. Use Cases
10/25/2016 Confidential 11
• Messaging
• Website activity tracking
• Metrics
• Log Aggregation
• Stream Processing
• Event Publishing

12. Comparison
10/25/2016 Confidential 12

13. Sooo fast… How come?
10/25/2016 Confidential 13
1. No wait.
2. efficient storage format.
Header -> 9bytes (144 bytes in AMQ(As per JMS)).
3. Zero copy
4. Compressing multiple messages – message set

14. Thank You
Yogesh..BG
10/25/2016 Confidential 14