Kafka is a distributed streaming platform that allows for publishing and subscribing to streams of records, such as logs and metrics, across a cluster of servers. It provides several key features including polyglot consumers and producers, scalability, configurable message retention, and guaranteed ordering. The document demonstrates how to set up a local Kafka cluster using Zookeeper for management and includes code examples for producers and consumers.

1. Kafka
Streaming Data Platform

2. Traditional Messaging System
• Queue
• Topic
• After Consumed Removed
• Out of order messaging

3. What is Kafka
• Messaging system
• Polyglot Consumers / Producers
• Topics and Partitions
• Scalable
• Configurable Message Retention
• Guaranteed order

4. Use Cases
• Ordered Messaging
• Log Aggregation
• Metrics
• Web Activity Tracking
• Stream Processing

5. Kafka Brokers – Clusters and Replication
• Topics can be replicated
• Data stored across various nodes
• Kafka clusters require broker.id=0
• Zookeeper
• Offsets
• Topic names
• partitions

6. Demo – Local Kafka
• Startup zookeeper
• bin/zookeeper-server-start.sh config/zookeeper.properties
• Start kafka
• bin/kafka-server-start.sh config/server.properties

7. Demo Command line tools
• bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-
factor 1 --partitions 1 --topic test
• bin/kafka-topics.sh --list --zookeeper localhost:2181
• bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
• bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic
test --from-beginning

8. Example Producer
• <CODE>

9. Example Consumer
• <CODE>

10. Deployment Options
• Stand alone deployment
• Confluent.io
• Horton Works
• AWS

11. HortonWorks Data Platform on AWS
Big Data in a one stop shop

12. Determine Cluster Sizing
• Implement a producer and consumer
• Use your data structures
• 3 Zookeeper nodes and 3 Kafka nodes
• Java Heap = 2GB
• Network Saturation (1 gigabit / 10 gigabit)
• Avro Data Serialization

13. Producer for testing throughput
• <CODE>

14. Architectural Possibilities
• Streaming data platform
• Common interface
• High throughput

15. WARNING
• Kafka 0.8.x has a major bug…deletes data
• Make sure to use 0.9.0.x

16. Question & Answer
bryancjacobs@gmail.com