A talk inside ThoughtWorks about Apache Kafka

1. Apache Kafka
Free Friday
Luiza Souza / Otávio Carvalho
lsouza@thoughtworks.com
ocarvalh@thoughtworks.com

2. Apache Kafka
● Apache Kafka is a distributed messaging system
○ Provides fast, highly scalable and redundant messaging
through a pub-sub model
● It was built at LinkedIn to be used as central hub for all of the
messaging communication between their systems
● Focus on scalability and fault tolerance

3. Motivation
● Microservices
○ "In short, the microservice architectural style is an approach to developing a
single application as a suite of small services, each running in its own process
and communicating with lightweight mechanisms, often an HTTP resource
API. These services are built around business capabilities and independently
deployable by fully automated deployment machinery."- Martin Fowler
● Monolith First
○ Using microservices as a way to decompose monolitical
infrastructures
● Message Queues
○ Asynchronous processing
○ Decoupling
○ Load balancing
○ Scalability

4. How is it different?
● High throughput
○ Millions of events per second per node
● Fault-tolerance guarantees
○ Relies on Apache Zookeeper for detection of node failures
and leader election
○ Maintains a structure called ISR (In-Sync Replica Set) in order
to be able to tolerate node failures
○ (Claims to) Guarantees up to f failures with f+1 replicas
without losing data
● Distributed
○ More nodes can be included and the system keeps its
high-performance and fault-tolerance capabilities

5. ● Broker-centric (AMQP)
○ AMQP implementations are usually broker-centric
○ Focus on delivery guarantees between producers/consumers
○ Transient preferred over durable messages
○ Use the broker itself to maintain state of what is consumed
(via message acknowledgements)
● Producer-centric (Kafka)
○ Partition a fire hose of event data into durable message
brokers with cursors (pointers)
○ Support to batch consumers that may be offline, or online
consumers that want messages at low latency
○ Doesn't have message acknowledgements, it assumes the
consumer tracks what has been consumed so far
Comparison with AMQP

6. Kafka Terminology
● Producers
○ Processes that publishes
msgs to topics
● Consumers
○ Processes that reads
msgs from topics
● Topic
○ Name of the feed to which
msgs are published
● Broker
○ Process running on a
single machine
● Cluster
○ Group of brokers working
together

7. Kafka Terminology
● Partitions
○ Subdivision of Topics
■ Scalability
■ Load balancing
○ Consumers control
their own offsets

8. ● Replication
○ In-Sync-Replica (ISR) sets
Kafka Terminology
Figure 1. A Kafka cluster with 4 brokers, 1 topic and 2 partitions, each
with 3 replicas

9. Use Cases
● Messaging
● Distributed log / Log aggregation
● Change Data Capture
● Stream Processing / Event Sourcing

10. Use Cases - Messaging
● Messaging
○ Simple Queueing
■ e.g. Queue for sending e-mails
○ Tracking user events
○ Near real-time metrics

11. Use Cases - Distributed Log
● Distributed log / Log aggregation
○ LinkedIn usage
■ The whole platform is built around a central log
■ 13 million messages/sec, 15 gigabytes per sec
■ Over 1100 brokers in more than 60 clusters

12. Use Cases - Change Data Capture

13. Use Cases - Stream Processing
● Stream Processing / Event Sourcing
LinkedIn's example Netflix's example

14. DEMO
14

15. ISSUES
15

18. Issues
● CAP theorem (Consistency, Availability, Partitioning)
○ "You can't sacrifice partition tolerance"
● Jepsen tests (@aphyr)
○ In order to force failures on Kafka, it needs to shrink ISR
(In-Sync Replica Set) to one node (the master) and then lose
the master itself
■ It will cause a leader election and a new leader will be
elected
● It causes Kafka to lose ~50% of writes done during this
partition time
■ Kafka users usually set a replication factor of 2 or 3
replicas for each partition on a given topic

20. THANK YOU
20
Luiza Souza / Otávio Carvalho
lsouza@thoughtworks.com
ocarvalh@thoughtworks.com

21. ● https://aphyr.com/posts/315-jepsen-rabbitmq
● https://aphyr.com/posts/293-jepsen-kafka
● https://thoughtworks.jiveon.com/people/tbartlet/blog/2015/11/
02/project-metamorphosis-with-kafka-spark
● https://thoughtworks.jiveon.com/message/1013489
● https://medium.com/@ikem/event-sourcing-and-cqrs-a-look-at-
kafka-e0c1b90d17d8#.x4f9ezrwn
● https://martin.kleppmann.com/2016/01/29/event-sourcing-stre
am-processing-at-ddd-europe.html
● http://microservices.io/patterns/microservices.html
● http://martinfowler.com/articles/microservices.html
● https://engineering.linkedin.com/kafka/running-kafka-scale
● https://engineering.linkedin.com/kafka/intra-cluster-replication-
apache-kafka
● http://martinfowler.com/bliki/MonolithFirst.html
Links

22. ● https://www.oreilly.com/learning/making-sense-of-stream-proc
essing/page/3/integrating-databases-and-kafka-with-change-da
ta-capture
● http://kafka.apache.org/documentation.html
● https://github.com/toddpalino/kafkafromscratch/blob/master/A
pache%20Kafka%20from%20Scratch.pdf
● http://www.javaworld.com/article/3060078/big-data/big-data-m
essaging-with-kafka-part-1.html
● https://sookocheff.com/post/kafka/kafka-in-a-nutshell/
Links

23. Use Cases - Change Data Capture
● Log compaction
○ Kafka + Kafka Connect

24. Partitioning
● Custom Partitioner
○ Write your own logic
● Default Partitioner
○ Manual
○ Hashing
■ The most common approach
■ Messages with the same key go to the same producer
○ Spraying
■ Random partitioning