Kafka is a distributed streaming platform that allows for publishing and subscribing to streams of records. It provides low latency and high throughput processing of streams of records. Key features include replication across clusters for fault tolerance, partitioning for scalability, and ordering of records. Common uses of Kafka include building event driven microservices, stream processing, and serving as the data backbone for real-time analytics applications.

1. Data Pipelines with
Apache Kafka
Ben Stopford @confluentinc

2. Today
• What is Kafka? (High level fluffy stuff)
• What makes it tick? (Low level geeky stuff)
• How can you use it? (Architect oriented stuff)

4. What is
Kafka?

5. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

6. The Log
Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

7. Clients
JVM & C native implementations, Go, Python, many more OS
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

8. Connectors
Plug into your database of choice
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

9. Streaming Engine
The declarative power of a database, wrapped into a Kafka client
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

10. Kafka:
The distributed Log
Today we’ll focus on

11. The log is a type of
messaging system

12. What is messaging in essence?
• Take a message, keep it safe, make it available to
consumers.
• Track what messages have been consumed
Kafka attacks these problems separately

13. What is a message
broker in essence?
Sender Receiver
Broker
(the log)

14. The log is a simple idea
Messages are added at the end of the log
Just think of the log as a file
Old New

15. Consumers have a position
Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan

16. Only Sequential Access
Old New
Read to offset & scan

17. No Random Access
Index
Disk
Kafka avoids Indexes by keeping the approach simple
(indexes impede scalability in this context)

18. Topics are Broadcast
Consumer
Consumer
Broker broadcast

19. Can also behave as a queue
Sender Receiver

20. The problem:
If you built a messaging
system for internet scale,
what would it look like?

21. Shard data to get scalability
Messages are sent to
different partitions
Producer (1) Producer (2) Producer (3)
Cluster of
machines Partitions live on
different machines

22. Replicate to get fault tolerance
replicate
msg
mastership
moves
machines
(1)
(2)
msg
leader
Machine A
Machine A
Machine B
Machine B

23. Kafka goes a step further
A single topic can be spread
over multiple consumers
(4 consuming machines
process a single topic)

24. Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!

25. Distributed Commit Log
Different to a traditional messaging system

26. Data is replicated

27. Strong Consistency
Send Message
3 replicas on different machines
• Only 1 elected leader
• Only leader can be written to, read from

28. Replication provides resiliency
Another replica takes
over on machine failure

29. Replication Protocol
Send Message

30. Optimistic Write
(single machine delivery)
Send Message
Get ack (optimistic)

31. Pessimistic Write
(wait for replication to complete)
Send Message
Get ack (pessimistic)

32. Replication Protocol
Writer
Messages can be read only after
replication completes
Reader

33. Replication Protocol
Number of replicas is
a soft quorum
(set min/max
tolerable values)
Writer
Reader

34. Replication is used for
resiliency. No need to flush
to disk synchronously.
You can flush if you wish, but no one does.

35. Advanced
Features

36. Consumers cluster too!
Consumer Group 1 Consumer Group 1

37. Consumers cluster too!

38. Compacted Topics
(Tabular View)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1
Version 2
Version 3
Version 5
All versions Latest Key
only

39. Multi Tenancy
Users isolated using
security features
Bandwidth segregated
per user

40. Use Cases

41. Microservice
Backbone

42. Always on, Event-Driven Services
The Log
(streams & tables)
Ingestion
Services
Services with
Polyglotic
persistence
Simple Services
Streaming Services

43. Event Buffer

44. Many producers,
small messages
Kafka
Hadoop etc

45. Stream Processing
for enrichment &
transformation

46. Kafka Streams Example
Orders
Customer
(Compacted)
Join
Customer Stream
Join, aggregate,
intermediary state
stored in Kafka
Kafka
Kafka Streams
Orders Stream
Dashboard
Query

47. Stream Data
Platform
(Kappa Architecture)

48. Allyourdata Stream Data platform
Views
Client
Client
Kafka
Stream
processor
Connectors

49. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

50. The end
@benstopford
http://benstopford.com