Gwen Shapira of Confluent presented episode #03 of Gluent New World series and talked about stream processing in modern enterprises using Apache Kafka. The video recording for this presentation is at: http://vimeo.com/gluent/

1. Apache Kafka and Real Time
Stream Processing
Gwen Shapira
System Architect
Confluent
@gwenshap

3. I’ll tell you about
• What is stream processing and
why it matters
• What is Apache Kafka
• How Kafka helps stream processing
Stay awake for
this part

4. What is Stream Processing?

5. Data Processing Paradigm
Request / Response
Batch
Stream Processing

6. Stream Processing Paradigm
• Data is generated at its own rate as “Streams”
• We can process as much or as little as we want
• Continuously
• Results are available in real-time
• But nothing waits for specific results
• Time for data availability?
• More than “few ms”
• Less than “hours”

8. This is the world changing bit
• Most of the business is…
• Not urgent enough to require immediate response
• But can’t wait for the next day
• “Streams of events” represents something fundamental
• Same way relational tables are fundamental

9. Ok, got the streams part.
But what about Apache Kafka?

10. Cross of messaging system
and file system

11. Kafka is all about LOGS

12. If you understand logs
You understand Kafka

13. Redo Log:
The most crucial structure for
recovery operations …
store all changes made to the
database as they occur.

14. Important Point
The redo log is the only reliable
source of information about current
state of the database.

15. But Logs are also a STREAM of events
And Kafka stores those logs
Allowing to read the past
and keep getting updates on the future

16. Stream Processing
Read a stream
modify it
output another stream

17. Example: CDC-based ETL

18. If we use Kafka for CDC,
does it mean it is ACID?

19. Stream Processing is Important
Kafka is a collection of logs.
How does Kafka help with stream processing?

20. First, How do we actually
do stream processing?

21. Method 1:
Do it yourself (Hipster stream processing)

22. Method 2:
The Stream Processing Frameworks
• Storm
• Spark
• Flink
• Samza
• Apex
• Nifi
• StreamBase
• InfoSphere Streams
• Google DataFlow (AKA Beam)
• I can go on for 5 more pages…

23. Few of those are really popular!
• Pro: They handle some hard problems
• Con: It can be too complex

24. What do I mean by too complex?
Hadoop Cluster II
Storage Processing
SolR
Hadoop Cluster I
ClientClient
Flume Agents
Hbase /
Memory
Spark
Streaming
HDFS
Hive/Imp
ala
Map/Red
uce
Spark
Search
Automated &
Manual
Analytical
Adjustments
and Pattern
detection
Fetching &
Updating Profiles
Adjusting NRT Stats
HDFSEventSink
SolR Sink
Batch Time Adjustments
Automated &
Manual
Review of NRT
Changes and
Counters
Local Cache
Kafka
Clients:
(Swipe
here!)
Web App

25. Why so many moving parts?
We needed…
Hbase to handle complex state
Spark requires HDFS
Ingest layer
Batch layer to handle re-calculations

26. What we really wanted was…
Inputs
Kafka
Processor
output

27. Enter KafkaStreams
3 Simplifications:
1. Uses Kafka
2. No Framework
3. Unify Tables and Streams

28. Don’t all stream processing use
Kafka?

29. We use Kafka for… Partitioning, Scalability,
Fault Tolerance
Kafka
A A A
Group A
B
B
Group B

30. Handling Time

31. No Framework
• It is just a library that does transformations
• We can add languages on top
• Kafka does everything we needed the framework to do
• You don’t need “framework” to run queries, why do you need it to
run queries continuously?

32. The really important bit:
Streams meet Tables

33. Streams: Things that happen. Events.
Tables: State of things as they are.

34. Databases: Only states.
Streams: Only events.

35. We can convert tables to streams and back:
Stream -> Apply -> Table
Table -> Change Capture -> Stream
This is called Table-Stream Duality.

36. Streams and Tables sometimes work
the same.
And sometimes are very different.
KafkaStreams handles both.

37. But…
Where do streams come from?

44. We really like streams
So we created a
Stream Data Platform

46. Where can we learn more?
• http://www.confluent.io/blog
• http://kafka.apache.org/docume
ntation.html
• http://docs.confluent.io/current