Slides from Neha Narkhede's keynote at the dotScale conference in Paris on April 24th, 2017. There is a tectonic shift happening in how data powers the core of a company's business. This shift is about the rise of real-time. Apache Kafka was built with the vision to help companies navigate this change and become the central nervous system that makes data available in real-time to all the applications that need to use it. This talk is about how you can put Apache Kafka to practice to help your company make this shift to real-time. And how the Connect and Streams API in Apache Kafka capture the entire scope of what it means to put streams into practice.

1. the rise of real time
1
Neha Narkhede, Co-founder & CTO, Confluent

2. THEN
2

3. NOW
3

4. What does transitioning to a
digital business look like?
4

5. not just a better
or faster version
of what has been
done before
5

6. but a fundamentally
different solution fit
for modern digital
needs
6

7. STREAMS
7

8. 8
“all your data is event streams”
bold claim

9. 9
sales
shipments
inventory
adjustments
price
adjustments
analytics
fraud
reordering
retail today

10. 10
event streams are relevant to
every business

11. 11
business events logs sensors
monitoring databases
event streams are related to every business

12. 12
two uses for streams
build streaming data pipelines
apps that react to, process, transform
streams

13. 13
how to companies actually do
this?

14. 14
A giant mess!
App App App App
search
HadoopDWH
monitoring security
MQ MQ
cache
cache

15. 15
key insight:
data pipeline = event stream

16. 16Confidential 16
streaming platform
DWH
Hadoop
security
App App App App
search
NoSQL
monitoring
request-response
messaging
OR
stream
processing
streaming data pipelines
changelogs

17. be it one application
app

18. … or a whole company
Hadoop
!
{Real-time
analysis
}Stream
processing apps
Data
Warehouse
DBs Apps

19. how do you begin the transition
to a streaming-first enterprise?

20. make a fundamental transition
to event-centric thinking

21. 21
event-centric thinking
Streaming
Platform
“A product was viewed”
HadoopWeb
app

22. 22
event-centric thinking
Streaming
Platform
“A product was viewed”
Hadoop
Web
app
mobile
app
APIs

23. 23
event-centric thinking
mobile
app
web
app
APIs
Streaming
Platform
Hadoop
Security
Monitoring
Rec
engine
“A product was viewed”

24. 24Confidential
event-centric thinking at a company-wide scale!
24

25. apache kafka: a distributed streaming platform
scalability of a
filesystem
● hundreds of MB/s
● many TBs per server
● commodity hardware
guarantees of a
database
● persistence
● ordering
distributed by
design
● replication
● partitioning
● horizontal scalability
● fault tolerance

26. 26Confidential
apache kafka
7 years ago
26

27. > 1,400,000,000,000
messages processed / day
27

28. 28
kafka is adopted at 1000s of companies
Financial Services Enterprise Tech Consumer Tech
Entertainment & Media Telecom Retail Travel & Leisure

29. how does Kafka put event-centric
thinking into practice?

30. kafka for the two uses for streams
30
build streaming data pipelines
apps that react to, process, transform
streams

31. kafka’s connect api =
streaming data pipelines made easy
NoSQL
RDBMS
hadoop
DWH
Search Monitoring
RT
analytics
apps apps apps
31

32. 32
ConnectAPI
ConnectAPI
source sink
pull push
kafka’s connect api =
streaming data pipelines made easy

33. kafka all the things!
33
Kafka’sconnectAPI

34. 34
kafka for the two uses for streams
build streaming data pipelines
react to, process, transform

35. 35Confidential
stream processing
35

36. 36Confidential
kafka’s streams api = stream
processing made easy!
36

37. 37
2 visions for stream processing
real-time mapreduce event-driven microservices

38. vision 1: real-time mapreduce
38

39. 39
vision 2: event-driven microservices
Streams API
microservice
Stream processing

40. vision 2: event-driven microservices
using kafka’s streams api
Streams API
microservice
stream processing
● simple but powerful Java library
● convenient DSL
● event-at-a-time processing; No micro
batching
● dataflow-style windowing
● local state
● reprocessing

41. 41
example: real-time dashboard for security monitoring

42. 42
kafka’s streams api: build apps, not clusters
vision 1
vision 2

43. kafka’s connect api + kafka’s streams api
Messaging API
Streams API
apps
apps
ConnectAPI
ConnectAPI
source sink
pull push
stream processing

44. 44Confidential
44
streaming platform
DWH Hadoop
security
App App App App
search
NoSQL
monitoring
request-response
messaging
OR
stream
processing
streaming data pipelines
changelogs
vision: all your data … everywhere … now

45. 45Confidential
45
streaming platform
DWH Hadoop
security
App App App App
search
NoSQL
monitoring
request-response
messaging
OR
stream
processing
streaming data pipelines
changelogs
vision: all your data … everywhere … now

46. confluent.io/download
46
@nehanarkhede