More info: https://cnfl.io/cloud-native-experience-for-kafka-in-cloud | Neha Narkhede is co-founder and CTO at Confluent, a company backing the popular Apache Kafka messaging system. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

1. 1
Event Streaming
Lessons from our cloud-native journey
Neha Narkhede
Co-Creator, Apache Kafka
Co-Founder & CTO, Confluent

2. 22
The New Business Reality
Past
Technology was a support function
Innovation required for growth
Running the business on yesterday’s data was
“good enough”
Today
Technology is the business
Innovation required for survival
Yesterday’s data = failure.
Modern, real-time data infrastructure
is required.

3. 3
The Rise Of Event Streaming

4. 4
The Rise Of Event Streaming

5. 5
The Rise Of Event Streaming
60%Fortune 100 Companies
Using Apache Kafka

6. 66
What We Are Building With Event Streaming
Online
Dating
Data-driven
Video Games
Driver-rider
match
Digital Publishing
Pipelines
Digital Banking
Assistants

7. 77
The Central Challenge
Connecting applications with data
ETL
What happened
in the world
Messaging
What is happening in
the world

8. 8

9. 9
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Highly Scalable
Durable
Persistent
Ordered
Fast (Low Latency)
What is happening
in the world
What happened
in the world

10. 1010
Highly Scalable
Durable
Persistent
Maintains Order
ETL/Data Integration MessagingETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
Fast (Low Latency)
What happened
in the world
What is happening
in the world
Highly Scalable
Durable
Persistent
Ordered
Fast (Low Latency)
Event Streaming Paradigm

11. 1111
Highly Scalable
Durable
Persistent
Maintains Order
ETL/Data Integration MessagingETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
Fast (Low Latency)Highly Scalable
Durable
Persistent
Ordered
Fast (Low Latency)
Event Streaming Paradigm
What is contextually happening in the world

12. 1212
Event-Driven App
(Location Tracking)
Only Real-Time Events
Messaging Queues and
Event Streaming
Platforms can do this
Contextual
Event-Driven App
(ETA)
Real-Time combined
with stored data
Only Event Streaming
Platforms can do this
Where is my driver? When will my driver
get here?
Where is my driver? When will my driver
get here?
Why Combine Real-time
With Historical Context?
2
min

13. 13
Event Streaming Paradigm
Highly Scalable
Durable
Persistent
Maintains Order
Fast (Low Latency)
Event Streaming Paradigm
To rethink data as not stored records
or transient messages, but instead as
a continually updating stream of events

14. 1414
Event Streaming Paradigm
Highly Scalable
Durable
Persistent
Maintains Order
Fast (Low Latency)
Event Streaming Paradigm

15. 15
Real-Time
Inventory
Real-Time
Fraud
Detection
Real-Time
Customer 360
Machine
Learning
Models
Real-Time
Data
Transformation
...
Contextual Event-Driven Applications
Universal Event Pipeline
Data Stores Logs 3rd Party Apps Custom Apps/Microservices
STREAMS
CONNECT CLIENTS

16. 1616
The Path To Success With
Event Streaming
Based on our first-hand experience

17. 1717
Our Path With
Confluent Cloud
Today
3 GB/s
2018
Start
3 Months
5 MB/s
6 Months
15 MB/s
9 Months
500 MB/s
12 Months
1 GB/s
15 Months
1.5 GB/s

18. 1818
Today
3 GB/s
Late 2019
25 GB/s
2020
100 GB/s
Our Path With
Confluent Cloud

19. 1919
Top 5 Lessons Learned
Things we learned that everyone
can benefit from

20. 2020
01
Lesson
There Is No Cloud-Native
Easy Button
20

21. 21

22. 22

23. 23
On
Premises
topic_name_01
topic_name_02
topic_name_03
Confluent
Replicator
Schema Registry
topic_name_01
topic_name_02
topic_name_03
Schema Registry
Public
Cloud

24. 2424
02
Lesson
Schemas Are Service APIs
For Event Streaming

25. 25
A Schema Is A Contract, An API Between Services
1
2
3
4
5
6
7
8
9
{
“name”: “received_at”,
“date”: “1448928000”,
“type”: {
“type”: “string”,
“avro.jaja.string”: “String”,
},
“doc”: “FIT’s time of assignment”,
},
1
2
3
4
5
6
7
8
9
{
“name”: “received_at”,
“date”: “Dec 1, 2015”,
“type”: {
“type”: “string”,
“avro.jaja.string”: “String”,
},
“doc”: “FIT’s time of assignment”,
},

26. 2626
Just Like APIs, Schemas Need To Be
02
Versioned
03
Validated +
Enforced
01
Discoverable
+ Searchable

27. 27

28. 28

29. 29

30. 3030
03
Lesson
Stream Processing Is
The Best Way To Observe
Stream Processing

31. 31

32. 32

33. 33

34. 34

35. 3535
04
Lesson
Elasticity Is A Cloud-Native
Imperative

36. 3636
Making Kafka Elastic In The Cloud
01
Instant access to
capacity with
seamless scalability
02
Work around the
myriad limits on
cloud infrastructure
03
Balance traffic in a
continuous and
intelligent fashion
04
Move data out
of local storage

37. 3737
05
Lesson
Life Is Better Without
Cluster Sizing

38. 3838
Cloud infrastructure costs
are confusing and hard
to understand
Overprovision OR Outage?
Pick One
Understanding cost
attribution for a service in
the cloud is even harder
Sizing is a moving target,
the available options
keep changing

39. 3939
The Cost Of Cloud Complexity
per year if you picked i3.2xlarge on-
demand instance vs r5.xlarge reserved
instance
+$250k
per year if you picked Private Link
instead of VPC Peering for networking
+$600k
per year if you picked SSD EBS storage
VS optimized EBS
+$700k
Kafka workload:
100 MB/s
writes and reads
30 day
retention
This adds up to over $1M per year!

40. 4040
03
Stream processing
is the best way
to observe stream
processing
02
Schemas are
service APIs for
Event Streaming
01
There is no cloud-
native
easy button
05
Life is better
without cluster
sizing
04
Elasticity is a cloud-
native imperative
Top 5 Lessons Learned

41. 41
Available Today In Confluent Cloud
● 0 to 100MBps: Self-service,
no need to size or provision
clusters
● 100 MBps to 10s of GBps:
scale with provisioned
capacity
Scale elastically to
100MBps and down
in seconds
● Start at less than $50/month
● No minimums or commitment
Pay only for what you
actually stream
● The most popular Kafka
tools available as a fully-
managed service
Fully-managed Schema
Registry, KSQL,
S3 Connector (Preview)

42. 4242

43. 4343

44. 4444

45. 4545
2 Easy Choices To Get Started
Confluent
Platform
All features are free
on 1 node
Confluent
Cloud
Pay as you go: Start at
less than $50 / month

46. 4646
What If We All Succeed With
Event Streaming?

47. 4747
Transportation
Without Event Streaming
No knowledge of driver
arrival
Call company for driver
availability
No sensor diagnostics

48. 4848
Transportation
Real-time ETA
Real-time driver-rider
match
Real-time sensor
diagnostics
With Event Streaming

49. 4949
Banking
Out of sync account
Information
Batch fraud checks
Batch internal risk reporting
Without Event Streaming

50. 5050
Banking
With Event Streaming
Real-time omni-channel
account updates
Real-time fraud detection
Real-time internal risk
calculations

51. 5151
Retail
Customer receives “out of
stock” email after ordering
Retailers receive sales
reports every 1-2 days
No ability to upsell through
personalization
Without Event Streaming

52. 5252
Retail
Real-time inventory across
web and store
Real-time point of sales
reporting
Real-time, contextual
recommendations
With Event Streaming

53. 5353
Event Streaming Paradigm Is The Future Of Data
Infrastructure
as code
Data as a continuous
stream of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

54. 54