Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).

1. 1
Events Everywhere
Jay Kreps
CEO, Confluent

2. 22
The world is changing.

3. 33
The world is changing.
Then
Hardware product
Up front purchase
Opaque
No data
Now
Hardware, Software & Global Internet Service
On-demand
Real-time visibility
Built on a foundation of data
Transportation

4. 4
Transportation

5. 55
This transformation is
happening everywhere...

6. 6
Banking

7. 7
Retail

8. 88
What enables this
transformation?

9. 99
Cloud Machine
Learning
Mobile Event
Streaming

10. 1010
Transportation
ETA
Real-time sensor
diagnostics
Driver-rider match
Banking
Fraud detection
Trading and risk
systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized news
feed
In-app purchases

11. 1111

12. 1212
This is a fundamental paradigm shift...
Infrastructure
as code
Data as a continuous
stream of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

13. 1313
The Event
Streaming Paradigm

14. 1414
Two Problems in Application Infrastructure
What’s the state
of the world?
What’s happening
in the world?
Solution:
Databases
Solution:
Messaging, RPC, ETL, etc.

15. 1515
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)

16. 1616
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Transient MessagesStored records

17. 1717

18. 1818
ETL/Data Integration Messaging
Transient MessagesStored records
ETL/Data Integration MessagingMessaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Event Streaming Paradigm
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)

19. 1919
To rethink data as not stored records
or transient messages, but instead as
a continually updating stream of events
Event Streaming Paradigm

20. 20

21. 21
Event Streaming
Platform
Universal Event Pipeline
Data Stores Logs 3rd Party Apps Custom Apps/Microservices

22. 22
Universal Event Pipeline
Data Stores Logs 3rd Party Apps Custom Apps/Microservices
Real-Time
Inventory
Real-Time
Fraud
Detection
Real-Time
Customer 360
Machine
Learning
Models
Real-Time
Data
Transformation
...
Contextual Event-Driven Apps
Event Streaming
Platform

23. 2323
Building an Event
Streaming Platform
with Kafka

24. 24
Consumer A Consumer B
Reads
Offset = 3 Offset = 6
Producers
Writes
91 2 3 4 5 6 7 8

25. 25

26. 26

27. 27
Kafka Connect and Connector Ecosystem

28. 28
Stream Processing with KSQL
Stream 01:
Stream 02:
Stream
Table
Processing streams of data to create more streams or tables
QueryQuery
Continuous
Query

29. 29
KSQL: a few use cases
A big step towards a
streaming-first world:
● Real-time monitoring
and analytics
● Streaming ETL, not
Batch ETL
● Application development

30. 30

31. 31
Securing your event streams
Authentication
Data
Confidentiality
Authorization

32. 32
Topics PartitionsMessages / sec Brokers
10000000 25000 1000000 1500
Topics PartitionsMessages / sec Brokers
250000 500 25000 25
Topics PartitionsMessages / sec Brokers
1 5 300 3
Operations

33. 33
Confluent Products: Software and SaaS
Confluent Platform
The Enterprise Distribution of
Apache Kafka
Self-Managed Software
Deploy on any platform, on-prem or cloud
VM
Confluent Cloud
Apache Kafka Re-engineered
for the Cloud
Fully-Managed Service
Available on the leading public clouds

34. 34
Announcing Confluent Platform 5.2
Accelerate
Development
● Free forever Confluent
Platform on single
broker
● Librdkafka 1.0
● New KSQL expressions
Operate
at Scale
● Control Center
○ Schema management
○ Dynamic broker config
○ Multi-cluster
KSQL/Connect
○ Improved scalability
Enable Hybrid
Streaming
● Schema Registry
replication from
Confluent Platform to
Confluent Cloud
Apache Kafka 2.2.0

35. 3535
This is happening now.

36. 363636
Kafka is in production in tens
of thousands of companies

37. 3737
70% 80% 90%10% 20% 50% 60%40%30%
60%
of the Fortune 100
Now

38. 3838
Every customer
experience
Every bank
transaction
Every
purchase
Every
car ride
Every
log in
Events Everywhere

39. 39