Jay Kreps is the CEO of Confluent, Inc., a company backing the popular Apache Kafka® messaging system. Prior to founding Confluent, he was formerly the lead architect for data infrastructure at LinkedIn. He is among the original authors of several open source projects including Project Voldemort (a key-value store). Apache Kafka (a distributed messaging system) and Apache Samza (a stream processing system).

1. 9
Overcome By Events:
Apache Kafka &
Event-oriented Architecture
Jay Kreps

2. 1010
Overcome By Events (OBE)
10
A term of military origin used when
a situation changes so rapidly that
previously proposed courses of
action are no longer relevant.

3. 11
What is an event?

4. 12
Events

5. 13
Events
A Sale
A Trade
An Invoice
A Customer
Experience

6. 1414
Here!
Where are they?
Events haven’t had a proper
home in infrastructure or
in code. They are implicit.

7. 15
Where do
events show
up?
● Microservices
● Monitoring
● Data Pipelines
● Analytics

8. 1616
Microservices Synchronous doesn’t scale

9. 1717
ServerlessMicroservices

10. 1818
Event SourcingMicroservices

11. 19
Old School
4 Pillars
● Metrics
● Alerting
● Distributed Tracing
● Log Aggregation
Monitoring

12. 20
Old School
4 Pillars
● Metrics
● Alerting
● Distributed Tracing
● Log Aggregation
Monitoring
Event Processing
New School

13. 21
Log AggregationMonitoring

14. 22
MetricsMonitoring

15. 23
Metrics
SELECT HOST,
COUNT(*) AS ERROR_RATE
FROM LOGS
WINDOW TUMBLING (SIZE 5 MINUTES)
WHERE LOG_SEVERITY='ERROR'
GROUP BY HOST;
Monitoring

16. 24
AlertingMonitoring

17. 25
Alerting
CREATE TABLE SLA_BREACH AS
SELECT HOST, COUNT(*) AS ERROR_RATE
FROM LOGS
WINDOW TUMBLING (SIZE 5 MINUTES)
WHERE LOG_SEVERITY='ERROR'
GROUP BY HOST
HAVING COUNT(*) > 10;
Monitoring

18. 26
Distributed TracingMonitoring

19. 27
Distributed TracingMonitoring
SELECT HOST, APP, TIMESTAMP,
USERID, USER_NAME, USER_EMAIL
FROM LOGS L
INNER JOIN USERS U
ON L.USERID=U.USERID
WHERE TRACE_ID='40e9bea3-de9b-4004';

20. 28
Data Pipelines

21. 29
Data Pipelines

22. 30
Data WarehousingAnalytics
LOCATION
PK Location_ID
Name
State
Pincode
TIME
PK Time_ID
Day
Month
Quarter
Year
BRANCH
PK Branch_ID
Branch_NM
Owner
ITEM
PK Item_Key
Item_Name
Brand
Sold_By
Category
FK ITEM_KEY
FK TIME_ID
FK BRANCH_ID
FK LOCATION_ID
QTY_SOLD
AMT_SOLD
AVG_SALES
Branch dimension table
Item dimension table
Location dimension table
Time dimension table
Sales Fact Table
STAR
SCHEMA

23. 31
Analytics
Real-time Analytics
and Machine
Learning on Event
Streams

24. 32
A business is a series of events and the
reactions to those events.
Events are everywhere

25. 3333
The Infrastructure for Events

26. 34
Traditionally,
there is none.

27. 3535
You need both these things
Tables
The current state of
the world
Events
What’s happening in
the world

28. 3636
Table Stream Duality

29. 37
The
Streaming
Platform

30. 38
Key Elements
of Apache
Kafka

31. 39
01
Immutable
log-centric
architecture

32. 40
02
Support
for tables
with Log
Compaction

33. 41
03
Horizontally
scalable
Custom Apps

34. 42
03
Horizontally
scalable
Custom Apps MonitoringMicroservices
Bloomberg SFDC
TwitterOracle Hadoop

35. 43
Custom Apps
Bloomberg
Microservices Monitoring AnalyticsWeb
SFDC
TwitterOracle Hadoop
Data
Warehouse
NoSQL
any sinkany source
03
Horizontally
scalable

36. 44
04
Connector
Architecture

37. 45
05
Transactional
semantics
for stream
processing
BEGIN
Read Input
Update State
Write Output
COMMIT

38. 46
Community
and
Ecosystem
112
Meetup groups
282,011
meetup members

39. 47
The
Streaming
Platform

40. 48