1. Event Streaming CTO Roundtable
Real-World Use Cases for Data in Motion with Cloud-native Architectures
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de

2. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

3. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

4. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
This is a fundamental paradigm shift...
4
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud-
Native
Event
Streaming

5. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Real-time Data beats Slow Data.
Logistics
Real-time sensor
diagnostics
Delivery planning
ETA updates
Payment
Fraud detection
Risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Cybersecurity
Threat detection
Intrusion detection
Incident response
Fraud detection

6. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6

7. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

8. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

9. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Lambda Architecture
Option 1: Unified serving layer
9
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Serving Layer
Real-Time App
(Data Processing in Motion)
Batch App
(Data Processing at Rest)
ms
min/hr

10. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
10
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Real-time Query
Mixed Query
ms
min/hr
Speed
View
Batch
View
Batch Query
Lambda Architecture
Option 2: Separate serving layers

11. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
11
Data
Source
Real-Time Layer
(Data Processing in Motion)
Real-Time App
(Data Processing in Motion)
Storage
Batch App
(Data Processing at Rest)
Storage
ms
min/hr
Storage
Kappa Architecture
One pipeline for real-time and batch consumers

12. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Kappa @ Uber
12

13. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Kappa @ Shopify
13
Kappa Building Blocks
The Log (Kafka)
Durability with Topic Compaction and Tiered Storage
Consistency via Exactly-Once Semantics (EOS)
Data Integration via Kafka Connect
Elasticity via dynamic Kafka clusters
Streaming Framework (Kafka Streams / Flink)
Reliability and scalability
Fault tolerance
State management
Sinks
Update/Upsert for simplified design:
RDBMS, NoSQL, Compacted Kafka Topics
Append-only: Regular Kafka Topics, Time Series

14. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Kappa @ Disney
14
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning without a Data Lake

15. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

16. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Mainframe Offloading
Journey from Mainframe
to Hybrid* and Cloud
PHASE 3
Hybrid
Replication
Mainframe
Replacement
PHASE 2
PHASE 1
* with or without the mainframe

17. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Strangler Design Pattern - A Big Bang will FAIL !!!
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/

18. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

19. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Year 1: Kafka for Decoupling between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

20. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

21. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Core Banking ‘2020’
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

22. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Integration Platform
for legacy and modern technologies

23. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Global Event Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking
23

24. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Domain
Data
Product
Focus on Business and Data Products with Decoupled Microservices
Data
Mesh
Mesh is a logical view,
not physical!

25. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

26. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Context-specific Customer 360
26
Electrical retailer
Hyper-personalized online retail experience,
turning each customer visit into a one-on-one
marketing opportunity
Correlation of historical customer data with real-
time digital signals
Maximize customer satisfaction and revenue
growth, increased customer conversions
https://www.confluent.io/customers/ao/

27. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Innovative Business Modell
• Clothing rental subscription service
• Very different from a typical e-commerce model
• Need for a real-time event driven architecture
Benefits of serverless Confluent Cloud
• Cut launch time from over a year to 6 months
• Stable production ops set up in 1 week vs. 6 months
• Administrative overhead reduced by 10
27
https://www.confluent.io/customers/nuuly/

28. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
28
https://medium.com/porschedev

29. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

30. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

31. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

32. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Wavelength
Cloud VPC
Carrier #1 5G
Wavelength
Carrier #2 5G
32
Hybrid Streaming Data Exchange

33. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
Event Topics Storage Partitions
Events / sec Kafka Servers
250,000 500 25,000 25
Event Topics Storage Partitions
Events / sec Kafka Servers
100 5 300 3
Kafka Scales with Your Business.

34. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
BMW Group
Mission-critical workloads at the edge and in the cloud
• Why Kafka? Decoupling. Transparency. Innovation.
• Why Confluent? Stability is key in manufacturing
• Decoupling between logistics and production systems
• Provide edge platform (self-managed) + Azure Cloud (fully-managed) + bidirectional
integration
• Use case
• Logistics and supply chain in global plants
• Right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence
• Lot of critical applications
34
Jay Kreps, Confluent CEO
Felix Böhm, BMW Plant Digitalization and Cloud Transformation
Keynote at Kafka Summit Eurpoe 2021:
https://www.youtube.com/watch?v=3cG2ud7TRs4
(My Notes from the BMW Keynote at Kafka Summit EU 2021)

35. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

36. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Apache Kafka as Infrastructure for ML

37. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client

38. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
38
Model X
(at a later time)

39. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Confluent Tiered Storage for Kafka
39

40. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
41

41. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Fraud Detection @ Grab
GrabDefence SaaS service build with Confluent Cloud, Kafka Streams and ML for stateful stream processing
Billions of fraud and safety detections performed daily for millions of transactions (1.6% is lost in fraud in Southeast Asia)

42. @KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

43. Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!