1.
@KaiWaehner - www.kai-waehner.de
Event Streaming with
Apache Kafka for the Connected World
Connected Vehicles, Manufacturing 4.0, Cybersecurity, Customer 360, and Smart City
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2.
@KaiWaehner - www.kai-waehner.de
Agenda
2
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

3.
@KaiWaehner - www.kai-waehner.de
Agenda
3
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

4.
@KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
4

5.
@KaiWaehner - www.kai-waehner.de
Stores become Software
5

6.
@KaiWaehner - www.kai-waehner.de
Banks become Software
6

7.
@KaiWaehner - www.kai-waehner.de
Taxis become Software
2
min
7

8.
@KaiWaehner - www.kai-waehner.de
Cars become Software
8

9.
@KaiWaehner - www.kai-waehner.de
Car Production becomes Software
9

10.
@KaiWaehner - www.kai-waehner.de
Collaboration is Key for Success
Car Maker
Supplier
Government
Telco
Mobility Services
Cloud Providers
Software Providers
10

11.
@KaiWaehner - www.kai-waehner.de
Example: Automotive Industry Point of View
CASE (Connected, Autonomous, Shared, Electrified)
https://wiprodigital.com/2019/08/26/digital-transformation-auto-industry-fueled-by-case/
11

12.
@KaiWaehner - www.kai-waehner.de
Huge demand to build an open,
flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Elasticity
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global
12

13.
@KaiWaehner - www.kai-waehner.de
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C C
C
Event Streaming
13

14.
@KaiWaehner - www.kai-waehner.de
An Event Streaming Platform is the
Underpinning of an Event-driven Architecture
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps
Supplier
Alert
Forecast
Inventor
y
Customer
Order
14

15.
@KaiWaehner - www.kai-waehner.de
Cloud-native Hybrid Architectures
Edge, On Premise and Public Cloud
VM
SELF MANAGED
FULLY MANAGED
15

16.
@KaiWaehner - www.kai-waehner.de
Agenda
16
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

17.
@KaiWaehner - www.kai-waehner.de
Use Cases
for Connected Vehicles
17

18.
@KaiWaehner - www.kai-waehner.de
Self-Driving Ecosystem
• Urban, interstate, private (e.g., factory)
• Example: driver (Aurora), environment (e.g., ride sharing with Uber)
Vehicle
Toyota
Hyundai
BMW
…
“Driver”
Waymo
Aurora
…
Environment
Uber
Lyft
Wunder Mobility
Here Technologies
…
18

19.
Connected Services

20.
CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Wavelength
Public 5G
Campus #1 5G
Wavelength
Campus #2 5G
20

21.
@KaiWaehner - www.kai-waehner.de
Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/
21

22.
@KaiWaehner - www.kai-waehner.de
MQTT Proxy
MongoDB
Storage
MongoDB
Dashboards
Search
Analytics
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka
Streams
Application
All
Data
Critical
Data
Ingest
Data
Potential
Detect
TensorFlow
Train
Analytic
Model
ksqlDB
Analytic
Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
Tiered
Storage
Mobile App
BI Tool
Demo: 100,000 Connected Cars
(Kafka + ksqlDB + MQTT + TensorFlow)
Kafka Ecosystem
TensorFlow
Other Components
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
22

23.
@KaiWaehner - www.kai-waehner.de
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id,
detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with Apache Kafka,
ksqlDB and TensorFlow
23

24.
@KaiWaehner - www.kai-waehner.de
Agenda
24
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

25.
@KaiWaehner - www.kai-waehner.de
Industry 4.0 / Industrial IoT (IIoT)
Evolution of the Convergence between IT and Industrial Automation
25

26.
Kafka is “Real-Time”, but NOT
“Hard Real-Time”

27.
@KaiWaehner - www.kai-waehner.de
OT - Connected Vehicle
(Car, Train, Drone)
OT - Manufacturing
(Field Bus, PLC, Machine,
Robot)
IT – Enterprise Software
(Data Center, Cloud, Car IT)
Central Data Center / Public Cloud
Connect
Vehicle Data (e.g. OBD2)
Robot Data All Data
C
C++
C
C++
Java
Python
Go
[#] Hard Real Time
= Deterministic network
with zero spikes + zero latency
[#] Soft Real Time
+ Near Real Time
+ Batch
27

28.
@KaiWaehner - www.kai-waehner.de
Global Event Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster
Linking
28

29.
Kafka as Data Historian
Continuous real time data ingestion,
processing and monitoring 24/7 at scale

30.
@KaiWaehner - www.kai-waehner.de
Analytics
Database
Data Lake
CRM
MQTT ModbusFirewall
Siemens
S7
Kafka Connect
Integration | Storage | Processing | Access | Cloud | Security
Storage
Kafka Streams
/
ksqlDB
OPC-UA
Stateless +
Stateful
Kafka
Connect
Kafka
Clients
Java, C++. Python, Go,
…
REST
Proxy
SCADA
DCS
ERP
MES
Syslog
File
X
Y
Z
A
B
C
Cloud
Factory

31.
@KaiWaehner - www.kai-waehner.de
Reduction in Production Cost –
Early Part Scrapping
Station 1 Station 2
Cost Before
100$
Total Cost
450$
(or more)
200$ 150$
Scrap? Scrap?
31

32.
@KaiWaehner - www.kai-waehner.de
BI
Tool
AI/ML
Machine Vision for Quality Assurance
and Yield Management
Apache Kafka and Applied Machine Learning
Filter, transform
aggregate,
orchestrate
APP
Real-time alerting
Sensor Data
SCADA
MES
PLCs
OT
Team
Plant
Manager
Images
from Products
of Assembly Lines
IT
Team
Live
Ops
Machine Vision for
Quality Inspection
Reporting
Backup
Data Science Team
Data Lake
32

33.
@KaiWaehner - www.kai-waehner.de
Construction site management
Collaborative planning
Inventory and asset management
Track, manage, and locate tools and equipment anytime and anywhere
https://www.confluent.io/customers/bosch/
https://events.confluent.io/online-talks/bosch-power-toolse-nables-real-time-analytics-on-iot-event-streams
33

34.
@KaiWaehner - www.kai-waehner.de
Agenda
36
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

35.
@KaiWaehner - www.kai-waehner.de
37

36.
@KaiWaehner - www.kai-waehner.de
Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Industry 4.0
Networking
Communication
Connectivity
Open standards
”Always-on”
38

37.
@KaiWaehner - www.kai-waehner.de
Legacy SIEM needs to evolve
Forwarder
Network traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
• Proprietary forwarders that can only send
data to single source
• Data locked from being shared
• Difficult to scale with growing data volumes
• Prohibitively high indexing costs
• Unable to filter out noisy data
• Slow batch processing
HTTP proxy logs
39

38.
@KaiWaehner - www.kai-waehner.de
AI/ML
Modernized security information and event management
(SIEM)
Filter,
transform,
aggregate
APP SIEM
Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
Syslog
CDC
Network traffic
Firewall logs
RDBMS
Application logs
Sensor Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
40

39.
@KaiWaehner - www.kai-waehner.de
Cyber Intelligence Platform
Leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
41

40.
@KaiWaehner - www.kai-waehner.de
Agenda
42
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

41.
Innovative new business models emerging…
https://www.wejo.com/
https://parknowgroup.com/on-street-cashless-mobile-parking-payments/
https://www.scheidt-bachmann.de/en/article/news/ticketless-parking-
management-system-the-future-has-begun-motorists-can-now-park-and-pay-
without/ 43

42.
@KaiWaehner - www.kai-waehner.de
Software and Digital Services
become the Key Differentiator
44
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

43.
@KaiWaehner - www.kai-waehner.de
Mobility as a Service and Customer 360
Hand in hand integration and data correlation in real-time
45

44.
@KaiWaehner - www.kai-waehner.de
https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/
https://www.confluent.io/thank-you/stream-processing-kafka-uber/
Trillions of messages and
multiple petabytes of data per day
46

45.
@KaiWaehner - www.kai-waehner.de
Ride-Sharing
More than just Messaging! Data Correlation in Real-Time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006
47

46.
@KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago
48

47.
@KaiWaehner - www.kai-waehner.de
Confluent Tiered Storage for Kafka
49

48.
@KaiWaehner - www.kai-waehner.de
‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
50
https://medium.com/porschedev

49.
@KaiWaehner - www.kai-waehner.de
Agenda
51
1. Software is Eating and Connecting the World
2. Connected Vehicles
3. Manufacturing 4.0
4. Cybersecurity
5. Mobility Services and Customer 360
6. Smart City

50.
@KaiWaehner - www.kai-waehner.de
What is a Smart City?
A smart city is an approach to
urbanization that uses innovative
technologies to enhance
community services and
economic opportunities, improves
city infrastructure, reduce costs
and resource consumption, and
increases civic engagement.”
—Smart Cities for Dummies,
Jonathan Reichental
“
52

51.
@KaiWaehner - www.kai-waehner.de
Digital Infrastructure
A Prerequisite for a Smart City
Digital
Infrastructure
Data Centers
Fixed and
Mobile
Connectivity
Network
Infrastructure
Internet of
Things and
Devices
Platforms
and
Applications
53

52.
@KaiWaehner - www.kai-waehner.de
Event Streaming
The Foundation for a Smart City
Mobility
Services
Citizen
Services
Smart
Buildings
Cybersecurity
Smart
Grid
Weather
Data
Citizen
Data
Traffic
Data
Security
Alerts
Streams of real time events
Personal Data
Weather Service
Connected
Cars
Routing
Service
Traffic Alerts
Surveillance
54

53.
@KaiWaehner - www.kai-waehner.de
Public Sector Use Case Examples
for Event Streaming
Citizen Services
• Health services, e.g. hospital modernization, track&trace—Covid distance control
• Efficient and digital citizen engagement, e.g. personal ID application process
• Open exchange, e.g. mobility services (working with partners such as car makers)
Smart City
• Smart driving, parking, buildings, environment
• Waste management
• Mobility services
Energy
• Smart grid and utilities infrastructure (energy distribution,
smart home, smart meters, smart water, etc.)
Security
• Law enforcement, surveillance
• Defense, military
• Cybersecurity

54.
@KaiWaehner - www.kai-waehner.de
Why the “right time” is “real time”…
56

55.
The Smarter City Nervous System
Producers Consumers
• Client Libraries
• Kafka Connectors
• Client Libraries
• Kafka Connectors
X
Consumer Groups 57
Streams

56.
@KaiWaehner - www.kai-waehner.de
The Smarter City Nervous System
Topic (observation-raw)
Metadata and Geo
(lat/long) Enrichment
Topic (observation-meta)
Streams
Health Score &
Incident Detection
Topic (observation-out)
Elasticsearch
Kafka Connect
Elasticsearch Sink
Connector
58

57.
@KaiWaehner - www.kai-waehner.de
NAV (Norwegian Work and Welfare
Department): Life is a Stream of Events
https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/
Assist people through all phases
of life within the domains of
work, family, health, retirement
and social security
59

58.
@KaiWaehner - www.kai-waehner.de
Why Confluent?

59.
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
61

60.
@KaiWaehner - www.kai-waehner.de
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
62

61.
@KaiWaehner - www.kai-waehner.de
Confluent Platform
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

62.
@KaiWaehner - www.kai-waehner.de
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!