The Rise of Data in Motion in the Automotive Industry
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
The world is changing.

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Stores become Software

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Banks become Software

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Taxis become Software
2
min

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Cars become Software

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Car Production becomes Software

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
This transformation is
happening everywhere

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Automotive and Transportation

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Software and Digital Services become the Key Differentiator
13
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
CASE (Connected, Autonomous, Shared, Electrified)
https://wiprodigital.com/2019/08/26/digital-transformation-auto-industry-fueled-by-case/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
What enables this
transformation?

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Real-time Data in Motion beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
What is Event Streaming?

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
The Rise of Data in Motion
with Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
25

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Real World Examples

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Connected Car Infrastructure at Audi
30
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
BMW
Decoupled Logistics and Manufacturing
Mission-critical workloads at the edge and in the cloud
• Why Kafka? Decoupling. Transparency. Innovation.
• Why Confluent? Stability is key in manufacturing
• Decoupling between logistics and production systems
• Provide edge platform (self-managed) + Azure Cloud (fully-managed)
+ bidirectional integration
• Use case
• Logistics and supply chain in global plants
• Right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence
• Lot of critical applications
• Things BMW couldn’t do before
• Get IoT data (without interfereing with others), get it to the right
place
• Collect once, process and consume several times (at different
times)
• Enable scalable real-time processing and improve time-to-market
with new applications
32
Jay Kreps, Confluent CEO
Felix Böhm, BMW Plant Digitalization and Cloud Transformation
Keynote at Kafka Summit EU 2021:
https://www.youtube.com/watch?v=3cG2ud7TRs4
(My Notes from the BMW Keynote at Kafka Summit EU 2021)

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
DriveCentric
A scalable real-time CRM for Automotive Dealerships
Customer 360 with effective customer engagement across all channels
Boost engagement, shorten sales cycles, and spur growth
Focus on business, not infrastructure with Confluent Cloud
34
https://www.confluent.io/customers/drivecentric/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
What is Kafka?

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Integrate and Process
your Events Streams

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Kafka Stores Your Data Durably.
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
Kafka is the leading system.
Account Activity Replay API to recover events
that weren’t delivered for various reasons
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/kafka-as-a-storage-system.html

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
44

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
45
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
Event Topics Storage Partitions
Events / sec Kafka Servers
250,000 500 25,000 25
Event Topics Storage Partitions
Events / sec Kafka Servers
100 5 300 3
Kafka Scales with Your Business.

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
The Real-Time Spectrum
https://www.embedded.com/introduction-to-real-time/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Kafka is “Real-Time”, but NOT “Hard Real-Time”
OT - Connected Vehicle (Car,
Train, Drone)
OT - Manufacturing
(Field Bus, PLC, Machine, Robot)
IT – Enterprise Software
(Data Center, Cloud, Car IT)
Central Data Center / Public Cloud
Connect
Vehicle Data (e.g. OBD2)
Robot Data All Data
C
C++
C
C++
Java
Python
Go
[#] Hard Real Time
= Deterministic network
with zero spikes + zero latency
[#] Soft Real Time
+ Near Real Time
+ Batch

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/
https://www.confluent.io/thank-you/stream-processing-kafka-uber/
Trillions of messages and
multiple petabytes of data per day

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Ride-Hailing at Lyft
More than just Messaging! Data Correlation in Real-Time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
55

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Hybrid Architectures

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
57

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
60
https://medium.com/porschedev

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Porsche’s Streamzilla
A central platform strategy across data centers, clouds, and regions
61
https://medium.com/porschedev

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Omnichannel Retail
Time
P
App1 App2
App3
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Omnichannel Retail
Time
P
App1 App2
App3
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Omnichannel Retail
Time
P
App1 App2
App3
Location-based
Customer Action
Digital Twin
(R&D, Manufacturing, Customer 360, Aftersales)
Manufacturing
Design Aftersales

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Cross-Company Streaming Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
(REST et al) is not
appropriate for streaming
data
Infosec and politics are your
biggest hurdle
OEM

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Here Technologies
Captures location content such as road networks, buildings, parks and traffic patterns
Sells or licenses mapping content, along with map related navigation and location services to other businesses
https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Open API Platform
for Car Data

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Event Streaming Is The Future Of Data
68
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Example Project:
100,000 Connected Cars

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
70
MQTT Proxy
MongoDB
Storage
MongoDB
Dashboards
Search
Analytics
Kafka Cluster Kafka Connect
Car Sensors
Kafka Streams
Application
All
Data
Critical
Data
Ingest
Data
Potential Detect
TensorFlow
Train Analytic
Model
ksqlDB
Analytic
Model
Preprocess Data Consume
Data
Deploy
Analytic Model
Tiered Storage
Mobile App
BI Tool
A Digital Twin for Predictive Maintenance
Example Project: 100,000 Connected Cars
Kafka Ecosystem
TensorFlow
Other Components
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
71

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Why Confluent?

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
73

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!