Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4.0, Mobility Services, Smart City)

The Rise of Data in Motion in the Automotive Industry
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

@KaiWaehner - www.kai-waehner.de – Data in Motion in the Automotive Industry
The world is changing.

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

Stores become Software

Banks become Software

Taxis become Software
2
min

Cars become Software

Car Production becomes Software

This transformation is
happening everywhere

Automotive and Transportation

Software and Digital Services become the Key Differentiator
13
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

CASE (Connected, Autonomous, Shared, Electrified)
https://wiprodigital.com/2019/08/26/digital-transformation-auto-industry-fueled-by-case/

What enables this
transformation?

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

Real-time Data in Motion beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

This is a fundamental paradigm shift...
19
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

What is Event Streaming?

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Banking
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example

The Rise of Data in Motion
with Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
25

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

Real World Examples

Connected Car Infrastructure at Audi
30
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

Tesla
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

BMW
Decoupled Logistics and Manufacturing
Mission-critical workloads at the edge and in the cloud
• Why Kafka? Decoupling. Transparency. Innovation.
• Why Confluent? Stability is key in manufacturing
• Decoupling between logistics and production systems
• Provide edge platform (self-managed) + Azure Cloud (fully-managed)
+ bidirectional integration
• Use case
• Logistics and supply chain in global plants
• Right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence
• Lot of critical applications
• Things BMW couldn’t do before
• Get IoT data (without interfereing with others), get it to the right
place
• Collect once, process and consume several times (at different
times)
• Enable scalable real-time processing and improve time-to-market
with new applications
32
Jay Kreps, Confluent CEO
Felix Böhm, BMW Plant Digitalization and Cloud Transformation
Keynote at Kafka Summit EU 2021:
https://www.youtube.com/watch?v=3cG2ud7TRs4
(My Notes from the BMW Keynote at Kafka Summit EU 2021)

BMW Group
Industry-ready NLP Service Framework Based on Kafka
https://www.confluent.io/kafka-summit-lon19/industry-ready-nlp-service-framework-kafka/

DriveCentric
A scalable real-time CRM for Automotive Dealerships
Customer 360 with effective customer engagement across all channels
Boost engagement, shorten sales cycles, and spur growth
Focus on business, not infrastructure with Confluent Cloud
34
https://www.confluent.io/customers/drivecentric/

What is Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Integrate and Process
your Events Streams

Kafka Loves Your Existing Systems.
...many more
Other
Systems
Other
Systems
Kafka
Connect
Kafka Cluster
Kafka
Connect

Kafka Stores Your Data Durably.
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
Kafka is the leading system.
Account Activity Replay API to recover events
that weren’t delivered for various reasons
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/kafka-as-a-storage-system.html

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
44

Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
45
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

The Real-Time Spectrum
https://www.embedded.com/introduction-to-real-time/

Kafka is “Real-Time”, but NOT “Hard Real-Time”
OT - Connected Vehicle (Car,
Train, Drone)
OT - Manufacturing
(Field Bus, PLC, Machine, Robot)
IT – Enterprise Software
(Data Center, Cloud, Car IT)
Central Data Center / Public Cloud
Connect
Vehicle Data (e.g. OBD2)
Robot Data All Data
C
C++
C
C++
Java
Python
Go
[#] Hard Real Time
= Deterministic network
with zero spikes + zero latency
[#] Soft Real Time
+ Near Real Time
+ Batch

https://www.confluent.io/thank-you/uber-kafka-uber-worlds-realtime-transit-infrastructure/
https://www.confluent.io/thank-you/stream-processing-kafka-uber/
Trillions of messages and
multiple petabytes of data per day

Ride-Hailing at Lyft
More than just Messaging! Data Correlation in Real-Time
for map-matching, ETA, cost calculation, and much more…
https://eng.lyft.com/a-new-real-time-map-matching-algorithm-at-lyft-da593ab7b006

FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
55

Hybrid Architectures

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
57

Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
60
https://medium.com/porschedev

Porsche’s Streamzilla
A central platform strategy across data centers, clouds, and regions
61
https://medium.com/porschedev

Omnichannel Retail
Time
P
App1 App2
App3
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

Omnichannel Retail
Time
P
App1 App2
App3
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

Omnichannel Retail
Time
P
App1 App2
App3
Location-based
Customer Action
Digital Twin
(R&D, Manufacturing, Customer 360, Aftersales)
Manufacturing
Design Aftersales

Cross-Company Streaming Exchange
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Cluster Linking
Tier 1 Mobility
Service
Streaming integration
between companies
API Management
(REST et al) is not
appropriate for streaming
data
Infosec and politics are your
biggest hurdle
OEM

Here Technologies
Captures location content such as road networks, buildings, parks and traffic patterns
Sells or licenses mapping content, along with map related navigation and location services to other businesses
https://developer.here.com/documentation/data-client-library/dev_guide/client/direct-kafka.html

Open API Platform
for Car Data

Event Streaming Is The Future Of Data
68
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Example Project:
100,000 Connected Cars

70
MQTT Proxy
MongoDB
Storage
MongoDB
Dashboards
Search
Analytics
Kafka Cluster Kafka Connect
Car Sensors
Kafka Streams
Application
All
Data
Critical
Data
Ingest
Data
Potential Detect
TensorFlow
Train Analytic
Model
ksqlDB
Analytic
Model
Preprocess Data Consume
Data
Deploy
Analytic Model
Tiered Storage
Mobile App
BI Tool
A Digital Twin for Predictive Maintenance
Example Project: 100,000 Connected Cars
Kafka Ecosystem
TensorFlow
Other Components
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
71

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
73

Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4.0, Mobility Services, Smart City)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4.0, Mobility Services, Smart City)

Similar to Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4.0, Mobility Services, Smart City) (20)

More from Kai Wähner

More from Kai Wähner (16)

Recently uploaded

Recently uploaded (20)

Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4.0, Mobility Services, Smart City)