Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Using MQTT, Kafka and InfluxDB 2.0 on Kubernetes | InfluxDays Virtual Experience London 2020

1. IoT Architectures for a Digital Twin
with Apache Kafka and InfluxDB
A Digital Replica of Things - Open, Scalable and Reliable
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
3

3. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
4

4. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Software and Digital Services become the Key Differentiator
5
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

5. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin – Merging the Physical and the Digital World
6
• Downtime reduction
• Inventory management
• Fleet management
• What-if simulations
• Operational planning
• Servitization
• Product development
• Healthcare
• Customer experience
“Virtual representation of something else (Physical thing, process, service)”
“A living model that drives a business outcome”
https://www.youtube.com/watch?v=Ri0TD7kYsIQ

6. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Smart Infrastructure:
Digital Solutions for Entire Building Lifecycle
7
https://new.siemens.com/global/en/products/buildings/digitalization/digital-building-lifecycle.html

7. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Connected Car Infrastructure
8
https://www.youtube.com/watch?v=yGLKi3TMJv8

8. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Twinning the Human Body to Enhance Medical Care
9
https://www.challenge.org/insights/digital-twin-in-healthcare/
https://youtu.be/H6JzPCbyVSM

9. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin and Artificial Intelligence (AI) / Machine Learning
• Complementary Concepts
• Continuous Learning, Monitoring and Acting
• (Good) Data is key for success
10
https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b

10. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
11

11. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
History of Automation Industry vs. Big Data and Cloud
https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf

12. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Trends: Evolution of Convergence between IT and Industrial Automation
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence

13. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Complexity, Cost and Scalability are Main Blockers
14

14. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

15. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
16

16. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka - The Rise of an Event Streaming Platform
17
=
Messaging
+
Storage
+
Integration
+
Processing

17. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
P
Decoupling of Producers and Consumers
Time
C2 C3C1
18

18. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
19

19. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
Confluent - Business Value per Use Case
20

20. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
10 Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
21

21. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin and AI / Machine Learning (with Kafka)
• Complementary Concepts
• Continuous Learning, Monitoring and Acting à Real time, scalable
• (Good) data is key for success à Integration, data processing
22

22. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Hold on…
Kafka is NOT
an IoT Platform!

23. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Device management
Unreliable networks
Connectivity beyond standards
Edge hardware
…

24. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
25

25. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
600+ IoT Platforms
26
https://iot-analytics.com/iot-platform-companies-landscape-2020/

26. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Proprietary IoT Platforms
27

27. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
IoT Offerings from Cloud Providers
28

28. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Standards-based / Open Source IoT Platforms
29

29. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
30

30. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Characteristics of Digital Twin Technology
• Connectivity
• Physical assets, enterprise software, customers
• Bidirectional communication
• Homogenization
• Decoupling and standardization
• Virtualization of information
• Shared with multiple agents
• Lower cost
• Reprogrammable and smart
• Adjust and improve characteristics
• Digital traces
• Diagnose problems
• Modularity
• Tweak modules of models and machines
31

31. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Scenario 1: Digital Twin Monolith
32
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Device Mgt.
Analytics
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

32. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Scenario 2: Digital Twin as External Database
33
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Device Mgt.
InfluxDB
Analytics
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

33. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache
Kafka
Scenario 3: Kafka as Backbone for the
Digital Twin and the Rest of the Enterprise
34
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
InfluxDB
Real
Time
App
Batch
App
Request
Response
App
Kafka
Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

34. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache Kafka
Scenario 4: Kafka as IoT Platform
35
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
Digital
Twin
InfluxD
B
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Storage Processing

35. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
36

36. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Building a Digital Twin with Kafka and InfluxDB
Apache Kafka
• Integration
• Decoupling and Backpressure
• Data Processing
• Ingest into InfluxDB
• Consume from InfluxDB
• Consumption by other Applications
InfluxDB
• Storage
• Batch and Real Time Analytics
• Dashboards
Þ Open
Þ Scalable
Þ Mission-critical
37
Data Lake
Batch Analytics
Kafka Streams /
ksqlDB
Stream
Processing
Databases
Message Queues
Sensors
Applications

37. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Edge Digital Twin
Single Broker
(or Cluster)
Digital Twin
Self-managed or
certified OEM Hardware
Kafka
Cluster
in DC /
Cloud
Replicator
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

38. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Centralized Digital Twin
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

39. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Global Digital Twin Architecture
Multiple Clusters and Aggregation
Factories à Analytics Cluster
Multi-Region Cluster
High Availability (Disaster Recovery)
Global Data Streaming
Outsourced
Development

40. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• Apache Kafka as Event Streaming Solution for IoT
• IoT Platforms
• Spoilt for Choice for a Digital Twin
• IoT Architectures with Kafka and InfluxDB
• A Digital Twin for 100000 Connected Cars
41

41. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
A Digital Twin with
Kafka, TensorFlow and InfluxDB
42
MQTT
Proxy
InfluxDB
Storage
InfluxDB
Dashboards
+
Analytics
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
InfluxDB
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic
Model
Consume
Data
Preprocess
Data
Analytic
Model
Deploy
Analytic
Model
Python
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Mobile App
BI Tool

42. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Architecture for 100000 Connected Cars
Kafka + KSQL + MQTT + TensorFlow + Kubernetes
43
https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/

43. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Kafka Connect Connector for InfluxDB
44
https://www.confluent.io/hub/confluentinc/kafka-connect-influxdb

44. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Key Takeaways
• A Digital Twin merges the physical and the digital world
• Apache Kafka + InfluxDB enable an open, scalable and reliable infrastructure for a Digital Twin
• Event Streaming complements IoT platforms and other backend applications / databases.
+
45

45. Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!