1. IoT Architectures for a Digital Twin
with Apache Kafka and Event Streaming
A Digital Replica of Things - Open, Scalable and Reliable
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

2. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Abstract
IoT Architectures for a Digital Twin with Apache Kafka and Event Streaming
A digital twin is a digital replica of a living or non-living physical entity. This session discusses the benefits and IoT architectures of a Digital Twin in Industrial IoT (IIoT)
and its relation to Apache Kafka and other IoT frameworks. Kafka is often used as central event streaming platform to build a scalable and reliable digital twin for real
time streaming sensor data. A live demo shows a scalable digital twin infrastructure for condition monitoring and predictive maintenance in real time for a connected
car infrastructure leveraging Kafka, MQTT and TensorFlow.
Key Take-Aways:
● Learn about use cases and characteristics of a digital twin in various industries
● Understand how to build a digital twin for every single (of tens of thousands) IoT device or machine
● See different IoT architectures with Kafka and other IoT technologies and products
● Bring added value to your IoT infrastructure by enabling use cases like predictive maintenance
● Understand how the Apache Kafka enables scalable and flexible end-to-end integration processing from IIoT data to various backend applications
● Watch a live demo of an end-to-end integration, real time processing and analytics of thousands of IoT devices

3. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Key Takeaways
• A Digital Twin merges the physical and the digital world
• Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin
• Event Streaming complements IoT platforms and other backend applications / databases.
+
3

4. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
4

5. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
5

6. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Software and Digital Services become the Key Differentiator
6
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

7. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin – Merging the Physical and the Digital World
7
• Downtime reduction
• Inventory management
• Fleet management
• What-if simulations
• Operational planning
• Servitization
• Product development
• Healthcare
• Customer experience
“Virtual representation of something else (Physical thing, process, service)”
“A living model that drives a business outcome”
https://www.youtube.com/watch?v=Ri0TD7kYsIQ

8. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Granularity of Digital Twins
8
https://www.youtube.com/watch?v=cfbKR48nSyQ
Remaining Useful Life

9. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Thread
9
Digital Twin vs. Digital Thread?
I only use the term Digital Twin in the following slides.
Both terms overlap, often meaning the same.
Span
the
entire
lifecycle

10. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Virtual Singapore:
A Digital Twin of the (Smart) City
10
Design, Monitor and Manage Cities
• Urban Planning (e.g. Crowd Simulation)
• Collaboration and Decision-Making
• Communication and Visualisation
• Improved Accessibility
• Analysis on Potential for Solar Energy
Production
• …
https://www.nrf.gov.sg/programmes/virtual-singapore

11. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Smart Infrastructure:
Digital Solutions for Entire Building Lifecycle
11
https://new.siemens.com/global/en/products/buildings/digitalization/digital-building-lifecycle.html
• Safer, more secure, more efficient and resilient buildings
• Continuously interaction, learning and adaption to create environments that care
• Follow the entire digital building lifecycle

12. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Connected Car Infrastructure
12
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

13. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Twinning the Human Body to Enhance Medical Care
13
• Monitoring and evaluation without being in close proximity
• Testing the impact of changes on the performance of a system
• Smart machines are more advanced than humans
• Determine what actions to take
• Modelling an individual’s genomic makeup, physiological characteristics, and lifestyle to create personalized medicine
• Capturing the human body will have multiple benefits for doctors such as discovering undeveloped illnesses,
experimenting with treatments, and improving preparation for surgeries
https://www.challenge.org/insights/digital-twin-in-healthcare/
https://youtu.be/H6JzPCbyVSM

14. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Market
2020
14
https://www.youtube.com/watch?v=w4TPeVO3dwo (Q1, 2020 Webinar IoT Analytics Research)

15. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin and Artificial Intelligence (AI) / Machine Learning
• Complementary Concepts
• Continuous Learning, Monitoring and Acting
• (Good) data is key for success
15
https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b

16. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin Applied…
16
https://www.youtube.com/watch?v=cfbKR48nSyQ

17. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin Applied…
17
https://www.youtube.com/watch?v=cfbKR48nSyQ

18. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin Applied…
18
https://www.youtube.com/watch?v=cfbKR48nSyQ

19. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin Applied…
19
https://www.youtube.com/watch?v=cfbKR48nSyQ

20. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
20

21. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
History of Automation Industry vs. Big Data and Cloud
https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf

22. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Challenges in Automation Industry
IoT != IIoT != Buildings != Healthcare …
• IoT = Connected cars, smart home, … à Large scale, secure, scalable, open,
modern technologies
• IIoT / Buildings = Not connected at all or slow, insecure, not scalable,
proprietary
• Healthcare = Often not connected at all yet (huge security requirements)
Legacy / Proprietary IIoT Technologies
• Usually incompatible protocols, typically proprietary
• Usually serial connections (very low latency, nanoseconds) - with TCP /
UDP wrapper around it to integrate with “external world”
• Siemens S7, Modbus, Beckhoff, Profinet, Allen Bradley, etc.
• OPC-UA (required machine update + license cost)
Product Lifecycles
• Long lifecycle (tens of years)
• Factories and buildings cost millions, no simple changes / upgrades
• Still using Windows 7 without Service Packs => Usability and security issues
• Mantra: “Stay with your well-known vendor forever”

23. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Challenges in Automation Industry
Monoliths
• No scalability
• No extendibility
• No real failover (start your backup machine)
Missing Security Capabilities
• Security in software development ==
Authentication, Authorization, Antivirus, SSL,
SASL, Kerberos
• Security in automation industry == Safety
• “if you press the red button, the machine stops
immediately”
• Insecure by nature => No Authentication /
Authorization / Encryption
• Mantra: “Our factory building and network is
secure, no access from outside”
• Contradicts with “move to cloud and big data
analytics”

24. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Trends: Evolution of Convergence between IT and Industrial Automation
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence

25. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Complexity, Cost and Scalability are Main Blockers
25

26. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

27. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
27

28. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
600+ IoT Platforms
28
https://iot-analytics.com/iot-platform-companies-landscape-2020/

29. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Proprietary IoT Platforms
• Sophisticated integration for related IIoT protocols (like Siemens S7, Modbus, etc.) and standards
(like OPC-UA)
• Not a single product (plenty of acquisitions, OEMs and different code bases are typically the
foundation)
• Typically very expensive
• Proprietary (just open interfaces)
• Limited scalability
29

30. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT Offerings from Cloud Providers
• Sophisticated tools for IoT management (devices, shadowing, …)
• Good integration with other cloud services (storage, analytics, …)
• Vendor lock-in
• No focus on hybrid and edge (but on prem products)
• Limited scalability
• Often high cost (beyond ’hello world’)
30

31. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Standards-based / Open Source IoT Platforms
• Open and standards-based (e.g. MQTT)
• Open source / open core business model
• Infrastructure-independent
• Different vendors behind the core
technologies
• Sometimes less mature or non-existent
connectivity (especially to legacy and
proprietary protocols)
• Trade-off: Solid offering for one standard
(e.g. HiveMQ for MQTT) or diversity but
not for mission-critical scale (e.g. Node-
RED)
31

32. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
32

33. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka - The Rise of an Event Streaming Platform
33
=
Messaging
+
Storage
+
Integration
+
Processing

34. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka is not just used for big data
** Kafka Is not just used by tech giants
34

35. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
Confluent - Business Value per Use Case
35

36. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
P
Decoupling of Producers and Consumers
Time
C2 C3C1
36

37. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Schema Enforcement (Client and Server Side)
37

38. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
MQTT
Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
OPC-UA
Integration
Real Time
Predictions
IoT Platform
Connector
Java / Python /
”you-name-it”
Schema
Registry
Event Streaming Platform
IoT Domain Legacy Domain Analytics Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
38

39. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
10 Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
39

40. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin and AI / Machine Learning (with Kafka)
• Complementary Concepts
• Continuous Learning, Monitoring and Acting à Real time, scalable
• (Good) data is key for success à Integration, data processing
40

41. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache Kafka and Machine Learning – More Details
41
Blog: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka:
https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/
Blog: Using Apache Kafka to Drive Cutting-Edge Machine Learning:
https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/
Blog: Machine Learning and Real-Time Analytics in Apache Kafka Applications:
https://www.confluent.io/blog/machine-learning-real-time-analytics-models-in-kafka-applications/
Blog: Streaming Machine Learning with Tiered Storage and Without a Data Lake:
https://www.confluent.io/blog/streaming-machine-learning-with-tiered-storage/
Slides + Video: Event-driven Model Scoring vs. RPC with Model Server:
https://www.confluent.io/kafka-summit-san-francisco-2019/event-driven-model-serving-stream-processing-vs-rpc-with-kafka-
and-tensorflow/
Few examples for real time scoring with Kafka Steams and TensorFlow, H2O, etc.:
https://github.com/kaiwaehner/kafka-streams-machine-learning-examples
Demo: Anomaly Detection with KSQL:
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
Demo + Video: Connected Car demo with Kafka + Streaming ML
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

42. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Hold on…
Kafka is NOT
an IoT Platform!

43. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Device management
Unreliable networks
Connectivity beyond standards
Edge hardware
…

44. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
(De facto) Standards for Processing IoT Data
A Match Made In Heaven
+ =
IoT
Platform

45. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
45

46. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Characteristics of Digital Twin Technology
• Connectivity
• Physical assets, enterprise software, customers
• Bidirectional communication to ingest, command and control
• Homogenization
• Decoupling and standardization
• Virtualization of information
• Shared with multiple agents, unconstrained by physical location or time
• Lower cost and easier testing, development and predictions
• Reprogrammable and smart
• Adjust and improve characteristics and develop new version of a product
• Digital traces
• Go back in time and analyse historical events to diagnose problems
• Modularity
• Design and customization of products and production modules
• Tweak modules of models and machines
46

47. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Twin Mapped to Apache Kafka
• Connectivity – Kafka Connect provides connectivity as scale in real time to IoT interfaces, big data
solutions and cloud services. The Kafka ecosystem is complementary, NOT competitive to other
Middleware and IoT Platforms.
• Homogenization – Real decoupling between clients (i.e. producers and consumers) is one of the
key strengths of Kafka. Schema management and enforcement leveraging different technologies
(JSON Schema, Avro, Profobuf, etc.) enables data awareness and standardization.
• Reprogrammable and smart – Kafka is the de facto standard for microservices for exactly this
reason: Separation of concerns and domain-driven design (DDD). Deploy new decoupled
applications and versions, do A/B testing, canarying.
• Digital traces – Kakfa is a distributed commit log. Events are appended, stored as long as you
want (potentially forever with rentention time = -1) and immutable. Seriously, what other
technology could be used better to build a digital trace for a digital twin?
• Modularity – The Kafka infrastructure itself is modular and scalable. This includes components like
Kafka brokers, Connect, Schema Registry, REST Proxy and client applications in different
languages like Java, Scala, Python, Go, .NET, C++ and others. With this modularity, you can easily
build the right Digital Twin architecture your your edge, hybrid or global scenarios and also
combine the Kafka components with any other IoT solutions.
47

48. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Simplified Pipeline for a Digital Twin
48
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Real
Time
App
Batch
App
Request
Response
App
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

49. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Scenario 1: Digital Twin Monolith
49
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Device Mgt.
Analytics
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

50. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Scenario 2: Digital Twin as External Database
50
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Device Mgt.
Database
XYZAnalytics
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

51. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache
Kafka
Scenario 3: Kafka as Backbone for the
Digital Twin and the Rest of the Enterprise
51
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Database
XYZ
Real
Time
App
Batch
App
Request
Response
App
Kafka
Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

52. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache Kafka
Scenario 4: Kafka as IoT Platform
52
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT Cloud
Platform
Digital
Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
Storage Processing

53. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache Kafka
Scenario 5: Kafka as Digital Twin
53
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
Digital Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Storage Processing
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

54. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Kafka as
Database?
Seriously?

55. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
P
Kafka’s Storage – A Distributed Commit Log
Time
C2 C3C1
55

56. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
(Only available in Confluent Platform)
www.kai-waehner.de | @KaiWaehner

57. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Distributed System with Replication and High Availability
on Server and Client Side
read,
write
Kafka Client Kafka Server Side
Materialized View
in the Client App
(In-memory, RocksDB)

58. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Stateful Kafka Client Applications
58

59. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Kafka as Data Storage?
59
https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-transactions-sql-nosql-data-lake/

60. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
60

61. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
No matter which Digital Twin
Architecture I use…
Most Architectures are Hybrid
(Edge, Data Center, Cloud)
and sometimes even Global!

62. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Edge Digital Twin
Single
Kafka Broker
(or Cluster)
Digital Twin
Self-managed or
certified OEM Hardware
Kafka
Cluster
in DC /
Cloud
Replicator
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

63. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Centralized Digital Twin
Digital Twin
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS

64. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Global Digital Twin Architecture
Multiple Clusters and Aggregation
Factories à Analytics Cluster
Multi-Region Cluster
High Availability (Disaster Recovery)
Global Data Streaming
Outsourced
Development

65. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Example of a Multi-Region Digital Twin Deployment
Order
Sensor
Order
Sensor
Logs Logs
synchronous
asynchronous
● Automate Disaster
Recovery
● Sync or Async Replication
per Topic
● Offset Preserving
● Automated Client Failover
with No Custom Code
Zero downtime, zero data loss
(even in cases of data center outage)
US West US East

66. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Architecture patterns for distributed, hybrid,
edge and global Apache Kafka deployments
www.kai-waehner.de | @KaiWaehner
https://www.kai-waehner.de/blog/2020/01/29/deployment-patterns-distributed-hybrid-edge-global-multi-data-center-kafka-architecture/

67. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
67

68. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
A Digital Twin with Kafka and TensorFlow
68
MQTT
Proxy
Elastic Grafana
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic
Model
Consume
Data
Preprocess
Data
Analytic
Model
Deploy
Analytic
Model
Python
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

69. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Architecture for 100000 Connected Cars
Kafka + KSQL + MQTT + TensorFlow + Kubernetes
69
https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/

70. IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Key Takeaways
• A Digital Twin merges the physical and the digital world
• Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin
• Event Streaming complements IoT platforms and other backend applications / databases.
+
70

71. Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!