IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning

IoT Architectures for a Digital Twin
with Apache Kafka and Event Streaming
A Digital Replica of Things - Open, Scalable and Reliable
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

IoT, Digital Twin and Event Streaming – @KaiWaehner - www.kai-waehner.de
Abstract
IoT Architectures for a Digital Twin with Apache Kafka and Event Streaming
A digital twin is a digital replica of a living or non-living physical entity. This session discusses the benefits and IoT architectures of a Digital Twin in Industrial IoT (IIoT)
and its relation to Apache Kafka and other IoT frameworks. Kafka is often used as central event streaming platform to build a scalable and reliable digital twin for real
time streaming sensor data. A live demo shows a scalable digital twin infrastructure for condition monitoring and predictive maintenance in real time for a connected
car infrastructure leveraging Kafka, MQTT and TensorFlow.
Key Take-Aways:
● Learn about use cases and characteristics of a digital twin in various industries
● Understand how to build a digital twin for every single (of tens of thousands) IoT device or machine
● See different IoT architectures with Kafka and other IoT technologies and products
● Bring added value to your IoT infrastructure by enabling use cases like predictive maintenance
● Understand how the Apache Kafka enables scalable and flexible end-to-end integration processing from IIoT data to various backend applications
● Watch a live demo of an end-to-end integration, real time processing and analytics of thousands of IoT devices

Key Takeaways
• A Digital Twin merges the physical and the digital world
• Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin
• Event Streaming complements IoT platforms and other backend applications / databases.
+
3

Agenda
• Digital Twin - Merging the Physical and the Digital World
• Real World Challenges
• IoT Platforms
• Apache Kafka as Event Streaming Solution for IoT
• Spoilt for Choice for a Digital Twin
• Global IoT Architectures
• A Digital Twin for 100000 Connected Cars
4

Agenda
• IoT Platforms
5

Software and Digital Services become the Key Differentiator
6
https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery

Digital Twin – Merging the Physical and the Digital World
7
• Downtime reduction
• Inventory management
• Fleet management
• What-if simulations
• Operational planning
• Servitization
• Product development
• Healthcare
• Customer experience
“Virtual representation of something else (Physical thing, process, service)”
“A living model that drives a business outcome”
https://www.youtube.com/watch?v=Ri0TD7kYsIQ

Granularity of Digital Twins
8
https://www.youtube.com/watch?v=cfbKR48nSyQ
Remaining Useful Life

Digital Thread
9
Digital Twin vs. Digital Thread?
I only use the term Digital Twin in the following slides.
Both terms overlap, often meaning the same.
Span
the
entire
lifecycle

Virtual Singapore:
A Digital Twin of the (Smart) City
10
Design, Monitor and Manage Cities
• Urban Planning (e.g. Crowd Simulation)
• Collaboration and Decision-Making
• Communication and Visualisation
• Improved Accessibility
• Analysis on Potential for Solar Energy
Production
• …
https://www.nrf.gov.sg/programmes/virtual-singapore

Smart Infrastructure:
Digital Solutions for Entire Building Lifecycle
11
https://new.siemens.com/global/en/products/buildings/digitalization/digital-building-lifecycle.html
• Safer, more secure, more efficient and resilient buildings
• Continuously interaction, learning and adaption to create environments that care
• Follow the entire digital building lifecycle

Connected Car Infrastructure
12
https://www.youtube.com/watch?v=yGLKi3TMJv8
• Real Time Data Analysis
• Swarm Intelligence
• Collaboration with Partners
• Predictive AI
• …

Twinning the Human Body to Enhance Medical Care
13
• Monitoring and evaluation without being in close proximity
• Testing the impact of changes on the performance of a system
• Smart machines are more advanced than humans
• Determine what actions to take
• Modelling an individual’s genomic makeup, physiological characteristics, and lifestyle to create personalized medicine
• Capturing the human body will have multiple benefits for doctors such as discovering undeveloped illnesses,
experimenting with treatments, and improving preparation for surgeries
https://www.challenge.org/insights/digital-twin-in-healthcare/
https://youtu.be/H6JzPCbyVSM

Market
2020
14
https://www.youtube.com/watch?v=w4TPeVO3dwo (Q1, 2020 Webinar IoT Analytics Research)

Digital Twin and Artificial Intelligence (AI) / Machine Learning
• Complementary Concepts
• Continuous Learning, Monitoring and Acting
• (Good) data is key for success
15
https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b

Digital Twin Applied…
16

17

18

19

Agenda
• IoT Platforms
20

History of Automation Industry vs. Big Data and Cloud
https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf

Challenges in Automation Industry
IoT != IIoT != Buildings != Healthcare …
• IoT = Connected cars, smart home, … à Large scale, secure, scalable, open,
modern technologies
• IIoT / Buildings = Not connected at all or slow, insecure, not scalable,
proprietary
• Healthcare = Often not connected at all yet (huge security requirements)
Legacy / Proprietary IIoT Technologies
• Usually incompatible protocols, typically proprietary
• Usually serial connections (very low latency, nanoseconds) - with TCP /
UDP wrapper around it to integrate with “external world”
• Siemens S7, Modbus, Beckhoff, Profinet, Allen Bradley, etc.
• OPC-UA (required machine update + license cost)
Product Lifecycles
• Long lifecycle (tens of years)
• Factories and buildings cost millions, no simple changes / upgrades
• Still using Windows 7 without Service Packs => Usability and security issues
• Mantra: “Stay with your well-known vendor forever”

Challenges in Automation Industry
Monoliths
• No scalability
• No extendibility
• No real failover (start your backup machine)
Missing Security Capabilities
• Security in software development ==
Authentication, Authorization, Antivirus, SSL,
SASL, Kerberos
• Security in automation industry == Safety
• “if you press the red button, the machine stops
immediately”
• Insecure by nature => No Authentication /
Authorization / Encryption
• Mantra: “Our factory building and network is
secure, no access from outside”
• Contradicts with “move to cloud and big data
analytics”

Trends: Evolution of Convergence between IT and Industrial Automation
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence

Complexity, Cost and Scalability are Main Blockers
25

Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

Agenda
• IoT Platforms
27

600+ IoT Platforms
28
https://iot-analytics.com/iot-platform-companies-landscape-2020/

Proprietary IoT Platforms
• Sophisticated integration for related IIoT protocols (like Siemens S7, Modbus, etc.) and standards
(like OPC-UA)
• Not a single product (plenty of acquisitions, OEMs and different code bases are typically the
foundation)
• Typically very expensive
• Proprietary (just open interfaces)
• Limited scalability
29

IoT Offerings from Cloud Providers
• Sophisticated tools for IoT management (devices, shadowing, …)
• Good integration with other cloud services (storage, analytics, …)
• Vendor lock-in
• No focus on hybrid and edge (but on prem products)
• Limited scalability
• Often high cost (beyond ’hello world’)
30

Standards-based / Open Source IoT Platforms
• Open and standards-based (e.g. MQTT)
• Open source / open core business model
• Infrastructure-independent
• Different vendors behind the core
technologies
• Sometimes less mature or non-existent
connectivity (especially to legacy and
proprietary protocols)
• Trade-off: Solid offering for one standard
(e.g. HiveMQ for MQTT) or diversity but
not for mission-critical scale (e.g. Node-
RED)
31

Agenda
• IoT Platforms
32

The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka - The Rise of an Event Streaming Platform
33
=
Messaging
+
Storage
+
Integration
+
Processing

Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka is not just used for big data
** Kafka Is not just used by tech giants
34

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
Confluent - Business Value per Use Case
35

P
Decoupling of Producers and Consumers
Time
C2 C3C1
36

Schema Enforcement (Client and Server Side)
37

Kafka Connect
Kafka Cluster
MQTT
Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
OPC-UA
Integration
Real Time
Predictions
IoT Platform
Connector
Java / Python /
”you-name-it”
Schema
Registry
Event Streaming Platform
IoT Domain Legacy Domain Analytics Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
38

10 Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
39

Digital Twin and AI / Machine Learning (with Kafka)
• Complementary Concepts
• Continuous Learning, Monitoring and Acting à Real time, scalable
• (Good) data is key for success à Integration, data processing
40

Apache Kafka and Machine Learning – More Details
41
Blog: How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka:
https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/
Blog: Using Apache Kafka to Drive Cutting-Edge Machine Learning:
https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/
Blog: Machine Learning and Real-Time Analytics in Apache Kafka Applications:
https://www.confluent.io/blog/machine-learning-real-time-analytics-models-in-kafka-applications/
Blog: Streaming Machine Learning with Tiered Storage and Without a Data Lake:
https://www.confluent.io/blog/streaming-machine-learning-with-tiered-storage/
Slides + Video: Event-driven Model Scoring vs. RPC with Model Server:
https://www.confluent.io/kafka-summit-san-francisco-2019/event-driven-model-serving-stream-processing-vs-rpc-with-kafka-
and-tensorflow/
Few examples for real time scoring with Kafka Steams and TensorFlow, H2O, etc.:
https://github.com/kaiwaehner/kafka-streams-machine-learning-examples
Demo: Anomaly Detection with KSQL:
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot
Demo + Video: Connected Car demo with Kafka + Streaming ML
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference

Hold on…
Kafka is NOT
an IoT Platform!

Device management
Unreliable networks
Connectivity beyond standards
Edge hardware
…

(De facto) Standards for Processing IoT Data
A Match Made In Heaven
+ =
IoT
Platform

Agenda
• IoT Platforms
45

Characteristics of Digital Twin Technology
• Connectivity
• Physical assets, enterprise software, customers
• Bidirectional communication to ingest, command and control
• Homogenization
• Decoupling and standardization
• Virtualization of information
• Shared with multiple agents, unconstrained by physical location or time
• Lower cost and easier testing, development and predictions
• Reprogrammable and smart
• Adjust and improve characteristics and develop new version of a product
• Digital traces
• Go back in time and analyse historical events to diagnose problems
• Modularity
• Design and customization of products and production modules
• Tweak modules of models and machines
46

Digital Twin Mapped to Apache Kafka
• Connectivity – Kafka Connect provides connectivity as scale in real time to IoT interfaces, big data
solutions and cloud services. The Kafka ecosystem is complementary, NOT competitive to other
Middleware and IoT Platforms.
• Homogenization – Real decoupling between clients (i.e. producers and consumers) is one of the
key strengths of Kafka. Schema management and enforcement leveraging different technologies
(JSON Schema, Avro, Profobuf, etc.) enables data awareness and standardization.
• Reprogrammable and smart – Kafka is the de facto standard for microservices for exactly this
reason: Separation of concerns and domain-driven design (DDD). Deploy new decoupled
applications and versions, do A/B testing, canarying.
• Digital traces – Kakfa is a distributed commit log. Events are appended, stored as long as you
want (potentially forever with rentention time = -1) and immutable. Seriously, what other
technology could be used better to build a digital trace for a digital twin?
• Modularity – The Kafka infrastructure itself is modular and scalable. This includes components like
Kafka brokers, Connect, Schema Registry, REST Proxy and client applications in different
languages like Java, Scala, Python, Go, .NET, C++ and others. With this modularity, you can easily
build the right Digital Twin architecture your your edge, hybrid or global scenarios and also
combine the Kafka components with any other IoT solutions.
47

Simplified Pipeline for a Digital Twin
48
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT
Platform
Digital
Twin
Real
Time
App
Batch
App
Request
Response
App
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity

Scenario 1: Digital Twin Monolith
49
IoT
Platform
Digital
Twin
Device Mgt.
Analytics
Connectivity
Homogenization
Digital traces
Modularity

Scenario 2: Digital Twin as External Database
50
IoT
Platform
Digital
Twin
Device Mgt.
Database
XYZAnalytics
Connectivity
Homogenization
Digital traces
Modularity

Apache
Kafka
Scenario 3: Kafka as Backbone for the
Digital Twin and the Rest of the Enterprise
51
IoT
Platform
Digital
Twin
Database
XYZ
Real
Time
App
Batch
App
Request
Response
App
Kafka
Connect
Connectivity
Homogenization
Digital traces
Modularity

Apache Kafka
Scenario 4: Kafka as IoT Platform
52
IoT Cloud
Platform
Digital
Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Connectivity
Homogenization
Digital traces
Modularity
Storage Processing

Apache Kafka
Scenario 5: Kafka as Digital Twin
53
Digital Twin
Real
Time
App
Batch
App
Request
Response
App
Kafka Connect
Storage Processing
Connectivity
Homogenization
Digital traces
Modularity

Kafka as
Database?
Seriously?

P
Kafka’s Storage – A Distributed Commit Log
Time
C2 C3C1
55

Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
(Only available in Confluent Platform)
www.kai-waehner.de | @KaiWaehner

Distributed System with Replication and High Availability
on Server and Client Side
read,
write
Kafka Client Kafka Server Side
Materialized View
in the Client App
(In-memory, RocksDB)

Stateful Kafka Client Applications
58

Kafka as Data Storage?
59
https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-replace-database-acid-storage-transactions-sql-nosql-data-lake/

Agenda
• IoT Platforms
60

No matter which Digital Twin
Architecture I use…
Most Architectures are Hybrid
(Edge, Data Center, Cloud)
and sometimes even Global!

Edge Digital Twin
Single
Kafka Broker
(or Cluster)
Digital Twin
Self-managed or
certified OEM Hardware
Kafka
Cluster
in DC /
Cloud
Replicator

Centralized Digital Twin
Digital Twin
Single
Kafka Broker
(or Cluster)
Self-managed or
Single
Kafka Broker
(or Cluster)
Self-managed or

Global Digital Twin Architecture
Multiple Clusters and Aggregation
Factories à Analytics Cluster
Multi-Region Cluster
High Availability (Disaster Recovery)
Global Data Streaming
Outsourced
Development

Example of a Multi-Region Digital Twin Deployment
Order
Sensor
Order
Sensor
Logs Logs
synchronous
asynchronous
● Automate Disaster
Recovery
● Sync or Async Replication
per Topic
● Offset Preserving
● Automated Client Failover
with No Custom Code
Zero downtime, zero data loss
(even in cases of data center outage)
US West US East

Architecture patterns for distributed, hybrid,
edge and global Apache Kafka deployments
www.kai-waehner.de | @KaiWaehner
https://www.kai-waehner.de/blog/2020/01/29/deployment-patterns-distributed-hybrid-edge-global-multi-data-center-kafka-architecture/

Agenda
• IoT Platforms
67

A Digital Twin with Kafka and TensorFlow
68
MQTT
Proxy
Elastic Grafana
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic
Model
Consume
Data
Preprocess
Data
Analytic
Model
Deploy
Analytic
Model
Python
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Connectivity
Homogenization
Digital traces
Modularity

Architecture for 100000 Connected Cars
Kafka + KSQL + MQTT + TensorFlow + Kubernetes
69
https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/

Key Takeaways
• A Digital Twin merges the physical and the digital world
• Apache Kafka enables an open, scalable and reliable infrastructure for a Digital Twin
• Event Streaming complements IoT platforms and other backend applications / databases.
+
70

Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning

Similar to IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Machine Learning