Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
IoT Architectures for a Digital Twin
with Apache Kafka and InfluxDB
A Digital Replica of Things - Open, Scalable and Relia...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Software and Digital Services becom...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin – Merging the Physical...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Smart Infrastructure:
Digital Solut...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Connected Car Infrastructure
8
http...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Twinning the Human Body to Enhance ...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin and Artificial Intelli...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
History of Automation Industry vs. ...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Trends: Evolution of Convergence be...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Complexity, Cost and Scalability ar...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Huge demand to build an open, flexi...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
The Log ConnectorsConnectors
Produc...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
P
Decoupling of Producers and Consu...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giant...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Improve
Customer
Experience
(CX)
In...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
10 Reasons for Event Streaming with...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Digital Twin and AI / Machine Learn...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Hold on…
Kafka is NOT
an IoT Platfo...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Device management
Unreliable networ...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
600+ IoT Platforms
26
https://iot-a...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Proprietary IoT Platforms
27
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
IoT Offerings from Cloud Providers
...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Standards-based / Open Source IoT P...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Characteristics of Digital Twin Tec...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Scenario 1: Digital Twin Monolith
3...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Scenario 2: Digital Twin as Externa...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache
Kafka
Scenario 3: Kafka as B...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Apache Kafka
Scenario 4: Kafka as I...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Building a Digital Twin with Kafka ...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Edge Digital Twin
Single Broker
(or...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Centralized Digital Twin
Single
Kaf...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Global Digital Twin Architecture
Mu...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Agenda
• Digital Twin - Merging the...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
A Digital Twin with
Kafka, TensorFl...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Architecture for 100000 Connected C...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Kafka Connect Connector for InfluxD...
IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de
Key Takeaways
• A Digital Twin merg...
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Question...
Upcoming SlideShare
Loading in …5
×

Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Using MQTT, Kafka and InfluxDB 2.0 on Kubernetes | InfluxDays Virtual Experience London 2020

153 views

Published on

Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Using MQTT, Kafka and InfluxDB 2.0 on Kubernetes | InfluxDays Virtual Experience London 2020

Published in: Technology
  • Be the first to comment

Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Using MQTT, Kafka and InfluxDB 2.0 on Kubernetes | InfluxDays Virtual Experience London 2020

  1. 1. IoT Architectures for a Digital Twin with Apache Kafka and InfluxDB A Digital Replica of Things - Open, Scalable and Reliable Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  2. 2. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 3
  3. 3. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 4
  4. 4. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Software and Digital Services become the Key Differentiator 5 https://www.mckinsey.com/industries/advanced-electronics/our-insights/iiot-platforms-the-technology-stack-as-value-driver-in-industrial-equipment-and-machinery
  5. 5. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Digital Twin – Merging the Physical and the Digital World 6 • Downtime reduction • Inventory management • Fleet management • What-if simulations • Operational planning • Servitization • Product development • Healthcare • Customer experience “Virtual representation of something else (Physical thing, process, service)” “A living model that drives a business outcome” https://www.youtube.com/watch?v=Ri0TD7kYsIQ
  6. 6. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Smart Infrastructure: Digital Solutions for Entire Building Lifecycle 7 https://new.siemens.com/global/en/products/buildings/digitalization/digital-building-lifecycle.html
  7. 7. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Connected Car Infrastructure 8 https://www.youtube.com/watch?v=yGLKi3TMJv8
  8. 8. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Twinning the Human Body to Enhance Medical Care 9 https://www.challenge.org/insights/digital-twin-in-healthcare/ https://youtu.be/H6JzPCbyVSM
  9. 9. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Digital Twin and Artificial Intelligence (AI) / Machine Learning • Complementary Concepts • Continuous Learning, Monitoring and Acting • (Good) Data is key for success 10 https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b
  10. 10. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 11
  11. 11. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de History of Automation Industry vs. Big Data and Cloud https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf
  12. 12. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Trends: Evolution of Convergence between IT and Industrial Automation https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
  13. 13. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Complexity, Cost and Scalability are Main Blockers 14
  14. 14. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Huge demand to build an open, flexible, scalable platform • Real time • Scalability • High availability • Decoupling • Cost reduction • Flexibility • Standards-based • Extendibility • Security • Infrastructure-independent • Multi-region / global
  15. 15. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 16
  16. 16. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka - The Rise of an Event Streaming Platform 17 = Messaging + Storage + Integration + Processing
  17. 17. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de P Decoupling of Producers and Consumers Time C2 C3C1 18
  18. 18. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data 19
  19. 19. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in- car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔ Example Case Studies (of many) Confluent - Business Value per Use Case 20
  20. 20. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de 10 Reasons for Event Streaming with Apache Kafka Real Time Scalable Cost Reduction 24/7 – Zero downtime, zero data loss Decoupling – Storage, Domain-driven Design Data (re-)processing and stateful client applications Integration – Connectivity to IoT, legacy, big data, everything Hybrid Architecture – On Premises, multi cloud, edge computing Fully managed cloud No vendor locking 21
  21. 21. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Digital Twin and AI / Machine Learning (with Kafka) • Complementary Concepts • Continuous Learning, Monitoring and Acting à Real time, scalable • (Good) data is key for success à Integration, data processing 22
  22. 22. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Hold on… Kafka is NOT an IoT Platform!
  23. 23. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Device management Unreliable networks Connectivity beyond standards Edge hardware …
  24. 24. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 25
  25. 25. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de 600+ IoT Platforms 26 https://iot-analytics.com/iot-platform-companies-landscape-2020/
  26. 26. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Proprietary IoT Platforms 27
  27. 27. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de IoT Offerings from Cloud Providers 28
  28. 28. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Standards-based / Open Source IoT Platforms 29
  29. 29. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 30
  30. 30. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Characteristics of Digital Twin Technology • Connectivity • Physical assets, enterprise software, customers • Bidirectional communication • Homogenization • Decoupling and standardization • Virtualization of information • Shared with multiple agents • Lower cost • Reprogrammable and smart • Adjust and improve characteristics • Digital traces • Diagnose problems • Modularity • Tweak modules of models and machines 31
  31. 31. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Scenario 1: Digital Twin Monolith 32 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Device Mgt. Analytics Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  32. 32. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Scenario 2: Digital Twin as External Database 33 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin Device Mgt. InfluxDB Analytics Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  33. 33. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Apache Kafka Scenario 3: Kafka as Backbone for the Digital Twin and the Rest of the Enterprise 34 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS IoT Platform Digital Twin InfluxDB Real Time App Batch App Request Response App Kafka Connect Connectivity Homogenization Reprogrammable and smart Digital traces Modularity
  34. 34. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Apache Kafka Scenario 4: Kafka as IoT Platform 35 Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Digital Twin InfluxD B Real Time App Batch App Request Response App Kafka Connect Connectivity Homogenization Reprogrammable and smart Digital traces Modularity Storage Processing
  35. 35. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 36
  36. 36. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Building a Digital Twin with Kafka and InfluxDB Apache Kafka • Integration • Decoupling and Backpressure • Data Processing • Ingest into InfluxDB • Consume from InfluxDB • Consumption by other Applications InfluxDB • Storage • Batch and Real Time Analytics • Dashboards Þ Open Þ Scalable Þ Mission-critical 37 Data Lake Batch Analytics Kafka Streams / ksqlDB Stream Processing Databases Message Queues Sensors Applications
  37. 37. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Edge Digital Twin Single Broker (or Cluster) Digital Twin Self-managed or certified OEM Hardware Kafka Cluster in DC / Cloud Replicator Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  38. 38. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Centralized Digital Twin Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  39. 39. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Global Digital Twin Architecture Multiple Clusters and Aggregation Factories à Analytics Cluster Multi-Region Cluster High Availability (Disaster Recovery) Global Data Streaming Outsourced Development
  40. 40. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Agenda • Digital Twin - Merging the Physical and the Digital World • Real World Challenges • Apache Kafka as Event Streaming Solution for IoT • IoT Platforms • Spoilt for Choice for a Digital Twin • IoT Architectures with Kafka and InfluxDB • A Digital Twin for 100000 Connected Cars 41
  41. 41. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de A Digital Twin with Kafka, TensorFlow and InfluxDB 42 MQTT Proxy InfluxDB Storage InfluxDB Dashboards + Analytics Kafka Cluster Kafka Connect Car Sensors Kafka Ecosystem TensorFlow InfluxDB Other Components Kafka Streams (Java) All Data Critical Data Ingest Data Potential Detect KSQL TensorFlow Train Analytic Model Consume Data Preprocess Data Analytic Model Deploy Analytic Model Python https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference Connectivity Homogenization Reprogrammable and smart Digital traces Modularity Mobile App BI Tool
  42. 42. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Architecture for 100000 Connected Cars Kafka + KSQL + MQTT + TensorFlow + Kubernetes 43 https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/
  43. 43. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Kafka Connect Connector for InfluxDB 44 https://www.confluent.io/hub/confluentinc/kafka-connect-influxdb
  44. 44. IoT and Digital Twin with Apache Kafka and InfluxDB – @KaiWaehner - www.kai-waehner.de Key Takeaways • A Digital Twin merges the physical and the digital world • Apache Kafka + InfluxDB enable an open, scalable and reliable infrastructure for a Digital Twin • Event Streaming complements IoT platforms and other backend applications / databases. + 45
  45. 45. Kai Waehner Technology Evangelist contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!

×