SlideShare a Scribd company logo
1 of 37
IoT Architectures for
Apache Kafka and Event Streaming
Industry 4.0, Digital Twin, PLCs, Edge and Hybrid Deployments
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Poll 1
What are you using Apache Kafka for today?
● Big data ingestion into data lake / analytics cluster for batch processing (Hadoop,
Spark, etc.)
● Streaming pipeline (integration layer, change data capture, streaming ETL)
● Critical business applications (microservice architectures, payment / fraud apps, etc.)
● IoT scenarios (sensor processing / analytics, device management, etc.)
● I don’t use Kafka today
www.kai-waehner.de | @KaiWaehner
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Agenda
• Use Cases
• Event Streaming and Apache Kafka
• IoT Architectures
• Live Demo – Connected Car Infrastructure
• Why Confluent
• Contact Us
Consumer IoT and Industrial IoT (IIoT)
Use Cases
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
5
Event-Driven App
(Location Tracking)
Only Real-Time Events
Messaging Queues can
do this
Contextual
Event-Driven App
(ETA)
Real-Time combined with
stored data
Only Event Streaming
Platforms can do this
Where is my driver? When will my driver
get here?
Where is my driver? When will my driver
get here?
Why add the word
Contextual?
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Front, rear and top
view cameras
Parking assistant
Environment pointer
Ultrasonic Sensors
Parking assistant with front
and rear camera plus
environment indicator
Crash Sensors
Front protection adaptivity
Side protection
Tail impact protection
Infrared Camera
Rearview assistance with
Pedestrian recognition
Front and Rear
Radar Sensors
ACC with stop and go function
Side assist
Front Camera
Audi Active lane assistant
Speed limit indicator
Adaptive light
The Future of the Automotive Industry
is a Real Time Data Cluster
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
The Future of the Automotive Industry
is a Real Time Data Cluster
Front, rear and top
view cameras
Ultrasonic SensorsCrash Sensors
Front Camera Infrared Camera
Front and Rear
Radar Sensors
Traffic Alerts
Hazard Alerts Personalization
Anomaly
Detection
MQTT MQTT
MQTT
MQTT MQTTMQTT
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Real Time Streaming ML at the Edge @ Severstal
www.kai-waehner.de | @KaiWaehner
https://www.confluent.io/customers/severstal/
with Apache Kafka
Event
Streaming
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
14
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
TRADITIONAL
DATABASE
EVENT STREAM
PROCESSING
SELECT * FROM
DB_TABLE
CREATE TABLE T
AS SELECT * FROM
EVENT_STREAM
Active Query: Passive Data:
DB Table
Active Data: Passive Query:
Event Stream
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
TABLES STREAMS
USER
JAY
SUE
FRED
CREDIT_SCORE
695
430
710V1
V3
V2
PAYMENTS42
18
65
...
USER
JAY
SUE
FRED
...
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
PUSH PULL
APP
Jay’s credit score is 670
Jay’s credit score is 710
Jay’s credit score is 695
What is Jay’s credit score now?
695
APP
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka - The Rise of an Event Streaming Platform
18
=
Messaging
+
Storage
+
Integration
+
Processing
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Apache Kafka at Scale at Tech Giants
> 7 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka is not just used for big data
** Kafka Is not just used by tech giants
19
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
10 Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
20
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Device management
Unreliable networks
Connectivity beyond standards
Edge hardware
…
is not an IoT Platform!
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Poll 2
How do you use or plan to use Kafka and its ecosystem?
● Build a core IoT even streaming platform (using Kafka Connect + connectors for MQTT
/ Syslog / PLC4X / etc.)
● Combine Kafka with other open source components (like Eclipse IoT, HiveMQ, …)
● Combine Kafka with commercial platforms (like Siemens MindSphere or GE Predix)
● Use Kafka just as messaging pipeline and not for IoT-related tasks
● I don’t use or plan to use Kafka at all for my IoT projects
www.kai-waehner.de | @KaiWaehner
Edge, Hybrid, Global
IoT
Architectures
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Digital Service at the Edge
Single
Kafka Broker
(or Cluster)
Digital
Service
Self-managed or
certified OEM Hardware
Kafka
Cluster
in DC /
Cloud
Replicator
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Centralized Digital Service
Digital
Service
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
Single
Kafka Broker
(or Cluster)
Self-managed or
certified OEM Hardware
Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Global IoT Architecture
Multiple Clusters and Aggregation
Factories à Analytics Cluster
Multi-Region Cluster
High Availability (Disaster Recovery)
Global Data Streaming
Outsourced
Development
IoT Infrastructure for 100000 Connected Cars
Example
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
IoT Infrastructure with
Kafka, TensorFlow and MongoDB
28
MQTT
Proxy
Kafka
Cluster
Kafka
Connect
Car Sensors
Kafka Ecosystem
TensorFlow
Other Components
Kafka
Streams
(Java)
All
Data
Critical
Data
Ingest
Data
Potential
Detect
KSQL
TensorFlow
Train
Analytic
Model
Consume
Data
Preprocess
Data
Analytic
Model
Deploy
Analytic
Model
Python
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference
Connectivity
Homogenization
Reprogrammable and smart
Digital traces
Modularity
MongoDB
Storage
MongoDB
Dashboards
Search
Analytics
Mobile App
BI Tool
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Architecture for 100000 Connected Cars
Kafka + KSQL + MQTT + TensorFlow + Kubernetes
29
https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/
Live Demo:
Connected Car Infrastructure with
Kafka, MQTT and TensorFlow
Why Confluent
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
32Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance & Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete Engagement Model
Revenue / Cost / Risk Impact
TCO / ROI
Executive Buyer
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
33
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
33
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
34C O N F I D E N T I A L
Future-proof event streaming
Kafka re-engineered as a fully-managed, cloud-native service by its original creators and
the team that wrote 80% of all Kafka commits
Global
Automated disaster
recovery
Global applications with
geo-awareness
Infinite
Efficient and infinite data
with tiered storage
Unlimited horizontal
scalability for clusters
Elastic
Easy multi-cloud
orchestration
Persistent bridge to
cloud from on-prem
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
Poll 3
What architectures do make most sense for your Kafka deployment?
● Deployment at the edge (factory, plant, data center) for local integration and data processing
● Replication between edge and cloud / other data center
● Global architecture (e.g. aggregation and analytics of data from different factories)
● Single Kafka Broker, e.g. at small sites (no data center, no high availability)
● Digital Twin for real time analytics of sensor data
www.kai-waehner.de | @KaiWaehner
IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
36
• Schedule a free one to one virtual session with a Confluent
Systems Engineer.
• Contact Robin North at rnorth@confluent.io to get started!
• Visit the Confluent IoT Website
https://www.confluent.io/solutions/internet-of-things-iot/
Next Steps
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®confluent
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptxAlex Ivy
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Araf Karsh Hamid
 
Microservices Architecture & Testing Strategies
Microservices Architecture & Testing StrategiesMicroservices Architecture & Testing Strategies
Microservices Architecture & Testing StrategiesAraf Karsh Hamid
 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPDatabricks
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...GetInData
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)Kai Wähner
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingAraf Karsh Hamid
 
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...Kai Wähner
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
Apache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningApache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningKai Wähner
 
Running Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsRunning Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann
 
Containers Docker Kind Kubernetes Istio
Containers Docker Kind Kubernetes IstioContainers Docker Kind Kubernetes Istio
Containers Docker Kind Kubernetes IstioAraf Karsh Hamid
 
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPBridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPconfluent
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks DeltaDatabricks
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafkaconfluent
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafkaconfluent
 

What's hot (20)

The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Flink vs. Spark
Flink vs. SparkFlink vs. Spark
Flink vs. Spark
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
Microservices Architecture & Testing Strategies
Microservices Architecture & Testing StrategiesMicroservices Architecture & Testing Strategies
Microservices Architecture & Testing Strategies
 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCP
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)Kafka Connect and Streams (Concepts, Architecture, Features)
Kafka Connect and Streams (Concepts, Architecture, Features)
 
Big Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb ShardingBig Data Redis Mongodb Dynamodb Sharding
Big Data Redis Mongodb Dynamodb Sharding
 
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
Apache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep LearningApache Kafka Streams + Machine Learning / Deep Learning
Apache Kafka Streams + Machine Learning / Deep Learning
 
Running Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsRunning Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration Options
 
Containers Docker Kind Kubernetes Istio
Containers Docker Kind Kubernetes IstioContainers Docker Kind Kubernetes Istio
Containers Docker Kind Kubernetes Istio
 
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCPBridge to Cloud: Using Apache Kafka to Migrate to GCP
Bridge to Cloud: Using Apache Kafka to Migrate to GCP
 
Introducing Databricks Delta
Introducing Databricks DeltaIntroducing Databricks Delta
Introducing Databricks Delta
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafka
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafka
 

Similar to IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digital Twin, PLCs, Edge and Hybrid Deployments

Apache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT WorldApache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT Worldconfluent
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertconfluent
 
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and ConfluentEvent Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and Confluentconfluent
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridPaolo Castagna
 
Set Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO RoundtableSet Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO Roundtableconfluent
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionKai Wähner
 
How to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent IIHow to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent IIconfluent
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...InfluxData
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming VisualizationGuido Schmutz
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...HostedbyConfluent
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...confluent
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingTimothy Spann
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemGuido Schmutz
 
StreamAnalytix - Multi-Engine Streaming Analytics Platform
StreamAnalytix - Multi-Engine Streaming Analytics PlatformStreamAnalytix - Multi-Engine Streaming Analytics Platform
StreamAnalytix - Multi-Engine Streaming Analytics PlatformAtul Sharma
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...Kai Wähner
 

Similar to IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digital Twin, PLCs, Edge and Hybrid Deployments (20)

Apache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT WorldApache Kafka® and Analytics in a Connected IoT World
Apache Kafka® and Analytics in a Connected IoT World
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniertFast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
 
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and ConfluentEvent Streaming in the Telco Industry with Apache Kafka® and Confluent
Event Streaming in the Telco Industry with Apache Kafka® and Confluent
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - Madrid
 
Set Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO RoundtableSet Your Data In Motion - CTO Roundtable
Set Your Data In Motion - CTO Roundtable
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy Production
 
How to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent IIHow to Build Streaming Apps with Confluent II
How to Build Streaming Apps with Confluent II
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...
Kai Waehner [Confluent] | Real-Time Streaming Analytics with 100,000 Cars Usi...
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and Streaming
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka Ecosystem
 
StreamAnalytix - Multi-Engine Streaming Analytics Platform
StreamAnalytix - Multi-Engine Streaming Analytics PlatformStreamAnalytix - Multi-Engine Streaming Analytics Platform
StreamAnalytix - Multi-Engine Streaming Analytics Platform
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...
Apache Kafka, Tiered Storage and TensorFlow for Streaming Machine Learning wi...
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryKai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaKai Wähner
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityKai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance Industry
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
 
Apache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart CityApache Kafka for Automotive Industry, Mobility Services & Smart City
Apache Kafka for Automotive Industry, Mobility Services & Smart City
 

Recently uploaded

User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeKaylee Miller
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevpmgdscunsri
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...MyFAA
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.Ritesh Kanjee
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptxAGATSoftware
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJpolinaucc
 
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...Maxim Salnikov
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Inc
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxBarakaMuyengi
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurPriyadarshini T
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfCloudMetic
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckNaval Singh
 
Enterprise Content Managements Solutions
Enterprise Content Managements SolutionsEnterprise Content Managements Solutions
Enterprise Content Managements SolutionsIQBG inc
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...jackiepotts6
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 

Recently uploaded (20)

User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller Resume
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data Mesh
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJ
 
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...
If your code could speak, what would it tell you? Let GitHub Copilot Chat hel...
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
 
20140812 - OBD2 Solution
20140812 - OBD2 Solution20140812 - OBD2 Solution
20140812 - OBD2 Solution
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deck
 
Enterprise Content Managements Solutions
Enterprise Content Managements SolutionsEnterprise Content Managements Solutions
Enterprise Content Managements Solutions
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 

IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digital Twin, PLCs, Edge and Hybrid Deployments

  • 1. IoT Architectures for Apache Kafka and Event Streaming Industry 4.0, Digital Twin, PLCs, Edge and Hybrid Deployments Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  • 2. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Poll 1 What are you using Apache Kafka for today? ● Big data ingestion into data lake / analytics cluster for batch processing (Hadoop, Spark, etc.) ● Streaming pipeline (integration layer, change data capture, streaming ETL) ● Critical business applications (microservice architectures, payment / fraud apps, etc.) ● IoT scenarios (sensor processing / analytics, device management, etc.) ● I don’t use Kafka today www.kai-waehner.de | @KaiWaehner
  • 3. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Agenda • Use Cases • Event Streaming and Apache Kafka • IoT Architectures • Live Demo – Connected Car Infrastructure • Why Confluent • Contact Us
  • 4. Consumer IoT and Industrial IoT (IIoT) Use Cases
  • 5. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 5 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why add the word Contextual?
  • 6. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Front, rear and top view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist Front Camera Audi Active lane assistant Speed limit indicator Adaptive light The Future of the Automotive Industry is a Real Time Data Cluster
  • 7. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Ultrasonic SensorsCrash Sensors Front Camera Infrared Camera Front and Rear Radar Sensors Traffic Alerts Hazard Alerts Personalization Anomaly Detection MQTT MQTT MQTT MQTT MQTTMQTT
  • 8. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
  • 9. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
  • 10. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
  • 11. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de
  • 12. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Real Time Streaming ML at the Edge @ Severstal www.kai-waehner.de | @KaiWaehner https://www.confluent.io/customers/severstal/
  • 14. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 14 STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC
  • 15. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de TRADITIONAL DATABASE EVENT STREAM PROCESSING SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream
  • 16. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de TABLES STREAMS USER JAY SUE FRED CREDIT_SCORE 695 430 710V1 V3 V2 PAYMENTS42 18 65 ... USER JAY SUE FRED ...
  • 17. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de PUSH PULL APP Jay’s credit score is 670 Jay’s credit score is 710 Jay’s credit score is 695 What is Jay’s credit score now? 695 APP
  • 18. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka - The Rise of an Event Streaming Platform 18 = Messaging + Storage + Integration + Processing
  • 19. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “You name it” * Kafka is not just used for big data ** Kafka Is not just used by tech giants 19
  • 20. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 10 Reasons for Event Streaming with Apache Kafka Real Time Scalable Cost Reduction 24/7 – Zero downtime, zero data loss Decoupling – Storage, Domain-driven Design Data (re-)processing and stateful client applications Integration – Connectivity to IoT, legacy, big data, everything Hybrid Architecture – On Premises, multi cloud, edge computing Fully managed cloud No vendor locking 20
  • 21. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Device management Unreliable networks Connectivity beyond standards Edge hardware … is not an IoT Platform!
  • 22. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Poll 2 How do you use or plan to use Kafka and its ecosystem? ● Build a core IoT even streaming platform (using Kafka Connect + connectors for MQTT / Syslog / PLC4X / etc.) ● Combine Kafka with other open source components (like Eclipse IoT, HiveMQ, …) ● Combine Kafka with commercial platforms (like Siemens MindSphere or GE Predix) ● Use Kafka just as messaging pipeline and not for IoT-related tasks ● I don’t use or plan to use Kafka at all for my IoT projects www.kai-waehner.de | @KaiWaehner
  • 24. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Digital Service at the Edge Single Kafka Broker (or Cluster) Digital Service Self-managed or certified OEM Hardware Kafka Cluster in DC / Cloud Replicator Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  • 25. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Centralized Digital Service Digital Service Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS Single Kafka Broker (or Cluster) Self-managed or certified OEM Hardware Siemens S7, Modbus, Allen Bradley, Beckhoff ADS
  • 26. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Global IoT Architecture Multiple Clusters and Aggregation Factories à Analytics Cluster Multi-Region Cluster High Availability (Disaster Recovery) Global Data Streaming Outsourced Development
  • 27. IoT Infrastructure for 100000 Connected Cars Example
  • 28. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de IoT Infrastructure with Kafka, TensorFlow and MongoDB 28 MQTT Proxy Kafka Cluster Kafka Connect Car Sensors Kafka Ecosystem TensorFlow Other Components Kafka Streams (Java) All Data Critical Data Ingest Data Potential Detect KSQL TensorFlow Train Analytic Model Consume Data Preprocess Data Analytic Model Deploy Analytic Model Python https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference Connectivity Homogenization Reprogrammable and smart Digital traces Modularity MongoDB Storage MongoDB Dashboards Search Analytics Mobile App BI Tool
  • 29. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Architecture for 100000 Connected Cars Kafka + KSQL + MQTT + TensorFlow + Kubernetes 29 https://www.kai-waehner.de/blog/2019/11/08/live-demo-iot-100-000-connected-cars-kubernetes-kafka-mqtt-tensorflow/
  • 30. Live Demo: Connected Car Infrastructure with Kafka, MQTT and TensorFlow
  • 32. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 32Confluent Platform Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer
  • 33. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 33 I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model 33 Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi- Region) Build Contextual Event- Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...
  • 34. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 34C O N F I D E N T I A L Future-proof event streaming Kafka re-engineered as a fully-managed, cloud-native service by its original creators and the team that wrote 80% of all Kafka commits Global Automated disaster recovery Global applications with geo-awareness Infinite Efficient and infinite data with tiered storage Unlimited horizontal scalability for clusters Elastic Easy multi-cloud orchestration Persistent bridge to cloud from on-prem
  • 35. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de Poll 3 What architectures do make most sense for your Kafka deployment? ● Deployment at the edge (factory, plant, data center) for local integration and data processing ● Replication between edge and cloud / other data center ● Global architecture (e.g. aggregation and analytics of data from different factories) ● Single Kafka Broker, e.g. at small sites (no data center, no high availability) ● Digital Twin for real time analytics of sensor data www.kai-waehner.de | @KaiWaehner
  • 36. IoT and Event Streaming – @KaiWaehner - www.kai-waehner.de 36 • Schedule a free one to one virtual session with a Confluent Systems Engineer. • Contact Robin North at rnorth@confluent.io to get started! • Visit the Confluent IoT Website https://www.confluent.io/solutions/internet-of-things-iot/ Next Steps