SlideShare a Scribd company logo
1 of 56
1Confidential
Flexible and Scalable Integration in
Automation Industry / Industrial IoT
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Kafka-Native End-to-End IIoT Data Integration and Processing
with Kafka Connect, KSQL and Apache PLC4X
2
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
3
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
4
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
Mobile Cloud Microservices Internet of Things Machine Learning
The world has changed!
5
Industry 4.0 / Industrial IoT (IIoT)
6
Some IIoT use cases
Analytics
• Ingest data into cloud for analytics
• Reduce cost: Leverage open frameworks instead of paying very expensive licenses per machine
• Flexible integration (select data to ingest, flexible changes over time)
• Machine Learning / Data Science
Manufacturing
• Collect data from machines à Preprocess + monitoring to optimize assembly line and reduce cost
• Aggregate data from different machines / companies —> Leverage (and sell?) insights
• Sell services on top of machines —> Predictive maintenance (remote)
• Scale up (add more sites, add more data)
Production Robots
• Ingest, process and monitor large volumes data (where the proprietary monolith does not scale)
Smart Factories
• Monitor and manage the whole factory (at scale, in real time, flexible)
• Integration with legacy proprietary protocols and modern cloud-native technologies
7
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
8
History of Automation Industry vs. Big Data and Cloud
Christofer Dutz (codecentric)
https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf
9
Challenges in Automation Industry
IoT != IIoT
• IoT = Connected cars, smart home, … à Large scale, secure, scalable, open,
modern technologies
• IIoT = Slow, insecure, not scalable, proprietary
Legacy / Proprietary IIoT Technologies
• Usually incompatible protocols, typically proprietary
• Usually serial connections (very low latency, nanoseconds) - with TCP / UDP
wrapper around it to integrate with “external world”
• Siemens S7, Modbus, Beckhoff, Profinet, Allen Bradley, etc.
• OPC-UA (required machine update + license cost)
Product Lifecycles
• Long lifecycle (tens of years)
• Factories cost millions, no simple changes / upgrades
• Still using Windows 7 without Service Packs => Usability and security issues
• Mantra: “Stay with your well known vendor forever”
10
Challenges in Automation Industry
Monoliths
• No scalability
• No extendibility
• No real failover (start your backup machine)
Missing Security Capabilities
• Security in software development == Authentication,
Authorization, Antivirus, SSL, SASL, Kerberos
• Security in automation industry == Safety
• “if you press the red button, the machine stops
immediately”
• Insecure by nature => No Authentication /
Authorization / Encryption
• Mantra: “Our factory building and network is secure,
no access from outside”
• Contradicts with “move to cloud and big data
analytics”
11
PLC (Programmable Logic Controller)
• Started early 70’s
• Control of manufacturing processes
• Small grey box
• ~100 messages per second, stored to CSV file, Windows Share
• Limited operations: Read (90+%), Write, Subscribe, Call
Functions, List Resources
• High reliability control, ease of programming and process
fault diagnosis
• Hardwire à softwire
• Has Input / Sensors, Output / Actors
• Firmware (= operating system)
• Mechanism to load user programs
• Highly fragmented market
• S7 (Siemens), Beckhoff ADS, Modbus (Asia), Ethernet/IP, KNX,
Emerson DeltaV, Profinet, Allen Bradley, etc.
• State of the art in automation industry
12
Example: Siemens S7 Communication
When communicating with S7 Devices
there is a whole family of protocols,
that can be used.
In general you can divide them
into Profinet protocols and S7
Comm protocols. The later are far
simpler in structure, but also far less
documented.
The S7 Comm protocols are generally
split up into two flavors: The
classic S7 Comm and a newer version
unofficially called S7 Comm Plus.
https://plc4x.apache.org/protocols/s7/index.html
13
Trends: ~50% of industrial assets in factories will be connected by 2020
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
14
Trends: Evolution of Convergence between IT and Industrial Automation
https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
15
How to get from legacy, proprietary to cloud, big data, machine learning?
16
Costly and inflexible legacy Integration between IIoT and other Systems
ModbusS7
Siemens
Integration
Middleware
Monolith
Schneider Electric
Integration
Middleware
Monolith
Integration
Middleware
17
Huge demand to build an open, flexible, scalable platform
• Cost reduction
• Flexibility
• Standards-based
• Scalability
• Extendibility
• Security
• Infrastructure-independent
18
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
19
?
IIoT Architecture (High Level)
Kafka BrokerKafka BrokerStreaming
Platform
Connect
w/ MQTT
connector
GatewayDevicesDevicesDevicesMachine
Sensor Analytics
(Real Time)
Predictive
Maintenance
(Near Real Time)
Machine Learning
(Batch)
Edge Data Center / Cloud
How to integrate and process data at scale and reliable?
20
Vendor-Neutral IoT Architectures across Edge, On Premise and Multi-Cloud
On-Premise / Edge
Deploy on bare-metal, VMs,
containers or Kubernetes in your
datacenter with Confluent Platform
and Confluent Operator
Public Cloud
Implement self-managed in the public
cloud or adopt a fully managed service
with Confluent Cloud
Hybrid Cloud
Build a persistent bridge between
datacenter and cloud with
Confluent Replicator
Confluent
Replicator
VM
SELF MANAGED FULLY MANAGED
Data Lake
Batch
Analytics
Event
Streaming
Platform
Batch
Integration
Real Time Pre-
processing
Machine Sensors
Streaming Platform
Other Components
Real Time
Processing
(6b) All Data
(7) Potential Defect
(3)
Read Data
Optimization
/ Analytics
(5)
Deploy
Optimization
Model
(8b) Alert Person (e.g. Mobile App)
(2)
Preprocess
Data (6a) Consume machine data
Model
Standard
based
Integration
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
Model Lite
Real Time App
Model Server
RPC
PLC Proprietary
based
Integration
Standard
Interface
Proprietary
Interface
(9) Manual user-based analytics
and reporting to find insights
and improve real time process
22
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
23
The beginning of a new Era
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
The first use case. This is why Kafka was created!
24
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka—The Rise of an Event Streaming Platform
25
● Global-scale
● Real-time
● Persistent Storage
● Stream Processing
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache Kafka
Apache Kafka: The De-facto Standard for Real-Time Event Streaming
26
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
27
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
28
Apache Kafka - A Distributed Commit Log
Writers
Kafka
cluster
Readers
29
Kafka Topics
my-topic
my-topic-partition-0
my-topic-partition-1
my-topic-partition-2
broker-1
broker-2
broker-3
30
P
Decoupled Producers and Consumers
Time
C2 C3C1
31
Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
32
Confluent Schema Registry
33
Kafka Streams
Your
app
sinksource
KafkaConnect
KafkaConnect
Kafka Cluster
Apache Kafka includes Kafka Connect and Kafka Streams
34
Kafka Streams
● No separate processing cluster required
● Develop on Mac, Linux, Windows
● Deploy to containers, VMs, bare metal, cloud
● Powered by Kafka: elastic, scalable, distributed,
battle-tested
● Perfect for small, medium, large use cases
● Fully integrated with Kafka security
● Exactly-once processing semantics
● Part of Apache Kafka
KStream<User, PageViewEvent> pageViews = builder.stream("pageviews-topic");
KTable<Windowed<User>, Long> viewsPerUserSession = pageViews
.groupByKey()
.count(SessionWindows.with(TimeUnit.MINUTES.toMillis(5)), "session-views");
https://docs.confluent.io/current/streams/
Write standard Java apps and microservices
to process your data in real-time
35
KSQL: Enable Stream Processing using SQL-like Semantics
Leverage Kafka Streams API
using simple SQL commands
KSQL server
Engine
(runs queries)
REST API
CLIClients
Confluent
Control Center
GUI
Kafka Cluster
Use any programming language
Connect via Control Center UI,
CLI, REST or deploy in headless
mode
36
streams
The streaming SQL engine for Apache Kafka
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;
Apache Kafka library to write
real-time applications and
microservices in Java and Scala
confluent.io/product/ksql
Confluent KSQL
You write only SQL. No Java, Python, or
other boilerplate to wrap around it!
Event Transformation with Stream Processing
37
Kafka Connect
● Centralized management and configuration
● Support for hundreds of technologies including
RDBMS, Elasticsearch, HDFS, S3
● Supports CDC ingest of events from RDBMS
● Preserves data schema
● Fault tolerant and automatically load balanced
● Extensible API
● Single Message Transforms
● Part of Apache Kafka
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://localhost:3306/demo?user=rmoff&password=foo",
"table.whitelist": "sales,orders,customers"
}
https://docs.confluent.io/current/connect/
Reliable and scalable integration of Kafka with other systems
38
Connect External Data Sources and Sinks with Connectors
SOURCES SINKS
CDC
Connectors developed and supported by Confluent, partners and the open source community available on
confluent.io/hub
39
IoT Integration with Kafka Connect, MQTT and REST Proxy
Video and Slides:
https://www.confluent.io/kafka-summit-sf18/processing-iot-data-from-end-to-end
40
Native, decoupled Integration between IIoT and other Systems
ModbusSiemens
S7
Siemens
S7
Siemens
S7
Modbus Modbus Modbus
Kafka Connect Kafka Connect
Siemens
S7
?
41
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
42
Apache PLC4X
• Top Level Apache project
• PLC 4 (for) X (anything)
• Goal: Open up PLC interfaces to outside world
• Vertical integration
• Write software independent of PLC
• JDBC-like Adapters for various protocols
https://plc4x.apache.org/
43
Code Example – Connection to Siemens S7 PLC
Feels like
JDBC
44
Native, decoupled Integration between IIoT and other Systems
ModbusSiemens
S7
Siemens
S7
Siemens
S7
Modbus Modbus ModbusSiemens
S7
Kafka Connect
45
One more thing à PLC4X vs. OPC-UA
• Open standard
• All the pros and cons of an open standard
(works with different vendors; slow adoption;
inflexible, etc.)
• Often poorly implemented
• Requires app server on top of PLC
• Every device has to be retrofitted with the
ability to speak a new protocol and use a
common client to speak with these devices
• Often overengineering for just reading the data
• Activating OPC-UA support on existing PLCs
greatly increases the load on the PLCs
• With licensing cost for every machine
• Open source framework (Apache 2.0 license)
• Provides unified API by implementing drivers
for communicating with most industrial
controllers in the protocols they natively
understand
• No need to modify existing hardware
• No increased load on the PLCs
• No need to pay for licenses to activate OPC-UA
support
• Drivers being implemented from the specs or
by reverse engineering protocols in order to be
fully Apache 2.0 licensed
• PLC4X adapter for OPC-UA available -> Both
can be used together!
46
Agenda
1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning
2) Automation Industry and its Challenges
3) Architecture for End-to-End Integration from Edge to Data Center / Cloud
4) Apache Kafka as Event Streaming Platform
5) Apache PLC4X for Edge Integration
6) Example: Supply Chain Optimization at Scale in Real Time
Spark
Notebooks
(Jupyter)
Kafka
Cluster
Kafka
Connect
KSQL
Machine Sensors
Kafka Ecosystem
Other Components Real Time
Kafka Streams
Application
(Java / Scala)
(6b) All Data
(7) Potential Defect
(3)
Read Data
TensorFlow I/O
TensorFlow
(5)
Deploy Model
(2)
Preprocess
Data (6a) Consume machine data
TensorFlow
File
HTTP
MQTT
ROS
(8a)
Stop Machine
(1)
Ingest Data
Real Time Edge
Computing
(C / librdkafka)
TensorFlow Lite
Real Time Kafka
App
TensorFlow
Serving
HTTP /
gRPC
(4)
Train Model
PLC
Beckhoff
S7
Modbus
Allen Bradley
OPC-UA
PLC4X
Connector
Kafka Connect
Standard
Interface
Proprietary
Interface
(8b) Alert Person (e.g. Mobile App)
(9) Manual user-based analytics
and reporting to find insights
and improve real time process
Example Project:
Supply Chain Optimization
in Real Time at Scale
Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
Streaming analytics using Confluent
Batch analytics using other frameworks
Physical operations
UI UI UIUI
(Reference use case implemented with our partner Expero)
Planners
forecast long
term schedule
Production
begins
IOT data from
production:
inventories,
manufacturing
machines,
yield metrics
Production
forecast
Forecasted
production -
plan diffs
Re optimize
plan based on
actuals
Change orders
to supply
chain:
inventory,
manufacturing
schedules
Change
operational
characteristics
: plant 223
needs new Al
extruder
Customer
delivery SLAs:
actuals vs.
plan
UI UI UIUI
Kafka
Connect
+
PLC4X
Connector
Machine
Sensors
Kafka
Cluster
KSQL
Tensor
Flow
Kafka
Connect
Notebooks
(Jupyter)
Spark
Real
Time
Kafka
App
Streaming analytics using Confluent
Batch analytics using other frameworks
Physical operations
TensorFlow
Serving
(Reference use case implemented with our partner Expero)
51
Supply Chain Optimization in Real Time at Scale
Slides and Video Recording:
http://www.kai-waehner.de/blog/2019/08/23/apache-kafka-machine-learning-for-real-time-supply-chain-iiot-opcua-modbus/
Why
for IIoT Projects?
53
Confluent Platform
The Event Streaming Platform Built by the Original Creators of Apache Kafka®
Operations and Security
Development & Stream Processing
Apache Kafka
Confluent Platform
Support,Services,
Training,&Partners
Mission-Critical Reliability
Complete Event
Streaming Platform
Freedom of Choice
Datacenter Public Cloud Confluent Cloud
Self-Managed Software Fully Managed Service
56
Confluent Platform – Benefits for IoT Projects
• Based on open source and de facto standards for IoT projects
• Low license / subscription costs for Confluent support / services / training (compared to traditional IoT vendors + their products)
• Spend budget for consulting to realize the project successfully
• Mission critical deployments at large scale in various industries
• Automotive, Manufacturing, Logistics, Oil&Gas, Retail, Telco, …
• Flexible architecture
• Lightweight infrastructure footprint on commodity hardware
• Pick what you need
• Deploy where you want
• Complementary to other frameworks, technologies (e.g. Siemens MindSphere, Cisco Kinetic) and cloud services (e.g. Google Cloud IoT)
• Customize and build for the specific customer use case
• Battle-tested at large scale
• Event Streaming Platform for real time integration and processing (plus integration to batch, file and other communication protocols)
• Security and reliability as core concepts
• Elastic scalability, start small and grow to extreme scale easily
• Partner (open source) technologies for specific integrations (like HiveMQ or PLC4X)
• Integration with any legacy and modern technology
• IoT standards like MQTT or OPC-UA
• Legacy and proprietary IIoT protocols like Modbus, Siemens S7, Beckhoff, Allen Bradley, etc.
• Modern technologies like S3, HDFS, MongoDB, etc.
• Modern applications (business services like Salesforce and IoT solutions like Siemens MindSphere)
57
Confluent and IoT Platform Solutions
Kafka
Cluster
Siemens
MindSphere
KSQL
Machine Sensors
File
HTTP
MQTT
ROS
PLC
Beckhoff
S7
Modbus
OPC-UA
“you-name-it”
PLC4X
Connector
Kafka Connect
Azure
IoT Hub
Framework or solution?
Or both as complementary technologies?
S7 PLC
58
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
 
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021StreamNative
 
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafkaReal time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafkaTimothy Spann
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...Lucas Jellema
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!Guido Schmutz
 
Introduction to Apache Kafka and Confluent... and why they matter
Introduction to Apache Kafka and Confluent... and why they matterIntroduction to Apache Kafka and Confluent... and why they matter
Introduction to Apache Kafka and Confluent... and why they matterconfluent
 
Databricks for Dummies
Databricks for DummiesDatabricks for Dummies
Databricks for DummiesRodney Joyce
 
Building Robust ETL Pipelines with Apache Spark
Building Robust ETL Pipelines with Apache SparkBuilding Robust ETL Pipelines with Apache Spark
Building Robust ETL Pipelines with Apache SparkDatabricks
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using KafkaKnoldus Inc.
 
Apache Kafka Introduction
Apache Kafka IntroductionApache Kafka Introduction
Apache Kafka IntroductionAmita Mirajkar
 
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 20190-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019confluent
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaKai Wähner
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaBest Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardParis Data Engineers !
 

What's hot (20)

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021
 
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafkaReal time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Introduction to Apache Kafka and Confluent... and why they matter
Introduction to Apache Kafka and Confluent... and why they matterIntroduction to Apache Kafka and Confluent... and why they matter
Introduction to Apache Kafka and Confluent... and why they matter
 
Databricks for Dummies
Databricks for DummiesDatabricks for Dummies
Databricks for Dummies
 
Building Robust ETL Pipelines with Apache Spark
Building Robust ETL Pipelines with Apache SparkBuilding Robust ETL Pipelines with Apache Spark
Building Robust ETL Pipelines with Apache Spark
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Apache Kafka Introduction
Apache Kafka IntroductionApache Kafka Introduction
Apache Kafka Introduction
 
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 20190-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
0-60: Tesla's Streaming Data Platform ( Jesse Yates, Tesla) Kafka Summit SF 2019
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Mainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache KafkaMainframe Integration, Offloading and Replacement with Apache Kafka
Mainframe Integration, Offloading and Replacement with Apache Kafka
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache KafkaBest Practices for Streaming IoT Data with MQTT and Apache Kafka
Best Practices for Streaming IoT Data with MQTT and Apache Kafka
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin AmbardDelta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
 

Similar to IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

Activeeon technology for Big Compute and cloud migration
Activeeon technology for Big Compute and cloud migrationActiveeon technology for Big Compute and cloud migration
Activeeon technology for Big Compute and cloud migrationActiveeon
 
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQBuilding a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQDominik Obermaier
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner
 
Apache kafka event_streaming___kai_waehner
Apache kafka event_streaming___kai_waehnerApache kafka event_streaming___kai_waehner
Apache kafka event_streaming___kai_waehnerconfluent
 
Red hat's updates on the cloud & infrastructure strategy
Red hat's updates on the cloud & infrastructure strategyRed hat's updates on the cloud & infrastructure strategy
Red hat's updates on the cloud & infrastructure strategyOrgad Kimchi
 
Processing Real-Time Data at Scale: A streaming platform as a central nervous...
Processing Real-Time Data at Scale: A streaming platform as a central nervous...Processing Real-Time Data at Scale: A streaming platform as a central nervous...
Processing Real-Time Data at Scale: A streaming platform as a central nervous...confluent
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of dataconfluent
 
Real-Time Analytics with Confluent and MemSQL
Real-Time Analytics with Confluent and MemSQLReal-Time Analytics with Confluent and MemSQL
Real-Time Analytics with Confluent and MemSQLSingleStore
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
 
GARE du MIDIH MIDIH, towards a flexible, modular and open source reference ...
GARE du MIDIH   MIDIH, towards a flexible, modular and open source reference ...GARE du MIDIH   MIDIH, towards a flexible, modular and open source reference ...
GARE du MIDIH MIDIH, towards a flexible, modular and open source reference ...MIDIH_EU
 
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...Rockwell Automation
 
Partner Keynote: Intel - The New Frontier of Cloud Computing
Partner Keynote: Intel - The New Frontier of Cloud ComputingPartner Keynote: Intel - The New Frontier of Cloud Computing
Partner Keynote: Intel - The New Frontier of Cloud ComputingAmazon Web Services
 
Latest trendsincloud computing
Latest trendsincloud computingLatest trendsincloud computing
Latest trendsincloud computingLiliana Ignat
 
InduSoft IoTView
InduSoft IoTViewInduSoft IoTView
InduSoft IoTViewAVEVA
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®confluent
 
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...confluent
 

Similar to IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X (20)

Activeeon technology for Big Compute and cloud migration
Activeeon technology for Big Compute and cloud migrationActiveeon technology for Big Compute and cloud migration
Activeeon technology for Big Compute and cloud migration
 
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQBuilding a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
 
Apache kafka event_streaming___kai_waehner
Apache kafka event_streaming___kai_waehnerApache kafka event_streaming___kai_waehner
Apache kafka event_streaming___kai_waehner
 
inmation Presentation_2017
inmation Presentation_2017inmation Presentation_2017
inmation Presentation_2017
 
Red hat's updates on the cloud & infrastructure strategy
Red hat's updates on the cloud & infrastructure strategyRed hat's updates on the cloud & infrastructure strategy
Red hat's updates on the cloud & infrastructure strategy
 
Processing Real-Time Data at Scale: A streaming platform as a central nervous...
Processing Real-Time Data at Scale: A streaming platform as a central nervous...Processing Real-Time Data at Scale: A streaming platform as a central nervous...
Processing Real-Time Data at Scale: A streaming platform as a central nervous...
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of data
 
Io t data streaming
Io t data streamingIo t data streaming
Io t data streaming
 
Real-Time Analytics with Confluent and MemSQL
Real-Time Analytics with Confluent and MemSQLReal-Time Analytics with Confluent and MemSQL
Real-Time Analytics with Confluent and MemSQL
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
GARE du MIDIH MIDIH, towards a flexible, modular and open source reference ...
GARE du MIDIH   MIDIH, towards a flexible, modular and open source reference ...GARE du MIDIH   MIDIH, towards a flexible, modular and open source reference ...
GARE du MIDIH MIDIH, towards a flexible, modular and open source reference ...
 
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...
RA TechED 2019 - SY08 - Developing Information Ready Applications using Smart...
 
Partner Keynote: Intel - The New Frontier of Cloud Computing
Partner Keynote: Intel - The New Frontier of Cloud ComputingPartner Keynote: Intel - The New Frontier of Cloud Computing
Partner Keynote: Intel - The New Frontier of Cloud Computing
 
Latest trendsincloud computing
Latest trendsincloud computingLatest trendsincloud computing
Latest trendsincloud computing
 
InduSoft IoTView
InduSoft IoTViewInduSoft IoTView
InduSoft IoTView
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
 
Apresentação Webinar industria 4.0
Apresentação Webinar industria 4.0   Apresentação Webinar industria 4.0
Apresentação Webinar industria 4.0
 
Digital transformation and AI @Edge
Digital transformation and AI @EdgeDigital transformation and AI @Edge
Digital transformation and AI @Edge
 
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...
Viele Autos, noch mehr Daten: IoT-Daten-Streaming mit MQTT & Kafka (Kai Waehn...
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 

Recently uploaded

Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024SimonedeGijt
 
Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...Varun Mithran
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxNeo4j
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfWSO2
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Conceptsthomashtkim
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmuxevmux96
 
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphGraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphNeo4j
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMarkus Moeller
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Henry Schreiner
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 

Recently uploaded (20)

Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...Incident handling is a clearly defined set of procedures to manage and respon...
Incident handling is a clearly defined set of procedures to manage and respon...
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphGraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

  • 1. 1Confidential Flexible and Scalable Integration in Automation Industry / Industrial IoT Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de Kafka-Native End-to-End IIoT Data Integration and Processing with Kafka Connect, KSQL and Apache PLC4X
  • 2. 2 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 3. 3 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 4. 4 Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency Mobile Cloud Microservices Internet of Things Machine Learning The world has changed!
  • 5. 5 Industry 4.0 / Industrial IoT (IIoT)
  • 6. 6 Some IIoT use cases Analytics • Ingest data into cloud for analytics • Reduce cost: Leverage open frameworks instead of paying very expensive licenses per machine • Flexible integration (select data to ingest, flexible changes over time) • Machine Learning / Data Science Manufacturing • Collect data from machines à Preprocess + monitoring to optimize assembly line and reduce cost • Aggregate data from different machines / companies —> Leverage (and sell?) insights • Sell services on top of machines —> Predictive maintenance (remote) • Scale up (add more sites, add more data) Production Robots • Ingest, process and monitor large volumes data (where the proprietary monolith does not scale) Smart Factories • Monitor and manage the whole factory (at scale, in real time, flexible) • Integration with legacy proprietary protocols and modern cloud-native technologies
  • 7. 7 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 8. 8 History of Automation Industry vs. Big Data and Cloud Christofer Dutz (codecentric) https://foss-backstage.de/sites/foss-backstage.de/files/2018-07/Revolutionizing%20Industrial%20IoT%20with%20Apache%20PLC4X.pdf
  • 9. 9 Challenges in Automation Industry IoT != IIoT • IoT = Connected cars, smart home, … à Large scale, secure, scalable, open, modern technologies • IIoT = Slow, insecure, not scalable, proprietary Legacy / Proprietary IIoT Technologies • Usually incompatible protocols, typically proprietary • Usually serial connections (very low latency, nanoseconds) - with TCP / UDP wrapper around it to integrate with “external world” • Siemens S7, Modbus, Beckhoff, Profinet, Allen Bradley, etc. • OPC-UA (required machine update + license cost) Product Lifecycles • Long lifecycle (tens of years) • Factories cost millions, no simple changes / upgrades • Still using Windows 7 without Service Packs => Usability and security issues • Mantra: “Stay with your well known vendor forever”
  • 10. 10 Challenges in Automation Industry Monoliths • No scalability • No extendibility • No real failover (start your backup machine) Missing Security Capabilities • Security in software development == Authentication, Authorization, Antivirus, SSL, SASL, Kerberos • Security in automation industry == Safety • “if you press the red button, the machine stops immediately” • Insecure by nature => No Authentication / Authorization / Encryption • Mantra: “Our factory building and network is secure, no access from outside” • Contradicts with “move to cloud and big data analytics”
  • 11. 11 PLC (Programmable Logic Controller) • Started early 70’s • Control of manufacturing processes • Small grey box • ~100 messages per second, stored to CSV file, Windows Share • Limited operations: Read (90+%), Write, Subscribe, Call Functions, List Resources • High reliability control, ease of programming and process fault diagnosis • Hardwire à softwire • Has Input / Sensors, Output / Actors • Firmware (= operating system) • Mechanism to load user programs • Highly fragmented market • S7 (Siemens), Beckhoff ADS, Modbus (Asia), Ethernet/IP, KNX, Emerson DeltaV, Profinet, Allen Bradley, etc. • State of the art in automation industry
  • 12. 12 Example: Siemens S7 Communication When communicating with S7 Devices there is a whole family of protocols, that can be used. In general you can divide them into Profinet protocols and S7 Comm protocols. The later are far simpler in structure, but also far less documented. The S7 Comm protocols are generally split up into two flavors: The classic S7 Comm and a newer version unofficially called S7 Comm Plus. https://plc4x.apache.org/protocols/s7/index.html
  • 13. 13 Trends: ~50% of industrial assets in factories will be connected by 2020 https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
  • 14. 14 Trends: Evolution of Convergence between IT and Industrial Automation https://iot-analytics.com/5-industrial-connectivity-trends-driving-the-it-ot-convergence
  • 15. 15 How to get from legacy, proprietary to cloud, big data, machine learning?
  • 16. 16 Costly and inflexible legacy Integration between IIoT and other Systems ModbusS7 Siemens Integration Middleware Monolith Schneider Electric Integration Middleware Monolith Integration Middleware
  • 17. 17 Huge demand to build an open, flexible, scalable platform • Cost reduction • Flexibility • Standards-based • Scalability • Extendibility • Security • Infrastructure-independent
  • 18. 18 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 19. 19 ? IIoT Architecture (High Level) Kafka BrokerKafka BrokerStreaming Platform Connect w/ MQTT connector GatewayDevicesDevicesDevicesMachine Sensor Analytics (Real Time) Predictive Maintenance (Near Real Time) Machine Learning (Batch) Edge Data Center / Cloud How to integrate and process data at scale and reliable?
  • 20. 20 Vendor-Neutral IoT Architectures across Edge, On Premise and Multi-Cloud On-Premise / Edge Deploy on bare-metal, VMs, containers or Kubernetes in your datacenter with Confluent Platform and Confluent Operator Public Cloud Implement self-managed in the public cloud or adopt a fully managed service with Confluent Cloud Hybrid Cloud Build a persistent bridge between datacenter and cloud with Confluent Replicator Confluent Replicator VM SELF MANAGED FULLY MANAGED
  • 21. Data Lake Batch Analytics Event Streaming Platform Batch Integration Real Time Pre- processing Machine Sensors Streaming Platform Other Components Real Time Processing (6b) All Data (7) Potential Defect (3) Read Data Optimization / Analytics (5) Deploy Optimization Model (8b) Alert Person (e.g. Mobile App) (2) Preprocess Data (6a) Consume machine data Model Standard based Integration (8a) Stop Machine (1) Ingest Data Real Time Edge Computing Model Lite Real Time App Model Server RPC PLC Proprietary based Integration Standard Interface Proprietary Interface (9) Manual user-based analytics and reporting to find insights and improve real time process
  • 22. 22 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 23. 23 The beginning of a new Era https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case. This is why Kafka was created!
  • 24. 24 The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka—The Rise of an Event Streaming Platform
  • 25. 25 ● Global-scale ● Real-time ● Persistent Storage ● Stream Processing Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka Apache Kafka: The De-facto Standard for Real-Time Event Streaming
  • 26. 26 Apache Kafka at Scale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
  • 27. 27 Confluent - Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
  • 28. 28 Apache Kafka - A Distributed Commit Log Writers Kafka cluster Readers
  • 30. 30 P Decoupled Producers and Consumers Time C2 C3C1
  • 31. 31 Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
  • 34. 34 Kafka Streams ● No separate processing cluster required ● Develop on Mac, Linux, Windows ● Deploy to containers, VMs, bare metal, cloud ● Powered by Kafka: elastic, scalable, distributed, battle-tested ● Perfect for small, medium, large use cases ● Fully integrated with Kafka security ● Exactly-once processing semantics ● Part of Apache Kafka KStream<User, PageViewEvent> pageViews = builder.stream("pageviews-topic"); KTable<Windowed<User>, Long> viewsPerUserSession = pageViews .groupByKey() .count(SessionWindows.with(TimeUnit.MINUTES.toMillis(5)), "session-views"); https://docs.confluent.io/current/streams/ Write standard Java apps and microservices to process your data in real-time
  • 35. 35 KSQL: Enable Stream Processing using SQL-like Semantics Leverage Kafka Streams API using simple SQL commands KSQL server Engine (runs queries) REST API CLIClients Confluent Control Center GUI Kafka Cluster Use any programming language Connect via Control Center UI, CLI, REST or deploy in headless mode
  • 36. 36 streams The streaming SQL engine for Apache Kafka CREATE STREAM fraudulent_payments AS SELECT * FROM payments WHERE fraudProbability > 0.8; Apache Kafka library to write real-time applications and microservices in Java and Scala confluent.io/product/ksql Confluent KSQL You write only SQL. No Java, Python, or other boilerplate to wrap around it! Event Transformation with Stream Processing
  • 37. 37 Kafka Connect ● Centralized management and configuration ● Support for hundreds of technologies including RDBMS, Elasticsearch, HDFS, S3 ● Supports CDC ingest of events from RDBMS ● Preserves data schema ● Fault tolerant and automatically load balanced ● Extensible API ● Single Message Transforms ● Part of Apache Kafka { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url": "jdbc:mysql://localhost:3306/demo?user=rmoff&password=foo", "table.whitelist": "sales,orders,customers" } https://docs.confluent.io/current/connect/ Reliable and scalable integration of Kafka with other systems
  • 38. 38 Connect External Data Sources and Sinks with Connectors SOURCES SINKS CDC Connectors developed and supported by Confluent, partners and the open source community available on confluent.io/hub
  • 39. 39 IoT Integration with Kafka Connect, MQTT and REST Proxy Video and Slides: https://www.confluent.io/kafka-summit-sf18/processing-iot-data-from-end-to-end
  • 40. 40 Native, decoupled Integration between IIoT and other Systems ModbusSiemens S7 Siemens S7 Siemens S7 Modbus Modbus Modbus Kafka Connect Kafka Connect Siemens S7 ?
  • 41. 41 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 42. 42 Apache PLC4X • Top Level Apache project • PLC 4 (for) X (anything) • Goal: Open up PLC interfaces to outside world • Vertical integration • Write software independent of PLC • JDBC-like Adapters for various protocols https://plc4x.apache.org/
  • 43. 43 Code Example – Connection to Siemens S7 PLC Feels like JDBC
  • 44. 44 Native, decoupled Integration between IIoT and other Systems ModbusSiemens S7 Siemens S7 Siemens S7 Modbus Modbus ModbusSiemens S7 Kafka Connect
  • 45. 45 One more thing à PLC4X vs. OPC-UA • Open standard • All the pros and cons of an open standard (works with different vendors; slow adoption; inflexible, etc.) • Often poorly implemented • Requires app server on top of PLC • Every device has to be retrofitted with the ability to speak a new protocol and use a common client to speak with these devices • Often overengineering for just reading the data • Activating OPC-UA support on existing PLCs greatly increases the load on the PLCs • With licensing cost for every machine • Open source framework (Apache 2.0 license) • Provides unified API by implementing drivers for communicating with most industrial controllers in the protocols they natively understand • No need to modify existing hardware • No increased load on the PLCs • No need to pay for licenses to activate OPC-UA support • Drivers being implemented from the specs or by reverse engineering protocols in order to be fully Apache 2.0 licensed • PLC4X adapter for OPC-UA available -> Both can be used together!
  • 46. 46 Agenda 1) Modern IIoT Use Cases around Cloud, Big Data, Machine Learning 2) Automation Industry and its Challenges 3) Architecture for End-to-End Integration from Edge to Data Center / Cloud 4) Apache Kafka as Event Streaming Platform 5) Apache PLC4X for Edge Integration 6) Example: Supply Chain Optimization at Scale in Real Time
  • 47. Spark Notebooks (Jupyter) Kafka Cluster Kafka Connect KSQL Machine Sensors Kafka Ecosystem Other Components Real Time Kafka Streams Application (Java / Scala) (6b) All Data (7) Potential Defect (3) Read Data TensorFlow I/O TensorFlow (5) Deploy Model (2) Preprocess Data (6a) Consume machine data TensorFlow File HTTP MQTT ROS (8a) Stop Machine (1) Ingest Data Real Time Edge Computing (C / librdkafka) TensorFlow Lite Real Time Kafka App TensorFlow Serving HTTP / gRPC (4) Train Model PLC Beckhoff S7 Modbus Allen Bradley OPC-UA PLC4X Connector Kafka Connect Standard Interface Proprietary Interface (8b) Alert Person (e.g. Mobile App) (9) Manual user-based analytics and reporting to find insights and improve real time process
  • 48. Example Project: Supply Chain Optimization in Real Time at Scale
  • 49. Planners forecast long term schedule Production begins IOT data from production: inventories, manufacturing machines, yield metrics Production forecast Forecasted production - plan diffs Re optimize plan based on actuals Change orders to supply chain: inventory, manufacturing schedules Change operational characteristics : plant 223 needs new Al extruder Customer delivery SLAs: actuals vs. plan Streaming analytics using Confluent Batch analytics using other frameworks Physical operations UI UI UIUI (Reference use case implemented with our partner Expero)
  • 50. Planners forecast long term schedule Production begins IOT data from production: inventories, manufacturing machines, yield metrics Production forecast Forecasted production - plan diffs Re optimize plan based on actuals Change orders to supply chain: inventory, manufacturing schedules Change operational characteristics : plant 223 needs new Al extruder Customer delivery SLAs: actuals vs. plan UI UI UIUI Kafka Connect + PLC4X Connector Machine Sensors Kafka Cluster KSQL Tensor Flow Kafka Connect Notebooks (Jupyter) Spark Real Time Kafka App Streaming analytics using Confluent Batch analytics using other frameworks Physical operations TensorFlow Serving (Reference use case implemented with our partner Expero)
  • 51. 51 Supply Chain Optimization in Real Time at Scale Slides and Video Recording: http://www.kai-waehner.de/blog/2019/08/23/apache-kafka-machine-learning-for-real-time-supply-chain-iiot-opcua-modbus/
  • 53. 53 Confluent Platform The Event Streaming Platform Built by the Original Creators of Apache Kafka® Operations and Security Development & Stream Processing Apache Kafka Confluent Platform Support,Services, Training,&Partners Mission-Critical Reliability Complete Event Streaming Platform Freedom of Choice Datacenter Public Cloud Confluent Cloud Self-Managed Software Fully Managed Service
  • 54. 56 Confluent Platform – Benefits for IoT Projects • Based on open source and de facto standards for IoT projects • Low license / subscription costs for Confluent support / services / training (compared to traditional IoT vendors + their products) • Spend budget for consulting to realize the project successfully • Mission critical deployments at large scale in various industries • Automotive, Manufacturing, Logistics, Oil&Gas, Retail, Telco, … • Flexible architecture • Lightweight infrastructure footprint on commodity hardware • Pick what you need • Deploy where you want • Complementary to other frameworks, technologies (e.g. Siemens MindSphere, Cisco Kinetic) and cloud services (e.g. Google Cloud IoT) • Customize and build for the specific customer use case • Battle-tested at large scale • Event Streaming Platform for real time integration and processing (plus integration to batch, file and other communication protocols) • Security and reliability as core concepts • Elastic scalability, start small and grow to extreme scale easily • Partner (open source) technologies for specific integrations (like HiveMQ or PLC4X) • Integration with any legacy and modern technology • IoT standards like MQTT or OPC-UA • Legacy and proprietary IIoT protocols like Modbus, Siemens S7, Beckhoff, Allen Bradley, etc. • Modern technologies like S3, HDFS, MongoDB, etc. • Modern applications (business services like Salesforce and IoT solutions like Siemens MindSphere)
  • 55. 57 Confluent and IoT Platform Solutions Kafka Cluster Siemens MindSphere KSQL Machine Sensors File HTTP MQTT ROS PLC Beckhoff S7 Modbus OPC-UA “you-name-it” PLC4X Connector Kafka Connect Azure IoT Hub Framework or solution? Or both as complementary technologies? S7 PLC