SlideShare a Scribd company logo
1 of 49
Download to read offline
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
IoT meets Confluent meets Data Platform
MQTT
Broker
OPC UA
gRPC
Proxy
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Goal
Partners Tech Talks are webinars where subject matter experts from a Partner talk about a
specific use case or project. The goal of Tech Talks is to provide best practices and
applications insights, along with inspiration, and help you stay up to date about innovations
in confluent ecosystem.
@yourtwitterhandle | developer.confluent.io
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
Confluent Perspective on
IoT
8
9
Business challenges Technical challenges
Limited scalability with messaging
brokers requiring manual horizontal
scaling via VMs.
Downtime and support tickets with
operational disruption.
Data schema incompatibility breaking
existing functionality or causing data loss.
Unprecedented data volume – collecting
data from your 100,000s of IoT devices can
be challenging to store and process.
Delayed time to insight from high latency
in batch data processing, hindering your
organization’s ability to react by hours or
more.
Data variety, as data from text, audio, and
video is challenging to analyze.
Data quality issues with noisy and
incomplete data in inconsistent formats
affecting the accuracy of your analysis.
10
Why Confluent
Stream
data everywhere, from IoT devices via MQTT,
on premises and in every major public cloud.
Connect
IoT sensors, objects, devices, and other systems
with pre-built, fully managed connectors to
build streaming data pipelines.
Process
data streams in flight to create live, refined,
ready-to-use IoT data products.
Govern
data to ensure quality, security, and
compliance while enabling teams to discover
and leverage existing data products.
Business impact
Create new revenue streams for your
business (e.g., route optimization modules for
your customers to save fuel costs and optimize
driver hours).
Unlock real-time analytics for new use cases
such as predictive maintenance.
Improve your platform reliability and
stability with Confluent’s 99.99% uptime SLA.
Seamlessly scale from 0.5 MBps to 50 MBps in
a matter of minutes.
INDUSTRY: ALL
MQTT: the natural candidate
➢ MQTT is lightweight and designed to address edge devices connectivity
○ Poor connectivity / High latency network
➢ MQTT can address many thousands connections with filtered distribution of data to
consumers (esp. devices)
➢ Many enterprise and open source MQTT broker implementations
○ Mosquitto, RabbitMQ, HiveMQ, VerneMQ
➢ MQTT is becoming a de facto standard in (I)IoT space
○ Both Edge & Cloud
➢ Many Client Libraries
○ C, C++, Java, C#, Python, Javascript, websockets, Arduino …
11
… But MQTT has caveats and is not enough
MQTT is designed for safe message delivery, not for stream processing
Once message is delivered, message is not retained.
In case of processing crash after message delivery, messages are lost and cannot be
re-processed, then corrupting business outputs
Real-time processing of your manufacturing data require stream processing
infrastructure: Apache Kafka
Recommended read : https://www.umh.app/post/tools-techniques-for-scalable-data-processing-in-industrial-iot
12
IoT Data
Ingestion at Edge
IoT Data
Aggregation & Processing
Other OT Protocols
IoT
Gateway
Custom
Integration
Edge Integration
Data Ingestion &
Processing
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Connect - Broker to Kafka
7
Complements Kafka's concurrent
connection limits
Complementing a large number of
simultaneous connections, which Kafka
is not good at, with a dedicated broker.
All Brokers provide Kafka or Confluent
connection functions and can be
connected seamlessly.
Broker selection according to
connection needs
Various brokers can be selected
according to the number of
connected devices, connection
type, traffic volume, and client
requirements.
Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
MQTT Confluent Connectors
14
Kafka Cluster
Kafka Connect Cluster
MQTT Source Connector
MQTT Sink Connector
Broker
Subscribe &
Consume
Publish
Publish
Subscribe &
Consume
➢ Relies on 3rd Party MQTT Broker
○ Some example of integration with HiveMQ :
https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realt
ime-iot-machine-learning-training-inference/tree/master/infrastruct
ure/terraform-gcp
➢ Handles both communication paths
➢ Available on confluent.io/hub
Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
MQTT Proxy
Gateways BROKER
Devices MQTT
Proxy
➢ MQTT Proxy exposes MQTT protocol and translates into Kafka Protocol
➢ Remove need to deploy and manage 3rd party MQTT brokers
➢ Different solutions
○ Confluent MQTT Proxy (only handles device-to-Kafka data flows)
○ Technology partner
@yourtwitterhandle | developer.confluent.io
What are the best practices to debug client applications
(producers/consumers in general but also Kafka Streams
applications)?
Starting soon…
STARTING SOOOOON..
Starting sooooon ..
About me
ALEXANDER KEIDEL
Head of Product & Alliance
in the fields of Business Intelligence/Big Data and
IoT/Smart City.
Main focus: Design and implementation of
heterogeneous software architectures with open
source components such as Kafka, Kong,
Pentaho and ThingsBoard.
Content
− Intro
− What is the Problem that the Unified namespace solves
− Implementation with HiveMQ
− UNF: Confluent+ Hive ..better together
− Recaps
The (I)IoT Space
OT
Cloud Services
IT
Fieldsensor
s
− Currently most challenging Area (Ton of Data Silos)
− Protocols for build OT
− Legacy hardware and limitations
− Avg. depreciation period for a maschine around 15Y
− Dominated by OPC-UA
Focus on OT
The Shopfloor in a Nutshell
− A machine (e.g. Robot) is interconnected via an Realtime-Protocol to a PLC that runs the
production routines
− Data is capatured by an MES and or SCADA for Ops Control
OT Protocol
(e.g.Profinet)
PL
C
SCAD
A
MES
Low grain data is key to AI
Level 4
ERP
Level 3 Ops
Control
(MES)
Level 2 Process Control
(SCADA)
Level 1 - Control (PLC)
Level 0 - Field Level (Sensors Actors)
min
hours
sec/m
s
Days
us/ms
OPC-UA and the problem it solves
• Unified Access for different machine vendors and types (Otherwise, the MES has to implement
the vendor-specific protocol)
• Addresses/Variables of the PLC Program are displayed in a hierarchical Namespace
OT
Protocol
Siemens
OT
Protocol
Beckhoff
OT
Protocol
Maxwell
OPC
-UA
Serv
er
MES
OPC-U
A
Client
1. Complexity
OPC UA offers a wide range of
features including complex data
models, security mechanisms, and
interoperability capabilities.
While these features are beneficial,
they can also make the
implementation and configuration of
OPC UA more complex compared to
simpler protocols.
Problems of OPC-UA (my Top 3)
3. Security Concerns
While OPC UA includes robust
security features, the configuration
and maintenance of these security
measures can be complex.
Incorrectly configured security
settings might leave systems
vulnerable.
Additionally, as with any networked
technology, the larger attack surface
of IIoT systems might expose OPC
UA implementations to cyber threats
if not properly secured.
2. Scalability
Although OPC UA is designed to be
scalable, managing a large number
of OPC UA servers and clients in a
vast IIoT network can become
challenging.
The protocol's sophisticated features
might require more resources,
making it harder to scale up
efficiently in large deployments
without significant resource
allocation and careful planning.
MQTT in a Nutshell
− MQTT in its first version
was developed in 1999
for monitoring oil
pipelines. Initially
designed for limited
bandwidth through radio
and satellite networks
− Implements
publish/subscribe pattern
− Payload variable
TCP-based (MQTT-S for
UDP)
− TLS supported
− Scalable for millions of
connections
MQTT 5.0 Features
Vanilla MQTT – Problems (Top 5)
1. MQTT does allow all payloads (Structure).
2. MQTT does allow the Publisher to set the QoS (Quality of Service level should be decided by
the Subscriber).
3. MQTT does allow the Publisher to set Retained Messages (creates load on the Broker).
4. MQTT does allow the Publisher to register LWT (Last Will and Testament) Messages (pertains
to Business Logic).
5. MQTT does not make suggestions regarding Namespaces (Governance) (e.g.,
/plant/sensorA, /dev/SensorB…SensorC/temp/C).
Sparkplug B
•
2015 Introduction of Sparkplug: Cirrus Link Solutions introduces Sparkplug, a specification designed to
enhance MQTT with a standard data format.
• 2017 Eclipse Tahu Project: Sparkplug B is contributed to the Eclipse Foundation under the Tahu Project to
promote community-driven development and standardization.
• 2017 Adoption and Iteration: is released adoption by industries starts to take place, with iterations made to
the Sparkplug B specification based on real-world use cases and feedback.
• 2019-2020 : Sparkplug 2.2 is released as Industry 4.0 gains momentum, Sparkplug B sees increased adoption
as a key enabler for interoperability in IoT platforms.
• 2022-ongoing : Sparkplug 3.0.0 is released, containing various Improvements, super seeding 2.2.
Sparkplug B
1. Uniform Data Structure: Sparkplug B defines a uniform data
structure that ensures data is transmitted in a standardized way
in an MQTT-based network.
2. Uniform Namespaces: Sparkplug B defines a standardized
method for managing IoT endpoints, including the transmission
of device metadata and status information.
3. Extensibility: Sparkplug B is an open framework that is
extendable to meet the requirements of different applications and
industries. It allows developers to tailor it to their specific needs
without the need to change the underlying functionality.
4. Specifications regarding QoS, LW and Retained Flags for all
Message Types
Sparkplug Basis Message Type
1. "Birth" - This message is used to announce the creation or presence
of a device or namespace at the broker.
2. "Data" - This message type contains the payload data for a specific
data element within a namespace.
3. "CMD" - This message type is used to transmit commands from the
broker to a device.
4. "Death" - This message is used to announce the disappearance or
decommissioning of a device or namespace from the network.
5. "State" - This message type contains the current state or data of a
device or namespace. Distinction between N (Edge of Network) and
D(Device) messages Example: NData / DData
The Idea of the Unified Namespace in a Nutshell
1. Lets use MQTT for interconnecton in all the IIoT Space
2. Lets think of all Data Sources like devices/sensors
3. Lets use SparkPlug
4. Lets use ISA95 for our Namespace Hierachy as a Start
5. The unfied Namespace should be the single source of
truth of IIoT
OT IT
Fieldsensors
Cloud
Unified Name
Space
Unlock the Power
of IIoT in
Smart Manufacturing
Contact Details
34
David Guschakowski
Senior Solution Engineer
📨 david.guschakowski@hivemq.com
The Enterprise
MQTT Platform
35
HiveMQ
Solves Reliability
Cluster
Zero message loss
Persistent messaging and replication
to disk, true Quality of Service (QoS)!
Reliable communication
Connection and cluster overload
protection, automatic throttling,
queueing, retained messages..
No single point of failure
Masterless cluster architecture.
Zero downtime upgrades
Broker spawning with nodes
seamlessly upgraded.
36
HiveMQ Solves Scalability
● Proven scalability – benchmarked to
200M active clients with 1.8B
messages/hour
● Elastic scaling – Masterless load
balancing, automatic data balancing,
smart message distribution across cluster
nodes
● Linear scalability – Scale from 2->100+
nodes with consistent hashing algorithm
both vertically and horizontally
37
HiveMQ Enables Edge
Edge Deployment
Address connectivity challenges of organizations
Enables Unified Namespace
Eliminate data silos by enabling UNS
API-based Operability
Enables data sharing with enterprise
Machine Protocols Supported
OPC UA, Modbus, MQTT SN, …
Addresses escalating deployment costs
Open source technologies
38
HiveMQ Improves Data Quality
Data Policies
Define set of rules and guidelines to enforce
how data and messages should be expected.
Data Schemas
Create the blueprint for how data is formatted.
JSON and Protobuf currently supported.
Control Center
Simple GUI to manage schemas, data and
behavior policies. Dashboard provides an
overview of quality metrics making it easy
to locate bad actors and bad data sources.
Data Validation and Transformation
Defining and enforcing data standards across
deployments.
Policy Actions
Describe what should happen to messages/data
that fail validation. Messages can be rerouted,
forwarded, or simply logged and ignored.
39
Build your own!
Java SDK
HiveMQ Solves Interoperability
Runs anywhere
Cloud, on-premises, public and private cloud
Connect from everything
Client support for Java, C, C++, C#, Python, …
Enterprise security
OAuth 2.0, LDAP, RBAC, …
Robust streaming support
Kafka, Amazon Kinesis, Google Pub/Sub, …
Database analytics support
Postgres, Snowflake, Databricks, MongoDB,
InfluxDB, …
40
● Create a federation of multiple clusters and
bidirectionally exchange IoT data between
geographically distributed areas (on-prem and
cloud)
● Allow low latency communication between devices
in local network
● Local broker serves as buffer in case of connection
loss to data center
41
Converge data in your central IT
42
HiveMQ + Kafka
HiveMQ and Kafka are better together. Kafka is designed for fault-tolerant, high throughput data
pipelines, and HiveMQ is designed for reliable, scalable real-time communication with constrained
IoT devices. They can work together to enable end-to-end data streaming and real-time data
processing scenarios in IoT deployments.
Why Hive and Confluent for UNF
MQTT
− Optimize for monitoring of devices &
sensors
− Deep topic structure, millions of topics
− Millions von connections
− Data Collection, feedback canal, M2M
Kafka
− Optimize for data provision for distribution
in companies
− Flat topic structure (scale over partitions)
− High throughput (e.g. Analytics, Big
Data…)
Why Confluent and Hive for UNS
− The UNS being MQTT-based does not contain any history of data
and only represents a snap-shot of the current state, relies on
Historian that is not designed for that
▪ Solution: Shadow the MQTT broker with Confluent to preserve
history of the
− MQTT was not build for training AI or running Analytics
(Small File Problem for e.g. get 1M Sensor Points for One Device
Type.)
▪ Solution: Fan in thousends MQTT Topics e.g. based on
Device Type into larger Kafka Topics
− MQTT is not fit for complex or high throughput Streamprocessing
Tasks
▪ Solution: Fan în Data in Kafka Process it with Flink/Kstreams
For Confluent Partners what Confluent Features
do support the Unified Name Space
− Schema Registry
▪ SparkPlug Messages are Protobuf, putting the Schema on the SchemaReg allows
for easy Structured Streaming
− On Cloud: Advanced Stream Governance
▪ Data Contracts and Business Tags help to put Business Context to the Data for Data
Scientiests
− Kafka Streams
▪ Microservices for Data Transformation even for small Volume Topics
− Cluster Linking:
▪ Hub and Spoke Architectures with local Clusters and Cloud Clusters
Sample Architecture
Recap & Take-Aways
− Unified Namespace is a promising concept for IIoT to allow harmonized device
interconnections
− Sparkplug B vs OPC-UA is benefical when thinking about cloud and field-sensor integration
− Using MQTT with confluent is benefical as it adds
▪ History
▪ Schematization
▪ Governance
▪ Streamprocessing
Q & A
Vielen Dank
für Ihre Aufmerksamkeit
it-novum GmbH Deutschland
Hauptsitz: Edelzeller Straße 44,
36043 Fulda
Niederlassung:
Kaiserswerther Str. 229,
40474 Düsseldorf
it-novum Schweiz
GmbH
Seestrasse 97
8800 Thalwil
Schweiz
it-novum Zweigniederlassung
Österreich
Ausstellungsstraße 50 /
Zugang C
1020 Wien
Alexander Keidel
Head of Product & Alliance
T +49 661 103-392
E
alexander.keidel@it-novum.co
m
data.it-novum.com
Thank you!

More Related Content

What's hot

What's hot (20)

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
Netflix Data Pipeline With Kafka
Netflix Data Pipeline With KafkaNetflix Data Pipeline With Kafka
Netflix Data Pipeline With Kafka
 
An Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a ServiceAn Introduction to Confluent Cloud: Apache Kafka as a Service
An Introduction to Confluent Cloud: Apache Kafka as a Service
 
Flink vs. Spark
Flink vs. SparkFlink vs. Spark
Flink vs. Spark
 
Building an Authorization Solution for Microservices Using Neo4j and OPA
Building an Authorization Solution for Microservices Using Neo4j and OPABuilding an Authorization Solution for Microservices Using Neo4j and OPA
Building an Authorization Solution for Microservices Using Neo4j and OPA
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
Apache Kafka Best Practices
Apache Kafka Best PracticesApache Kafka Best Practices
Apache Kafka Best Practices
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
Introducing Confluent Cloud: Apache Kafka as a Service
Introducing Confluent Cloud: Apache Kafka as a Service Introducing Confluent Cloud: Apache Kafka as a Service
Introducing Confluent Cloud: Apache Kafka as a Service
 
How Apache Kafka® Works
How Apache Kafka® WorksHow Apache Kafka® Works
How Apache Kafka® Works
 
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...
 
Various Cloud offerings AWS/AZURE/GCP
Various Cloud offerings AWS/AZURE/GCPVarious Cloud offerings AWS/AZURE/GCP
Various Cloud offerings AWS/AZURE/GCP
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
 
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
스타트업을 위한 Confluent 웨비나 3탄
스타트업을 위한 Confluent 웨비나 3탄스타트업을 위한 Confluent 웨비나 3탄
스타트업을 위한 Confluent 웨비나 3탄
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
What's New in API Connect & DataPower Gateway in 1H 2018
What's New in API Connect & DataPower Gateway in 1H 2018What's New in API Connect & DataPower Gateway in 1H 2018
What's New in API Connect & DataPower Gateway in 1H 2018
 

Similar to Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and SparkPlug

MuleSoft Meetup Singapore #8 March 2021
MuleSoft Meetup Singapore #8 March 2021MuleSoft Meetup Singapore #8 March 2021
MuleSoft Meetup Singapore #8 March 2021
Julian Douch
 

Similar to Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and SparkPlug (20)

IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
 
Kafka Summit 2021 - Why MQTT and Kafka are a match made in heaven
Kafka Summit 2021 - Why MQTT and Kafka are a match made in heavenKafka Summit 2021 - Why MQTT and Kafka are a match made in heaven
Kafka Summit 2021 - Why MQTT and Kafka are a match made in heaven
 
Connext eng
Connext engConnext eng
Connext eng
 
DEVELOPMENT AND IMPLEMENTATION OF LOW COST IIOT GATEWAY WITH EDGE COMPUTING F...
DEVELOPMENT AND IMPLEMENTATION OF LOW COST IIOT GATEWAY WITH EDGE COMPUTING F...DEVELOPMENT AND IMPLEMENTATION OF LOW COST IIOT GATEWAY WITH EDGE COMPUTING F...
DEVELOPMENT AND IMPLEMENTATION OF LOW COST IIOT GATEWAY WITH EDGE COMPUTING F...
 
Realtime mobile&iot solutions using mqtt and message sight
Realtime mobile&iot solutions using mqtt and message sightRealtime mobile&iot solutions using mqtt and message sight
Realtime mobile&iot solutions using mqtt and message sight
 
Open platform communication
Open platform communicationOpen platform communication
Open platform communication
 
MuleSoft Meetup Singapore #8 March 2021
MuleSoft Meetup Singapore #8 March 2021MuleSoft Meetup Singapore #8 March 2021
MuleSoft Meetup Singapore #8 March 2021
 
Session 1908 connecting devices to the IBM IoT Cloud
Session 1908   connecting devices to the  IBM IoT CloudSession 1908   connecting devices to the  IBM IoT Cloud
Session 1908 connecting devices to the IBM IoT Cloud
 
HiveMQ + Kafka - The Ideal Solution for IoT MQTT Data Integration
HiveMQ + Kafka - The Ideal Solution for IoT MQTT Data IntegrationHiveMQ + Kafka - The Ideal Solution for IoT MQTT Data Integration
HiveMQ + Kafka - The Ideal Solution for IoT MQTT Data Integration
 
SuperConnectivity: One company’s heroic mission to deliver on the promises of...
SuperConnectivity: One company’s heroic mission to deliver on the promises of...SuperConnectivity: One company’s heroic mission to deliver on the promises of...
SuperConnectivity: One company’s heroic mission to deliver on the promises of...
 
Is your MQTT broker IoT ready?
Is your MQTT broker IoT ready?Is your MQTT broker IoT ready?
Is your MQTT broker IoT ready?
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
InduSoft Web Studio and MQTT for Internet of Things Applications
InduSoft Web Studio and MQTT for Internet of Things ApplicationsInduSoft Web Studio and MQTT for Internet of Things Applications
InduSoft Web Studio and MQTT for Internet of Things Applications
 
Best Practices for Streaming Connected Car Data with MQTT & Kafka
Best Practices for Streaming Connected Car Data with MQTT & KafkaBest Practices for Streaming Connected Car Data with MQTT & Kafka
Best Practices for Streaming Connected Car Data with MQTT & Kafka
 
InduSoft IoTView
InduSoft IoTViewInduSoft IoTView
InduSoft IoTView
 
Programming IoT Gateways with macchina.io
Programming IoT Gateways with macchina.ioProgramming IoT Gateways with macchina.io
Programming IoT Gateways with macchina.io
 
Web of things
Web of thingsWeb of things
Web of things
 
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQBuilding a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
 
Whitepaper: Mobile Networks in a smart digital future - deploying a platform ...
Whitepaper: Mobile Networks in a smart digital future - deploying a platform ...Whitepaper: Mobile Networks in a smart digital future - deploying a platform ...
Whitepaper: Mobile Networks in a smart digital future - deploying a platform ...
 
Flexible and Scalable Integration in the Automation Industry/Industrial IoT
Flexible and Scalable Integration in the Automation Industry/Industrial IoTFlexible and Scalable Integration in the Automation Industry/Industrial IoT
Flexible and Scalable Integration in the Automation Industry/Industrial IoT
 

More from confluent

More from confluent (20)

Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 

Recently uploaded

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdfThe Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
The Evolution of Web App Testing_ An Ultimate Guide to Future Trends.pdf
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Novo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMsNovo Nordisk: When Knowledge Graphs meet LLMs
Novo Nordisk: When Knowledge Graphs meet LLMs
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and SparkPlug

  • 1. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 2. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON..
  • 3. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. IoT meets Confluent meets Data Platform MQTT Broker OPC UA gRPC Proxy
  • 4. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. IoT meets Confluent meets Data Platform MQTT Broker OPC UA gRPC Proxy
  • 5. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)?
  • 6. Goal Partners Tech Talks are webinars where subject matter experts from a Partner talk about a specific use case or project. The goal of Tech Talks is to provide best practices and applications insights, along with inspiration, and help you stay up to date about innovations in confluent ecosystem.
  • 7. @yourtwitterhandle | developer.confluent.io Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 9. 9 Business challenges Technical challenges Limited scalability with messaging brokers requiring manual horizontal scaling via VMs. Downtime and support tickets with operational disruption. Data schema incompatibility breaking existing functionality or causing data loss. Unprecedented data volume – collecting data from your 100,000s of IoT devices can be challenging to store and process. Delayed time to insight from high latency in batch data processing, hindering your organization’s ability to react by hours or more. Data variety, as data from text, audio, and video is challenging to analyze. Data quality issues with noisy and incomplete data in inconsistent formats affecting the accuracy of your analysis.
  • 10. 10 Why Confluent Stream data everywhere, from IoT devices via MQTT, on premises and in every major public cloud. Connect IoT sensors, objects, devices, and other systems with pre-built, fully managed connectors to build streaming data pipelines. Process data streams in flight to create live, refined, ready-to-use IoT data products. Govern data to ensure quality, security, and compliance while enabling teams to discover and leverage existing data products. Business impact Create new revenue streams for your business (e.g., route optimization modules for your customers to save fuel costs and optimize driver hours). Unlock real-time analytics for new use cases such as predictive maintenance. Improve your platform reliability and stability with Confluent’s 99.99% uptime SLA. Seamlessly scale from 0.5 MBps to 50 MBps in a matter of minutes. INDUSTRY: ALL
  • 11. MQTT: the natural candidate ➢ MQTT is lightweight and designed to address edge devices connectivity ○ Poor connectivity / High latency network ➢ MQTT can address many thousands connections with filtered distribution of data to consumers (esp. devices) ➢ Many enterprise and open source MQTT broker implementations ○ Mosquitto, RabbitMQ, HiveMQ, VerneMQ ➢ MQTT is becoming a de facto standard in (I)IoT space ○ Both Edge & Cloud ➢ Many Client Libraries ○ C, C++, Java, C#, Python, Javascript, websockets, Arduino … 11
  • 12. … But MQTT has caveats and is not enough MQTT is designed for safe message delivery, not for stream processing Once message is delivered, message is not retained. In case of processing crash after message delivery, messages are lost and cannot be re-processed, then corrupting business outputs Real-time processing of your manufacturing data require stream processing infrastructure: Apache Kafka Recommended read : https://www.umh.app/post/tools-techniques-for-scalable-data-processing-in-industrial-iot 12 IoT Data Ingestion at Edge IoT Data Aggregation & Processing Other OT Protocols IoT Gateway Custom Integration Edge Integration Data Ingestion & Processing
  • 13. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Connect - Broker to Kafka 7 Complements Kafka's concurrent connection limits Complementing a large number of simultaneous connections, which Kafka is not good at, with a dedicated broker. All Brokers provide Kafka or Confluent connection functions and can be connected seamlessly. Broker selection according to connection needs Various brokers can be selected according to the number of connected devices, connection type, traffic volume, and client requirements.
  • 14. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. MQTT Confluent Connectors 14 Kafka Cluster Kafka Connect Cluster MQTT Source Connector MQTT Sink Connector Broker Subscribe & Consume Publish Publish Subscribe & Consume ➢ Relies on 3rd Party MQTT Broker ○ Some example of integration with HiveMQ : https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realt ime-iot-machine-learning-training-inference/tree/master/infrastruct ure/terraform-gcp ➢ Handles both communication paths ➢ Available on confluent.io/hub
  • 15. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. MQTT Proxy Gateways BROKER Devices MQTT Proxy ➢ MQTT Proxy exposes MQTT protocol and translates into Kafka Protocol ➢ Remove need to deploy and manage 3rd party MQTT brokers ➢ Different solutions ○ Confluent MQTT Proxy (only handles device-to-Kafka data flows) ○ Technology partner
  • 16. @yourtwitterhandle | developer.confluent.io What are the best practices to debug client applications (producers/consumers in general but also Kafka Streams applications)? Starting soon… STARTING SOOOOON.. Starting sooooon ..
  • 17. About me ALEXANDER KEIDEL Head of Product & Alliance in the fields of Business Intelligence/Big Data and IoT/Smart City. Main focus: Design and implementation of heterogeneous software architectures with open source components such as Kafka, Kong, Pentaho and ThingsBoard.
  • 18. Content − Intro − What is the Problem that the Unified namespace solves − Implementation with HiveMQ − UNF: Confluent+ Hive ..better together − Recaps
  • 19. The (I)IoT Space OT Cloud Services IT Fieldsensor s
  • 20. − Currently most challenging Area (Ton of Data Silos) − Protocols for build OT − Legacy hardware and limitations − Avg. depreciation period for a maschine around 15Y − Dominated by OPC-UA Focus on OT
  • 21. The Shopfloor in a Nutshell − A machine (e.g. Robot) is interconnected via an Realtime-Protocol to a PLC that runs the production routines − Data is capatured by an MES and or SCADA for Ops Control OT Protocol (e.g.Profinet) PL C SCAD A MES
  • 22. Low grain data is key to AI Level 4 ERP Level 3 Ops Control (MES) Level 2 Process Control (SCADA) Level 1 - Control (PLC) Level 0 - Field Level (Sensors Actors) min hours sec/m s Days us/ms
  • 23. OPC-UA and the problem it solves • Unified Access for different machine vendors and types (Otherwise, the MES has to implement the vendor-specific protocol) • Addresses/Variables of the PLC Program are displayed in a hierarchical Namespace OT Protocol Siemens OT Protocol Beckhoff OT Protocol Maxwell OPC -UA Serv er MES OPC-U A Client
  • 24. 1. Complexity OPC UA offers a wide range of features including complex data models, security mechanisms, and interoperability capabilities. While these features are beneficial, they can also make the implementation and configuration of OPC UA more complex compared to simpler protocols. Problems of OPC-UA (my Top 3) 3. Security Concerns While OPC UA includes robust security features, the configuration and maintenance of these security measures can be complex. Incorrectly configured security settings might leave systems vulnerable. Additionally, as with any networked technology, the larger attack surface of IIoT systems might expose OPC UA implementations to cyber threats if not properly secured. 2. Scalability Although OPC UA is designed to be scalable, managing a large number of OPC UA servers and clients in a vast IIoT network can become challenging. The protocol's sophisticated features might require more resources, making it harder to scale up efficiently in large deployments without significant resource allocation and careful planning.
  • 25. MQTT in a Nutshell − MQTT in its first version was developed in 1999 for monitoring oil pipelines. Initially designed for limited bandwidth through radio and satellite networks − Implements publish/subscribe pattern − Payload variable TCP-based (MQTT-S for UDP) − TLS supported − Scalable for millions of connections
  • 27. Vanilla MQTT – Problems (Top 5) 1. MQTT does allow all payloads (Structure). 2. MQTT does allow the Publisher to set the QoS (Quality of Service level should be decided by the Subscriber). 3. MQTT does allow the Publisher to set Retained Messages (creates load on the Broker). 4. MQTT does allow the Publisher to register LWT (Last Will and Testament) Messages (pertains to Business Logic). 5. MQTT does not make suggestions regarding Namespaces (Governance) (e.g., /plant/sensorA, /dev/SensorB…SensorC/temp/C).
  • 28. Sparkplug B • 2015 Introduction of Sparkplug: Cirrus Link Solutions introduces Sparkplug, a specification designed to enhance MQTT with a standard data format. • 2017 Eclipse Tahu Project: Sparkplug B is contributed to the Eclipse Foundation under the Tahu Project to promote community-driven development and standardization. • 2017 Adoption and Iteration: is released adoption by industries starts to take place, with iterations made to the Sparkplug B specification based on real-world use cases and feedback. • 2019-2020 : Sparkplug 2.2 is released as Industry 4.0 gains momentum, Sparkplug B sees increased adoption as a key enabler for interoperability in IoT platforms. • 2022-ongoing : Sparkplug 3.0.0 is released, containing various Improvements, super seeding 2.2.
  • 29. Sparkplug B 1. Uniform Data Structure: Sparkplug B defines a uniform data structure that ensures data is transmitted in a standardized way in an MQTT-based network. 2. Uniform Namespaces: Sparkplug B defines a standardized method for managing IoT endpoints, including the transmission of device metadata and status information. 3. Extensibility: Sparkplug B is an open framework that is extendable to meet the requirements of different applications and industries. It allows developers to tailor it to their specific needs without the need to change the underlying functionality. 4. Specifications regarding QoS, LW and Retained Flags for all Message Types
  • 30. Sparkplug Basis Message Type 1. "Birth" - This message is used to announce the creation or presence of a device or namespace at the broker. 2. "Data" - This message type contains the payload data for a specific data element within a namespace. 3. "CMD" - This message type is used to transmit commands from the broker to a device. 4. "Death" - This message is used to announce the disappearance or decommissioning of a device or namespace from the network. 5. "State" - This message type contains the current state or data of a device or namespace. Distinction between N (Edge of Network) and D(Device) messages Example: NData / DData
  • 31. The Idea of the Unified Namespace in a Nutshell 1. Lets use MQTT for interconnecton in all the IIoT Space 2. Lets think of all Data Sources like devices/sensors 3. Lets use SparkPlug 4. Lets use ISA95 for our Namespace Hierachy as a Start 5. The unfied Namespace should be the single source of truth of IIoT
  • 33. Unlock the Power of IIoT in Smart Manufacturing
  • 34. Contact Details 34 David Guschakowski Senior Solution Engineer 📨 david.guschakowski@hivemq.com
  • 36. HiveMQ Solves Reliability Cluster Zero message loss Persistent messaging and replication to disk, true Quality of Service (QoS)! Reliable communication Connection and cluster overload protection, automatic throttling, queueing, retained messages.. No single point of failure Masterless cluster architecture. Zero downtime upgrades Broker spawning with nodes seamlessly upgraded. 36
  • 37. HiveMQ Solves Scalability ● Proven scalability – benchmarked to 200M active clients with 1.8B messages/hour ● Elastic scaling – Masterless load balancing, automatic data balancing, smart message distribution across cluster nodes ● Linear scalability – Scale from 2->100+ nodes with consistent hashing algorithm both vertically and horizontally 37
  • 38. HiveMQ Enables Edge Edge Deployment Address connectivity challenges of organizations Enables Unified Namespace Eliminate data silos by enabling UNS API-based Operability Enables data sharing with enterprise Machine Protocols Supported OPC UA, Modbus, MQTT SN, … Addresses escalating deployment costs Open source technologies 38
  • 39. HiveMQ Improves Data Quality Data Policies Define set of rules and guidelines to enforce how data and messages should be expected. Data Schemas Create the blueprint for how data is formatted. JSON and Protobuf currently supported. Control Center Simple GUI to manage schemas, data and behavior policies. Dashboard provides an overview of quality metrics making it easy to locate bad actors and bad data sources. Data Validation and Transformation Defining and enforcing data standards across deployments. Policy Actions Describe what should happen to messages/data that fail validation. Messages can be rerouted, forwarded, or simply logged and ignored. 39
  • 40. Build your own! Java SDK HiveMQ Solves Interoperability Runs anywhere Cloud, on-premises, public and private cloud Connect from everything Client support for Java, C, C++, C#, Python, … Enterprise security OAuth 2.0, LDAP, RBAC, … Robust streaming support Kafka, Amazon Kinesis, Google Pub/Sub, … Database analytics support Postgres, Snowflake, Databricks, MongoDB, InfluxDB, … 40
  • 41. ● Create a federation of multiple clusters and bidirectionally exchange IoT data between geographically distributed areas (on-prem and cloud) ● Allow low latency communication between devices in local network ● Local broker serves as buffer in case of connection loss to data center 41 Converge data in your central IT
  • 42. 42 HiveMQ + Kafka HiveMQ and Kafka are better together. Kafka is designed for fault-tolerant, high throughput data pipelines, and HiveMQ is designed for reliable, scalable real-time communication with constrained IoT devices. They can work together to enable end-to-end data streaming and real-time data processing scenarios in IoT deployments.
  • 43. Why Hive and Confluent for UNF MQTT − Optimize for monitoring of devices & sensors − Deep topic structure, millions of topics − Millions von connections − Data Collection, feedback canal, M2M Kafka − Optimize for data provision for distribution in companies − Flat topic structure (scale over partitions) − High throughput (e.g. Analytics, Big Data…)
  • 44. Why Confluent and Hive for UNS − The UNS being MQTT-based does not contain any history of data and only represents a snap-shot of the current state, relies on Historian that is not designed for that ▪ Solution: Shadow the MQTT broker with Confluent to preserve history of the − MQTT was not build for training AI or running Analytics (Small File Problem for e.g. get 1M Sensor Points for One Device Type.) ▪ Solution: Fan in thousends MQTT Topics e.g. based on Device Type into larger Kafka Topics − MQTT is not fit for complex or high throughput Streamprocessing Tasks ▪ Solution: Fan în Data in Kafka Process it with Flink/Kstreams
  • 45. For Confluent Partners what Confluent Features do support the Unified Name Space − Schema Registry ▪ SparkPlug Messages are Protobuf, putting the Schema on the SchemaReg allows for easy Structured Streaming − On Cloud: Advanced Stream Governance ▪ Data Contracts and Business Tags help to put Business Context to the Data for Data Scientiests − Kafka Streams ▪ Microservices for Data Transformation even for small Volume Topics − Cluster Linking: ▪ Hub and Spoke Architectures with local Clusters and Cloud Clusters
  • 47. Recap & Take-Aways − Unified Namespace is a promising concept for IIoT to allow harmonized device interconnections − Sparkplug B vs OPC-UA is benefical when thinking about cloud and field-sensor integration − Using MQTT with confluent is benefical as it adds ▪ History ▪ Schematization ▪ Governance ▪ Streamprocessing
  • 48. Q & A
  • 49. Vielen Dank für Ihre Aufmerksamkeit it-novum GmbH Deutschland Hauptsitz: Edelzeller Straße 44, 36043 Fulda Niederlassung: Kaiserswerther Str. 229, 40474 Düsseldorf it-novum Schweiz GmbH Seestrasse 97 8800 Thalwil Schweiz it-novum Zweigniederlassung Österreich Ausstellungsstraße 50 / Zugang C 1020 Wien Alexander Keidel Head of Product & Alliance T +49 661 103-392 E alexander.keidel@it-novum.co m data.it-novum.com Thank you!