Guido Schmutz, profile picture
Uploaded byGuido Schmutz
PDF, PPTX1,996 views

IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?

The document discusses the Internet of Things (IoT) and its architectural considerations, including the integration of devices, data analysis, and enterprise infrastructure adaptation. It outlines key challenges and solutions associated with building IoT applications, such as connectivity, data processing, and analytics, while emphasizing the need for appropriate architecture based on specific use cases. Additionally, it highlights the significance of advanced analytics and real-time processing in handling the influx of data generated by IoT devices.

Embed presentation

Download as PDF, PPTX
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH IoT Architecture Guido Schmutz @gschmutz guidoschmutz.wordpress.com Are Traditional Architectures Good Enough or do we Need New Approaches?
Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
Agenda 1. Introduction 2. Towards an IoT Architecture 3. Summary
Introduction
Internet of Things (IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist Source: Ericsson, June 2016
Reasons why IoT opportunity is occurring now ? Affordable hardware • Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware • Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility • Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools • Big data tools & cloud based infrastructure have become widely available Mass market awareness • IoT has surpassed a critical tipping point • Vision of a connected world has reached such a followership that companies have initiated IoT developments • Commitment is irreversible
The Sensing-as-a-Service Model
Towards an IoT Architecture
Key Challenges for building an IoT application 1. Connect: How to collect data from intelligent devices? • Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? • Reduce noise and detect business event at real-time • Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? • Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices
Today) Existing Service-/API Architecture as a base 10 Mobile Apps DB Rich (Web) Client Apps API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST / SOAP REST / SOAP SOAP Various SQL SOAP REST Service BusOracle Data Integrator API Gateway SOA Suite BPM Suite Business Activity Monitoring = one way = request/response DB
REST / SOAP REST / SOAP IoT 1a) Reuse exiting Service-/API-based Architecture IoT Smart Devices 11 Mobile Apps D B Rich (Web) Client Apps D B Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST REST JMS / REST SOAP Various SQL SOAP REST WebSocket JMS Service BusOracle Data Integrator API Gateway API Gateway JMS JMS WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
IoT 1a) Challenges • Are IoT devices powerful enough to communicate over the internet (HTTP protocol) ? • Device only collects data (sense) or is there also the way back necessary (actuator) ? • Can JMS be used from external devices (firewalls allow traffic over JMS) ? • How many IoT devices are planned short and long term ? • How frequently do they produce data ? • Is JMS server and ESB capable for dealing with resulting message volume ? • What operations are needed on IoT data (messages / events) ‘ • only simple transformations, filter and routing operations (VETRO pattern) ?
REST / SOAP REST / SOAP IoT 1b) Reuse existing Service-/API-based Architecture IoT Smart Devices 13 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST HTTP REST REST HTTP JMS API Gateway Service BusOracle Data Integrator SOAP Various SQL SOAP REST WebSocket JMS JMS JMS REST / SOAP WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
REST / SOAP REST / SOAP IoT 2) Adding Event Hub and optional IoT Gateway 14 Mobile Apps D B Rich (Web) Client Apps D B ESB / Data Integration IoT Devices IoT Gateways IoT Smart Devices Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST Kafka / MQTT / REST REST Kafka / MQTT / REST MQTT Kura Event Hub Kafka Service Bus Oracle Data Integrator REST REST SOAP Various SQL SOAP REST WebSocket Kafka JMS JMS API Gateway API Gateway REST Kafka REST / SOAP REST REST SOA Suite BPM Suite Business Activity Monitoring MQTT MQTT = one way = request/response
How to implement an Event Hub? Apache Kafka to the rescue • Distributed publish-subscribe messaging system • Designed for processing of high-volume, real time activity stream data • Stateless (passive) architecture, offset-based consumption • Provides Topics, but does not implement JMS standard • Initially developed at LinkedIn, now part of Apache • Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound
Oracle’s Service Bus as a consumer of Kafka Service Bus 12c Cloud Apps Business Service Cloud Proxy Service Kafka Cloud API Mobile Apps Pipeline Routing Kafka Sensor / IoT Web Apps Business Service REST Business Service WSDL Backend Apps REST Backend Apps WSDL Proxy Service Kafka Pipeline Routing Database DB CDC Stream Processing
IoT 2) Solutions & Challenges Solutions • Event Hub solves the potential scalability issue of JMS • Event Hub can interoperate / integrate with existing SOA platform (ESB) • IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges • Where to do complex analytics on the events? Is it scalable? • Can we really send all data down to backend? Network bandwidth?
REST / SOAP REST / SOAP IoT 3) Adding Stream Processing / Analytics in Backend 18 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response KafkaMQTT
IoT 3) Solutions & Challenges Solutions • Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges • Can we really send all data down to backend? Network bandwidth?
Oracle’s Stream Analytics as consumer of Kafka/MQTT Oracle Stream Analytics Stream Analytics Kafka Kafka Mobile Apps Kafka Sensor / IoT Web Apps Machine Data DB CDC Kafka MQTT
REST / SOAP REST / SOAP IoT 4) Adding Industry 4.0 Data Sources (machine data) 21 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST KafkaMQTT
GoldenGate Gateway Oracle’s GoldenGate for Change Data Capture of existing database Machine Data DB Kafka Oracle GoldenGate Delivery Capture Pump JMS Machine Data DB Oracle GoldenGate Delivery Capture Pump HBase HDFS
REST / SOAP REST / SOAP IoT 5) Adding Stream Processing / Analytics at Edge 23 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka
IoT 5) Solutions & Challenges Solutions • Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges • How to handle integration between the components on the IoT side (fan in)
REST / SOAP REST / SOAP IoT 6) Adding Data Flow Engines 25 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka NiFiNiFi
Continuous Ingestion / Fan-In from the Edge DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 26 Dataflow GW Topic Topic Queue Message GW Topic Dataflow GW Dataflow TopicREST 26 File Source Log Log Log Social Native Topic Topic
Challenges for Ingesting Sensor Data • Multitude of sensors • Multiple Firmware versions • Bad Data from damaged sensors • Data Quality
IoT 6) Solutions & Challenges Solutions • Data Flow engines allow for passing information onwards, potentially over multiple hops (fan-in) Challenges • What if we want to keep the raw data? Where to persist it?
REST / SOAP REST / SOAP IoT 7) Adding Raw Data Storage and Batch Analytics 29 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
IoT 7) Solutions & Challenges Solutions • Adding Big Data platform allows to store all raw data in a distributed file system in a scalable and reliable manner Challenges • How can we leverage the Big Data platform for more than just storing raw data? • How does it integrate with the stream processing?
REST / SOAP REST / SOAP IoT 7a) Adding Data Mining / Machine Learning and Model execution 31 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
REST / SOAP REST / SOAP Oracle Cloud Services and Internet of Things / Big Data 32 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices Gateway IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB = one way = request/response Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI Gateway Kafka / REST REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processin g DB Kafka Kafka HDFS MQTT Hadoop / Spark Oracle IoT Cloud Service Oracle Big Data Cloud Service Oracle BI Cloud Service Oracle Integration Cloud Service Oracle Messaging Cloud Service Oracle Process Cloud Service Oracle SOA Cloud Service Oracle Mobile Cloud Service ESP/CEP Edge Analytics Oracle IoT CS Gateway Oracle IoT CS Client Library = one way = request/response
Summary
Summary Treat events as events! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary? • What is the network bandwidth between device/gateway and cloud/backend? • Centralized or Decentralized IoT solution?
J
Q & A Guido Schmutz Technology Manager guido.schmutz@trivadis.com
IoT Services IoT Logical Reference Architecture IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Internet of Things (IoT) and Big Data37
IoT Services IoT Logical Reference Architecture – Oracle on premises IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Business Activity Monitoring SOA Suite BPM Suite Service Bus Oracle Data Integrator Stream Analytics Big Data Appliance Stream Analytics Service Bus API Gateway Internet of Things (IoT) and Big Data38 Oracle IoT CS Gateway Oracle IoT CS Client Library
IoT Services IoT Logical Reference Architecture – Oracle Cloud Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Oracle BI CS Oracle Big Data CS Oracle SOA CS Oracle Integration CS Oracle IoT CS Oracle Streaming Analytics CS Oracle Messaging CS Oracle Big Data Discovery CS Oracle Mobile CS Internet of Things (IoT) and Big Data39 Oracle IoT CS Gateway Oracle IoT CS Client Library Oracle Process CS Oracle DataFlow ML CS Big Data Preparation CS Application Container CS Container CS
IoT Services IoT Logical Reference Architecture – Microsoft Azure IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Azure IoTHub Azure Streaming Analytics Azure Service Bus Azure Power BI HD Insight Azure Storage Azure IoT Gateway SDK Azure Event Hub Azure Storage Azure Machine Learning Internet of Things (IoT) and Big Data40
IoT Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired IoT Logical Architecture – Open Source Internet of Things (IoT) and Big Data41

More Related Content

PDF
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
byKai Wähner
 
PDF
Azure and Predix
byAltoros
 
PDF
batbern43 Events - Lessons learnt building an Enterprise Data Bus
byBATbern
 
PDF
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
byKai Wähner
 
PDF
Sukumar Nayak-Agile-DevOps-Cloud Management
bySukumar Nayak
 
PDF
L'Internet des objets (IDO)
byCisco Canada
 
PDF
Cloud Native Middleware Microservices - 10 Lessons Learned
byKai Wähner
 
PDF
Internet of Things - Are traditional architectures good enough?
byGuido Schmutz
 
IoT Architectures for a Digital Twin with Apache Kafka, IoT Platforms and Mac...
byKai Wähner
 
Azure and Predix
byAltoros
 
batbern43 Events - Lessons learnt building an Enterprise Data Bus
byBATbern
 
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Ser...
byKai Wähner
 
Sukumar Nayak-Agile-DevOps-Cloud Management
bySukumar Nayak
 
L'Internet des objets (IDO)
byCisco Canada
 
Cloud Native Middleware Microservices - 10 Lessons Learned
byKai Wähner
 
Internet of Things - Are traditional architectures good enough?
byGuido Schmutz
 

What's hot

PDF
Enterprise Integration Patterns Revisited (again) for the Era of Big Data, In...
byKai Wähner
 
PDF
Webinar Data Mesh - Part 3
byJeffrey T. Pollock
 
PDF
Reactive Java Robotics & IoT with Spring Reactor
byTrayan Iliev
 
PDF
Open Source IoT Project Flogo - Introduction, Overview and Architecture
byKai Wähner
 
PPTX
Modernizing your Application Architecture with Microservices
byconfluent
 
PDF
Event Mesh: The Architecture Layer That Will Power Your Digital Transformation
bySolace
 
PDF
Apache Kafka and the Data Mesh | Michael Noll, Confluent
byHostedbyConfluent
 
PPTX
Stream me to the Cloud (and back) with Confluent & MongoDB
byconfluent
 
PPTX
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
byHostedbyConfluent
 
PPTX
Kafka Summit 2021 - Why MQTT and Kafka are a match made in heaven
byDominik Obermaier
 
PPSX
Microservices, DevOps & SRE
byAraf Karsh Hamid
 
PPTX
Event Driven Architecture
bySistek Yazılım
 
PDF
Lead confluent HQ Dec 2019
bySabri Skhiri
 
PDF
Building a Real-Time Forecasting Engine with Scala and Akka
byLightbend
 
PPTX
Gartner event mesh solace - phil scanlon - gold coast
byPhil Scanlon
 
PDF
Blockchain - The Next Big Thing for Middleware
byKai Wähner
 
PPTX
Trends in Cloud and Mobile Computing - Alain Azagury, IBM
byCodemotion Tel Aviv
 
PDF
ETL Is Dead, Long-live Streams
byC4Media
 
PDF
"Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about...
byKai Wähner
 
PDF
Streaming Analytics - Comparison of Open Source Frameworks and Products
byKai Wähner
 
Enterprise Integration Patterns Revisited (again) for the Era of Big Data, In...
byKai Wähner
 
Webinar Data Mesh - Part 3
byJeffrey T. Pollock
 
Reactive Java Robotics & IoT with Spring Reactor
byTrayan Iliev
 
Open Source IoT Project Flogo - Introduction, Overview and Architecture
byKai Wähner
 
Modernizing your Application Architecture with Microservices
byconfluent
 
Event Mesh: The Architecture Layer That Will Power Your Digital Transformation
bySolace
 
Apache Kafka and the Data Mesh | Michael Noll, Confluent
byHostedbyConfluent
 
Stream me to the Cloud (and back) with Confluent & MongoDB
byconfluent
 
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
byHostedbyConfluent
 
Kafka Summit 2021 - Why MQTT and Kafka are a match made in heaven
byDominik Obermaier
 
Microservices, DevOps & SRE
byAraf Karsh Hamid
 
Event Driven Architecture
bySistek Yazılım
 
Lead confluent HQ Dec 2019
bySabri Skhiri
 
Building a Real-Time Forecasting Engine with Scala and Akka
byLightbend
 
Gartner event mesh solace - phil scanlon - gold coast
byPhil Scanlon
 
Blockchain - The Next Big Thing for Middleware
byKai Wähner
 
Trends in Cloud and Mobile Computing - Alain Azagury, IBM
byCodemotion Tel Aviv
 
ETL Is Dead, Long-live Streams
byC4Media
 
"Hadoop and Data Warehouse (DWH) – Friends, Enemies or Profiteers? What about...
byKai Wähner
 
Streaming Analytics - Comparison of Open Source Frameworks and Products
byKai Wähner
 

Similar to IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?

PDF
Internet of Things (IoT) and Big Data
byGuido Schmutz
 
PDF
IoT Architecture - are traditional architectures good enough?
byGuido Schmutz
 
PPTX
unit3-241119090144-4cgfhfxxfh7d92930.pptx
byravirajghodake98
 
PDF
Iot 3 architectures architecture of iot.
bykunal2819052023
 
PDF
Role of cloud and analytics in IoT
bySelvaraj Kesavan
 
PPTX
Successful Industrial IoT patterns
byJohn Mathon
 
PDF
Successful Industrial IoT Patterns
byWSO2
 
PPTX
Internet of Things Anatomy
byTibbo
 
PDF
Internet of Things and Big Data
bySwiss Data Forum Swiss Data Forum
 
PDF
Internet of Things (IoT)
byTrivadis
 
PDF
Internet of Things IoT Guido Schmutz
byDésirée Pfister
 
PDF
Eurotech and Red Hat collaboration simplifies Internet of Things integration ...
byEurotech
 
PDF
Azure IoT services - overview, SenZations 2015
bySenZations Summer School
 
PPTX
Fin fest 2014 - Internet of Things and APIs
byRobert Greiner
 
PPTX
Io t first(1)
byMuhammadAbduArRahman
 
PDF
Simplify Internet of Things with an Intelligent Gateway
byEurotech
 
PDF
Designing Internet of things
byMahdi Hosseini Moghaddam
 
PDF
WSO2Con USA 2017: Building Enterprise Grade IoT Architectures for Digital Tra...
byWSO2
 
PDF
IoT Overview
byAshu Joshi
 
PDF
WSO2Con EU 2015: IoT in Finance
byWSO2
 
Internet of Things (IoT) and Big Data
byGuido Schmutz
 
IoT Architecture - are traditional architectures good enough?
byGuido Schmutz
 
unit3-241119090144-4cgfhfxxfh7d92930.pptx
byravirajghodake98
 
Iot 3 architectures architecture of iot.
bykunal2819052023
 
Role of cloud and analytics in IoT
bySelvaraj Kesavan
 
Successful Industrial IoT patterns
byJohn Mathon
 
Successful Industrial IoT Patterns
byWSO2
 
Internet of Things Anatomy
byTibbo
 
Internet of Things and Big Data
bySwiss Data Forum Swiss Data Forum
 
Internet of Things (IoT)
byTrivadis
 
Internet of Things IoT Guido Schmutz
byDésirée Pfister
 
Eurotech and Red Hat collaboration simplifies Internet of Things integration ...
byEurotech
 
Azure IoT services - overview, SenZations 2015
bySenZations Summer School
 
Fin fest 2014 - Internet of Things and APIs
byRobert Greiner
 
Io t first(1)
byMuhammadAbduArRahman
 
Simplify Internet of Things with an Intelligent Gateway
byEurotech
 
Designing Internet of things
byMahdi Hosseini Moghaddam
 
WSO2Con USA 2017: Building Enterprise Grade IoT Architectures for Digital Tra...
byWSO2
 
IoT Overview
byAshu Joshi
 
WSO2Con EU 2015: IoT in Finance
byWSO2
 

More from Guido Schmutz

PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualisation
byGuido Schmutz
 
PDF
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
PDF
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
PDF
Kafka as an event store - is it good enough?
byGuido Schmutz
 
PDF
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
PDF
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Streaming Visualisation
byGuido Schmutz
 
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
Kafka as an event store - is it good enough?
byGuido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 

Recently uploaded

PDF
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
PDF
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
PDF
API206-S: Transforming Supply Chains with Amazon Bedrock AgentCore - AWS re:I...
byChris Bingham
 
PDF
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
PPTX
Basics of Identity Access Management In mordern Infrastructure
byPrinceXavier18
 
PDF
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
PDF
Agentic Automation for Testers – A Hands-On Deep Dive
byUiPathCommunity
 
PDF
The year in review - MarvelClient in 2025
bypanagenda
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
PPTX
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
PDF
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
PDF
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
PDF
AI Technology important knowledge ......
byutkarshj2007
 
PDF
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PPTX
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
API206-S: Transforming Supply Chains with Amazon Bedrock AgentCore - AWS re:I...
byChris Bingham
 
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
Basics of Identity Access Management In mordern Infrastructure
byPrinceXavier18
 
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
Agentic Automation for Testers – A Hands-On Deep Dive
byUiPathCommunity
 
The year in review - MarvelClient in 2025
bypanagenda
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
AI Technology important knowledge ......
byutkarshj2007
 
The Ultimate Guide to Problem Management Dashboards for IT Teams.pdf
byomnex systems
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
Combining Induction and Transduction for Abstract Reasoning.pptx
byallen578468
 

IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?

  • 1.
    BASEL BERN BRUGGDÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH IoT Architecture Guido Schmutz @gschmutz guidoschmutz.wordpress.com Are Traditional Architectures Good Enough or do we Need New Approaches?
  • 2.
    Guido Schmutz Working forTrivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
  • 3.
    Agenda 1. Introduction 2. Towardsan IoT Architecture 3. Summary
  • 4.
  • 5.
    Internet of Things(IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist Source: Ericsson, June 2016
  • 6.
    Reasons why IoTopportunity is occurring now ? Affordable hardware • Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware • Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility • Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools • Big data tools & cloud based infrastructure have become widely available Mass market awareness • IoT has surpassed a critical tipping point • Vision of a connected world has reached such a followership that companies have initiated IoT developments • Commitment is irreversible
  • 7.
  • 8.
    Towards an IoTArchitecture
  • 9.
    Key Challenges forbuilding an IoT application 1. Connect: How to collect data from intelligent devices? • Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? • Reduce noise and detect business event at real-time • Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? • Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices
  • 10.
    Today) Existing Service-/APIArchitecture as a base 10 Mobile Apps DB Rich (Web) Client Apps API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST / SOAP REST / SOAP SOAP Various SQL SOAP REST Service BusOracle Data Integrator API Gateway SOA Suite BPM Suite Business Activity Monitoring = one way = request/response DB
  • 11.
    REST / SOAP REST / SOAP IoT 1a) Reuseexiting Service-/API-based Architecture IoT Smart Devices 11 Mobile Apps D B Rich (Web) Client Apps D B Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST REST JMS / REST SOAP Various SQL SOAP REST WebSocket JMS Service BusOracle Data Integrator API Gateway API Gateway JMS JMS WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
  • 12.
    IoT 1a) Challenges •Are IoT devices powerful enough to communicate over the internet (HTTP protocol) ? • Device only collects data (sense) or is there also the way back necessary (actuator) ? • Can JMS be used from external devices (firewalls allow traffic over JMS) ? • How many IoT devices are planned short and long term ? • How frequently do they produce data ? • Is JMS server and ESB capable for dealing with resulting message volume ? • What operations are needed on IoT data (messages / events) ‘ • only simple transformations, filter and routing operations (VETRO pattern) ?
  • 13.
    REST / SOAP REST / SOAP IoT 1b) Reuseexisting Service-/API-based Architecture IoT Smart Devices 13 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST HTTP REST REST HTTP JMS API Gateway Service BusOracle Data Integrator SOAP Various SQL SOAP REST WebSocket JMS JMS JMS REST / SOAP WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
  • 14.
    REST / SOAP REST / SOAP IoT 2) AddingEvent Hub and optional IoT Gateway 14 Mobile Apps D B Rich (Web) Client Apps D B ESB / Data Integration IoT Devices IoT Gateways IoT Smart Devices Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST Kafka / MQTT / REST REST Kafka / MQTT / REST MQTT Kura Event Hub Kafka Service Bus Oracle Data Integrator REST REST SOAP Various SQL SOAP REST WebSocket Kafka JMS JMS API Gateway API Gateway REST Kafka REST / SOAP REST REST SOA Suite BPM Suite Business Activity Monitoring MQTT MQTT = one way = request/response
  • 15.
    How to implementan Event Hub? Apache Kafka to the rescue • Distributed publish-subscribe messaging system • Designed for processing of high-volume, real time activity stream data • Stateless (passive) architecture, offset-based consumption • Provides Topics, but does not implement JMS standard • Initially developed at LinkedIn, now part of Apache • Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound
  • 16.
    Oracle’s Service Busas a consumer of Kafka Service Bus 12c Cloud Apps Business Service Cloud Proxy Service Kafka Cloud API Mobile Apps Pipeline Routing Kafka Sensor / IoT Web Apps Business Service REST Business Service WSDL Backend Apps REST Backend Apps WSDL Proxy Service Kafka Pipeline Routing Database DB CDC Stream Processing
  • 17.
    IoT 2) Solutions& Challenges Solutions • Event Hub solves the potential scalability issue of JMS • Event Hub can interoperate / integrate with existing SOA platform (ESB) • IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges • Where to do complex analytics on the events? Is it scalable? • Can we really send all data down to backend? Network bandwidth?
  • 18.
    REST / SOAP REST / SOAP IoT 3) AddingStream Processing / Analytics in Backend 18 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response KafkaMQTT
  • 19.
    IoT 3) Solutions& Challenges Solutions • Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges • Can we really send all data down to backend? Network bandwidth?
  • 20.
    Oracle’s Stream Analyticsas consumer of Kafka/MQTT Oracle Stream Analytics Stream Analytics Kafka Kafka Mobile Apps Kafka Sensor / IoT Web Apps Machine Data DB CDC Kafka MQTT
  • 21.
    REST / SOAP REST / SOAP IoT 4) AddingIndustry 4.0 Data Sources (machine data) 21 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST KafkaMQTT
  • 22.
    GoldenGate Gateway Oracle’s GoldenGatefor Change Data Capture of existing database Machine Data DB Kafka Oracle GoldenGate Delivery Capture Pump JMS Machine Data DB Oracle GoldenGate Delivery Capture Pump HBase HDFS
  • 23.
    REST / SOAP REST / SOAP IoT 5) AddingStream Processing / Analytics at Edge 23 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka
  • 24.
    IoT 5) Solutions& Challenges Solutions • Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges • How to handle integration between the components on the IoT side (fan in)
  • 25.
    REST / SOAP REST / SOAP IoT 6) AddingData Flow Engines 25 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka NiFiNiFi
  • 26.
    Continuous Ingestion /Fan-In from the Edge DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 26 Dataflow GW Topic Topic Queue Message GW Topic Dataflow GW Dataflow TopicREST 26 File Source Log Log Log Social Native Topic Topic
  • 27.
    Challenges for IngestingSensor Data • Multitude of sensors • Multiple Firmware versions • Bad Data from damaged sensors • Data Quality
  • 28.
    IoT 6) Solutions& Challenges Solutions • Data Flow engines allow for passing information onwards, potentially over multiple hops (fan-in) Challenges • What if we want to keep the raw data? Where to persist it?
  • 29.
    REST / SOAP REST / SOAP IoT 7) AddingRaw Data Storage and Batch Analytics 29 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
  • 30.
    IoT 7) Solutions& Challenges Solutions • Adding Big Data platform allows to store all raw data in a distributed file system in a scalable and reliable manner Challenges • How can we leverage the Big Data platform for more than just storing raw data? • How does it integrate with the stream processing?
  • 31.
    REST / SOAP REST / SOAP IoT 7a) AddingData Mining / Machine Learning and Model execution 31 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
  • 32.
    REST / SOAP REST / SOAP Oracle Cloud Servicesand Internet of Things / Big Data 32 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices Gateway IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB = one way = request/response Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI Gateway Kafka / REST REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processin g DB Kafka Kafka HDFS MQTT Hadoop / Spark Oracle IoT Cloud Service Oracle Big Data Cloud Service Oracle BI Cloud Service Oracle Integration Cloud Service Oracle Messaging Cloud Service Oracle Process Cloud Service Oracle SOA Cloud Service Oracle Mobile Cloud Service ESP/CEP Edge Analytics Oracle IoT CS Gateway Oracle IoT CS Client Library = one way = request/response
  • 33.
  • 34.
    Summary Treat events asevents! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary? • What is the network bandwidth between device/gateway and cloud/backend? • Centralized or Decentralized IoT solution?
  • 35.
  • 36.
    Q & A GuidoSchmutz Technology Manager guido.schmutz@trivadis.com
  • 37.
    IoT Services IoT LogicalReference Architecture IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Internet of Things (IoT) and Big Data37
  • 38.
    IoT Services IoT LogicalReference Architecture – Oracle on premises IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Business Activity Monitoring SOA Suite BPM Suite Service Bus Oracle Data Integrator Stream Analytics Big Data Appliance Stream Analytics Service Bus API Gateway Internet of Things (IoT) and Big Data38 Oracle IoT CS Gateway Oracle IoT CS Client Library
  • 39.
    IoT Services IoT LogicalReference Architecture – Oracle Cloud Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Oracle BI CS Oracle Big Data CS Oracle SOA CS Oracle Integration CS Oracle IoT CS Oracle Streaming Analytics CS Oracle Messaging CS Oracle Big Data Discovery CS Oracle Mobile CS Internet of Things (IoT) and Big Data39 Oracle IoT CS Gateway Oracle IoT CS Client Library Oracle Process CS Oracle DataFlow ML CS Big Data Preparation CS Application Container CS Container CS
  • 40.
    IoT Services IoT LogicalReference Architecture – Microsoft Azure IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Azure IoTHub Azure Streaming Analytics Azure Service Bus Azure Power BI HD Insight Azure Storage Azure IoT Gateway SDK Azure Event Hub Azure Storage Azure Machine Learning Internet of Things (IoT) and Big Data40
  • 41.
    IoT Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event HubEvent Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired IoT Logical Architecture – Open Source Internet of Things (IoT) and Big Data41