IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?

431 views

Published on

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
431
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
28
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

IoT Architecture - Are Traditional Architectures Good Enough or do we Need New Approaches?

  1. 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH IoT Architecture Guido Schmutz @gschmutz guidoschmutz.wordpress.com Are Traditional Architectures Good Enough or do we Need New Approaches?
  2. 2. Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
  3. 3. Agenda 1. Introduction 2. Towards an IoT Architecture 3. Summary
  4. 4. Introduction
  5. 5. Internet of Things (IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist Source: Ericsson, June 2016
  6. 6. Reasons why IoT opportunity is occurring now ? Affordable hardware • Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware • Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility • Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools • Big data tools & cloud based infrastructure have become widely available Mass market awareness • IoT has surpassed a critical tipping point • Vision of a connected world has reached such a followership that companies have initiated IoT developments • Commitment is irreversible
  7. 7. The Sensing-as-a-Service Model
  8. 8. Towards an IoT Architecture
  9. 9. Key Challenges for building an IoT application 1. Connect: How to collect data from intelligent devices? • Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? • Reduce noise and detect business event at real-time • Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? • Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices
  10. 10. Today) Existing Service-/API Architecture as a base 10 Mobile Apps DB Rich (Web) Client Apps API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST / SOAP REST / SOAP SOAP Various SQL SOAP REST Service BusOracle Data Integrator API Gateway SOA Suite BPM Suite Business Activity Monitoring = one way = request/response DB
  11. 11. REST / SOAP REST / SOAP IoT 1a) Reuse exiting Service-/API-based Architecture IoT Smart Devices 11 Mobile Apps D B Rich (Web) Client Apps D B Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST REST JMS / REST SOAP Various SQL SOAP REST WebSocket JMS Service BusOracle Data Integrator API Gateway API Gateway JMS JMS WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
  12. 12. IoT 1a) Challenges • Are IoT devices powerful enough to communicate over the internet (HTTP protocol) ? • Device only collects data (sense) or is there also the way back necessary (actuator) ? • Can JMS be used from external devices (firewalls allow traffic over JMS) ? • How many IoT devices are planned short and long term ? • How frequently do they produce data ? • Is JMS server and ESB capable for dealing with resulting message volume ? • What operations are needed on IoT data (messages / events) ‘ • only simple transformations, filter and routing operations (VETRO pattern) ?
  13. 13. REST / SOAP REST / SOAP IoT 1b) Reuse existing Service-/API-based Architecture IoT Smart Devices 13 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST HTTP REST REST HTTP JMS API Gateway Service BusOracle Data Integrator SOAP Various SQL SOAP REST WebSocket JMS JMS JMS REST / SOAP WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring = one way = request/response
  14. 14. REST / SOAP REST / SOAP IoT 2) Adding Event Hub and optional IoT Gateway 14 Mobile Apps D B Rich (Web) Client Apps D B ESB / Data Integration IoT Devices IoT Gateways IoT Smart Devices Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST Kafka / MQTT / REST REST Kafka / MQTT / REST MQTT Kura Event Hub Kafka Service Bus Oracle Data Integrator REST REST SOAP Various SQL SOAP REST WebSocket Kafka JMS JMS API Gateway API Gateway REST Kafka REST / SOAP REST REST SOA Suite BPM Suite Business Activity Monitoring MQTT MQTT = one way = request/response
  15. 15. How to implement an Event Hub? Apache Kafka to the rescue • Distributed publish-subscribe messaging system • Designed for processing of high-volume, real time activity stream data • Stateless (passive) architecture, offset-based consumption • Provides Topics, but does not implement JMS standard • Initially developed at LinkedIn, now part of Apache • Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound
  16. 16. Oracle’s Service Bus as a consumer of Kafka Service Bus 12c Cloud Apps Business Service Cloud Proxy Service Kafka Cloud API Mobile Apps Pipeline Routing Kafka Sensor / IoT Web Apps Business Service REST Business Service WSDL Backend Apps REST Backend Apps WSDL Proxy Service Kafka Pipeline Routing Database DB CDC Stream Processing
  17. 17. IoT 2) Solutions & Challenges Solutions • Event Hub solves the potential scalability issue of JMS • Event Hub can interoperate / integrate with existing SOA platform (ESB) • IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges • Where to do complex analytics on the events? Is it scalable? • Can we really send all data down to backend? Network bandwidth?
  18. 18. REST / SOAP REST / SOAP IoT 3) Adding Stream Processing / Analytics in Backend 18 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response KafkaMQTT
  19. 19. IoT 3) Solutions & Challenges Solutions • Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges • Can we really send all data down to backend? Network bandwidth?
  20. 20. Oracle’s Stream Analytics as consumer of Kafka/MQTT Oracle Stream Analytics Stream Analytics Kafka Kafka Mobile Apps Kafka Sensor / IoT Web Apps Machine Data DB CDC Kafka MQTT
  21. 21. REST / SOAP REST / SOAP IoT 4) Adding Industry 4.0 Data Sources (machine data) 21 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST KafkaMQTT
  22. 22. GoldenGate Gateway Oracle’s GoldenGate for Change Data Capture of existing database Machine Data DB Kafka Oracle GoldenGate Delivery Capture Pump JMS Machine Data DB Oracle GoldenGate Delivery Capture Pump HBase HDFS
  23. 23. REST / SOAP REST / SOAP IoT 5) Adding Stream Processing / Analytics at Edge 23 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka
  24. 24. IoT 5) Solutions & Challenges Solutions • Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges • How to handle integration between the components on the IoT side (fan in)
  25. 25. REST / SOAP REST / SOAP IoT 6) Adding Data Flow Engines 25 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka NiFiNiFi
  26. 26. Continuous Ingestion / Fan-In from the Edge DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 26 Dataflow GW Topic Topic Queue Message GW Topic Dataflow GW Dataflow TopicREST 26 File Source Log Log Log Social Native Topic Topic
  27. 27. Challenges for Ingesting Sensor Data • Multitude of sensors • Multiple Firmware versions • Bad Data from damaged sensors • Data Quality
  28. 28. IoT 6) Solutions & Challenges Solutions • Data Flow engines allow for passing information onwards, potentially over multiple hops (fan-in) Challenges • What if we want to keep the raw data? Where to persist it?
  29. 29. REST / SOAP REST / SOAP IoT 7) Adding Raw Data Storage and Batch Analytics 29 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
  30. 30. IoT 7) Solutions & Challenges Solutions • Adding Big Data platform allows to store all raw data in a distributed file system in a scalable and reliable manner Challenges • How can we leverage the Big Data platform for more than just storing raw data? • How does it integrate with the stream processing?
  31. 31. REST / SOAP REST / SOAP IoT 7a) Adding Data Mining / Machine Learning and Model execution 31 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka NiFiNiFi
  32. 32. REST / SOAP REST / SOAP Oracle Cloud Services and Internet of Things / Big Data 32 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices Gateway IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB = one way = request/response Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI Gateway Kafka / REST REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processin g DB Kafka Kafka HDFS MQTT Hadoop / Spark Oracle IoT Cloud Service Oracle Big Data Cloud Service Oracle BI Cloud Service Oracle Integration Cloud Service Oracle Messaging Cloud Service Oracle Process Cloud Service Oracle SOA Cloud Service Oracle Mobile Cloud Service ESP/CEP Edge Analytics Oracle IoT CS Gateway Oracle IoT CS Client Library = one way = request/response
  33. 33. Summary
  34. 34. Summary Treat events as events! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary? • What is the network bandwidth between device/gateway and cloud/backend? • Centralized or Decentralized IoT solution?
  35. 35. J
  36. 36. Q & A Guido Schmutz Technology Manager guido.schmutz@trivadis.com
  37. 37. IoT Services IoT Logical Reference Architecture IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Internet of Things (IoT) and Big Data37
  38. 38. IoT Services IoT Logical Reference Architecture – Oracle on premises IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Business Activity Monitoring SOA Suite BPM Suite Service Bus Oracle Data Integrator Stream Analytics Big Data Appliance Stream Analytics Service Bus API Gateway Internet of Things (IoT) and Big Data38 Oracle IoT CS Gateway Oracle IoT CS Client Library
  39. 39. IoT Services IoT Logical Reference Architecture – Oracle Cloud Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Oracle BI CS Oracle Big Data CS Oracle SOA CS Oracle Integration CS Oracle IoT CS Oracle Streaming Analytics CS Oracle Messaging CS Oracle Big Data Discovery CS Oracle Mobile CS Internet of Things (IoT) and Big Data39 Oracle IoT CS Gateway Oracle IoT CS Client Library Oracle Process CS Oracle DataFlow ML CS Big Data Preparation CS Application Container CS Container CS
  40. 40. IoT Services IoT Logical Reference Architecture – Microsoft Azure IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Azure IoTHub Azure Streaming Analytics Azure Service Bus Azure Power BI HD Insight Azure Storage Azure IoT Gateway SDK Azure Event Hub Azure Storage Azure Machine Learning Internet of Things (IoT) and Big Data40
  41. 41. IoT Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired IoT Logical Architecture – Open Source Internet of Things (IoT) and Big Data41

×