Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Architecting Azure (I)IoT Solutions @ IoT Saturday 2019

138 views

Published on

Disoriented about all the Azure services in the IoT and Industrial IoT that you can use for building a modern Architecture on the Cloud and on the Edge? Well, this session aims to describe a reference architecture like Lambda and to map it to Azure services like Event Hubs, IoT Hubs just to mention a few. It also presents different approaches on how to handle communication from a more commercial devices to discrete manufacturing ones, with different standards like OPC UA. All those bricks will also help you to use already-build solutions like our Accelerators and IoT Central.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Architecting Azure (I)IoT Solutions @ IoT Saturday 2019

  1. 1. #iotsatpn saturday 2019 Internet of Things Architecting Azure (I)IoT solutions Pietro Brambati, Microsoft Twitter:@pietrobr pietrobr@microsoft.com
  2. 2. #iotsatpn Azure IoT portfolio Device SDKs Azure Sphere Windows IoT Data Box Edge Certified Devices IoT Edge Stream Analytics Azure ML Azure SQL Functions Cognitive Services IoT Hub Event Hub IoT Hub Device Provisioning Service Time Series Insight (TSI) Maps Stream Analytics CosmosDB Logic Apps Cognitive / AI Web Apps PowerBI Azure IoT Solution Accelerators Azure IoT Central AzureSecurityCenterforIoT App Insight Blob Service Bus
  3. 3. #iotsatpn IoT Logical Architecture IoT DeviceIoT DeviceIoT Device IoT DeviceIoT Device IoT Edge Device Cloud Gateway Stream Processing Data Transformation Warm Store Cold Store UI & Reporting User Management Business Integration Machine Learning Fast path – real time processing Slow path – Batch processing
  4. 4. #iotsatpn Azure IoT Architecture – simplified view IoT DeviceIoT DeviceIoT Device IoT DeviceIoT Device IoT Edge Device Stream Processing UI & Reporting Machine Learning Fast path – real time processing Slow path – Batch processing Cloud Gateway IoT Hub Data Transformation Functions Warm CosmosDB Cold Blob Azure Stream Analytics Power BI Business Integration Logic Apps User Management Azure Active Directory Machine Learning
  5. 5. #iotsatpn Lambda Architecture A generic, scalable, robust and fault-tolerant data processing architecture New data Storage (HDFS) Real-time view Batch layer Batch view Batch view Query Real-time view Pros Increase throghput Reduce latency Negligible errors Useful with event sourcing Serving layer Speed layer Cons Different tecnologies No easy to mantain Process (Map Reduce) Process (Spark/Storm)
  6. 6. #iotsatpn Kappa Architecture Real-time processing of live streaming data - an alternative to Lambda, not a replacement Pros Focus on speed Less code resource Fewer resources: ML done in real-time Re-processing only if code changes Cons More errors, no batch layer New data Real-time view Query Real-time view Serving layer Process (Spark/Storm) Speed layer
  7. 7. #iotsatpn Slow path – batch processing Fast path – real-time processing Azure IoT Logical Architecture – Lambda Recommended data flow Cloud GatewayIoT DeviceIoT DeviceIoT Device Route Analyze (low latency) Store Display Action Analyze (complex) Action Store Display Partitioning Message props.
  8. 8. #iotsatpn Azure IoT Physical Architecture – Lambda Stateless and Static rules, no reference to external data Scheduled batch processing (training , Machile Learning) Real-time device telemetry (visualization) IoT Hub IoT DeviceIoT DeviceIoT Device IoT Hub Route IoT Hub Route Event Hub Functions Logic App IoT Hub Route Blob/ADL (Cold) Azure ML CosmosDB Real-time analytics (Alert, Actions) Event Hub Functions When to use: Processing rule with no history Records in JSON Small number of rules required Define conditions in payload of IoT Hub Route
  9. 9. #iotsatpn Azure IoT Physical Architecture – Lambda Statefull and Dynamic rules, ability to reference external data Scheduled batch processing (training , Machile Learning) Real-time device telemetry (visualization) IoT Hub IoT DeviceIoT DeviceIoT Device IoT Hub Route Event Hub Functions Logic App IoT Hub Route Storage (Cold) Azure ML CosmosDB Real-time analytics (Alert, Actions) Azure Stream Analytics FunctionsWhen to use: Complex analysis: time windows, join with external data source Processing logic consists of severval rules that can grow Input data in binary format (like Avro)
  10. 10. #iotsatpn Ops….
  11. 11. #iotsatpn IoT and Industrial IoT (IIoT) Industrial ready solutions IoT Consumer apps and device Good Internet connectivity Well know set of protocols «Close enough» for good latency Usually no real-time requirements Simple application with low-risk impact Industrial IoT Industrial apps (agricaulture and manifacturing, etc.) Low internet connectivity A «junglue» of communication protocols lack of interoperability and legacy technologies: SCADA, M2M, etc Security is a major complain (e.g take down an electrical grid) Must integrate with ERP Precision is business as usual Low latency for near-real-time operations Reliability and Resilence: 20-30 years, e.g. loss of sensors, loss of connectivity
  12. 12. #iotsatpn Edge Connectivity IP-based communication Device IoT Client Device Device Device IoT Client Device Device Field gateway IoT Client Cloud Gateway (IoT Hub) AMQP, MQTT, HTTPS AMQP, MQTT, HTTPS Custom Cloud Gateway Custom protocols Field gateway OPC, HTTP, CoAP VPN/ExR CoAP,AllJoyn, OPC CoAP,AllJoyn, OPC
  13. 13. #iotsatpn OPC Unified Architecture(UA) Local network communication OPC «Classic» Object Linking and Embedding for Process Control, first defined in 1995. A Client/Server based communication architecture Evolved over the years adding more functionalities (protocols): Data Access, Alarm & Events, etc.. OPC Unified Application Secure, platform-independent, SOA, in 2008 OPC-UA gateway read from OPC-UA server via Publish/Subscribe (JSON payload over MQTT or AMQP)
  14. 14. #iotsatpn Field Gateway (IoT Edge) OPC UA with Microsoft Technologies Microsoft invests heavily in OPC-UA by providing several solutions IoT DeviceIoT Device Industrial Device OPC UA Server OPC Publisher «Container» Cloud Gateway OPC Client OPC Server Cert. Store Telemetry Data Flow Scenario OCP-UA Publish/Subscribe
  15. 15. #iotsatpn MIRAI
  16. 16. #iotsatpn Security Connected device – some consideration Inexpensive device trading away cryptographic capability They should be constrained on local network used by field gateway Use device that support cryptographic capabilities Encryption algorithms Digital signature TLS 1.2, DTLS 1.2 Updatable key-store per device and firmware must allow updates Choose MCU with TPM Secure boot loader Sensor to detect intrusion attemps Use a relay service to remote connect to a device No direct SSH to it
  17. 17. #iotsatpn IoT Hub Ingestion to the cloud Event processing Device provisioning and management Device business logic, connectivity monitoring
  18. 18. #iotsatpn IoT Device Provisioning Service (DPS) Zero touch, just in time device provisioning to the right IoT Hub No hard-coding IoT Hub into device at the factory Load balancing Lowest latency Connect to the right IoT Hub in Mutitenancy scenario Roll Certificate (e.g in case of a breach) Reprovisioning IoT DeviceIoT Device IoT Edge Device Device Provisioning Service (DPS) Enrollment List Cloud Gateway (IoT Hub) X509, TPM, Sym. Key 1 2 3 4 5 6 7-8
  19. 19. #iotsatpn Event Hub Architecture- event ingestor at high-scale HTTPS AMQP Kafka Consumer group Consumer group Event Producers MB/s → GB/s Event Receivers Each consumer app its own cursor/offset Durable: 1-7 days for Standard, 90 days Dedicated
  20. 20. #iotsatpn Event Hub Scenario : Clean Event Hubs
  21. 21. #iotsatpn Event Hub Scenario : Checkpoints It doesn’t start from the beginning
  22. 22. #iotsatpn Azure Stream Analytics event-processing engine IoT Hubs Archiving for long term storage/ batch analytics Real-time dashboard Stream Analytics Automation to kick-off workflowsMachine LearningReference Data (Blobs and SQL DB) Event Hubs Blobs Devices & Gateways Power BI Applications
  23. 23. #iotsatpn Azure Stream Analytics event-processing engine Stream UnitInput Stream output Stream stream From With Into Stream UnitInput Stream output Stream From With IntoJoin
  24. 24. #iotsatpn SAQL Query Language Scaling Extensions WITH PARTITION BY OVER Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP String Functions Len Concat CharIndex Substring PatIndex Temporal Functions Lag, IsFirst CollectTop DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER/LEFT OUTER JOIN UNION CROSS/OUTER APPLY CAST INTO ORDER BY ASC, DSC Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd
  25. 25. #iotsatpn Azure Stream Analytics Tumbling Window Tell me the count of tweets per time zone every 10 seconds 1 5 4 26 8 6 5 Time (secs) 1 5 4 26 8 6 A 10-second Tumbling Window 3 6 1 5 3 6 1 1 5 4 26 1 SELECT TimeZone, COUNT(*) AS Count FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY TimeZone, TumblingWindow(second,10)
  26. 26. #iotsatpn Azure Stream Analytics Hoping Window SELECT Topic, COUNT(*) AS TotalTweets, AVG(SentimentScore) FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY Topic, HoppingWindow(second, 10 , 5) Every 5 seconds give me the count of tweets and the average sentiment score over the last 10 seconds 1 5 4 26 8 7 A 10-second Hopping Window with a 5-second “Hop” 4 26 8 6 5 3 6 1 1 5 4 26 8 6 5 3 6 15 3
  27. 27. #iotsatpn Azure Stream Analytics Sliding Window SELECT Topic, COUNT(*) FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY Topic, SlidingWindow(second, 10) HAVING COUNT(*) > 10 Give me the count of tweets for all topics which are tweeted more than 10 times in the last 10 seconds 1 5 A 10-second Sliding Window 8 8 51 9 51 9 1
  28. 28. #iotsatpn Azure Stream Analytics Session Window Tell me the count of tweets that occur within 5 minutes each other 1 5 A 10-second Sliding Window 9 5 76 8 1 6 7 8 9
  29. 29. #iotsatpn Event Grid event routing service for publish and subscribe model Retry mechanism Filter to route events to different endpoints single endpoint for SaaS apps. Fine- grained auth with AAD Dead-letter Events != Messages
  30. 30. #iotsatpn Time Series Insights Analytics for Industrial IoT Analytics, storage and visualization service for time series data Operation Analysis for both warm and cold data Storage in-memory and SSD Easy connection with IoT Hub SQL like filtering and aggregation Data explorer dashboard and API, JavaScript library Latency 30-60 seconds
  31. 31. #iotsatpn The Edge
  32. 32. #iotsatpn Why Azure IoT Edge?
  33. 33. #iotsatpn IoT Edge runtime Installs and updates workloads on the device. Maintains Azure IoT Edge security standards on the device. Ensures that IoT Edge modules are always running. Reports module health to the cloud for remote monitoring. Facilitates communication between downstream leaf devices and the IoT Edge device. Facilitates communication between modules on the IoT Edge device. Facilitates communication between the IoT Edge device and the cloud
  34. 34. #iotsatpn IoT Edge runtime Standard deployment model Azure IoT Hub Azure IoT Edge Device IoT Edge Runtime
  35. 35. #iotsatpn IoT Edge runtime Kubernetes integration Kubernetes Cluster Devices Azure IoT Hub IoT Edge Runtime Kubernetes IoT Edge Runtime Kubernetes IoT Edge Runtime Kubernetes
  36. 36. #iotsatpn
  37. 37. #iotsatpn 9 BILLION new MCU devices built and deployed every year Microcontrollers (MCUs) low-cost, single chip computers
  38. 38. #iotsatpn Fewer than 1% of MCUs are connected today.
  39. 39. #iotsatpn Azure Sphere
  40. 40. #iotsatpn Solution accelerators
  41. 41. #iotsatpn Azure IoT solution accelerators Dashboards, visualization and insights Completely customizable Command and control Device connectivity and management Predictive Maintenance Connected FactoryRemote Monitoring Workflow automation and integration Preconfigured solutions End-to-end implementation Device Simulation Open-source microservices based architecture
  42. 42. #iotsatpn
  43. 43. #iotsatpn
  44. 44. #iotsatpn
  45. 45. #iotsatpn Azure IoT central Analytics, dashboards and visualization Extensibilities (Flow, Dynamics, Webhooks, etc.) Monitoring rules and triggered actions Fully hosted and managed by Microsoft Device connectivity and management Risk-free trial with simplified pricing No cloud development expertise required
  46. 46. #iotsatpn Summary IoT projects are complex, but… Azure Reference Architecture + PaaS Service + Accelerators + IoT edge + Edge components = make them simpler!
  47. 47. #iotsatpn Grazie e buona continuazione Pietro Brambati, Microsoft
  48. 48. #iotsatpn Resources • Microsoft Azure IoT Reference Architecture pdf • IoT PaaS Services • Lambda architecture • Lamba vs K architecture • OPC UA Publisher • Connected Factory solution accelerator walkthrough

×