IIoT leverages the power of machines and realtime analytics to pick up on industrial inefficiencies and problems sooner, and save time and money in addition to supporting BI efforts. In a myriad of reference architectures it is up to experience and trial-error to find out what really works in a real life scenario.. We will review the challenges and solutions in building an IIoT platform from the ground up on the edge between Azure and open source in order to have the best from both worlds. Technical focus will be on IoT Edge, TS Insights, Stream Analytics, IoT Hub, App Insights, Event Grid, Service Bus, ARM templates, Influx DB, Grafana and more - all neatly glued together by Azure Functions.

1. September 14
Industrial IoT from the Ground up
with Azure and Open Source
Industry 4.0 MDC system with advanced
analytics in the cloud

2. About me
• Software Architect @
o 17+ years professional experience
• Microsoft Azure MVP
• External Expert Horizon 2020
• External Expert Eurostars-Eureka, InnoFund Denmark
• Business Interests
o Web Development, SOA, Integration
o IoT, Machine Learning, Computer Intelligence
o Security & Performance Optimization
• Contact
ivelin.andreev@icb.bg
www.linkedin.com/in/ivelin
www.slideshare.net/ivoandreev

3. Thanks to our Sponsors
With the support of:

4. Agenda
• Do your Homework
• Architecture
• Time Series DB, IoT Edge, Functions
• CDM & Power BI
• Event Grid, Service Bus, Notifications
• Deployment, API Management
• Monitoring, Cost & Security
• Demo

5. Azure
IoT Central
• Features
• Fully managed SaaS solution
• Low skill levels required
• Simple predictable pricing
(€1.7 device, €4.3 1M msg)
• Drawbacks
• Straightforward IoT solutions
w/o deep customization
• Infrastructure not accessible
• Basic business analytics
• Data retention (30d)
• Integration (machines, LoB systems)
• Prerequisite: IoT Hub SDK used by the device
• IoT Device Bridge support for HTTP ingestion (i.e. other clouds)

6. IoT Solution
Accelerators
• Features
• Enterprise-grade PaaS
• For custom IoT and
maximum flexibility
• Access underlying services
• Drawbacks
• High skills required to
customize
• Business analytics still to be
developed
• External integration (OPCUA)
• Not maintained actively
*Supported scenarios:
• Remote Monitoring
• Connected Factory
• Predictive Maintenance
*Supported deployment:
• Standard
• Basic
• Local

7. Platform
Overview
• Features
• End-end solution
• Azure PaaS (whenever justified)
• Open Source components
• Cost-optimized
• ARM deployment & config.
• Individual AZ subscription
• Drawbacks
• Trial & error (3y) €xp€ns€s
• “Early adopter tax” hiccups
• Microsoft move fast

8. IoT is Synonymous with Time Series
• Writes
o 95%-99% of all operations
o Streaming live data from multiple devices
o Typically sequential appends
• Updates to modify values are rare
• Deletes are bulk on large ranges (days, months, years)
• Queries
o Typically sequential
o Concurrent reads are common
• Performance issues are typically I/O bound
o Caching does not work well for BigData
o Systems are typically distributed by design
Credits: Baron Schwartz

9. Azure Time Series Insights (GA)
What is
o Managed cloud service for TS data storage, analysis and visualization (TS DB as a Service)
o Scalable – V-Scale (S1, S2), H-Scale (up to 10 units)
o GA (Nov 2017)
Features
o Schema-less JSON
o Ingest, Store, Explore, Visualize, Analyze events
o Out of the box IoT Hub/Event Hub integration
o TSI Explorer
Issues
o Retention limited to 400 days
o Pricing (per unit) from 126 EUR/month (30M events)
o Deletion support (only through retention policy)
o Few TS data aggregation functions

10. Azure Time Series Insights (Preview)
What is
o Comprehensive end-to-end PaaS for IoT data exploration and analyzing IoT-scale time series
Feature Improvements
o Infinite Retention
• User storage with Apache Parquet files
o Cost optimized TS data store
• Price - € 25 / Unit / month (100GB)
• Queries – €0.009 / GB scanned
o JavaScript SDK ( controls), Service APIs
o Improved TSI Preview Explorer
o Integration
Hand-on Demo (Contoso Wind Farm )
https://insights.timeseries.azure.com/preview/demo

11. Mongo DB for TS Workloads
What is
o #1 document DB (NoSQL), Open source
o Promoted as suitable for TS workloads
Features
o On-prem H-Scalability (management is a )
o SaaS as Azure Cosmos DB
o Aggregation Framework - preparation, transformations, and analysis
TS Scenario Issues
o Non-trivial TS scenario setup and DB design
o Initial design is not flexible and may be limiting
o Poor ad-hoc aggregation (preaggregation in subdocuments for each level)
o Price (Cosmos DB) – from 400RUs
• https://cosmos.azure.com/capacitycalculator/
o Performance: 2.4x slower (Write), 20x larger (Storage), 5.7x slower (Read)

12. What is
o Top #1 TS DB, Open core model
o Influx DB 2.0 Alpha on its way (with Flux)
Features
o Easy setup (5 min), no external dependencies
o Cross-platform (Linux, Windows, OS X)
o .NET, Java, JS, R, PHP, Python, Ruby, Go, Node.js
o SQL-like syntax, REST API
o Up to 10M series/node, up to 1M Writes/sec
o Built-in TimeSeries functions
Issues
o H-scale not supported on open source
o No fine-grain auth on open source (requires Cloud or
Enterprise)
o Tag update requires datapoint deletion
o No TS Context (as in OSISoft PI event frames)

13. VisualizationVisualization
Platform
• Purpose
• Real-time dashboards
• KPI dashboards
• Management reports
• System health
• Expectations
• Charting framework
• Flexibile configuration
• Open & extensible
• Data sources
• Ownership cost

14. What is
o Web-based analytical and visualization, highly popular, open source (excl. Cloud & Enterprise)
Features
o Easy to install, Fast, GUI (Angular, React)
o Plugin-based (59 official, not all are useful)
o Multiple data sources (68 official)
o Alerts and notifications
o Customizable - Variables, Templates, Settings, Playlists
Issues
o Very few UI themes (looks industrial)
o Authorization (filter not implicitly added to DB query)
o Read-only plugins
o Secure DB interaction requires R/O user

15. Azure IoT Edge
IoT Edge v.2 Evolution (May 2018)
o Edge Hub - lightweight local IoT Hub + Broker
o Edge Agent - run modules
o Edge Security Manager – security and integrity
o Modules (C#, Python, C, Node.js, Java)
• Containerized (easy deployment)
• Deployed remotely from Azure IoT Hub
• Configured remotely (module twins)
IoT Edge v.1 (Build Conference – May 10, 2017)
o Inspired by industrial IoT
o Azure workload to the edge ( , , )
o Low latency, 2-way communication
o Open source (in C), X-platform (Windows, Linux)
Architecture
o Broker – declarative message routing
o Modules – unit of execution (log, filter, analysis)
o Messages – flow from module to module

16. Azure IoT Edge Hiccups
Challenge
o Some industrial control libraries require COM, full .NET and Windows OS (i.e. Fanuc Robot Interface, Heidenhain)
IoT Edge Runtime Issues
1. Windows containers run on Windows only (Windows 10 IoT Enterprise, build 17763)
2. Docker for Windows allows simulating Linux containers (not vice-versa)
3. Linux containers on Windows not supported for production Azure IoT Edge
4. Large Full .NET 4.7 runtime image 7.5GB (IoT edge download timeout)
5. Docker for Windows didn’t run on target environment
Workaround
o Primary GW: IoT Edge on Linux with Linux containers
o Secondary GW: Windows service + IoT Hub device SDK

17. IoT Edge on Linux didn’t go Smooth too
“Early adopters tax” in action
o EdgeHub: Disconnecting module with exception. Restart helps for ~20min
• Symptom: “[WRN] Error sending messages to module ”
• Ref: https://github.com/Azure/iotedge/issues/673
• Fix: Update edge runtime (Agent, Hub) (1.0->1.0.6); client SDK (1.18.1->1.20)
o EdgeAgent: Module is kept in the failed state by the edgeAgent
• Symptom: “[WRN] - Reconcile failed because of the an exception System.ObjectDisposedException”
• Ref: https://github.com/Azure/azure-iot-sdk-csharp/issues/953
• Ref: https://github.com/Azure/iotedge/pull/1329
• Ref: https://github.com/Azure/iotedge/issues/1336
• Fix: Update edge runtime (Agent, Hub) (1.0.6->1.0.8);

18. Infrastructure as Code (IaC)
What is
o Declaratively managing and provision infrastructure instead of via manual config. tools
Benefits
o Behaviour consistency (no hardcoded configurations and dependencies)
o Efficiency, repeatable deployment results
o DevOps Continuous Delivery precondition
How
o Azure Resource Manager (ARM) templates for resources
o PowerShell for AAD objects (i.e. Applications)
o Per customer subscriptions minimize security issues
o Easy cost distribution
Downside
o Template maintenance is time consuming
o More expensive per user, resources not fully utilized

19. Azure Functions
What is
o Serverless compute service without managed infrastructure
o The glue among components (Azure service and open source)
o 2 Runtime versions (1.x - .NET 4.7, 2.x - .NET Core 2.2 and other)
Features
o Simplicity, Scalability
• Consumption plan – per execution (€0.17 per 1M, 1M free), scale on load
• App Service plan – pricey, dedicated VMs, unlimited timeout, always on
o Azure Stream Analytics alternative for async processing scalable units
o Trigger-based integration (HTTP, Timer, Storage Queue, EventGrid, EventHub, ServiceBus)
Issues
o Runtime 1.x has compatibility issues - .NET, EF, Nuget (i.e.Newtonsoft Json)
o Long running calls, Stateless, Idempotent

20. Common Data Model (CDM) Concept
What is
o Concept for a standardized data model for a domain
• i.e. https://microsoft.github.io/CDM/
o Data structure to import data in
Benefits
o Simplified integration and interpretation of data, Faster analytics
o Structural and semantic consistency across applications
Challenges
o Obtain sufficient knowledge of similar systems to design the CDM
• Enterprise Resource Planning (ERP)
• Advance Planning and Scheduling (APS)
• Condition-based Maintenance Management Systems (CMMS)
o Develop IoT Edge module to import on premises source
• i.e. IoT Edge Module with SQL change tracking (event based)

21. Power BI
What is
o Powerful business analytics solution with self-service and AI capabilities
PBI Portfolio
o On-Premises: Desktop, Mobile, Report Server
o Service: Free, Pro, Premium(EM/P SKU), Embedded (A SKU)
Our Approach
o Data mart (Azure SQL DB) tabular OLAP
o ETL with Report analyzer
o Staging DB (optional)
o Direct Query DataSource
o Interactive PBI reports
Challenges
o On-demand A SKUs for premium features
o No support for Influx DB DataSource
A SKU EM SKU P SKU
Purchase Azure O365 O365
Sharing
Use Case
Embedded rep. Embedded rep.
SharePoint
MS Teams
Embedded rep.
SharePoint
MS Teams
PBI Apps
Billing Hourly Monthly Monthly
Commitment No Year/Month Year/Month
PBI Service
User
Licensing
PBI Free PBI Pro
Capacity
Licensing
P SKU EM SKU A SKU

22. APIs, GraphQL, Swagger, API Management Service
What is
o Individually scalable services
o Support external integration
Azure API Management Service (GA May 2019)
o Abstract API implementation and host under the same URL
o Implement insights, analytics, security and test APIs
Configuration
o Design and Document API with Swagger open framework (Editor, UI and Codegen)
o Import API from OpenAPI URL (i.e. ([ApiUrl]/swagger/v1/swagger.json)
o Organize APIs in product groups
o Define inbound policy (50+ types)
• Transform, cache and protect to hide implementation
• Restrictions and limit policy (throttling)
Pricing – Consumption tier – 1M calls free, €0.03 / 1000 req.

23. Alerts & Notifications
• What is
o Expression-based notifications for events of interest
• Why
o Allow operators to leave workplace and monitor remotely
• How
1. Persister
• Stores telemetry to TS DB
• Checks event definitions, sends on Service Bus topic
2. Event Processor
• Generates event
• Checks notification definitions, sends on Service Bus topic
3. Logic Apps workflow from Service Bus trigger
4. Notification Server
• Identify notification context and send via channel
• Email (SendGrid), SMS (Twilio), Push (IFTTT, Viber)
5. Event Grid subscription for notification config. Changes

24. Message Channel Highlights
TWILIO
• Programmable SMS
o Caller IDs, Local phone numbers
o €0.056 W.Europe, €0.072 Bulgaria
• Other Features
o MMS, In-App chat SDK, preferred user device
o Azure Logic App Web Hook-Twilio integration
• Pay-as-you-go
IF THIS-THAN-THAT
• Applets
o Free web service for chain of conditional statements
• Integration
o Trigger endpoint for last 50 events, checked on 15 min
o Realtime API to notify IFTTT for changes
• Issues
o IFTTT Android App errors and messages w/o sound
MICROSOFT FLOW
• Features
o IFTTT-like with focus on office; 275+ connectors
• Plans
o Plan 2 (15’000 runs, check per 1 min) €13.5 user/mo
• Issues
o Made for other purpose (1 msg/min, app compatibility)
• AZ Monitor Action Group Alerts
VIBER
• Public accounts for large companies
o Can use Viber API to send to mobile phone
• Business messages by global aggregators
o €0.01 W.Europe, €0.02 Bulgaria, Min €100/month
• Viber Bot
o Web Service integrated with Viber API
o Subscribe to Viber Bot from deep link

25. Environment
Monitoring
• Purpose
• Resource usage
• Errors or malfunctioning
system components
• How
• Collect health data (device
and module heartbeat)
• Azure Monitor Service
• Dashboards, alerts,
notifications
• Tools
• App Insights, Log Analytics
• Grafana
• Azure Monitor plugin
• Influx _internal DB

26. Environment Monitoring Hiccups
25 Mar 2019 18:00
It All Started
• Lost connection from test
env. to machine
• Log connection attempt
failures in a loop (x1000/s)
Cost Update
• 29 Mar 2019 € 215.37
• 30 Mar 2019 € 366.77
• Cost updated 4d later
Warning
• Cost warning received
for Azure subscription
• Period forecast was
OK until 12 Apr 2019
Suspended
• VMs stopped
• Forecast still OK
• LogAnalytics €305
Reason
o Cost update is really slow; Check actual cost on “Subscription > Manage”
o App Insight Metrics 157GB (154M) logs for 2 days at €2.5 per 1GB (5GB free per month)
Takeaways
o Impose Application Insights daily limit to 1GB (default is 100GB) and warning at 60% quota
o Minimize log verbosity; Current log size is ~12GB/month
30 Mar 201927 Mar 2019 09:00 27 Mar 2019 16:30

27. Operational Cost
• Cost Optimized
Environment
• Average Cost
• 100 €/Month (10 machines)
• 140 €/Month (20 machines)
• Top Spendings
• VM (Influx DB)
• Storage (Disk transactions)
• IoTHub
• VM (Grafana)
• Storage (Blob)

28. Security Challenges
• Web Apps & Web APIs
o Authorize access with OAuth 2.0 and Azure enterprise applications
• Azure AD user management
o Graph API in Beta – for user to role membership
• License Manager
o Retain control over the environment with 24h renewable tokens
• Grafana
o Use read-only DB users to SQL and InfluxDB
o Create write endpoint API to Influx DB and custom plugins
o Filtering on current user for backend (Warning: insecure)
• Grafana proxy (GoLang) with filtering of query and results
• InfluxDB
o Fine grain access control to points available in Influx Enterprise

29. DEMO
• Live Dashboards
• KPI Dashboards
• System Health
• Platform Configuration
• Azure Environment

30. Thanks to our Sponsors
With the support of:

31. Upcoming Events
SQLSaturday #878 (Sofia), October 12
http://www.sqlsaturday.com/878/
JS Talks (Sofia), November 23
http://jstalks.net/
Global Microsoft 365 Developer
Bootcamp, December 14
http://aka.ms/m365devbootcamp