Being able to analyze data in real-time is a very hot topic already and it will be more and more in. From product recommendations to fraud detection alarms a lot of stuff would be perfect if it could happen in real time. In this session a sample solution using the serverless capabilities of Azure will be developed, right from the ingestion of sensor data to their analysis and recommendation using AI in real time. Come to see how you could do the same in your environment, moving your application capabilities to the next level.

1. Building a Real-Time IoT
monitoring application
with Azure
Davide Mauri | dm@sensoriainc.com | @mauridb
Director SW Engineering & Lead data architect
Sensoria Inc.

2. About Me
Microsoft Data Platform MVP since 2006
Working with the Microsoft Data Platform from 1997
• Specialized in Data Solution Architecture, Database Design, Performance Tuning,
High-Performance Data Warehousing, BI, Big Data
• Very strong developer background
Loves this community!
• President of UGISS (Italian SQL Server UG) for 11 Years
• AppDev VG Leader since 2017
Regular Speaker @ Microsoft Data Platform events
Director SW & Cloud @ Sensoria: http://www.sensoriafitness.com/
E-mail: dm@sensoriainc.com-Twitter: @mauridb
Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

3. About the Session
Showcase how it is possible to create an IoT solution on Azure, using a 100% PaaS
solution
Ref: http://lambda-architecture.net/

4. IoT in Sensoria
We’re applying IoT PaaS as shown to the Internet-Of-Me (IoT applied to human
beings)
Sports / Fashion: garments, smart shoes, smart socks
Healthcare: rehabilitation, fall prevention, etc. etc.

5. Steps and Technologies involved
Ingestion with Event Hub
Real-Time Data Processing via Stream Analytics
Real-Time Data Scoring via Machine Learning
Real-Time Event Handling via Event Grid/Azure Functions
Real-Time Dashboard with Power BI
Time Series analysis with Azure Time Series
Application data backbone with Azure SQL or CosmosDb
Long-Term storage and Enterprise BI with Azure SQL DW
Create a Data Lake & do Data Science with Azure Data Lake & Machine Learning

6. Architecture
100% PaaS
Lambda Architecture
Supports all business
needs:
From Field Applications
To Strategic Dashboards

7. Event Hub
Telemetry ingestion service
Handles up to millions events per second
Supports HTTPS and AMQP
Native client libraries for .NET and JAVA or REST API
Optional automatic near-real time archiving of
ingested data
https://docs.microsoft.com/en-us/azure/event-hubs/event-
hubs-capture-enable-through-portal

8. Event Hub
Other Option: IoT Hub
Centralized Device Management
IoT Hub if you need Cloud-To-Device communication
MQTT Protocol
Comparison of Azure IoT Hub and Azure Event Hubs
https://docs.microsoft.com/en-us/azure/iot-hub/iot-
hub-compare-event-hubs

9. Stream Analytics
Can handle millions of events per second
Based on the REEF project (now Apache incubated)
Streaming Query Engine
Take date from one or more input
Send resulting data to one or more output
Support most common data types:
bigint, float, unicode strings, datetime
key-value pairs
arrays

10. Stream Analytics
Stream Analytics Query Language Reference
https://msdn.microsoft.com/library/azure/dn834998.aspx
SQL-Like Query Language
With specific temporal extension
Time is a known entity to the engine
Compare the current “Scored Label” value
against the same value existing 1 minute ago

11. Stream Analytics
Native support to geospatial queries
https://msdn.microsoft.com/library/mt778980(Azure.100).a
spx
Native function to detect anomalies:
ANOMALYDETECTION
https://docs.microsoft.com/en-us/azure/stream-
analytics/stream-analytics-machine-learning-anomaly-
detection
Functions can be applied to the streamed data
Javascript
Machine Learning Web Services REST calls

12. Machine Learning
Online editor for defining training and scoring
processes
A lot of native components
Support Python and R (BYOM: Bring your own model)
Natively supports Jupyter notebooks
Once created models can be called via REST API
New version just released: Azure Machine Learning
Desktop Workbench
https://docs.microsoft.com/en-us/azure/machine-
learning/preview/quickstart-installation

13. Azure Event Grid
React to events that happens in Azure or in your
Applications
Natively provide a subscription mechanism
Eg: As soon as a new file has been archived by
Event Hub, call a function
to move it into Data Lake and/or Data Warehouse
start ML re-training

14. Serverless solution: just write the code you need, all the
infrastructural code in already done for you
Increase development efficiency *a lot*
Easily and well integrated with other Azure solutions
Supports WebHooks
Supports a variety of languages
C#, Python, PHP, Javascript…
Azure Functions

15. Power BI Reports and Dashboard
Online and desktop Dashboard solution
Desktop is free to use
Online has a free tier too
Online version supports streaming data sets
Custom or coming from Stream Analytics

16. Time Series Insights
Specific tool to analyze Time-Series data
Just Perfect for IoT
Automatically infer schema of incoming data
Easily allow visual analysis of time-series data
Heat-Map especially useful

17. Azure SQL
The “usual” SQL Server, just PaaS Just a few
highlights:
In-Memory (Lock-Free structures)
Columnar Storage
JSON support
In-Database Analytics with Python (coming soon) and R
(available now)
Graph

18. Cosmos DB
NoSQL distributed database
Key-Value, Columnar, Document, Graph
Multiple API supported
DocumentDB, MongoDB, Table, Gremlin
Guaranteed SLA on latency
99% of reads in <10 ms and writes in <15 ms

19. Data Lake
Made of two separate services
Pay only for what you use
Data Lake Store
Store any type of data at any scale
Almost no limits in size and quantity
Supports HDFS
Data Lake Analytics
Distributed computing platform
Integrated with Visual Studio
Great developer experience
U-SQL:
Declarative language,
SQL-Like with C# Data Types,
Extensible with .NET, R & Python and other languages

20. U-SQL
Declarative, composable language
Approach similar to Apache Pig
Procedure, Functions, Views are also
supported
Input and Output are files or U-SQL
databases
Script is analyzed, split across nodes
and executed in parallel

21. Azure DW
MPP data warehouse solution
Ideal for long-term organized storage
Separate charge for storage and compute
Based on SQL Server
Columnstore
Polybase allows integration with Hadoop

22. Are all these technologies
really needed?
Quite likely. Why?
My complete answer is here:
“Data Juice”
https://www.slideshare.net/davidemauri/data-juice
https://medium.com/@mauridb/data-juice-
5a49b159fd51

23. Conclusions
100% IoT ingestion/monitoring/analysis solution is possible
Focus on business not on infrastructure
Simpler if fits your use case
Otherwise the Apache stack (Kafka, Storm, Spark, etc is still the most flexible option)
Why we went with Azure?
Great integration with development tools
Supports almost all languages and all platforms
Our background was python and .net and sql server
Lots of new products, very fast evolution of existing one
Stay updated and informed

24. Thanks!