1) Sam Vanhoutte discusses using Azure services like IoT Edge, IoT Hub, Stream Analytics, and Azure Databricks for real-time data analytics in IoT from edge to cloud.
2) A traffic camera scenario is presented where IoT Edge is used at the edge for tasks like license plate recognition while the cloud is used for analytics like detecting speeding tickets and suspicious vehicles.
3) Stream Analytics is used both at the edge and in the cloud to process streaming data in real-time while Azure Databricks is used for structured streaming and continuous aggregations using Apache Spark.
2. Connecting end to end
Connect things to the cloud
Cloud strategy & architecture
Automate business processes
Expose API’s
2
3. About Codit
3
2000 Belgium
2004 France
2013 Portugal
2016 Switzerland
2016 UK
2016 The Netherlands
2017 Malta
180
worldwide
Largest Microsoft
partner in Europe for
integration, API
management, IoT
and Azure Solutions
9. Architecture
9
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
10. 2. Time Series Insights : Data exploration
10
• Get near real-time
insights in seconds
• Start in seconds, scale in
minutes
• Create a global view of
your IoT-scale data
• Leverage the power of
Time Series Insights in
your Apps and Solutions
11. Architecture
11
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
12. Architecture
12
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
13. 3. Stream Analytics: in the cloud & on the edge
13
Presentation &
Action
Storage &
Batch Analysis
Stream
Analytics
Event Queuing
& Stream
Ingestion
Event
production
IoT Hubs
Applications
Archiving for long
term storage/
batch analytics
Real-time dashboard
Stream
Analytics
Automation to
kick-off workflows
Machine Learning
Reference Data
Event Hubs
Blobs
Devices &
Gateways PowerBI
14. Stream analytics on the edge
14
Camera simulator
Stream Analytics Display simulator
Cloud hubEdge hub
$upstream
Rules Engine{
"routes": {
"SpeedCalculator": "FROM /messages/modules/camera/outputs/*
INTO BrokeredEndpoint("/modules/traffic-speeding-detection-edge/inputs/iot-speed-events")",
"DisplayWarning": "FROM /messages/modules/traffic-speeding-detection-edge/outputs/*
INTO BrokeredEndpoint("/modules/display/inputs/camera")",
"Filter": "FROM /messages/modules/camera/outputs/*
INTO BrokeredEndpoint("/modules/rules-engine/inputs/nebulus")",
"Cloud": "FROM /messages/modules/rules-engine/outputs/*
INTO $upstream"
}
}
1
2
3
4
1
2
3
4
15. Architecture
15
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
16. Architecture
16
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
17. 4. Structured data streaming: Azure Data Bricks
17
Optimized Databricks Runtime Engine
DATABRICKS I/O SERVERLESS
Collaborative Workspace
Cloud storage
Data warehouses
Hadoop storage
IoT / streaming data
Rest APIs
Machine learning models
BI tools
Data exports
Data warehouses
Azure Databricks
Deploy Production Jobs & Workflows
APACHE SPARK
MULTI-STAGE PIPELINES
DATA ENGINEER
JOB SCHEDULER NOTIFICATION & LOGS
DATA SCIENTIST BUSINESS ANALYST
18. Streaming pipeline
| readStream…load() creates
a streaming DataFrame, does
not start any computation
val input = spark.readStream
.format("json")
.load("source-path")
val output = input
.select(“clientid“, “querytime”)
.where(“querytime > 100")
output.writeStream
.format("json")
.start(“dest-path")
24. Continuous windowed aggregations
• Simplifies event-time stream processing
• Works in both, streaming and batch jobs
input.groupBy(
$“carmake”,
window($“event-time”, “10 min”))
.avg(“speed”)
25. Joining streams with static data
| Join streaming data from IoT Hub with
static dataset from Azure Blob to enrich
streaming data
val iotStream = spark.readStream
.format(“eventhubs”)
.load()
val staticDataset = spark.read
.json(“/mnt/json”, multiline=True)
val joinedDataset =
iotStream.join(
staticDataset, “segmentId”)
26. Query management
| query: a handle to the running streaming
computation
| Multiple queries can be active at the same time
| Each query has unique name to keep track of
it’s state
val query = result.writeStream
.format(“parquet”)
.outputMode(“append”)
.start(“dest-path”)
query.stop()
query.awaitTermination()
query.exception()
query.sourceStatuses()
query.sinkStatuses()
27. Architecture
27
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
28. Architecture
28
IoT Hub
Time Series
Stream Analytics
Data Lake Gen2
Logic Apps
Fine creation
Event Grid
Detect speeding cars
Detect
Suspected cars
Dashboard
Logic Apps
Alerting !
IoT Edge
29. Takeaways
29
| Azure IoT Edge for connectivity & AI
close to the devices
| Azure IoT Hub as secure , high
performant service that connects it
all
| Data analytics options available
| Stream Analytics for quick starting and
easy query logic
| Azure Data Bricks as 1st class citizen for
streaming, machine learning and
translation
| Multiple data integration options
available
The value of IoT
is defined by the
data and
integration
We have a broad array of offerings to suit your business needs. We do consultancy, we design solutions, we use our own turnkey software such as Invictus and Nebulus – designed specifically for businesses like yours- to easily implement solutions. Not only do we design and build your soltuion, our Managed Services team can run it for you 24/7.
{
"routes": {
"SpeedCalculator": "FROM /messages/modules/camera/outputs/* INTO BrokeredEndpoint(\"/modules/traffic-speeding-detection-edge/inputs/iot-speed-events\")",
"DisplayWarning": "FROM /messages/modules/traffic-speeding-detection-edge/outputs/* INTO BrokeredEndpoint(\"/modules/display/inputs/camera\")",
"Filter": "FROM /messages/modules/camera/outputs/* INTO BrokeredEndpoint(\"/modules/rules-engine/inputs/nebulus\")",
"Cloud": "FROM /messages/modules/rules-engine/outputs/* INTO $upstream"
}
}