Microsoft Cloud
Workshop
INGEST STORE PREP & TRAIN MODEL & SERVE
R E F E R E N C E A R C H I T E C T U R E S :
I N T E L L I G E N T A P P L I C A T I O N S
Azure Blob Storage
Business/custom apps
(Structured)
Logs, files and media
(unstructured)
Azure SQL Data
Warehouse
Azure Analysis
Services
Azure Data Factory
Sensors and IoT
(unstructured)
Azure HDInsight (Kafka)
Azure IoT Hub
Cosmos DB
Predictive apps
Operational Reports
&
Analytical Dashboards
(PowerBI)
SQL Database
SQL
Polybase
Azure Databricks /
HDInsight
https://azure.microsoft.com/en-au/solutions/architecture/
Azure Stream
Analytics
New monetization avenues
due to IoT-related services
Companies that increased
revenue as a result of IoT
implementation
Average increase in
operating income (avg. 8%)
among the most digitally
transformed enterprises
p
p
p
Data +
intelligence
Optimize
operations
Transform
products
Engage
customers
Empower
employees
Connected “things” by 2025
generating 180ZB of data
p
Azure Data Lake
Azure Time Series
Insights
Azure Maps
Azure Stream Analytics
Azure Machine Learning
Azure HD Insight
Spark, Storm, Kafka
Azure Event Hubs
Azure IoT Hub Device
Provisioning Service
Azure IoT Hub
Windows 10 IoT Core
Azure IoT Edge
Azure Sphere
Azure Certified for IoT
Azure IoT Device SDK
Azure Logic Apps
Azure Websites
Azure Monitor
Azure Event Grid
Microsoft Power BI
Microsoft Flow
Azure Functions
PlatformUse|Solutions
IoT Edge
Azure IoT Central
Analytics, dashboards and visualization
User roles and permissions
Monitoring rules and triggered actions
Fully hosted and managed by Microsoft
Device connectivity and management
Risk-free trial with simplified pricing
No cloud development expertise required
Azure IoT solution
accelerators
Predictive Maintenance
Connected FactoryRemote Monitoring
Device Simulation
End-to-end implementation
Completely customizable
Open-source microservices based architecture
Device connectivity and management
Dashboards, visualization, and insights
Workflow automation and integration
Command and control
Preconfigured solutions
Core Subsystems
Things Insights Actions
Provision and
send data from
device to cloud
Device
Management
Stream processing and
rules evaluation over data
Store data Integrate with business processes
Visualize data and learnings
aka.ms/iotrefarch
© Microsoft Corporation
Serverless Azure service
Billions of devices and assets
Supports Linux, iOS, Android, Linux, Windows, and
real-time operating system (RTOS) devices
Supports reliable bi-directional communication—
device-to-cloud and cloud-to-device
Ingests data in real-time
Can manage IoT devices at scale with device
management
Extends the power of the cloud to edge devices
with Azure IoT Edge
Azure IoT Hub
A fully-managed service to connect, monitor, and manage billions of IoT assets
© Microsoft Corporation
Per device security - IoT Hub lets you set up individual identities and credentials for your connected devices. This helps maintain
the confidentiality of cloud-to-device and device-to-cloud authentication. You can also selectively revoke the access rights to
specific devices as needed.
Device monitoring and management - Administrators can remotely maintain, update, and manage IoT devices at scale from the
cloud. Service personnel will rarely have to travel to the asset location. It also monitors device connectivity.
Provisioning - With IoT Hub Device Provisioning Service you can register and provision devices with zero-touch in a secure and
scalable way. Devices can be provisioned via the Azure portal.
Bi-directional communication - IoT Hub lets you establish bi-directional communication between Azure and IoT devices. Use
device-to-cloud telemetry data to understand the state of your devices and assets. In cloud-to-device messages, reliably send
commands and notifications to your connected devices—and track message delivery with acknowledgement receipts. Device
messages are sent in a durable way to accommodate intermittently connected devices.
Device twins - Using device twins, you can store, synchronize, and query device metadata and state information. Device twins
are JSON documents that store device state information like metadata, configurations, and conditions. IoT Hub maintains a
device twin for each device that you connect to IoT Hub.
Route messages - IoT Hub enables you to define message routes based on routing rules to control where your hub sends
device-to-cloud messages. Routing rules don’t require you to write any code, and can take the place of custom post-ingestion
message dispatchers.
Azure IoT Hub capability overview
© Microsoft Corporation
Full list of all the IoT Hub endpoints: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-endpoints
Azure IoT Hub endpoints
Azure IoT Hub exposes several endpoints for different purposes
Some important endpoints
IoT Hub
Device id
Send/receive data via IoT Hub
C2D queue
endpoint
D2C send
endpoint D2C receive
endpoint
C2D send and
feedback endpoints
Event processing
(hot path)
Device
Client SDKs
• C
• Java
• C# (.Net
Standard 1.3)
• Python
Device management, device business logic,
Connectivity monitoring
Client SDKs
• .Net
• Node
• Java
• Python
IoT Hub
Device Twin
IoT Hub device management
Device
Device management, device business logic,
Connectivity monitoring
Properties
Tags
Desired
Reported
Properties
Desired
Reported
Method
• Simple "plug and play” provisioning
• Minimize manual connection
requirements
• Enhanced security through HSM
• Global availability
IoT Solution US IoT Solution ChinaIoT Solution Germany
Azure IoT Edge
Compatible with popular operating systems
Code symmetry between cloud and edge
for easy development and testing
Secure solution from chipset to cloud
Move cloud and custom workloads
to the edge, securely
Seamless deployment of AI and
advanced analytics
Configure, update and monitor
from the cloud
Azure Time Series Insights
IoT scale time-series data store
Easy IoT Hub connection
Store, query, and visualize billions of events
Get near real-time insights in seconds
Schema-less store, just send data
Build apps using Time Series Insights APIs
Azure Maps
Render maps and satellite imagery across many
geographies
Integrate rich mapping visualizations into applications
Calculate routes from N to N points for optimal
calculations
Convert places and addresses to coordinates; or,
convert coordinates to addresses or cross streets
Show real time traffic information
Obtain time zone and current time information
Azure Sphere
Control
Measure
Insight,
optimization
Quantified Organization
Assistance,
Task completion
Intent,
Context
Artificial Intelligence
Measure
Insight,
optimization
Quantified SelfDIGITAL
WORLD
Data
ControlData
ControlData
VI
PEOPLE
SPACES
PHYSICAL
WORLD
© Microsoft Corporation
Canonical operations
Streaming
Connect, collect, and store
Ingest
Process and analyze
Analytics
Connect, collect, and store
Actions
A
B
C
© Microsoft Corporation
 Input Capacity: 1 MB/s per TU*
 Output Capacity: 2 MB/s per TU*
 Latency: 50 ms avg, 99% < 100ms
 Events/second: 1,000
 Max message size: 256 KB
*In Azure Event Hubs, capacity is purchased in throughput units (TU). Add TUs to increase capacity.
Event
publisher
Partition
Partition
Partition
Reader
Reader
Reader
Event
Consumer
Event hubs
Azure Event Hubs:
Scale and performance
Azure Event Hubs
A highly scalable, fully-managed telemetry ingestion service
Stream
Analytics on
IoT Edge
Presentation &
Action
Storage &
Batch Analysis
Stream
Analytics
Event Queuing
& Stream
Ingestion
Event
production
IoT Hub
Applications
Archiving for long
term storage/
batch analytics
Real-time dashboard
Azure
Stream
Analytics
Automation to
kick-off workflows
Machine Learning
Reference Data
Event Hubs
Blobs
Devices &
Gateways
© Microsoft Corporation
Azure Blob Storage
Azure IoT Hub
Azure Event Hubs
Reference data
Streaming data
Streaming data
Azure Stream
Analytics
Integration with Azure Event Hubs and IoT Hub
Azure Stream Analytics has built-in, first class integration with Azure Event Hubs and IoT Hub
Data from Azure Event Hubs and Azure IoT Hub
can be sources of streaming data to Azure Stream Analytics
The connections can be established through the Azure Portal without any coding
Azure Blob Storage is supported as a source of reference data
Azure Stream Analytics supports compression across all data
stream input sources—Event Hubs, IoT Hub, and Blob Storage
Better
Data
Better
Data
Optimized Databricks Runtime Engine
DATABRICKS I/O SERVERLESS
Collaborative Workspace
Cloud storage
Data warehouses
Hadoop storage
IoT / streaming data
Rest APIs
Machine learning models
BI tools
Data exports
Data warehouses
Azure Databricks
Enhance Productivity
Deploy Production Jobs & Workflows
APACHE SPARK
MULTI-STAGE PIPELINES
DATA ENGINEER
JOB SCHEDULER NOTIFICATION & LOGS
DATA SCIENTIST BUSINESS ANALYST
Build on secure & trusted cloud Scale without limits
Azure Databricks
© Microsoft Corporation
For archiving for compliance (raw data)
For batch analysis using Big Data analytics
To train machine learning models Event Hubs IoT Hub Kafka on HDInsight
Stream
Analytics
Storm on
HDInsight
Azure Databricks
(Spark Streaming)
Stream ingestion
Long-term storage
Stream analytics
Raw data Processed data
Why store streaming data?
Azure Storage
Object Storage
Data Transport
File storage
Hybrid Storage
Lift and shift of legacy
applications that require file
shares to the cloud
Secure, centralized storage
target for backup/disaster
recovery
Move or migrate data into
Azure
Secure, intelligent data tiering
between on-premises and cloud
storage
Azure Blobs Azure Files
Azure NetApp Files
Azure Import/Export
Azure DataBox
Azure StorSimple
Azure File Sync
Avere
Disk Storage
Reliable, persistent, high
performing storage for Virtual
Machines
Premium
Standard
Blob Level
Tiering
Blob-Level Tiering
Individual blobs can move
between tiers
All tiers co-exist in the same
storage account
New API to set blob tier:
SetBlobTier
Acknowledged immediately from
service
Get APIs (GetBlobProperties
and ListBlobs) return current tier
and archive status
New headers "x-ms-access-tier”
and “x-ms-archive-status”
Preview @ Build, GA CY18:
Automated Lifecycle
Management
Blob REST API
Hot Tier
Lower Transaction cost
Cool Tier
Lower Capacity cost
Archive Tier
Lowest Capacity cost
© Microsoft Corporation
No limits on number of files, size of individual files, total
amount of data stored, how long data can be stored, or
ingestion throughput
Low latency and high throughput workloads can be used
for ingesting streaming data
Stores all data types
Is Hadoop-compatible via WebHDFS REST API.
Supported by leading Hadoop distros and HDInsight
Provides POSIX-style permissions for RBAC
Integrates with Azure AD for authentication
Azure Data Lake store
A highly scalable, parallel, file system in the cloud that’s specifically optimized for Big Data analytics
Backend storage in Azure
Data node Data node Data node Data node Data nodeData node
Sh
ard
Sh
ardBlock
Block 1 Block 2
Block
n…
Azure Data Lake Store file
Sh
ard
Sh
ardBlock
Sh
ard
Sh
ardBlock
Sh
ard
Sh
ardBlock
Sh
ard
Sh
ardBlock
Sh
ard
Sh
ardBlock
Azure Cosmos DB
SQL
MongoDB
Table API
Turnkey global
distribution
Elastic scale out
of storage & throughput
Guaranteed low latency
at the 99th percentile
Comprehensive
SLAs
Five well-defined
consistency models
DocumentColumn-family
Key-value Graph
A globally distributed, massively scalable, multi-model database service
Any data, any way, anywhere
© Microsoft Corporation
Big Data streaming pattern with Azure
Real-time applications
Real-time dashboards
Sensors and IoT
(unstructured)
Event hubs IoT hub Kafka on HDInsight Azure Stream
Analytics
Storm on
HDInsight
Azure Databricks
(Spark Streaming)
Azure ML
Studio
R Server Azure Databricks
(Spark ML)
Machine learning
Stream ingestion
Long-term storage
Stream analytics
Data Lake Store SQL DB Cosmos DB Azure Blob Storage
Business/custom apps
(structured)
Logs, files, and media
(unstructured)
Power BI
Azure IoT School
aka.ms/iotschool
Azure IoT Summary

Azure IoT Summary

  • 1.
  • 2.
    INGEST STORE PREP& TRAIN MODEL & SERVE R E F E R E N C E A R C H I T E C T U R E S : I N T E L L I G E N T A P P L I C A T I O N S Azure Blob Storage Business/custom apps (Structured) Logs, files and media (unstructured) Azure SQL Data Warehouse Azure Analysis Services Azure Data Factory Sensors and IoT (unstructured) Azure HDInsight (Kafka) Azure IoT Hub Cosmos DB Predictive apps Operational Reports & Analytical Dashboards (PowerBI) SQL Database SQL Polybase Azure Databricks / HDInsight https://azure.microsoft.com/en-au/solutions/architecture/ Azure Stream Analytics
  • 3.
    New monetization avenues dueto IoT-related services Companies that increased revenue as a result of IoT implementation Average increase in operating income (avg. 8%) among the most digitally transformed enterprises p p p Data + intelligence Optimize operations Transform products Engage customers Empower employees Connected “things” by 2025 generating 180ZB of data p
  • 4.
    Azure Data Lake AzureTime Series Insights Azure Maps Azure Stream Analytics Azure Machine Learning Azure HD Insight Spark, Storm, Kafka Azure Event Hubs Azure IoT Hub Device Provisioning Service Azure IoT Hub Windows 10 IoT Core Azure IoT Edge Azure Sphere Azure Certified for IoT Azure IoT Device SDK Azure Logic Apps Azure Websites Azure Monitor Azure Event Grid Microsoft Power BI Microsoft Flow Azure Functions PlatformUse|Solutions IoT Edge
  • 5.
    Azure IoT Central Analytics,dashboards and visualization User roles and permissions Monitoring rules and triggered actions Fully hosted and managed by Microsoft Device connectivity and management Risk-free trial with simplified pricing No cloud development expertise required
  • 6.
    Azure IoT solution accelerators PredictiveMaintenance Connected FactoryRemote Monitoring Device Simulation End-to-end implementation Completely customizable Open-source microservices based architecture Device connectivity and management Dashboards, visualization, and insights Workflow automation and integration Command and control Preconfigured solutions
  • 7.
    Core Subsystems Things InsightsActions Provision and send data from device to cloud Device Management Stream processing and rules evaluation over data Store data Integrate with business processes Visualize data and learnings aka.ms/iotrefarch
  • 8.
    © Microsoft Corporation ServerlessAzure service Billions of devices and assets Supports Linux, iOS, Android, Linux, Windows, and real-time operating system (RTOS) devices Supports reliable bi-directional communication— device-to-cloud and cloud-to-device Ingests data in real-time Can manage IoT devices at scale with device management Extends the power of the cloud to edge devices with Azure IoT Edge Azure IoT Hub A fully-managed service to connect, monitor, and manage billions of IoT assets
  • 9.
    © Microsoft Corporation Perdevice security - IoT Hub lets you set up individual identities and credentials for your connected devices. This helps maintain the confidentiality of cloud-to-device and device-to-cloud authentication. You can also selectively revoke the access rights to specific devices as needed. Device monitoring and management - Administrators can remotely maintain, update, and manage IoT devices at scale from the cloud. Service personnel will rarely have to travel to the asset location. It also monitors device connectivity. Provisioning - With IoT Hub Device Provisioning Service you can register and provision devices with zero-touch in a secure and scalable way. Devices can be provisioned via the Azure portal. Bi-directional communication - IoT Hub lets you establish bi-directional communication between Azure and IoT devices. Use device-to-cloud telemetry data to understand the state of your devices and assets. In cloud-to-device messages, reliably send commands and notifications to your connected devices—and track message delivery with acknowledgement receipts. Device messages are sent in a durable way to accommodate intermittently connected devices. Device twins - Using device twins, you can store, synchronize, and query device metadata and state information. Device twins are JSON documents that store device state information like metadata, configurations, and conditions. IoT Hub maintains a device twin for each device that you connect to IoT Hub. Route messages - IoT Hub enables you to define message routes based on routing rules to control where your hub sends device-to-cloud messages. Routing rules don’t require you to write any code, and can take the place of custom post-ingestion message dispatchers. Azure IoT Hub capability overview
  • 10.
    © Microsoft Corporation Fulllist of all the IoT Hub endpoints: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-endpoints Azure IoT Hub endpoints Azure IoT Hub exposes several endpoints for different purposes Some important endpoints
  • 11.
    IoT Hub Device id Send/receivedata via IoT Hub C2D queue endpoint D2C send endpoint D2C receive endpoint C2D send and feedback endpoints Event processing (hot path) Device Client SDKs • C • Java • C# (.Net Standard 1.3) • Python Device management, device business logic, Connectivity monitoring Client SDKs • .Net • Node • Java • Python
  • 12.
    IoT Hub Device Twin IoTHub device management Device Device management, device business logic, Connectivity monitoring Properties Tags Desired Reported Properties Desired Reported Method
  • 13.
    • Simple "plugand play” provisioning • Minimize manual connection requirements • Enhanced security through HSM • Global availability IoT Solution US IoT Solution ChinaIoT Solution Germany
  • 14.
    Azure IoT Edge Compatiblewith popular operating systems Code symmetry between cloud and edge for easy development and testing Secure solution from chipset to cloud Move cloud and custom workloads to the edge, securely Seamless deployment of AI and advanced analytics Configure, update and monitor from the cloud
  • 16.
    Azure Time SeriesInsights IoT scale time-series data store Easy IoT Hub connection Store, query, and visualize billions of events Get near real-time insights in seconds Schema-less store, just send data Build apps using Time Series Insights APIs
  • 18.
    Azure Maps Render mapsand satellite imagery across many geographies Integrate rich mapping visualizations into applications Calculate routes from N to N points for optimal calculations Convert places and addresses to coordinates; or, convert coordinates to addresses or cross streets Show real time traffic information Obtain time zone and current time information
  • 19.
  • 20.
    Control Measure Insight, optimization Quantified Organization Assistance, Task completion Intent, Context ArtificialIntelligence Measure Insight, optimization Quantified SelfDIGITAL WORLD Data ControlData ControlData VI PEOPLE SPACES PHYSICAL WORLD
  • 21.
    © Microsoft Corporation Canonicaloperations Streaming Connect, collect, and store Ingest Process and analyze Analytics Connect, collect, and store Actions A B C
  • 22.
    © Microsoft Corporation Input Capacity: 1 MB/s per TU*  Output Capacity: 2 MB/s per TU*  Latency: 50 ms avg, 99% < 100ms  Events/second: 1,000  Max message size: 256 KB *In Azure Event Hubs, capacity is purchased in throughput units (TU). Add TUs to increase capacity. Event publisher Partition Partition Partition Reader Reader Reader Event Consumer Event hubs Azure Event Hubs: Scale and performance Azure Event Hubs A highly scalable, fully-managed telemetry ingestion service
  • 23.
    Stream Analytics on IoT Edge Presentation& Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production IoT Hub Applications Archiving for long term storage/ batch analytics Real-time dashboard Azure Stream Analytics Automation to kick-off workflows Machine Learning Reference Data Event Hubs Blobs Devices & Gateways
  • 24.
    © Microsoft Corporation AzureBlob Storage Azure IoT Hub Azure Event Hubs Reference data Streaming data Streaming data Azure Stream Analytics Integration with Azure Event Hubs and IoT Hub Azure Stream Analytics has built-in, first class integration with Azure Event Hubs and IoT Hub Data from Azure Event Hubs and Azure IoT Hub can be sources of streaming data to Azure Stream Analytics The connections can be established through the Azure Portal without any coding Azure Blob Storage is supported as a source of reference data Azure Stream Analytics supports compression across all data stream input sources—Event Hubs, IoT Hub, and Blob Storage
  • 25.
  • 26.
  • 27.
    Optimized Databricks RuntimeEngine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Enhance Productivity Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST Build on secure & trusted cloud Scale without limits Azure Databricks
  • 28.
    © Microsoft Corporation Forarchiving for compliance (raw data) For batch analysis using Big Data analytics To train machine learning models Event Hubs IoT Hub Kafka on HDInsight Stream Analytics Storm on HDInsight Azure Databricks (Spark Streaming) Stream ingestion Long-term storage Stream analytics Raw data Processed data Why store streaming data?
  • 29.
    Azure Storage Object Storage DataTransport File storage Hybrid Storage Lift and shift of legacy applications that require file shares to the cloud Secure, centralized storage target for backup/disaster recovery Move or migrate data into Azure Secure, intelligent data tiering between on-premises and cloud storage Azure Blobs Azure Files Azure NetApp Files Azure Import/Export Azure DataBox Azure StorSimple Azure File Sync Avere Disk Storage Reliable, persistent, high performing storage for Virtual Machines Premium Standard
  • 30.
    Blob Level Tiering Blob-Level Tiering Individualblobs can move between tiers All tiers co-exist in the same storage account New API to set blob tier: SetBlobTier Acknowledged immediately from service Get APIs (GetBlobProperties and ListBlobs) return current tier and archive status New headers "x-ms-access-tier” and “x-ms-archive-status” Preview @ Build, GA CY18: Automated Lifecycle Management Blob REST API Hot Tier Lower Transaction cost Cool Tier Lower Capacity cost Archive Tier Lowest Capacity cost
  • 31.
    © Microsoft Corporation Nolimits on number of files, size of individual files, total amount of data stored, how long data can be stored, or ingestion throughput Low latency and high throughput workloads can be used for ingesting streaming data Stores all data types Is Hadoop-compatible via WebHDFS REST API. Supported by leading Hadoop distros and HDInsight Provides POSIX-style permissions for RBAC Integrates with Azure AD for authentication Azure Data Lake store A highly scalable, parallel, file system in the cloud that’s specifically optimized for Big Data analytics Backend storage in Azure Data node Data node Data node Data node Data nodeData node Sh ard Sh ardBlock Block 1 Block 2 Block n… Azure Data Lake Store file Sh ard Sh ardBlock Sh ard Sh ardBlock Sh ard Sh ardBlock Sh ard Sh ardBlock Sh ard Sh ardBlock
  • 32.
    Azure Cosmos DB SQL MongoDB TableAPI Turnkey global distribution Elastic scale out of storage & throughput Guaranteed low latency at the 99th percentile Comprehensive SLAs Five well-defined consistency models DocumentColumn-family Key-value Graph A globally distributed, massively scalable, multi-model database service
  • 33.
    Any data, anyway, anywhere
  • 34.
    © Microsoft Corporation BigData streaming pattern with Azure Real-time applications Real-time dashboards Sensors and IoT (unstructured) Event hubs IoT hub Kafka on HDInsight Azure Stream Analytics Storm on HDInsight Azure Databricks (Spark Streaming) Azure ML Studio R Server Azure Databricks (Spark ML) Machine learning Stream ingestion Long-term storage Stream analytics Data Lake Store SQL DB Cosmos DB Azure Blob Storage Business/custom apps (structured) Logs, files, and media (unstructured) Power BI
  • 35.