With the recent addition of three new services - Azure Stream Analytics, Azure Data Factory and Azure Event Hubs - Microsoft is making progress in building the best cloud platform for both big data solutions as well as enabling the Internet of Things (IoT). These additions will allow you to process, manage and orchestrate data from Internet of Things (IoT) devices and sensors and turn this data into valuable insights for your business. The above mentioned new services extend Microsoft's existing big data offering based on HDInsight and Azure Machine Learning. HDInsight is Microsoft's offering of Hadoop functionality on Microsoft Azure. It simplifies the setup and configuration of Hadoop cluster by offering it as an elastic service. Azure Machine Learning is a new Microsoft Azure-based tool that helps organization build predictive models using built in machine learning algorithms all from a web console.

1. JUNE 25, 2015 | SLIDE 1
www.realdolmen.com
WHAT’S NEW ON THE MICROSOFT AZURE DATA
PLATFORM
JORIS.POELMANS@REALDOLMEN.COM

2. JUNE 25, 2015 | SLIDE 2
#Name: Joris Poelmans #Function: Solution Architect
#Email:joris.poelmans@realdolmen.com #Twitter: jopxtwits
#Blog: jopx.blogspot.com
#Slideshare: www.slideshare.net/jplq631
Company: www.realdolmen.com

3. JUNE 25, 2015 | SLIDE 3
“In God we trust, all
others bring data.”
William E. Deming

4. JUNE 25, 2015 | SLIDE 4
TRENDS DRIVING THE NEW DATA PLATFORM

5. JUNE 25, 2015 | SLIDE 5
CONTOSO ELECTRONICS CASE
Who is interacting
with a product to
purchase?
Which products to
showcase?
How to automate
customer support?

6. JUNE 25, 2015 | SLIDE 6
INSIDE THE STORE OF THE FUTURE
… but 73% think it’s
a plus when an online store
also has an offline sales
outlet
35% of Amazon purchases
based on personalized
recommendations, 75%
for Netflix
DELIVER A PERSONALIZED EXPERIENCE WITH A HUMAN TOUCH

7. JUNE 25, 2015 | SLIDE 7
CUSTOMER CENTRICITY THROUGH DATA
Event / Data
producers
Screen
interaction
In-Store Activity
Social Data
Ingest Transform Long-term
storage
Cloud
storage
Predictive
Analytics
Predictive/
prescriptive
analytics
Presentation
and action
On premise

8. JUNE 25, 2015 | SLIDE 8
… POWERED BY MICROSOFT CLOUD
Event / Data
producers
Web logs
In-Store Activity
Social Data
Ingest Transform Long-term
storage
Azure SQL
Database & Azure
Storage
Predictive
Analytics
Azure
Machine
Learning
Presentation
and action
Azure Event Hubs
Azure Stream
Analytics
Azure HDInsight Azure ML
On premise

9. JUNE 25, 2015 | SLIDE 9
INGESTING RAW DATA AT SCALE
Azure EventHub

10. JUNE 25, 2015 | SLIDE 10
Event / Data
producers
Web logs
In-Store Activity
Social Data
Ingest
Azure Event Hubs
Azure Stream
Analytics
Azure HDInsight Azure ML

11. JUNE 25, 2015 | SLIDE 11
BUSINESS SCENARIOS

12. JUNE 25, 2015 | SLIDE 12
EVENT VELOCITY
 Device telemetry
 Thermostats report data
every 15 minutes
 Cars send telemetry data every minute
 Application telemetry
 Application performance counters are measured
every second per server
 Mobile app telemetry is captured for
every action on your app!
 Application and operational events

13. JUNE 25, 2015 | SLIDE 13
AZURE EVENT HUBS
Event Producers
HTTPS
AMQP 1.0
Throughput Units:
• 1 ≤ TUs ≤ Partition Count
• TU: 1 MB/s writes, 2 MB/s
reads
Event Producers
AMQP 1.0

14. JUNE 25, 2015 | SLIDE 14
ACTING ON DATA IN MOTION
Azure Streaming Analytics

15. JUNE 25, 2015 | SLIDE 15
… POWERED BY MICROSOFT CLOUD
Event / Data
producers
Web logs
In-Store Activity
Social Data
Ingest Transform
Azure Event Hubs
Azure Stream
Analytics
Azure HDInsight Azure ML

16. JUNE 25, 2015 | SLIDE 16
DATA AT REST
DATA AT REST DATA IN MOTION
SELECT COUNT(*) FROM
PARKINGLOT
WHERE type=‘CAR’
AND color=‘RED’
?

17. JUNE 25, 2015 | SLIDE 17
TRACK SHELF INVENTORY IN REAL TIME
1 Track your products through shelf sensors,
RFIDs, price tags, or Wi-Fi Way Finding. As
inventory is removed from the store shelf, the
store info updates in real time.
2 Configure the system to notify an employee to
restock when inventory drops below a
preconfigured range.
4 Track inventory end-to-end; from the
manufacturer, through shipments and
stocking, to the floor, and to sale.
3 Access real-time inventory data
on your devices.

18. JUNE 25, 2015 | SLIDE 18
Intake millions of events per
second
Process data from connected devices/apps
Integrated with highly-scalable publish-subscriber ingestor
Easy processing on
continuous streams of data
Transform, augment, correlate, temporal operations
Detect patterns and anomalies in streaming data
Correlate streaming with
reference data

19. JUNE 25, 2015 | SLIDE 19
AZURE STREAMING ANALYTICS
No hardware acquisition
and maintenance
No software provisioning
and maintenance
Up and running in a few
clicks
Elasticity of the cloud for
scale up or scale down
Low startup cost
Built in monitoring
SQL like language
available to create stream
processing solutions
Development and
debugging experience
through Azure Portal
Integrated with
EventHub, Azure Blobs
and Azure SQL DB

20. JUNE 25, 2015 | SLIDE 20
AZURE STREAMING ANALYTICS

21. JUNE 25, 2015 | SLIDE 21
HADOOP AS A SERVICE
Azure HDInsight

22. JUNE 25, 2015 | SLIDE 22
Event / Data
producers
Web logs
In-Store Activity
Social Data
Ingest Transform Long-term
storage
Azure SQL Database
& Azure Storage
Azure Event Hubs
Azure Stream
Analytics
Azure HDInsight Azure ML
On premise

23. JUNE 25, 2015 | SLIDE 23
HADOOP AND MODERN DATA ARCHITECTURE
 Apache Hadoop is an
open source framework that supports
data-intensive distributed applications
 Uses HDFS storage to enable applications to
work with 1000s of nodes and petabytes of data
using a scale-out model
 Uses MapReduce to process data
 Inspired by Google
 MapReduce
 Google File System
 Related projects:
 HBase, Hive, Mahout, Pig,Sqoop, Ambari, Storm,
Zookeeper, ... And many more

24. JUNE 25, 2015 | SLIDE 24
AZURE HDINSIGHT
Data Node Data Node Data Node Data Node
Task Tracker Task Tracker Task Tracker Task Tracker
Name Node
Job Tracker
HMaster
Coordination
Region Server Region Server Region Server Region Server
Pay for what you use
Use Azure Blob storage
Extend with HBase as a columnar NoSQL transactional database
Support for additional Apache projects such as Storm and Mahout

25. JUNE 25, 2015 | SLIDE 25
Telecommunications Financial Services Health Care Industry/Utility
HOW ORGANIZATIONS ARE USING HADOOP
Churn prediction, CDR
analysis, network
monitoring, next best
offer, …
Customer 360°, fraud
detection,
Clinical trial selection,
patent mining,
personalized medicine,…
Predictive maintenance,
supply chain and
inventory optimization,
smart metering,…

26. JUNE 25, 2015 | SLIDE 26

27. JUNE 25, 2015 | SLIDE 27
MACHINE LEARNING AS A SERVICE
Azure Machine Learning

28. JUNE 25, 2015 | SLIDE 28
Event / Data
producers
Web logs
In-Store Activity
Social Data
Ingest Transform Long-term
storage
Azure SQL Database
& Azure Storage
Predictive
Analytics
Azure
Machine
Learning
Presentation
and action
Azure Event Hubs
Azure Stream
Analytics
Azure HDInsight Azure ML
On premise

29. JUNE 25, 2015 | SLIDE 29
TYPES OF ANALYTICS
Traditional BI Deployed ML

30. JUNE 25, 2015 | SLIDE 30
• Formal definition: “A computer program is said to learn from
experience E with respect to some class of tasks T and performance
measure P, if its performance at tasks in T, as measured by P,
improves with experience E” - Tom M. Mitchell
• Another definition: “The goal of machine learning is to program
computers to use example data or past experience to solve a given
problem.” – Introduction to Machine Learning, 2nd Edition, MIT Press
• ML often involves two primary techniques:
– Supervised Learning: Finding the mapping between inputs and outputs using
correct values to “train” a model
– Unsupervised Learning: Finding patterns in the input data (similar to Density
Estimates in Statistics)
DEFINING MACHINE LEARNING

31. JUNE 25, 2015 | SLIDE 31
Vision Analytics
Recommenda-
tion engines
Advertising
analysis
Weather
forecasting for
business planning
Social network
analysis
Legal
discovery and
document
archiving
Pricing analysis
Fraud
detection
Churn
analysis
Equipment
monitoring
Location-based
tracking and
services
Personalized
Insurance

32. JUNE 25, 2015 | SLIDE 32
PREDICTIVE ANALYTICS/MACHINE LEARNING
Developing predictive analytics must
be simpler, today it requires
specialized skills:
• Data management
• Data exploration
• Math & statistics
• Domain expertise
• Machine learning
• Software development
• Data visualization
65% of enterprise feel they have a
strategic shortage of data scientists, a
role many did not know existed 12
months ago …

33. JUNE 25, 2015 | SLIDE 33
AZURE MACHINE LEARNING

34. JUNE 25, 2015 | SLIDE 34

35. JUNE 25, 2015 | SLIDE 35

36. JUNE 25, 2015 | SLIDE 36

37. JUNE 25, 2015 | SLIDE 37
WINDOWS AZURE
A 10,000 feet view

40. vpn

41. SUMMARY

42. JUNE 25, 2015 | SLIDE 42
3 days 10 days 15 days
Goal Architecture Architecture +
basic POC
Architecture +
POC
Pre-engagement questionnaire √ √ √
Define basic architecture √ √ √
Refine architecture √ √
Basic POC* (Project Setup, Mobile Services, Web API, Identity,
Scalability, Azure Machine Learning, HDInsight)
√ √
Extended POC (possible topics a.o. Integration with backoffice system,
Notifications, Search, Azure Machine Learning, HDInsight, …) *
√
*: scope for POCs to be discussed/Usage costs of Azure will be billed separately
AZURE POC OFFERING

43. JUNE 25, 2015 | SLIDE 43
Seamless
Integration
Scalable
Processing
Actionable
Insights
• Predictive,
Prescriptive &
Cognitive Analytics
• Data Mining &
Machine Learning
• Enriched Applications
• Distributed Data Platform
• Processing Engine
• Hybrid Architecture
• Interaction &
Transaction Data
• Existing & Emerging
Data Sources
• Open Data sets &
Commercial Data Providers
1 2 3
END-TO-END SMART DATA SOLUTIONS

44. JUNE 25, 2015 | SLIDE 44
QUIT TALKING AND BEGIN DOING.”
“THE WAY TO GET STARTED IS TO
Walt Disney