4. Implement Data Warehouse
Reporting &
Analytics
Development
Reporting &
Analytics Design
Physical DesignDimension Modelling
ETL
Development
ETL Design
Install and TuneSetup Infrastructure
Traditional data warehousing approach
Data sources
ETL
BI and analytics
Data warehouse
Understand
Corporate
Strategy
Gather
Requirements
Business
Requirements
Technical
Requirements
5. The Data Lake approach
Ingest all data
regardless of
requirements
Store all data
in native format
without schema
definition
Do analysis
Using analytic
engines like Hadoop
Interactive queries
Batch queries
Machine Learning
Data warehouse
Real-time analytics
Devices
6. Source: ComScore 2009-2015 Search Report US
9%
11%
15%
16%
18%
19%
20%
0%
5%
10%
15%
20%
25%
2009 2010 2011 2012 2013 2014 2015
MICROSOFT DOUBLES SEARCH SHARE
How Microsoft has used
Big Data
We needed to better leverage data and
analytics to win in search
We changed our approach
• More experiments by more people!
So we…
Built an Exabyte-scale data lake for everyone
to put their data.
Built tools approachable by any developer.
Built machine learning tools for collaborating
across large experiment models.
11. No limits to SCALE
Store ANY DATA in its native format
HADOOP FILE SYSTEM (HDFS) for the cloud
ENTERPRISE GRADE access control, encryption
at rest
Optimized for analytic workload
PERFORMANCE
Azure Data Lake
Store
A hyper scale repository for big
data analytics workloads
IN PREVIEW
12. Data Lake Store: Built for the cloud
Secure Must be highly secure to prevent unauthorized access (especially as all data is in one place).
Native format Must permit data to be stored in its ‘native format’ to track lineage and for data provenance.
Low latency Must have low latency for high-frequency operations.
Must support multiple analytic frameworks—Batch, Real-time, Streaming, Machine Learning, etc.
No one analytic framework can work for all data and all types of analysis.
Multiple analytic
frameworks
Details Must be able to store data with all details; aggregation may lead to loss of details.
Throughput Must have high throughput for massively parallel processing via frameworks such as Hadoop and Spark.
Reliable Must be highly available and reliable (no permanent loss of data).
Scalable Must be highly scalable. When storing all data indefinitely, data volumes can quickly add up.
All sources Must be able ingest data from a variety of sources-LOB/ERP, Logs, Devices, Social NWs etc.
15. FULLY SUPPORTED Hadoop for the cloud
Available on LINUX and WINDOWS
Works on AZURE STORAGE or DATA LAKE
STORE
100% OPEN SOURCE Apache Hadoop (HDP 2.3)
Clusters up and RUNNING IN MINUTES
Use familiar BI TOOLS FOR ANALYSIS like Excel
Azure HDInsight
Hadoop Platform as a
Service on Azure
18. ADLA complements HDInsight
Target the same scenarios, tools, and customers
HDInsight
For developers familiar with the
Open Source: Java, Eclipse, Hive, etc.
Clusters offer customization, control,
and flexibility in a managed Hadoop
cluster
ADLA
Enables customers to leverage
existing experience with C#, SQL &
PowerShell
Offers convenience, efficiency,
automatic scale, and management in
a “job service” form factor
19. No limits to SCALE
Includes U-SQL, a language that unifies the
benefits of SQL with the expressive power of C#
Optimized to work with ADL STORE
FEDERATED QUERY across Azure data sources
ENTERPRISE GRADE role-based access control
and auditing
Pay PER QUERY and scale PER QUERY
Azure Data Lake
Analytics
A distributed analytics service
built on Apache YARN that
dynamically scales to your
needs
IN PREVIEW
21. Work across all cloud data
Azure Data Lake
Analytics
Azure SQL DW Azure SQL DB
Azure
Storage Blobs
Azure
Data Lake Store
SQL DB in an
Azure VM
22.
23. Simplified management and administration
Web-based management
in Azure Portal
Automate tasks using
PowerShell
Role-based access control
with Azure AD
Monitor service
operations and activity
24. Get started
Log in to Azure Create an ADLA
account
Write and
submit an ADLA
job with U-SQL
(or Hive/Pig)
The job reads
and writes data
from storage
1 2 3 4
30 seconds
ADLS
Azure Blobs
Azure DB
…
26. Account Management
Create new account
List accounts
Update account properties
Delete account
Transferring Data
Upload into store from local
disk
Download from store to
local disk
Files and Folders
List contents of
folder
Create
Move
Delete
Does file exist
Security
Get ACLs
Update ACLs
Get Owner
Set Owner
File Content
Set file content
Append file content
Get file content
Merge files
27. Account Management
Create new account
List accounts
Update account properties
Delete account
Data Sources
Add a data source
List data sources
Update data source
Delete data source
Compute
List jobs
Submit job
Cancel job
Catalog Items
List items in U-SQL catalog
Update item
Catalog Secrets
Create catalog secret
List catalog secrets
Delete catalog secrets
29. Management
Create and manage ADLA accounts
Jobs
Submit and manage jobs
Catalog
Explore catalog items
Management
Create and manage ADLS accounts
File System
Upload, download, list, delete, rename, append
(WebHDFS)
Analytics Store
The opportunity – more data than ever before to use
The challenge – how to find value in that data, structured/unstructured
The Data Warehouses leverages the top-down approach where there is a well-architected information store and enterprisewide BI solution. To build a data warehouse follows the top-down approach where the company’s corporate strategy is defined first. This is followed by gathering of business and technical requirements for the warehouse. The data warehouse is then implemented by dimension modelling and ETL design followed by the actual development of the warehouse. This is all done prior to any data being collected. It utilizes a rigorous and formalized methodology because a true enterprise data warehouse supports many users/applications within an organization to make better decisions.
A data lake is an enterprise wide repository of every type of data collected in a single place. Data of all types can be arbitrarily stored in the data lake prior to any formal definition of requirements or schema for the purposes of operational and exploratory analytics. Advanced analytics can be done using Hadoop, Machine Learning tools, or act as a lower cost data preparation location prior to moving curated data into a data warehouse. In these cases, customers would load data into the data lake prior to defining any transformation logic.
This is bottom up because data is collected first and the data itself gives you the insight and helps derive conclusions or predictive models.
Other points to make here, but not called out above
Built on Apache YARN
Scales dynamically with the turn of a dial
Supports Azure AD for access control, roles, and integration with on-prem identity systems
U-SQL’s scalable runtime processes data across multiple Azure data sources
ADLA allows you to compute on data anywhere and a join data from multiple cloud sources.
Tell them what they need to know in order to create this sample (pre-reqs)