Find out how you can get actionable insights and drive better decision making with Azure HDInsight!

1. Introduction to
Microsoft Azure HDInsight
Dattatrey Sindhol

2. 2
Agenda
Introduction
Hadoop Distributions
Microsoft Azure HDInsight
Microsoft BI and Data Platform
HDInsight - Use Cases
HDInsight - Typical Implementation
Further Learning

3. 3
Introduction
What is Big Data?
“Big Data is a collection
of data sets so large and
complex that it becomes
difficult to process using
on-hand database
management tools or
traditional data processing
applications”.

4. 4
Introduction
Hadoop is an open source
framework, from Apache foundation,
capable of processing very large
volumes of heterogeneous data sets
in a distributed fashion across clusters
of commodity computers and
hardware using a simplified
programming model.
What is Hadoop?

5. 5
Introduction
Conclusion
In simple terms, Big Data is the Challenge and Hadoop is the Solution.

6. 6
Hadoop Distributions
Amazon Elastic
Map Reduce
(EMR)
Cloudera Hortonworks
IBM
InfoSphere
BigInsights
MapR
Pivotal Teradata Intel
Azure
HDInsight
Reference: How the 9 Leading Commercial Hadoop Distributions Stack Up

7. 7
Which Distribution Should I Use?
Cost
Scalability
Availability
Existing Technology Stack
Existing Infrastructure
Existing Skillset

8. 8
HDInsight - Overview
Microsoft’s
Hadoop
Distribution in
the Cloud
Offers Hadoop
on Windows
Platform
Based on
Hortonworks
Data Platform
(HDP)
Tightly
integrated
with Microsoft
Technology
Stack

9. 9
HDInsight - Architecture

10. 10
Microsoft Data Platform and Enterprise BI Ecosystem

11. 11
Why HDInsight?
Microsoft Stack
Runs on Windows
Create & Destroy
On-Demand
DFS Implementation
in Blob Storage
DFS Implementation
in Blob Storage
Store data on Blob
Storage for Later Use
Automation using
PowerShell
Orchestration/Work
flow using SSIS
Scheduling using
SQL Agent
BI & Analytics with
Power BI

12. 12
Considerations
Requires dropping and
re-creating the cluster to
scale-up/down
Storage and Cluster should be in
the same Data Center

13. 13
HDInsight Versions
COMPONENT VERSION 1.6 VERSION 2.1 VERSION 3.0
VERSION 3.1
(Current/Default)
Hortonworks Data Platform (HDP) 1.1 1.3 2.0 2.1.7
Apache Hadoop & YARN 1.0.3 1.2.0 2.2.0 2.4.0
Tez 0.4.0
Apache Pig 0.9.3 0.11.0 0.12.0 0.12.1
Apache Hive & HCatalog 0.9.0 0.11.0 0.12.0 0.13.1
HBase 0.98.0
Apache Sqoop 1.4.2 1.4.3 1.4.4 1.4.4
Apache Oozie 3.2.0 3.3.2 4.0.0 4.0.0
Apache HCatalog 0.4.1 Merged with Hive Merged with Hive Merged with Hive
Apache Templeton 0.1.4 Merged with Hive Merged with Hive Merged with Hive
Ambari API v1.0 1.4.1 >=1.5.1
Zookeeper 3.4.5 3.4.5
Storm 0.9.1
Mahout 0.9.0
Phoenix 4.0.0.2.1.7.0-2162

14. 14
HDInsight Use Case - Iterative Exploration

15. 15
HDInsight Use Case - Data Warehouse on Demand

16. 16
HDInsight Use Case - ETL Automation

17. 17
HDInsight Use Case - BI Integration

18. 18
Typical Implementation
Transactional
Social
Warehouse
Azure
Blob
Blob Blob
Blob Blob
Multi-Node
HDInsight Cluster
MapReduce
• Hive
• Java
Reporting and Analytics
• SSRS
• Excel
• Power BI
Web Logs
Clickstream
Files
(TXT, XML, JSON, ..)
Collaboration
Office 365 / SharePoint

19. 19
Typical Implementation (Contd…)
E-CommerceInternalSystems
OLTP
Transactional
Internal Systems
Customers
Internal Systems
Team
Sqoop
Or AzCopy
Hive Metastore
MapReduce
Hive
Multi-Node
HDInsight Cluster
MapReduce
• Hive
• Pig
• Java
• Python
Collaboration, Reporting, and Analytics• SSRS
• Excel
• Power BI
PowerShell / SSIS / SQL Agent
Subscription & Cluster Management | Data Movement | Job Execution
Warehouse
Web Logs
Social
Web Logs
Azure
Blob Storage
Blob
Blob Blob
Blob
Blob
Blob
Blob

20. 20
Further Reading and Learning Resources
• HDInsight Emulator
• http://azure.microsoft.com
• Learning map for HDInsight: http://azure.microsoft.com/en-us/documentation/articles/hdinsight-learn-map

21. 21
References
• http://msdn.microsoft.com/en-us/library/dn749804.aspx
• http://azure.microsoft.com/en-us/documentation/articles/hdinsight-
component-versioning/
• http://msdn.microsoft.com/en-us/library/dn749848.aspx
• http://msdn.microsoft.com/en-us/library/dn749787.aspx
• http://msdn.microsoft.com/en-us/library/dn749805.aspx
• http://msdn.microsoft.com/en-us/library/dn749876.aspx

22. 22
Related Apache Projects
Term Description
Ambari / HUE Deployment, Configuration, and Monitoring
Avro / Parquet / RC / Sequence Data serialization system
Flume / S4 / Storm Collection and import of log and event data
Hbase / Cassandra Column-oriented database scaling to billions of rows
HCatalog Schema and Data Type Sharing over Pig, Hive, and MapReduce
Hive / Drill / Impala Data Warehouse with SQL-Like Access
Hive-QL/HQL SQL-Like Language to Query Hive
Mahout Library of machine learning and data mining algorithms
Pig High-level programming for Hadoop computations
Oozie Orchestration and workflow management
Sqoop Imports data from relational databases
Tez Application framework for graph
Whirr Cloud-agnostic deployment of clusters
MapReduce / YARN
MapReduce is a programming model for distributed data processing. MapReduce has undergone a
complete overhaul in hadoop-0.23 and we now have Map-Reduce 2.0 (MRv2) or YARN.
Zookeeper Configuration management and coordination

23. THANK YOU

24. 24
Top 10
Mobile Companies
Top 5
Outsourced Product Development Companies
2012 Partner of the year
Windows Azure, Finalist
40
GLOBAL OFFICES
7500
EMPLOYEES
23
COUNTRIES
Excellence Award
Technology Agency of the Year