4. Devices
&
Sensors
Speed
Layer
Data Lake Store Gen 2
Blob
Storage
Corporate
Data
SaaS
Data
Web
Data
Streaming/Real-
Time/
Application
Advanced Analytics
& Data Science
Machine Learning
R, Python, APIs
Analytics
Data Exploration
Corporate
Reporting
Self-Service BI
ETL Serving LayerStorage
Hive LLAP
@ashishth
5. Devices
&
Sensors
Speed
Layer
Data Lake Store Gen 2
Blob
Storage
Corporate
Data
SaaS
Data
Web
Data
Streaming/Real-
Time/
Application
Advanced Analytics
& Data Science
Machine Learning
R, Python, APIs
Analytics
Data Exploration
Corporate
Reporting
Self-Service BI
ETL Serving LayerStorage
Hive LLAP
?
?
?
?
@ashishth
7. Spark Pig Hive
Designed for ETL ETL Data warehousing
Adoption High, increasing Low, decreasing Stable
Number of connectors Highest High High
Languages Python, R, Scala, Java, SQL Pig SQL
Performance High Medium Medium
@ashishth
8. Spark Structured Streaming Storm
Adoption High, increasing Decreasing
Event processing guarantee Exactly once At least once
Throughput High Low
Processing Model Micro Batch Real-Time
Latency High Low
Event time support Yes Yes
Languages Python, R, Scala, Java,
SQL
Java
@ashishth
9. Capability Hive LLAP Spark SQL Presto
Interactive Query Speed High High Medium
Scale High High Low
Caching Yes Yes Early Support
Result Caching Yes No No
Intelligent Cache
Eviction
Yes No No
Materialized Views Yes No No
Complex Fact to Fact Joins Yes Yes No
Transactions Yes No No
Query Concurrency High Low Low
Row , Column level
security
Yes [Apache Ranger+ AAD] Medium Medium
Rich end user Tools Yes Yes Yes
Language Support SQL, UDF SQL, Scala, Python SQL
Data Source Connector
Support
Storage Handlers Data Sources High number of
connectors @ashishth
11. ADF Airflow Oozie
Service management Azure PaaS IaaS VM HDInsight
Code JSON Python Java
GUI ADF V2 has great UX Good UX Below Average UX
Community Microsoft Growing (10893 Stars) Declining (454 Stars)
On-demand clusters Yes No, but extensible No
Extensibility Custom action-only Full, graph + actions Custom action-only
Pipeline definition JSON/UX Python/ UX XML/UX
Devops-first design Yes Yes Yes
Pipeline monitoring Yes Yes Yes
Scheduling Event, Time Event Event, Time
@ashishth
12. Motivation and benefits
Architecture best practices
Infrastructure best practices
Storage best practices
Data migration best practices
Security and DevOps best practices
https://azure.microsoft.com/en-us/blog/migrating-on-premises-hadoop-infrastructure-to-azure-hdinsight/
@ashishth
13. Data
Sources
Apps
Sensors
and
devices
Data Ingestion Advanced Analytics BI/ Visualization
People
Automated
Systems
Apps
Web
Mobile
Bots
Data catalog/ Governance/ Lineage
Connectors: JDBC, ODBC
Productivity Tools
Enterprise grade add-ons (hybrid, backup, DR, security, performance)
Data Prep/
Management
@ashishth
16. Data Qty Network Bandwidth
45 Mbps (T3) 100 Mbps 1 Gbps
1 TB 2 days 1 day 2 hours
10 TB 22 days 10 days 1 day
35 TB 76 days 34 days 3 days
80 TB 173 days 78 days 8 days
100 TB 216 days 97 days 10 days
200 TB 1 year 194 days 19 days
500 TB 3 years 1 year 49 days
1 PB 6 years 3 years 97 days
2 PB 12 years 5 years 194 days
@ashishth
17. Network Transfer with TLS
• Over Internet
• Express Route
• Databox online Transfer
Shipping data offline
• Import / Export service
• Data Box offline data transfer
@ashishth
20. Type Latency Consistency Workloads Bandwidth Key Benefits
ADLS Gen 1 Hierarchical 10-100ms Low HDInsight 3.6(
No HBase)
High Atomic Rename,
File Folder level
ACL’s
ADLS Gen 2 Hierarchical 10-50ms Medium HDInsight 3.6 &
4.0
Unconstrained Atomic Rename,
File Folder level
ACL’s
Standard
BLOB
Object Store 10-50ms Medium HDInsight 3.6 &
4.0
Unconstrained Mature
Premium
BLOB
Object Store ~5ms High HBase in Preview Unconstrained Fast
Premium
Managed
Disks
Hierarchical ~5ms High Kafka, HBase in
preview
Based on disk Consistent
@ashishth
21. Scenario Supported Workaround
HDInsight 3.6 & 4.0 with Standard Blob as Primary
and/ or secondary
Yes
HDInsight 3.6 & 4.0 with ADLS Gen2 as primary Yes
HDInsight 3.6 & 4.0 with ADLS Gen2 as primary &
Blob as additional
Yes
HDInsight 3.6 & 4.0 with Blob as primary & ADLS
Gen2 as additional
No
HDInsight 3.6 with multiple ADLS Gen2 accounts Yes
HDInsight 3.6 & 4.0 with ADLS Gen1 and ADLS Gen 2 No Distcp across two
clusters
HDInsight 4.0 with ADLS Gen 1 No Distcp across two
clusters
@ashishth
32. Workload Caching Options Key benefits
Spark Spark IO Cache Up to ~8 to 10x perf improvements
HBase &
Phoenix
Bucket cache Up 5-10x perf gains on recently read or written
data
Hive + LLAP LLAP Intelligent cache/Result Cache Up to ~4-100X gain on cached data
33. Azure Data Lake Storage
INSTANCE CORE RAM TEMP SSD
D1 v2 1 3.50 GiB 50 GiB
D2 v2 2 7.00 GiB 100 GiB
D3 v2 4 14.00 GiB 200 GiB
D4 v2 8 28.00 GiB 400 GiB
D5 v2 16 56.00 GiB 800 GiB
• Significant Spark performance speed up
with IO cache (up to 9X perf gains)
• Automatic cache resource management
• DRAM + Temp SSD makes large cache
pool
@ashishth
Azure HDInsight is a secure and managed platform for building data lakes on Azure based on the Apache Hadoop and Spark frameworks. So, what all does HDInsight have to offer?
Reliable Open Source analytics with an Industry leading SLAHDInsight allows you to easily spin up open source cluster types guaranteed with the industry’s best 99.9% SLA and 24/7 support. We guarantee this SLA for the entire big data solution, not just the VM instances. HDInsight is architected for full redundancy and high availability including head node replication, data geo-replication, and built-in standby NameNode making HDInsight resilient to critical failures not addressed in standard Hadoop implementations. Azure also offers cluster monitoring and 24x7 enterprise support backed by Microsoft and Hortonworks with 37 combined committers for Hadoop core, more than all other managed cloud providers combined to support your deployment and the ability to fix and commit code back to Hadoop.
Enterprise Grade Security & MonitoringHDInsight protects your data assets and easily extends your on-premise security and governance controls to the cloud. We feature single sign-on (SSO), multi-factor authentication and seamless management of millions of identities through Azure Active Directory. You can authorize users and groups with fine-grained access control policies over all your enterprise data with Apache Ranger. HDInsight meets HIPAA, PCI, SOC compliance, ensuring your enterprise data assets are always protected with the highest security and regulatory compliance. To ensure the highest level of business continuity, HDInsight extends capabilities for alerting, monitoring, defining pre-emptive actions, and enhanced workload protection through native integration with Azure Operations Management Suite (OMS).
Most Productive platform for developers and scientists HDInsight offers developers tailored experiences through rich productivity suites for Hadoop & Spark with integrated development environments using Visual Studio, Eclipse, and IntelliJ supporting Scala, Python, R, Java, and .Net. HDInsight gives data scientists the ability to create narratives that combine code, statistical equations, and visualizations that tell a story about the data through integration to the two most popular notebooks: Jupyter and Zeppelin. HDInsight is also the only managed cloud Hadoop solution with integration to Microsoft R Server. Multi-threaded math libraries and transparent parallelization in R Server means handling up to 1000x more data and up to 50x faster speeds than open source R—helping you train more accurate models for better predictions than previously possible.
Cost effective cloud scaleHDInsight has decoupled compute and storage, enabling you to cost-effectively scale workloads up or down, independent of storage. Local storage can still be used for caching and fast I/O. Spark and interactive Hive users can choose SSD memory for interactive performance; while Kafka users can retain all streaming data in premium managed disks. You only pay for the compute and storage you use and are given the ability to choose any Azure VM types that enables the best utilization of resources. A recent study showed HDInsight delivering 63% lower TCO than deploying Hadoop on premises over 5 years.*
Integration with leading Productivity ApplicationsIn the broader ecosystem for Hadoop, there is a thriving market of independent software vendors (ISVs) who provide value added solutions. Through a unique design where every cluster is extended with edge nodes and script action, HDInsight lets customers spin up Hadoop and Spark clusters pre-integrated and pre-tuned with any ISV application out-of-the-box. Datameer, Cask, AtScale, StreamSets are few such applications, which are very popular on the HDInsight platform today.
Easy for administrators to manageWith HDInsight, administrators can deploy Hadoop in the cloud without buying new hardware or incurring other up-front costs. There’s also no time-consuming installation or set up. There is also no need to patch the operating system or upgrade the Hadoop versions. Azure does it for you. Launch your first cluster in minutes.
Build 2015
The new world of HDInsight 4.0 with Hadoop 3.0, brings the Spark and Hive worlds closer together. Lets see, how…
Before Hadoop 3.0, the Spark executors would directly access the Hive metastore. While, on the surface, this seems like a fine thing to do, it is rife with problems.
The new architecture instead requires explicit registration of Hive transactional tables as Spark external tables through Hive Warehouse Connector. While it adds one extra step during configuration, this approach greatly increases the reliability of data access. Hive Warehouse Connector supports efficient predicate pushdown and Apache Arrow-based communication between Spark executors and Hive LLAP daemons. This results in overall small overhead of communication between two systems. With Hive Warehouse Connector, Apache Spark on HDInsight 4.0 gets mature transactional capabilities.
The new integration between Apache Spark and Hive LLAP in HDInsight 4.0 delivers new capabilities for business analysts, data scientists, and data engineers. Business analysts get a performant SQL engine in the form of Hive LLAP (Interactive Query) while data scientists and data engineers get a great platform for ML experimentation and ETL with Apache Spark over transactional data in Hive tables.
Transfer data over network with TLS
Over internet - You can transfer data to Azure storage over a regular internet connection using any one of several tools such as: Azure Storage Explorer, AzCopy, Azure Powershell, and Azure CLI. See Moving data to and from Azure Storage for more information.
Express Route - ExpressRoute is an Azure service that lets you create private connections between Microsoft datacenters and infrastructure that’s on your premises or in a colocation facility. ExpressRoute connections do not go over the public Internet, and offer higher security, reliability, and speeds with lower latencies than typical connections over the Internet. For more information, see Create and modify an ExpressRoute circuit.
Data Box online data transfer - Data Box Edge and Data Box Gateway are online data transfer products that act as network storage gateways to manage data between your site and Azure. Data Box Edge, an on-premises network device, transfers data to and from Azure and uses artificial intelligence (AI)-enabled edge compute to process data. Data Box Gateway is a virtual appliance with storage gateway capabilities. For more information, see Azure Data Box Documentation - Online Transfer.
Shipping data Offline
Import / Export service - you can send physical disks to Azure and they will be uploaded for you. For more information, see What is Azure Import/Export service?.
Data Box offline data transfer - Data Box, Data Box Disk, and Data Box Heavy devices help you transfer large amounts of data to Azure when the network isn’t an option. These offline data transfer devices are shipped between your organization and the Azure datacenter. They use AES encryption to help protect your data in transit, and they undergo a thorough post-upload sanitization process to delete your data from the device. For more information, see Azure Data Box Documentation - Offline Transfer.
Before I describe specific capabilities and value propositions of HDInsight, let us take a quick look at the architecture of a HDInsight cluster. We will build upon this when we talk about security later on in the presentation.
First off, a key difference between an on-premise Hadoop cluster and a HDInsight cluster is that with HDInsight, the storage and compute layers are separated. This allows for storage and compute to be scaled independently of each other. We have seen in numerous customer cases, that trying to combine storage and compute on to a single cluster often leads to underutilization of one or the other or both. With HDInsight, you can keep loading data in to Azure Storage Gen1 or Gen2 or in WASB. And you can create small or large clusters as and when needed.
Each HDInsight cluster comes with 2 gateway nodes, 2 head nodes and 3 ZooKeeper nodes. In most cases, these are free of charge. As we will discuss later, we provision multiple of these nodes to ensure high availability.
Each HDInsight cluster lives within a VNET. The gateway nodes are the ONLY public endpoints accessible from outside the VNET. As we will see later, this architecture allows you to securely lock down your HDInsight cluster.
Let’s start with network security. Previously you could have injected a HDInsight cluster within a VNET and secured access to it from the public internet using NSG firewalls. Now you can ensure that any resources that the cluster needs to accesss e.g. Azure Storage accounts, Hive metastores etc. can themselves be secured. With the new service endpoint capability, Azure resources such as Azure Storage, Azure DB, Cosmos DB etc. can be secured via service endpoints. HDInsight now integrates with this capability.
Let me walk you through how this would work. [WALK THROUGH THE ANIMATION]