Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building Big Data Applications using Spark, Hive, HBase and Kafka
Similar to Building Big Data Applications using Spark, Hive, HBase and Kafka (20)
More from Ashish Thapliyal
More from Ashish Thapliyal (12)
Recently uploaded
Recently uploaded (20)
Building Big Data Applications using Spark, Hive, HBase and Kafka
- 2. @ashishth
- 3. @ashishth
- 4. hot path cold path Serving-layer data sources consumers Governance HDFS Compliant Storage (Data Lake) Meta data Management Security / Access Control Ingest real-time data Real Time NOSQL Store ETL Ingest batch data AdHoc Query in DataLake Downstream Applications Store real-time data for long term analysis Orchestration Corporate Data Devices & Sensors Advanced Analytics & Data Science Machine Learning R, Python, APIs Analytics Data Exploration Corporate Reporting Self-Service BI Streaming/Real- Time/ Application Stream Processing @ashishth
- 5. AZURE SDK AZURE DATA FACTORY AZURE IMPORT EXPORT SERVICE AZURE CLI COGNITIVE SERVICESBOT SERVICE AZURE SEARCH AZURE DATA CATALOG AZURE EXPRESSROUTE AZURE NETWORK SECURITY GROUPS AZURE FUNCTIONS VISUAL STUDIOOPERATIONS MANAGEMENT SUITE AZURE ACTIVE DIRECTORY AZURE KEY MANAGEMENT SERVICE AZURE STORAGE BLOBS AZURE DATA LAKE STORAGE AZURE IOT HUB AZURE EVENT HUBS KAFKA ON AZURE HDINSIGHT AZURE SQL DATA WAREHOUSEAZURE SQL DB AZURE COSMOS DB AZURE ANALYSIS SERVICES POWER BI AZURE HDINSIGHT AZURE DATABRICKS AZURE STREAM ANALYTICS AZURE ML ML SERVER AZURE DATABRICKS @ashishth
- 6. @ashishth
- 7. ⢠The most trusted and compliant platform Azure HDInsight A secure and managed Apache Hadoop and Spark platform for building data lakes in the Cloud @ashishth
- 8. Monitoring & Security Presto or Hive LLAP? Which storage system? How to Transfer the Data ADF/Airflow or Oozie? Pig, Hive or Spark Spark Streaming or Storm ETL Serving Layer Storage Orchestration Event Processing @ashishth
- 9. @ashishth
- 10. Pig Designed for ETL ETL Data warehousing Adoption High, increasing Low, decreasing Stable Number of connectors Highest High High Languages Python, R, Scala, Java, SQL Pig SQL Performance High Medium Medium @ashishth
- 11. Spark Structured Streaming Storm Adoption High, increasing Decreasing Event processing guarantee Exactly once At least once Throughput High Low Processing Model Micro Batch Real-Time Latency High Low Event time support Yes Yes Languages Python, R, Scala, Java, SQL Java @ashishth
- 12. Capability Hive LLAP Interactive Query Speed High High Medium Scale High High Low Caching Yes Yes Early Support Result Caching Yes No No Intelligent Cache Eviction Yes No No Materialized Views Yes No No Complex Fact to Fact Joins Yes Yes No Transactions Yes No No Query Concurrency High Low Low Row , Column level security Yes [Apache Ranger+ AAD] Medium Medium Rich end user Tools Yes Yes Yes Language Support SQL, UDF SQL, Scala, Python SQL Data Source Connector Support Storage Handlers Data Sources connectors @ashishth
- 13. @ashishth
- 14. Hive Metadata Spark Metadata Hive Metadata Azure HDInsight 3.6 with Hadoop 2.6 Azure HDInsight 4.0 with Hadoop 3.x Hive Metastore migration tool: https://azure.microsoft.com/en-us/blog/hdinsight-metastore-migration-tool- open-source-release-now-available/ @ashishth
- 15. ADF Airflow Oozie Service management Azure PaaS IaaS VM HDInsight Code JSON Python Java GUI ADF V2 has great UX Good UX Below Average UX Community Microsoft Growing (12,133 Stars) Declining (483 Stars) On-demand clusters Yes No, but extensible No Extensibility Custom action-only Full, graph + actions Custom action-only Pipeline definition JSON/UX Python/ UX XML/JAVA/UX Devops-first design Yes Yes Yes Pipeline monitoring Yes Yes Yes Scheduling Event, Time Event Event, Time @ashishth
- 16. @ashishth
- 18. Data Qty Network Bandwidth 45 Mbps (T3) 100 Mbps 1 Gbps 1 TB 2 days 1 day 2 hours 10 TB 22 days 10 days 1 day 35 TB 76 days 34 days 3 days 80 TB 173 days 78 days 8 days 100 TB 216 days 97 days 10 days 200 TB 1 year 194 days 19 days 500 TB 3 years 1 year 49 days 1 PB 6 years 3 years 97 days 2 PB 12 years 5 years 194 days @ashishth
- 19. Network Transfer with TLS ⢠Over Internet ⢠Express Route ⢠Data Box online Transfer Shipping data offline ⢠Data Box offline data transfer @ashishth
- 20. USB 3.1 SSD disks Order up to 5 in each pack Ruggedized, self-contained appliances 100 TB 8 TB, up to 40 TB 1 PB @ashishth Use Azure Data Box to migrate data from an on-premises HDFS store to Azure Storage
- 21. Type Latency ( Consistency of latency) Workloads Bandwidth Key Benefits ADLS Gen 2 Hierarchical 10-50ms (Medium) HDInsight 3.6 & 4.0 Unconstrained Atomic Rename, File Folder level ACLâs Standard BLOB Object Store 10-50ms (Medium) HDInsight 3.6 & 4.0 Unconstrained Mature Premium BLOB Object Store ~5ms (High) HBase in Preview Unconstrained Fast Premium Managed Disks Hierarchical ~5ms (High) Kafka, HBase in preview Based on disk Consistent latency ADLS Gen 1 Hierarchical 10-100ms (Low) HDInsight 3.6( No HBase) High Atomic Rename, File Folder level ACLâs @ashishth
- 22. @ashishth
- 24. RegionServer Storage Client -Put -Update -Get -Delete Log Flusher Remote store write path challenges with Write Ahead Log Insert Update Get Delete Sync Operation ⢠Inconsistent Latencies ⢠High latencies @ashishth
- 25. RegionServer Premium Managed Disk(s) Client -Put -Update -Get -Delete Log Flusher Insert Update Get Delete Sync Operation Introducing Premium Managed disk for WAL ⢠Consistent Latencies ⢠Low latencies ⢠Data Durability @ashishth
- 26. RegionServer Client -Put -Delete -Get Region Region Region Log Flusher Memstore HFile Memstore HFile Memstore HFile Low latency workload HBase/ Small write @ashishth Storage Premium Managed Disk(s)
- 27. @ashishth
- 28. @ashishth
- 30. @ashishth
- 31. Workload Caching Options Key benefits Spark Spark IO Cache Up to ~8 to 10x perf improvements HBase & Phoenix Bucket cache Up 5-10x perf gains on recently read or written data Hive + LLAP LLAP Intelligent cache/Result Cache Up to ~4-100X gain on cached data @ashishth
- 32. Azure Data Lake Storage INSTANCE CORE RAM TEMP SSD D1 v2 1 3.50 GiB 50 GiB D2 v2 2 7.00 GiB 100 GiB D3 v2 4 14.00 GiB 200 GiB D4 v2 8 28.00 GiB 400 GiB D5 v2 16 56.00 GiB 800 GiB ⢠Significant Spark performance speed up with IO cache (up to 9X perf gains) ⢠Automatic cache resource management ⢠DRAM + Temp SSD makes large cache pool @ashishth@ashishth
- 33. @ashishth
- 34. PERIMETER Isolate clusters within VNETs Service Endpoint support for WASB, Azure DB, Cosmos DB Restrict outbound traffic using NVAs* AUTHENTICATION Azure Active Directory Kerberos with Active Directory AUTHORIZATION Role-Based Access Control Apache Ranger based Access Control DATA PROTECTION Encryption on-the-wire with HTTPS enforced Encryption at Rest using Azure Key Vault Auditing of all data operations and configuration changes @ashishth
- 35. @ashishth
- 36. @ashishth
- 37. Apache Ranger ADLS Gen 2 ACLs @ashishth
- 38. Scenario Authorizing Component Yarn: Submit-App Apache Ranger: Yarn Plugin Hive Operations: Select , Drop, index, Lock, Read, Write, Masking, Row level filter on Hive Database, Table & Columns Apache Ranger: Hive Plugin Create/ Alter Table with storage location reference Apache Ranger + ADLS Gen 2 ACLâs Spark SQL access with Hive Metastore Apache Ranger: Hive Plugin HBase Access Policies Apache Ranger/ HBase plugin Kafka Access Policies Apache ranger/ Kafka Plugin Access Azure Data Lake Storage Gen2 using the Spark DataFrame API ADLS Gen 2 ACLs Access Azure Data Lake Storage Gen2 using the RDD API ADLS Gen 2 ACLs HDFS operations: Mkdir, ls, put, copyFromLocal, get, cat, mv, cp etc ADLS Gen 2 ACLs Running Map Reduce jobs ADLS Gen 2 ACLs @ashishth
- 39. @ashishth
- 40. @ashishth
- 41. ⢠hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode get # A report that shows the ⢠details of HDFS state: hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -report # Get HDFS ⢠out of safe mode hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave # Get ⢠HDFS into safe mode hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode enter @ashishth
- 42. SetupAutoscale Customize to your own scenario Pay for ONLY what you need Monitoring scaling history easily Graceful Scale Down @ashishth
- 43. HDInsight Cluster Gateways Head Node 1 Head Node 2 Worker Node Worker Node Worker Node Worker Node Zookeeper1 Zookeeper1 Zookeeper1 Hive Metastore YARN https://cluster.azurehdinsight.net/APIs @ashishth
- 44. Workload DR Option Spark / Hive Manual, Partner solution HBase HBase replication, Snapshot export, Import Export, Copy Tables Kafka Mirror Maker @ashishth https://github.com/anagha-microsoft/hdi-spark-dr https://github.com/anagha-microsoft/hdi-kafka-dr https://docs.microsoft.com/en- us/azure/hdinsight/hbase/apache-hbase-backup-replication
- 45. @ashishth
- 46. Apache Ambari Azure Log Analytics IntegrationHDInsight Cluster Metrics @ashishth
- 47. @ashishth
- 48. Motivation and benefits Architecture best practices Infrastructure best practices Storage best practices Data migration best practices Security and DevOps best practices https://azure.microsoft.com/en-us/blog/migrating-on-premises-hadoop-infrastructure-to-azure-hdinsight/ @ashishth
Editor's Notes
- Azure HDInsight is a secure and managed platform for building data lakes on Azure based on the Apache Hadoop and Spark frameworks. So, what all does HDInsight have to offer? Reliable Open Source analytics with an Industry leading SLA HDInsight allows you to easily spin up open source cluster types guaranteed with the industryâs best 99.9% SLA and 24/7 support. We guarantee this SLA for the entire big data solution, not just the VM instances. HDInsight is architected for full redundancy and high availability including head node replication, data geo-replication, and built-in standby NameNode making HDInsight resilient to critical failures not addressed in standard Hadoop implementations. Azure also offers cluster monitoring and 24x7 enterprise support backed by Microsoft and Hortonworks with 37 combined committers for Hadoop core, more than all other managed cloud providers combined to support your deployment and the ability to fix and commit code back to Hadoop. Enterprise Grade Security & Monitoring HDInsight protects your data assets and easily extends your on-premise security and governance controls to the cloud. We feature single sign-on (SSO), multi-factor authentication and seamless management of millions of identities through Azure Active Directory. You can authorize users and groups with fine-grained access control policies over all your enterprise data with Apache Ranger. HDInsight meets HIPAA, PCI, SOC compliance, ensuring your enterprise data assets are always protected with the highest security and regulatory compliance. To ensure the highest level of business continuity, HDInsight extends capabilities for alerting, monitoring, defining pre-emptive actions, and enhanced workload protection through native integration with Azure Operations Management Suite (OMS). Most Productive platform for developers and scientists HDInsight offers developers tailored experiences through rich productivity suites for Hadoop & Spark with integrated development environments using Visual Studio, Eclipse, and IntelliJ supporting Scala, Python, R, Java, and .Net. HDInsight gives data scientists the ability to create narratives that combine code, statistical equations, and visualizations that tell a story about the data through integration to the two most popular notebooks: Jupyter and Zeppelin. HDInsight is also the only managed cloud Hadoop solution with integration to Microsoft R Server. Multi-threaded math libraries and transparent parallelization in R Server means handling up to 1000x more data and up to 50x faster speeds than open source Râhelping you train more accurate models for better predictions than previously possible. Cost effective cloud scale HDInsight has decoupled compute and storage, enabling you to cost-effectively scale workloads up or down, independent of storage. Local storage can still be used for caching and fast I/O. Spark and interactive Hive users can choose SSD memory for interactive performance; while Kafka users can retain all streaming data in premium managed disks. You only pay for the compute and storage you use and are given the ability to choose any Azure VM types that enables the best utilization of resources. A recent study showed HDInsight delivering 63% lower TCO than deploying Hadoop on premises over 5 years.* Integration with leading Productivity Applications In the broader ecosystem for Hadoop, there is a thriving market of independent software vendors (ISVs) who provide value added solutions. Through a unique design where every cluster is extended with edge nodes and script action, HDInsight lets customers spin up Hadoop and Spark clusters pre-integrated and pre-tuned with any ISV application out-of-the-box. Datameer, Cask, AtScale, StreamSets are few such applications, which are very popular on the HDInsight platform today. Easy for administrators to manage With HDInsight, administrators can deploy Hadoop in the cloud without buying new hardware or incurring other up-front costs. Thereâs also no time-consuming installation or set up. There is also no need to patch the operating system or upgrade the Hadoop versions. Azure does it for you. Launch your first cluster in minutes.
- The new world of HDInsight 4.0 with Hadoop 3.0, brings the Spark and Hive worlds closer together. Lets see, how⌠Before Hadoop 3.0, the Spark executors would directly access the Hive metastore. While, on the surface, this seems like a fine thing to do, it is rife with problems. The new architecture instead requires explicit registration of Hive transactional tables as Spark external tables through Hive Warehouse Connector. While it adds one extra step during configuration, this approach greatly increases the reliability of data access. Hive Warehouse Connector supports efficient predicate pushdown and Apache Arrow-based communication between Spark executors and Hive LLAP daemons. This results in overall small overhead of communication between two systems. With Hive Warehouse Connector, Apache Spark on HDInsight 4.0 gets mature transactional capabilities.â The new integration between Apache Spark and Hive LLAP in HDInsight 4.0 delivers new capabilities for business analysts, data scientists, and data engineers. Business analysts get a performant SQL engine in the form of Hive LLAP (Interactive Query) while data scientists and data engineers get a great platform for ML experimentation and ETL with Apache Spark over transactional data in Hive tables.â
- Reference https://azure.microsoft.com/en-us/blog/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines/
- Build 2015
- Transfer data over network with TLS Over internet - You can transfer data to Azure storage over a regular internet connection using any one of several tools such as: Azure Storage Explorer, AzCopy, Azure Powershell, and Azure CLI. See Moving data to and from Azure Storage for more information. Express Route - ExpressRoute is an Azure service that lets you create private connections between Microsoft datacenters and infrastructure thatâs on your premises or in a colocation facility. ExpressRoute connections do not go over the public Internet, and offer higher security, reliability, and speeds with lower latencies than typical connections over the Internet. For more information, see Create and modify an ExpressRoute circuit. Data Box online data transfer - Data Box Edge and Data Box Gateway are online data transfer products that act as network storage gateways to manage data between your site and Azure. Data Box Edge, an on-premises network device, transfers data to and from Azure and uses artificial intelligence (AI)-enabled edge compute to process data. Data Box Gateway is a virtual appliance with storage gateway capabilities. For more information, see Azure Data Box Documentation - Online Transfer. Shipping data Offline Import / Export service - you can send physical disks to Azure and they will be uploaded for you. For more information, see What is Azure Import/Export service?. Data Box offline data transfer - Data Box, Data Box Disk, and Data Box Heavy devices help you transfer large amounts of data to Azure when the network isnât an option. These offline data transfer devices are shipped between your organization and the Azure datacenter. They use AES encryption to help protect your data in transit, and they undergo a thorough post-upload sanitization process to delete your data from the device. For more information, see Azure Data Box Documentation - Offline Transfer.
- Azure Data Lake Storage Gen2 is Azureâs Storage platform for high performance analytics It is built on the strong foundation of Blob Storage which is Azureâs object storage platform and has been serving customers and various use cases (including some analytics use cases) for over a decade ADLS Gen2 is designed with native file system semantics and optimized for high performance analytics (example: rename folder operations which is very common in spark workloads is a single metadata operation as opposed to a large number for individual object operations) ADLS Gen2 also supports POSIX ACLâs which is an open source industry standard ADLS Gen2 currently only supports a small subset of the Blob capabilities (authentication and redundancy) but several of the other Blob capabilities will light up once we support âinteroperabilityâ â which is the ability to run multiple protocols on the same account. This is planned to roll out in waves throughout the calendar year Blob interoperability also lights up integrations such as ASA, Event Hubs etc since this was previously done for Blob storage. This includes SDKâs as the Blob SDKâs can be used on the account as opposed to writing brand new SDKâs. ADLS Gen2 is GA and available in all Azure regions and is the recommended storage platform for analytics pipelines
- Before I describe specific capabilities and value propositions of HDInsight, let us take a quick look at the architecture of a HDInsight cluster. We will build upon this when we talk about security later on in the presentation. First off, a key difference between an on-premise Hadoop cluster and a HDInsight cluster is that with HDInsight, the storage and compute layers are separated. This allows for storage and compute to be scaled independently of each other. We have seen in numerous customer cases, that trying to combine storage and compute on to a single cluster often leads to underutilization of one or the other or both. With HDInsight, you can keep loading data in to Azure Storage Gen1 or Gen2 or in WASB. And you can create small or large clusters as and when needed. Each HDInsight cluster comes with 2 gateway nodes, 2 head nodes and 3 ZooKeeper nodes. In most cases, these are free of charge. As we will discuss later, we provision multiple of these nodes to ensure high availability. Each HDInsight cluster lives within a VNET. The gateway nodes are the ONLY public endpoints accessible from outside the VNET. As we will see later, this architecture allows you to securely lock down your HDInsight cluster.