Downloaded 30 times
![Capability Interactive Query Spark SQL Presto
Interactive Query Speed High High Medium
Scale High High Low
Caching Yes Yes Early Support
Intelligent Cache Eviction Yes No No
Complex Fact to Fact Joins Yes Yes No
Transactions Yes No No
Query Concurrency High Low Low
Row , Column level security Yes [Apache Ranger+ AAD] High Medium
Rich end user Tools Yes Yes Yes
Language Support SQL, UDF SQL, Scala, Python SQL
Data Source Connector
Support
Storage Handlers Data Sources High number of
connectors](https://image.slidesharecdn.com/zeroetlanalyticsllapazurehdinsight1-180427211635/75/Zero-ETL-analytics-with-LLAP-in-Azure-HDInsight-33-2048.jpg)
Uploaded byDataWorks Summit
Zero ETL analytics with LLAP in Azure HDInsight
The document discusses the integration of Azure HDInsight with Azure Blob Storage and Azure Data Lake Store for processing and serving data. It details the architecture, capabilities, and performance benchmarks of various clusters including LLAP, Spark, and Presto, along with an illustrative pricing estimate for using Azure services. Additionally, it highlights the security features and monitoring tools available for effective data management within the platform.
More Related Content
Similar to Zero ETL analytics with LLAP in Azure HDInsight
More from DataWorks Summit
Zero ETL analytics with LLAP in Azure HDInsight
- 2.
- 3.
- 4.
- 5.
- 6.
- 8.
- 9.
- 10. HDInsight cluster Azure BLOBStore/ Azure Data Lake Store Network @ashishth
- 11. Network HDInsight cluster AzureBLOB Store/ Azure Data Lake Store @ashishth
- 12. Network Hadoop Cluster Azure BLOB Store/ AzureData Lake Store Spark Cluster LLAP Cluster Presto Cluster @ashishth
- 13.
- 14. On Demand ProcessingClusters Data Serving Clusters Azure BLOB Store/ Azure Data Lake Store Common Hive Metastore @ashishth
- 15. Storage Storage HDInsight Spark/Hive/MRcluster 1. Create cluster 2. Submit jobs 6. Drop cluster jobs @ashishth
- 16. Azure BLOB Store/ AzureData Lake Store Common Hive Metastore @ashishth
- 17.
- 18.
- 19. Ingest Transform Convert to ORC/Parquet Load to Relational Store Serve @ashishth
- 20. Ingest Transform Convert to ORC/Parquet Load to Relational Store Serve Time @ashishth
- 21.
- 22.
- 23.
- 24.
- 25. • Hive LowLatency and Analytical Processing (LLAP) • Serves queries directly from Azure BLOB/ADLS • Works with TEXT, JSON, CSV, TSV, ORC, Parquet • Super fast performance with TEXT data • Modern scalable query concurrency architecture • Security with Apache Ranger and Active Directory @ashishth
- 26. HDInsight Interactive Queryarchitecture Memory + SSD cache @ashishth
- 27. Intelligent cache Automatically reactsto changes in underlying data o Shared cache between queries o Cache eviction is based on source file last modified date o Every query will check modified date, and reload if a new file has arrived DRAM SSD ADLS/BLOBStore Updates @ashishth
- 28. • LLAP, Spark,and Presto against 1 TB derived from the TPC-DS benchmark • Out of the box HDInsight Configuration • 45 queries derived from TPC-DS benchmark that ran on all engines successfully @ashishth
- 29.
- 31. • We usednumber of different concurrency levels to test the concurrency performance • 99 queries on 1 TB data with 32 worker node cluster with max concurrency set to 32. Test 1: Run all 99 queries, 1 at a time - Concurrency = 1 Test 2: Run all 99 queries, 2 at a time - Concurrency = 2 Test 3: Run all 99 queries, 4 at a time - Concurrency = 4 Test 4: Run all 99 queries, 8 at a time - Concurrency = 8 Test 5: Run all 99 queries, 16 at a time - Concurrency = 16 Test 6: Run all 99 queries, 32 at a time - Concurrency = 32 Test 7: Run all 99 queries, 64 at a time - Concurrency = 64 @ashishth
- 32.
- 33. Capability Interactive QuerySpark SQL Presto Interactive Query Speed High High Medium Scale High High Low Caching Yes Yes Early Support Intelligent Cache Eviction Yes No No Complex Fact to Fact Joins Yes Yes No Transactions Yes No No Query Concurrency High Low Low Row , Column level security Yes [Apache Ranger+ AAD] High Medium Rich end user Tools Yes Yes Yes Language Support SQL, UDF SQL, Scala, Python SQL Data Source Connector Support Storage Handlers Data Sources High number of connectors
- 34. Microsoft Azure Estimate YourEstimate Service type Custom name Region Description Estimated Cost HDInsight East US Interactive Query Component: 2 A3 (4 cores, 7 GB RAM) Head nodes x 730 Hours, 6 D14V2 (16 cores, 112 GB RAM) Region nodes x 730 Hours, 3 A1 (1 cores, 1.75 GB RAM) Zookeeper nodes x 730 Hours, 0 D4V2 (8 cores, 28 GB RAM) Edge nodes x 730 Hours $7,163.27 Storage East US Block Blob Storage, General Purpose V2, LRS Redundancy, Hot Access Tier, 100 TB Capacity, 10,000,000 Write operations, 100,000 List and Create Container Operations, 99,999,000 Read operations, 9,990,000 Other operations. 500 TB Data Retrieval, 50 TB Data Write $2,181.82 Support Support $0.00 Monthly Total $9,345.09 Annual Total $112,141.06 Disclaimer All prices shown are in US Dollar ($). This is a summary estimate, not a quote. For up to date pricing information please visit https://azure.microsoft.com/pricing/calculator/ This estimate was created at 4/13/2018 7:48:34 PM UTC. @ashishth
- 40.
- 50. OMS Agent for Linux HDInsightnodes (Head, Worker , Zookeeper ) FluentD HDInsight plugin 1. Plugin for ‘in_tail’ for all Logs, allows regexp to create JSON object 2. Filter for WARN and above for each Log Type. `grep` filter plugin 3. Output to out_oms_api Type 4. Exec plugin for Metrics HBaseConfigosmconfig Spark Hive/ LLAP Storm Kafka Config Config Config Config Log Analytics(OMS) Service HDInsight Log Analytics Architecture
- 51.
- 52.