This document discusses HDInsight interactive query architecture and performance. It summarizes that: 1. HDInsight uses LLAP (Low Latency Analytical Processing) clusters to serve queries directly from Azure blob storage and data lake store for fast performance on text data. 2. Testing showed LLAP had high query concurrency and interactive query speed compared to Spark SQL and Presto. 3. The document also outlines HDInsight's logging architecture where the OMS agent collects logs and metrics from HDInsight clusters and sends them to Log Analytics for analysis.

2. @ashishth

4. LLAP

10. HDInsight cluster
Azure BLOB Store/
Azure Data Lake
Store
Network

11. Network
HDInsight cluster
Azure BLOB Store/
Azure Data Lake
Store

12. Network
Hadoop
Cluster
Azure BLOB Store/
Azure Data Lake
Store
Spark Cluster
LLAP Cluster
Presto Cluster

14. On Demand Processing Clusters
Data Serving Clusters
Azure BLOB Store/
Azure Data Lake
Store
Common Hive
Metastore

15. Storage Storage
HDInsight Spark/Hive/MR cluster
1. Create cluster
2. Submit jobs
6. Drop cluster jobs

16. Azure BLOB Store/
Azure Data Lake
Store
Common Hive
Metastore

17. Azure HDInsight
Analyst
Power
User
Data
Engineer
Data
Scientist

19. Ingest Transform
Convert to
ORC/ Parquet
Load to
Relational
Store
Serve

20. Ingest Transform
Convert to
ORC/ Parquet
Load to
Relational
Store
Serve
Time

23. Ingest Serve
Time

24. • Hive Low Latency and Analytical Processing (LLAP)
• Serves queries directly from Azure BLOB/ADLS
• Works with TEXT, JSON, CSV, TSV, ORC, Parquet
• Super fast performance with TEXT data
• Modern scalable query concurrency architecture
• Security with Apache Ranger and Active Directory

25. HDInsight Interactive Query architecture
Memory + SSD cache

26. Intelligent cache
Automatically reacts to changes in underlying data
o Shared cache between queries
o Cache eviction is based on source file last modified date
o Every query will check modified date, and reload if a new file has
arrived
DRAM
SSD
ADLS/BLOBStore
Updates

27. • LLAP, Spark, and Presto against 1 TB derived from the TPC-DS benchmark
• Out of the box HDInsight Configuration
• 45 queries derived from the derived TPC-DS benchmark that ran on all
engines successfully

30. • We used number of different concurrency levels to test the concurrency
performance
• 99 queries on 1 TB data with 32 worker node cluster with max concurrency set
to 32.
Test 1: Run all 99 queries, 1 at a time - Concurrency = 1
Test 2: Run all 99 queries, 2 at a time - Concurrency = 2
Test 3: Run all 99 queries, 4 at a time - Concurrency = 4
Test 4: Run all 99 queries, 8 at a time - Concurrency = 8
Test 5: Run all 99 queries, 16 at a time - Concurrency = 16
Test 6: Run all 99 queries, 32 at a time - Concurrency = 32
Test 7: Run all 99 queries, 64 at a time - Concurrency = 64

32. Capability Interactive Query Spark SQL Presto
Interactive Query Speed High High Medium
Scale High High Low
Caching Yes Yes Early Support
Intelligent Cache Eviction Yes No No
Complex Fact to Fact Joins Yes Yes No
Transactions Yes No No
Query Concurrency High Low Low
Row , Column level security Yes [Apache Ranger+ AAD] High Medium
Rich end user Tools Yes Yes Yes
Language Support SQL, UDF SQL, Scala, Python SQL
Data Source Connector
Support
Storage Handlers Data Sources High number of
connectors

38. DataLakeProbe
HBaseHealthProbe
HBaseMetricsProbe
HBaseProbe
HdfsProbe
HdinsightZookeeperProbe
……..
EdgenodeSSHWatchdog
GatewayTCPPingWatchdog
SSHTCPPingWatchdog
RStudioWatchdog
CertRolloverWatchdog
JobSubmissionPingWatchdog
OozieWatchdog
DataNodesUpWatchdog
NodeManagersUpWatchdog
ResourceHealthWatchdog
AzureNodeStatusWatchdog
ClusterMALoggingHashWatchdo
g
ClusterAvailabilityWatchdog
ClusterHealthWatchdog
……..
namenode_ha_health
ams_metrics_collector_process
ams_metrics_collector_autostart
ams_metrics_collector_hbase_master_p
rocess
namenode_last_checkpoint
namenode_webui
increase_nn_heap_usage_daily
hive_metastore_process
ambari_server_stale_alerts
ambari_server_agent_heartbeat
metrics_monitor_process_percent
……….

48. OMS Agent for
Linux
HDInsight nodes (Head, Worker ,
Zookeeper )
FluentD
HDInsight
plugin
1. Plugin for ‘in_tail’ for all Logs, allows
regexp to create JSON object
2. Filter for WARN and above for each
Log Type. `grep` filter plugin
3. Output to out_oms_api Type
4. Exec plugin for Metrics
HBaseConfigosmconfig
Spark
Hive/ LLAP
Storm
Kafka
Config
Config
Config
Config
Log Analytics(OMS) Service
HDInsight Log Analytics Architecture

49. Perimeter level security
Authentication
Authorization
Data security

50. Roadmap US Gov