How to create and run TPCDS benchmark for Spark , Hive, LLAP and Presto in HDInsight?

1. DIY: TPCDS HDInsight Benchmark
Ashish Thapliyal

2. Agenda
Typical Benchmarking
Scenarios with
HDInsight
01
How to Generate
TPCDS Benchmark
02
Run Benchmarks
- Spark
- Hadoop
- LLAP
- Presto
03

3. Benchmarking scenarios
Data Landing
Distcp
01
Batch/ETL
TeraSort
02
Relational Data
Analysis
TPCDS
03
Streaming
Pipeline
Yahoo Streaming
Benchmark
04
No SQL
YCSB
05

4. Most asked
benchmarking
question?
• ADLS vs Azure BLOB
• Which engine is faster?
• LLAP [Low Latency Analytical
processing]
• Spark Sql
• Presto
• Hive on Tez

5. Oozie Metastore
Hive Metastore
ADLS
Storage (Azure)
GW1
Head Node 1
Worker Node
Worker Node
Worker Node
Worker Node
Zookeeper1
Edge Node
VNET
Subnet
GW2Head Node 2
Zookeeper2 Zookeeper3

6. Hive Metastore
Data prep/Process AreaIngestion Area Consumption /BI Area
Spark Cluster
Kafka Cluster
Storm Cluster
HBase Cluster
Pig Cluster
Hive Cluster
Spark Cluster
Interactive Hive
[LLAP]
Spark SQL
Presto Cluster

7. TPCDS
The TPC Benchmark DS (TPC-DS) is a
decision support benchmark that models
several generally applicable aspects of a
decision support system, including
queries and data maintenance.
http://www.tpc.org/tpc_documents_curr
ent_versions/pdf/tpc-ds_v2.1.0.pdf

9. Key Scenarios
• Examine large volumes of data
• Give answers to real-world business questions
• Execute queries of various operational
requirements and complexities (e.g., ad-hoc,
reporting, iterative OLAP, data mining)
• Are characterized by high CPU and IO load
• Are periodically synchronized with source OLTP
databases through database maintenance
functions

10. TPCDS Queries [Total
99]
• Reporting – 41
• Ad Hoc – 59
• Iterative – 4
• Data Mining - 23

11. Scale Factor
• 1TB, 3TB, 10TB, 30TB, 100TB

12. How to setup your
TestBench?
Spark
Cluster
Spark
Cluster
Spark
Cluster
Spark
Cluster
SQL DB
[Metastore]
Azure Storage(WASB)
BI /Consumption Area

13. Best Practices
• Keep all cluster types similar [VM’s]
• Metastore needs to be S2 or greater
• Add the data storage account as
additional storage account to the cluster
• Do not run benchmark on multiple
clusters at the same time
• Keep Monitoring Yarn/Tez for failures
• Use Byobu mode while running queries
• Beeline is the best client to run query
• Hive view is the worst for running
benchmark queries
• Install Presto from
https://github.com/hdinsight/presto-
hdinsight

14. Let’s do it
https://github.com/hdinsight/tpcds-
datagen-as-hive-query

16. Thank You