SlideShare a Scribd company logo
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
SQL-on-Hadoop Tutorial
VLDB 2015
9/30/2015SQL-on-Hadoop Tutorial
1
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
Fatma Özcan
IBM Research
IBM Big SQL
Ippokratis Pandis
Cloudera
Cloudera Impala
Daniel Abadi
Yale University and
Teradata
HadoopDB/Hadapt
Shivnath Babu
Duke University
Starfish
2
Presenters
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Why SQL-on-Hadoop?
 People need to process data in parallel
 Hadoop is by far the leading open source parallel data
processing platform
 Low costs of HDFS results in heavy usage
9/30/2015SQL-on-Hadoop Tutorial
3
Lots of data in Hadoop with appetite to process it
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce is not the answer
 MapReduce is a powerful primitive to do many kinds of parallel data
processing
 BUT
 Little control of data flow
 Fault tolerance guarantees not always necessary
 Simplicity leads to inefficiencies
 Does not interface with existing analysis software
 Industry has existing training in SQL
9/30/2015SQL-on-Hadoop Tutorial
4
SQL interface for Hadoop critical for mass adoption
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Decades of research in parallel database systems
Efficient data flow
Load balancing in the face of skew
Query optimization
Vectorized processing
Dynamic compilation of query operators
Co-processing of queries
9/30/2015SQL-on-Hadoop Tutorial
5
Massive talent war between SQL-on-Hadoop companies
for members of database community
The database community knows how to
process data
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
SQL-on-Hadoop is not a direct
implementation of parallel DBMSs
Little control of storage
 Most deployments must be over HDFS
Append-only file system
 Must support many different storage formats
Avro, Parquet, RCFiles, ORC, Sequence Files
Little control of metadata management
 Optimizer may have limited access to statistics
Little control of resource management
 YARN still in its infancy
9/30/2015SQL-on-Hadoop Tutorial
6
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
SQL-on-Hadoop is not a direct
implementation of parallel DBMSs
Hadoop often used a data dump (swamp?)
 Data often unclean, irregular, and unreliable
Data not necessarily relational
 HDFS does not enforce structure in the data
 Nested data stored as JSON extremely popular
Scale larger than previous generation parallel database
systems
 Fault tolerance vs. query performance
Most Hadoop components written in Java
Want to play nicely with the entire Hadoop ecosystem 9/30/2015SQL-on-Hadoop Tutorial
7
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Outline of Tutorial
This session [13:30-15:00]
SQL-on-Hadoop Technologies
Storage
Run-time engine
Query optimization
Q&A
9/30/2015SQL-on-Hadoop Tutorial
8
Second Session [15:30-17:00]
SQL-on-Hadoop examples
HadoopDB/Hadapt
Presto
Impala
BigSQL
SparkSQL
Phoenix/Spice Machine
Research directions
Q&A
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Storage
9/30/2015SQL-on-Hadoop Tutorial
9
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Quick Look at HDFS
9/30/2015SQL-on-Hadoop Tutorial
10
…
NameNode
DataNode DataNode DataNode
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Good for
Storing large files
Write once and read many times
“Cheap” commodity hardware
Not good for
Low-latency reads
Short-circuit reads and HDFS caching help
Large amounts of small files
Multiple writers
9/30/2015SQL-on-Hadoop Tutorial
11 HDFS is
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
In-situ Data Processing
HDFS as the data dump
Store the data first, figure out what to do later
Most data arrive in text format
Transform, cleanse the data
Create data marts in columnar formats
Lost of nested, JSON data
Some SQL in data transformations, but mostly other
languages, such as Pig, Cascading, etc..
Columnar formats are good for analytics
9/30/2015SQL-on-Hadoop Tutorial
12
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Most SQL-on-Hadoop systems do not control or own the data
 Hive, Impala, Presto, Big SQL, Spark SQL, Drill
 Other SQL-on-Hadoop systems tolerate HDFS data, but work better
with their own proprietary storage
 HadoopDB/Hadapt
 HAWQ, Actian Vortex, and HP Vertica
9/30/2015SQL-on-Hadoop Tutorial
13 SQL-on-Hadoop according to storage formats
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Only support native Hadoop formats with open-source
reader/writers
Any Hadoop tool can generate their data
Pig, Cascading and other ETL tools
They are more of a query processor than a database
Indexing is a challenge !!
No co-location of multiple tables
Due to HDFS
9/30/2015SQL-on-Hadoop Tutorial
14 Query Processors with HDFS Native Formats
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Almost all exploit some existing database systems
They store their own binary format on HDFS
Hadapt stores the data in a single node database, like
postgres
Can exploit Postgres indexes
HAWQ, Actian, HP Vertica, and Hadapt all control how
tables are partitioned, and can support co-located joins
9/30/2015SQL-on-Hadoop Tutorial
15 Systems with Proprietary Formats
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
CSV files are most common for ETL-like workloads
Lots of nested and complex data
Arrays, structs, maps, collections
Two major columnar formats
ORCFile
Parquet
Data serialization
JSON and Avro
Protocol buffers and Thrift
9/30/2015SQL-on-Hadoop Tutorial
16 HDFS Native Formats
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
17
9/30/2015SQL-on-Hadoop Tutorial
17
Parquet
 PAX format, supporting nested data
 Idea came from the Google‘s Dremel System
 Major contributors: Twitter & Cloudera
 Provides dictionary encoding and several compressions
 Preffered format for Impala, IBM Big SQL, and Drill
 Can use Thrift or Avro to describe the schema
Nested data
 A natural schema
 Flexible
 Less duplication applying
denormalization
Columnar storage
 Fast compression
 Schema projection
 Efficient encoding
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Parquet, cont.
 A table with N columns is split
into M row groups.
 The file metadata contains the
locations of all the column
metadata start locations.
 Metadata is written after the
data to allow for single pass
writing.
 There are three types of
metadata: file metadata, column
(chunk) metadata and page
header metadata.
 Row group metadata includes
 Min-max values for skipping
9/30/2015SQL-on-Hadoop Tutorial
18
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Second generation, following RC file
 PAX formats with all data in a single file
 Hortonworks is the major contributor, together with Microsoft
 Preferred format for Hive, and Presto
 Supports
 Dictionary encoding
 Fast compression
 File, and stripe level metadata
 Stripe indexing for skipping
 Now metadata even includes bloom filters for point query lookups
9/30/2015SQL-on-Hadoop Tutorial
19 ORCFile
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
20
ORCFile Layout
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 No updates in HDFS
 Appends to HDFS files are supported,
but not clear how much they are used in
production
 Updates are collected in delta files
 At the time of read delta and main files
are merged
 Special inputFormats
 Lazy compaction to merge delta files
and main files
 When delta files reach a certain size
 Scheduled intervals
9/30/2015SQL-on-Hadoop Tutorial
21
File A
delta
1
delta
2
delta
n
…
Handling Updates in HDFS
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
SQL on NoSQL!
 Put a NoSQL solution on top of HDFS
 For the record, you can avoid HDFS completely
 But, this is a SQL-on-Hadoop tutorial
 NoSQL solutions can provide CRUD at scale
 CRUD = Create, Read, Update, Delete
 And, then run SQL on it?
 Sounds crazy? Well, lets see
9/30/2015SQL-on-Hadoop Tutorial
22
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
23 HBase: The Hadoop Database
 Not HadoopDB, which we will see later in the tutorial
 HBase is a data store built on top of HDFS based on Google Bigtable
 Data is logically organized into tables, rows, and columns
 Although, Key-Value storage principles are used at multiple points in the design
 Columns are organized into Column Families (CF)
 Supports record-level CRUD, record-level lookup, random updates
 Supports latency-sensitive operations
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HBase Architecture24
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HBase Architecture25
HBase stores three types of files
on HDFS:
• WALs
• HFiles
• Links
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HBase Read and Write Paths26
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HFile Structure
27
• Immutable
• Created on flush or compaction
• Sequential writes
• Read randomly or sequentially
• Data is in blocks
• HFile blocks are not HDFS blocks
• Default data block size == 64K
• Default index block size == 128K
• Default bloom filter block size
== 128K
• Use smaller block sizes for
faster random lookup
• Use larger block sizes for faster scans
• Compression is recommended
• Block encoding is recommended
HFile Format
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Run-time Engine
9/30/2015SQL-on-Hadoop Tutorial
28
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Low Latency
High Throughput
Degree of tolerance to faults
Scalability in data size
Scalability in cluster size
Resource elasticity
Multi-tenancy
Ease of installation in existing environments
9/30/2015SQL-on-Hadoop Tutorial
29 Design Decisions: Influencers
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Push computation to data
Columnar data formats
Vectorization
Support for multiple data formats
Support for UDFs
9/30/2015SQL-on-Hadoop Tutorial
30 Accepted across SQL-on-Hadoop Solutions
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
What is the Lowest Common Execution Unit
Use of Push Vs. Pull
On the JVM or not
Fault tolerance: Intra-query or inter-query
Support for multi-tenancy
9/30/2015SQL-on-Hadoop Tutorial
31 Differences across SQL-on-Hadoop Solutions
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Hive
Tenzing
9/30/2015SQL-on-Hadoop Tutorial
32 SQL on MapReduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
33 Hive
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
34 Example: Joins in MapReduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Having a MapReduce Job as the Lowest Execution Unit quickly
becomes restrictive
Query execution plans become MapReduce workflows
9/30/2015SQL-on-Hadoop Tutorial
35 Limitations
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
D1 D2
D3
D4
D6D5
D7
D01 D02
J1 J2
J3
J4
J5 J6
J7
MapReduce Jobs
36
Datasets
MapReduce Workflows
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
On efficient joins in the MapReduce paradigm
On reducing the number of MapReduce jobs by
packing/collapsing the MapReduce workflow
Horizontally
Shared scans
Vertically
Making using of static and dynamic partitioning
On efficient management of intermediate data
9/30/2015SQL-on-Hadoop Tutorial
37 Research Done to Address these Limitations
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Dryad
Tez
9/30/2015SQL-on-Hadoop Tutorial
38 From MapReduce to DAGs
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
39 Dryad: Dataflows as First-class Citizens
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
40 Smart DAG Execution in Dryad
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Tez: Inspired by Dryad and Powered by YARN
9/30/2015SQL-on-Hadoop Tutorial
41
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 The Hadoop Community realized that
MapReduce cannot be the Lowest
Execution Unit for all data apps
 Separated out the resource management
aspects from application management
 YARN is best seen as an Operating
System for Data Processing Apps
Recall the 80s: Databases and
Operating Systems: Friends or Foes?
9/30/2015SQL-on-Hadoop Tutorial
42 Quick Detour on YARN
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
An Example of What Tez Enables
9/30/2015SQL-on-Hadoop Tutorial
43
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
A Tez Slide on Tez
9/30/2015SQL-on-Hadoop Tutorial
44
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
map filter reduceBykey map saveAsTextFile
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
RDD0 RDD1 RDD2 RDD3 RDD4 HDFS
sc.textFile(hdfsPath)
.map(parseInput)
.filter(subThreshold)
.reduceByKey(tallyCount)
.map(formatOutput)
.saveAsTextFile(outPath)
Spark: A Different Way to Look at a Dataflow
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
sc.textFile(hdfsPath)
.map(parseInput)
.filter(subThreshold)
.reduceByKey(tallyCount)
.map(formatOutput)
.saveAsTextFile(outPath)
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
map filter reduceBykey map saveAsTextFile
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
HDFS
Stage 0 Stage 1
RDD0 RDD1 RDD2 RDD3 RDD4
Spark: A Different Way to Look at a Dataflow
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
map filter reduceBykey
map
saveAsTextFile
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
RDD0 RDD1 RDD2 RDD3 RDD4 HDFS
Stage 0 Stage 1
sc.textFile(hdfsPath)
.map(parseInput)
.filter(subThreshold)
.reduceByKey(tallyCount)
.map(formatOutput)
.saveAsTextFile(outPath)
Spark: A Different Way to Look at a Dataflow
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
map filter reduceBykey map saveAsTextFile
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
part-
0
part-
1
part-
2
part-
3
Exec0Exec1Exec2
RDD0 RDD1 RDD2 RDD3 RDD4 HDFS
sc.textFile(hdfsPath)
.map(parseInput)
.filter(subThreshold)
.reduceByKey(tallyCount)
.map(formatOutput)
.saveAsTextFile(outPath)
Stage 0 Stage 1
Spark: A Different Way to Look at a Dataflow
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Spark: A Different Way to Look at a Dataflow
9/30/2015SQL-on-Hadoop Tutorial
49
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
50
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
51
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
52
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
53
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
54
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
55
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
56
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
57
0
20
40
60
80
100
120
140
160
180
200
Fault tolerance Slowdown tolerance
PercentageSlowdown Traditional
DBMS
MapReduce
• SELECT sourceIP,
SUM(adRevenue)
FROM UserVisits
GROUP BY sourceIP
• Node fails (or slows down
by factor of 2) in the middle
of query
Fault Tolerance
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Downsides of MapReduce Fault Tolerance
9/30/2015SQL-on-Hadoop Tutorial
58
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
HDFSHDFS
Map
Map
Map
Map
Map
Reduce
Reduce
Reduce
Reduce
Map output written
to disk
Reduce output
written to HDFS
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Spark RDDs
Stores intermediate results in memory rather than disk
Advantage: Performance
Disadvantage: Memory requirements
9/30/2015SQL-on-Hadoop Tutorial
59
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Resource Management
9/30/2015SQL-on-Hadoop Tutorial
60
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Resource Management
(At least) Two dimension problem:
1. RM across different frameworks
Usually not a dedicated cluster
Shared across multiple frameworks
ETL (MapReduce, Spark), Hbase
SQL-on-Hadoop processing
2. RM across concurrent queries
9/30/2015SQL-on-Hadoop Tutorial
61
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
RM -- Across frameworks
 YARN – Yet Another Resource Negotiator
 Centralized, cluster-wide resource management system
Allows frameworks to share resources without
partitioning between them
 Designed for batch-mostly processing
Not mature
Not good for interactive analytics
Not meant for long running processes
Approaches: Llama and Slider
9/30/2015SQL-on-Hadoop Tutorial
62
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
RM -- LLAMA (low-latency application master)
Introduced by Cloudera
LLAMA acts as a proxy between Impala and YARN
Mitigates some of the batch-centric design aspects of YARN:
High resource acquisition latency -> solves via resource caching
Resource request is immutable -> solves via expansion request
Resource allocation is incremental -> solves via gang scheduling
6
3
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Slider allows running non-YARN enabled applications on YARN
 Without having to write your own custom Application Master
 Existing applications are packaged as Slider applications
 Encapsulates a set of one or more application components or roles
 Deployed by Slider, runs in containers across a YARN cluster
 Pre-built packages for HBase, Accumulo, Storm, and jmemcached
 Packages need to be custom built for other applications
 Some notable Slider features
 Applications can be stopped and started later  state is persisted
 Container failures are automatically detected by Slider and restarted
64
64 RM -- Apache Slider
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Query Optimization
9/30/2015SQL-on-Hadoop Tutorial
65
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Some Techniques We Know and Love Are
not Directly Applicable
Indexing
Zone-maps
Co-located joins
Query rewrites
Cost-based
optimization
 Databases own their storage
SQL-on-Hadoop systems do not
 Metadata management is tricky
 Data inserted/loaded without
SQL system knowledge
 No co-location of related tables
 HDFS is for most practical
purposes, read-only
9/30/2015SQL-on-Hadoop Tutorial
66
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Hive Partition tables maintain metadata values as one folder/ directory in
HDFS, per distinct value:
 Example: PARTITIONED BY (country STRING, year INT, month INT, day INT) ;
Folder/Directory created for country=US/year=2012/month=12/day=22
Partitioning only logical, not physical
 Partition pruning eliminates reading files that are not needed
 Almost all SQL-on-Hadoop offerings support this
 Hive, Impala, SparkSQL, IBM BigSQL, ….
9/30/2015SQL-on-Hadoop Tutorial
67 I/O Elimination for HDFS Data: Partition-level
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 ORCFile broken into Stripes (250MB default)
 Index with Min/Max values stored for each Column
 Data is a “stream” of columns
 Bloom filters for each stripe in ORCFile allow fast lookups
 Parquet also supports min/max values
 Works well when data is sorted, not very effective otherwise
9/30/2015SQL-on-Hadoop Tutorial
68 I/O Elimination for HDFS Data: Rowblock-level
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Quick look at query optimizers
 Two types of optimization
 Logical transformations to transform query into equivalent but simpler form
 Cost-based enumeration of alternative execution plans
 Most systems support the first one
 Cost-based optimization depends on good statistics and a good model of
the execution environment
 Without controlling data storage, statistics are “gestimates”
9/30/2015SQL-on-Hadoop Tutorial
69
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Selection/projection pushdown
 Nested SQL queries require more sophisticated rewrites, such
as decorrelation
 New systems all have rewrites but lack complex decorrelation
and subquery optimization ones
 Hive, Impala, Presto, Spark SQL
 Systems that leverage mature DB technology offer more
sophisticated rewrite engines
 IBM SQL, Hadapt, HP Vertica
9/30/2015SQL-on-Hadoop Tutorial
70 Query Rewrite
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Hive analyze table collects basic statistics
 Column value distributions, min-max, no-of-distinct values
 No control of data  data changes without the systems’
knowledge
 Multi-tenant system makes it harder to build a cost model
 More complex system behavior
9/30/2015SQL-on-Hadoop Tutorial
71
More adaptive query processing is needed
Cost-based Optimization
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Co-partitioning two tables on the join key enables local joins
 HDFS default block placement policy scatters blocks in the cluster
 Actian Vortex changes HDFS default block placement to enforce co-
located joins
9/30/2015SQL-on-Hadoop Tutorial
72
 Files A & B are co-located
 Files C & D are co-located
File A File B
File DFile C
Co-located joins
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Outline of Tutorial
This session [13:30-15:00]
SQL-on-Hadoop Technologies
Storage
Run-time engine
Query optimization
Q&A
9/30/2015SQL-on-Hadoop Tutorial
73
Second Session [15:30-17:00]
SQL-on-Hadoop examples
HadoopDB/Hadapt
Presto
Impala
BigSQL
SparkSQL
Phoenix/Spice Machine
Research directions
Q&A
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HadoopDB
 First of avalanche of SQL-on-Hadoop solutions to claim 100x faster than Hive (on
certain types of queries)
 Used Hadoop MapReduce to coordinate execution of multiple independent (typically
single node, open source) database systems
 Maintained MapReduce’s fault tolerance
 Sped up single-node processing via leveraging database performance optimizations:
Compression
Vectorization
Partitioning
Column-orientation
Query optimization
Broadcast joins
 Flexible query interface (both SQL and MapReduce)
74
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HadoopDB Architecture75
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HadoopDB SMS Planner76
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HadoopDB History
 Paper published in 2009
 Company founded in 2010 (Hadapt) to commercialize
HadoopDB
 Added support for search in 2011 (for major insurance
customer)
 Added JSON support in 2012
 Added interactive query engine in 2013
 Acquired by Teradata in 2014
9/30/2015SQL-on-Hadoop Tutorial
77
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Teradata Unified Data Architecture: QueryGrid
Marketing
Executives
Operational
Systems
Customers
& Partners
Frontline
Workers
Business
Analysts
Data
Scientists
Engineers &
Programmers
SQL-H SQL SQL, NOSQL
DATA
PLATFORM
HADOOP OR
TERADATA
INTEGRATED
DATA
WAREHOUSE
TERADATA
DATABASE
ASTER
DATABASE
DISCOVERY
PLATFORM
COMPUTE
CLUSTERS
SAS, PYTHON,
R, PERL, RUBY…
OTHER
DATABASES
ORACLE,
MONGODB, ETC
SQL VARIOUS
TERADATA OR ASTER DATABASE
TERADATA QUERYGRID
PUSH DOWN / REMOTE PROCESSING
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Remote Processing On Hadoop
Query through Teradata
Leaves of query plan
sent to SQL-on-Hadoop
engine
Results returned to
Teradata
Additional query
processing done in
Teradata
Final results sent back to
application/user
Teradata 15.0
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Bi-directional data movement
 Read and write data to Hadoop
 Create new table in Hadoop or insert records
 Query push-down
 Execute query on Hadoop
 Qualify rows and columns to reduce data returned
 Easy configuration and simplified queries
 Create “Hadoop server” definition once
 Use @foreign_server name to access Hadoop
Teradata QueryGrid Teradata-Hadoop
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
History of Presto
FALL 2012
6 developers
start Presto
development
FALL 2014
88 releases
41 contributors
3943 commits
SPRING 2015
Teradata
provides first
commercial
support for
Presto +
roadmap
SPRING 2013
Presto rolled out
within Facebook
FALL 2013
Facebook
open sources
Presto
FALL 2008
Facebook
open sources
Hive
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Hive
Reduce Reduce
Map Map
Reduce Reduce
Map Map
Disk
Disk
Disk
Wait
between
stages
Write to Disk
• Fault Tolerance
• IO Overhead
Presto
Task Task
Task Task
Task
Task Task
All stages are pipelined
• Reduced wait time
• No Fault Tolerance
Memory-to-memory
Data transfer
• No disc IO
• Data chunk must
fit in memory
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Uses Hive metastore
 Bytecode query compilation
 Approximate queries
 Return X% sample rows
 Limitations
 Manual join SQL ordering
 Non-equi joins not supported
 Not YARN enabled
 No Avro support
 No spill-to-disk
 Written in Java
 100% ANSI SQL goal
 Numerous built-in functions
 Window functions
 Array/map support
 Plug-in architecture
 Join across data stores
 Hive, Cassandra, Kafka, MySQL
 Amazon S3
Presto at a Glance
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Presto Pipeline Architecture
Data stream API
Worker
Data stream API
Worker
Coordinator
Data Location
API
Metadata
API
Parser/
analyzer
Planner Scheduler
Worker
Client
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Presto Connectors
Client
Presto worker Presto worker Presto worker Presto worker
Presto Coordinator
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Github: Presto Plug-in Connectors
 Hive tables and HCatalog
 Apache Cassandra
 Apache Kafka
 Kafka topics = Presto tables, messages = rows
 MySQL
 Single node access only -- no sharding
 Postgres
 Single node access only
 HBase
 not released
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Cloudera Impala
9/30/2015SQL-on-Hadoop Tutorial
87
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
8
8
Query Executor
SQL App
ODBC
Hive Metastore HDFS NameNode Statestore
Query Planner
Query Coordinator
HDFS DN HBase
Impalad
HDFS DN HBase
Impalad
HDFS DN HBase
Impalad
Catalog
Query Executor
Query Planner Query Planner
Query Coordinator Query Coordinator
Query Executor
SQL request
Plan Fragments
Results
Query execution at the high level88
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
89
HashJoinScan: t1
Scan: t3
Scan: t2
HashJoin
TopN
Pre-Agg
MergeAgg
TopN
Broadcast
Merge
hash t2.idhash t1.id1
hash
t1.custid
at HDFS DN
at HBase RS
at coordinator
HashJoin
Scan: t1
Scan: t3
Scan: t2
HashJoin
TopN
Agg
Single-Node
Plan
Query Planning: Distributed Plans
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Written in C++ for minimal cycle and memory overhead
Leverages decades of parallel DB research
Partitioned parallelism
Pipelined relational operators
Batch-at-a-time runtime
Focussed on speed and efficiency
Intrinsics/machine code for text parsing, hashing, etc.
Runtime code generation with LLVM
90
Execution Engine
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Uses llvm to jit-compile the runtime-intensive parts of a
query
 Effect the same as custom-coding a query:
 Remove branches, unroll loops
 Propagate constants, offsets, pointers, etc.
 Inline function calls
 Optimized execution for modern CPUs (instruction
pipelines)
91
Runtime Code Generation
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
92
interpreted codegen’d
IntVal my_func(const IntVal& v1, const IntVal& v2) {
return IntVal(v1.val * 7 / v2.val);
}
SELECT my_func(col1 + 10, col2) FROM ...
my_fu
nc
col
2
+
10
col
1
function
pointer
function
pointer
function
pointer
function
pointer
(col1 + 10) * 7 / col2
function
pointer
Runtime Code Generation — Example
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
93
10 node cluster (12 disks / 48GB RAM / 8 cores per node)
~40 GB / ~60M row Avro dataset
Impala Runtime Code Generation - Performance
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Codegen is not the panacea!94
TPC-DS 500GB,10-node clusterTPC-H 300GB,10-node cluster
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Admission control and Yarn-based RM cater to different workloads
Use admission control for:
 Low-latency, high-throughput workloads
 Mostly running Impala, or resource partitioning is feasible
Use Llama/Yarn for:
 Mixed workloads (Impala, MR, Spark, …) and resource partitioning is
impractical
 Latency and throughput SLAs are relatively relaxed
Resource Management in Impala95
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
96
 Nested data: Structs, arrays, maps in Parquet, Avro, JSON, …
 Natural extension of SQL: expose nested structures as tables
 No limitation on nesting levels or number of nested fields in single query
 Multithreaded execution past scan operator
 Resource management and admission control
 low-latency, high-throughput mixed workloads without resource partitioning
 More SQL: ROLLUP/GROUPING SETS, INTERSECT/MINUS,
MERGE
 Improved query planning, using statistics
 Physical tuning
Roadmap: Impala 2.3+
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Ibis: Scaling the Python Data Experience
Target user:
Data scientists and data engineers (“Python data users”)
Goals:
Mirror single-node Python experience, maximize productivity
Complete support for SQL engines with Pandas-like API (same
designer)
High-performance Python user-defined functions
Integration with Python data ecosystem / libraries
97
http://www.ibis-project.org/
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Ibis/Impala Joint Roadmap
• More natural data modeling
• Complex types support
• Integration with full Python data ecosystem
• Advanced analytics + machine learning
• Enable use of performance computing tools
• User extensibility with native performance
• In-memory columnar format
• Python-to-LLVM IR compilation
• Workflow and usability tools
99
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Code at github (https://github.com/cloudera/Impala/)
Impala Developer Docker Images & Chef scripts
https://registry.hub.docker.com/u/cloudera/impala-dev/
Minimal (7GB) — ready to compile, latest code
Default (33GB) — includes test data, e.g. TPC-H
Shout out to Spyros Blanas (Ohio State)
http://web.cse.ohio-state.edu/~sblanas/5242/
Impala JIRAs, ramp-up tasks
100
Academic Challenge
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
IBM Big SQL
9/30/2015SQL-on-Hadoop Tutorial
101
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Head (coordinator) node
 Compiles and optimizes the query
 Coordinates the execution of the query
 Big SQL worker processes reside on compute nodes (some or all)
 Worker nodes stream data between each other as needed
Mgmt Node
Big SQL
Mgmt Node
Hive Metastore
Mgmt Node
Name Node
Mgmt Node
Job Tracker
•••
Compute Node
Task
Tracker
Data Node
Compute Node
Task
Tracker
Data Node
Compute Node
Task
Tracker
Data Node
Compute Node
Task
Tracker
Data Node
•••
Big
SQL
Big
SQL
Big
SQL
Big
SQL
HDFS
102 Big SQL – Architecture
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 For common table formats a native I/O engine is utilized
 e.g. delimited, RC, SEQ, Parquet, …
 For all others, a java I/O engine is used
 Maximizes compatibility with existing tables
 Allows for custom file formats and SerDe's
 All Big SQL built-in functions are native code
 Customer built UDF's can be developed in C++ or Java
Mgmt Node
Big SQL
Compute Node
Task
Tracker
Data Node
Big
SQL
Big SQL Worker
Native I/O
Engine
Java I/O
Engine
Runtime
Java UDFs
Native UDFs
103
103 Big SQL – Architecture (cont.)
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 All data is Hadoop data
 In files in HDFS
 SEQ, ORC, delimited, Parquet …
 Never need to copy data to a proprietary representation
 All data is catalog-ed in the Hive metastore
 It is the Hadoop catalog
 It is flexible and extensible
104
104 Big SQL works with Hadoop
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 The scheduler is the main RDBMS↔Hadoop service interface
 Interfaces with Hive metastore for table metadata
 SQL compiler ask it for some "hadoop" metadata, such as partitioning columns
 Acts like the MapReduce job tracker for Big SQL
 Big SQL provides query predicates for scheduler to perform partition elimination
 Determines splits for each “table” involved in the query
 Schedules splits on available Big SQL nodes
(with best effort data locality)
 Decides which I/O library to use and serves
work (splits) to them
 Coordinates “commits” after INSERTs
Management Node
Big SQL
Master Node
Big SQL
Scheduler
DDL
FMP
UDF
FMP
Mgmt Node
Database
Service
Hive
Metastore
Big SQL
Worker Node
Java
I/O
FMP
Native
I/O
FMP
HDFS Data
Node
MRTask
TrackerUDF
FMP
105 Scheduler Service
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Query Rewrite
 There are many ways to express the same query
 Query generators often produce suboptimal queries and don't permit "hand optimization"
 Complex queries often result in redundancy, especially with views
 For large data volumes optimal access plans more crucial as penalty for poor planning is
greater
select sum(l_extendedprice) / 7.0
avg_yearly
from tpcd.lineitem, tpcd.part
where p_partkey = l_partkey
and p_brand = 'Brand#23'
and p_container = 'MED BOX'
and l_quantity < ( select 0.2 *
avg(l_quantity) from tpcd.lineitem
where l_partkey = p_partkey);
select sum(l_extendedprice) / 7.0 as avg_yearly
from temp (l_quantity, avgquantity, l_extendeprice)
as
(select l_quantity, avg(l_quantity) over
(partition by l_partkey)
as avgquantity, l_extenedprice
from tpcd.lineitem, tpcd.part
where p_partkey = l_partkey
and p_brand = 'BRAND#23'
and p_container = 'MED BOX')
where l_quantity < 0.2 * avgquantity
• Query correlation eliminated
• Lineitem table accessed only once
• Execution time reduced in half!
106
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Cost-based Optimization
|
2.66667e-08
HSJOIN
( 7)
1.1218e+06
8351
/--------+--------
5.30119e+08 3.75e+07
BTQ NLJOIN
( 8) ( 11)
948130 146345
7291 1060
| /----+----
5.76923e+08 1 3.75e+07
LTQ GRPBY FILTER
( 9) ( 12) ( 20)
855793 114241 126068
7291 1060 1060
| | |
5.76923e+08 13 7.5e+07
TBSCAN TBSCAN BTQ
( 10) ( 13) ( 21)
802209 114241 117135
7291 1060 1060
| | |
7.5e+09 13 5.76923e+06
TABLE: TPCH5TB_PARQ TEMP LTQ
ORDERS ( 14) ( 22)
Q1 114241 108879
1060 1060
| |
13 5.76923e+06
DTQ TBSCAN
( 15) ( 23)
114241 108325
1060 1060
| |
1 7.5e+08
GRPBY TABLE: TPCH5TB_PARQ
( 16) CUSTOMER
114241 Q5
1060
|
1
LTQ
( 17)
114241
1060
|
1
GRPBY
( 18)
114241
1060
|
5.24479e+06
TBSCAN
( 19)
113931
1060
|
7.5e+08
TABLE: TPCH5TB_PARQ
CUSTOMER
Q2
Few extensions required to the Cost Model
Scan operator cost model extended to evaluate
cost of reading from Hadoop
# of files, size of files, # of partitions, # of
nodes
Data not hash partitioned on a particular columns
(aka “Scattered partitioned”)
New parallel join strategy
Every node read data from HDFS, instead of
one reading and broadcasting
Optimizer now knows in which subset of nodes the
data resides => better costing!
Sophisticated statistics for cardinality estimation
107
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Statistics
 Big SQL utilizes Hive statistics collection with
some extensions:
Additional support for column groups,
histograms and frequent values
Automatic determination of partitions that
require statistics collection vs. explicit
Partitioned tables: added table-level
versions of NDV, Min, Max, Null count,
Average column length
Hive catalogs as well as database engine
catalogs are also populated
We are restructuring the relevant code for
submission back to Hive
 Capability for statistic fabrication if no stats
available at compile time
Table statistics
•Cardinality (count)
•Number of Files
•Total File Size
Column statistics
•Minimum value (all types)
•Maximum value (all types)
•Cardinality (non-nulls)
•Distribution (Number of Distinct Values
NDV)
•Number of null values
•Average Length of the column value (all
types)
•Histogram - Number of buckets configurable
•Frequent Values (MFV) – Number
configurable
Column group statistics
108
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Big SQL supports HBase tables
 Big SQL with HBase – basic operations
– Create tables and views
– LOAD / INSERT data
– Query data with full SQL breadth
 HBase-specific design points
 Column mapping
 Dense / composite columns
– FORCE KEY UNIQUE option
– Secondary indexes
– . . . .
109
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Big SQL works under YARN
 Big SQL integrates with YARN via the Slider
project
 YARN chooses suitable hosts for Big SQL
worker nodes
 Big SQL resources are accounted for by
YARN
 Size of the Big SQL cluster may dynamically
grow or shrink as needed
 Configured by user (not by installation
default)
 More Big SQL workers are added when more
resources are needed
 When demand wears off, Big SQL workers
are shut down
Data Management
SPARK
Hive
Pig
BigSQL
Data Access
HDFS
MapReduce
YARN
110
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Summary
 Big SQL provides rich, robust, standards-based SQL support for data stored in HDFS and HBase
 Uses IBM common client ODBC/JDBC drivers
 Big SQL fully integrates with SQL applications and tools
 Existing queries run with no or few modifications*
 Existing JDBC and ODBC compliant tools can be leveraged
 Big SQL provides faster and more reliable performance
 Big SQL uses more efficient access paths to the data
 Big SQL is optimized to more efficiently move data over the network
 Big SQL is capable of executing all 22 TPC-H and all 99 TPC-DS queries without modification
 Big SQL provides and enterprise grade data management
 Security, Auditing, workload management …
111
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
SparkSQL
9/30/2015SQL-on-Hadoop Tutorial
112
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
From:
What is so great about Spark?113
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Distributed data analytics engine, generalizing Map Reduce
 Core engine, with streaming, SQL, machine learning, and graph processing modules
114 OK, but what exactly is Spark?
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
RDDs
Distributed collection of objects
Can be cached in memory
Built via parallel transformations (map, filter, …)
Automatically rebuilt on failure based on lineage
DAGs of RDDs and Transformations can be (lazily)
executed via actions
Examples: Export to HDFS, count number of objects
Spark Core:
RDDs, Transformations & Actions
115
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
116 Spark’s DAG Execution
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Building a real-world big data application without and with Spark:
…
HDFS
read
HDFS
write
ETL
HDFS
read
HDFS
write
train
HDFS
read
HDFS
write
query
HDFS
HDFS
read
ETL
train
query
With Spark: Interactive
analysis
Why Application Developers love Spark117
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Raw JSON Tweets
SQL Machine Learning
Streaming
An Example App118
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)import org.apache.spark.sql._
val ctx = new org.apache.spark.sql.SQLContext(sc)
val tweets = sc.textFile("hdfs:/twitter")
val tweetTable = JsonTable.fromRDD(sqlContext, tweets, Some(0.1))
tweetTable.registerAsTable("tweetTable")
ctx.sql("SELECT text FROM tweetTable LIMIT 5").collect.foreach(println)
ctx.sql("SELECT lang, COUNT(*) AS cnt FROM tweetTable 
GROUP BY lang ORDER BY cnt DESC LIMIT 10").collect.foreach(println)
val texts = sql("SELECT text FROM tweetTable").map(_.head.toString)
def featurize(str: String): Vector = { ... }
val vectors = texts.map(featurize).cache()
val model = KMeans.train(vectors, 10, 10)
sc.makeRDD(model.clusterCenters, 10).saveAsObjectFile("hdfs:/model")
val ssc = new StreamingContext(new SparkConf(), Seconds(1))
val model = new KMeansModel(
ssc.sparkContext.objectFile(modelFile).collect())
// Streaming
val tweets = TwitterUtils.createStream(ssc, /* auth */)
val statuses = tweets.map(_.getText)
val filteredTweets = statuses.filter {
t => model.predict(featurize(t)) == clusterNumber
}
filteredTweets.print()
ssc.start()
Fitsononeslide
119
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
120
SQL, SQL, SQL, …
Databricks says that 100% of their customers use
some SQL
Schema is very useful
Even in complex pipelines that process a lot of
un/semi-structured data
Separation of logical from physical plan is critical for
performance and scalability
Why SparkSQL?
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Plan Optimization & Execution
121
SQL AST
DataFrame
Unresolved
Logical Plan
Logical Plan
Optimized
Logical Plan
RDDs
Selected
Physical
Plan
Analysis
Logical
Optimization
Physical
Planning
CostModel
Physical
Plans
Code
Generation
Catalog
DataFrames and SQL share the same optimization/execution pipeline
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
122
1. A distributed collection of rows organized into named
columns
2. An abstraction for selecting, filtering, aggregating and
plotting structured data (cf. R, Pandas, Ibis)
DataFrame
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Catalyst Optimizer: Tree Transformations
 Developers express tree transformations as PartialFunction[TreeType,TreeType]
1. If the function does apply to an operator, that operator is replaced with the result.
2. When the function does not apply to an operator, that operator is left unchanged.
3. The transformation is applied recursively to all children.
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Prior Work: Optimizer Generators
 Volcano / Cascades:
• Create a custom language for expressing rules that rewrite trees of relational
operators.
• Build a compiler that generates executable code for these rules.
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
An Example Catalyst Transformation
1. Find filters on top of projections.
2. Check that the filter can be evaluated
without the result of the project.
3. If so, switch the operators.
Project
name
Project
id,name
Filter
id = 1
People
Original
Plan
Project
name
Project
id,name
Filter
id = 1
People
Filter
Push-Down
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Filter Push Down Transformation
val newPlan = queryPlan transform {
case f @ Filter(_, p @ Project(_, grandChild))
if(f.references subsetOf grandChild.output) =>
p.copy(child = f.copy(child = grandChild)
}
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Community-Contributed Transformations
127
110 line patch took this user’s query from
“never finishing” to 200s.
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Project Tungsten: Getting Spark to Run Well
on the JVM
Overcoming JVM limitations:
• Memory Management and Binary Processing:
leveraging application semantics to manage memory
explicitly and eliminate the overhead of JVM object model
and garbage collection
• Cache-aware computation: algorithms and data
structures to exploit memory hierarchy
• Code generation: using code generation to exploit
modern compilers and CPUs
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
129
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Use sun.misc.Unsafe
JVM internal API
Can manipulate memory without safety checks
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
131
Null bits
Inline fixed-length values
Align on 8-byte word boundaries
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Apache Phoenix
9/30/2015SQL-on-Hadoop Tutorial
132
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
133
SQL compiler and execution engine for HBase
Query engine transforms SQL into native HBase APIs: put, delete, parallel
scans (instead of, say, MapReduce)
Supports features not provided by HBase: Secondary Indexing,
Multi-tenancy, simple Hash Join, etc.
The Phoenix Approach
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
134 Phoenix Architecture
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Open (Research) Challenges
9/30/2015SQL-on-Hadoop Tutorial
135
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
 Cost-based optimizer relies on
 Statistics over base relations
 Formulas for cost estimation
 Rules for plan enumeration
 Problems:
 Stats not reliable, do not own the data
 Prominent use of UDFs
 Independence assumption between predicates do not hold
 More nested data, harder to estimate selectivities
 Bad plans over big data may run “forever”
Defer more cost-based decisions to run-time; robust, adaptive
query optimization
Challenge 1: Query optimization
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
9/30/2015SQL-on-Hadoop Tutorial
137
 No single framework owns the
data!
 Multiple frameworks, with
different resource
requirements
HDFS, S3
Hadoop Yarn
Spark/MR/Tez/..
Streaming
SQL Graph Machine
learning
ETL and
batch
processing
 How to share the data?
 How to share resources?
 How to work together
seamlessly?
Challenge 2: Multi-framework environment
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
HDFS is a problem for transactional workloads
Workarounds do not lend itself to high-performance OLAP
Object-stores
Interesting combinations are emerging
Hive LLAP + Phoenix, Splice Machine + Spark
Need more tightly integrated solutions
Need an updatable, fast, distributed file system
9/30/2015SQL-on-Hadoop Tutorial
138 Challenge 3: Transactions and analytics in
one system
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References
 http://www.slideshare.net/enissoz/hbase-and-hdfs-understanding-filesystem-usage
 https://www.mapr.com/blog/in-depth-look-hbase-architecture
 Apache Drill. http://drill.apache.org/.
 Apache Phoenix. http://phoenix.apache.org/.
 Hive on spark. https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark .
 Splice machine. http://www.splicemachine.com/.
 Teradata query grid. http://www.teradata.com/Teradata- QueryGrid/#tabbable=0&tab1=0&tab2=0.
 M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi, J. Erickson, M. Grund, D. Hecht,
M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson, D. Rorke, S. Rus, J.
Russell, D. Tsirogiannis, S. Wanderman-Milne, and M. Yoder. “Impala: A Modern, Open-Source SQL
Engine for Hadoop.” In Proc. CIDR, 2015.
 Y. Huai, A. Chauhan, A. Gates, G. Hagleitner, E. N. Hanson, O. O'Malley, J. Pandey, Y. Yuan, R. Lee,
and X. Zhang. “Major technical advancements in apache hive.” In Proc. SIGMOD, 2014.
9/30/2015SQL-on-Hadoop Tutorial
139
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 A. Ailamaki, D. J. DeWitt, M. D. Hill, and M. Skounakis. “Weaving Relations for Cache Performance.” In
Proc. of the 27th International Conference on Very Large Data Bases, 2001.
 Y. He, R. Lee, Y. Huai, Z. Shao, N. Jain, X. Zhang, and Z. Xu. “RCFile: A fast and space-efficient data
placement structure in MapReduce-based warehouse systems.” In Proc. of ICDE, 2011.
 H. Lim, H. Herodotou, and S. Babu. “Stubby: a transformation-based optimizer for MapReduce
workflows.” PVLDB, 2012.
 T. Neumann. “Efficiently compiling efficient query plans for modern hardware.” PVLDB, 2011.
 D. Simmen, E. Shekita, and T. Malkemus. “Fundamental techniques for order optimization.” In Proc. of
ACM SIGMOD, 1996.
 A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy.” Hive
- A Petabyte Scale Data Warehouse Using Hadoop.” In ICDE, 2010.
 T. Willhalm, N. Popovici, Y. Boshmaf, H. Plattner, A. Zeier, and J. Schaffner. “SIMD-scan: ultra fast in-
memory table scan using on-chip vector processing units.” PVLDB, 2, 2009.
9/30/2015SQL-on-Hadoop Tutorial
140
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 V. Raman, G. Attaluri, R. Barber, N. Chainani, D. Kalmuk, V. KulandaiSamy, J. Leenstra, S. Light- stone,
S. Liu, G. M. Lohman, T. Malkemus, R. Mueller, I. Pandis, B. Schiefer, D. Sharpe, R. Sidle, A. Storm,
and L. Zhang. “DB2 with BLU Acceleration: So much more than just a column store.” PVLDB, 6, 2013.
 A. Abouzeid, K. Bajda-Pawlikowski, D. J. Abadi, A. Rasin, and A. Silberschatz. “HadoopDB: An
Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.” PVLDB, 2009.
 A. Abouzied, D. J. Abadi, and A. Silberschatz. “Invisible loading: Access-driven data transfer from raw
files into database systems.” In EDBT, 2013.
 M. Amburst, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A.
Ghodsi, and M. Zaharia. “Spark SQL: Relational data processing in Spark.” In ACM SIGMOD, 2015.
 V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H.
Shah, S. Seth, B. Saha, C. Curino, O. O’Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache
Hadoop YARN: Yet another resource negotiator. In SOCC, 2013.
9/30/2015SQL-on-Hadoop Tutorial
141
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 K. Bajda-Pawlikowski, D. J. Abadi, A. Silberschatz, and E. Paulson. “Efficient processing of data
warehousing queries in a split execution environment.” In ACM SIGMOD, 2011.
 P. Boncz. Vortex: Vectorwise goes Hadoop. http://databasearchitects.blogspot.com/2014/05/vectorwise-
goes-hadoop.html.
 L. Chang, Z. Wang, T. Ma, L. Jian, L. Ma, A. Goldshuv, L. Lonergan, J. Cohen, C. Welton,
G. Sherry, and M. Bhandarkar. “HAWQ: A massively parallel processing SQL engine in hadoop.” In ACM
SIGMOD, 2014.
 G. Graefe. “Encapsulation of parallelism in the Volcano query processing system.” In ACM SIGMOD,
1990.
 S. Gray, F. Özcan, H. Pereyra, B. van der Linden, and A. Zubiri. “IBM Big SQL 3.0: SQL-on-Hadoop
without compromise.” http://public.dhe.ibm.com/common/
ssi/ecm/en/sww14019usen/SWW14019USEN.PDF, 2014.
 F.Özcan, D. Hoa, K. S. Beyer, A. Balmin, C. J. Liu, and Y. Li. “Emerging trends in the enterprise data
analytics: Connecting Hadoop and DB2 warehouse.” In ACM SIGMOD, 2011.
9/30/2015SQL-on-Hadoop Tutorial
142
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. “Dremel:
Interactive analysis of web-scale datasets.” PVLDB, 2010.
 S. Padmanabhan, T. Malkemus, R. C. Agarwal, and A. Jhingran. “Block oriented processing of relational
database operations in modern computer architectures.” In ICDE, 2001.
 B. Saha, H. Shah, S. Seth, G. Vijayaraghavan, A. Murthy, and C. Curino. “Apache Tez: A unifying
framework for modeling and building data processing applications.” In ACM SIGMOD, 2015.
 P. Seshadri, H. Pirahesh, and T. Y. C. Leung. “Complex query decorrelation.” In ICDE, 1996.
 A. Floratou, U. F. Minhas, and F. Özcan. “SQL-on- Hadoop: Full circle back to shared-nothing database
architectures.” PVLDB 7(12), 2014.
 M. Traverso. Presto: Interacting with petabytes of data at Facebook.
https://www.facebook.com/notes/facebook- engineering/presto-interacting-with-petabytes-of-data- at-
facebook/10151786197628920.
 S. Wanderman-Milne and N. Li. Runtime code generation in Cloudera Impala. IEEE Data Eng. Bull.,
2014.
9/30/2015SQL-on-Hadoop Tutorial
143
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 R. S. Xin, J. Rosen, M. Zaharia, M. J. Franklin, S. Shenker, and I. Stoica. “Shark: SQL and rich analytics
at scale.” In ACM SIGMOD, 2013.
 M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. “Spark: Cluster computing with
working sets.” In HotCloud, 2010.
 C. Zuzarte, H. Pirahesh, W. Ma, Q. Cheng, L. Liu, and K. Wong. “WinMagic : Subquery elimination
using window aggregation.” In ACM SIGMOD, 2003.
 F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R.
Gruber. “Bigtable: A Distributed Storage System for Structured Data.”, In OSDI 2006
 B. Chattopadhyay, L. Lin, W. Liu, S. Mittal, P. Aragonda, V. Lychagina, Y. Kwon, and M. Wong. “Tenzing:
A SQL Implementation on the MapReduce Framework.” In VLDB 2011.
 T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. “MRShare: Sharing Across Multiple
Queries in MapReduce.” In VLDB 2010.
9/30/2015SQL-on-Hadoop Tutorial
144
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 G. Wang and C.-Y. Chan. “Multi-Query Optimization in MapReduce Framework.” In VLDB, 2013.
 F. Afrati and J. Ullman. “Optimizing Multiway Joins in a Map-Reduce Environment.” In TKDE 2011.
 Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. “SkewTune: Mitigating Skew in MapReduce
Applications.” In SIGMOD 2012.
 M. Eltabakh, F. Özcan, Y. Sismanis, P. J. Haas, H. Pirahesh, and J. Vondrak, “Eagle-eyed elephant:
split-oriented indexing in Hadoop”, in EDBT 2014.
 M. Eltabakh, Y. Tian, F. Özcan, R. Gemulla, A. Krettek, and J. McPherson, “CoHadoop: Flexible Data
Placement and Its Exploitation in Hadoop”, in PVLDB 4(9), 2011.
 J. Shi, Y. Qiu, U. F. Minhas, L. Jiao, C. Wang, B. Reinwald, and F. Özcan, “Clash of the Titans:
MapReduce vs. Spark for Large Scale Data Analytics” in PVLDB 8(13), 2015
 J. Dittrich, J-A. Quiane-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad, “Hadoop++: Making a Yellos
Elephant Run Like a Cheetah (without it even noticing)”, in PVLDB 3(1-2), 2010.
9/30/2015SQL-on-Hadoop Tutorial
145
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 D. Jiang, B. C. Ooi, L. Shi, and S. Wu, “The Performance of MapReduce: An In-depth Study”, in PVLDB
3(1-2), 2010.
 D. J. DeWitt, R. V. Nehme, S. Shankar, J. Aguilar-Saborit, A. Avanes, M. Flasza, and J. Gramling, “Split
Query Processing in Polybase”, in SIGMOD 2013.
 M. Stonebraker, D. J. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin, “MapReduce
and parallel DBMSs: Friends or Foes?” CACM, 53(1):64–71, 2010.
 A. Floratou, J. M. Patel, E. J. Shekita, and S. Tata, “Column-oriented Storage Techniques for
MapReduce”, in PVLDB, 4(7):419–429, 2011
 S. Harris, A. Sundararajan, E. Branish, and K. Chen, “Blistering Fast SQL Access to Hadoop using IBM
BigInsights 3.0 with Big SQL 3.0”
 S. Blanas and et al., “A comparison of join algorithms for log processing in mapreduce”, in SIGMOD
2010.
9/30/2015SQL-on-Hadoop Tutorial
146
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
References (cont.)
 HDFS caching, http://hadoop.apache.org/docs/current/hadoopproject-dist/hadoop-
hdfs/CentralizedCacheManagement.html.
 S. Babu and H. Herodotou, “Massively Parallel Databases and MapReduce Systems”, in Foundations
and Trends in Databases 5(1), 2013.
 N. Bruno, Y. Kwon, and M-C Wu, “Advanced Join Strategies for Large-Scale Distributed Computation”,
in PVLDB 7(13), 2014
 K. Karanasos, A. Balmin, M. Kutsch, F. Özcan, V. Ercegovac, C. Xia, and J. Jackson, “Dynamically
optimizing queries over large scale data platforms”, in SIGMOD 2014
 J. Dean, and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, in OSDI
2004.
9/30/2015SQL-on-Hadoop Tutorial
147
Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
Thank you!

More Related Content

What's hot

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Databricks
 
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache IcebergData Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Anant Corporation
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
Databricks
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Cloudera, Inc.
 
File Format Benchmarks - Avro, JSON, ORC, & Parquet
File Format Benchmarks - Avro, JSON, ORC, & ParquetFile Format Benchmarks - Avro, JSON, ORC, & Parquet
File Format Benchmarks - Avro, JSON, ORC, & Parquet
Owen O'Malley
 
HBase Application Performance Improvement
HBase Application Performance ImprovementHBase Application Performance Improvement
HBase Application Performance Improvement
Biju Nair
 
Hive 3 - a new horizon
Hive 3 - a new horizonHive 3 - a new horizon
Hive 3 - a new horizon
Thejas Nair
 
Hive tuning
Hive tuningHive tuning
Hive tuning
Michael Zhang
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
Spark Summit
 
Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep dive
t3rmin4t0r
 
Hive Does ACID
Hive Does ACIDHive Does ACID
Hive Does ACID
DataWorks Summit
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxData
 
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark Summit
 
Local Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache PhoenixLocal Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache Phoenix
Rajeshbabu Chintaguntla
 
Optimizing Hive Queries
Optimizing Hive QueriesOptimizing Hive Queries
Optimizing Hive Queries
Owen O'Malley
 
What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?
DataWorks Summit
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Databricks
 
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital KediaTuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
Databricks
 

What's hot (20)

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
 
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache IcebergData Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg
 
The Apache Spark File Format Ecosystem
The Apache Spark File Format EcosystemThe Apache Spark File Format Ecosystem
The Apache Spark File Format Ecosystem
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
File Format Benchmarks - Avro, JSON, ORC, & Parquet
File Format Benchmarks - Avro, JSON, ORC, & ParquetFile Format Benchmarks - Avro, JSON, ORC, & Parquet
File Format Benchmarks - Avro, JSON, ORC, & Parquet
 
HBase Application Performance Improvement
HBase Application Performance ImprovementHBase Application Performance Improvement
HBase Application Performance Improvement
 
Hive 3 - a new horizon
Hive 3 - a new horizonHive 3 - a new horizon
Hive 3 - a new horizon
 
Hive tuning
Hive tuningHive tuning
Hive tuning
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
 
Hive+Tez: A performance deep dive
Hive+Tez: A performance deep diveHive+Tez: A performance deep dive
Hive+Tez: A performance deep dive
 
Hive Does ACID
Hive Does ACIDHive Does ACID
Hive Does ACID
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
 
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
 
Local Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache PhoenixLocal Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache Phoenix
 
Optimizing Hive Queries
Optimizing Hive QueriesOptimizing Hive Queries
Optimizing Hive Queries
 
What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?What is new in Apache Hive 3.0?
What is new in Apache Hive 3.0?
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
 
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital KediaTuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
 

Similar to SQL-on-Hadoop Tutorial

Big Data Training in Amritsar
Big Data Training in AmritsarBig Data Training in Amritsar
Big Data Training in Amritsar
E2MATRIX
 
Big Data Training in Mohali
Big Data Training in MohaliBig Data Training in Mohali
Big Data Training in Mohali
E2MATRIX
 
Big Data Training in Ludhiana
Big Data Training in LudhianaBig Data Training in Ludhiana
Big Data Training in Ludhiana
E2MATRIX
 
Hadoop online training
Hadoop online training Hadoop online training
Hadoop online training
Keylabs
 
Hadoop HDFS and Oracle
Hadoop HDFS and OracleHadoop HDFS and Oracle
Hadoop HDFS and Oracle
Johan Louwers
 
12 SQL On-Hadoop Tools
12 SQL On-Hadoop Tools12 SQL On-Hadoop Tools
12 SQL On-Hadoop Tools
Xplenty
 
Future of-hadoop-analytics
Future of-hadoop-analyticsFuture of-hadoop-analytics
Future of-hadoop-analytics
MapR Technologies
 
Hadoop Tutorial for Beginners
Hadoop Tutorial for BeginnersHadoop Tutorial for Beginners
Hadoop Tutorial for Beginners
business Corporate
 
Big Data Hoopla Simplified - TDWI Memphis 2014
Big Data Hoopla Simplified - TDWI Memphis 2014Big Data Hoopla Simplified - TDWI Memphis 2014
Big Data Hoopla Simplified - TDWI Memphis 2014
Rajan Kanitkar
 
Oracle Unified Information Architeture + Analytics by Example
Oracle Unified Information Architeture + Analytics by ExampleOracle Unified Information Architeture + Analytics by Example
Oracle Unified Information Architeture + Analytics by Example
Harald Erb
 
Big SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor LandscapeBig SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor Landscape
Nicolas Morales
 
Hadoop .pdf
Hadoop .pdfHadoop .pdf
Hadoop .pdf
SudhanshiBakre1
 
Big data or big deal
Big data or big dealBig data or big deal
Big data or big deal
eduarderwee
 
What is Apache Hadoop and its ecosystem?
What is Apache Hadoop and its ecosystem?What is Apache Hadoop and its ecosystem?
What is Apache Hadoop and its ecosystem?
tommychauhan
 
Overview of big data & hadoop version 1 - Tony Nguyen
Overview of big data & hadoop   version 1 - Tony NguyenOverview of big data & hadoop   version 1 - Tony Nguyen
Overview of big data & hadoop version 1 - Tony Nguyen
Thanh Nguyen
 
Overview of Big data, Hadoop and Microsoft BI - version1
Overview of Big data, Hadoop and Microsoft BI - version1Overview of Big data, Hadoop and Microsoft BI - version1
Overview of Big data, Hadoop and Microsoft BI - version1
Thanh Nguyen
 
Hadoop and Big Data: Revealed
Hadoop and Big Data: RevealedHadoop and Big Data: Revealed
Hadoop and Big Data: Revealed
Sachin Holla
 
Introduction to Apache hadoop
Introduction to Apache hadoopIntroduction to Apache hadoop
Introduction to Apache hadoop
Omar Jaber
 
hadoop_module
hadoop_modulehadoop_module
Hadoop in action
Hadoop in actionHadoop in action
Hadoop in action
Mahmoud Yassin
 

Similar to SQL-on-Hadoop Tutorial (20)

Big Data Training in Amritsar
Big Data Training in AmritsarBig Data Training in Amritsar
Big Data Training in Amritsar
 
Big Data Training in Mohali
Big Data Training in MohaliBig Data Training in Mohali
Big Data Training in Mohali
 
Big Data Training in Ludhiana
Big Data Training in LudhianaBig Data Training in Ludhiana
Big Data Training in Ludhiana
 
Hadoop online training
Hadoop online training Hadoop online training
Hadoop online training
 
Hadoop HDFS and Oracle
Hadoop HDFS and OracleHadoop HDFS and Oracle
Hadoop HDFS and Oracle
 
12 SQL On-Hadoop Tools
12 SQL On-Hadoop Tools12 SQL On-Hadoop Tools
12 SQL On-Hadoop Tools
 
Future of-hadoop-analytics
Future of-hadoop-analyticsFuture of-hadoop-analytics
Future of-hadoop-analytics
 
Hadoop Tutorial for Beginners
Hadoop Tutorial for BeginnersHadoop Tutorial for Beginners
Hadoop Tutorial for Beginners
 
Big Data Hoopla Simplified - TDWI Memphis 2014
Big Data Hoopla Simplified - TDWI Memphis 2014Big Data Hoopla Simplified - TDWI Memphis 2014
Big Data Hoopla Simplified - TDWI Memphis 2014
 
Oracle Unified Information Architeture + Analytics by Example
Oracle Unified Information Architeture + Analytics by ExampleOracle Unified Information Architeture + Analytics by Example
Oracle Unified Information Architeture + Analytics by Example
 
Big SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor LandscapeBig SQL Competitive Summary - Vendor Landscape
Big SQL Competitive Summary - Vendor Landscape
 
Hadoop .pdf
Hadoop .pdfHadoop .pdf
Hadoop .pdf
 
Big data or big deal
Big data or big dealBig data or big deal
Big data or big deal
 
What is Apache Hadoop and its ecosystem?
What is Apache Hadoop and its ecosystem?What is Apache Hadoop and its ecosystem?
What is Apache Hadoop and its ecosystem?
 
Overview of big data & hadoop version 1 - Tony Nguyen
Overview of big data & hadoop   version 1 - Tony NguyenOverview of big data & hadoop   version 1 - Tony Nguyen
Overview of big data & hadoop version 1 - Tony Nguyen
 
Overview of Big data, Hadoop and Microsoft BI - version1
Overview of Big data, Hadoop and Microsoft BI - version1Overview of Big data, Hadoop and Microsoft BI - version1
Overview of Big data, Hadoop and Microsoft BI - version1
 
Hadoop and Big Data: Revealed
Hadoop and Big Data: RevealedHadoop and Big Data: Revealed
Hadoop and Big Data: Revealed
 
Introduction to Apache hadoop
Introduction to Apache hadoopIntroduction to Apache hadoop
Introduction to Apache hadoop
 
hadoop_module
hadoop_modulehadoop_module
hadoop_module
 
Hadoop in action
Hadoop in actionHadoop in action
Hadoop in action
 

More from Daniel Abadi

Leopard: Lightweight Partitioning and Replication for Dynamic Graphs
Leopard: Lightweight Partitioning and Replication  for Dynamic Graphs Leopard: Lightweight Partitioning and Replication  for Dynamic Graphs
Leopard: Lightweight Partitioning and Replication for Dynamic Graphs
Daniel Abadi
 
The Power of Determinism in Database Systems
The Power of Determinism in Database SystemsThe Power of Determinism in Database Systems
The Power of Determinism in Database Systems
Daniel Abadi
 
Beckman abadi-5min-pres
Beckman abadi-5min-presBeckman abadi-5min-pres
Beckman abadi-5min-pres
Daniel Abadi
 
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
Daniel Abadi
 
Shared slides-edbt-keynote-03-19-13
Shared slides-edbt-keynote-03-19-13Shared slides-edbt-keynote-03-19-13
Shared slides-edbt-keynote-03-19-13
Daniel Abadi
 
Invisible loading
Invisible loadingInvisible loading
Invisible loading
Daniel Abadi
 
Boston Hadoop Meetup, April 26 2012
Boston Hadoop Meetup, April 26 2012Boston Hadoop Meetup, April 26 2012
Boston Hadoop Meetup, April 26 2012
Daniel Abadi
 
Hadoop and Graph Data Management: Challenges and Opportunities
Hadoop and Graph Data Management: Challenges and OpportunitiesHadoop and Graph Data Management: Challenges and Opportunities
Hadoop and Graph Data Management: Challenges and Opportunities
Daniel Abadi
 
CAP, PACELC, and Determinism
CAP, PACELC, and DeterminismCAP, PACELC, and Determinism
CAP, PACELC, and Determinism
Daniel Abadi
 
Column-Stores vs. Row-Stores: How Different are they Really?
Column-Stores vs. Row-Stores: How Different are they Really?Column-Stores vs. Row-Stores: How Different are they Really?
Column-Stores vs. Row-Stores: How Different are they Really?
Daniel Abadi
 
Daniel Abadi HadoopWorld 2010
Daniel Abadi HadoopWorld 2010Daniel Abadi HadoopWorld 2010
Daniel Abadi HadoopWorld 2010
Daniel Abadi
 
VLDB 2009 Tutorial on Column-Stores
VLDB 2009 Tutorial on Column-StoresVLDB 2009 Tutorial on Column-Stores
VLDB 2009 Tutorial on Column-Stores
Daniel Abadi
 
Daniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 PanelDaniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 Panel
Daniel Abadi
 

More from Daniel Abadi (13)

Leopard: Lightweight Partitioning and Replication for Dynamic Graphs
Leopard: Lightweight Partitioning and Replication  for Dynamic Graphs Leopard: Lightweight Partitioning and Replication  for Dynamic Graphs
Leopard: Lightweight Partitioning and Replication for Dynamic Graphs
 
The Power of Determinism in Database Systems
The Power of Determinism in Database SystemsThe Power of Determinism in Database Systems
The Power of Determinism in Database Systems
 
Beckman abadi-5min-pres
Beckman abadi-5min-presBeckman abadi-5min-pres
Beckman abadi-5min-pres
 
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
From HadoopDB to Hadapt: A Case Study of Transitioning a VLDB paper into Real...
 
Shared slides-edbt-keynote-03-19-13
Shared slides-edbt-keynote-03-19-13Shared slides-edbt-keynote-03-19-13
Shared slides-edbt-keynote-03-19-13
 
Invisible loading
Invisible loadingInvisible loading
Invisible loading
 
Boston Hadoop Meetup, April 26 2012
Boston Hadoop Meetup, April 26 2012Boston Hadoop Meetup, April 26 2012
Boston Hadoop Meetup, April 26 2012
 
Hadoop and Graph Data Management: Challenges and Opportunities
Hadoop and Graph Data Management: Challenges and OpportunitiesHadoop and Graph Data Management: Challenges and Opportunities
Hadoop and Graph Data Management: Challenges and Opportunities
 
CAP, PACELC, and Determinism
CAP, PACELC, and DeterminismCAP, PACELC, and Determinism
CAP, PACELC, and Determinism
 
Column-Stores vs. Row-Stores: How Different are they Really?
Column-Stores vs. Row-Stores: How Different are they Really?Column-Stores vs. Row-Stores: How Different are they Really?
Column-Stores vs. Row-Stores: How Different are they Really?
 
Daniel Abadi HadoopWorld 2010
Daniel Abadi HadoopWorld 2010Daniel Abadi HadoopWorld 2010
Daniel Abadi HadoopWorld 2010
 
VLDB 2009 Tutorial on Column-Stores
VLDB 2009 Tutorial on Column-StoresVLDB 2009 Tutorial on Column-Stores
VLDB 2009 Tutorial on Column-Stores
 
Daniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 PanelDaniel Abadi: VLDB 2009 Panel
Daniel Abadi: VLDB 2009 Panel
 

Recently uploaded

Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3
DianaGray10
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
janagijoythi
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
Priyanka Aash
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
The Impact of the Internet of Things (IoT) on Smart Homes and Cities
The Impact of the Internet of Things (IoT) on Smart Homes and CitiesThe Impact of the Internet of Things (IoT) on Smart Homes and Cities
The Impact of the Internet of Things (IoT) on Smart Homes and Cities
Arpan Buwa
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
FIDO Alliance
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
Zilliz
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Zilliz
 

Recently uploaded (20)

Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3Communications Mining Series - Zero to Hero - Session 3
Communications Mining Series - Zero to Hero - Session 3
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
The Impact of the Internet of Things (IoT) on Smart Homes and Cities
The Impact of the Internet of Things (IoT) on Smart Homes and CitiesThe Impact of the Internet of Things (IoT) on Smart Homes and Cities
The Impact of the Internet of Things (IoT) on Smart Homes and Cities
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Retrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with RagasRetrieval Augmented Generation Evaluation with Ragas
Retrieval Augmented Generation Evaluation with Ragas
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...
 

SQL-on-Hadoop Tutorial

  • 1. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) SQL-on-Hadoop Tutorial VLDB 2015 9/30/2015SQL-on-Hadoop Tutorial 1
  • 2. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial Fatma Özcan IBM Research IBM Big SQL Ippokratis Pandis Cloudera Cloudera Impala Daniel Abadi Yale University and Teradata HadoopDB/Hadapt Shivnath Babu Duke University Starfish 2 Presenters
  • 3. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Why SQL-on-Hadoop?  People need to process data in parallel  Hadoop is by far the leading open source parallel data processing platform  Low costs of HDFS results in heavy usage 9/30/2015SQL-on-Hadoop Tutorial 3 Lots of data in Hadoop with appetite to process it
  • 4. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce is not the answer  MapReduce is a powerful primitive to do many kinds of parallel data processing  BUT  Little control of data flow  Fault tolerance guarantees not always necessary  Simplicity leads to inefficiencies  Does not interface with existing analysis software  Industry has existing training in SQL 9/30/2015SQL-on-Hadoop Tutorial 4 SQL interface for Hadoop critical for mass adoption
  • 5. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Decades of research in parallel database systems Efficient data flow Load balancing in the face of skew Query optimization Vectorized processing Dynamic compilation of query operators Co-processing of queries 9/30/2015SQL-on-Hadoop Tutorial 5 Massive talent war between SQL-on-Hadoop companies for members of database community The database community knows how to process data
  • 6. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) SQL-on-Hadoop is not a direct implementation of parallel DBMSs Little control of storage  Most deployments must be over HDFS Append-only file system  Must support many different storage formats Avro, Parquet, RCFiles, ORC, Sequence Files Little control of metadata management  Optimizer may have limited access to statistics Little control of resource management  YARN still in its infancy 9/30/2015SQL-on-Hadoop Tutorial 6
  • 7. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) SQL-on-Hadoop is not a direct implementation of parallel DBMSs Hadoop often used a data dump (swamp?)  Data often unclean, irregular, and unreliable Data not necessarily relational  HDFS does not enforce structure in the data  Nested data stored as JSON extremely popular Scale larger than previous generation parallel database systems  Fault tolerance vs. query performance Most Hadoop components written in Java Want to play nicely with the entire Hadoop ecosystem 9/30/2015SQL-on-Hadoop Tutorial 7
  • 8. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Outline of Tutorial This session [13:30-15:00] SQL-on-Hadoop Technologies Storage Run-time engine Query optimization Q&A 9/30/2015SQL-on-Hadoop Tutorial 8 Second Session [15:30-17:00] SQL-on-Hadoop examples HadoopDB/Hadapt Presto Impala BigSQL SparkSQL Phoenix/Spice Machine Research directions Q&A
  • 9. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Storage 9/30/2015SQL-on-Hadoop Tutorial 9
  • 10. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Quick Look at HDFS 9/30/2015SQL-on-Hadoop Tutorial 10 … NameNode DataNode DataNode DataNode
  • 11. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Good for Storing large files Write once and read many times “Cheap” commodity hardware Not good for Low-latency reads Short-circuit reads and HDFS caching help Large amounts of small files Multiple writers 9/30/2015SQL-on-Hadoop Tutorial 11 HDFS is
  • 12. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) In-situ Data Processing HDFS as the data dump Store the data first, figure out what to do later Most data arrive in text format Transform, cleanse the data Create data marts in columnar formats Lost of nested, JSON data Some SQL in data transformations, but mostly other languages, such as Pig, Cascading, etc.. Columnar formats are good for analytics 9/30/2015SQL-on-Hadoop Tutorial 12
  • 13. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Most SQL-on-Hadoop systems do not control or own the data  Hive, Impala, Presto, Big SQL, Spark SQL, Drill  Other SQL-on-Hadoop systems tolerate HDFS data, but work better with their own proprietary storage  HadoopDB/Hadapt  HAWQ, Actian Vortex, and HP Vertica 9/30/2015SQL-on-Hadoop Tutorial 13 SQL-on-Hadoop according to storage formats
  • 14. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Only support native Hadoop formats with open-source reader/writers Any Hadoop tool can generate their data Pig, Cascading and other ETL tools They are more of a query processor than a database Indexing is a challenge !! No co-location of multiple tables Due to HDFS 9/30/2015SQL-on-Hadoop Tutorial 14 Query Processors with HDFS Native Formats
  • 15. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Almost all exploit some existing database systems They store their own binary format on HDFS Hadapt stores the data in a single node database, like postgres Can exploit Postgres indexes HAWQ, Actian, HP Vertica, and Hadapt all control how tables are partitioned, and can support co-located joins 9/30/2015SQL-on-Hadoop Tutorial 15 Systems with Proprietary Formats
  • 16. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) CSV files are most common for ETL-like workloads Lots of nested and complex data Arrays, structs, maps, collections Two major columnar formats ORCFile Parquet Data serialization JSON and Avro Protocol buffers and Thrift 9/30/2015SQL-on-Hadoop Tutorial 16 HDFS Native Formats
  • 17. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 17 9/30/2015SQL-on-Hadoop Tutorial 17 Parquet  PAX format, supporting nested data  Idea came from the Google‘s Dremel System  Major contributors: Twitter & Cloudera  Provides dictionary encoding and several compressions  Preffered format for Impala, IBM Big SQL, and Drill  Can use Thrift or Avro to describe the schema Nested data  A natural schema  Flexible  Less duplication applying denormalization Columnar storage  Fast compression  Schema projection  Efficient encoding
  • 18. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Parquet, cont.  A table with N columns is split into M row groups.  The file metadata contains the locations of all the column metadata start locations.  Metadata is written after the data to allow for single pass writing.  There are three types of metadata: file metadata, column (chunk) metadata and page header metadata.  Row group metadata includes  Min-max values for skipping 9/30/2015SQL-on-Hadoop Tutorial 18
  • 19. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Second generation, following RC file  PAX formats with all data in a single file  Hortonworks is the major contributor, together with Microsoft  Preferred format for Hive, and Presto  Supports  Dictionary encoding  Fast compression  File, and stripe level metadata  Stripe indexing for skipping  Now metadata even includes bloom filters for point query lookups 9/30/2015SQL-on-Hadoop Tutorial 19 ORCFile
  • 20. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 20 ORCFile Layout
  • 21. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  No updates in HDFS  Appends to HDFS files are supported, but not clear how much they are used in production  Updates are collected in delta files  At the time of read delta and main files are merged  Special inputFormats  Lazy compaction to merge delta files and main files  When delta files reach a certain size  Scheduled intervals 9/30/2015SQL-on-Hadoop Tutorial 21 File A delta 1 delta 2 delta n … Handling Updates in HDFS
  • 22. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) SQL on NoSQL!  Put a NoSQL solution on top of HDFS  For the record, you can avoid HDFS completely  But, this is a SQL-on-Hadoop tutorial  NoSQL solutions can provide CRUD at scale  CRUD = Create, Read, Update, Delete  And, then run SQL on it?  Sounds crazy? Well, lets see 9/30/2015SQL-on-Hadoop Tutorial 22
  • 23. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 23 HBase: The Hadoop Database  Not HadoopDB, which we will see later in the tutorial  HBase is a data store built on top of HDFS based on Google Bigtable  Data is logically organized into tables, rows, and columns  Although, Key-Value storage principles are used at multiple points in the design  Columns are organized into Column Families (CF)  Supports record-level CRUD, record-level lookup, random updates  Supports latency-sensitive operations
  • 24. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HBase Architecture24
  • 25. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HBase Architecture25 HBase stores three types of files on HDFS: • WALs • HFiles • Links
  • 26. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HBase Read and Write Paths26
  • 27. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HFile Structure 27 • Immutable • Created on flush or compaction • Sequential writes • Read randomly or sequentially • Data is in blocks • HFile blocks are not HDFS blocks • Default data block size == 64K • Default index block size == 128K • Default bloom filter block size == 128K • Use smaller block sizes for faster random lookup • Use larger block sizes for faster scans • Compression is recommended • Block encoding is recommended HFile Format
  • 28. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Run-time Engine 9/30/2015SQL-on-Hadoop Tutorial 28
  • 29. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Low Latency High Throughput Degree of tolerance to faults Scalability in data size Scalability in cluster size Resource elasticity Multi-tenancy Ease of installation in existing environments 9/30/2015SQL-on-Hadoop Tutorial 29 Design Decisions: Influencers
  • 30. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Push computation to data Columnar data formats Vectorization Support for multiple data formats Support for UDFs 9/30/2015SQL-on-Hadoop Tutorial 30 Accepted across SQL-on-Hadoop Solutions
  • 31. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) What is the Lowest Common Execution Unit Use of Push Vs. Pull On the JVM or not Fault tolerance: Intra-query or inter-query Support for multi-tenancy 9/30/2015SQL-on-Hadoop Tutorial 31 Differences across SQL-on-Hadoop Solutions
  • 32. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Hive Tenzing 9/30/2015SQL-on-Hadoop Tutorial 32 SQL on MapReduce
  • 33. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 33 Hive
  • 34. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 34 Example: Joins in MapReduce
  • 35. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Having a MapReduce Job as the Lowest Execution Unit quickly becomes restrictive Query execution plans become MapReduce workflows 9/30/2015SQL-on-Hadoop Tutorial 35 Limitations
  • 36. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) D1 D2 D3 D4 D6D5 D7 D01 D02 J1 J2 J3 J4 J5 J6 J7 MapReduce Jobs 36 Datasets MapReduce Workflows
  • 37. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) On efficient joins in the MapReduce paradigm On reducing the number of MapReduce jobs by packing/collapsing the MapReduce workflow Horizontally Shared scans Vertically Making using of static and dynamic partitioning On efficient management of intermediate data 9/30/2015SQL-on-Hadoop Tutorial 37 Research Done to Address these Limitations
  • 38. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Dryad Tez 9/30/2015SQL-on-Hadoop Tutorial 38 From MapReduce to DAGs
  • 39. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 39 Dryad: Dataflows as First-class Citizens
  • 40. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 40 Smart DAG Execution in Dryad
  • 41. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Tez: Inspired by Dryad and Powered by YARN 9/30/2015SQL-on-Hadoop Tutorial 41
  • 42. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  The Hadoop Community realized that MapReduce cannot be the Lowest Execution Unit for all data apps  Separated out the resource management aspects from application management  YARN is best seen as an Operating System for Data Processing Apps Recall the 80s: Databases and Operating Systems: Friends or Foes? 9/30/2015SQL-on-Hadoop Tutorial 42 Quick Detour on YARN
  • 43. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) An Example of What Tez Enables 9/30/2015SQL-on-Hadoop Tutorial 43
  • 44. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) A Tez Slide on Tez 9/30/2015SQL-on-Hadoop Tutorial 44
  • 45. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 map filter reduceBykey map saveAsTextFile part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Spark: A Different Way to Look at a Dataflow
  • 46. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 map filter reduceBykey map saveAsTextFile part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 HDFS Stage 0 Stage 1 RDD0 RDD1 RDD2 RDD3 RDD4 Spark: A Different Way to Look at a Dataflow
  • 47. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 map filter reduceBykey map saveAsTextFile part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS Stage 0 Stage 1 sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Spark: A Different Way to Look at a Dataflow
  • 48. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 map filter reduceBykey map saveAsTextFile part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 part- 0 part- 1 part- 2 part- 3 Exec0Exec1Exec2 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1 Spark: A Different Way to Look at a Dataflow
  • 49. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Spark: A Different Way to Look at a Dataflow 9/30/2015SQL-on-Hadoop Tutorial 49
  • 50. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 50
  • 51. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 51 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 52. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 52 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 53. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 53 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 54. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 54 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 55. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 55 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 56. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 56 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce
  • 57. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 57 0 20 40 60 80 100 120 140 160 180 200 Fault tolerance Slowdown tolerance PercentageSlowdown Traditional DBMS MapReduce • SELECT sourceIP, SUM(adRevenue) FROM UserVisits GROUP BY sourceIP • Node fails (or slows down by factor of 2) in the middle of query Fault Tolerance
  • 58. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Downsides of MapReduce Fault Tolerance 9/30/2015SQL-on-Hadoop Tutorial 58 Map Map Map Map Map Reduce Reduce Reduce Reduce HDFSHDFS Map Map Map Map Map Reduce Reduce Reduce Reduce Map output written to disk Reduce output written to HDFS
  • 59. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Spark RDDs Stores intermediate results in memory rather than disk Advantage: Performance Disadvantage: Memory requirements 9/30/2015SQL-on-Hadoop Tutorial 59
  • 60. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Resource Management 9/30/2015SQL-on-Hadoop Tutorial 60
  • 61. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Resource Management (At least) Two dimension problem: 1. RM across different frameworks Usually not a dedicated cluster Shared across multiple frameworks ETL (MapReduce, Spark), Hbase SQL-on-Hadoop processing 2. RM across concurrent queries 9/30/2015SQL-on-Hadoop Tutorial 61
  • 62. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) RM -- Across frameworks  YARN – Yet Another Resource Negotiator  Centralized, cluster-wide resource management system Allows frameworks to share resources without partitioning between them  Designed for batch-mostly processing Not mature Not good for interactive analytics Not meant for long running processes Approaches: Llama and Slider 9/30/2015SQL-on-Hadoop Tutorial 62
  • 63. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) RM -- LLAMA (low-latency application master) Introduced by Cloudera LLAMA acts as a proxy between Impala and YARN Mitigates some of the batch-centric design aspects of YARN: High resource acquisition latency -> solves via resource caching Resource request is immutable -> solves via expansion request Resource allocation is incremental -> solves via gang scheduling 6 3
  • 64. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Slider allows running non-YARN enabled applications on YARN  Without having to write your own custom Application Master  Existing applications are packaged as Slider applications  Encapsulates a set of one or more application components or roles  Deployed by Slider, runs in containers across a YARN cluster  Pre-built packages for HBase, Accumulo, Storm, and jmemcached  Packages need to be custom built for other applications  Some notable Slider features  Applications can be stopped and started later  state is persisted  Container failures are automatically detected by Slider and restarted 64 64 RM -- Apache Slider
  • 65. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Query Optimization 9/30/2015SQL-on-Hadoop Tutorial 65
  • 66. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Some Techniques We Know and Love Are not Directly Applicable Indexing Zone-maps Co-located joins Query rewrites Cost-based optimization  Databases own their storage SQL-on-Hadoop systems do not  Metadata management is tricky  Data inserted/loaded without SQL system knowledge  No co-location of related tables  HDFS is for most practical purposes, read-only 9/30/2015SQL-on-Hadoop Tutorial 66
  • 67. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Hive Partition tables maintain metadata values as one folder/ directory in HDFS, per distinct value:  Example: PARTITIONED BY (country STRING, year INT, month INT, day INT) ; Folder/Directory created for country=US/year=2012/month=12/day=22 Partitioning only logical, not physical  Partition pruning eliminates reading files that are not needed  Almost all SQL-on-Hadoop offerings support this  Hive, Impala, SparkSQL, IBM BigSQL, …. 9/30/2015SQL-on-Hadoop Tutorial 67 I/O Elimination for HDFS Data: Partition-level
  • 68. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  ORCFile broken into Stripes (250MB default)  Index with Min/Max values stored for each Column  Data is a “stream” of columns  Bloom filters for each stripe in ORCFile allow fast lookups  Parquet also supports min/max values  Works well when data is sorted, not very effective otherwise 9/30/2015SQL-on-Hadoop Tutorial 68 I/O Elimination for HDFS Data: Rowblock-level
  • 69. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Quick look at query optimizers  Two types of optimization  Logical transformations to transform query into equivalent but simpler form  Cost-based enumeration of alternative execution plans  Most systems support the first one  Cost-based optimization depends on good statistics and a good model of the execution environment  Without controlling data storage, statistics are “gestimates” 9/30/2015SQL-on-Hadoop Tutorial 69
  • 70. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Selection/projection pushdown  Nested SQL queries require more sophisticated rewrites, such as decorrelation  New systems all have rewrites but lack complex decorrelation and subquery optimization ones  Hive, Impala, Presto, Spark SQL  Systems that leverage mature DB technology offer more sophisticated rewrite engines  IBM SQL, Hadapt, HP Vertica 9/30/2015SQL-on-Hadoop Tutorial 70 Query Rewrite
  • 71. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Hive analyze table collects basic statistics  Column value distributions, min-max, no-of-distinct values  No control of data  data changes without the systems’ knowledge  Multi-tenant system makes it harder to build a cost model  More complex system behavior 9/30/2015SQL-on-Hadoop Tutorial 71 More adaptive query processing is needed Cost-based Optimization
  • 72. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Co-partitioning two tables on the join key enables local joins  HDFS default block placement policy scatters blocks in the cluster  Actian Vortex changes HDFS default block placement to enforce co- located joins 9/30/2015SQL-on-Hadoop Tutorial 72  Files A & B are co-located  Files C & D are co-located File A File B File DFile C Co-located joins
  • 73. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Outline of Tutorial This session [13:30-15:00] SQL-on-Hadoop Technologies Storage Run-time engine Query optimization Q&A 9/30/2015SQL-on-Hadoop Tutorial 73 Second Session [15:30-17:00] SQL-on-Hadoop examples HadoopDB/Hadapt Presto Impala BigSQL SparkSQL Phoenix/Spice Machine Research directions Q&A
  • 74. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HadoopDB  First of avalanche of SQL-on-Hadoop solutions to claim 100x faster than Hive (on certain types of queries)  Used Hadoop MapReduce to coordinate execution of multiple independent (typically single node, open source) database systems  Maintained MapReduce’s fault tolerance  Sped up single-node processing via leveraging database performance optimizations: Compression Vectorization Partitioning Column-orientation Query optimization Broadcast joins  Flexible query interface (both SQL and MapReduce) 74
  • 75. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HadoopDB Architecture75
  • 76. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HadoopDB SMS Planner76
  • 77. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HadoopDB History  Paper published in 2009  Company founded in 2010 (Hadapt) to commercialize HadoopDB  Added support for search in 2011 (for major insurance customer)  Added JSON support in 2012  Added interactive query engine in 2013  Acquired by Teradata in 2014 9/30/2015SQL-on-Hadoop Tutorial 77
  • 78. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Teradata Unified Data Architecture: QueryGrid Marketing Executives Operational Systems Customers & Partners Frontline Workers Business Analysts Data Scientists Engineers & Programmers SQL-H SQL SQL, NOSQL DATA PLATFORM HADOOP OR TERADATA INTEGRATED DATA WAREHOUSE TERADATA DATABASE ASTER DATABASE DISCOVERY PLATFORM COMPUTE CLUSTERS SAS, PYTHON, R, PERL, RUBY… OTHER DATABASES ORACLE, MONGODB, ETC SQL VARIOUS TERADATA OR ASTER DATABASE TERADATA QUERYGRID PUSH DOWN / REMOTE PROCESSING
  • 79. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Remote Processing On Hadoop Query through Teradata Leaves of query plan sent to SQL-on-Hadoop engine Results returned to Teradata Additional query processing done in Teradata Final results sent back to application/user Teradata 15.0
  • 80. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Bi-directional data movement  Read and write data to Hadoop  Create new table in Hadoop or insert records  Query push-down  Execute query on Hadoop  Qualify rows and columns to reduce data returned  Easy configuration and simplified queries  Create “Hadoop server” definition once  Use @foreign_server name to access Hadoop Teradata QueryGrid Teradata-Hadoop
  • 81. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) History of Presto FALL 2012 6 developers start Presto development FALL 2014 88 releases 41 contributors 3943 commits SPRING 2015 Teradata provides first commercial support for Presto + roadmap SPRING 2013 Presto rolled out within Facebook FALL 2013 Facebook open sources Presto FALL 2008 Facebook open sources Hive
  • 82. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Hive Reduce Reduce Map Map Reduce Reduce Map Map Disk Disk Disk Wait between stages Write to Disk • Fault Tolerance • IO Overhead Presto Task Task Task Task Task Task Task All stages are pipelined • Reduced wait time • No Fault Tolerance Memory-to-memory Data transfer • No disc IO • Data chunk must fit in memory
  • 83. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Uses Hive metastore  Bytecode query compilation  Approximate queries  Return X% sample rows  Limitations  Manual join SQL ordering  Non-equi joins not supported  Not YARN enabled  No Avro support  No spill-to-disk  Written in Java  100% ANSI SQL goal  Numerous built-in functions  Window functions  Array/map support  Plug-in architecture  Join across data stores  Hive, Cassandra, Kafka, MySQL  Amazon S3 Presto at a Glance
  • 84. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Presto Pipeline Architecture Data stream API Worker Data stream API Worker Coordinator Data Location API Metadata API Parser/ analyzer Planner Scheduler Worker Client
  • 85. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Presto Connectors Client Presto worker Presto worker Presto worker Presto worker Presto Coordinator
  • 86. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Github: Presto Plug-in Connectors  Hive tables and HCatalog  Apache Cassandra  Apache Kafka  Kafka topics = Presto tables, messages = rows  MySQL  Single node access only -- no sharding  Postgres  Single node access only  HBase  not released
  • 87. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Cloudera Impala 9/30/2015SQL-on-Hadoop Tutorial 87
  • 88. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 8 8 Query Executor SQL App ODBC Hive Metastore HDFS NameNode Statestore Query Planner Query Coordinator HDFS DN HBase Impalad HDFS DN HBase Impalad HDFS DN HBase Impalad Catalog Query Executor Query Planner Query Planner Query Coordinator Query Coordinator Query Executor SQL request Plan Fragments Results Query execution at the high level88
  • 89. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 89 HashJoinScan: t1 Scan: t3 Scan: t2 HashJoin TopN Pre-Agg MergeAgg TopN Broadcast Merge hash t2.idhash t1.id1 hash t1.custid at HDFS DN at HBase RS at coordinator HashJoin Scan: t1 Scan: t3 Scan: t2 HashJoin TopN Agg Single-Node Plan Query Planning: Distributed Plans
  • 90. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Written in C++ for minimal cycle and memory overhead Leverages decades of parallel DB research Partitioned parallelism Pipelined relational operators Batch-at-a-time runtime Focussed on speed and efficiency Intrinsics/machine code for text parsing, hashing, etc. Runtime code generation with LLVM 90 Execution Engine
  • 91. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Uses llvm to jit-compile the runtime-intensive parts of a query  Effect the same as custom-coding a query:  Remove branches, unroll loops  Propagate constants, offsets, pointers, etc.  Inline function calls  Optimized execution for modern CPUs (instruction pipelines) 91 Runtime Code Generation
  • 92. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 92 interpreted codegen’d IntVal my_func(const IntVal& v1, const IntVal& v2) { return IntVal(v1.val * 7 / v2.val); } SELECT my_func(col1 + 10, col2) FROM ... my_fu nc col 2 + 10 col 1 function pointer function pointer function pointer function pointer (col1 + 10) * 7 / col2 function pointer Runtime Code Generation — Example
  • 93. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 93 10 node cluster (12 disks / 48GB RAM / 8 cores per node) ~40 GB / ~60M row Avro dataset Impala Runtime Code Generation - Performance
  • 94. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Codegen is not the panacea!94 TPC-DS 500GB,10-node clusterTPC-H 300GB,10-node cluster
  • 95. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Admission control and Yarn-based RM cater to different workloads Use admission control for:  Low-latency, high-throughput workloads  Mostly running Impala, or resource partitioning is feasible Use Llama/Yarn for:  Mixed workloads (Impala, MR, Spark, …) and resource partitioning is impractical  Latency and throughput SLAs are relatively relaxed Resource Management in Impala95
  • 96. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 96  Nested data: Structs, arrays, maps in Parquet, Avro, JSON, …  Natural extension of SQL: expose nested structures as tables  No limitation on nesting levels or number of nested fields in single query  Multithreaded execution past scan operator  Resource management and admission control  low-latency, high-throughput mixed workloads without resource partitioning  More SQL: ROLLUP/GROUPING SETS, INTERSECT/MINUS, MERGE  Improved query planning, using statistics  Physical tuning Roadmap: Impala 2.3+
  • 97. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Ibis: Scaling the Python Data Experience Target user: Data scientists and data engineers (“Python data users”) Goals: Mirror single-node Python experience, maximize productivity Complete support for SQL engines with Pandas-like API (same designer) High-performance Python user-defined functions Integration with Python data ecosystem / libraries 97 http://www.ibis-project.org/
  • 98. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)
  • 99. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Ibis/Impala Joint Roadmap • More natural data modeling • Complex types support • Integration with full Python data ecosystem • Advanced analytics + machine learning • Enable use of performance computing tools • User extensibility with native performance • In-memory columnar format • Python-to-LLVM IR compilation • Workflow and usability tools 99
  • 100. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Code at github (https://github.com/cloudera/Impala/) Impala Developer Docker Images & Chef scripts https://registry.hub.docker.com/u/cloudera/impala-dev/ Minimal (7GB) — ready to compile, latest code Default (33GB) — includes test data, e.g. TPC-H Shout out to Spyros Blanas (Ohio State) http://web.cse.ohio-state.edu/~sblanas/5242/ Impala JIRAs, ramp-up tasks 100 Academic Challenge
  • 101. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) IBM Big SQL 9/30/2015SQL-on-Hadoop Tutorial 101
  • 102. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Head (coordinator) node  Compiles and optimizes the query  Coordinates the execution of the query  Big SQL worker processes reside on compute nodes (some or all)  Worker nodes stream data between each other as needed Mgmt Node Big SQL Mgmt Node Hive Metastore Mgmt Node Name Node Mgmt Node Job Tracker ••• Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node Compute Node Task Tracker Data Node ••• Big SQL Big SQL Big SQL Big SQL HDFS 102 Big SQL – Architecture
  • 103. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  For common table formats a native I/O engine is utilized  e.g. delimited, RC, SEQ, Parquet, …  For all others, a java I/O engine is used  Maximizes compatibility with existing tables  Allows for custom file formats and SerDe's  All Big SQL built-in functions are native code  Customer built UDF's can be developed in C++ or Java Mgmt Node Big SQL Compute Node Task Tracker Data Node Big SQL Big SQL Worker Native I/O Engine Java I/O Engine Runtime Java UDFs Native UDFs 103 103 Big SQL – Architecture (cont.)
  • 104. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  All data is Hadoop data  In files in HDFS  SEQ, ORC, delimited, Parquet …  Never need to copy data to a proprietary representation  All data is catalog-ed in the Hive metastore  It is the Hadoop catalog  It is flexible and extensible 104 104 Big SQL works with Hadoop
  • 105. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  The scheduler is the main RDBMS↔Hadoop service interface  Interfaces with Hive metastore for table metadata  SQL compiler ask it for some "hadoop" metadata, such as partitioning columns  Acts like the MapReduce job tracker for Big SQL  Big SQL provides query predicates for scheduler to perform partition elimination  Determines splits for each “table” involved in the query  Schedules splits on available Big SQL nodes (with best effort data locality)  Decides which I/O library to use and serves work (splits) to them  Coordinates “commits” after INSERTs Management Node Big SQL Master Node Big SQL Scheduler DDL FMP UDF FMP Mgmt Node Database Service Hive Metastore Big SQL Worker Node Java I/O FMP Native I/O FMP HDFS Data Node MRTask TrackerUDF FMP 105 Scheduler Service
  • 106. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Query Rewrite  There are many ways to express the same query  Query generators often produce suboptimal queries and don't permit "hand optimization"  Complex queries often result in redundancy, especially with views  For large data volumes optimal access plans more crucial as penalty for poor planning is greater select sum(l_extendedprice) / 7.0 avg_yearly from tpcd.lineitem, tpcd.part where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX' and l_quantity < ( select 0.2 * avg(l_quantity) from tpcd.lineitem where l_partkey = p_partkey); select sum(l_extendedprice) / 7.0 as avg_yearly from temp (l_quantity, avgquantity, l_extendeprice) as (select l_quantity, avg(l_quantity) over (partition by l_partkey) as avgquantity, l_extenedprice from tpcd.lineitem, tpcd.part where p_partkey = l_partkey and p_brand = 'BRAND#23' and p_container = 'MED BOX') where l_quantity < 0.2 * avgquantity • Query correlation eliminated • Lineitem table accessed only once • Execution time reduced in half! 106
  • 107. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Cost-based Optimization | 2.66667e-08 HSJOIN ( 7) 1.1218e+06 8351 /--------+-------- 5.30119e+08 3.75e+07 BTQ NLJOIN ( 8) ( 11) 948130 146345 7291 1060 | /----+---- 5.76923e+08 1 3.75e+07 LTQ GRPBY FILTER ( 9) ( 12) ( 20) 855793 114241 126068 7291 1060 1060 | | | 5.76923e+08 13 7.5e+07 TBSCAN TBSCAN BTQ ( 10) ( 13) ( 21) 802209 114241 117135 7291 1060 1060 | | | 7.5e+09 13 5.76923e+06 TABLE: TPCH5TB_PARQ TEMP LTQ ORDERS ( 14) ( 22) Q1 114241 108879 1060 1060 | | 13 5.76923e+06 DTQ TBSCAN ( 15) ( 23) 114241 108325 1060 1060 | | 1 7.5e+08 GRPBY TABLE: TPCH5TB_PARQ ( 16) CUSTOMER 114241 Q5 1060 | 1 LTQ ( 17) 114241 1060 | 1 GRPBY ( 18) 114241 1060 | 5.24479e+06 TBSCAN ( 19) 113931 1060 | 7.5e+08 TABLE: TPCH5TB_PARQ CUSTOMER Q2 Few extensions required to the Cost Model Scan operator cost model extended to evaluate cost of reading from Hadoop # of files, size of files, # of partitions, # of nodes Data not hash partitioned on a particular columns (aka “Scattered partitioned”) New parallel join strategy Every node read data from HDFS, instead of one reading and broadcasting Optimizer now knows in which subset of nodes the data resides => better costing! Sophisticated statistics for cardinality estimation 107
  • 108. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Statistics  Big SQL utilizes Hive statistics collection with some extensions: Additional support for column groups, histograms and frequent values Automatic determination of partitions that require statistics collection vs. explicit Partitioned tables: added table-level versions of NDV, Min, Max, Null count, Average column length Hive catalogs as well as database engine catalogs are also populated We are restructuring the relevant code for submission back to Hive  Capability for statistic fabrication if no stats available at compile time Table statistics •Cardinality (count) •Number of Files •Total File Size Column statistics •Minimum value (all types) •Maximum value (all types) •Cardinality (non-nulls) •Distribution (Number of Distinct Values NDV) •Number of null values •Average Length of the column value (all types) •Histogram - Number of buckets configurable •Frequent Values (MFV) – Number configurable Column group statistics 108
  • 109. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Big SQL supports HBase tables  Big SQL with HBase – basic operations – Create tables and views – LOAD / INSERT data – Query data with full SQL breadth  HBase-specific design points  Column mapping  Dense / composite columns – FORCE KEY UNIQUE option – Secondary indexes – . . . . 109
  • 110. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Big SQL works under YARN  Big SQL integrates with YARN via the Slider project  YARN chooses suitable hosts for Big SQL worker nodes  Big SQL resources are accounted for by YARN  Size of the Big SQL cluster may dynamically grow or shrink as needed  Configured by user (not by installation default)  More Big SQL workers are added when more resources are needed  When demand wears off, Big SQL workers are shut down Data Management SPARK Hive Pig BigSQL Data Access HDFS MapReduce YARN 110
  • 111. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Summary  Big SQL provides rich, robust, standards-based SQL support for data stored in HDFS and HBase  Uses IBM common client ODBC/JDBC drivers  Big SQL fully integrates with SQL applications and tools  Existing queries run with no or few modifications*  Existing JDBC and ODBC compliant tools can be leveraged  Big SQL provides faster and more reliable performance  Big SQL uses more efficient access paths to the data  Big SQL is optimized to more efficiently move data over the network  Big SQL is capable of executing all 22 TPC-H and all 99 TPC-DS queries without modification  Big SQL provides and enterprise grade data management  Security, Auditing, workload management … 111
  • 112. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) SparkSQL 9/30/2015SQL-on-Hadoop Tutorial 112
  • 113. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) From: What is so great about Spark?113
  • 114. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Distributed data analytics engine, generalizing Map Reduce  Core engine, with streaming, SQL, machine learning, and graph processing modules 114 OK, but what exactly is Spark?
  • 115. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) RDDs Distributed collection of objects Can be cached in memory Built via parallel transformations (map, filter, …) Automatically rebuilt on failure based on lineage DAGs of RDDs and Transformations can be (lazily) executed via actions Examples: Export to HDFS, count number of objects Spark Core: RDDs, Transformations & Actions 115
  • 116. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 116 Spark’s DAG Execution
  • 117. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Building a real-world big data application without and with Spark: … HDFS read HDFS write ETL HDFS read HDFS write train HDFS read HDFS write query HDFS HDFS read ETL train query With Spark: Interactive analysis Why Application Developers love Spark117
  • 118. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Raw JSON Tweets SQL Machine Learning Streaming An Example App118
  • 119. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)import org.apache.spark.sql._ val ctx = new org.apache.spark.sql.SQLContext(sc) val tweets = sc.textFile("hdfs:/twitter") val tweetTable = JsonTable.fromRDD(sqlContext, tweets, Some(0.1)) tweetTable.registerAsTable("tweetTable") ctx.sql("SELECT text FROM tweetTable LIMIT 5").collect.foreach(println) ctx.sql("SELECT lang, COUNT(*) AS cnt FROM tweetTable GROUP BY lang ORDER BY cnt DESC LIMIT 10").collect.foreach(println) val texts = sql("SELECT text FROM tweetTable").map(_.head.toString) def featurize(str: String): Vector = { ... } val vectors = texts.map(featurize).cache() val model = KMeans.train(vectors, 10, 10) sc.makeRDD(model.clusterCenters, 10).saveAsObjectFile("hdfs:/model") val ssc = new StreamingContext(new SparkConf(), Seconds(1)) val model = new KMeansModel( ssc.sparkContext.objectFile(modelFile).collect()) // Streaming val tweets = TwitterUtils.createStream(ssc, /* auth */) val statuses = tweets.map(_.getText) val filteredTweets = statuses.filter { t => model.predict(featurize(t)) == clusterNumber } filteredTweets.print() ssc.start() Fitsononeslide 119
  • 120. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 120 SQL, SQL, SQL, … Databricks says that 100% of their customers use some SQL Schema is very useful Even in complex pipelines that process a lot of un/semi-structured data Separation of logical from physical plan is critical for performance and scalability Why SparkSQL?
  • 121. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Plan Optimization & Execution 121 SQL AST DataFrame Unresolved Logical Plan Logical Plan Optimized Logical Plan RDDs Selected Physical Plan Analysis Logical Optimization Physical Planning CostModel Physical Plans Code Generation Catalog DataFrames and SQL share the same optimization/execution pipeline
  • 122. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 122 1. A distributed collection of rows organized into named columns 2. An abstraction for selecting, filtering, aggregating and plotting structured data (cf. R, Pandas, Ibis) DataFrame
  • 123. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Catalyst Optimizer: Tree Transformations  Developers express tree transformations as PartialFunction[TreeType,TreeType] 1. If the function does apply to an operator, that operator is replaced with the result. 2. When the function does not apply to an operator, that operator is left unchanged. 3. The transformation is applied recursively to all children.
  • 124. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Prior Work: Optimizer Generators  Volcano / Cascades: • Create a custom language for expressing rules that rewrite trees of relational operators. • Build a compiler that generates executable code for these rules.
  • 125. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) An Example Catalyst Transformation 1. Find filters on top of projections. 2. Check that the filter can be evaluated without the result of the project. 3. If so, switch the operators. Project name Project id,name Filter id = 1 People Original Plan Project name Project id,name Filter id = 1 People Filter Push-Down
  • 126. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Filter Push Down Transformation val newPlan = queryPlan transform { case f @ Filter(_, p @ Project(_, grandChild)) if(f.references subsetOf grandChild.output) => p.copy(child = f.copy(child = grandChild) }
  • 127. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Community-Contributed Transformations 127 110 line patch took this user’s query from “never finishing” to 200s.
  • 128. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Project Tungsten: Getting Spark to Run Well on the JVM Overcoming JVM limitations: • Memory Management and Binary Processing: leveraging application semantics to manage memory explicitly and eliminate the overhead of JVM object model and garbage collection • Cache-aware computation: algorithms and data structures to exploit memory hierarchy • Code generation: using code generation to exploit modern compilers and CPUs
  • 129. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 129
  • 130. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Use sun.misc.Unsafe JVM internal API Can manipulate memory without safety checks
  • 131. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 131 Null bits Inline fixed-length values Align on 8-byte word boundaries
  • 132. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Apache Phoenix 9/30/2015SQL-on-Hadoop Tutorial 132
  • 133. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 133 SQL compiler and execution engine for HBase Query engine transforms SQL into native HBase APIs: put, delete, parallel scans (instead of, say, MapReduce) Supports features not provided by HBase: Secondary Indexing, Multi-tenancy, simple Hash Join, etc. The Phoenix Approach
  • 134. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 134 Phoenix Architecture
  • 135. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Open (Research) Challenges 9/30/2015SQL-on-Hadoop Tutorial 135
  • 136. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015)  Cost-based optimizer relies on  Statistics over base relations  Formulas for cost estimation  Rules for plan enumeration  Problems:  Stats not reliable, do not own the data  Prominent use of UDFs  Independence assumption between predicates do not hold  More nested data, harder to estimate selectivities  Bad plans over big data may run “forever” Defer more cost-based decisions to run-time; robust, adaptive query optimization Challenge 1: Query optimization
  • 137. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) 9/30/2015SQL-on-Hadoop Tutorial 137  No single framework owns the data!  Multiple frameworks, with different resource requirements HDFS, S3 Hadoop Yarn Spark/MR/Tez/.. Streaming SQL Graph Machine learning ETL and batch processing  How to share the data?  How to share resources?  How to work together seamlessly? Challenge 2: Multi-framework environment
  • 138. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) HDFS is a problem for transactional workloads Workarounds do not lend itself to high-performance OLAP Object-stores Interesting combinations are emerging Hive LLAP + Phoenix, Splice Machine + Spark Need more tightly integrated solutions Need an updatable, fast, distributed file system 9/30/2015SQL-on-Hadoop Tutorial 138 Challenge 3: Transactions and analytics in one system
  • 139. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References  http://www.slideshare.net/enissoz/hbase-and-hdfs-understanding-filesystem-usage  https://www.mapr.com/blog/in-depth-look-hbase-architecture  Apache Drill. http://drill.apache.org/.  Apache Phoenix. http://phoenix.apache.org/.  Hive on spark. https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark .  Splice machine. http://www.splicemachine.com/.  Teradata query grid. http://www.teradata.com/Teradata- QueryGrid/#tabbable=0&tab1=0&tab2=0.  M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson, D. Rorke, S. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-Milne, and M. Yoder. “Impala: A Modern, Open-Source SQL Engine for Hadoop.” In Proc. CIDR, 2015.  Y. Huai, A. Chauhan, A. Gates, G. Hagleitner, E. N. Hanson, O. O'Malley, J. Pandey, Y. Yuan, R. Lee, and X. Zhang. “Major technical advancements in apache hive.” In Proc. SIGMOD, 2014. 9/30/2015SQL-on-Hadoop Tutorial 139
  • 140. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  A. Ailamaki, D. J. DeWitt, M. D. Hill, and M. Skounakis. “Weaving Relations for Cache Performance.” In Proc. of the 27th International Conference on Very Large Data Bases, 2001.  Y. He, R. Lee, Y. Huai, Z. Shao, N. Jain, X. Zhang, and Z. Xu. “RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems.” In Proc. of ICDE, 2011.  H. Lim, H. Herodotou, and S. Babu. “Stubby: a transformation-based optimizer for MapReduce workflows.” PVLDB, 2012.  T. Neumann. “Efficiently compiling efficient query plans for modern hardware.” PVLDB, 2011.  D. Simmen, E. Shekita, and T. Malkemus. “Fundamental techniques for order optimization.” In Proc. of ACM SIGMOD, 1996.  A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Anthony, H. Liu, and R. Murthy.” Hive - A Petabyte Scale Data Warehouse Using Hadoop.” In ICDE, 2010.  T. Willhalm, N. Popovici, Y. Boshmaf, H. Plattner, A. Zeier, and J. Schaffner. “SIMD-scan: ultra fast in- memory table scan using on-chip vector processing units.” PVLDB, 2, 2009. 9/30/2015SQL-on-Hadoop Tutorial 140
  • 141. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  V. Raman, G. Attaluri, R. Barber, N. Chainani, D. Kalmuk, V. KulandaiSamy, J. Leenstra, S. Light- stone, S. Liu, G. M. Lohman, T. Malkemus, R. Mueller, I. Pandis, B. Schiefer, D. Sharpe, R. Sidle, A. Storm, and L. Zhang. “DB2 with BLU Acceleration: So much more than just a column store.” PVLDB, 6, 2013.  A. Abouzeid, K. Bajda-Pawlikowski, D. J. Abadi, A. Rasin, and A. Silberschatz. “HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.” PVLDB, 2009.  A. Abouzied, D. J. Abadi, and A. Silberschatz. “Invisible loading: Access-driven data transfer from raw files into database systems.” In EDBT, 2013.  M. Amburst, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi, and M. Zaharia. “Spark SQL: Relational data processing in Spark.” In ACM SIGMOD, 2015.  V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, B. Saha, C. Curino, O. O’Malley, S. Radia, B. Reed, and E. Baldeschwieler. Apache Hadoop YARN: Yet another resource negotiator. In SOCC, 2013. 9/30/2015SQL-on-Hadoop Tutorial 141
  • 142. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  K. Bajda-Pawlikowski, D. J. Abadi, A. Silberschatz, and E. Paulson. “Efficient processing of data warehousing queries in a split execution environment.” In ACM SIGMOD, 2011.  P. Boncz. Vortex: Vectorwise goes Hadoop. http://databasearchitects.blogspot.com/2014/05/vectorwise- goes-hadoop.html.  L. Chang, Z. Wang, T. Ma, L. Jian, L. Ma, A. Goldshuv, L. Lonergan, J. Cohen, C. Welton, G. Sherry, and M. Bhandarkar. “HAWQ: A massively parallel processing SQL engine in hadoop.” In ACM SIGMOD, 2014.  G. Graefe. “Encapsulation of parallelism in the Volcano query processing system.” In ACM SIGMOD, 1990.  S. Gray, F. Özcan, H. Pereyra, B. van der Linden, and A. Zubiri. “IBM Big SQL 3.0: SQL-on-Hadoop without compromise.” http://public.dhe.ibm.com/common/ ssi/ecm/en/sww14019usen/SWW14019USEN.PDF, 2014.  F.Özcan, D. Hoa, K. S. Beyer, A. Balmin, C. J. Liu, and Y. Li. “Emerging trends in the enterprise data analytics: Connecting Hadoop and DB2 warehouse.” In ACM SIGMOD, 2011. 9/30/2015SQL-on-Hadoop Tutorial 142
  • 143. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. “Dremel: Interactive analysis of web-scale datasets.” PVLDB, 2010.  S. Padmanabhan, T. Malkemus, R. C. Agarwal, and A. Jhingran. “Block oriented processing of relational database operations in modern computer architectures.” In ICDE, 2001.  B. Saha, H. Shah, S. Seth, G. Vijayaraghavan, A. Murthy, and C. Curino. “Apache Tez: A unifying framework for modeling and building data processing applications.” In ACM SIGMOD, 2015.  P. Seshadri, H. Pirahesh, and T. Y. C. Leung. “Complex query decorrelation.” In ICDE, 1996.  A. Floratou, U. F. Minhas, and F. Özcan. “SQL-on- Hadoop: Full circle back to shared-nothing database architectures.” PVLDB 7(12), 2014.  M. Traverso. Presto: Interacting with petabytes of data at Facebook. https://www.facebook.com/notes/facebook- engineering/presto-interacting-with-petabytes-of-data- at- facebook/10151786197628920.  S. Wanderman-Milne and N. Li. Runtime code generation in Cloudera Impala. IEEE Data Eng. Bull., 2014. 9/30/2015SQL-on-Hadoop Tutorial 143
  • 144. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  R. S. Xin, J. Rosen, M. Zaharia, M. J. Franklin, S. Shenker, and I. Stoica. “Shark: SQL and rich analytics at scale.” In ACM SIGMOD, 2013.  M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. “Spark: Cluster computing with working sets.” In HotCloud, 2010.  C. Zuzarte, H. Pirahesh, W. Ma, Q. Cheng, L. Liu, and K. Wong. “WinMagic : Subquery elimination using window aggregation.” In ACM SIGMOD, 2003.  F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. “Bigtable: A Distributed Storage System for Structured Data.”, In OSDI 2006  B. Chattopadhyay, L. Lin, W. Liu, S. Mittal, P. Aragonda, V. Lychagina, Y. Kwon, and M. Wong. “Tenzing: A SQL Implementation on the MapReduce Framework.” In VLDB 2011.  T. Nykiel, M. Potamias, C. Mishra, G. Kollios, and N. Koudas. “MRShare: Sharing Across Multiple Queries in MapReduce.” In VLDB 2010. 9/30/2015SQL-on-Hadoop Tutorial 144
  • 145. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  G. Wang and C.-Y. Chan. “Multi-Query Optimization in MapReduce Framework.” In VLDB, 2013.  F. Afrati and J. Ullman. “Optimizing Multiway Joins in a Map-Reduce Environment.” In TKDE 2011.  Y. Kwon, M. Balazinska, B. Howe, and J. Rolia. “SkewTune: Mitigating Skew in MapReduce Applications.” In SIGMOD 2012.  M. Eltabakh, F. Özcan, Y. Sismanis, P. J. Haas, H. Pirahesh, and J. Vondrak, “Eagle-eyed elephant: split-oriented indexing in Hadoop”, in EDBT 2014.  M. Eltabakh, Y. Tian, F. Özcan, R. Gemulla, A. Krettek, and J. McPherson, “CoHadoop: Flexible Data Placement and Its Exploitation in Hadoop”, in PVLDB 4(9), 2011.  J. Shi, Y. Qiu, U. F. Minhas, L. Jiao, C. Wang, B. Reinwald, and F. Özcan, “Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics” in PVLDB 8(13), 2015  J. Dittrich, J-A. Quiane-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad, “Hadoop++: Making a Yellos Elephant Run Like a Cheetah (without it even noticing)”, in PVLDB 3(1-2), 2010. 9/30/2015SQL-on-Hadoop Tutorial 145
  • 146. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  D. Jiang, B. C. Ooi, L. Shi, and S. Wu, “The Performance of MapReduce: An In-depth Study”, in PVLDB 3(1-2), 2010.  D. J. DeWitt, R. V. Nehme, S. Shankar, J. Aguilar-Saborit, A. Avanes, M. Flasza, and J. Gramling, “Split Query Processing in Polybase”, in SIGMOD 2013.  M. Stonebraker, D. J. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin, “MapReduce and parallel DBMSs: Friends or Foes?” CACM, 53(1):64–71, 2010.  A. Floratou, J. M. Patel, E. J. Shekita, and S. Tata, “Column-oriented Storage Techniques for MapReduce”, in PVLDB, 4(7):419–429, 2011  S. Harris, A. Sundararajan, E. Branish, and K. Chen, “Blistering Fast SQL Access to Hadoop using IBM BigInsights 3.0 with Big SQL 3.0”  S. Blanas and et al., “A comparison of join algorithms for log processing in mapreduce”, in SIGMOD 2010. 9/30/2015SQL-on-Hadoop Tutorial 146
  • 147. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) References (cont.)  HDFS caching, http://hadoop.apache.org/docs/current/hadoopproject-dist/hadoop- hdfs/CentralizedCacheManagement.html.  S. Babu and H. Herodotou, “Massively Parallel Databases and MapReduce Systems”, in Foundations and Trends in Databases 5(1), 2013.  N. Bruno, Y. Kwon, and M-C Wu, “Advanced Join Strategies for Large-Scale Distributed Computation”, in PVLDB 7(13), 2014  K. Karanasos, A. Balmin, M. Kutsch, F. Özcan, V. Ercegovac, C. Xia, and J. Jackson, “Dynamically optimizing queries over large scale data platforms”, in SIGMOD 2014  J. Dean, and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, in OSDI 2004. 9/30/2015SQL-on-Hadoop Tutorial 147
  • 148. Re-use permitted when acknowledging the original © Daniel Abadi, Shivnath Babu, Fatma Ozcan, and Ippokratis Pandis (2015) Thank you!

Editor's Notes

  1. https://github.com/Parquet/parquet-format
  2. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  3. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  4. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  5. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  6. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  7. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  8. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  9. Companies, researchers, etc use workflows to do data analytics. MapReduce is a popular choice for running Workflows. This is an example of a 7 –job MapReduce workflows. MapReduce jobs with prod-consumer relationship Tall person… basic idea when most people write workflow, they focus on the operators, which result in tall and thing workflows. We want it to be short and broad by packing functions into smaller number of jobs Big Performance Gap between Unoptimized Contribution: Stubby.. Automatic optimization Stubby! Turns the 7 job workflow into a 2job workflow
  10. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  11. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  12. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  13. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  14. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  15. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  16. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  17. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  18. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  19. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  20. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  21. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  22. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  23. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  24. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  25. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  26. Pay the conversion price once for query many times workloads. What data type for DECIMAL data? Dates probably stored as STRING in YYYY-MM-DD format Optionally stored as INTs Removed: For DECIMAL data use FLOAT/DOUBLE
  27. 102
  28. 103
  29. 104
  30. 106
  31. 107
  32. 108
  33. 111
  34. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.
  35. http://www.slideshare.net/alanfgates/strata-stingertalk-oct2013 Slide 22 mentions Stripe-based Input Splits Hive-5632 (Eliminate splits based on SARGs using stripe statistics in ORC): Includes a comment from committer: ORC already stores hierarchical min/max metadata. At the lowest level, ORC stores min/max for every 10,000 rows (called as rowgroups). The size of the rowgroup can be configured using the table property "orc.row.index.stride". At a higher level, HIVE-5562 adds min/max metadata to stripe level. There is also file level min/max values as well at the file footer.Stripe levels stats are stored in file footer, stripes that doesn't satisfy the predicates can be skipped while computing the splits.