SlideShare a Scribd company logo
1 of 44
Hive+Tez: A Performance
deep dive
gopalv@apache.org
© Hortonworks Inc. 2014.
Stinger Project
(announced February 2013)
Batch AND Interactive SQL-IN-Hadoop
Stinger Initiative
A broad, community-based effort to
drive the next generation of HIVE
Hive 0.13, April, 2013
• Hive on Apache Tez
• Query Service
• Buffer Cache
• Cost Based Optimizer (Optiq)
• Vectorized Processing
Hive 0.11, May 2013:
• Base Optimizations
• SQL Analytic Functions
• ORCFile, Modern File Format
Hive 0.12, October 2013:
• VARCHAR, DATE Types
• ORCFile predicate pushdown
• Advanced Optimizations
• Performance Boosts via YARN
Speed
Improve Hive query performance by 100X to
allow for interactive query times (seconds)
Scale
The only SQL interface to Hadoop designed
for queries that scale from TB to PB
SQL
Support broadest range of SQL semantics for
analytic applications running against Hadoop
…all IN Hadoop
Goals:
© Hortonworks Inc. 2014.
SQL: Enhancing SQL Semantics
Hive SQL Datatypes Hive SQL Semantics
INT SELECT, INSERT
TINYINT/SMALLINT/BIGINT GROUP BY, ORDER BY, SORT BY
BOOLEAN JOIN on explicit join key
FLOAT Inner, outer, cross and semi joins
DOUBLE Sub-queries in FROM clause
STRING ROLLUP and CUBE
TIMESTAMP UNION
BINARY Windowing Functions (OVER, RANK, etc)
DECIMAL Custom Java UDFs
ARRAY, MAP, STRUCT, UNION Standard Aggregation (SUM, AVG, etc.)
DATE Advanced UDFs (ngram, Xpath, URL)
VARCHAR Sub-queries in WHERE, HAVING
CHAR Expanded JOIN Syntax
SQL Compliant Security (GRANT, etc.)
INSERT/UPDATE/DELETE (ACID)
Common Table Expressions
Hive 0.12
Available
Hive 0.13*
SQL Compliance
Hive 12 provides a wide
array of SQL data types
and semantics so your
existing tools integrate
more seamlessly with
Hadoop
© Hortonworks Inc. 2014.
SPEED: Increasing Hive Performance
Key Highlights
– Tez: New execution engine
– Vectorized Query Processing
– Startup time improvement
– Statistics to accelerate query execution
– Cost Based Optimizer: Optiq
Interactive Query Times across ALL use cases
• Simple and advanced queries in seconds
• Integrates seamlessly with existing tools
• Currently a >100x improvement in just nine months
Elements of Fast SQL Execution
• Query Planner/Cost Based
Optimizer w/ Statistics
• Query Startup
• Query Execution
• I/O Path
© Hortonworks Inc. 2014.
Statistics and Cost-based optimization
• Statistics:
– Hive has table and column level statistics
– Used to determine parallelism, join selection
• Optiq: Open source, Apache licensed query execution framework in Java
– Used by Apache Drill, Apache Cascading, Lucene DB, …
– Based on Volcano paper
– 20 man years dev, more than 50 optimization rules
• Goals for hive
– Ease of Use – no manual tuning for queries, make choices automatically based on cost
– View Chaining/Ad hoc queries involving multiple views
– Help enable BI Tools front-ending Hive
– Emphasis on latency reduction
• Cost computation will be used for
 Join ordering
 Join algorithm selection
 Tez vertex boundary selection
Page 5
HIVE-5775
© Hortonworks Inc. 2014.
Apache Tez (“Speed”)
• Replaces MapReduce as primitive for Pig, Hive, Cascading etc.
– Smaller latency for interactive queries
– Higher throughput for batch queries
– 22 contributors: Hortonworks (13), Facebook, Twitter, Yahoo, Microsoft
YARN ApplicationMaster to run DAG of Tez Tasks
Task with pluggable Input, Processor and Output
Tez Task - <Input, Processor, Output>
Task
ProcessorInput Output
© Hortonworks Inc. 2014.
Hive – MR Hive – Tez
Hive-on-MR vs. Hive-on-Tez
SELECT g1.x, g1.avg, g2.cnt
FROM (SELECT a.x, AVERAGE(a.y) AS avg FROM a GROUP BY a.x) g1
JOIN (SELECT b.x, COUNT(b.y) AS avg FROM b GROUP BY b.x) g2
ON (g1.x = g2.x)
ORDER BY avg;
GROUP a BY a.x
JOIN (a,b)
GROUP b BY b.x
ORDER BY
M M M
R R
M M
R
M M
R
M
R
HDFS HDFS
HDFS
M M M
R R
R
M M
R
GROUP BY a.x
JOIN (a,b)
ORDER BY
GROUP BY x
Tez avoids
unnecessary writes
to HDFS
HIVE-4660
© Hortonworks Inc. 2014.
Shuffle Join
SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand
FROM inventory inv
JOIN store_sales ss
ON (inv.inv_item_sk = ss.ss_item_sk);
Hive – MR Hive – Tez
© Hortonworks Inc. 2014.
Broadcast Join
• Similar to map-join w/o the need to build a hash table on the
client
• Will work with any level of sub-query nesting
• Uses stats to determine if applicable
• How it works:
– Broadcast result set is computed in parallel on the cluster
– Join processor are spun up in parallel
– Broadcast set is streamed to join processor
– Join processors build hash table
– Other relation is joined with hashtable
• Tez handles:
– Best parallelism
– Best data transfer of the hashed relation
– Best scheduling to avoid latencies
© Hortonworks Inc. 2014.
Broadcast Join
Hive – MR Hive – Tez
M M M
M
HDFS
M MM
M M
HDFS
SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand
FROM store_sales ss
JOIN inventory inv
ON (inv.inv_item_sk = ss.ss_item_sk);
HDFS
Inventory
scan (Runs
as single
local map
task)
Store Sales
scan and Join
(Inventory hash
table read as
side file)
Inventory scan
(Runs on cluster
potentially more
than 1 mapper)
Store Sales
scan and Join
Broadcast
edge
© Hortonworks Inc. 2014.
Broadcast Join
SELECT ss.ss_item_sk, ss.ss_quantity, avg_price, inv.inv_quantity_on_hand
FROM (select avg(ss_sold_price) as avg_price, ss_item_sk, ss_quantity_sk from store_sales
group by ss_item_sk) ss
JOIN inventory inv
ON (inv.inv_item_sk = ss.ss_item_sk);
Hive – MR Hive – Tez
M
M
M
M M
HDFS
Store Sales
scan. Group by
and aggregation
reduce size of
this input.
Inventory scan
and Join
Broadcast
edge
M M M
HDFS
Store Sales
scan. Group by
and aggregation.
Inventory and Store
Sales (aggr.) output
scan and shuffle
join.
R R
R R
RR
M
MMM
HDFS
© Hortonworks Inc. 2014.
1-1 Edge
• Typical star schema join involve join between large number of
tables
• Dimension aren’t always tiny (Customer dimension)
• Might not be able to handle all dimensions in single vertex as
broadcast joins
• Tez allows streaming records from one processor to the next via
a 1-1 Edge
– Transfer details (streaming, files, etc) are handled transparently
– Scheduling/cluster capacity is worked out by Tez
• Allows hive to build a pipeline of in memory joins which we can
stream records through
© Hortonworks Inc. 2014.
Dynamically partitioned Hash join
• Kicks in when large table is bucketed
– Bucketed table
– Dynamic as part of query processing
• Uses custom edge to match the partitioning on the smaller table
• Allows hash-join in cases where broadcast would be too large
• Tez gives us the option of building custom edges and vertex
managers
– Fine grained control over how the data is replicated and partitioned
– Scheduling and actual data transfer is handled by Tez
© Hortonworks Inc. 2014.
Dynamically Partitioned Hash Join
SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand
FROM store_sales ss
JOIN inventory inv
ON (inv.inv_item_sk = ss.ss_item_sk);
Hive – MR Hive – Tez
M MM
M M
HDFS
Inventory scan
(Runs on
cluster
potentially more
than 1 mapper)
Store Sales
scan and Join
(Custom vertex
reads both
inputs – no side
file reads)
Custom
edge (routes
outputs of
previous stage to
the correct
Mappers of the
next stage)
M MM
M
HDFS
Inventory scan
(Runs as single
local map task)
Store Sales
scan and Join
(Inventory hash
table read as
side file)
HDFS
© Hortonworks Inc. 2014.
Dynamically Partitioned Hash Join
Plans look very similar to map join but the way things work change between
MR and Tez.
Hive – MR (Bucket map-join) Hive – Tez
• Not dynamically partitioned.
• Both tables need to be bucketed by the join
key.
• Local task that generates the hash table
writes n files corresponding to n buckets.
• Number of mappers for the join must be
same as the number of buckets.
• Each of these mappers reads the
corresponding bucket file of the local task to
perform the join.
• Only one of the sides needs to be bucketed
and the other side is dynamically bucketed.
• Also works if neither side is explicitly
bucketed, but another operation forced
bucketing in the pipeline (traits)
• No writing to HDFS.
• There can be more mappers than number of
buckets but splits do not span multiple
buckets.
• The dynamically bucketed mappers have as
many outputs as number of buckets and a
custom tez routing ensures these outputs
reach the right mappers.
© Hortonworks Inc. 2014.
Union all
• Common operation in decision support queries
• Caused additional no-op stages in MR plans
– Last stage spins up multi-input mapper to write result
– Intermediate unions have to be materialized before additional processing
• Tez has union that handles these cases transparently w/o any
intermediate steps
© Hortonworks Inc. 2014.
Union all
SELECT count(*) FROM (
SELECT distinct ss_customer_sk from store_sales where ss_store_sk = 1
UNION ALL
SELECT distinct ss_customer_sk from store_sales where ss_store_sk = 2) as customers
Hive – MR Hive – Tez
M M M
R
M M M
HDFS
R
M
R
HDFS
M M M
R
M M M
HDFS
R
R
Two MR jobs to
do the distinct
Both sub-queries
are materialized
onto HDFS
Single map
reads both sides
and aggregates
In Tez the sub-query
output is pre-aggregated
and send directly to a
common final node
© Hortonworks Inc. 2014.
Multi-insert queries
• Allows the same input to be split and written to different tables
or partitions
– Avoids duplicate scans/processing
– Useful for ETL
– Similar to “Splits” in PIG
• In MR a “split” in the operator pipeline has to be written to HDFS
and processed by multiple additional MR jobs
• Tez allows to send the mulitple outputs directly to downstream
processors
© Hortonworks Inc. 2014.
Multi-insert queries
FROM (SELECT * FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk
and d_year = 2000)
INSERT INTO TABLE t1 SELECT distinct ss_item_sk
INSERT INTO TABLE t2 SELECT distinct ss_customer_sk;
Hive – MR Hive – Tez
M MM
M
HDFS
Map join
date_dim/store
sales
Two MR jobs to
do the distinct
M MM
M M
HDFS
RR
HDFS
M M M
R
M M M
R
HDFS
Broadcast Join
(scan date_dim,
join store sales)
Distinct for
customer + items
Materialize join on
HDFS
© Hortonworks Inc. 2014.
Execution
“A good plan violently executed now is better
than a perfect plan executed next week.
George S. Patton
© Hortonworks Inc. 2014.
Faster Query Setup
• AM per-session instead of per-query
– Reused across JDBC connections
• No more local tasks
– Except fetch aggregation
• Metastore fetches are much faster
– Metastore direct sql fast-path
– Partition filters pushed to metastore
• Use distributed cache efficiently for hive-exec.jar
– /home/$user/.hiveJars
• UDF Jars as well
– .jar.<sha1> identifier to avoid conflicts
– Multiple version compatibility easily
– YARN localizes the jars once per node (not per query)
• Kryo instead of XML to serialize operators
– Works better on jdk7
Page 24
© Hortonworks Inc. 2014.
Faster Operator Pipeline
• Previously on hive
© Hortonworks Inc. 2014.
Operator Vectorization
• Avoid Writable objects & use primitive int/long
– Allows efficient JIT code for primitive types
• Generate per-type loops & avoid runtime type-checks
• The classes generated look like
– LongColEqualDoubleColumn
– LongColEqualLongColumn
– LongColEqualLongScalar
• Avoid duplicate operations on repeated values
– isRepeating & hasNulls
© Hortonworks Inc. 2014.
Optimized Row Columnar File
• ORC Vectorized Reader
• Logical Compression helps reader
– isRepeating
• Split per-stripe
• Row-group level indexes
• Stripe level indexes
• PPD avoids a lot of IO
– Column conditions are ANDed
© Hortonworks Inc. 2014.
Faster Statistics
• ORC stripe footers aggregate stats per-column
– Min/Max/Sum/Count
• set hive.stats.autogather=true;
• ANALYZE TABLE <table> compute statistics partialscan;
– Reads only ORC footers
• Predicate computation without Tez/MR tasks
© Hortonworks Inc. 2014.
Faster Execution: Tez
• Multiple edge types
– Broadcast
– Shuffle
– One-to-One
• Multiple output types
– Sorted
– Unsorted
– Unsorted Partitioned
• Per-vertex configurations
– Instead of one configuration between M&R tasks
© Hortonworks Inc. 2014.
Tez I/O speed-ups
• Tez shuffle can use keep-alive over HTTP
• Shuffle scheduler can optimize connection count
– Can fetch all map outputs from one node via 1 connection
• Can skip fetching 0 sized partitions from a mapper
– Speeds up group-by queries with high locality
– Reducers finish shuffle faster
• Shuffle threads are re-used in container re-use
– Secure shuffle has crypto thread-local inits
© Hortonworks Inc. 2014.
Skewed Reducers: auto-parallelism
• Often queries are slow because of one slow reducer
• Skewed data is too common in real life queries
• This avoids running too many reducers with with very little data
• Future
– This can be extended to group by input size
– This mechanism can actually speculate on stalling reducers better (split into 3)
© Hortonworks Inc. 2014.
A Query in motion
Page 32
• 4-way Map join + map reduce reduce query
• Timeline in left to right, each lane represents one container
© Hortonworks Inc. 2014.
Defer/Skip tasks
Page 33
• No more uploading hive-exec.jar/UDFs for every query
• No more spinning up an AM for each stage
• No more computation on hive client (local task)
© Hortonworks Inc. 2014.
Concurrency of small tasks
Page 34
• Hive used to run several lightweight tasks in a local VM
• LocalTask was a bottleneck
– No locality
– No parallelism
– Small VM
• Tez Broadcast edges solve that problem
© Hortonworks Inc. 2014.
Concurrent Split Generation
Page 35
• Tez input intializers are run parallel
• No more spinning up an AM for each stage
• No more computation on hive client (local task)
© Hortonworks Inc. 2014.
Split Elimination
Page 36
• ORC comes with Predicate Push Down in the reader
• Queries with SARGable where clauses
– http://en.wikipedia.org/wiki/Sargable
• Run the SARGs in the AM, using ORC footer data
– Eliminate splits before task spinups, avoid container costs
• Offers a soft cache for the ORC footers
• Zero splits offers an early exit for data validity checks (i.e price < 0)
© Hortonworks Inc. 2014.
Pipelining Split->Task
Page 37
• The task only depends on its own input
• It starts talking to YARN immediately once its inputs are ready
• Faster generation of dimension tables
• Fact tables can optimize on this further
– Will break existing FileSplit mechanism
© Hortonworks Inc. 2014.
Filling up the pipeline
Page 38
• Tez allows grouping splits dynamically
• Obsoletes CombineFileInputFormat
• Grouped according to locality
–1.7 x available containers (or any factor actually)
• Allow query to use up 100% of queue capacity
–Without tuning mapred split size for each data-set
© Hortonworks Inc. 2014.
ORC Split extras
• RCFile had horrible split performance
– rcfile::sync() was slow to find a sync point
• ORC Reader allows exact splits for stripes
• ORC Writer can pad a stripe to an HDFS block
– 5%-7% overhead measured on table
– 100% locality of a stripe in a block
© Hortonworks Inc. 2014.
Container reuse
• Tez specific feature
• Run an entire DAG using the same containers
• Different vertices use same container
• Saves time talking to YARN for new containers
© Hortonworks Inc. 2014.
Container reuse (II)
• Tez provides an object registry within a vertex
• This can be used to cache map-join hash-tables
• JVM JIT kicks in and optimizes better on re-use
© Hortonworks Inc. 2014.
Container re-use (Session)
• Keep a container group alive between queries
• Fast query spin-up and skip YARN queue
• Even better JIT performance on >1 queries
© Hortonworks Inc. 2014.
HiveServer2 and Sessions
• HiveServer2 can keep sessions alive
–Between different JDBC queries
• New security model helps
–All secure queries run as “hive” user
• Ideal for short exploratory queries
• Uses same JARs (no download for task)
• Even better JIT performance on >1 queries
© Hortonworks Inc. 2014.
Supersize it!
• 78 vertex + 8374 tasks on 50 containers
Page 44
© Hortonworks Inc. 2014.
Query overload #2
• 5000 hive query test-set
• Only 3.9k triggered compute tasks
• Rest was optimized away into fetch tasks or metadata tasks
• Gets progressively faster as the JVM JIT improves the native code
Page 45
© Hortonworks Inc. 2014.
Big picture
1501.895
1176.479
631.027
4.872
0
200
400
600
800
1000
1200
1400
1600
Text Columnar Partitioned Stinger
Latency
© Hortonworks Inc. 2014.
Roadmap
• Expand uses for CBO
– Join Algorithm selection
– Tez checkpoint selection (recovery)
• (In-memory) Temp Tables
– Explicit/Implicit
– Speed up sharing of intermediate results
• Materialized views
– Pre-compute common results/aggregations
– Transparently route via CBO
• Join/Grouping w/o sort
– Tez decouples algorithm from data transfer
• Sort-merge bucket in Tez
– Leverage vertex manager
– Co-locate partitions on HDFS
• Inline sampling/range partitioning with Tez
– Sample/create histogram dynamically for skew joins and total order sort
Page 47

More Related Content

What's hot

Hive and Apache Tez: Benchmarked at Yahoo! Scale
Hive and Apache Tez: Benchmarked at Yahoo! ScaleHive and Apache Tez: Benchmarked at Yahoo! Scale
Hive and Apache Tez: Benchmarked at Yahoo! Scale
DataWorks Summit
 
Scaling HDFS to Manage Billions of Files with Key-Value Stores
Scaling HDFS to Manage Billions of Files with Key-Value StoresScaling HDFS to Manage Billions of Files with Key-Value Stores
Scaling HDFS to Manage Billions of Files with Key-Value Stores
DataWorks Summit
 
De-Bugging Hive with Hadoop-in-the-Cloud
De-Bugging Hive with Hadoop-in-the-CloudDe-Bugging Hive with Hadoop-in-the-Cloud
De-Bugging Hive with Hadoop-in-the-Cloud
DataWorks Summit
 
Hive analytic workloads hadoop summit san jose 2014
Hive analytic workloads hadoop summit san jose 2014Hive analytic workloads hadoop summit san jose 2014
Hive analytic workloads hadoop summit san jose 2014
alanfgates
 
Pig on Tez - Low Latency ETL with Big Data
Pig on Tez - Low Latency ETL with Big DataPig on Tez - Low Latency ETL with Big Data
Pig on Tez - Low Latency ETL with Big Data
DataWorks Summit
 
Hive at Yahoo: Letters from the trenches
Hive at Yahoo: Letters from the trenchesHive at Yahoo: Letters from the trenches
Hive at Yahoo: Letters from the trenches
DataWorks Summit
 

What's hot (19)

Indexed Hive
Indexed HiveIndexed Hive
Indexed Hive
 
Hive and Apache Tez: Benchmarked at Yahoo! Scale
Hive and Apache Tez: Benchmarked at Yahoo! ScaleHive and Apache Tez: Benchmarked at Yahoo! Scale
Hive and Apache Tez: Benchmarked at Yahoo! Scale
 
Scaling HDFS to Manage Billions of Files with Key-Value Stores
Scaling HDFS to Manage Billions of Files with Key-Value StoresScaling HDFS to Manage Billions of Files with Key-Value Stores
Scaling HDFS to Manage Billions of Files with Key-Value Stores
 
Quick Introduction to Apache Tez
Quick Introduction to Apache TezQuick Introduction to Apache Tez
Quick Introduction to Apache Tez
 
De-Bugging Hive with Hadoop-in-the-Cloud
De-Bugging Hive with Hadoop-in-the-CloudDe-Bugging Hive with Hadoop-in-the-Cloud
De-Bugging Hive with Hadoop-in-the-Cloud
 
Optimizing Hive Queries
Optimizing Hive QueriesOptimizing Hive Queries
Optimizing Hive Queries
 
Apache Tez - A unifying Framework for Hadoop Data Processing
Apache Tez - A unifying Framework for Hadoop Data ProcessingApache Tez - A unifying Framework for Hadoop Data Processing
Apache Tez - A unifying Framework for Hadoop Data Processing
 
Achieving 100k Queries per Hour on Hive on Tez
Achieving 100k Queries per Hour on Hive on TezAchieving 100k Queries per Hour on Hive on Tez
Achieving 100k Queries per Hour on Hive on Tez
 
Apache Tez – Present and Future
Apache Tez – Present and FutureApache Tez – Present and Future
Apache Tez – Present and Future
 
February 2014 HUG : Pig On Tez
February 2014 HUG : Pig On TezFebruary 2014 HUG : Pig On Tez
February 2014 HUG : Pig On Tez
 
October 2014 HUG : Hive On Spark
October 2014 HUG : Hive On SparkOctober 2014 HUG : Hive On Spark
October 2014 HUG : Hive On Spark
 
Hive analytic workloads hadoop summit san jose 2014
Hive analytic workloads hadoop summit san jose 2014Hive analytic workloads hadoop summit san jose 2014
Hive analytic workloads hadoop summit san jose 2014
 
Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing Apache Tez: Accelerating Hadoop Query Processing
Apache Tez: Accelerating Hadoop Query Processing
 
Evolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage SubsystemEvolving HDFS to a Generalized Distributed Storage Subsystem
Evolving HDFS to a Generalized Distributed Storage Subsystem
 
Tuning up with Apache Tez
Tuning up with Apache TezTuning up with Apache Tez
Tuning up with Apache Tez
 
Pig on Tez - Low Latency ETL with Big Data
Pig on Tez - Low Latency ETL with Big DataPig on Tez - Low Latency ETL with Big Data
Pig on Tez - Low Latency ETL with Big Data
 
Hive at Yahoo: Letters from the trenches
Hive at Yahoo: Letters from the trenchesHive at Yahoo: Letters from the trenches
Hive at Yahoo: Letters from the trenches
 
Hive on spark is blazing fast or is it final
Hive on spark is blazing fast or is it finalHive on spark is blazing fast or is it final
Hive on spark is blazing fast or is it final
 
Apache Drill
Apache DrillApache Drill
Apache Drill
 

Viewers also liked

Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
Email &amp; Social Media Training
Email &amp; Social Media TrainingEmail &amp; Social Media Training
Email &amp; Social Media Training
William Mann
 
CMG Recommendation Letter PSB JPG
CMG Recommendation Letter PSB JPGCMG Recommendation Letter PSB JPG
CMG Recommendation Letter PSB JPG
Pia Sanchez
 
Child domestic labor handbook
Child domestic labor handbookChild domestic labor handbook
MS Foods Plan A Quarterly Newsletter February 2016
MS Foods Plan A Quarterly Newsletter February 2016MS Foods Plan A Quarterly Newsletter February 2016
MS Foods Plan A Quarterly Newsletter February 2016
Ray McCreadie
 
Using Oracle Applications on your iPad
Using Oracle Applications on your iPadUsing Oracle Applications on your iPad
Using Oracle Applications on your iPad
Oracle Day
 

Viewers also liked (17)

Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
 
Tao tai khoan google play
Tao tai khoan google playTao tai khoan google play
Tao tai khoan google play
 
Pengumuman sipencatar 2015
Pengumuman sipencatar 2015Pengumuman sipencatar 2015
Pengumuman sipencatar 2015
 
Email &amp; Social Media Training
Email &amp; Social Media TrainingEmail &amp; Social Media Training
Email &amp; Social Media Training
 
Chord siti badriah senandung cinta
Chord siti badriah senandung cintaChord siti badriah senandung cinta
Chord siti badriah senandung cinta
 
BÍ QUYẾT QUẢN LÝ HIỆU QUẢ CÂU LẠC BỘ CHA MẸ VÀ VỊ THÀNH NIÊN, THANH NIÊN
BÍ QUYẾT QUẢN LÝ HIỆU QUẢ CÂU LẠC BỘ CHA MẸ VÀ VỊ THÀNH NIÊN, THANH NIÊNBÍ QUYẾT QUẢN LÝ HIỆU QUẢ CÂU LẠC BỘ CHA MẸ VÀ VỊ THÀNH NIÊN, THANH NIÊN
BÍ QUYẾT QUẢN LÝ HIỆU QUẢ CÂU LẠC BỘ CHA MẸ VÀ VỊ THÀNH NIÊN, THANH NIÊN
 
Primary EFL Reading Competition
Primary EFL Reading CompetitionPrimary EFL Reading Competition
Primary EFL Reading Competition
 
Kengur
KengurKengur
Kengur
 
Show me your hands
Show me your handsShow me your hands
Show me your hands
 
Sailing after stroke
Sailing after strokeSailing after stroke
Sailing after stroke
 
Jdj Foss Java Tools
Jdj Foss Java ToolsJdj Foss Java Tools
Jdj Foss Java Tools
 
CMG Recommendation Letter PSB JPG
CMG Recommendation Letter PSB JPGCMG Recommendation Letter PSB JPG
CMG Recommendation Letter PSB JPG
 
Cele Album
Cele AlbumCele Album
Cele Album
 
Child domestic labor handbook
Child domestic labor handbookChild domestic labor handbook
Child domestic labor handbook
 
MS Foods Plan A Quarterly Newsletter February 2016
MS Foods Plan A Quarterly Newsletter February 2016MS Foods Plan A Quarterly Newsletter February 2016
MS Foods Plan A Quarterly Newsletter February 2016
 
Pak 1974-na-committe-ahmadiyya.vOL 5
Pak 1974-na-committe-ahmadiyya.vOL 5Pak 1974-na-committe-ahmadiyya.vOL 5
Pak 1974-na-committe-ahmadiyya.vOL 5
 
Using Oracle Applications on your iPad
Using Oracle Applications on your iPadUsing Oracle Applications on your iPad
Using Oracle Applications on your iPad
 

Similar to Performance Hive+Tez 2

Stinger hadoop summit june 2013
Stinger hadoop summit june 2013Stinger hadoop summit june 2013
Stinger hadoop summit june 2013
alanfgates
 
Mapreduce over snapshots
Mapreduce over snapshotsMapreduce over snapshots
Mapreduce over snapshots
enissoz
 
Big Data and Cloud Computing
Big Data and Cloud ComputingBig Data and Cloud Computing
Big Data and Cloud Computing
Farzad Nozarian
 
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times FasterApril 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
Yahoo Developer Network
 

Similar to Performance Hive+Tez 2 (20)

La big datacamp2014_vikram_dixit
La big datacamp2014_vikram_dixitLa big datacamp2014_vikram_dixit
La big datacamp2014_vikram_dixit
 
An In-Depth Look at Putting the Sting in Hive
An In-Depth Look at Putting the Sting in HiveAn In-Depth Look at Putting the Sting in Hive
An In-Depth Look at Putting the Sting in Hive
 
Stinger hadoop summit june 2013
Stinger hadoop summit june 2013Stinger hadoop summit june 2013
Stinger hadoop summit june 2013
 
HBase and Hadoop at Urban Airship
HBase and Hadoop at Urban AirshipHBase and Hadoop at Urban Airship
HBase and Hadoop at Urban Airship
 
Tez Data Processing over Yarn
Tez Data Processing over YarnTez Data Processing over Yarn
Tez Data Processing over Yarn
 
Gunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stingerGunther hagleitner:apache hive & stinger
Gunther hagleitner:apache hive & stinger
 
Big data processing engines, Atlanta Meetup 4/30
Big data processing engines, Atlanta Meetup 4/30Big data processing engines, Atlanta Meetup 4/30
Big data processing engines, Atlanta Meetup 4/30
 
Impala for PhillyDB Meetup
Impala for PhillyDB MeetupImpala for PhillyDB Meetup
Impala for PhillyDB Meetup
 
Unit II Hadoop Ecosystem_Updated.pptx
Unit II Hadoop Ecosystem_Updated.pptxUnit II Hadoop Ecosystem_Updated.pptx
Unit II Hadoop Ecosystem_Updated.pptx
 
Mapreduce over snapshots
Mapreduce over snapshotsMapreduce over snapshots
Mapreduce over snapshots
 
Cheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduceCheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduce
 
Incredible Impala
Incredible Impala Incredible Impala
Incredible Impala
 
hive_slides_Webinar_Session_1.pptx
hive_slides_Webinar_Session_1.pptxhive_slides_Webinar_Session_1.pptx
hive_slides_Webinar_Session_1.pptx
 
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive DemosHadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos
Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos
 
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
 
Hive
HiveHive
Hive
 
Big Data and Cloud Computing
Big Data and Cloud ComputingBig Data and Cloud Computing
Big Data and Cloud Computing
 
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times FasterApril 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
April 2013 HUG: The Stinger Initiative - Making Apache Hive 100 Times Faster
 
Mar 2012 HUG: Hive with HBase
Mar 2012 HUG: Hive with HBaseMar 2012 HUG: Hive with HBase
Mar 2012 HUG: Hive with HBase
 
Hive big-data meetup
Hive big-data meetupHive big-data meetup
Hive big-data meetup
 

Recently uploaded

CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 

Recently uploaded (20)

WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 

Performance Hive+Tez 2

  • 1. Hive+Tez: A Performance deep dive gopalv@apache.org
  • 2. © Hortonworks Inc. 2014. Stinger Project (announced February 2013) Batch AND Interactive SQL-IN-Hadoop Stinger Initiative A broad, community-based effort to drive the next generation of HIVE Hive 0.13, April, 2013 • Hive on Apache Tez • Query Service • Buffer Cache • Cost Based Optimizer (Optiq) • Vectorized Processing Hive 0.11, May 2013: • Base Optimizations • SQL Analytic Functions • ORCFile, Modern File Format Hive 0.12, October 2013: • VARCHAR, DATE Types • ORCFile predicate pushdown • Advanced Optimizations • Performance Boosts via YARN Speed Improve Hive query performance by 100X to allow for interactive query times (seconds) Scale The only SQL interface to Hadoop designed for queries that scale from TB to PB SQL Support broadest range of SQL semantics for analytic applications running against Hadoop …all IN Hadoop Goals:
  • 3. © Hortonworks Inc. 2014. SQL: Enhancing SQL Semantics Hive SQL Datatypes Hive SQL Semantics INT SELECT, INSERT TINYINT/SMALLINT/BIGINT GROUP BY, ORDER BY, SORT BY BOOLEAN JOIN on explicit join key FLOAT Inner, outer, cross and semi joins DOUBLE Sub-queries in FROM clause STRING ROLLUP and CUBE TIMESTAMP UNION BINARY Windowing Functions (OVER, RANK, etc) DECIMAL Custom Java UDFs ARRAY, MAP, STRUCT, UNION Standard Aggregation (SUM, AVG, etc.) DATE Advanced UDFs (ngram, Xpath, URL) VARCHAR Sub-queries in WHERE, HAVING CHAR Expanded JOIN Syntax SQL Compliant Security (GRANT, etc.) INSERT/UPDATE/DELETE (ACID) Common Table Expressions Hive 0.12 Available Hive 0.13* SQL Compliance Hive 12 provides a wide array of SQL data types and semantics so your existing tools integrate more seamlessly with Hadoop
  • 4. © Hortonworks Inc. 2014. SPEED: Increasing Hive Performance Key Highlights – Tez: New execution engine – Vectorized Query Processing – Startup time improvement – Statistics to accelerate query execution – Cost Based Optimizer: Optiq Interactive Query Times across ALL use cases • Simple and advanced queries in seconds • Integrates seamlessly with existing tools • Currently a >100x improvement in just nine months Elements of Fast SQL Execution • Query Planner/Cost Based Optimizer w/ Statistics • Query Startup • Query Execution • I/O Path
  • 5. © Hortonworks Inc. 2014. Statistics and Cost-based optimization • Statistics: – Hive has table and column level statistics – Used to determine parallelism, join selection • Optiq: Open source, Apache licensed query execution framework in Java – Used by Apache Drill, Apache Cascading, Lucene DB, … – Based on Volcano paper – 20 man years dev, more than 50 optimization rules • Goals for hive – Ease of Use – no manual tuning for queries, make choices automatically based on cost – View Chaining/Ad hoc queries involving multiple views – Help enable BI Tools front-ending Hive – Emphasis on latency reduction • Cost computation will be used for  Join ordering  Join algorithm selection  Tez vertex boundary selection Page 5 HIVE-5775
  • 6. © Hortonworks Inc. 2014. Apache Tez (“Speed”) • Replaces MapReduce as primitive for Pig, Hive, Cascading etc. – Smaller latency for interactive queries – Higher throughput for batch queries – 22 contributors: Hortonworks (13), Facebook, Twitter, Yahoo, Microsoft YARN ApplicationMaster to run DAG of Tez Tasks Task with pluggable Input, Processor and Output Tez Task - <Input, Processor, Output> Task ProcessorInput Output
  • 7. © Hortonworks Inc. 2014. Hive – MR Hive – Tez Hive-on-MR vs. Hive-on-Tez SELECT g1.x, g1.avg, g2.cnt FROM (SELECT a.x, AVERAGE(a.y) AS avg FROM a GROUP BY a.x) g1 JOIN (SELECT b.x, COUNT(b.y) AS avg FROM b GROUP BY b.x) g2 ON (g1.x = g2.x) ORDER BY avg; GROUP a BY a.x JOIN (a,b) GROUP b BY b.x ORDER BY M M M R R M M R M M R M R HDFS HDFS HDFS M M M R R R M M R GROUP BY a.x JOIN (a,b) ORDER BY GROUP BY x Tez avoids unnecessary writes to HDFS HIVE-4660
  • 8. © Hortonworks Inc. 2014. Shuffle Join SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand FROM inventory inv JOIN store_sales ss ON (inv.inv_item_sk = ss.ss_item_sk); Hive – MR Hive – Tez
  • 9. © Hortonworks Inc. 2014. Broadcast Join • Similar to map-join w/o the need to build a hash table on the client • Will work with any level of sub-query nesting • Uses stats to determine if applicable • How it works: – Broadcast result set is computed in parallel on the cluster – Join processor are spun up in parallel – Broadcast set is streamed to join processor – Join processors build hash table – Other relation is joined with hashtable • Tez handles: – Best parallelism – Best data transfer of the hashed relation – Best scheduling to avoid latencies
  • 10. © Hortonworks Inc. 2014. Broadcast Join Hive – MR Hive – Tez M M M M HDFS M MM M M HDFS SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand FROM store_sales ss JOIN inventory inv ON (inv.inv_item_sk = ss.ss_item_sk); HDFS Inventory scan (Runs as single local map task) Store Sales scan and Join (Inventory hash table read as side file) Inventory scan (Runs on cluster potentially more than 1 mapper) Store Sales scan and Join Broadcast edge
  • 11. © Hortonworks Inc. 2014. Broadcast Join SELECT ss.ss_item_sk, ss.ss_quantity, avg_price, inv.inv_quantity_on_hand FROM (select avg(ss_sold_price) as avg_price, ss_item_sk, ss_quantity_sk from store_sales group by ss_item_sk) ss JOIN inventory inv ON (inv.inv_item_sk = ss.ss_item_sk); Hive – MR Hive – Tez M M M M M HDFS Store Sales scan. Group by and aggregation reduce size of this input. Inventory scan and Join Broadcast edge M M M HDFS Store Sales scan. Group by and aggregation. Inventory and Store Sales (aggr.) output scan and shuffle join. R R R R RR M MMM HDFS
  • 12. © Hortonworks Inc. 2014. 1-1 Edge • Typical star schema join involve join between large number of tables • Dimension aren’t always tiny (Customer dimension) • Might not be able to handle all dimensions in single vertex as broadcast joins • Tez allows streaming records from one processor to the next via a 1-1 Edge – Transfer details (streaming, files, etc) are handled transparently – Scheduling/cluster capacity is worked out by Tez • Allows hive to build a pipeline of in memory joins which we can stream records through
  • 13. © Hortonworks Inc. 2014. Dynamically partitioned Hash join • Kicks in when large table is bucketed – Bucketed table – Dynamic as part of query processing • Uses custom edge to match the partitioning on the smaller table • Allows hash-join in cases where broadcast would be too large • Tez gives us the option of building custom edges and vertex managers – Fine grained control over how the data is replicated and partitioned – Scheduling and actual data transfer is handled by Tez
  • 14. © Hortonworks Inc. 2014. Dynamically Partitioned Hash Join SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand FROM store_sales ss JOIN inventory inv ON (inv.inv_item_sk = ss.ss_item_sk); Hive – MR Hive – Tez M MM M M HDFS Inventory scan (Runs on cluster potentially more than 1 mapper) Store Sales scan and Join (Custom vertex reads both inputs – no side file reads) Custom edge (routes outputs of previous stage to the correct Mappers of the next stage) M MM M HDFS Inventory scan (Runs as single local map task) Store Sales scan and Join (Inventory hash table read as side file) HDFS
  • 15. © Hortonworks Inc. 2014. Dynamically Partitioned Hash Join Plans look very similar to map join but the way things work change between MR and Tez. Hive – MR (Bucket map-join) Hive – Tez • Not dynamically partitioned. • Both tables need to be bucketed by the join key. • Local task that generates the hash table writes n files corresponding to n buckets. • Number of mappers for the join must be same as the number of buckets. • Each of these mappers reads the corresponding bucket file of the local task to perform the join. • Only one of the sides needs to be bucketed and the other side is dynamically bucketed. • Also works if neither side is explicitly bucketed, but another operation forced bucketing in the pipeline (traits) • No writing to HDFS. • There can be more mappers than number of buckets but splits do not span multiple buckets. • The dynamically bucketed mappers have as many outputs as number of buckets and a custom tez routing ensures these outputs reach the right mappers.
  • 16. © Hortonworks Inc. 2014. Union all • Common operation in decision support queries • Caused additional no-op stages in MR plans – Last stage spins up multi-input mapper to write result – Intermediate unions have to be materialized before additional processing • Tez has union that handles these cases transparently w/o any intermediate steps
  • 17. © Hortonworks Inc. 2014. Union all SELECT count(*) FROM ( SELECT distinct ss_customer_sk from store_sales where ss_store_sk = 1 UNION ALL SELECT distinct ss_customer_sk from store_sales where ss_store_sk = 2) as customers Hive – MR Hive – Tez M M M R M M M HDFS R M R HDFS M M M R M M M HDFS R R Two MR jobs to do the distinct Both sub-queries are materialized onto HDFS Single map reads both sides and aggregates In Tez the sub-query output is pre-aggregated and send directly to a common final node
  • 18. © Hortonworks Inc. 2014. Multi-insert queries • Allows the same input to be split and written to different tables or partitions – Avoids duplicate scans/processing – Useful for ETL – Similar to “Splits” in PIG • In MR a “split” in the operator pipeline has to be written to HDFS and processed by multiple additional MR jobs • Tez allows to send the mulitple outputs directly to downstream processors
  • 19. © Hortonworks Inc. 2014. Multi-insert queries FROM (SELECT * FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk and d_year = 2000) INSERT INTO TABLE t1 SELECT distinct ss_item_sk INSERT INTO TABLE t2 SELECT distinct ss_customer_sk; Hive – MR Hive – Tez M MM M HDFS Map join date_dim/store sales Two MR jobs to do the distinct M MM M M HDFS RR HDFS M M M R M M M R HDFS Broadcast Join (scan date_dim, join store sales) Distinct for customer + items Materialize join on HDFS
  • 20. © Hortonworks Inc. 2014. Execution “A good plan violently executed now is better than a perfect plan executed next week. George S. Patton
  • 21. © Hortonworks Inc. 2014. Faster Query Setup • AM per-session instead of per-query – Reused across JDBC connections • No more local tasks – Except fetch aggregation • Metastore fetches are much faster – Metastore direct sql fast-path – Partition filters pushed to metastore • Use distributed cache efficiently for hive-exec.jar – /home/$user/.hiveJars • UDF Jars as well – .jar.<sha1> identifier to avoid conflicts – Multiple version compatibility easily – YARN localizes the jars once per node (not per query) • Kryo instead of XML to serialize operators – Works better on jdk7 Page 24
  • 22. © Hortonworks Inc. 2014. Faster Operator Pipeline • Previously on hive
  • 23. © Hortonworks Inc. 2014. Operator Vectorization • Avoid Writable objects & use primitive int/long – Allows efficient JIT code for primitive types • Generate per-type loops & avoid runtime type-checks • The classes generated look like – LongColEqualDoubleColumn – LongColEqualLongColumn – LongColEqualLongScalar • Avoid duplicate operations on repeated values – isRepeating & hasNulls
  • 24. © Hortonworks Inc. 2014. Optimized Row Columnar File • ORC Vectorized Reader • Logical Compression helps reader – isRepeating • Split per-stripe • Row-group level indexes • Stripe level indexes • PPD avoids a lot of IO – Column conditions are ANDed
  • 25. © Hortonworks Inc. 2014. Faster Statistics • ORC stripe footers aggregate stats per-column – Min/Max/Sum/Count • set hive.stats.autogather=true; • ANALYZE TABLE <table> compute statistics partialscan; – Reads only ORC footers • Predicate computation without Tez/MR tasks
  • 26. © Hortonworks Inc. 2014. Faster Execution: Tez • Multiple edge types – Broadcast – Shuffle – One-to-One • Multiple output types – Sorted – Unsorted – Unsorted Partitioned • Per-vertex configurations – Instead of one configuration between M&R tasks
  • 27. © Hortonworks Inc. 2014. Tez I/O speed-ups • Tez shuffle can use keep-alive over HTTP • Shuffle scheduler can optimize connection count – Can fetch all map outputs from one node via 1 connection • Can skip fetching 0 sized partitions from a mapper – Speeds up group-by queries with high locality – Reducers finish shuffle faster • Shuffle threads are re-used in container re-use – Secure shuffle has crypto thread-local inits
  • 28. © Hortonworks Inc. 2014. Skewed Reducers: auto-parallelism • Often queries are slow because of one slow reducer • Skewed data is too common in real life queries • This avoids running too many reducers with with very little data • Future – This can be extended to group by input size – This mechanism can actually speculate on stalling reducers better (split into 3)
  • 29. © Hortonworks Inc. 2014. A Query in motion Page 32 • 4-way Map join + map reduce reduce query • Timeline in left to right, each lane represents one container
  • 30. © Hortonworks Inc. 2014. Defer/Skip tasks Page 33 • No more uploading hive-exec.jar/UDFs for every query • No more spinning up an AM for each stage • No more computation on hive client (local task)
  • 31. © Hortonworks Inc. 2014. Concurrency of small tasks Page 34 • Hive used to run several lightweight tasks in a local VM • LocalTask was a bottleneck – No locality – No parallelism – Small VM • Tez Broadcast edges solve that problem
  • 32. © Hortonworks Inc. 2014. Concurrent Split Generation Page 35 • Tez input intializers are run parallel • No more spinning up an AM for each stage • No more computation on hive client (local task)
  • 33. © Hortonworks Inc. 2014. Split Elimination Page 36 • ORC comes with Predicate Push Down in the reader • Queries with SARGable where clauses – http://en.wikipedia.org/wiki/Sargable • Run the SARGs in the AM, using ORC footer data – Eliminate splits before task spinups, avoid container costs • Offers a soft cache for the ORC footers • Zero splits offers an early exit for data validity checks (i.e price < 0)
  • 34. © Hortonworks Inc. 2014. Pipelining Split->Task Page 37 • The task only depends on its own input • It starts talking to YARN immediately once its inputs are ready • Faster generation of dimension tables • Fact tables can optimize on this further – Will break existing FileSplit mechanism
  • 35. © Hortonworks Inc. 2014. Filling up the pipeline Page 38 • Tez allows grouping splits dynamically • Obsoletes CombineFileInputFormat • Grouped according to locality –1.7 x available containers (or any factor actually) • Allow query to use up 100% of queue capacity –Without tuning mapred split size for each data-set
  • 36. © Hortonworks Inc. 2014. ORC Split extras • RCFile had horrible split performance – rcfile::sync() was slow to find a sync point • ORC Reader allows exact splits for stripes • ORC Writer can pad a stripe to an HDFS block – 5%-7% overhead measured on table – 100% locality of a stripe in a block
  • 37. © Hortonworks Inc. 2014. Container reuse • Tez specific feature • Run an entire DAG using the same containers • Different vertices use same container • Saves time talking to YARN for new containers
  • 38. © Hortonworks Inc. 2014. Container reuse (II) • Tez provides an object registry within a vertex • This can be used to cache map-join hash-tables • JVM JIT kicks in and optimizes better on re-use
  • 39. © Hortonworks Inc. 2014. Container re-use (Session) • Keep a container group alive between queries • Fast query spin-up and skip YARN queue • Even better JIT performance on >1 queries
  • 40. © Hortonworks Inc. 2014. HiveServer2 and Sessions • HiveServer2 can keep sessions alive –Between different JDBC queries • New security model helps –All secure queries run as “hive” user • Ideal for short exploratory queries • Uses same JARs (no download for task) • Even better JIT performance on >1 queries
  • 41. © Hortonworks Inc. 2014. Supersize it! • 78 vertex + 8374 tasks on 50 containers Page 44
  • 42. © Hortonworks Inc. 2014. Query overload #2 • 5000 hive query test-set • Only 3.9k triggered compute tasks • Rest was optimized away into fetch tasks or metadata tasks • Gets progressively faster as the JVM JIT improves the native code Page 45
  • 43. © Hortonworks Inc. 2014. Big picture 1501.895 1176.479 631.027 4.872 0 200 400 600 800 1000 1200 1400 1600 Text Columnar Partitioned Stinger Latency
  • 44. © Hortonworks Inc. 2014. Roadmap • Expand uses for CBO – Join Algorithm selection – Tez checkpoint selection (recovery) • (In-memory) Temp Tables – Explicit/Implicit – Speed up sharing of intermediate results • Materialized views – Pre-compute common results/aggregations – Transparently route via CBO • Join/Grouping w/o sort – Tez decouples algorithm from data transfer • Sort-merge bucket in Tez – Leverage vertex manager – Co-locate partitions on HDFS • Inline sampling/range partitioning with Tez – Sample/create histogram dynamically for skew joins and total order sort Page 47

Editor's Notes

  1. With Hive and Stinger we are focused on enabling the SQL ecosystem and to do that we’ve put Hive on a clear roadmap to SQL compliance. That includes adding critical datatypes like character and date types as well as implementing common SQL semantics seen in most databases.
  2. checkcast
  3. Tpch query 1 and query 6. Before:
  4. 1Tb of tpc-hdata compreses to 200Gb of ORC data. 30Tb of tpc-ds data compresses to approx ~6Tb of ORC data.