© Hortonworks Inc. 2013 Page 1
Apache Tez : Accelerating Hadoop
Query Processing
Bikas Saha
@bikassaha
© Hortonworks Inc. 2013
Tez – Introduction
Page 2
• Distributed execution framework
targeted towards data-processing
appli...
© Hortonworks Inc. 2012© Hortonworks Inc. 2013. Confidential and Proprietary.
Tez – Hadoop 1 ---> Hadoop 2
HADOOP 1.0
HDFS...
© Hortonworks Inc. 2013
Tez – Empowering Applications
Page 4
• Tez solves hard problems of running on a distributed Hadoop...
© Hortonworks Inc. 2013
Tez – End User Benefits
• Better performance of applications
• Built-in performance + Application ...
© Hortonworks Inc. 2013
Tez – Design considerations
Don’t solve problems that have already been solved. Or else
you will h...
© Hortonworks Inc. 2013
Tez – Problems that it addresses
• Expressing the computation
• Direct and elegant representation ...
© Hortonworks Inc. 2013
Tez – Simplifying Operations
• No deployments to do. No side effects. Easy and safe to try it out!...
© Hortonworks Inc. 2013
Tez – Expressing the computation
Page 9
Aggregate Stage
Partition Stage
Preprocessor Stage
Sampler...
© Hortonworks Inc. 2013
Tez – Expressing the computation
Page 10
Tez provides the following APIs to define the processing
...
© Hortonworks Inc. 2013
Tez – DAG API
// Define DAG
DAG dag = new DAG();
// Define Vertex
Vertex Map1 = new Vertex(Process...
© Hortonworks Inc. 2013
Tez – DAG API
Page 12
• Data movement – Defines routing of data between tasks
– One-To-One : Data ...
© Hortonworks Inc. 2013
Tez – Logical DAG expansion at Runtime
Page 13
Reduce1
Map2
Reduce2
Join
Map1
© Hortonworks Inc. 2013
Tez – Runtime API
Flexible Inputs-Processor-Outputs Model
• Thin API layer to wrap around arbitrar...
© Hortonworks Inc. 2013
Tez – Library of Inputs and Outputs
Page 15
Classical ‘Map’ Classical ‘Reduce’
Intermediate ‘Reduc...
© Hortonworks Inc. 2013
Tez – Composable Task Model
Page 16
Hive Processor
HDFS
Input
Remote
File
Server
Input
HDFS
Output...
© Hortonworks Inc. 2013
Tez – Performance
• Benefits of expressing the data processing as a DAG
• Reducing overheads and q...
© Hortonworks Inc. 2013
Tez – Benefits of DAG execution
Faster Execution and Higher Predictability
• Eliminate replicated ...
© Hortonworks Inc. 2013
Tez – Container Re-Use
• Reuse YARN containers/JVMs to launch new tasks
• Reduce scheduling and la...
© Hortonworks Inc. 2013
Tez – Sessions
Page 20
Application Master
Client
Start
Session
Submit
DAG
Task Scheduler
Container...
© Hortonworks Inc. 2013
Tez – Re-Use in Action
Task Execution Timeline
Containers
Time 
© Hortonworks Inc. 2013
Tez – Customizable Core Engine
Page 22
Vertex-2
Vertex-1
Start
vertex
Vertex Manager
Start
tasks
D...
© Hortonworks Inc. 2013
Tez – Event Based Control Plane
Page 23
Reduce Task 2
Input1 Input2
Map Task 2
Output1
Output2
Out...
© Hortonworks Inc. 2013
Tez – Automatic Reduce Parallelism
Page 24
Map Vertex
Reduce Vertex
App Master
Vertex Manager
Data...
© Hortonworks Inc. 2013
Tez – Reduce Slow Start/Pre-launch
Page 25
Map Vertex
Reduce Vertex
App Master
Vertex Manager
Task...
© Hortonworks Inc. 2013
Tez – Theory to Practice
• Performance
• Scalability
Page 26
© Hortonworks Inc. 2013
Tez – Hive TPC-DS Scale 200GB latency
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Replicated
Join (2.8x)
Join +
Groupby
(1.5x)
Join +
Groupby +
Orderby
(...
© Hortonworks Inc. 2013
Tez – Observations on Performance
• Number of stages in the DAG
• Higher the number of stages in t...
© Hortonworks Inc. 2013
Tez – Data at scale
Page 30
Hive TPC-DS
Scale 10TB
© Hortonworks Inc. 2013
Tez – DAG definition at scale
Page 31
Hive : TPC-DS Query 64 Logical DAG
© Hortonworks Inc. 2013
Tez – Container Reuse at Scale
78 vertices + 8374 tasks on 50 containers (TPC-DS Query 4)
Page 32
© Hortonworks Inc. 2013
Tez – Real World Use Cases for the API
Page 33
© Hortonworks Inc. 2013
Tez – Broadcast Edge
SELECT ss.ss_item_sk, ss.ss_quantity, avg_price, inv.inv_quantity_on_hand
FRO...
© Hortonworks Inc. 2013
Tez – Multiple Outputs
Page 35
Pig : Split & Group-by
f = LOAD ‘foo’ AS (x, y, z);
g1 = GROUP f BY...
© Hortonworks Inc. 2013
Tez – Multiple Outputs
FROM (SELECT * FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk...
© Hortonworks Inc. 2013
Tez – One to One Edge
Page 37
Aggregate
Sample L
Join
Stage sample map
on distributed cache
l = LO...
© Hortonworks Inc. 2013
Tez – Custom Edge
SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand
FROM store_sales ...
© Hortonworks Inc. 2013
Tez – Bringing it all together
Page 39
Architecting the Future of Big Data
Tez Session populates
c...
© Hortonworks Inc. 2013
Tez – Bridging the Data Spectrum
Page 40
Fact Table
Dimension
Table 1
Result
Table 1
Dimension
Tab...
© Hortonworks Inc. 2013
Tez – Current status
• Apache Incubator Project
–Rapid development. Over 1000 jiras opened. Over 7...
© Hortonworks Inc. 2013
Tez – Adoption
• Apache Hive
• Hadoop standard for declarative access via SQL-like interface
• Apa...
© Hortonworks Inc. 2013
Tez – Adoption Path
Pre-requisite : Hadoop 2 with YARN
Tez has zero deployment pain. No side effec...
© Hortonworks Inc. 2013
Tez – Roadmap
• Richer DAG support
– Addition of vertices at runtime
– Shared edges for shared out...
© Hortonworks Inc. 2013
Tez – Community
• Early adopters and code contributors welcome
– Adopters to drive more scenarios....
© Hortonworks Inc. 2013
Tez – Takeaways
• Distributed execution framework that models processing as
dataflow graphs
• Cust...
© Hortonworks Inc. 2013
Tez
Thanks for your time and attention!
Video with Deep Dive on Tez
http://youtu.be/-7YhVwqky6M
ht...
Upcoming SlideShare
Loading in …5
×

Apache Tez : Accelerating Hadoop Query Processing

3,011 views

Published on

Apache Tez is the new data processing framework in the Hadoop ecosystem. It runs on top of YARN - the new compute platform for Hadoop 2. Learn how Tez is built from the ground up to tackle a broad spectrum of data processing scenarios in Hadoop/BigData - ranging from interactive query processing to complex batch processing. With a high degree of automation built-in, and support for extensive customization, Tez aims to work out of the box for good performance and efficiency. Apache Hive and Pig are already adopting Tez as their platform of choice for query execution.

Published in: Technology, Education
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,011
On SlideShare
0
From Embeds
0
Number of Embeds
232
Actions
Shares
0
Downloads
56
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide
  • For anyone who has been working on MapReduce, there is this age-old problem around “how do I figure out the correct number of reducers?”. We guess some number at compile-time and usually that turns out to be incorrect at run-time. Let’s see how we can use the Tez model to fix that. So here is this Map Vertex and this Reduce Vertex, which have these tasks running and you have the Vertex Manager running inside the framework …

    [CLICK] The Map Tasks can send Data Size Statistics to the Vertex Manager, which can then extrapolate those statistics to figure out “what would be the final size of the data when all of these Maps finish?”. Based on that, it can realize that the data size is actually smaller than expected, and I can actually run two reduce tasks instead of three.

    [CLICK] The Vertex Manager sends a Set Paralellism command to the framework which changes the routing information in-between these two tasks and also cancels the last task.
  • So the new plan looks like this … which is how Tez handles the concurrency problem of plan re-configuration. What about “when do you actually want to launch those tasks?”?

    In MapReduce, there is an interesting concept of pre-launching the Reducers, in which even though the reducers are going to read the output from 100 mappers, we may want to launch the Reducers when only 50 mappers have finished – because then they can use the capacity in the cluster to start pre-fetching the data from the completed mappers while the rest of their inputs still complete. How can we do this in Tez? Again, the same Vertex Manager comes into the picture.

    [CLICK] As the Map Tasks complete they send Task Completed notifications to the Vertex Manager. There it can decide that enough Map tasks have finished that I need to start one of my reducers so that it can start pre-fetching the data.

    [CLICK] It sends a Start Task notification to the framework, which results in one of the Reduce Tasks starting.
  • query 1: SELECT pageURL, pageRank FROM rankings WHERE pageRank > X
  • 1.5x to 3x speedup on some of the Pigmix queries.
  • Hive has written it’s own processor
  • Hive has written it’s own processor
  • Apache Tez : Accelerating Hadoop Query Processing

    1. 1. © Hortonworks Inc. 2013 Page 1 Apache Tez : Accelerating Hadoop Query Processing Bikas Saha @bikassaha
    2. 2. © Hortonworks Inc. 2013 Tez – Introduction Page 2 • Distributed execution framework targeted towards data-processing applications. • Based on expressing a computation as a dataflow graph. • Highly customizable to meet a broad spectrum of use cases. • Built on top of YARN – the resource management framework for Hadoop. • Open source Apache incubator project and Apache licensed.
    3. 3. © Hortonworks Inc. 2012© Hortonworks Inc. 2013. Confidential and Proprietary. Tez – Hadoop 1 ---> Hadoop 2 HADOOP 1.0 HDFS (redundant, reliable storage) MapReduce (cluster resource management & data processing) Pig (data flow) Hive (sql) Others (cascading) HDFS2 (redundant, reliable storage) YARN (cluster resource management) Tez (execution engine) HADOOP 2.0 Data Flow Pig SQL Hive Others (Cascading) Batch MapReduce Real Time Stream Processing Storm Online Data Processing HBase, Accumulo Monolithic • Resource Management • Execution Engine • User API Layered • Resource Management – YARN • Execution Engine – Tez • User API – Hive, Pig, Cascading, Your App!
    4. 4. © Hortonworks Inc. 2013 Tez – Empowering Applications Page 4 • Tez solves hard problems of running on a distributed Hadoop environment • Apps can focus on solving their domain specific problems • This design is important to be a platform for a variety of applications App Tez • Custom application logic • Custom data format • Custom data transfer technology • Distributed parallel execution • Negotiating resources from the Hadoop framework • Fault tolerance and recovery • Horizontal scalability • Resource elasticity • Shared library of ready-to-use components • Built-in performance optimizations • Security
    5. 5. © Hortonworks Inc. 2013 Tez – End User Benefits • Better performance of applications • Built-in performance + Application define optimizations • Better predictability of results • Minimization of overheads and queuing delays • Better utilization of compute capacity • Efficient use of allocated resources • Reduced load on distributed filesystem (HDFS) • Reduce unnecessary replicated writes • Reduced network usage • Better locality and data transfer using new data patterns • Higher application developer productivity • Focus on application business logic rather than Hadoop internals Page 5
    6. 6. © Hortonworks Inc. 2013 Tez – Design considerations Don’t solve problems that have already been solved. Or else you will have to solve them again! • Leverage discrete task based compute model for elasticity, scalability and fault tolerance • Leverage several man years of work in Hadoop Map-Reduce data shuffling operations • Leverage proven resource sharing and multi-tenancy model for Hadoop and YARN • Leverage built-in security mechanisms in Hadoop for privacy and isolation Page 6 Look to the Future with an eye on the Past
    7. 7. © Hortonworks Inc. 2013 Tez – Problems that it addresses • Expressing the computation • Direct and elegant representation of the data processing flow • Interfacing with application code and new technologies • Performance • Late Binding : Make decisions as late as possible using real data from at runtime • Leverage the resources of the cluster efficiently • Just work out of the box! • Customizable engine to let applications tailor the job to meet their specific requirements • Operation simplicity • Painless to operate, experiment and upgrade Page 7
    8. 8. © Hortonworks Inc. 2013 Tez – Simplifying Operations • No deployments to do. No side effects. Easy and safe to try it out! • Tez is a completely client side application. • Simply upload to any accessible FileSystem and change local Tez configuration to point to that. • Enables running different versions concurrently. Easy to test new functionality while keeping stable versions for production. • Leverages YARN local resources. Page 8 Client Machine Node Manager TezTask Node Manager TezTaskTezClient HDFS Tez Lib 1 Tez Lib 2 Client Machine TezClient
    9. 9. © Hortonworks Inc. 2013 Tez – Expressing the computation Page 9 Aggregate Stage Partition Stage Preprocessor Stage Sampler Task-1 Task-2 Task-1 Task-2 Task-1 Task-2 Samples Ranges Distributed Sort Distributed data processing jobs typically look like DAGs (Directed Acyclic Graph). • Vertices in the graph represent data transformations • Edges represent data movement from producers to consumers
    10. 10. © Hortonworks Inc. 2013 Tez – Expressing the computation Page 10 Tez provides the following APIs to define the processing • DAG API • Defines the structure of the data processing and the relationship between producers and consumers • Enable definition of complex data flow pipelines using simple graph connection API’s. Tez expands the logical DAG at runtime • This is how all the tasks in the job get specified • Runtime API • Defines the interfaces using which the framework and app code interact with each other • App code transforms data and moves it between tasks • This is how we specify what actually executes in each task on the cluster nodes
    11. 11. © Hortonworks Inc. 2013 Tez – DAG API // Define DAG DAG dag = new DAG(); // Define Vertex Vertex Map1 = new Vertex(Processor.class); // Define Edge Edge edge = Edge(Map1, Reduce1, SCATTER_GATHER, PERSISTED, SEQUENTIAL, Output.class, Input.class); // Connect them dag.addVertex(Map1).addEdge(edge)… Page 11 Defines the global processing flow Map1 Map2 Reduce1 Reduce2 Join Scatter Gather Scatter Gather
    12. 12. © Hortonworks Inc. 2013 Tez – DAG API Page 12 • Data movement – Defines routing of data between tasks – One-To-One : Data from the ith producer task routes to the ith consumer task. – Broadcast : Data from a producer task routes to all consumer tasks. – Scatter-Gather : Producer tasks scatter data into shards and consumer tasks gather the data. The ith shard from all producer tasks routes to the ith consumer task. • Scheduling – Defines when a consumer task is scheduled – Sequential : Consumer task may be scheduled after a producer task completes. – Concurrent : Consumer task must be co-scheduled with a producer task. • Data source – Defines the lifetime/reliability of a task output – Persisted : Output will be available after the task exits. Output may be lost later on. – Persisted-Reliable : Output is reliably stored and will always be available – Ephemeral : Output is available only while the producer task is running Edge properties define the connection between producer and consumer tasks in the DAG
    13. 13. © Hortonworks Inc. 2013 Tez – Logical DAG expansion at Runtime Page 13 Reduce1 Map2 Reduce2 Join Map1
    14. 14. © Hortonworks Inc. 2013 Tez – Runtime API Flexible Inputs-Processor-Outputs Model • Thin API layer to wrap around arbitrary application code • Compose inputs, processor and outputs to execute arbitrary processing • Event routing based control plane architecture • Applications decide logical data format and data transfer technology • Customize for performance • Built-in implementations for Hadoop 2.0 data services – HDFS and YARN ShuffleService. Built on the same API. Your impls are as first class as ours! Page 14
    15. 15. © Hortonworks Inc. 2013 Tez – Library of Inputs and Outputs Page 15 Classical ‘Map’ Classical ‘Reduce’ Intermediate ‘Reduce’ for Map-Reduce-Reduce Map Processor HDFS Input Sorted Output Reduce Processor Shuffle Input HDFS Output Reduce Processor Shuffle Input Sorted Output • What is built in? – Hadoop InputFormat/OutputFormat – SortedGroupedPartitioned Key-Value Input/Output – UnsortedGroupedPartitioned Key-Value Input/Output – Key-Value Input/Output
    16. 16. © Hortonworks Inc. 2013 Tez – Composable Task Model Page 16 Hive Processor HDFS Input Remote File Server Input HDFS Output Local Disk Output Your Processor HDFS Input Remote File Server Input HDFS Output Local Disk Output Your Processor RDMA Input Native DB Input Kakfa Pub-Sub Output Amazon S3 Output Adopt Evolve Optimize
    17. 17. © Hortonworks Inc. 2013 Tez – Performance • Benefits of expressing the data processing as a DAG • Reducing overheads and queuing effects • Gives system the global picture for better planning • Efficient use of resources • Re-use resources to maximize utilization • Pre-launch, pre-warm and cache • Locality & resource aware scheduling • Support for application defined DAG modifications at runtime for optimized execution • Change task concurrency • Change task scheduling • Change DAG edges • Change DAG vertices Page 17
    18. 18. © Hortonworks Inc. 2013 Tez – Benefits of DAG execution Faster Execution and Higher Predictability • Eliminate replicated write barrier between successive computations. • Eliminate job launch overhead of workflow jobs. • Eliminate extra stage of map reads in every workflow job. • Eliminate queue and resource contention suffered by workflow jobs that are started after a predecessor job completes. • Better locality because the engine has the global picture Page 18 Pig/Hive - MR Pig/Hive - Tez
    19. 19. © Hortonworks Inc. 2013 Tez – Container Re-Use • Reuse YARN containers/JVMs to launch new tasks • Reduce scheduling and launching delays • Shared in-memory data across tasks • JVM JIT friendly execution Page 19 YARN Container / JVM TezTask Host TezTask1 TezTask2 SharedObjects YARN Container Tez Application Master Start Task Task Done Start Task
    20. 20. © Hortonworks Inc. 2013 Tez – Sessions Page 20 Application Master Client Start Session Submit DAG Task Scheduler ContainerPool Shared Object Registry Pre Warmed JVM Sessions • Standard concepts of pre-launch and pre-warm applied • Key for interactive queries • Represents a connection between the user and the cluster • Multiple DAGs executed in the same session • Containers re-used across queries • Takes care of data locality and releasing resources when idle
    21. 21. © Hortonworks Inc. 2013 Tez – Re-Use in Action Task Execution Timeline Containers Time 
    22. 22. © Hortonworks Inc. 2013 Tez – Customizable Core Engine Page 22 Vertex-2 Vertex-1 Start vertex Vertex Manager Start tasks DAG Scheduler Get Priority Get Priority Start vertex Task Scheduler Get container Get container • Vertex Manager • Determines task parallelism • Determines when tasks in a vertex can start. • DAG Scheduler Determines priority of task • Task Scheduler Allocates containers from YARN and assigns them to tasks
    23. 23. © Hortonworks Inc. 2013 Tez – Event Based Control Plane Page 23 Reduce Task 2 Input1 Input2 Map Task 2 Output1 Output2 Output3 Map Task 1 Output1 Output2 Output3 AM Router Scatter-Gather Edge • Events used to communicate between the tasks and between task and framework • Data Movement Event used by producer task to inform the consumer task about data location, size etc. • Input Error event sent by task to the engine to inform about errors in reading input. The engine then takes action by re-generating the input • Other events to send task completion notification, data statistics and other control plane information Data Event Error Event
    24. 24. © Hortonworks Inc. 2013 Tez – Automatic Reduce Parallelism Page 24 Map Vertex Reduce Vertex App Master Vertex Manager Data Size Statistics Vertex State Machine Set Parallelism Cancel Task Re-Route Event Model Map tasks send data statistics events to the Reduce Vertex Manager. Vertex Manager Pluggable application logic that understands the data statistics and can formulate the correct parallelism. Advises vertex controller on parallelism
    25. 25. © Hortonworks Inc. 2013 Tez – Reduce Slow Start/Pre-launch Page 25 Map Vertex Reduce Vertex App Master Vertex Manager Task Completed Vertex State Machine Start Tasks Start Event Model Map completion events sent to the Reduce Vertex Manager. Vertex Manager Pluggable application logic that understands the data size. Advises the vertex controller to launch the reducers before all maps have completed so that shuffle can start.
    26. 26. © Hortonworks Inc. 2013 Tez – Theory to Practice • Performance • Scalability Page 26
    27. 27. © Hortonworks Inc. 2013 Tez – Hive TPC-DS Scale 200GB latency
    28. 28. 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Replicated Join (2.8x) Join + Groupby (1.5x) Join + Groupby + Orderby (1.5x) 3 way Split + Join + Groupby + Orderby (2.6x) Timeinsecs MR Tez Tez – Pig performance gains • Demonstrates performance gains from a basic translation to a Tez DAG • Deeper integration in the works for further improvements
    29. 29. © Hortonworks Inc. 2013 Tez – Observations on Performance • Number of stages in the DAG • Higher the number of stages in the DAG, performance of Tez (over MR) will be better. • Cluster/queue capacity • More congested a queue is, the performance of Tez (over MR) will be better due to container reuse. • Size of intermediate output • More the size of intermediate output, the performance of Tez (over MR) will be better due to reduced HDFS usage. • Size of data in the job • For smaller data and more stages, the performance of Tez (over MR) will be better as percentage of launch overhead in the total time is high for smaller jobs. • Offload work to the cluster • Move as much work as possible to the cluster by modelling it via the job DAG. Exploit the parallelism and resources of the cluster. E.g. MR split calculation. • Vertex caching • The more re-computation can be avoided the better is the performance. Page 29
    30. 30. © Hortonworks Inc. 2013 Tez – Data at scale Page 30 Hive TPC-DS Scale 10TB
    31. 31. © Hortonworks Inc. 2013 Tez – DAG definition at scale Page 31 Hive : TPC-DS Query 64 Logical DAG
    32. 32. © Hortonworks Inc. 2013 Tez – Container Reuse at Scale 78 vertices + 8374 tasks on 50 containers (TPC-DS Query 4) Page 32
    33. 33. © Hortonworks Inc. 2013 Tez – Real World Use Cases for the API Page 33
    34. 34. © Hortonworks Inc. 2013 Tez – Broadcast Edge SELECT ss.ss_item_sk, ss.ss_quantity, avg_price, inv.inv_quantity_on_hand FROM (select avg(ss_sold_price) as avg_price, ss_item_sk, ss_quantity_sk from store_sales group by ss_item_sk) ss JOIN inventory inv ON (inv.inv_item_sk = ss.ss_item_sk); Hive – MR Hive – Tez M M M M M HDFS Store Sales scan. Group by and aggregation reduce size of this input. Inventory scan and Join Broadcast edge M M M HDFS Store Sales scan. Group by and aggregation. Inventory and Store Sales (aggr.) output scan and shuffle join. R R R R RR M MMM HDFS Hive : Broadcast Join
    35. 35. © Hortonworks Inc. 2013 Tez – Multiple Outputs Page 35 Pig : Split & Group-by f = LOAD ‘foo’ AS (x, y, z); g1 = GROUP f BY y; g2 = GROUP f BY z; j = JOIN g1 BY group, g2 BY group; Group by y Group by z Load foo Join Load g1 and Load g2 Group by y Group by z Load foo Join Multiple outputs Reduce follows reduce HDFS HDFS Split multiplex de-multiplex Pig – MR Pig – Tez
    36. 36. © Hortonworks Inc. 2013 Tez – Multiple Outputs FROM (SELECT * FROM store_sales, date_dim WHERE ss_sold_date_sk = d_date_sk and d_year = 2000) INSERT INTO TABLE t1 SELECT distinct ss_item_sk INSERT INTO TABLE t2 SELECT distinct ss_customer_sk; Hive – MR Hive – Tez M MM M HDFS Map join date_dim/store sales Two MR jobs to do the distinct M MM M M HDFS RR HDFS M M M R M M M R HDFS Broadcast Join (scan date_dim, join store sales) Distinct for customer + items Materialize join on HDFS Hive : Multi-insert queries
    37. 37. © Hortonworks Inc. 2013 Tez – One to One Edge Page 37 Aggregate Sample L Join Stage sample map on distributed cache l = LOAD ‘left’ AS (x, y); r = LOAD ‘right’ AS (x, z); j = JOIN l BY x, r BY x USING ‘skewed’; Load & Sample Aggregate Partition L Join Pass through input via 1-1 edge Partition R HDFS Broadcast sample map Partition L and Partition R Pig – MR Pig – Tez Pig : Skewed Join
    38. 38. © Hortonworks Inc. 2013 Tez – Custom Edge SELECT ss.ss_item_sk, ss.ss_quantity, inv.inv_quantity_on_hand FROM store_sales ss JOIN inventory inv ON (inv.inv_item_sk = ss.ss_item_sk); Hive – MR Hive – Tez M MM M M HDFS Inventory scan (Runs on cluster potentially more than 1 mapper) Store Sales scan and Join (Custom vertex reads both inputs – no side file reads) Custom edge (routes outputs of previous stage to the correct Mappers of the next stage)M MM M HDFS Inventory scan (Runs as single local map task) Store Sales scan and Join (Inventory hash table read as side file) HDFS Hive : Dynamically Partitioned Hash Join
    39. 39. © Hortonworks Inc. 2013 Tez – Bringing it all together Page 39 Architecting the Future of Big Data Tez Session populates container pool Dimension table calculation and HDFS split generation in parallel Dimension tables broadcasted to Hive MapJoin tasks Final Reducer pre- launched and fetches completed inputs TPCDS – Query-27 with Hive on Tez
    40. 40. © Hortonworks Inc. 2013 Tez – Bridging the Data Spectrum Page 40 Fact Table Dimension Table 1 Result Table 1 Dimension Table 2 Result Table 2 Dimension Table 3 Result Table 3 Broadcast Join Shuffle Join Typical pattern in a TPC-DS query Fact Table Dimension Table 1 Dimension Table 1 Dimension Table 1 Broadcast join for small data sets Based on data size, the query optimizer can run either plan as a single Tez job Broadcast Join
    41. 41. © Hortonworks Inc. 2013 Tez – Current status • Apache Incubator Project –Rapid development. Over 1000 jiras opened. Over 700 resolved –Growing community of contributors and users –Latest release is 0.4 • Focus on stability –Testing and quality are highest priority –Code ready and deployed on multi-node environments at scale • Support for a vast topology of DAGs – Already functionally equivalent to Map Reduce. Existing Map Reduce jobs can be executed on Tez with few or no changes –Apache Hive 0.13 release supports Tez as an execution engine (HIVE- 4660) –Apache Pig port to Tez close to completion (PIG-3446) Page 41
    42. 42. © Hortonworks Inc. 2013 Tez – Adoption • Apache Hive • Hadoop standard for declarative access via SQL-like interface • Apache Pig • Hadoop standard for procedural scripting and pipeline processing • Cascading • Developer friendly Java API and SDK • Scalding (Scala API on Cascading) • Commercial Vendors • ETL : Use Tez instead of MR or custom pipelines • Analytics Vendors : Use Tez as a target platform for scaling parallel analytical tools to large data-sets Page 42
    43. 43. © Hortonworks Inc. 2013 Tez – Adoption Path Pre-requisite : Hadoop 2 with YARN Tez has zero deployment pain. No side effects or traces left behind on your cluster. Low risk and low effort to try out. • Using Hive, Pig, Cascading, Scalding • Try them with Tez as execution engine • Already have MapReduce based pipeline • Use configuration to change MapReduce to run on Tez by setting ‘mapreduce.framework.name’ to ‘yarn-tez’ in mapred-site.xml • Consolidate MR workflow into MR-DAG on Tez • Change MR-DAG to use more efficient Tez constructs • Have custom pipeline • Wrap custom code/scripts into Tez inputs-processor-outputs • Translate your custom pipeline topology into Tez DAG • Change custom topology to use more efficient Tez constructs Page 43
    44. 44. © Hortonworks Inc. 2013 Tez – Roadmap • Richer DAG support – Addition of vertices at runtime – Shared edges for shared outputs – Enhance Input/Output library • Performance optimizations – Improve support for high concurrency – Improve locality aware scheduling. – Add framework level data statistics – HDFS memory storage integration • Usability – Stability and testability – API ease of use – Tools for performance analysis and debugging Page 44
    45. 45. © Hortonworks Inc. 2013 Tez – Community • Early adopters and code contributors welcome – Adopters to drive more scenarios. Contributors to make them happen. • Tez meetup for developers and users – http://www.meetup.com/Apache-Tez-User-Group • Technical blog series – http://hortonworks.com/blog/apache-tez-a-new-chapter-in-hadoop-data- processing • Useful links – Work tracking: https://issues.apache.org/jira/browse/TEZ – Code: https://github.com/apache/incubator-tez – Developer list: dev@tez.incubator.apache.org User list: user@tez.incubator.apache.org Issues list: issues@tez.incubator.apache.org Page 45
    46. 46. © Hortonworks Inc. 2013 Tez – Takeaways • Distributed execution framework that models processing as dataflow graphs • Customizable execution architecture designed for extensibility and user defined performance optimizations • Works out of the box with the platform figuring out the hard stuff • No install – Safe and Easy to Evaluate and Experiment • Addresses common needs of a broad spectrum of applications and usage patterns • Open source Apache project – your use-cases and code are welcome • It works and is already being used by Apache Hive and Pig Page 46
    47. 47. © Hortonworks Inc. 2013 Tez Thanks for your time and attention! Video with Deep Dive on Tez http://youtu.be/-7YhVwqky6M http://www.infoq.com/presentations/apache-tez Questions? @bikassaha Page 47

    ×