February 2014 HUG : Pig On Tez


Published on

February 2014 HUG: Pig On Tez

Published in: Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Pigmix queries
  • - Either turn off container reuse or fix code
  • - Both Algebraic and Accumulator UDFs
  • - Both Algebraic and Accumulator UDFs
  • February 2014 HUG : Pig On Tez

    1. 1. Pig on Tez PRESENTED BY Cheolsoo Park, Netflix R o h i n i P a l a n i s w a m y , Ya h o o ! The Apache Software Foundation
    2. 2. Apache Pig on Tez Team Name Role Company Apache Pig Contributor Linkedin Cheolsoo Park VP. Apache Pig Netflix Daniel Dai Apache Pig PMC Hortonworks Mark Wagner Apache Pig Committer Linkedin Olga Natkovich Apache Pig PMC, Pig on Tez Project Manager Yahoo! Rohini Palaniswamy Apache Pig PMC Yahoo! Alex Bain The Apache Software Foundation 2
    3. 3. Agenda  Overview  Pig and Hive  Pig on Tez  Why Tez?  Benefits of Tez  Design  Operator DAGs  Performance  Known Issues  Where are we?  What next? The Apache Software Foundation 3
    4. 4. Pig Overview  Apache top-level project for ETL on hadoop.  PIG Latin - Procedural scripting language that translates sequence of data processing steps into MapReduce jobs.  Easy to write, read and reuse and very extensible.  Feature parity with SQL (FILTER BY, CROSS, JOIN (OUTER, INNER), ORDER BY, LIMIT, RANK, ROLLUP, CUBE), Custom Loader and Storer, User defined functions (java and non-java), Nested ForEach, Streaming, macros and much more PAGEVIEWS = LOAD ‘/data/pageviews’ as (user, url); GRP = GROUP PAGEVIEWS BY user; CNT = FOREACH GRP GENERATE group, COUNT(url) as numvisits; STORE CNT into ‘/data/visited’ using PigStorage(‘,’); The Apache Software Foundation 4
    5. 5. Pig and Hive Pig Language Hive PIG Latin - Procedural SQL - Declarative Features Feature rich. Can easily add new operators and constructs. For eg: Nested Foreach, Switch case, Macros, Scalars. Limited to SQL operators Developer code Load/StoreFunc, Algebraic and Accumulator java UDFs, non-java UDFs (jython, python, javascript, groovy, jruby), Custom Partitioners. StorageHandler, java UDFs Complex Processing Well suited. Multi-query works well with 1000s of lines of pig script. Not a good fit Server Only client. Can work with Hive Metastore using HCatLoader/Storer. Requires Metastore server and data has to be registered in it. HiverServer2 for jdbc The Apache Software Foundation 5
    6. 6. Pig and Hive - Continued Pig Tez as execution engine Hive Planned for 0.14 Planned for Hive 0.13 ORCFile Support Patch available. Currently through HCatLoader From Hive 0.12 onwards. Huge performance gains Vectorization No. May be in future. Yes. Huge performance gains Transactions No Yes. In works Cost-based optimizer No Yes. In works JDBC support, Integration with BI tools No Yes. HiveServer2 with Microstrategy/Tableau Area of application Pipeline processing language standard Interactive Analytics /Reporting Platform The Apache Software Foundation 6
    7. 7. Why Tez?  Built on top of YARN   Multi-tenancy (queues, capacity management) Resource allocation  DAG execution framework   Natural fit for Pig and Hive than MR as their execution plans are DAGs. Better than running a DAG of MR jobs passing data in between jobs using HDFS as intermediate store.  Different types of edges  ONE_ONE, BROADCAST, SHUFFLE  Flexible Input-Processor-Output runtime model        Custom Vertex Processors. For eg: Map Processor, Reduce Processor, Pig Processor Custom Inputs. For eg: MRFileInput (input to map), ShuffledMergedInput (input to reduce) Custom Outputs. For eg: OnFileSortedOutput (output of map), MRFileOutput (output of reduce) Multiple inputs and outputs Highly extensible Security Support from Tez Community and Hive Community The Apache Software Foundation 7
    8. 8. Why Tez? – As a end user     Better Performance Reduced Resource Usage (Containers/Memory/CPU) Reduced Network I/O Reduced Namenode and Datanode load The Apache Software Foundation 8
    9. 9. Benefits of Tez Features Benefits • No intermediate data storage • • • Single AM for whole DAG The Apache Software Foundation • Less pressure on Namenode - Lesser calls for listing and getting block locations - Smaller namespace usage - Cuts down on GC Less pressure on Datanode - Cuts down on IO in network for both writing and reading. - Saves space as there are no 3 replicas Eliminates extra step of map reads from HDFS in every intermediate job in DAG - Saves on capacity by eliminating the need for map task containers Saves on capacity. For a 5 stage MR job, there would be 5 AM containers launched. Eliminates issue of queue and resource contention faced in MR for jobs started after previous job in DAG completes. 9
    10. 10. Benefits of Tez - Continued Features Benefits • Container reuse • Reduced launch overhead - Container request and release overhead - Resource localization overhead - JVM launch time overhead Reduced network IO - Reduce tasks can be launched on same node as Map - 1-1 edge tasks can be launched on same node • Memory structures like small tables used for join can be cached in jvm and reused for next task on container reuse. Provides significant performance speedup. • Using unsorted input and output where possible saves a lot of CPU usage and increases performance • Saves on capacity. Can have reducers based on data size instead of having fixed number of reducers. Vertex caching Custom inputs and outputs Dynamic reducer estimation The Apache Software Foundation 10
    11. 11. Pig on Tez - Design Logical Plan LogToPhyTranslationVisitor Physical Plan TezCompiler MRCompiler Tez Plan MR Plan Tez Execution Engine MR Execution Engine The Apache Software Foundation 11
    12. 12. Pig on Tez – Join Left split Right split Left split Load L and R Right split l = LOAD ‘left’ AS (x, y); r = LOAD ‘right’ AS (x, z); j = JOIN l BY x, r BY x; Configuration per input Configuration per job Join The Apache Software Foundation Left split Left split Right split Load L Load R Join 12 Right split
    13. 13. Pig on Tez – Split + Group-by Load foo Split multiplex de-multiplex Group by y Group by z HDFS f = LOAD ‘foo’ AS (x, y, z); g1 = GROUP f BY y; g2 = GROUP f BY z; j = JOIN g1 BY group, g2 BY group; Load foo Multiple outputs Group by y HDFS Group by z Load g1 and Load g2 Reduce follows reduce Join The Apache Software Foundation Join 13
    14. 14. Pig on Tez – Order-by Aggregate HDFS Load & Sample f = LOAD ‘foo’ AS (x, y); o = ORDER f BY x; Sample Aggregate Stage sample map on distributed cache Pass through input via 1-1 edge Broadcast sample map Partition Partition Sort Sort The Apache Software Foundation 14
    15. 15. Pig on Tez – Skewed join l = LOAD ‘left’ AS (x, y); r = LOAD ‘right’ AS (x, z); j = JOIN l BY x, r BY x USING ‘skewed’; Sample L Load & Sample Aggregate HDFS Aggregate Pass through input via 1-1 edge Stage sample map on distributed cache Broadcast sample map Partition L Partition R Partition L and Partition R Join Join The Apache Software Foundation 15
    16. 16. Time in secs Performance numbers 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0 MR Tez Replicated Join (2.8x) The Apache Software Foundation Join + Groupby (1.5x) Join + 3 way Split + Groupby + Join + Orderby Groupby + (1.5x) Orderby (2.6x) 16
    17. 17. Factors affecting performance  Number of stages in the DAG  Higher the number of stages in the DAG, performance of Tez over MR will be better.  Cluster/queue capacity  More congested a queue is, the performance of Tez over MR will be better due to container reuse.  Size of intermediate output  More the size of intermediate output, the performance of Tez over MR will be better due to reduced HDFS usage.  Size of data in the job  For smaller data and more stages, the performance of Tez over MR will be better as percentage of launch overhead in the total time is high for smaller jobs.  Vertex caching The Apache Software Foundation 17
    18. 18. Container usage MR Tez Savings Tez with container reuse 7563 7562 1 180 Join + Groupby + Orderby 7655 7603 52 180 Join + Groupby + Orderby 7663 7609 54 180 3 way Split + Join + Groupby + Order by 622 563 59 180 Query Replicated Join Note. The cluster size was 25 nodes with 180 containers (1.5G each) and Tez reused them again and again for tasks. The Apache Software Foundation 18
    19. 19. Known issues  Container reuse will have issues when there are  Static variables in LoadFunc, StoreFunc, UDFs  Memory leaks in LoadFunc, StoreFunc, UDFs  With single DAG execution of whole script, AM retries can be very costly until Tez supports checkpointing and resuming. The Apache Software Foundation 19
    20. 20. Where are we?  Major operators   Split, Union  Group-by, Distinct, Limit  Order-by        Load, Store, Filter-by, Foreach Hash join, Replicated join, Skewed join, Merge join UDFs (Java and non-Java) Streaming Multi-query on and off Macros Scalars 95% of e2e tests pass for finished features. The Apache Software Foundation 20
    21. 21. What next?  Feature Parity with MR    Local mode Port all unit and e2e tests Support for remaining Operators   CROSS, RANK, CUBE, ROLLUP Support for Native Mapreduce (Low priority)  Merge tez branch with trunk  Stability  Handling failures  Testing and tuning for large data and DAGs with > 10 stages  Usability  Counters  Progress Information  Log information and debuggability The Apache Software Foundation 21
    22. 22. What next? – Performance Improvements › Dynamic Reducer Estimation › Better memory management › Calculate input splits in AM and let Tez do combining of input splits for pig.maxCombinedSplitSize › Vertex Grouping to write data directly into one output directory from multiple vertices in case of union › Using unsorted shuffle in Union, Orderby, Skewed Join, etc to improve performance. › Shared Edges for multiple outputs if same data has to go to multiple downstream vertices. For eg: Multi-query off, skewed join sample aggregation output. › HDFS Caching The Apache Software Foundation 22
    23. 23. C ontri butors Wel come The Apache Software Foundation
    24. 24. Pi g User Group Meetup at Li nkedIn 14 th March 2014 The Apache Software Foundation
    25. 25. Questi ons ??? The Apache Software Foundation