Download to read offline
Uploaded byGal Vinograd
Tuning up with Apache Tez
This document discusses tuning a pipeline with Apache Tez. It provides an overview of Tez, including that it aims to enhance scenarios not well served by MapReduce by supporting Hive and Pig. It details lessons learned from using Tez, such as some Pig tasks not compiling and occasional freezing, lack of DistributedCache support for S3, and poor Amazon support. The document concludes by stating that Tez is good for early adopters of Pig and Hive and those with bounded workloads.
![[2C1] 아파치 피그를 위한 테즈 연산 엔진 개발하기 최종](https://cdn.slidesharecdn.com/ss_thumbnails/2b1-140929191628-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Tuning up with Apache Tez
- 1. Tuning Up WithApache Tez Gal Vinograd @ Crosswise - 2016/03/09
- 3. Agenda The Pipeline The Problem Whywe chose Tez Lessons Learned Demo
- 4.
- 5.
- 6. 250 c3.2xlarge X 30hours 10TB per Batch
- 8. “Tez aims tobe a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce. In the short term the major focus is to support Hive and Pig ...” Tez Design v1.1
- 9. “Tez aims tobe a general purpose execution runtime that enhances various scenarios that are not well served by classic Map-Reduce. In the short term the major focus is to support Hive and Pig ...” Tez Design v1.1
- 10.
- 11.
- 12.
- 13. Logical and PhysicalGraphs PhysicalLogical Hortonworks
- 14.
- 15. Optimizations No Barrier BetweenJobs Project GroupBy Project Project Distinct Project Distinct GroupBy Tez MR
- 16. Optimizations No Redundant ResourceAllocation Project Project Distinct GroupBy Project Project Distinct GroupBy Pig Process Pig Process Tez MR
- 17.
- 18. Lessons Learned Some PigTasks Did Not Compile Occasionaly Froze No DistributedCache Support For S3 Poor Amazon Support No Pre-Built Releases Additional Deployment for Tez UI
- 19. What is itgood for? Earily Adopters Pig Hive Bounded
- 20.
Editor's Notes
- #19 -Dpig.tez.opt.union=false