This document discusses tuning a pipeline with Apache Tez. It provides an overview of Tez, including that it aims to enhance scenarios not well served by MapReduce by supporting Hive and Pig. It details lessons learned from using Tez, such as some Pig tasks not compiling and occasional freezing, lack of DistributedCache support for S3, and poor Amazon support. The document concludes by stating that Tez is good for early adopters of Pig and Hive and those with bounded workloads.

1. Tuning Up With Apache Tez
Gal Vinograd @ Crosswise - 2016/03/09

3. Agenda
The Pipeline
The Problem
Why we chose Tez
Lessons Learned
Demo

4. The Batch
Internet
Labels
Data

5. Internet
Labels
Data
~200
Scripts

6. 250 c3.2xlarge X
30 hours
10TB per Batch

8. “Tez aims to be a general purpose execution
runtime that enhances various scenarios
that are not well served by classic
Map-Reduce. In the short term the major
focus is to support Hive and Pig ...”
Tez Design v1.1

9. “Tez aims to be a general purpose execution
runtime that enhances various scenarios that are
not well served by classic Map-Reduce. In the
short term the major focus is to support
Hive and Pig ...”
Tez Design v1.1

10. Hortonworks

11. The Batch
Internet
Labels
Data
~200
Scripts

12. Tez Atomic Components
Tokenizer
Aggregator
Edge
Vertex
Vertex

13. Logical and Physical Graphs
PhysicalLogical
Hortonworks

14. Optimizations
No “NOP” Map
Project
Distinct
GroupBy
NOP
Project
Distinct
GroupBy
Tez MR

15. Optimizations
No Barrier Between Jobs
Project
GroupBy
Project
Project
Distinct
Project
Distinct
GroupBy
Tez MR

16. Optimizations
No Redundant Resource Allocation
Project
Project
Distinct
GroupBy
Project
Project
Distinct
GroupBy
Pig
Process
Pig
Process
Tez MR

17. Optimizations
Sessions
Allocate
Submit 2
Submit 1
Cleanup
Client

18. Lessons Learned
Some Pig Tasks Did Not Compile Occasionaly Froze
No DistributedCache Support For S3
Poor Amazon Support
No Pre-Built Releases
Additional Deployment for Tez UI

19. What is it good for?
Earily
Adopters
Pig Hive
Bounded

20. Thanks for Listening!