November 2014 HUG: Apache Tez - A Performance View into Large Scale Data-processing, Shuffle Throughput, Reducer parallelism and Reducer Skew

Tez: UI & Debugging
Fall 2014
Version 1.0
Page 1 © Hortonworks Inc. 2014
gopalv@apache.org

TEZ (nomenclature)
• DAG
• Vertex
• Task
• Attempt
• Container
• Edge
Page 2 © Hortonworks Inc. 2014 FOR: BAY AREA HADOOP USER GROUP MEETUP #49

Directed Acyclic Graphs

How to view raw DAGs from logs
• Tez Application logs contain .dot files in Graphviz format
• To generate images: dot –Tpng –o dag.png dag.dot
• OR javascript version: http://people.apache.org/~gopalv/dagviz/

TEZ-8 JIRA & branch
• TEZ UI for progress tracking and history
• https://issues.apache.org/jira/browse/TEZ-8
• https://github.com/apache/tez/tree/TEZ-8
• UI-centric branch

Tez-UI: Landing page

Tez UI: DAG view

Tez UI: Vertex view

Tez UI: Vertex -> Tasks view

Tez UI: Task logs
Task logs

Tez UI: Task counters
Task counters

Tez UI: Task counters
Search for
counters

Tez UI: Per-edge shuffle counters
Map 3 to Map 1 only

Tez UI: Payload view

Tez UI: Failed DAGs (diagnostic)

Tez UI: Failed tasks indication
Failed tasks

Tez UI: Failed tasks

Tez UI: Failed attempts

Post-hoc/Ad-hoc analysis helpers
• tez/tez-tools ships with two helper tools
• swimlanes
• tez-tfile-parser

Swimlanes
• ./yarn-swimlanes.sh application_1415860665053_0098

TFile parser
• Tez logs can be parsed via PIG
• Allows us to treat our logs exactly like we treat our big-data
• Processing using “pig –x tez” + UDFs [1]
rawLogs = load ‘/app-logs/root/logs/application_1409012059361_0539/*' using
org.apache.tez.tools.TFileLoader() as (machine:chararray, key:chararray, line:chararray);
[1] - https://github.com/rajeshbalamohan/tez_log_parser/blob/master/src/main/resources/pig/udf.groovy

TFile parser (contd)
• Parsing INFO logs for shuffle for instance (for time taken + machine)
Problematic machine

TFile parser (node/rack traffic at 350 nodes)
Problematic machine
Fetcher in node-100 is always slow
(irrespective of where its pulling data from)
Other faulty nodes
Mapout served from node-100 to node-120
To any node is always slow

Questions?
• Thanks all tez contributors for their efforts!
• FYI, Hadoop Summit 2015 (Europe) Call for papers is out

November 2014 HUG: Apache Tez - A Performance View into Large Scale Data-processing, Shuffle Throughput, Reducer parallelism and Reducer Skew

More Related Content

What's hot

Viewers also liked

Similar to November 2014 HUG: Apache Tez - A Performance View into Large Scale Data-processing, Shuffle Throughput, Reducer parallelism and Reducer Skew

More from Yahoo Developer Network

Recently uploaded

November 2014 HUG: Apache Tez - A Performance View into Large Scale Data-processing, Shuffle Throughput, Reducer parallelism and Reducer Skew