Why is my Hadoop* job slow?
•Download as PPTX, PDF•
1 like•1,773 views
This document discusses various tools and techniques for diagnosing slow Hadoop jobs, including metrics and monitoring, logging and correlation, and tracing and analysis. It describes how to use Ambari metrics and Grafana dashboards to monitor cluster health and performance. It also explains how to leverage Hadoop audit logs and caller context to correlate job activity. Techniques for application tracing using YARN timeline service and tools like Zeppelin and Tez analyzer are presented to enable deep performance analysis of Hadoop jobs.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Why is my Hadoop* job slow?
Similar to Why is my Hadoop* job slow? (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Why is my Hadoop* job slow?
- 1. Why is my Hadoop* job slow? Rajesh Balamohan @rajeshbalamohan *Apache Hadoop, Falcon, Atlas, Tez, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie, Zeppelin and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Metrics and Monitoring Logging and Correlation Tracing and Analysis
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metrics and Monitoring Metrics as high level pointers Ambari Metrics System Ambari Grafana Integration HBase, HDFS, YARN Dashboards Metrics based alerting
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metrics as high level pointers Machine level metrics like CPU load Application level metrics like HDFS counters Metrics at point of time Metrics anomalies along a time series Correlated anomalies Problem is to need to know what to look for
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Metrics Service - Motivation Limited Ganglia capabilities OpenTSDB – GPL license and needs a Hadoop cluster Need service level aggregation as well as time based Alerts based on metrics system Ability to scale past a 1000 nodes Ability to perform analytics based on a use case Allow fine grained control over aspects like: retention, collection intervals, aggregation Pluggable and Extensible First version released with Ambari 2.0.0
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Grafana Integration Open source dashboard builder integrated with AMS. Available from Ambari-2.2.2 Pre-defined host level and service level (HDFS, HBase, Yarn etc) dashboards. Added to Ambari through API after upgrade
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HBase Dashboard
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Dashboard
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN Dashboard
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metrics based Alerting Top N support to quickly identify potential offenders Alerting based on time series
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Metrics and Monitoring Logging and Correlation Tracing and Analysis
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Logging and Correlation HDFS, YARN Audit logs Caller Context YARN Application Timeline Service Lineage tracking of operations across workloads Ambari Log Search
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDFS Audit Logs and Caller Context FSNamesystem.audit: allowed=true ugi=userA (auth:SIMPLE) ip=/172.22.68.32 cmd=create src=/tmp/in/_temporary/1/_temporary/attempt_14644848874070_0009_m_009995_0/part-m-09995 dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=tez_ta:attempt_1464484887407_0009_1_00_009995_0 FSNamesystem.audit: allowed=true ugi=userA (auth:SIMPLE) ip=/172.22.68.33 cmd=create src=/tmp/in2/_temporary/1/_temporary/attempt_1464484887407_0011_m_000097_0/part-m-00097 dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=mr_attempt_1464484887407_0011_m_000097_0 FSNamesystem.audit: allowed=true ugi=userB (auth:SIMPLE) ip=/172.22.68.34 cmd=create src=/tmp/in2/_temporary/1/_temporary/attempt_1464484887407_0011_m_000095_0/part-m-00095 dst=null perm=root:hdfs:rw-r--r-- proto=rpc callerContext=mr_attempt_1464484887407_0011_m_000095_0
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved ResourceManager Audit Logs and Caller Context resourcemanager.RMAuditLogger: USER=userA IP=172.22.68.32 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1464484887407_0001 CALLERCONTEXT=PIG-pigSmoke.sh-8a052588-0013-4e39-83b1-ebad699d8e2e resourcemanager.RMAuditLogger: USER=userA IP=172.22.68.30 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1464484887407_0009 CALLERCONTEXT=CLI resourcemanager.RMAuditLogger: USER=userB IP=172.22.68.34 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1464484887407_0008 CALLERCONTEXT=mr_attempt_1464484887407_0007_m_000000_0 resourcemanager.RMAuditLogger: USER=userB IP=172.22.68.30 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1464484887407_0012 CALLERCONTEXT=HIVE_SSN_ID:f3aadf99-9e36-494b-84a1-99b685ac344b
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN Application Timeline Service YARN service for fine grained application level tracing Enables complex metadata to be recorded as the YARN app makes progress Allows retrieval of this timeline data based on filters Can be used to drive limited online analytics and extensive post-hoc analysis
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lineage Tracking using YARN Timeline Timeline:8188/ws/v1/timeline/TEZ_DAG_ID/dag_1464484887407_0013_1 dagContext: { callerId: "root_20160529021115_006f8007-5840-4c64-9970-c1b506f68db2", callerType: "HIVE_QUERY_ID", context: "HIVE", description: "select user, count(visit_id) as visits from users group by user order by visits” } Timeline:8188/ws/v1/timeline/HIVE_QUERY_ID/root_20160529021115_006f8007- 5840-4c64-9970-c1b506f68db2 hiveContext: { callerId: “workflow_abcd", callerType: “OOZIE_ID", context: “OOZIE", description: “Daily ETL Summary Job” }
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Log Search
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari Log Search
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Metrics and Monitoring Logging and Correlation Tracing and Analysis
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing and Analysis Use Big Data methods to solve Big Data problems Apache Zeppelin as analytical tool Hive/Tez/YARN notebook for analysis
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Zeppelin for Ad-hoc Analytics
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved YARN Analyzer
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez Analyzer
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez Analyzer
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez Swimlane View
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez UI to Download Timeline Data
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enable Task Level Debug Logs in Tez Enable debug logs for specific class tez.task.log.level="INFO;org.apache.hadoop.hive.ql.io.orc=DEBUG;” For specific task in specific vertex – hive --hiveconf tez.task-specific.launch.cmd-opts.list="Map 1[0]" --hiveconf tez.task- specific.log.level="INFO;org.apache=DEBUG;” – Adds DEBUG logs for Task 0 in Map 1.
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Swimlanes TEZ-1332
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tez Analyzer
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You
Editor's Notes
- - We will be looking at 3 aspects mainly metrics and monitoring, logging and co-relation and tracing &analysis Metrics are at the bottom level where you have counters and system level metrics gathered at runtime and exposed to high level for processing Logs are mainly at the service level, for example, HDFS, YARN, HIVE, HBASE etc Tracing and Analysis is at the application level where you are running a specific hive job or hadoop job.
- When we say about metrics and monitoring, we have heard saying “there is no smoke without fire” and metrics is something equivalent to the smoke. And that is something which provides info on where the problem could be. Usually metrics do not go wrong unless something is very broken.
- So what do I mean by metrics as high level pointers. Let me start with a simple example of one of the issues we dealt with. <customer story>
- Similarly there is YARN dashboard. YARN is a resource management layer where you arrange a bunch of queues in the cluster which have a certain amount of memory and cpu allocated to it and you run multiple jobs,and it arbitrates the cpu/memory among these jobs.
- HDFS and YARN provides the audit logs to track of the various state changes in the system.
- You can use log search directly query directly .So here we are search for give me top 5 files which are being used.
- You have lots of data distributed across systems about the metrics, audit logs, lineage etc and it is possible to make use big data tools itself to mine these logs.
- You could potentially fetch data from ambari metrics system, from audit logs, from yarn application time serivce etc and start mining it via the notebooks.
- Summarize: There are many kinds of data that are available for debugging. Like counters and system counters which are emitted at the stack level. There are audit log details to provide information about the kind of operation that are happening in the system And Also the application time line service which captures the metadata about the events. They all have ways in which each of them can be co-related to get meaningful information.