This document summarizes Ryu Kobayashi's presentation on HDP2 and YARN operations. The presentation introduced YARN, the resource management framework in Hadoop 2.0, describing its architecture and how it differs from the previous MapReduce v1 framework. It highlighted important considerations for YARN resource management and potential bugs in older versions of Hadoop.

1. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
HDP2
and
YARN operations point
Ryu Kobayashi
Treasure Data Tech Talk
11 and 12 Mar 2015

2. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Who am I?
• Ryu Kobayashi
• @ryu_kobayashi
• https://github.com/ryukobayashi
• Treasure Data, Inc.
• Software Engineer
• Background
• Hadoop, Cassandra, Machine Learning, ...
• I developed Huahin(Hadoop) Framework.
http://huahinframework.org/

3. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
What is YARN?

4. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
YARN(Yet Another Resource Negotiator)
Architecture

5. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• MRv1
• JobTracker
• TaskTracker

6. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• YARN
• ResourceManager
• NodeManager
• ApplicationMaster
• Job History Server
• YARN Timeline Server

7. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• MRv1
• JobTracker
• TaskTracker
• YARN
• ResourceManager
• NodeManager
• ApplicationMaster
• Job History Server
(We
can
not
see
the
log
job
history
If
it
do
not
install)
• YARN Timeline Server
(We
can
not
see
the
log
YARN
history
If
it
do
not
install)

8. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
YARN Timeline Server
• It is included container info

9. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Note!!!

10. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Use the Hadoop 2.4.0
and later!!!

11. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• The versions which must not be used
• Apache Hadoop 2.2.0
• Apache Hadoop 2.3.0
• HDP 2.0(2.2.0 based)

12. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• Currently
• Apache Hadoop 2.6.0
• CDH 5.3.2(2.5.0 based and patch)
• HDP 2.2(2.6.0 based and patch)

13. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• Why should not use?
• Capacity Scheduler
• There is a bug
• Fair Scheduler
• There is a bug

14. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
• Any bugs?
• Each Scheduler will cause
a deadlock

15. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
•In fact, there is a bug in 2.4.0 and 2.6.0…
•It is better to use the new version.
•Note: 2.7.0 and later is a different thing

16. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Backport Patch
• I was backport some patch
• https://github.com/ryukobayashi/patches

17. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Backport Patch
• Included dead lock patch
• Format of the counter
• Application kill in Web UI.

18. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Format of the counter

19. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Format of the counter

20. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Application kill in Web UI

21. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Application kill in Web UI

22. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Application kill in Web UI
• Job kill in Web UI
•
(default false)
• Application kill in Web UI
•
(default true)
mapreduce.jobtracker.webinterface.trusted
yarn.resourcemanager.webapp.ui-actions.enabled

23. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Backport Patch
• We want the next…
• Job task attempt kill in Web UI patch
(in development)
• Currently, only command line
$ mapred job -kill-task attempt_*

24. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 2

25. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 2
Cluster
Application
App
Master
Container
Application
App
Master
Container

26. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4

27. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
Cluster
Application
App
Master
Application
App
Master
Application
App
Master
Application
App
Master

28. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
Cluster
Application
App
Master
Application
App
Master
Application
App
Master
Application
App
Master

29. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
Cluster
Application
App
Master
Application
App
Master
Application
App
Master
Application
App
Master
Livelock!

30. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
Cluster
Application
App
Master
Application
App
Master
Application
App
Master
Container
Application
App
Master
Kill

31. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
• ^ squeeze the number of applications

32. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
• ^ squeeze the number of applications
• set the root maxRunningApps

33. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Matter of resources
• total container = 4
• concurrent application = 4
• root maxRunningApps = 2
Cluster
Application
App
Master Container
Application
App
Master
Application
App
Master Container
Application
App
Master
Pending

34. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
YANR Resource Management
yarn-site.xml
• yarn.nodemanager.resource.memory-mb
• (yarn.nodenamager.vmem-pmem-ratio)
• (yarn.scheduler.minimum-allocation-mb)
mapred-site.xml
• yarn.app.mapreduce.am.resource.mb
• mapreduce.map.memory.mb
• mapreduce.reduce.memory.mb
fair-scheduler.xml
• maxResources, minResources
etc…

35. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
e.g.
Use hdp-configuration-utils.py script
http://goo.gl/L2hxyq
Use Ambari
http://ambari.apache.org/
See the Cloudera’s document
http://goo.gl/EBreca
YANR Resource Management

36. Copyright
©2015
Treasure
Data.
All
Rights
Reserved.
Thanks!!!