The job throughput and Apache Hadoop cluster utilization benefits of YARN and MapReduce v2 are widely known. Who wouldn’t want job throughput increased by 2x? Most likely you’ve heard (repeatedly) about the key benefits that could be gained from migrating your Hadoop cluster from MapReduce v1 to YARN: namely around improved job throughput and cluster utilization, as well as around permitting different computational frameworks to run on Hadoop. What you probably haven’t heard about are the configuration tweaks needed to ensure your existing MR v1 jobs can run on your YARN cluster as well as YARN specific configuration settings. In this session we’ll start with a list of recommended YARN configurations, and then step through the most common use-cases we’ve seen in the field. Production migrations can quickly go awry without proper guidance. Learn from others’ misconfigurations to get your YARN cluster configured right the first time.

1. Yarns about YARN
Migrating to MapReduce v2
Kathleen Ting, kate@cloudera.com
Strata Hadoop Barcelona, 21 November 2014

2. © 2014 Cloudera, Inc. All rights reserved. 2
$ whoami
Kathleen Ting
• Joined Cloudera in 2011
• Former customer operations
engineer
• Technical account manager
• Apache Sqoop Cookbook co-author

3. Cloudera and Apache Hadoop
• Apache Hadoop is an open source software framework for distributed
storage and distributed processing of Big Data on clusters of
commodity hardware.
• Cloudera is revolutionizing enterprise data management by offering the
first unified Platform for Big Data, an enterprise data hub built on
Apache Hadoop.
• Distributes CDH, a distribution Hadoop Distribution.
• Teaches, consults, and supports customers building applications on the Hadoop
stack.
• The world-wide Cloudera Customer Operations Engineering team has closed tens
of thousands of support incidents over six years.
© 2014 Cloudera, Inc. All rights reserved. 3

4. © 2014 Cloudera, Inc. All rights reserved. 4
Outline
• MR2 motivation
• Upgrading MR1 to MR2
• YARN upgrade pitfalls
• YARN applications

5. © 2014 Cloudera and/or its affiliates. All rights reserved. 5
MR2 motivation

6. Search: Solr
Hadoop
Daemons
© 2014 Cloudera, Inc. All rights reserved. 6
Hadoop
Languages/APIs: Hive, Pig, Crunch, Kite,
Machine
Hadoop
Daemons
JVM
Linux
Disk, CPU, Mem
Mahout
Resource Management: YARN
Machine
Hadoop
Daemons
JVM
Linux
Disk, CPU, Mem
Machine
Hadoop
Daemons
JVM
Linux
Disk, CPU, Mem
Machine
JVM
Linux
Disk, CPU, Mem
In mem processing:
Spark
Interactive SQL:
Batch processing Impala
MapReduce
Coordination:
ZooKeeper;
Security: Sentry
Event ingest:
Flume, Kafka
DB Import
Export: Sqoop
User interface: HUE
Other
systems:
httpd, sas,
custom
apps
etc
Random Access Storage: HBase
File Storage: HDFS

7. © 2014 Cloudera, Inc. All rights reserved. 7

8. © 2014 Cloudera, Inc. All rights reserved. 8
MapReduce v1
Job Tracker
Task
Tracker
Task
Tracker
Task
Tracker
Map
Map
Reduce
Map
Map
Reduce
Client
Client
Karthik Kambatla, http://www.meetup.com/LA-HUG/events/184628072/

9. © 2014 Cloudera, Inc. All rights reserved. 9
MapReduce v2
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/

10. © 2014 Cloudera, Inc. All rights reserved. 10
YARN motivation
MR1 MR2
Scalability JobTracker tracks all jobs, tasks
Max out at 4k nodes , 40k tasks
Split up tracking between
ResourceManager, ApplicationMaster
Scale up to 10k nodes, 100k tasks
Availability JT HA RM HA & for per-application basis
Utilization Fixed size slots for map, reduce Allocate only as many resources as needed,
allows cluster utilization > 70%
Multi-tenancy N/A Cluster resource management system
Data locality & lowered operational costs
from sharing resources between frameworks

11. Upgrading MR1 to MR2
© 2014 Cloudera and/or its affiliates. All rights reserved. 11

12. © 2014 Cloudera, Inc. All rights reserved. 12
MR1 to MR2 functionality mapping
• Completely revamped architecture in MR2 on YARN
• While most translate, some configurations don’t

13. © 2014 Cloudera, Inc. All rights reserved. 13
MR1 to MR2 functionality mapping
MR1 MR2
TaskTracker NodeManager (launch containers, slave)
JobTracker JobHistoryServer (store completed job, http://rmhost:19888/jobhistory)
ResourceManager (scheduler, master)
ApplicationMaster (manage MR job)
Slots Containers (fungible)

14. © 2014 Cloudera, Inc. All rights reserved. 14
MR1 to MR2 memory allocation mapping
MR1 MR2
Memory
allocations
mapred.child.java.opts
mapred.child.ulimit
3 mappers :: 2 reducers ratio
Memory slots =
yarn.nodemanager.resource.memory-mb ÷
mapreduce.[map|reduce].memory.mb
CPU slots =
yarn.nodemanager.resource.cpu-vcores ÷
mapreduce.[map|reduce].cpu.vcores
Set yarn.nodemanager.resource.cpu-vcores =
# of physical cores
(prevents swapping or OOM)

15. © 2014 Cloudera, Inc. All rights reserved. 15
MR1 to MR2 task mapping
MR1 MR2
Tasks
launched
per node
mapred.map.tasks.maximum
mapred.reduce.tasks.maximum
total tasks < total # of cores
Note this makes for an
underutilized cluster
minimum (memory slots,
CPU slots)
Note no distinguishing
between map and reduce
tasks

16. © 2014 Cloudera, Inc. All rights reserved. 16
YARN compatibility
Migration path Binary
support?
MR1 (CDH4) to MR2 (CDH5) ✔
MR1 (CDH4) to MR1 (CDH5) ✔
MR2 (CDH4) to MR1/MR2 (CDH5) ✖
• Migrating to MR2 on YARN
• For Operators:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-
2-on-yarn-for-operators/
• For Users:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-
2-on-yarn-for-users/
• http://blog.cloudera.com/blog/2014/04/apache-hadoop-
yarn-avoiding-6-time-consuming-gotchas/
• Getting MR2 Up to Speed
• http://blog.cloudera.com/blog/2014/02/getting-mapreduce-
2-up-to-speed/
• Avoiding YARN Gotchas
• http://blog.cloudera.com/blog/2014/04/apache-hadoop-
yarn-avoiding-6-time-consuming-gotchas/
CDH has complete binary compatibility and
source compatibility for almost all programs.

17. © 2014 Cloudera and/or its affiliates. All rights reserved. 17
YARN upgrade pitfalls

18. © 2014 Cloudera, Inc. All rights reserved. 18
The Law of Cluster Inertia
A cluster in a good state stays in a good state,
and
a cluster in a bad state stays in a bad state,
unless
acted upon by an external force.

19. © 2014 Cloudera, Inc. All rights reserved. 19

20. © 2014 Cloudera, Inc. All rights reserved. 20
An upgrade is an external force
Upgrades
• Linux
• Hadoop
• Java
Misconfiguration
• Memory Mismanagement
• Thread Mismanagement
• Disk Mismanagement

21. Tips on mitigating upgrades (from Eric Sammer’s Hadoop Operations)
Order matters KISS Check units
© 2014 Cloudera, Inc. All rights reserved. 21
Understand the precedence of
configuration value overrides.
Get the cluster working with the
minimal viable set before you
add on security and
performance parameters.
Double-check the unit of each
parameter you set.

22. Tips on mitigating upgrades (from Eric Sammer’s Hadoop Operations)
Audit Cloudera Manager Hello World
© 2014 Cloudera, Inc. All rights reserved. 22
Frequently confirm whether
parameters changed in meaning
or scope.
Use a configuration
management and deployment
system to ensure that all files are
up-to-date on all hosts.
Before and after making
configuration changes, run
benchmarks to evaluate the
performance and resource impact.

23. © 2014 Cloudera, Inc. All rights reserved. 23
Frozen jobs after upgrade to YARN
Symptom:
After upgrading to
YARN, applications
were waiting 7+
hours to launch

24. © 2014 Cloudera, Inc. All rights reserved. 24
Frozen jobs after upgrade to YARN
Evidence:
• Initially thought it was a GC
issue
• RM had frozen and without
RM, no tasks or containers
can be launched, rendering
cluster useless

25. © 2014 Cloudera, Inc. All rights reserved. 25
Frozen jobs after upgrade to YARN
Solution:
• Fixed by YARN-713 (CDH5.1.1/
Hadoop 2.4.0), YARN-2313
(CDH5.2/Hadoop 2.6.0),
YARN-2359 (CDH5.1.1/Hadoop
2.6.0)
• Continuous scheduling reduces the
allocation time of containers when there is
capacity available
Workaround:
• Known JDK7 issue (sorting algorithm)
• -Djava.util.Arrays.useLegacyMergeSort=true
• Disable Fair Scheduler Continuous
Scheduling (reduces RM load by
coupling container scheduling
decisions with NM heartbeats)
• Configure RM HA to recover apps
running on the cluster when RM died

26. © 2014 Cloudera, Inc. All rights reserved. 26
Killing of tasks after upgrade to YARN
Symptom:
After upgrading to
YARN, tasks were
being killed

27. © 2014 Cloudera, Inc. All rights reserved. 27
Killing of tasks after upgrade to YARN
Evidence:
Even with sufficient physical
memory available, RHEL6
bug was causing JVM to take
up to 100x as much virtual
memory as physical memory
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-
avoiding-6-time-consuming-gotchas/
Application
application_1392853856445_0900
failed 2 times due to AM
Container for
appattempt_1392853856445_0900_
000002 exited with exitCode:
143 due to: Container
[pid=27408,containerID=contain
er_1392853856445_0900_02_00000
1] is running beyond virtual
memory limits.
Current usage: 337.6 MB of 1
GB physical memory used; 2.2
GB of 2.1 GB virtual memory
used. Killing container.

28. © 2014 Cloudera, Inc. All rights reserved. 28
Killing of tasks after upgrade to YARN
Solution:
• Turn off virtual memory limits as
YARN can kill tasks perceived to
even virtually take up too much
memory
• Set yarn.nodemanager.vmem-check-
enabled = false (fixed in
CDH5.0.0)

29. © 2014 Cloudera and/or its affiliates. All rights reserved. 29
YARN applications

30. © 2014 Cloudera, Inc. All rights reserved. 30
Configuring YARN for Spark
• Designed for interactive queries and
iterative algorithms
• In-memory caching, DAG engine, and APIs
• Set yarn.scheduler.maximum-allocation-mb
as high as 64G on a machine with 192GB of
memory
• Won’t run with small (< 1 GB) containers
because starts fixed number of executors
on the cluster
http://blog.cloudera.com/blog/2014/05/apache-spark-
resource-management-and-yarn-app-models/

31. © 2014 Cloudera, Inc. All rights reserved. 31
YARN applications
• Llama (Low Latency Application MAster)
• Reserves memory in YARN for short-lived processes (e.g. Impala)
• Registers one long-lived AM per YARN pool
• Caches resources allocated by YARN for a short time, so that they can be quickly re-allocated
to Impala queries
• Long-term solution is to run Impala on YARN but currently recommend setting up admission
control

32. © 2014 Cloudera, Inc. All rights reserved. 32
YARN applications
• Apache Slider (incubating) née Hoya
• Runs long-lived persistent services on YARN (e.g. HBase)
• Not currently recommended as it doesn’t provide IO isolation

33. © 2014 Cloudera and/or its affiliates. All rights reserved. 33
Conclusion

34. © 2014 Cloudera, Inc. All rights reserved. 34
YARN performance
• Improved cluster utilization
• Can run more jobs in smaller clusters
• Run in uber mode for smaller jobs (reduces AM
overhead)
• Dynamic resource sharing between
frameworks
• One framework can use the entire cluster
• Tom White’s Hadoop: The Definitive Guide
4th Ed (to be released)
• Chapter 4 is on YARN

35. YARN vs Mesos: Resource Manager’s role
© 2014 Cloudera, Inc. All rights reserved. 35
YARN Mesos
Asks for resources Offers resources
Evolved into a resource manager Evolved into managing Hadoop
Written in Java Written in C++
Locality aware More customizable

36. Cloudera Live Cloudera Community In Europe!
© 2014 Cloudera, Inc. All rights reserved. 36
Shameless plug
A full demo cluster + tutorial and sample
data/queries in the cloud – free access
for two weeks (plus, clusters in Tableau
and Zoomdata flavors!)
cloudera.com/live
Ask technical questions/get
technical answers from Cloudera
Software Engineers and other
users
cloudera.com/community
Account Executives, Solutions
Architects, Sales Engineers,
Trainers
cloudera.com/careers

37. @kate_ting
Office hours
@Table C in 10 min