The document summarizes the process Yahoo! undertook to upgrade its Hadoop clusters from version 0.20 to 0.23. It involved providing a 420-node sandbox cluster running 0.23 for customer testing, configuring queues and memory settings, and adding Kerberos authentication. Init scripts were modified for the DataNode, NameNode, SecondaryNameNode, HistoryServer, ResourceManager and NodeManager to start the new Hadoop versions.

1. Status of Hadoop 0.23
Operations at Yahoo!
From Inception to Customer Validation
Charles Wimmer, Staff Site Reliability
Engineer at LinkedIn

2. Summary of This Talk
● Includes
○ Operational changes required to support 0.23
● Does not include
○ Specifics about customer testing
○ Deployment into Research or Production clusters

3. Scope of This Change at Yahoo!
● 42,000+ Hadoop servers
● 20+ clusters
● Three tiers: Sandbox, Research, Production
● 0.20.205.x

4. Overview of the Process
● Provide customers a 0.23 Sandbox cluster
● Provide customers enough data to test their
applications
● Provide developer support to address
application issues quickly
● Upgrade Research and Production clusters
as applications are certified to work with 0.23

5. Test Cluster
● 420 Nodes
● 2 x Westmere 4 core processors
● 24G RAM
● 12 x 2T Disks
● No Federation

6. Configuration
● Hierarchical Queues
● Memory Configuration
● Kerberos

7. Hierarchical Queues
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>BIZUNIT-A,BIZUNIT-U,BIZUNIT-C,unfunded</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>

8. Hierarchical Queues
<property>
<name>yarn.scheduler.capacity.root.BIZUNIT-A.capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.BIZUNIT-U.capacity</name>
<value>30</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.BIZUNIT-C.capacity</name>
<value>15</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.unfunded.capacity</name>
<value>5</value>
</property>

9. Hierarchical Queues
<property>
<name>yarn.scheduler.capacity.root.BIZUNIT-U.proj-a.
capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.BIZUNIT-U.proj-b.
capacity</name>
<value>50</value>
</property>

10. Hierarchical Queues

11. Memory Configuration
<property> <name>yarn.
nodemanager.resource.memory-
mb</name>
<value>21504</value>
</property>

12. Kerberos Configuration
<property>
<name>yarn.resourcemanager.principal</name>
<value>mapred/clustername-jt1.domain.name.com@REALM.
NAME.COM</value>
</property>
<property>
<name>yarn.nodemanager.principal</name>
<value>tt/_HOST@REALM.NAME.COM</value>
</property>

13. Init Scripts
● DataNode/NameNode
● SecondaryNameNode
● HistoryServer
● NodeManager
● ResourceManager

14. DataNode/NameNode
start_20(){
. . .
}
start_next(){
. . .
}
if [ -x /home/gs/hadoop/current/bin/hdfs ] ; then
start_next $@
else
start_20 $@
fi

15. SecondaryNameNode
function clean_checkpoint_dir {
CHECKPOINT_DIR=/grid/0/tmp/hadoop-
hdfs/dfs/namesecondary/current
if [ -d "$CHECKPOINT_DIR" ] ; then
DELETE_DIR=`mktemp -p /grid/0/tmp -d delete-XXXXXX`
if [ $? -eq 0 ] ; then
echo "moving $CHECKPOINT_DIR to ${DELETE_DIR}/ "
mv $CHECKPOINT_DIR ${DELETE_DIR}/
cat<<EOF | at now+1min 2>/dev/null
if [ -d $DELETE_DIR ] ; then
rm -rf --preserve-root $DELETE_DIR
fi
EOF
fi
fi
}

16. HistoryServer
case "$1" in
start)
su $HADOOP_USER -s /bin/sh -c
"$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh
--config $HADOOP_CONF_DIR start
historyserver"
RET=$?
;;

17. ResourceManager/NodeManager
case "$1" in
start)
su $HADOOP_USER -s /bin/sh -c
"$HADOOP_PREFIX/sbin/yarn-daemon.sh --config
$HADOOP_CONF_DIR start $PROC"
RET=$?
;;

18. Questions?
Charles Wimmer
@cwimmer
charles@wimmer.net
cwimmer@linkedin.com