This document provides instructions for installing Hadoop on a cluster. It outlines prerequisites like having multiple Linux machines with Java installed and SSH configured. The steps include downloading and unpacking Hadoop, configuring environment variables and configuration files, formatting the namenode, starting HDFS and Yarn processes, and running a sample MapReduce job to test the installation.

1. Hadoop Installation
Xuhong Zhang, Jiangling Yin
Advisor: Dr. Jun Wang

2. Hadoop Installation
• Documentation
Goes to http://hadoop.apache.org/

3. Documentation

4. Install Hadoop in a Cluster
UCF CASS:
http://cass.eecs.ucf.edu/ganglia/?p=2&c=CASS

5. Prerequisites
• Several machines
• Linux for production platform (Linux centos in
this example)
• Java installed (Version 6 or later)
• SSH installed

6. Install Hadoop
• Two steps:
1. Download Hadoop from release page
http://hadoop.apache.org/releases.html#Download
In this example, we use Hadoop 2.2.0
1. Config configuration files

7. Download and unpack Hadoop
• Unpackage
$ tar hadoop-2.2.0.tar.gz
• ‘cd’ to directory hadoop
$ cd hadoop-2.2.0/
• Inside hadoop-2.2.0 directory

8. Configuration (1)
• All configuration files are under hadoop-
2.2.0/etc/hadoop directory:

9. Configuration (2) –
Environment variables
• Java
Set JAVA_HOME to the location of your jdk
for example:
$export JAVA_HOME=/home/ji453898/jan/jdk1.7.0_03
• Hadoop
Set HADOOP_HOME to the location of your hadoop
folder
export HADOOP_HOME=/home/xzhang/hadoop-2.2.0
export PATH=$PATH:$HADOOP_HOME/bin

10. Configuration (3) –
core-site.xml

11. Configuration (4) –
hdfs-site.xml

12. Configuration (4) –
hdfs-site.xml

13. Configuration (5) –
mapred-site.xml

14. Configuration (5) –
mapred-site.xml

15. Configuration (6) –
yarn-site.xml

16. Configuration (7) –
slaves
A list of machines (one per line) that each run
a datanode and a tasktracker.

17. Configuration-SSH passwordless
login
SSH passwordless login from master to slaves
• Generate SSH key pairs(public and private)

18. Configuration-SSH passwordless
login
• Append public key into authorized_keys
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
• Copy id_rsa.pub file into all slave’s ~/.ssh/ folder

19. Run Hadoop
• Format Name node
• Start dfs(HDFS)
• Start Yarn(resourcemanager,nodemanager)
• Check if success

20. Format Namenode
• Inside bin directory
$./hadoop namenode –format

21. Start DFS and Yarn
• Start hadoop

22. Check
• On namenode
• On datanode

23. Run a sample MapReduce (1)
• Upload one file into dfs:
Link to hadoop file system shell documentation
http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/FileSystemShell.html

24. Run a sample MapReduce (2)
• Run a MapReduce job:
$ hadoop jar hadoop-mapreduce-examples-2.4.2-SNAPSHOT.jar wordcount
/wordcount/input /wordcount/output
……./hadoop/share/hadoop/mapreduce/

25. compiling
• compile WordCount.java
$ javac -classpath hadoop-core-0.20.203.0.jar -d
wordcount WordCount.java
• create a jar
$jar -cvf ./word.jar -C wordcount .
• Look up the clasess:
$ jar tf word.jar