The document describes the steps to set up a Hadoop cluster with one master node and three slave nodes. It includes installing Java and Hadoop, configuring environment variables and Hadoop files, generating SSH keys, formatting the namenode, starting services, and running a sample word count job. Additional sections cover adding and removing nodes and performing health checks on the cluster.

1. 구성도
Master
192.168.1.11
node1
LG xNote
Slave1 Slave2 Slave3
192.168.1.11
node1
LG xNote
192.168.1.12
node2
LG Desktop
192.168.1.13
node3
IBM x356
2014-12-03 1

2. Master
User: root
설치할 Software
•JDK 1.8
•Hadoop 1.2.1
Software 설치 Directory
•JDK 1.8: /usr/java/jdk
•Hadoop: /home/hadoop
java 설치
https://java.com/ko/download/ie_manual.jsp?locale=ko
Hadoop 설치
•http://mirror.apache-kr.org/hadoop/common/stable/ 에서 hadoop-
1.2.1.tar.gz download
•tar zxvf hadoop-1.2.1.tar.gz
2014-12-03 2

3. Master
환경변수 설정
•vi .bash_profile
export JAVA_HOME=/usr/java/jdk
export HADOOP_HOME=/home/hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH
•source .bash_profile
/home/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk
export HADOOP_HOME=/home/hadoop
export HADOOP_HOME_WARN_SUPPRESS=TRUE
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
export HADOOP_SLAVES=${HADOOP_HOME/conf/slaves
export HADOOP_PID_DIR=${HADOOP_HOME}/pids
/home/hadoop/conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-${user.name}</value>
</property>
2014-12-03 3

4. Master
/home/hadoop/conf/hdfs-site.xml
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.name.edits.dir</name>
<value>${dfs.name.dir}</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/dfs/data</value>
</property>
2014-12-03 4

5. Master
/home/hadoop/conf/mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>node1:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>${hadoop.tmp.dir}/mapred/system</value>
</property>
/home/hadoop/conf/masters
node1
/home/hadoop/conf/slaves
node1
node2
node3
2014-12-03 5

6. Master
Master Node SSH Key 생성 및 복사
ssh-keygen –t dsa –P ‘’ –f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp ~/.ssh/id_dsa.pub node2:/root/.ssh/authorized_keys
scp ~/.ssh/id_dsa.pub node3:/root/.ssh/authorized_keys
scp ~/.ssh/id_dsa.pub node4:/root/.ssh/authorized_keys
scp ~/.ssh/id_dsa.pub node5:/root/.ssh/authorized_keys
Namenode format
hadoop namenode –format
*Datanode 구동시 ID관련한 에러가 발생하면
/home/hadoop/dfs/data 안의 내용을 삭제
Hadoop 구동
bin/start-all.sh
bin/start-mapred.sh
bin/start-dfs.sh
bin/stop-all.sh
bin/stop-mapred.sh
bin/stop-dfs.sh
2014-12-03 6

7. Master
jps
master node
jobtracker / namenode / datanode
slave nodes
tasktracker / datanode
UI 확인 (Slave까지 설치한 후에)
http://node1:50030 MapReduce
http://node1:50070 HDFS
Word Count 실행
bin/hadoop dfs –mkdir input
bin/hadoop dfs –put CHANGE.txt input/
bin/hadop jar /home/hadoop/hadoop-examples-1.2.1.jar wordcount
input output
2014-12-03 7

8. 구성도
Master
192.168.1.11
node1
LG xNote
Slave1 Slave2 Slave3
192.168.1.11
node1
LG xNote
192.168.1.12
node2
LG Desktop
192.168.1.13
node3
IBM x356
2014-12-03 8

9. Slaves
User: root
설치할 Software
•JDK 1.8
•Hadoop 1.2.1
Software 설치 Directory
•JDK 1.8: /usr/java/jdk
•Hadoop: /home/hadoop
java 설치
https://java.com/ko/download/ie_manual.jsp?locale=ko
Hadoop 설치
•http://mirror.apache-kr.org/hadoop/common/stable/ 에서 hadoop-
1.2.1.tar.gz download
•tar zxvf hadoop-1.2.1.tar.gz
2014-12-03 9

10. Slaves
환경변수 설정
•vi .bash_profile
export JAVA_HOME=/usr/java/jdk
export HADOOP_HOME=/home/hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH
•source .bash_profile
/home/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk
export HADOOP_HOME=/home/hadoop
export HADOOP_HOME_WARN_SUPPRESS=TRUE
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
export HADOOP_SLAVES=${HADOOP_HOME/conf/slaves
export HADOOP_PID_DIR=${HADOOP_HOME}/pids
/home/hadoop/conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://node1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-${user.name}</value>
</property>
2014-12-03 10

11. Slaves
/home/hadoop/conf/hdfs-site.xml
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.name.edits.dir</name>
<value>${dfs.name.dir}</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/dfs/data</value>
</property>
2014-12-03 11

12. Slaves
/home/hadoop/conf/mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>node1:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>${hadoop.tmp.dir}/mapred/system</value>
</property>
/home/hadoop/conf/masters
node1
/home/hadoop/conf/slaves
node1
node2
node3
2014-12-03 12

13. Slaves
jps
master node
jobtracker / namenode / datanode
slave nodes
tasktracker / datanode
2014-12-03 13

14. File Handling Commands
hadoop fs –cmd
hadoop fs –ls
hadoop fs –ls /
hadop fs –mkdir /user/chuck
hadoop fs –lsr /
hadoop fs –put example.txt
hadoop fs –get example.txt
hadoop fs –cat example.txt
hadoop fs –tail example.txt
hadoop fs –rm example.txt
/home/hadoop Directory에서
hadoop fs –mkdir /input
hadoop fs –put ivy.xml input/
hadoop jar /home/hadoop/hadoop-examples-1.2.1.jar wordcount input
output
2014-12-03 14

15. Node 삭제/추가
노드 삭제
•제외할 노드 파일 리스트 생성
•Namenode가 참조하는 설정 “dfs.hosts.exclude”에 해당 항목을 지정
•bin/hadoop dfsadmin –refreshNodes
새로운 노드 등록
•신규노드에 hadoop 설치, 구성
•bin/hadoop/datanode 실행
•bin/start-balancer.sh
•bin/stop-balancer.sh
2014-12-03 15

16. System Healthcheck
bin/hadoop fsck /
bin/hadoop fsck /user/hadoop/test –files –blocks –location –racks
bin/hadoop dfsadmin –report
bin/hadoop –metasave <filename>
Exception in CreateBlockOutputStream error
service iptables save
service iptables stop
chkconfig iptables off
2014-12-03 16