HADOOP 실제 구성 사례, Multi-Node 구성
•Download as PPTX, PDF•
1 like•2,498 views
Namenode 1개, 3개의 데이터노드로 HADOOP을 구성한 사례입니다. 데이터 노드는 같은 구성으로 계속 늘려가면 됩니다.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to HADOOP 실제 구성 사례, Multi-Node 구성
Similar to HADOOP 실제 구성 사례, Multi-Node 구성 (20)
Recently uploaded
Recently uploaded (20)
HADOOP 실제 구성 사례, Multi-Node 구성
- 1. 구성도 Master 192.168.1.11 node1 LG xNote Slave1 Slave2 Slave3 192.168.1.11 node1 LG xNote 192.168.1.12 node2 LG Desktop 192.168.1.13 node3 IBM x356 2014-12-03 1
- 2. Master User: root 설치할 Software •JDK 1.8 •Hadoop 1.2.1 Software 설치 Directory •JDK 1.8: /usr/java/jdk •Hadoop: /home/hadoop java 설치 https://java.com/ko/download/ie_manual.jsp?locale=ko Hadoop 설치 •http://mirror.apache-kr.org/hadoop/common/stable/ 에서 hadoop- 1.2.1.tar.gz download •tar zxvf hadoop-1.2.1.tar.gz 2014-12-03 2
- 3. Master 환경변수 설정 •vi .bash_profile export JAVA_HOME=/usr/java/jdk export HADOOP_HOME=/home/hadoop export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH •source .bash_profile /home/hadoop/conf/hadoop-env.sh export JAVA_HOME=/usr/java/jdk export HADOOP_HOME=/home/hadoop export HADOOP_HOME_WARN_SUPPRESS=TRUE export HADOOP_LOG_DIR=${HADOOP_HOME}/logs export HADOOP_SLAVES=${HADOOP_HOME/conf/slaves export HADOOP_PID_DIR=${HADOOP_HOME}/pids /home/hadoop/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://node1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hadoop-${user.name}</value> </property> 2014-12-03 3
- 4. Master /home/hadoop/conf/hdfs-site.xml <property> <name>dfs.name.dir</name> <value>/home/hadoop/dfs/name</value> </property> <property> <name>dfs.name.edits.dir</name> <value>${dfs.name.dir}</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/dfs/data</value> </property> 2014-12-03 4
- 5. Master /home/hadoop/conf/mapred-site.xml <property> <name>mapred.job.tracker</name> <value>node1:9001</value> </property> <property> <name>mapred.local.dir</name> <value>${hadoop.tmp.dir}/mapred/local</value> </property> <property> <name>mapred.system.dir</name> <value>${hadoop.tmp.dir}/mapred/system</value> </property> /home/hadoop/conf/masters node1 /home/hadoop/conf/slaves node1 node2 node3 2014-12-03 5
- 6. Master Master Node SSH Key 생성 및 복사 ssh-keygen –t dsa –P ‘’ –f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys scp ~/.ssh/id_dsa.pub node2:/root/.ssh/authorized_keys scp ~/.ssh/id_dsa.pub node3:/root/.ssh/authorized_keys scp ~/.ssh/id_dsa.pub node4:/root/.ssh/authorized_keys scp ~/.ssh/id_dsa.pub node5:/root/.ssh/authorized_keys Namenode format hadoop namenode –format *Datanode 구동시 ID관련한 에러가 발생하면 /home/hadoop/dfs/data 안의 내용을 삭제 Hadoop 구동 bin/start-all.sh bin/start-mapred.sh bin/start-dfs.sh bin/stop-all.sh bin/stop-mapred.sh bin/stop-dfs.sh 2014-12-03 6
- 7. Master jps master node jobtracker / namenode / datanode slave nodes tasktracker / datanode UI 확인 (Slave까지 설치한 후에) http://node1:50030 MapReduce http://node1:50070 HDFS Word Count 실행 bin/hadoop dfs –mkdir input bin/hadoop dfs –put CHANGE.txt input/ bin/hadop jar /home/hadoop/hadoop-examples-1.2.1.jar wordcount input output 2014-12-03 7
- 8. 구성도 Master 192.168.1.11 node1 LG xNote Slave1 Slave2 Slave3 192.168.1.11 node1 LG xNote 192.168.1.12 node2 LG Desktop 192.168.1.13 node3 IBM x356 2014-12-03 8
- 9. Slaves User: root 설치할 Software •JDK 1.8 •Hadoop 1.2.1 Software 설치 Directory •JDK 1.8: /usr/java/jdk •Hadoop: /home/hadoop java 설치 https://java.com/ko/download/ie_manual.jsp?locale=ko Hadoop 설치 •http://mirror.apache-kr.org/hadoop/common/stable/ 에서 hadoop- 1.2.1.tar.gz download •tar zxvf hadoop-1.2.1.tar.gz 2014-12-03 9
- 10. Slaves 환경변수 설정 •vi .bash_profile export JAVA_HOME=/usr/java/jdk export HADOOP_HOME=/home/hadoop export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH •source .bash_profile /home/hadoop/conf/hadoop-env.sh export JAVA_HOME=/usr/java/jdk export HADOOP_HOME=/home/hadoop export HADOOP_HOME_WARN_SUPPRESS=TRUE export HADOOP_LOG_DIR=${HADOOP_HOME}/logs export HADOOP_SLAVES=${HADOOP_HOME/conf/slaves export HADOOP_PID_DIR=${HADOOP_HOME}/pids /home/hadoop/conf/core-site.xml <property> <name>fs.default.name</name> <value>hdfs://node1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hadoop-${user.name}</value> </property> 2014-12-03 10
- 11. Slaves /home/hadoop/conf/hdfs-site.xml <property> <name>dfs.name.dir</name> <value>/home/hadoop/dfs/name</value> </property> <property> <name>dfs.name.edits.dir</name> <value>${dfs.name.dir}</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/dfs/data</value> </property> 2014-12-03 11
- 12. Slaves /home/hadoop/conf/mapred-site.xml <property> <name>mapred.job.tracker</name> <value>node1:9001</value> </property> <property> <name>mapred.local.dir</name> <value>${hadoop.tmp.dir}/mapred/local</value> </property> <property> <name>mapred.system.dir</name> <value>${hadoop.tmp.dir}/mapred/system</value> </property> /home/hadoop/conf/masters node1 /home/hadoop/conf/slaves node1 node2 node3 2014-12-03 12
- 13. Slaves jps master node jobtracker / namenode / datanode slave nodes tasktracker / datanode 2014-12-03 13
- 14. File Handling Commands hadoop fs –cmd hadoop fs –ls hadoop fs –ls / hadop fs –mkdir /user/chuck hadoop fs –lsr / hadoop fs –put example.txt hadoop fs –get example.txt hadoop fs –cat example.txt hadoop fs –tail example.txt hadoop fs –rm example.txt /home/hadoop Directory에서 hadoop fs –mkdir /input hadoop fs –put ivy.xml input/ hadoop jar /home/hadoop/hadoop-examples-1.2.1.jar wordcount input output 2014-12-03 14
- 15. Node 삭제/추가 노드 삭제 •제외할 노드 파일 리스트 생성 •Namenode가 참조하는 설정 “dfs.hosts.exclude”에 해당 항목을 지정 •bin/hadoop dfsadmin –refreshNodes 새로운 노드 등록 •신규노드에 hadoop 설치, 구성 •bin/hadoop/datanode 실행 •bin/start-balancer.sh •bin/stop-balancer.sh 2014-12-03 15
- 16. System Healthcheck bin/hadoop fsck / bin/hadoop fsck /user/hadoop/test –files –blocks –location –racks bin/hadoop dfsadmin –report bin/hadoop –metasave <filename> Exception in CreateBlockOutputStream error service iptables save service iptables stop chkconfig iptables off 2014-12-03 16