Hadoop 3.1.1 single node

Taiwan Steel Union
Hadoop 3.1.1
Single-node Installation on Ubuntu
康志強
2018/8/23

Hadoop 3.1.1 (Signle-node) Installation on Ubuntu
1
一、前言................................................................................................................................. 2
二、安裝環境................................................................................................................................. 3
三、安裝步驟................................................................................................................................. 4
1. 安裝環境說明................................................................................................................. 4
2. 設定................................................................................................................................. 5
3. Single node 設定.............................................................................................................. 6
4. Cloud001 的 SSH 無密碼登入........................................................................................ 7
5. 安裝 JDK......................................................................................................................... 8
6. Hadoop 3.1.1 安裝......................................................................................................... 10
7. Hadoop 3.1.1 啟動......................................................................................................... 15
五、REFERENCE:....................................................................................................................... 18

2
一、前言
 http://hadoop.apache.org/
 Hadoop Common: The common utilities that support the other Hadoop modules.
 Hadoop Distributed File System (HDFS™): A distributed file system that
provides high-throughput access to application data.
 Hadoop YARN: A framework for job scheduling and cluster resource
management.
 Hadoop MapReduce: A YARN-based system for parallel processing of large data
sets.


3
二、安裝環境
VM Platform AWS t3.large
VM Guest OS ubuntu-16.04
VMRAM 8.0 GB
VM HD EBS

4
三、安裝步驟
1. 安裝環境說明
這裡我們建構一個由一台機器組成的叢集
Hostname User/Password Cluster 角色 OS
ubuntu/ ubuntu Name node
Secondary Name node
Resource manager
ubuntu-16.04 64 bits

5
2. 設定
(1) 修改 hostname , 改成 cloud001
vim /etc/hostname
(2) 修改 hduser 權限 : (跳過)
vim /etc/sudoers
(3) 系统升级到最新
sudo apt-get update
sudo apt-get upgrade
基本上先把 cloud001 裝好,再 clone 成 002,003 後,改 hotname 就可以了

6
3. Single node 設定
ubuntu@ip-172-30-1-102:~$ vim /etc/hosts

7
4. Cloud001 的 SSH 無密碼登入
(1) 安裝 SSH
sudo apt-get install ssh
(2) 產生金鑰(一直 Enter 就可以)
ubuntu@ip-172-30-1-102:~$ ssh-keygen -t rsa
(3) 把 id_rsa.pub 追加到授權的 key 裡面去
ubuntu@ip-172-30-1-102:~$ cat id_rsa.pub >> authorized_keys
(4) 重啟 SSH 服務
ubuntu@ip-172-30-1-102:~$ service ssh restart

8
測試
ssh localhost
5. 安裝 JDK
安裝 Java (OpenJDK 1.8.0) 及設定環境變數
ubuntu@ip-172-30-1-102:~$ sudo apt-get install openjdk-8-jre openjdk-8-jdk

9
環境變數
ubuntu@ip-172-30-1-102:~$ vim ~/.bashrc
最後面增加
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
[ NOTE ] : 上述環境變數設定中的指令 :
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
是為了解決下列出現 "無法使用本地 Java library” 的警告訊息 :
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable
輸入下面的命令來使之生效
ubuntu@ip-172-30-1-102:~$ source ~/.bashrc
測試
ubuntu@ip-172-30-1-102:~$ java -version

10
6. Hadoop 3.1.1 安裝
(1) 下載檔案 hadoop-3.1.1.tar.gz, 解壓到/usr/local 路径下
ubuntu@ip-172-30-1-102:~$wget http://www-
us.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
# 直接解壓縮到 /usr/local
ubuntu@ip-172-30-1-102:~$ sudo tar -zxvf ./hadoop-3.1.1.tar.gz -C /usr/local
ubuntu@ip-172-30-1-102:~$ cd /usr/local
# 將檔案夾名稱改為 hadoop
ubuntu@ip-172-30-1-102:~$ sudo mv ./hadoop-3.1.1/ ./hadoop
(2) hadoop 配置
配置之前，需要在 cloud001 新增以下資料夾
/home/hduser/dfs/name
/home/hduser/dfs/data
/home/hduser/temp
sudo rm -rf /usr/local/hadoop/hadoopdata/hdfs
mkdir -p /usr/local/hadoop/hadoopdata/hdfs/namenode
mkdir -p /usr/local/hadoop/hadoopdata/hdfs/datanode
sudo chown -R ubuntu:ubuntu /usr/local/hadoop

11
修改相關設定擋案內容,清單如下
/usr/local/hadoop/etc/hadoop/core-site.xml
/usr/local/hadoop/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
/usr/local/hadoop/etc/hadoop/yarn-site.xml
/usr/local/hadoop/etc/hadoop/mapred-site.xml
修改 hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
修改 core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://Cloud001:9000</value>
</property>
</configuration>
修改 hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>Cloud001:9001</value>
</property>

12
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value> file:/usr/local/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value> file:/usr/local/hadoop/hadoopdata/hdfs/datanode</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>

13
修改 mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
</property>
<property>
<name>mapreduce.reduce.env</name>
</property>
</configuration>
修改 yarn-site.xml
<configuration>

14
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8192</value>
</property>
</configuration>

15
7. Hadoop 3.1.1 啟動
(1) 格式化 HDFS NameNode：
hadoop namenode -format
(2) 格式化 HDFS DataNode
hadoop datanode -format
(3) 啟動 hdfs
start-dfs.sh
(4) 啟動 YARN
start-yarn.sh
(4) 檢查 Java processes
jps
(5) 查看叢集狀態
hdfs dfsadmin –report

16
(6) 查看文件組成
hdfs fsck / -files –blocks

17
(7) 查看 HDFS
(8) 查看 RM

18
五、REFERENCE:
1. “How to Setup Hadoop 3.1 on CentOS, Ubuntu and LinuxMint”
https://tecadmin.net/setup-hadoop-single-node-cluster-on-centos-redhat/
2. “How to Install Hadoop On Ubuntu 17.10 And Run Your First MapReduce Program”
https://www.digitalvidya.com/blog/install-hadoop-on-ubuntu-and-run-your-first-mapreduce-
program/
3. How do I set or change the root password for my EC2 Linux instance?
https://aws.amazon.com/tw/premiumsupport/knowledge-center/set-change-root-linux/

Hadoop 3.1.1 single node

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hadoop 3.1.1 single node

Similar to Hadoop 3.1.1 single node (20)

More from 康志強大人

More from 康志強大人 (8)

Recently uploaded

Recently uploaded (20)