Hadoop 安装

1,203 views

Published on

hadoop, hbase, zookeeper,hive,sqoop install

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,203
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hadoop 安装

  1. 1. 目录1 Hadoop .............................................................................................................................. 2 1.1 机器 ............................................................................................................................ 2 1.2 安装 ............................................................................................................................ 2 1.3 启动 ............................................................................................................................ 5 1.4 管理页面 .................................................................................................................... 5 1.5 FAQ............................................................................................................................ 52 Hive .................................................................................................................................... 6 2.1 机器 ............................................................................................................................ 6 2.2 安装 Mysql ................................................................................................................ 6 2.3 安装 Hive ................................................................................................................... 7 2.4 测试 ............................................................................................................................ 83 Sqoop ................................................................................................................................. 8 3.1 机器 ............................................................................................................................ 8 3.2 安装 ............................................................................................................................ 8 3.3 测试 ............................................................................................................................ 84 Zookeeper.......................................................................................................................... 9 4.1 机器 ............................................................................................................................ 9 4.2 安装 ............................................................................................................................ 9 4.3 启动 .......................................................................................................................... 10 4.4 测试 .......................................................................................................................... 105 HBase ............................................................................................................................... 10 5.1 机器 .......................................................................................................................... 10 5.2 安装 .......................................................................................................................... 10 5.3 启动 .......................................................................................................................... 12 5.4 管理界面 .................................................................................................................. 12
  2. 2. 1 Hadoop 1.1 机器 192.168.38.0(master) 192.168.38.1(slave) 192.168.38.2(slave) 192.168.38.3(slave) (上述 ip 为例子,操作时以实际 ip 为准)1 创建 hadoop 账号 (root 权限)2 设置 Master 到 Slave 的 hadoop 账号 相互之间无密码登陆(包括自己到自己)3 设置每台机器的 hostname 192.168.38.0 hadoopcm0 192.168.38.1 hadoopcm1 192.168.38.2 hadoopcm2 192.168.38.3 hadoopcm34 配置每台的/etc/hosts 192.168.38.0 hadoopcm0 192.168.38.1 hadoopcm1 192.168.38.2 hadoopcm2 192.168.38.3 hadoopcm3 1.2 安装1 JDK 1.1 安装 JDK, 1.6 以上 1.2 设置环境变量 JAVA_HOME, /usr/jdk/jdk1.6.0_13/ 1.3 将 $JAVA_HOME/bin 添加到 Path 目录以下操作使用 master 机器 hadoopcm02 下载 Hadoop 1.0.1 的发布wget http://labs.renren.com/apache-mirror//hadoop/common/hadoop-1.0.1/hadoop-1.0.1.tar.gz3 解压发布包到一个目录下, 在/home/ hadoop 下 tar xzf hadoop-1.0.1.tar.gz4 配置 hadoop 的环境变量,将以下两行添加进 ~/.bashrc 中:
  3. 3. export HADOOP_HOME=/home/hadoop/hadoop-1.0.1 export PATH=$PATH:$HADOOP_HOME/bin source ~/.bashrc5 检查 Hadoop 环境变量设置 hadoop version6 配置 hadoop 6.1 进入 conf 目录,conf 位于 hadoop-1.0.1 目录下 6.2 core-site.xml<configuration> <property> <name>fs.default.name</name> <value>hdfs://hadoopcm0:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> </property></configuration> 6.3 mapred-site.xml<configuration> <property> <name>mapred.job.tracker</name> <value>hadoopcm0:9001</value> </property> <property> <name>mapred.child.java.opts</name> <value>-Xmx2048m</value> </property></configuration> 6.4 hdfs-site.xml<configuration> <property> <name>dfs.replication</name> <value>2</value> </property></configuration>
  4. 4. 6.5 hadoop-env.sh# The java implementation to use. Required.export JAVA_HOME=/usr/jdk/jdk1.6.0_13/ 6.6 mastershadoopcm0 6.7 slaveshadoopcm1hadoopcm2hadoopcm37 将/home/hadoop 从 Master 复制到其他 Slave 机器scp –r /home/hadoop hadoopcm1:/home/hadoopscp –r /home/hadoop hadoopcm2:/home/hadoopscp –r /home/hadoop hadoopcm3:/home/hadoop8 Master 上 初始化hadoop namenode –format9 每台机器上设置环境变量export JAVA_HOME=/usr/jdk/jdk1.6.0_13export PATH=$JAVA_HOME/bin:$PATHexport HADOOP_HOME=/home/hadoop/hadoop-1.0.1export PATH=$HADOOP_HOME/bin:$PATHexport HIVE_HOME=/home/hadoop/hive-0.9.0export PATH=$HIVE_HOME/bin:$PATHexport SQOOP_HOME=/home/hadoop/sqoop-1.4.1export PATH=$SQOOP_HOME/bin:$PATHexport HBASE_HOME=/home/hadoop/hbase-0.94.0export PATH=$HBASE_HOME/bin:$PATHexport ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.4.3export PATH=$ZOOKEEPER_HOME/bin:$PATHexport HADOOP_HOME_WARN_SUPPRESS=1
  5. 5. 1.3 启动登陆 Master hadoopcm01. 启动start-all.sh2. 测试,进入 hadoop 安装目录hadoop jar hadoop-1.0.1-examples.jar wordcountfile:///home/hadoop/hadoop-1.0.1/README.txt outputhadoop fs -cat output/part-r-000003. 关闭stop-all.sh 1.4 管理页面Jobtracker:http://hadoopcm0:50030/Namenode:http://hadoopcm0:50070/ 1.5 FAQ1 Hadoop tasks: “execvp: permission denied” 运行 hadoop 程序时,报错java.lang.Throwable: Child Error at org.apache.hardoop.mapred.TaskRunner.run(TaskRunner.java:271)Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hardoop.mapred.TaskRunner.run(TaskRunner.java:258) 在 task 节点,%hadoop_home%/logs/userlogs/job*/*/stderr 报错”execvp: permissiondenied” 解决:增加文件 %JAVA_HOME%/jre/bin/java 的执行权限2 TODO
  6. 6. 2 Hive 2.1 机器单机安装 hadoopcm0 2.2 安装 Mysql如果已经安装 mysql,则跳到第 5 步增加数据库下面为 rpm 安装 Mysql5.51. 增加用户 mysql groupadd mysql useradd -g mysql mysql2. 下载 mysql rpm 包 wget http://cdn.mysql.com/Downloads/MySQL-5.5/MySQL-server-5.5.25a-1.linux2.6.x86_64.rpm wget http://cdn.mysql.com/Downloads/MySQL-5.5/MySQL-client-5.5.25a-1.linux2.6.x86_64.rpm3. 安装 mysql rpm -ivh MySQL-server-5.5.25a-1.linux2.6.x86_64.rpm rpm -ivh MySQL-client-5.5.25a-1.linux2.6.x86_64.rpm4. 启动 mysql 添加系统启动项/sbin/chkconfig --add mysql 启动/etc/init.d/mysql start5. 增加数据库 create database hive; grant all on hive.* to hive@% identified by hive; flush privileges;
  7. 7. 2.3 安装 Hive1 下载 hive wget http://labs.renren.com/apache-mirror/hive/hive-0.9.0/hive-0.9.0.tar.gz2 解压 hive tar –xzf hive-0.9.0.tar.gz3 增加 mysql 的连接 jar 包 3.1 下载 mysql 的 jar 包 wget http://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.21.tar.gz 3.2 放到 hive-0.9.0/lib 目录下4 配置 hive 4.1 进入 conf 目录,conf 位于 hive-0.9.0 目录下 4.2 hive-site.xml<configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://hadoopcm0:3306/hive?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName </name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>hive.stats.dbconnectionstring</name> <value>jdbc:mysql://hadoopcm0:3306/HiveStats?createDatabaseIfNotExist=true&amp;user=hive&amp;password=hive</value> </property></configuration> 4.3 hive-config.shexport JAVA_HOME=/usr/jdk/jdk1.6.0_13/export HIVE_HOME=/home/hadoop/hive-0.9.0/export HADOOP_HOME=/home/hadoop/hadoop-1.0.1 4.4 hive-log4j.propertiescp hive-log4j.properties.template hive-log4j.properties
  8. 8. 增加下面一行到 hive-log4j.propertieslog4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter 2.4 测试进入 hive 安装目录bin/hiveshow tables;能正常显示3 Sqoop 3.1 机器 单机安装 192.168.38.0 hadoopcm0 3.2 安装1、下载 sqoopwget http://labs.renren.com/apache-mirror/sqoop/1.4.1-incubating/sqoop-1.4.1-incubating__hadoop-1.0.0.tar.gz2、解压tar -xf sqoop-1.4.1-incubating__hadoop-1.0.0.tar.gzmv sqoop-1.4.1-incubating__hadoop-1.0.0 sqoop-1.4.1将解压后的 sqoop-1.4.1 文件放在系统的/home/hadoop/中。 3.3 测试输入 sqoop version出现 sqoop-1.4.1-incubating,测试通过
  9. 9. 4 Zookeeper 4.1 机器 使用其中的三台 Slave 192.168.38.1 hadoopcm1 192.168.38.2 hadoopcm2 192.168.38.3 hadoopcm3 4.2 安装1 下载 zookeeperwget http://labs.renren.com/apache-mirror/zookeeper/zookeeper-3.4.3/zookeeper-3.4.3.tar.gz2 解压tar -xf zookeeper-3.4.3.tar.gz将解压后的 zookeeper-3.4.3 文件放在系统的/home/hadoop/中3 配置 3.1 将 zookeeper-3.4.3/conf 目录下的 zoo_sample.cfg 文件拷贝一份,命名为为“zoo.cfg” 3.2 修改 zoo.cfgtickTime=2000dataDir=/home/hadoop/zookeeper-3.4.3/zookeeper-datadataLogDir=/home/hadooptest/zookeeper-3.4.3/logsclientPort=2181initLimit=5syncLimit=2server.1=hadoopcm1:2888:3888server.2=hadoopcm2:2888:3888server.3=hadoopcm3:2888:3888 3.3 创建 zookpper-data 目录mkdir /home/hadoop/zookeeper3.4.3/zookeeper-data 3.4 生成 myid 文件touch myid4 将/home/hadoop/ zookeeper-3.4.3 复制到其他 Slave 机器scp –r /home/hadoop/zookeeper-3.4.3 hadoopcm1:/home/hadoop
  10. 10. scp –r /home/hadoop/zookeeper-3.4.3 hadoopcm2:/home/hadoop5 修改 myid 文件 hadoopcm1 机器中 myid 写入 1 hadoopcm2 机器中 myid 写入 2 hadoopcm3 机器中 myid 写入 3 4.3 启动每台机器中执行1. 启动bin/zkServer.sh start2. 关闭bin/zkServer.sh stop 4.4 测试bin/zkCli.sh -server 127.0.0.1:21815 HBase 5.1 机器 同上,使用其中的三台 slaves 192.168.38.1 hadoopcm1 192.168.38.2 hadoopcm2 192.168.38.3 hadoopcm3 5.2 安装1 下载 HBase 0.94 的发布 wget http://labs.renren.com/apache-mirror/hbase/hbase-0.94.0/hbase-0.94.0.tar.gz2 解压发布包到一个目录 /home/hadoop tar xzf hbase-0.94.0.tar.gz
  11. 11. 3 配置 hbase 3.1 进入 conf 目录,conf 位于 hbase-0.94.0 目录下 3.2 hbase-site.xml<configuration> <property> <name>hbase.master</name> <value>hadoopcm1:60000</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://hadoopcm0:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>hadoopcm1, hadoopcm2, hadoopcm3</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/hadoop/zookeeper</value> </property></configuration> 3.3 regionservershadoopcm1hadoopcm2hadoopcm34 将/home/hadoop/hbase-0.94.0 复制到其他机器scp –r /home/hadoop/hbase-0.94.0 hadoopcm2:/home/hadoopscp –r /home/hadoop/hbase-0.94.0 hadoopcm3:/home/hadoop
  12. 12. 5.3 启动在 Hbase 安装目录下1. 启动$./bin/start-hbase.sh2. 关闭$ ./bin/stop-hbase.sh 5.4 管理界面HBase Master:http://192.168.38.2:60010/

×