Hadoop on ec2

Hadoop on EC2 Configuring and running Hadoop clusters, using Cloudera distribution

My farm

Confirm

OK, it's running

Set /etc/hosts

Logging into an EC2 machine
ec2_login.sh:
#!/bin/sh ssh -i ~/.ssh/shmsoft_hadoop.pem ubuntu@$1
For example,
ec2_login sh1

Run command on the cluster
run_on_cluster.sh:

#!/bin/bash for i in {1..6} do ssh -i ~/.ssh/shmsoft_hadoop.pem ubuntu@sh$i $1 done
For example:
run_on_cluster.sh 'ifconfig | grep cast'

Result of running ifconfig
inet addr:10.220.141.227 Bcast:10.220.141.255
inet addr:10.95.31.140 Bcast:10.95.31.255
inet addr:10.220.214.15 Bcast:10.220.215.255
inet addr:10.94.245.56 Bcast:10.94.245.255
inet addr:10.127.17.143 Bcast:10.127.17.255
inet addr:10.125.79.225 Bcast:10.125.79.255

Edit local conf
gedit masters core-site.xml mapred-site.xml slaves
10.220.141.227 - masters, core-site.xml 10.95.31.140 - mapred-site.xml 10.220.214.15 - slaves... 10.94.245.56 10.127.17.143 10.125.79.225

masters
10.220.141.227

core-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  <configuration> <property> <name>fs.default.name</name> <value>hdfs://10.220.141.227</value> </property> </configuration>

mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  <configuration> <property> <name>mapred.job.tracker</name> <value>10.95.31.140:54311</value> </property> <property> <name>mapred.local.dir</name> <value>/tmp/mapred</value> </property> </configuration>

slaves
10.220.214.15 10.94.245.56 10.127.17.143 10.125.79.225

update-hadoop-cluster.sh
#!/bin/bash for i in {1..6} do scp -i ~/.ssh/hadoop.pem -r ~/projects//hadoop/conf ubuntu@hc$i:/home/ubuntu/ done run-hadoop-cluster.sh 'sudo cp /home/ubuntu/conf/* /etc/hadoop/conf/'

Important gotchas
sudo chkconfig hadoop-0.20-namenode off
repeat for each installed service

Important gotchas - 2
On EC2 in /etc/hosts you have
# Added by cloud-init 127.0.1.1 domU-12-31-38-04-AA-53.compute-1.internal domU-12-31-38-04-AA-53
Instead, do 127.0.0.1
# Added by me - the developer 127.0.0.1 domU-12-31-38-04-AA-53.compute-1.internal domU-12-31-38-04-AA-53
# !!! for remote access, use internal ip:
10.220.169.157 domU-12-31-38-04-AA-53.compute-1.internal domU-12-31-38-04-AA-53

Now start Hadoop services
On each node
for service in /etc/init.d/hadoop-0.20-*; do sudo $service start; done
Do it with script
run_my_cluster.sh ''

Verify HDFS and MR
Verify!
hadoop fs -ls /
copy to
copy from
run MR

better yet...

w3m http://localhost:50070

w3m next screen

Start HBase
start-hbase.sh
#!/bin/sh sudo /etc/init.d/hadoop-hbase-master start sudo /etc/init.d/hadoop-zookeeper-server start sudo /etc/init.d/hadoop-hbase-regionserver start

w3m http://localhost:60010

Stop HBase
1. Do compaction
2. stop-hbase.sh
#!/bin/sh sudo /etc/init.d/hadoop-hbase-master stop sleep 5 sudo /etc/init.d/hadoop-zookeeper-server stop sudo /etc/init.d/hadoop-hbase-regionserver stop

Amazon EMR

Hadoop on ec2

by markkerzner

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Hadoop on ec2 Presentation Transcript