This document discusses options for running Hadoop clusters on Amazon EC2, including using tools like Whirr to automate cluster setup, limitations of Whirr, using Amazon EMR, manually setting up clusters, and advanced options like monitoring cluster health. It also provides context on Hadoop, clouds, and related technologies like HBase, Cassandra, and different Hadoop distributions from Cloudera, MapR, and others.

1. Automated Hadoop
Clusters on EC2
Mark Kerzner
SHMsoft

2. What is Hadoop? :) :) :)
Everybody knows that
... What is your definition?

3. What is a cloud?
Everybody knows that, but
1. Elastic resources
2. Internet delivery
3. SAAS
4. Virtualization
5. Device-enabled
6. Only (1) or all of the above

4. You are the Hadoop programmer
... and you need tools
What are your alternatives?
● IDE
● Local "cluster"
● Pseudo-distributed cluster
● EC2

5. You are the Hadoop programmer
... and you need tools
What are your alternatives?
● IDE - compile and run the code
● Local "cluster" - local file system
● Pseudo-distributed cluster - test outside
● EC2 - test on the cluster, test for scale

6. What are your resources
● Tom White, "Hadoop, the Definitive Guide"
● www.hadoopilluminated.com

7. For real play, you need a cluster

8. Hadoop+ (oh, by the way...)
HBase, Cassandra, MongoDB, NoSQL,
Dynamo, BigTable, Dryad (MS), Azure (MS),
MapReduce, MapR (EMC), Cloudera
distribution, EMC distribution, IBM distribution...

9. Whirr
Setup
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
Install
curl -O http://www.apache.org/dist/whirr/whirr-0.7.1/whirr-0.7.1.tar.gz
tar zxf whirr-0.7.1.tar.gz; cd whirr-0.7.1
Generate key
sssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr
Run
bin/whirr launch-cluster --config recipes/zookeeper-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr

10. Whirr limitations
● No EBS
● All or nothing
● Generates configuration artifacts
● Takes over your computer, no more local
development - uses proxy
● Hard to customize

11. Amazon EMR

12. EMR limitations
● No choice of image
● Fixed architecture
● Hard to debug
● Hard to customize

13. You do it
Repeat the manual procedure, only automate it
Prepare
AMI, Java, Hadoop
On-the-fly
Start AMI, login, configure, start services,
verify, run test jobs

14. You do it - advanced
On startup
Under-provision, over-provision, progress
On-the-fly
Monitor, run test jobs, watch for cluster
deterioration

15. Cloudera Manager

16. MapR Manager

17. On the large scale
Hadoop 0.20 - up to 4,000 nodes
Hadoop 0.23 - up to 20,000
GridGain - 100's of 1,000's

18. Thank you
Questions?