[Harvard CS264] 08a - Cloud Computing, Amazon EC2, MIT StarCluster (Justin Riley, MIT)
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


[Harvard CS264] 08a - Cloud Computing, Amazon EC2, MIT StarCluster (Justin Riley, MIT)







Total Views
Views on SlideShare
Embed Views



2 Embeds 2

http://www.cs264.org 1
http://www.docseek.net 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

[Harvard CS264] 08a - Cloud Computing, Amazon EC2, MIT StarCluster (Justin Riley, MIT) Presentation Transcript

  • 1. CS264: Introduction to Cloud Computing Justin Riley Software Tools for Academics and Researchers Office of Educational Innovation and Technology Massachusetts Institute of Technology
  • 2. What is Cloud Computing Anyway?“Cloud computing” is a very fuzzyterm in generalOften includes everything and thekitchen sinkThree broad categories:Software as a Service (SaaS)Platform as a Service (PaaS)Infrastructure as a Service (IaaS)Image Credit: http://tomlambert.com/cloud-computing-will-rule-the-world/
  • 3. Infrastructure as a Service (IaaS) Hardware On Demand Pay for what you use Full root access – you control the OS and Software Stack Ability to scale computing resources up and down No dealing with racks, networks, power, cooling, housing, etc.
  • 4. Amazon Web Services
  • 5. Amazon Simple Storage Solution (S3) “... a simple web service interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web”Read, write, and delete objectscontaining from 1 byte to 5terabytes of data each.Number of objects you can storeis unlimited.Each object stored in a bucketand retrieved via a unique, user-assigned key
  • 6. Amazon Elastic Compute Cloud (EC2) Resizable Compute Capacity As much as you need, when you need it. Scale up or down in minutes. Complete Control via API Create, scale, & manage instances programmatically. Variety of Instance Sizes CPU Power, Cores, RAM, Disk. Wide Variety of Pre-built AMIs (Amazon Machine Images) Hit the ground running with minimal system building effort. Now: Linux, Windows, and OpenSolaris. Secure & Flexible Network Security Model Full control of access for each running instance. Keypair required for SSH access.
  • 7. Amazon EC2 Instance Types Micro Standard High High Cluster Cluster Memory CPU Compute GPU Extra Extra Micro Small Large XL 2 XL 4 XL Medium 4 XL 4XL Large LargeBits 32/64 32 64 64 64 64 64 32 64 64 64RAM 7.5 613 MB 1.7 GB 15 GB 17.1 34.2 68.4 1.7 GB 7 GB 23 22 GBDisk 850 850 1690 420 GB 160 GB 1690 GB 420 350 GB 1690 GB 1690 GB 1690 GB GB GB GBVirtual 2 NVIDIACores Tesla 1 1 2 4 2 4 8 2 8 8 “Fermi” GPUsEC2Compute 2 (BurstUnits 1 4 8 6.5 13 26 5 20 33.5 33.5 )Firewall Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes On-Demand PricingLinuxPer Hour $0.02 $0.085 $0.34 $0.68 $0.50 $1.00 $2.00 $0.17 $0.68 1.60 2.10Window $0.03 $0.12 $0.48 $0.96 $0.62 $1.24 $2.48 $0.29 $1.16 N/A N/As
  • 8. “Spot” Instances Bid for unused AWS capacity Prices controlled by AWS based on supply and demand AWS can terminate Spot Instances without notice Best approach to temporary requests for large numbers of servers Default maximum = 100 servers (instead of 20 on-demand)
  • 9. Amazon Machine Images (AMI) Contains an entire operating system and software stack that can be loaded onto one or more virtual machines Instance AMI Instance Instance Instance Instance
  • 10. Amazon Elastic Block Storage (EBS) Persistent storage Volume lifetime is independent of any particular EC2 instance. General purpose Raw, unformatted, block device. Use from Linux, Solaris or Windows. High performance Equal to or better than local EC2 drive. High reliability Built-in redundancy within availability zone. AFR (Annual Failure Rate) between 0.1% and 1%. Scalable Volume sizes ranging from 1 GB to 1 TB. Easy to create, attach, back up, restore, and delete volumes.
  • 11. Amazon Elastic Block Storage Pricing EBS Volumes  $0.10 per GB-month of provisioned storage  $0.10 per 1 million I/O requests  No charge for mounting/unmounting volume EBS Snapshots to Amazon S3  $0.14 per GB-month of data stored  $0.01 per 1,000 PUT requests (when saving a snapshot)  $0.01 per 10,000 GET requests (when loading a snapshot)
  • 12. Amazon EC2 Regions and Availability Zones US East Region EU West Region Availability Availability Zone A Zone B Availability Availability Zone A Zone B Availability Zone C US West Region Singapore Availability Availability Availability Availability Zone A Zone B Zone A Zone B Note: Conceptual drawing only. The number of Availability Zones may vary
  • 13. Notes on Using EBS Volumes ● EBS volumes can only be used with instances in the same availability zone they were created in ● Analogous to a virtual “pen drive” ● Can only attach a volume to one instance at a time.
  • 14. Amazon Web Services ConsoleWeb-based management console for all AWS services http://aws.amazon.com/console
  • 15. Elastic MapReduce Easily launch Map/Reducejobs on Amazon EC2 Uses Hadoop Define Map/Reduce workflows either at command line orfrom AWS console Mapper/Reducer code mustbe stored on S3 Input/output data stored on S3
  • 16. Introducing StarClusterDeveloped at MITUnder active developmentOpen sourceWeb site: http://web.mit.edu/stardev/cluster/Easy to install and use ($ easy_install starcluster)Simplifies creation and management of EC2 clusters
  • 17. Why StarCluster?EC2 provides raw compute powerThere’s work to be done to create a usable cluster: Software installation AMI creation AWS / SSH key management and distribution Persistent Disk Storage and File Sharing Configuration management Higher-level management (cluster vs. instance)
  • 18. StarCluster FeaturesPrebuilt 32 and 64 bit AMIsLaunch a cluster of EC2 instances: One command (“starcluster”) to rule them all Passwordless SSH pre-configured Security group for SSH access Shared disk volume (NFS) Preinstalled libraries (OpenMPI, NumPy, SciPy, etc.)Easy to install, configure, and use
  • 19. StarCluster Architecture / Terminology AWS Region Cluster Cluster Cluster Master Node001 NodeN Client EC2 EC2 … EC2EC2 or DesktopRunning Linux Master DiskConfig File
  • 20. PrerequisitesClient computer running Mac/LinuxAWS security credentials: Access Key ID Secret Access Key Public Key (Keypair)Cluster-aware application (something to run)
  • 21. StepsInstall StarCluster on clientConfigure StarClusterStart cluster(s)Use themStop cluster(s)
  • 22. Configure StarClusterDownload your keypair to clientEdit .starcluster/config
  • 23. Edit .starcluster/config AWS Credentials Must match KEYNAME Name and location of file downloaded in last slide Name of EC2 keypair
  • 24. Additional Configuration Options Cluster size AMI for nodes Node instance type Master instance type AMI for master
  • 25. Start Cluster<client>: starcluster start mycluster
  • 26. Start Cluster (Output - 1)StarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to starcluster@mit.edu>>> Using default cluster template: smallcluster>>> Validating cluster template settings...>>> Cluster template settings are valid>>> Starting cluster...>>> Launching a 5-node cluster...>>> Launching master node...>>> Master AMI: ami-d1c42db8>>> Creating security group @sc-jb1...Reservation:r-edb9bd87>>> Launching worker nodes...>>> Node AMI: ami-d1c42db8Reservation:r-e1b9bd8b>>> Waiting for cluster to start...
  • 27. Start Cluster (Output - 2)>>> Waiting for cluster to start...>>> The master node is ec2-50-16-41-160.compute-1.amazonaws.com>>> Setting up the cluster...>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Creating cluster user: sgeadmin>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Configuring scratch space for user: sgeadmin>>> Configuring /etc/hosts on each node>>> Configuring NFS...>>> Configuring passwordless ssh for root>>> Configuring passwordless ssh for user: sgeadmin>>> Generating local RSA ssh keys for user: sgeadmin>>> Installing Sun Grid Engine...>>> Done Configuring Sun Grid Engine>>>
  • 28. Start Cluster (Output – 3)The cluster has been started and configured.Login to the master node as root by running: $ starcluster sshmaster jb1or manually as sgeadmin: $ ssh -i /home/ec2-user/keys-jbarr-us-east.pem sgeadmin@ec2-50-16-41-160.compute-1.amazonaws.comWhen you are finished using the cluster, run: $ starcluster stop jb1to shutdown the cluster and stop paying for service>>> start took 5.337 mins
  • 29. Check Cluster Status<client>: starcluster listclustersStarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to starcluster@mit.edu-----------------------------jb1 (security group: @sc-jb1)-----------------------------Launch time: 2011-01-14T05:43:44.000ZZone: us-east-1cKeypair: keys-jbarr-us-eastCluster nodes: master running i-3fad6653 ec2-50-16-41-160.compute-1.amazonaws.com node001 running i-3bad6657 ec2-184-73-107-91.compute-1.amazonaws.com node002 running i-35ad6659 ec2-174-129-124-218.compute-1.amazonaws.com node003 running i-37ad665b ec2-50-16-32-211.compute-1.amazonaws.com node004 running i-31ad665d ec2-50-16-31-114.compute-1.amazonaws.com
  • 30. Access Cluster SSH to master node as root:<client>: starcluster sshmaster mycluster SSH to any given node:<client>: starcluster sshnode mycluster node001
  • 31. StarCluster AMIUbuntu-based (8.10, 9.04, 10.04)Automatically installs/configures: OpenMPI Oracle Grid Engine (formerly Sun Grid Engine)Other pre-installed libraries: ATLAS LAPACK NumPy SciPy
  • 32. Using Sun Grid Engine Run all commands on master, as user sgeadmin:<client>: starcluster sshmaster mycluster<master>: su – sgeadmin Important commands:  qstat – Examine work queue  qsub – Submit work  qhost – List hosts in grid
  • 33. Sun Grid Engine – Queue and Host StatusCheck the queue status using qstat:Check the host status using qhost:
  • 34. Sun Grid Engine – Running Scripts - #!/bin/bash echo -n "Hello from script running on host " hostname time find /lib -type f -exec ls -l {} ; echo "Goodbye from script"<master-sge>: qsub -V -cwd exercise.shYour job 9 ("exercise.sh") has been submitted The argument “-V” is used to pass the current environment to the job once its executed.
  • 35. Watch Cluster in AWS Console
  • 36. Other StarCluster Commands listclusters listinstances createimage createvolume (EBS) listvolumes showconsole
  • 37. Stop ClusterAWS charges accrue as long as the cluster is running!Easy to start, easy to stop, so be parsimonious.To stop the cluster: <client>: starcluster stop jb1 StarCluster - (http://web.mit.edu/starcluster) Software Tools for Academics and Researchers (STAR) Please submit bug reports to starcluster@mit.edu Shutdown cluster jb1 (y/n)? y >>> Shutting down i-3fad6653 >>> Shutting down i-3bad6657 >>> Shutting down i-35ad6659 >>> Shutting down i-37ad665b >>> Shutting down i-31ad665d >>> Removing cluster security group @sc-jb1 <client>:
  • 38. Stop Cluster – Verify in Console
  • 39. Creating EBS Volumes (made easy)$ starcluster createvolume 100 us-east-1a Automagically handles:  Launching instance in specified zone  Creating and attaching an EBS volume to the instance  Partitioning/formatting the EBS volume
  • 40. Creating a Custom AMICreate a custom AMI (image):  Launch instance of AMI  Install and configure desired libraries, tools, apps$ starcluster createimage i-9c9c9c myimg myimgbucket
  • 41. StarCluster Plugin System Example Code (ubuntu.py):Specify your own custominstall routinesExecuted after defaultcluster setup routinesPlugins currently exist for:Hadoop (MapReduce),ipcluster (Ipython cluster),MPICH2, and more Example Config:
  • 42. Discussion / Q&A