[Harvard CS264] 08a - Cloud Computing, Amazon EC2, MIT StarCluster (Justin Riley, MIT)
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

[Harvard CS264] 08a - Cloud Computing, Amazon EC2, MIT StarCluster (Justin Riley, MIT)

  • 3,105 views
Uploaded on

http://cs264.org

http://cs264.org

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
3,105
On Slideshare
3,103
From Embeds
2
Number of Embeds
2

Actions

Shares
Downloads
62
Comments
0
Likes
0

Embeds 2

http://www.cs264.org 1
http://www.docseek.net 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. CS264: Introduction to Cloud Computing Justin Riley Software Tools for Academics and Researchers Office of Educational Innovation and Technology Massachusetts Institute of Technology
  • 2. What is Cloud Computing Anyway?“Cloud computing” is a very fuzzyterm in generalOften includes everything and thekitchen sinkThree broad categories:Software as a Service (SaaS)Platform as a Service (PaaS)Infrastructure as a Service (IaaS)Image Credit: http://tomlambert.com/cloud-computing-will-rule-the-world/
  • 3. Infrastructure as a Service (IaaS) Hardware On Demand Pay for what you use Full root access – you control the OS and Software Stack Ability to scale computing resources up and down No dealing with racks, networks, power, cooling, housing, etc.
  • 4. Amazon Web Services
  • 5. Amazon Simple Storage Solution (S3) “... a simple web service interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web”Read, write, and delete objectscontaining from 1 byte to 5terabytes of data each.Number of objects you can storeis unlimited.Each object stored in a bucketand retrieved via a unique, user-assigned key
  • 6. Amazon Elastic Compute Cloud (EC2) Resizable Compute Capacity As much as you need, when you need it. Scale up or down in minutes. Complete Control via API Create, scale, & manage instances programmatically. Variety of Instance Sizes CPU Power, Cores, RAM, Disk. Wide Variety of Pre-built AMIs (Amazon Machine Images) Hit the ground running with minimal system building effort. Now: Linux, Windows, and OpenSolaris. Secure & Flexible Network Security Model Full control of access for each running instance. Keypair required for SSH access.
  • 7. Amazon EC2 Instance Types Micro Standard High High Cluster Cluster Memory CPU Compute GPU Extra Extra Micro Small Large XL 2 XL 4 XL Medium 4 XL 4XL Large LargeBits 32/64 32 64 64 64 64 64 32 64 64 64RAM 7.5 613 MB 1.7 GB 15 GB 17.1 34.2 68.4 1.7 GB 7 GB 23 22 GBDisk 850 850 1690 420 GB 160 GB 1690 GB 420 350 GB 1690 GB 1690 GB 1690 GB GB GB GBVirtual 2 NVIDIACores Tesla 1 1 2 4 2 4 8 2 8 8 “Fermi” GPUsEC2Compute 2 (BurstUnits 1 4 8 6.5 13 26 5 20 33.5 33.5 )Firewall Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes On-Demand PricingLinuxPer Hour $0.02 $0.085 $0.34 $0.68 $0.50 $1.00 $2.00 $0.17 $0.68 1.60 2.10Window $0.03 $0.12 $0.48 $0.96 $0.62 $1.24 $2.48 $0.29 $1.16 N/A N/As
  • 8. “Spot” Instances Bid for unused AWS capacity Prices controlled by AWS based on supply and demand AWS can terminate Spot Instances without notice Best approach to temporary requests for large numbers of servers Default maximum = 100 servers (instead of 20 on-demand)
  • 9. Amazon Machine Images (AMI) Contains an entire operating system and software stack that can be loaded onto one or more virtual machines Instance AMI Instance Instance Instance Instance
  • 10. Amazon Elastic Block Storage (EBS) Persistent storage Volume lifetime is independent of any particular EC2 instance. General purpose Raw, unformatted, block device. Use from Linux, Solaris or Windows. High performance Equal to or better than local EC2 drive. High reliability Built-in redundancy within availability zone. AFR (Annual Failure Rate) between 0.1% and 1%. Scalable Volume sizes ranging from 1 GB to 1 TB. Easy to create, attach, back up, restore, and delete volumes.
  • 11. Amazon Elastic Block Storage Pricing EBS Volumes  $0.10 per GB-month of provisioned storage  $0.10 per 1 million I/O requests  No charge for mounting/unmounting volume EBS Snapshots to Amazon S3  $0.14 per GB-month of data stored  $0.01 per 1,000 PUT requests (when saving a snapshot)  $0.01 per 10,000 GET requests (when loading a snapshot)
  • 12. Amazon EC2 Regions and Availability Zones US East Region EU West Region Availability Availability Zone A Zone B Availability Availability Zone A Zone B Availability Zone C US West Region Singapore Availability Availability Availability Availability Zone A Zone B Zone A Zone B Note: Conceptual drawing only. The number of Availability Zones may vary
  • 13. Notes on Using EBS Volumes ● EBS volumes can only be used with instances in the same availability zone they were created in ● Analogous to a virtual “pen drive” ● Can only attach a volume to one instance at a time.
  • 14. Amazon Web Services ConsoleWeb-based management console for all AWS services http://aws.amazon.com/console
  • 15. Elastic MapReduce Easily launch Map/Reducejobs on Amazon EC2 Uses Hadoop Define Map/Reduce workflows either at command line orfrom AWS console Mapper/Reducer code mustbe stored on S3 Input/output data stored on S3
  • 16. Introducing StarClusterDeveloped at MITUnder active developmentOpen sourceWeb site: http://web.mit.edu/stardev/cluster/Easy to install and use ($ easy_install starcluster)Simplifies creation and management of EC2 clusters
  • 17. Why StarCluster?EC2 provides raw compute powerThere’s work to be done to create a usable cluster: Software installation AMI creation AWS / SSH key management and distribution Persistent Disk Storage and File Sharing Configuration management Higher-level management (cluster vs. instance)
  • 18. StarCluster FeaturesPrebuilt 32 and 64 bit AMIsLaunch a cluster of EC2 instances: One command (“starcluster”) to rule them all Passwordless SSH pre-configured Security group for SSH access Shared disk volume (NFS) Preinstalled libraries (OpenMPI, NumPy, SciPy, etc.)Easy to install, configure, and use
  • 19. StarCluster Architecture / Terminology AWS Region Cluster Cluster Cluster Master Node001 NodeN Client EC2 EC2 … EC2EC2 or DesktopRunning Linux Master DiskConfig File
  • 20. PrerequisitesClient computer running Mac/LinuxAWS security credentials: Access Key ID Secret Access Key Public Key (Keypair)Cluster-aware application (something to run)
  • 21. StepsInstall StarCluster on clientConfigure StarClusterStart cluster(s)Use themStop cluster(s)
  • 22. Configure StarClusterDownload your keypair to clientEdit .starcluster/config
  • 23. Edit .starcluster/config AWS Credentials Must match KEYNAME Name and location of file downloaded in last slide Name of EC2 keypair
  • 24. Additional Configuration Options Cluster size AMI for nodes Node instance type Master instance type AMI for master
  • 25. Start Cluster<client>: starcluster start mycluster
  • 26. Start Cluster (Output - 1)StarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to starcluster@mit.edu>>> Using default cluster template: smallcluster>>> Validating cluster template settings...>>> Cluster template settings are valid>>> Starting cluster...>>> Launching a 5-node cluster...>>> Launching master node...>>> Master AMI: ami-d1c42db8>>> Creating security group @sc-jb1...Reservation:r-edb9bd87>>> Launching worker nodes...>>> Node AMI: ami-d1c42db8Reservation:r-e1b9bd8b>>> Waiting for cluster to start...
  • 27. Start Cluster (Output - 2)>>> Waiting for cluster to start...>>> The master node is ec2-50-16-41-160.compute-1.amazonaws.com>>> Setting up the cluster...>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Creating cluster user: sgeadmin>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Configuring scratch space for user: sgeadmin>>> Configuring /etc/hosts on each node>>> Configuring NFS...>>> Configuring passwordless ssh for root>>> Configuring passwordless ssh for user: sgeadmin>>> Generating local RSA ssh keys for user: sgeadmin>>> Installing Sun Grid Engine...>>> Done Configuring Sun Grid Engine>>>
  • 28. Start Cluster (Output – 3)The cluster has been started and configured.Login to the master node as root by running: $ starcluster sshmaster jb1or manually as sgeadmin: $ ssh -i /home/ec2-user/keys-jbarr-us-east.pem sgeadmin@ec2-50-16-41-160.compute-1.amazonaws.comWhen you are finished using the cluster, run: $ starcluster stop jb1to shutdown the cluster and stop paying for service>>> start took 5.337 mins
  • 29. Check Cluster Status<client>: starcluster listclustersStarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to starcluster@mit.edu-----------------------------jb1 (security group: @sc-jb1)-----------------------------Launch time: 2011-01-14T05:43:44.000ZZone: us-east-1cKeypair: keys-jbarr-us-eastCluster nodes: master running i-3fad6653 ec2-50-16-41-160.compute-1.amazonaws.com node001 running i-3bad6657 ec2-184-73-107-91.compute-1.amazonaws.com node002 running i-35ad6659 ec2-174-129-124-218.compute-1.amazonaws.com node003 running i-37ad665b ec2-50-16-32-211.compute-1.amazonaws.com node004 running i-31ad665d ec2-50-16-31-114.compute-1.amazonaws.com
  • 30. Access Cluster SSH to master node as root:<client>: starcluster sshmaster mycluster SSH to any given node:<client>: starcluster sshnode mycluster node001
  • 31. StarCluster AMIUbuntu-based (8.10, 9.04, 10.04)Automatically installs/configures: OpenMPI Oracle Grid Engine (formerly Sun Grid Engine)Other pre-installed libraries: ATLAS LAPACK NumPy SciPy
  • 32. Using Sun Grid Engine Run all commands on master, as user sgeadmin:<client>: starcluster sshmaster mycluster<master>: su – sgeadmin Important commands:  qstat – Examine work queue  qsub – Submit work  qhost – List hosts in grid
  • 33. Sun Grid Engine – Queue and Host StatusCheck the queue status using qstat:Check the host status using qhost:
  • 34. Sun Grid Engine – Running Scripts - #!/bin/bash echo -n "Hello from script running on host " hostname time find /lib -type f -exec ls -l {} ; echo "Goodbye from script"<master-sge>: qsub -V -cwd exercise.shYour job 9 ("exercise.sh") has been submitted The argument “-V” is used to pass the current environment to the job once its executed.
  • 35. Watch Cluster in AWS Console
  • 36. Other StarCluster Commands listclusters listinstances createimage createvolume (EBS) listvolumes showconsole
  • 37. Stop ClusterAWS charges accrue as long as the cluster is running!Easy to start, easy to stop, so be parsimonious.To stop the cluster: <client>: starcluster stop jb1 StarCluster - (http://web.mit.edu/starcluster) Software Tools for Academics and Researchers (STAR) Please submit bug reports to starcluster@mit.edu Shutdown cluster jb1 (y/n)? y >>> Shutting down i-3fad6653 >>> Shutting down i-3bad6657 >>> Shutting down i-35ad6659 >>> Shutting down i-37ad665b >>> Shutting down i-31ad665d >>> Removing cluster security group @sc-jb1 <client>:
  • 38. Stop Cluster – Verify in Console
  • 39. Creating EBS Volumes (made easy)$ starcluster createvolume 100 us-east-1a Automagically handles:  Launching instance in specified zone  Creating and attaching an EBS volume to the instance  Partitioning/formatting the EBS volume
  • 40. Creating a Custom AMICreate a custom AMI (image):  Launch instance of AMI  Install and configure desired libraries, tools, apps$ starcluster createimage i-9c9c9c myimg myimgbucket
  • 41. StarCluster Plugin System Example Code (ubuntu.py):Specify your own custominstall routinesExecuted after defaultcluster setup routinesPlugins currently exist for:Hadoop (MapReduce),ipcluster (Ipython cluster),MPICH2, and more Example Config:
  • 42. Discussion / Q&A