• Save
High Performance Cloud Computing
Upcoming SlideShare
Loading in...5
×
 

High Performance Cloud Computing

on

  • 3,771 views

Talk at AWS Genomics Event

Talk at AWS Genomics Event

Statistics

Views

Total Views
3,771
Views on SlideShare
1,138
Embed Views
2,633

Actions

Likes
5
Downloads
0
Comments
0

9 Embeds 2,633

http://mnemosyne.de-blog.jp 1299
http://mnemosyne.de-blog.jp 1299
http://paper.li 16
http://mndoci.github.com 7
http://deepaksingh.net 5
http://www.linkedin.com 4
http://213.8.145.174:82 1
http://webcache.googleusercontent.com 1
http://cache.yahoofs.jp 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

High Performance Cloud Computing High Performance Cloud Computing Presentation Transcript

  • High Performance Cloud Computing Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r
  • Via butteryflysha under a CC-BY license
  • Image: Simon Cockell under CC-BY
  • “High Performance” Computing
  • using a large number ofcomputers at the same time for a single task
  • Image: pennstatelive
  • batch
  • tightly coupled
  • data intensive
  • 4
  • 1. Infrastructure
  • ec2-run-instances
  • instance types
  • t1.micro standard (m1)high memory (m2) high CPU (c1)
  • elastic
  • programmable
  • cluster computing
  • MPI
  • Cluster Compute Instance
  • 2*Intel Xeon 5570 23 GB RAM 1.7 TB disk
  • 10 gig E
  • Placement Group
  • Placement group
  • linpack
  • Cores 7040R max 41.82R peak 82.51
  • 231
  • 450
  • WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100kCredit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
  • HPC is evolving
  • 2*Intel Xeon 5570 22 GB RAM 1.7 TB disk2*NVidia M2050
  • 2. Provision & Manage
  • AWS CloudFormation
  • bootstrap
  • chef/puppet
  • include_recipe "packages"include_recipe "ruby"include_recipe "apache2"if platform?("centos","redhat") if dist_only? # just the gem, well install the apache module within apache2 package "rubygem-passenger" return else package "httpd-devel" endelse %w{ apache2-prefork-dev libapr1-dev }.each do |pkg| package pkg do action :upgrade end endendgem_package "passenger" do version node[:passenger][:version]endexecute "passenger_module" do command echo -en "nnnn" | passenger-install-apache2-module creates node[:passenger][:module_path]end
  • familiar tools
  • Oracle Grid Engine
  • LSF
  • Moab/Torque
  • Condor
  • StackIQ Rocks+
  • combine worlds
  • MIT Starcluster
  • $ starcluster start mycluster$ starcluster listclusters
  • http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
  • 30,472 cores
  • ‘nuff said
  • AmazonElastic MapReduce
  • S3 Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  • 3. Applications
  • http://usegalaxy.org/cloud
  • http://cloudbiolinux.org/
  • mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
  • 4. PeopleCredit: Pieter Musterd a CC-BY-NC-ND license
  • most valuable
  • removing barriers
  • TasksInstances
  • TasksQueueInstances
  • TasksQueueInstances
  • Tasks Queue Instances Increaseinstance count
  • TasksQueueInstancesResultsStore
  • TasksQueueOn-premiseInstancesResultsStore
  • TasksQueueOn-premiseInstancesResultsStore
  • optimize for cost
  • on-demand
  • reserved
  • spot
  • http://aws.amazon.com/ec2/spot-and-science/
  • Credit: Angel Pizzaro, U. Penn
  • NASA JPL
  • Stochastic Dual Dynamic Programming44,000 CPU hrs in Oct 2010 http://aws.amazon.com/solutions/case-studies/psr/
  • Credit: Angel Pizzaro, U. Penn
  • 4
  • 1. Infrastructure
  • 2. Provision & Manage
  • 3. Applications
  • 4. People
  • deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.github.com Inspiration and ideas from Matt Wood, James Hamilton & Larry LessigCredit” Oberazzi under a CC-BY-NC-SA license