High Performance
   Cloud Computing



                                 Deepak Singh
         P r i n c i p a l   P r o d u c t   M a n a g e r
Via butteryflysha under a CC-BY license
Image: Simon Cockell under CC-BY
“High Performance”
    Computing
using a large number of
computers at the same
 time for a single task
Image: pennstatelive
batch
tightly coupled
data intensive
4
1. Infrastructure
ec2-run-instances
instance types
t1.micro




  standard (m1)
high memory (m2)
  high CPU (c1)
elastic
programmable
cluster computing
MPI
Cluster Compute
    Instance
2*Intel Xeon 5570
   23 GB RAM
   1.7 TB disk
10 gig E
Placement
  Group
Placement
  group
linpack
Cores      7040
R   max
           41.82
R   peak
           82.51
231
450
WIEN2K Parallel
                                                                    Performance

                                                                          H size 56,000 (25GB)
                                                                     Runtime (16x8 processors)
                                                                        Local (Infiniband) 3h:48
                                                                   Cloud (10Gbps) 1h:30 ($40)




                    1200 atom unit cell; SCALAPACK+MPI
                    diagonalization, matrix size 50k-100k

Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
HPC is evolving
2*Intel Xeon 5570
   22 GB RAM
   1.7 TB disk
2*NVidia M2050
2. Provision & Manage
AWS CloudFormation
bootstrap
chef/puppet
include_recipe "packages"
include_recipe "ruby"
include_recipe "apache2"

if platform?("centos","redhat")
  if dist_only?
     # just the gem, we'll install the apache module within apache2
     package "rubygem-passenger"
     return
  else
     package "httpd-devel"
  end
else
  %w{ apache2-prefork-dev libapr1-dev }.each do |pkg|
     package pkg do
       action :upgrade
     end
  end
end

gem_package "passenger" do
  version node[:passenger][:version]
end

execute "passenger_module" do
  command 'echo -en "nnnn" | passenger-install-apache2-module'
  creates node[:passenger][:module_path]
end
familiar tools
Oracle Grid Engine
LSF
Moab/Torque
Condor
StackIQ Rocks+
combine worlds
MIT Starcluster
$ starcluster start mycluster
$ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
30,472 cores
‘nuff said
Amazon
Elastic MapReduce
S3

        Input data




Code     Elastic              Name                            Output
       MapReduce              node                          S3 + SimpleDB


                         Queries
                                                     HDFS
                          + BI
                     Via JDBC, Pig, Hive
                                           Elastic
                                           cluster
3. Applications
http://usegalaxy.org/cloud
http://cloudbiolinux.org/
mapreduce for
  genomics
 http://bowtie-bio.sourceforge.net/crossbow/index.shtml
            http://contrail-bio.sourceforge.net
   http://bowtie-bio.sourceforge.net/myrna/index.shtml
4. People


Credit: Pieter Musterd a CC-BY-NC-ND license
most valuable
removing barriers
Tasks




Instances
Tasks



Queue




Instances
Tasks



Queue




Instances
Tasks



            Queue




            Instances




 Increase
instance
  count
Tasks



Queue




Instances




Results



Store
Tasks



Queue



On-premise


Instances


Results



Store
Tasks



Queue



On-premise


Instances


Results



Store
optimize for cost
on-demand
reserved
spot
http://aws.amazon.com/ec2/spot-and-science/
Credit: Angel Pizzaro, U. Penn
NASA JPL
Stochastic Dual Dynamic Programming




44,000 CPU hrs in Oct 2010
            http://aws.amazon.com/solutions/case-studies/psr/
Credit: Angel Pizzaro, U. Penn
4
1. Infrastructure
2. Provision & Manage
3. Applications
4. People
deesingh@amazon.com
                                                            Twitter:@mndoci
                                               http://slideshare.net/mndoci
                                                   http://mndoci.github.com




        Inspiration and ideas from
        Matt Wood, James Hamilton
               & Larry Lessig

Credit” Oberazzi under a CC-BY-NC-SA license

High Performance Cloud Computing