High Performance Cloud Computing

3,937 views
3,824 views

Published on

Talk at AWS Genomics Event

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,937
On SlideShare
0
From Embeds
0
Number of Embeds
1,398
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

High Performance Cloud Computing

  1. 1. High Performance Cloud Computing Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r
  2. 2. Via butteryflysha under a CC-BY license
  3. 3. Image: Simon Cockell under CC-BY
  4. 4. “High Performance” Computing
  5. 5. using a large number ofcomputers at the same time for a single task
  6. 6. Image: pennstatelive
  7. 7. batch
  8. 8. tightly coupled
  9. 9. data intensive
  10. 10. 4
  11. 11. 1. Infrastructure
  12. 12. ec2-run-instances
  13. 13. instance types
  14. 14. t1.micro standard (m1)high memory (m2) high CPU (c1)
  15. 15. elastic
  16. 16. programmable
  17. 17. cluster computing
  18. 18. MPI
  19. 19. Cluster Compute Instance
  20. 20. 2*Intel Xeon 5570 23 GB RAM 1.7 TB disk
  21. 21. 10 gig E
  22. 22. Placement Group
  23. 23. Placement group
  24. 24. linpack
  25. 25. Cores 7040R max 41.82R peak 82.51
  26. 26. 231
  27. 27. 450
  28. 28. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100kCredit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
  29. 29. HPC is evolving
  30. 30. 2*Intel Xeon 5570 22 GB RAM 1.7 TB disk2*NVidia M2050
  31. 31. 2. Provision & Manage
  32. 32. AWS CloudFormation
  33. 33. bootstrap
  34. 34. chef/puppet
  35. 35. include_recipe "packages"include_recipe "ruby"include_recipe "apache2"if platform?("centos","redhat") if dist_only? # just the gem, well install the apache module within apache2 package "rubygem-passenger" return else package "httpd-devel" endelse %w{ apache2-prefork-dev libapr1-dev }.each do |pkg| package pkg do action :upgrade end endendgem_package "passenger" do version node[:passenger][:version]endexecute "passenger_module" do command echo -en "nnnn" | passenger-install-apache2-module creates node[:passenger][:module_path]end
  36. 36. familiar tools
  37. 37. Oracle Grid Engine
  38. 38. LSF
  39. 39. Moab/Torque
  40. 40. Condor
  41. 41. StackIQ Rocks+
  42. 42. combine worlds
  43. 43. MIT Starcluster
  44. 44. $ starcluster start mycluster$ starcluster listclusters
  45. 45. http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
  46. 46. 30,472 cores
  47. 47. ‘nuff said
  48. 48. AmazonElastic MapReduce
  49. 49. S3 Input dataCode Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
  50. 50. 3. Applications
  51. 51. http://usegalaxy.org/cloud
  52. 52. http://cloudbiolinux.org/
  53. 53. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
  54. 54. 4. PeopleCredit: Pieter Musterd a CC-BY-NC-ND license
  55. 55. most valuable
  56. 56. removing barriers
  57. 57. TasksInstances
  58. 58. TasksQueueInstances
  59. 59. TasksQueueInstances
  60. 60. Tasks Queue Instances Increaseinstance count
  61. 61. TasksQueueInstancesResultsStore
  62. 62. TasksQueueOn-premiseInstancesResultsStore
  63. 63. TasksQueueOn-premiseInstancesResultsStore
  64. 64. optimize for cost
  65. 65. on-demand
  66. 66. reserved
  67. 67. spot
  68. 68. http://aws.amazon.com/ec2/spot-and-science/
  69. 69. Credit: Angel Pizzaro, U. Penn
  70. 70. NASA JPL
  71. 71. Stochastic Dual Dynamic Programming44,000 CPU hrs in Oct 2010 http://aws.amazon.com/solutions/case-studies/psr/
  72. 72. Credit: Angel Pizzaro, U. Penn
  73. 73. 4
  74. 74. 1. Infrastructure
  75. 75. 2. Provision & Manage
  76. 76. 3. Applications
  77. 77. 4. People
  78. 78. deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.github.com Inspiration and ideas from Matt Wood, James Hamilton & Larry LessigCredit” Oberazzi under a CC-BY-NC-SA license

×