Utility Supercomputing on AWS - Stowe

794 views

Published on

Published in: Technology
  • Be the first to comment

Utility Supercomputing on AWS - Stowe

  1. 1. CycleCloud: Utility supercomputing for everyone Jason A. Stowe CEO, @jasonastowe
  2. 2. Change in capability forscience / math / analysis workloads
  3. 3. Helps do what we’re capable of
  4. 4. Utility Supercomputing
  5. 5. On-demand creation of small to massive HPC,as a metered, pay-for-use service.
  6. 6. Software that creates Cloud HPC from 50 to 50,000 cores
  7. 7. Hartford, JPMorgan,… Lockheed Martin,Purdue DiaGrid, Duke, U.Wisc
  8. 8. Cycle’s Utility Supercomputing inWSJ, NY Times, Wired, Bio-IT World BusinessWeek
  9. 9. Today50,000-core CycleCloud Utility Supercomputer on AWS
  10. 10. With Nimbus Discovery,looking at a cancer drug target
  11. 11. How big is 50,000 cores? Why does this matter?
  12. 12. (W.H.O./Globocan 2008)
  13. 13. Every day iscrucial and costly
  14. 14. Let’s look at drug designerstrying to treat these diseases
  15. 15. Develops leading computational chemistry algorithms
  16. 16. Drug Design involves identifying protein targets
  17. 17. Flickr: AMagill Which are like locks
  18. 18. Then finding molecules that may fit
  19. 19. Find the matches in millions of molecules
  20. 20. But, internal HPC clusters are too small when you need them most… and too large every other time.
  21. 21. Challenge: To run a virtual screen with ahigher accuracy algorithm & 21 million compounds
  22. 22. 50,000-core CycleCloud on AWS
  23. 23. Scales to 50 - 50,000+ cores - Automation & Error Handling - Encryption & Key Mgmt - Elastic, Massive CPU/FS Cloud HPC Software - Data scheduling, migration - Security, Security, Security… Deploying, Auto-scalingInternal HPC HPC clusters Hardware Audit & Reporting Applications Traditional Scheduler Security Automation, Monitoring Encryption/Key Mgmt Chef (SW config) Shared FS Storage/Data Scheduling OS / File System Scale Certification Instances Job/Cluster APIs EBS User Application Templates S3 EBS
  24. 24. Now I could bore you with infrastructure details
  25. 25. Details: http://blog.cyclecomputing.com
  26. 26. Details: http://blog.cyclecomputing.com
  27. 27. Details: http://blog.cyclecomputing.com
  28. 28. Resource CountTotal Instances 6,742 ($20+ million datacenter spend)- c1.xlarge instances 5,435- c1.medium instances 779- cc1.4xlarge instances 297- cc2.8xlarge instance 231cores 51,132RAM 58.78-TBAWS Regions All (7: us-east, us-west1, us-west2, eu-west, sa-east, ap-northeast, ap-southeast )
  29. 29. Metric CountCompute Hours of Work 109,927 hoursCompute Days of Work 4,580 daysCompute Years of Work 12.55 yearsLigand Count ~21 million ligands Using CycleCloud & AWSThe impossible run finished in…
  30. 30. 3 Hours…For $4828.85/hr
  31. 31. TodayFor < $1,000/hr
  32. 32. Instead of $20+ Million in infrastructure: At Peak Cost: <$1,000/hr to run
  33. 33. With Nimbus Discovery,already acquiring the “new leads”
  34. 34. Every day iscrucial and costly...
  35. 35. Not just in cancer researchAnd not just for big jobs.
  36. 36. With analysis,humans can be the bottleneck
  37. 37. We only have so much time on the planet
  38. 38. Big 10 Pharma with peak loads: <1,000 core use case
  39. 39. Researcher/ CycleServer SubmitOnce™ Engineer Big 10 Pharma JobsHPC / Workload Analytics CycleCloud clusters look exactly like internal environments- SubmitOnce™ routes workloads - Computational Fluid Dynamics- Agile Workload placement, Application-aware scheduling™ - Clinical Trial Simulation File File File File System System System System
  40. 40. Lab-generated Image Analysis Workload (very similar to satellite imagery) CycleCloud HPC Cluster - Linux and Windows workload Processing - Cluster spun up only during analysis - Similar to any imagery, document, or audio analysis Imagery FileAnalyst Data File System System
  41. 41. Engineering, Modeling 40 – 800 cores
  42. 42. EDA/Engineering company running 100s of cores, once a month
  43. 43. CycleCloud clusters look exactly like internal environments Compute Jobs -Periodic Usage, far 10,000% cheaper -Costs < $100/month Data FileResearcher File System System
  44. 44. Financial Modeling, Analogous toTax revenue modeling 80 – 1,600 cores
  45. 45. Researcher/ CycleServer SubmitOnce™ Quant Fortune 500 FinSer/Insurance JobsHPC / Workload Analytics CycleCloud clusters look exactly like internal environments- SubmitOnce™ routes workloads - Windows Cluster for Hedging analysis- Data scheduling seamless to user - Data migrated seamlessly File File File File System System System System
  46. 46. Tax Revenue modelingImagery/document/audio Analysis Infectious disease analysis Time is of the essence…
  47. 47. Tax Revenue modelingSatellite Imagery AnalysisInfectious disease analysisTime is of the essence…
  48. 48. LastlyCell identification
  49. 49. CycleBigScienceChallenge
  50. 50. Victor Ruotti,Morgridge Institute for Research
  51. 51. Big Data, meet Big Compute (1 million hours,78 TBs Genome data)
  52. 52. Our software aided a ComputeCentury™ In a week For ~$19,555
  53. 53. Will hopefully help withinfectious/other disease treatment
  54. 54. Every day iscrucial and costly...
  55. 55. Utility Supercomputing can help
  56. 56. To that end, Cycle wants to help you
  57. 57. Will help 3FED/SLED projectsget started this month
  58. 58. Talk to me afterwards:jstowe@cyclecomputing.com
  59. 59. Utility Supercomputing, Cycle & AWS
  60. 60. Make IT a force for good,a force for the public good
  61. 61. Make impossible BigScience, possible.
  62. 62. Thank you, Thanks AWS! Questions? Follow us at: @cyclecomputing http://www.cyclecomputing.com We’re hiring! jobs@cyclecomputing.com

×