Utility HPC: Right Systems, Right Scale, Right Science

  • 1,245 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,245
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
7
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Utility HPC:Right Systems, Right Scale,Right ScienceJason Stowe, CEO@jasonastowe, @cyclecomputing
  • 2. I’m here to recruit you,for a cause
  • 3. We believeutility access to compute powermakes impossible science,possible.
  • 4. Dynamic, utility access tocompute poweris as important as uptime
  • 5. (that’s why coded infrastructureis critical)
  • 6. Skeptical?Flickr:  Tourist  on  Earth  
  • 7. In prior years (today?)Researchers/engineers waitedfor computing
  • 8. For  the  horsepower  
  • 9. For  the  place    to  put  it  
  • 10. For  it  to  be    Configured..  Flickr: vaxomatic
  • 11. Yesterday, high performanceengineering, science clusterswere…Too smallwhen you need it most,Too largeevery other time.
  • 12. The Innovation Bottleneck:Researchers/Scientists/EngineersForced to size questions to theinfrastructure you have
  • 13.  Multi-­‐tenant  systems  create  float  capacity  That  is  critical  to  innovation    
  • 14. The60’sThe70’sThe80’sThe90’sThe00’sFrom centralized to decentralized, collaborative to independentand right back again!The10’sMainframes VAX   The  PC   Beowulf Clusters Central  Clouds  100% 60% 0% 40% ??? %SHARING  ~  0Mbit   ~ 1Mbit ~ 10Mbit ~  1000  Mbit   ~ 10,000 MbitBigger, better but further and further away from the scientist’s lab
  • 15. Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific MethodTest and Analyze stagesrequire the most time,compute, and data
  • 16. Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific MethodAny improvements to thiscycle yield multiplicativebenefits
  • 17. A Challenge Across Industries— 3 of Top 5 Insurance— 6 of Top 8 Pharmaceutical— 2 of Top 3 Banks— 2 of Top 3 Genomics Sequencing— 1 of Top 2 FPGA
  • 18. Utility HPC in the NewsWSJ, NYTimes, Wired, Bio-IT World BusinessWeek
  • 19. To accelerate science, we needautomation
  • 20. Management SoftwareCC1/CCGInstancesEBSS3SharedFSEBSUtility  HPC  Cluster  -­‐ Scales  to  50,000+  cores  -­‐ Data  Scheduling  -­‐ Workload  portability  Data &ApplicationAwareMovementTraditionalSchedulerMassive ScaleBased upon workloadSecure, HPCClusterUserHPCReporting &Audit
  • 21. 50,000-core CycleCloudUsing Chef and AWSChefConf 2012
  • 22. 10,600-instance clusteragainst cancer targetChefConf 2013
  • 23. Created in 2 hoursConfigured with Search,with Data bags
  • 24. one Chef 11 server
  • 25. We make software tools to easily orchestrate complexworkloads and data access across Utility HPCToday is a survey of use cases…10,600 instanceLife ScienceMolecularModeling600 coreManufacturingNuclear PowerPlant for safetysimulationGenomicAnalysisRNA forStem Cells
  • 26. Dynamic, utility access tocompute poweris as important as uptime
  • 27. Why?
  • 28. #1: “Better” Science =“Answer the question we want toask”, not constrained to what fitson local compute power
  • 29. #2 “Faster” Science =Run this “better” science,that would have takenmonths or yearsin hours or days
  • 30. Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
  • 31. Life Sciences & Compute?ComputeData/BandwidthGenomicsMolecularModelingCAD/CAMAll SampleAnalysisProteomicsBiomarker/Image AnalysisSensor Data ImportCreating fakeCharts, withFake Data
  • 32. Why is this important?
  • 33. (W.H.O./Globocan 2008)
  • 34. ~2 million Type 2 diabetics,~200k Type 1
  • 35. Every day iscrucial and costly
  • 36. Before:Trade-off compute time vs.accuracyNow:Accurate analysis, fewer falsenegatives, fasterInitialCoarseScreenHigherQualityAnalysisBestQualityProcess for Drug DesignHigherQualityAnalysisBestQuality
  • 37. Big 10 PharmaBuilt 10,600 instance cluster($44M) in 2 hours, ran40 years of sciencein 11 hours for $4,372
  • 38. Most Recent Utility Supercomputerserver count:
  • 39. AWS Console view:
  • 40. Cycle’s view of this cluster:One Chef 11 Server
  • 41. Earlier Drug DesignNovartis discussed at BioIT2012— Needed—  Push-button Utility Supercomputer for molecularmodeling— Created—  30,000 core run across US/EU Cloud (AWS)—  10 years of compute in 8 hours for $10,000—  Found 3 compounds now in the wetlab as a result
  • 42. —  Capacity is no longer an issue—  Hardware = software—  Testing (error handling, unit testing, etc.)e.g. Cycle spent ~$1M dollars on AWS over 5 years—  The only way to do this is to automateLessons learned
  • 43.  Servers  are  not    house  plants    
  • 44.  Servers  are  wheat    
  • 45. Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
  • 46. Nuclear Power Plant simulation
  • 47. We don’t’ know what they’rerunning, but it has “Safety”
  • 48. 600-core CAD/CAM3 Quarters of a year wait became 3 weeksSiteDataCorporateFirewall3 Weeks insteadOf 3 QuartersSecureHPCClusterTBs FSExternal Cloud  ~600 CPU clusterScheduledDataEngineer
  • 49. Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
  • 50. Gene Expression AnalysisMorgridge Institute for ResearchRun holistic comparison of all 78 terabyte stem cellRNA samples to build a unique gene expressiondatabaseMake it easier to replicate disease in petri dishes w/induced stem cells
  • 51. 78 TB of Stem Cell RNA
  • 52. 1 Million compute hours,115 years of computing in1 week for $19,555
  • 53. Gene Expression AnalysisMorgridge Institute for Research— Cluster details—  5,000 to 10,000 cores for a week—  Very long individual analysis were check-pointed =Spot instance usage possible
  • 54. Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
  • 55. Code can accelerate Science
  • 56. Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific Method on Utility HPCYield “Better”, “Faster”Research for less $
  • 57. Dynamic, utility access tocompute poweris as important as uptime
  • 58. I’m here to recruit you,for a cause
  • 59. Contribute to Chef.Make the community better.And you will help Cyclemake impossible science,possible.
  • 60. 2013 BigScience Challenge$10,000 of free computing to sciencebenefitting humanity2012 winner: 115yr Genomic analysisEnter at:http://cyclecomputing.com/big-science-challenge/enter
  • 61. Thank You! Questions?