• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Utility HPC: Right Systems, Right Scale, Right Science
 

Utility HPC: Right Systems, Right Scale, Right Science

on

  • 1,274 views

 

Statistics

Views

Total Views
1,274
Views on SlideShare
1,196
Embed Views
78

Actions

Likes
1
Downloads
6
Comments
0

3 Embeds 78

http://www.opscode.com 60
http://www.getchef.com 17
https://www.google.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Utility HPC: Right Systems, Right Scale, Right Science Utility HPC: Right Systems, Right Scale, Right Science Presentation Transcript

    • Utility HPC:Right Systems, Right Scale,Right ScienceJason Stowe, CEO@jasonastowe, @cyclecomputing
    • I’m here to recruit you,for a cause
    • We believeutility access to compute powermakes impossible science,possible.
    • Dynamic, utility access tocompute poweris as important as uptime
    • (that’s why coded infrastructureis critical)
    • Skeptical?Flickr:  Tourist  on  Earth  
    • In prior years (today?)Researchers/engineers waitedfor computing
    • For  the  horsepower  
    • For  the  place    to  put  it  
    • For  it  to  be    Configured..  Flickr: vaxomatic
    • Yesterday, high performanceengineering, science clusterswere…Too smallwhen you need it most,Too largeevery other time.
    • The Innovation Bottleneck:Researchers/Scientists/EngineersForced to size questions to theinfrastructure you have
    •  Multi-­‐tenant  systems  create  float  capacity  That  is  critical  to  innovation    
    • The60’sThe70’sThe80’sThe90’sThe00’sFrom centralized to decentralized, collaborative to independentand right back again!The10’sMainframes VAX   The  PC   Beowulf Clusters Central  Clouds  100% 60% 0% 40% ??? %SHARING  ~  0Mbit   ~ 1Mbit ~ 10Mbit ~  1000  Mbit   ~ 10,000 MbitBigger, better but further and further away from the scientist’s lab
    • Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific MethodTest and Analyze stagesrequire the most time,compute, and data
    • Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific MethodAny improvements to thiscycle yield multiplicativebenefits
    • A Challenge Across Industries— 3 of Top 5 Insurance— 6 of Top 8 Pharmaceutical— 2 of Top 3 Banks— 2 of Top 3 Genomics Sequencing— 1 of Top 2 FPGA
    • Utility HPC in the NewsWSJ, NYTimes, Wired, Bio-IT World BusinessWeek
    • To accelerate science, we needautomation
    • Management SoftwareCC1/CCGInstancesEBSS3SharedFSEBSUtility  HPC  Cluster  -­‐ Scales  to  50,000+  cores  -­‐ Data  Scheduling  -­‐ Workload  portability  Data &ApplicationAwareMovementTraditionalSchedulerMassive ScaleBased upon workloadSecure, HPCClusterUserHPCReporting &Audit
    • 50,000-core CycleCloudUsing Chef and AWSChefConf 2012
    • 10,600-instance clusteragainst cancer targetChefConf 2013
    • Created in 2 hoursConfigured with Search,with Data bags
    • one Chef 11 server
    • We make software tools to easily orchestrate complexworkloads and data access across Utility HPCToday is a survey of use cases…10,600 instanceLife ScienceMolecularModeling600 coreManufacturingNuclear PowerPlant for safetysimulationGenomicAnalysisRNA forStem Cells
    • Dynamic, utility access tocompute poweris as important as uptime
    • Why?
    • #1: “Better” Science =“Answer the question we want toask”, not constrained to what fitson local compute power
    • #2 “Faster” Science =Run this “better” science,that would have takenmonths or yearsin hours or days
    • Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
    • Life Sciences & Compute?ComputeData/BandwidthGenomicsMolecularModelingCAD/CAMAll SampleAnalysisProteomicsBiomarker/Image AnalysisSensor Data ImportCreating fakeCharts, withFake Data
    • Why is this important?
    • (W.H.O./Globocan 2008)
    • ~2 million Type 2 diabetics,~200k Type 1
    • Every day iscrucial and costly
    • Before:Trade-off compute time vs.accuracyNow:Accurate analysis, fewer falsenegatives, fasterInitialCoarseScreenHigherQualityAnalysisBestQualityProcess for Drug DesignHigherQualityAnalysisBestQuality
    • Big 10 PharmaBuilt 10,600 instance cluster($44M) in 2 hours, ran40 years of sciencein 11 hours for $4,372
    • Most Recent Utility Supercomputerserver count:
    • AWS Console view:
    • Cycle’s view of this cluster:One Chef 11 Server
    • Earlier Drug DesignNovartis discussed at BioIT2012— Needed—  Push-button Utility Supercomputer for molecularmodeling— Created—  30,000 core run across US/EU Cloud (AWS)—  10 years of compute in 8 hours for $10,000—  Found 3 compounds now in the wetlab as a result
    • —  Capacity is no longer an issue—  Hardware = software—  Testing (error handling, unit testing, etc.)e.g. Cycle spent ~$1M dollars on AWS over 5 years—  The only way to do this is to automateLessons learned
    •  Servers  are  not    house  plants    
    •  Servers  are  wheat    
    • Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
    • Nuclear Power Plant simulation
    • We don’t’ know what they’rerunning, but it has “Safety”
    • 600-core CAD/CAM3 Quarters of a year wait became 3 weeksSiteDataCorporateFirewall3 Weeks insteadOf 3 QuartersSecureHPCClusterTBs FSExternal Cloud  ~600 CPU clusterScheduledDataEngineer
    • Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
    • Gene Expression AnalysisMorgridge Institute for ResearchRun holistic comparison of all 78 terabyte stem cellRNA samples to build a unique gene expressiondatabaseMake it easier to replicate disease in petri dishes w/induced stem cells
    • 78 TB of Stem Cell RNA
    • 1 Million compute hours,115 years of computing in1 week for $19,555
    • Gene Expression AnalysisMorgridge Institute for Research— Cluster details—  5,000 to 10,000 cores for a week—  Very long individual analysis were check-pointed =Spot instance usage possible
    • Survey of Use Casesþ  Drug Designþ  CAD/CAMþ  Genomics…
    • Code can accelerate Science
    • Ask aQuestion Hypothesize PredictExperiment /Test Analyze Final Results        The Scientific Method on Utility HPCYield “Better”, “Faster”Research for less $
    • Dynamic, utility access tocompute poweris as important as uptime
    • I’m here to recruit you,for a cause
    • Contribute to Chef.Make the community better.And you will help Cyclemake impossible science,possible.
    • 2013 BigScience Challenge$10,000 of free computing to sciencebenefitting humanity2012 winner: 115yr Genomic analysisEnter at:http://cyclecomputing.com/big-science-challenge/enter
    • Thank You! Questions?