Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Monica Rut Avellino, profile picture
Uploaded byMonica Rut Avellino
1,196 views

AWS and Scientific Computing

This document provides an overview of how Amazon Web Services (AWS) can be used for high-performance computing (HPC). It discusses how AWS's large scale and flexibility allows users to provision resources on demand and pay only for what they use. Tools like AWS CloudFormation and cfnCluster make it easy to provision HPC clusters in AWS that utilize services like EC2, EBS, and VPC.

Technology◦
Download to read offline
! This!Amazon!Web!Services,!Inc.!(AWS)!package!is!provided!for!informational!purposes!only.!The!services!included! in!this!package!are!standard!commercial!services.!This!package!may!include!a!set!of!suggested!solutions!for!this! opportunity!that!are!based!on!our!limited!information,!and!should!not!be!construed!as!a!binding!offer!from!AWS.!For! current!prices!for!AWS!services,!please!refer!to!the!AWS!website!at!www.aws.amazon.com.! ! ! ! ! Amazon!Web!Services!and!! High5Performance!Computing!(HPC)!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!1! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! ! ! 20155Aug511! ! ! ! ! Prepared!by:! ! Brendan!Bouffler!<bouffler@amazon.com>! ! Amazon!Web!Services,!Inc.! 410!Terry!Avenue!North! Seattle,!WA!98109D5210! !
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!2! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! ! ! ! ! ! ! ! Why!AWS!for!High5Performance!Computing!(HPC)?! When!Amazon!Web!Services,!Inc.!(AWS)!emerged!from!the!Amazon.com!retail! business!in!2006,!the!scale!of!our!data!centers!already!rivaled!that!of!all!but!the!more! ambitious!TOP!500!sites!around!the!world,!allowing!an!efficient!exploitation!of!scale! economics!that!the!company!had!been!enjoying!for!some!time.!This!can!now!be! extended!to!our!customers!directly!through!what!was!then!a!unique!offering!of!“rentable! cycles.”! The!virtualization!technology!used! allowed!the!provisioning!of!new!servers! (and!subsequently!new!services)!in! minutes,!compared!to!what!previously! took!weeks,!meaning!that!experimenting! with!new!services!and!ideas!became!a! lowDrisk,!highDreward!opportunity!and! thus!new!ideas!faced!a!low!barrier!to! entry.!It!also!meant!that!successful!ideas! could!be!permitted!to!scale!more!quickly.! This!flexibility!is!one!of!the!immediate! reasons!why!cloud!computing—with! AWS!as!its!pioneer—has!seen!broad! success!in!the!enterprise!space!and!government!sector.! The!other!key!factor!supporting!the!cloud’s!success!stems!from!a!simply!stated!maxim! that!you!“only+pay+for+what+you+use.”!The!obverse—also!true—is!much!more! compelling:!“you+don’t+pay+for+what+you+don’t+use.”!This!represents!a!new!economic! model!that!is!already!driving!the!next!era!in!scientific!computing:!the!ability!to!procure! specialist!system!components,!in!almost!any!quantity,!for!a!finite!period!of!time!and!to! discard!them!when!the!task!is!completed.! This!payment!model!is!important!not!only!because!of!the!potential!to!save!costs!for! lightly!loaded!systems!(and!in!smallD!to!mediumDsized!deployments,!there!are!a!lot!of! those)!but!also!for!the!variety!it!offers!and!the!chance!for!experimentation.!A!researcher! can!form!a!hypothesis!that!a!particular!problem!may!be!tractable!as!a!mapDreduce!type! workload!(for!example)!and!quickly!build!and!run!the!model!to!test!it!without!ever!having! to!own!a!mapDreduce!cluster.!If!the!hypothesis!fails!the!test,!the!researcher!can!dispose! of!the!mapDreduce!facility!and!move!on,!without!penalty.!In!enterprise!parlance,!this!is! Figure!1!–!Millions!of!uncoupled,!uncorrelated! workloads!combine!to!fill!up!the!cloud.!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!3! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! called!“failing!fast,”!but!the!process!of!testing!ideas!and!allowing!them!to!fail!is! fundamental!to!science.+! The!realization!of!not!paying!for!what!you!don’t!use!is!only!really!possible!at!scale!and! is!definitely!difficult!to!selfDadminister.!The!reason,!of!course,!is!that!unused! infrastructure!still!needs!to!be!paid!for!by!someone!and!having!sufficient!capacity!of! multiple!different!shapes,!sizes,!and!strengths!available!at!any!one!time!for!adDhoc! availability!means!that!when!it’s!not!being!used!by!one!particular!user,!the!facility!owner! has!sufficient!confidence!that!another!user!will!need!or!want!it.!To!have!that!level!of! confidence!without!exact!planning!or!direct!insight!into!users’!minds,!it’s!best!to!have!a! large!number!of!users!with!multiple,!uncorrelated!workloads.!In!AWS’s!case,!we!have! more!than!one!million!such!customers—definitely!sufficient!scale!for!making!statistical! inferences!work.!In!many!ways,!the!cloud!can!really!be!seen!as!a!crowdDsourcing! mechanism!for!sharing!infrastructure.! Another!benefit!of!this! ownership!model!(or,!rather,! this!lack5of5ownership!model)! is!that!some!workloads!can! be!sped!up!dramatically.!An! embarrassingly!parallel! workload!is!characterized!by! a!lack!of!interaction!or!strong! dependency!between!the! multiple!(sometimes! thousands)!of!arbitrary! threads.!Great!examples!of! this!abound!in!the!sciences— free!energy!perturbation!in! chemistry,!ensemble!Fourier!transforms,!whole!classes!of!algorithms!in!bioinformatics,! and!Monte!Carlo!methods!all!share!this!characteristic.!This!class!of!problems!commonly! shares!the!property!that!a!single!core!laboring!for!hundreds!of!hours!will!produce!the! same!output!as!hundreds!of!cores!laboring!for!one!hour.!In!a!cloud!model!like!that! described!here,!the!cost!to!the!end!user!will!be!the!same,!but!in!one!of!these!cases,!the! workload!returns!its!result!after!an!hour!and!the!other!nearly!a!week.! The!Spot!Market! Depth—the!absolute!scale!of!resources—is!the!final!element!to!ensuring!that!highly! scalable!workloads!like!these!are!able!to!benefit.!Think!of!it!like!the!sheer!number!of! cores!available!for!compute,!for!example.!Ideally,!a!user!with!a!characteristic!need!like! that!discussed!above!would!like!the!ability!to!reach!out!and!find!an!almost!unlimited! pool!of!resources!in!order!to!scale!their!workload!up!(i.e.,!shorten!their!effective!waiting! Figure!2!–!Trading!time!for!capacity:!In!the!case!of! embarrassingly!parallel!workloads,!speed!increases!can!be! dramatic!and!virtually!free.!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!4! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! time)!by!many!multiples.!In!the!first!instance,!the!scale!of!resources!demanded!by!more! than!a!million!customers!provides!a!lot!of!depth.1 ! But!to!cope!with!some!of!those!users!suddenly!and!dramatically!departing!from!their! usual!usage!profile,!we!need!to!provide!sufficient!headroom!or!reserve!capacity!and! this,!as!we!know,!can!present!a!risk,!since!that!excess!capacity!can!effectively!cost! every!user!of!the!cloud!if!it’s!allowed!to!go!unused.!For!this!reason!(and!many!others),! AWS!operates!a!Spot!market!for!unused!compute!cores.!Similar!to!any!market,!the! buyers!place!bids!for!resources!that!are!in!varying!states!of!supply!(depending!on! demand!and!their!characteristics)!and!so!effectively!participate!in!the!price!decisions! being!made!minute!to!minute!in!the!cloud.!This!means!that!a!buyer!can!be!found!for! most!resources!available!in!the!cloud!almost!regardless!of!the!state!of!the!market.!It!will! just!be!the!price!that!varies.!For!example,!GPUDenabled!resources!might!become! especially!cheap!(“in!excess!supply”)!when!a!large,!VDIDbased!call!center!closes!for! business.!The!skills!needed!to!adapt!to!the!rigors!of!a!floating!market!like!this!can!be! trivial!for!some!and!hard!for!others.!However,!with!a!large!enough!user!base,!sufficient! numbers!of!users!will!be!present!to!develop!the!techniques!necessary!to!exploit!the! Spot!market!and!thus!find!it!interesting!in!which!to!participate.! The!scientific!community!has!been!a!very!large!and!enthusiastic!user!of!the!Spot! market,!since!scientific!computing!algorithms!span!the!length!and!breadth!of!workload! possibilities.!It!is!for!these!reasons!that!scientific!computing!is!such!a!complementary! workload!for!the!cloud.!Moreover,!for!workloads!that!are!not!timeDsensitive,!the!costs! can!be!extraordinarily!low.! Tools!for!Provisioning!HPC!Clusters!in!the!Cloud! AWS!is!built!upon!a!broad!and!deep!core!set!of!cloud!infrastructure!services!that!run! the!full!spectrum!of!requirements!for!compute!workloads.! In!the!simplest!case,!elastic!compute,!network,!and!storage!services!can!be!combined! in!a!limitless!variety!of!ways!to!create!very!complex,!resilient,!and!powerful!solutions.! The!most!common!HPC!workloads!typically!make!use!of!some!very!common! components!to!build!the!basic!HPC!structure!and!then!expand!in!interesting!new!ways! to!exploit!new!AWS!cloud!services!as!they!are!released.!AWS’s!goal!is!to!constantly! free!you!up!to!do!great!science,!rather!than!the!undifferentiated!heavy!lifting!of!building! and!running!a!cluster!of!IT!services.! The!AWS!cfnCluster!tool!(discussed!in!more!detail!below),!for!example,!makes!use!of:! •! Amazon!Elastic!Compute!Cloud!(Amazon!EC2)!compute!services!to!provide:! o! Hosts!running!CentOS!Linux!with!highDperformance!Intel!Xeon!(Ivy!Bridge! and!Haswell)!cores.! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1 !See!James!Hamilton,!our!VP!and!Distinguished!Engineer,!talking!about!the!scale!of!our!infrastructure!here:! https://www.youtube.com/watch?v=JIQETrFC_SQ.!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!5! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! o! Auto!Scaling!Groups!to!allow!for!expanding!the!number!of!compute!node! instances!when!the!cluster!is!under!load.! •! Amazon!Elastic!Block!Store!(Amazon!EBS)!to!provide!highDperformance!block! devices!that!are!then!formatted!with!the!ext4!file!system!and!ready!for!local! scratch!or!shared!file!system!uses.! •! Enhanced!networking!and!Virtual!Private!Cloud!(VPC)!to!provide!highD performance!10!Gb/s!networking!in!an!incredibly!secure!environment!to!ensure! complete!customer!privacy!and!segregation.! •! AWS!CloudFormation,!an!AWS!cloud!service!built!around!customizable! templates!that!gives!developers!and!systems!administrators!an!easy!way!to! create!and!manage!a!collection!of!related!AWS!resources,!provisioning!and! updating!them!in!an!orderly!and!predictable!fashion.! There!are!many!dozens!of!services!offered!by!AWS,!as!shown!in!the!diagram!below.! They!range!from!straightforward!infrastructure!services!to!complex!offerings!like! Amazon!Elastic!MapReduce!(Amazon!EMR)!or!highDperformance!message!services! like!Amazon!Kinesis.!New!ideas!and!new!services!are!created!every!day!by!more!than! a!million!customers!using!these!building!blocks.!Our!continual!innovation!ensures!that! customers!maintain!stateDofDtheDart!IT!infrastructure,!enhancing!performance!and! versatility.!AWS!has!launched!a!total!of!1,515!new!features!and/or!services!since! inception!in!2006.! ! Figure!3!–!High5Level!Overview!of!AWS!Cloud!Services! More!Information! The!complete!set!of!AWS!cloud!services!is!too!large!to!simply!outline!here!and!is! constantly!expanding.!The!most!upDtoDdate!reference!for!available!services!is!the!cloud! itself.!There!is!a!lot!of!good!information!available!at!aws.amazon.com,!in!our!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!6! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! documentation!at!aws.amazon.com/documentation,!and!in!the!whitepapers!we’ve! written!at!aws.amazon.com/whitepapers.! Multiple!Approaches!to!HPC!in!AWS! The!sections!below!outline!several!approaches!to!HPC!in!AWS.! cfnCluster! cfnCluster!is!AWS’s!own!set!of!openDsource!and!freely!available!tools!for!quickly! provisioning!an!HPC!cluster!in!the!cloud.!In!true!openDsource!spirit,!it’s!also!freely! modifiable!to!suit!your!own!needs.! Based!on!Python,!and!using!key!AWS!cloud!services!such!as!Amazon!EC2,!Amazon! EBS,!and!AWS!CloudFormation,!cfnCluster!allows!for!easy!creation!of!traditional!multiD node!HPC!clusters!using!CentOS!and!a!selection!of!standard!MPI!libraries!and! common!schedulers.! These!clusters!are!elastic!in!that!they!can!dynamically!expand!(up!to!your!preconfigured! limit)!to!service!a!growing!queue!of!jobs!and!shrink!(when!the!queue!is!depleted).!A! single!site!can!have!any!arbitrary!number!of!clusters!operating,!some!or!all!of!which!can! be!ephemeral—simply!ceasing!to!exist!when!they!are!no!longer!needed.!Individual! clusters!can!share!common!file!systems!or!be!completely!different,!allowing!for!some! software!stacks!to!be!“freezeDdried”!in!the!cloud!and!rehydrated!at!a!later!date!should!it! be!necessary!to!recreate!a!workload!or!revive!a!calculation!method!for!further!work.! cfnCluster!is!able!to!take!full!advantage!of!the!AWS!Spot!market,!making!it!easy!to! consume!thousands!of!potentially!inexpensive!cores!to!accelerate!progress!in!the!quiet! hours!of!the!night!when!the!rest!of!your!selected!region’s!workloads!are!sleeping—a! unique!proposition!that!science!is!truly!able!to!take!advantage!of!in!many!cases.! Bright!Cluster!Manager! Bright’s!Cluster!Manager!comes!to!the!cloud!from!a!long!history!of!managing!onD premises!traditional!HPC!clusters.! Bright!enables!the!creation!of!a!complete!cluster!in!Amazon!EC2!or!easily!extends!your! onsite!HPC!cluster!into!the!cloud.!Bright’s!Cluster!Manager!provides!a!“single!pane!of! glass”!to!both!your!onDpremises!and!cloud!resources,!enabling!you!to!dynamically!add! capacity!and!manage!cloud!nodes!as!part!of!your!onDsite!cluster.!Cloud!utilization!can! be!achieved!in!just!a!few!mouse!clicks,!without!the!need!for!expert!knowledge!of!Linux! or!cloud!computing.!Queues!can!be!created!and!resources!directed!to!the!cloud,! allowing!an!efficient!control!of!resource!provisioning!and!utilization.! A!great!advantage!of!Bright!is!that!the!software!environment!in!the!cloud!instances!is! completely!identical!to!that!which!is!provisioned!onDpremises—meaning!less!software! complexity!for!users!to!deal!with!and!fewer!points!of!management!for!the!maintainers!of! the!clusters.!Bright!is!also!able!to!take!advantage!of!the!AWS!Spot!market.!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!7! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! While!it’s!still!a!task!for!an!HPC!facility!manager!to!decide!what!workloads!should!be! enabled!for!cloud!extension!and!how/when!or!if!their!data!should!be!preDprovisioned!for! them!in!the!cloud!(since!these!aspects!of!the!workflow!are!highly!specific!to!a! workgroup),!Bright!Cluster!Manager!facilitates!the!provisioning!and!management!step,! allowing!for!a!much!easier!approach!to!managing!these!tasks.! Rescale! Rescale!operates!an!awardDwinning!cloud!simulation!and!HPC!platform!and!can!be! deployed!within!minutes!to!any!organization.! The!ScaleX!Pro™!platform!is!designed!for!independent!professionals!and!businesses!to! perform!their!complex!engineering!and!scientific!simulations!with!speed,!performance,! and!confidence.! ScaleX!Developer™!is!designed!for!external!application!developers!and!Independent! Software!Vendors!(ISVs)!to!build,!test,!and!deploy!software!directly!to!Rescale’s! platforms!and!perform!native!software!integration!with!Rescale’s!powerful!backend.! ScaleX™!Enterprise!is!the!enterprise!deployment!of!Rescale’s!industryDleading!cloud! simulation!and!HPC!platform.!ScaleX!Enterprise!features!a!unified!enterprise!simulation! platform!and!a!powerful!administrative!portal,!along!with!direct!integrations!and! management!of!onDpremises!HPC!resources,!schedulers,!and!software!licenses.! Cycle!Computing! Cycle!Computing!(Cycle)!migrates!and!orchestrates!big!data!and!big!compute! workloads!across!private!and!public!cluster!and!cloud!resources.!Cycle!has!experience! helping!customers!leverage!the!cloud!for!a!wide!variety!of!applications,!including:! ! •! OpenDsource!and!commercial!genomic!and!drug!discovery!applications!for!life! sciences.! •! Commercial!CFD!and!CAD!software!for!manufacturing.! •! Monte!Carlo!and!risk!analysis!applications!for!financial!services.! •! Internally!created!code!for!enterprise!or!government.! ! Cycle!operates!an!AWS!test!drive!facility!that!allows!our!customers!to!gain!handsDon! experience!with!real!workloads!in!order!to!explore!the!possibilities!of!what!Cycle’s! platform!can!offer.! Conclusion! The!cloud!truly!has!become!the!new!normal!in!science!and!is!delivering!huge!benefits! due!to!its!inherent!flexibility!in!conducting!experiments!and!trials!with!what!was! previously!expensive!and!difficultDtoDmanage!infrastructure.!It!permits!access!to! resources!at!a!scale!that!most!researchers!would!rarely!be!familiar!with!and!promotes!a! mechanism!for!sharing!data!and!workloads!that!leaves!everyone!free!to!focus!on!their! science.!
! AWS!and!HighDPerformance!Computing!(HPC)! ! ! 11!Aug!2015! ! Page!8! ! Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!! ! AWS!has!solution!architects!around!the!globe!who!are!very!interested!in!working!with! the!research!computing!community!to!better!understand!what!they!can!build!in!AWS.!! AWS!also!offers!a!scientific!computing!organization!that!works!every!day!to!make!the! cloud!more!and!more!useful!to!scientists!and!researchers!everywhere.! !