Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High-Performance-Computing-on-AWS-and-Industry-Simulation

High Performance Computing on AWS enables engineers, analysts, and researchers to think beyond the limitations of on-premises HPC infrastructure. AWS HPC solutions address the infrastructure capacity, secure global collaboration, technology obsolescence, and capital expenditure constraints associated with on-premises HPC clusters to give you the freedom to tackle the most challenging HPC workloads and get to your results faster. In this session. we will provide a quick overview of the services that make up the HPC on AWS solution, and share customer success stories across multiple industries, such as Financial Services and Life Sciences.

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

High-Performance-Computing-on-AWS-and-Industry-Simulation

  1. 1. S U M M I T HONGKONG
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High Performance Computing on AWS Driving innovation without infrastructure constraints David Pellerin Head WW Business Development, Hitech/Semiconductor AWS Anh Tran Sr. HPC Solutions Architect AWS
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Flexible configuration and virtually unlimited scalability to grow and shrink your infrastructure as your HPC workloads dictate, not the other way around HPC on AWS
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Why HPC on AWS Virtually unlimited infrastructure enabling scaling and agility not attainable on-premises Flexible configuration options quickly iterate resource selection and ensure cost optimization Increased collaboration with secure access to clusters around the world Better ROI Faster time to results
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Mapping HPC use cases - Fluid Dynamics/Res.Sim. - Weather Forecasting - Materials Simulations - Crash Simulations - Metagenomics - Astrophysics - Deep Learning - Coupled DL & Simulation - Seismic Processing - Animation and VFX - Semiconductor Verification - Image Processing/GIS - Genomics - Risk Simulations - Molecular Modeling - Contextual Search - Logistics Simulations Clustered (Tightly Coupled) Distributed/Grid (Loosely Coupled) Data Light Minimal requirement for high performance storage Data Heavy Benefits from access to high performance storage
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Components of a well-architected HPC environment
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T HPC on AWS solution components Automation and orchestration AWS Batch AWS ParallelCluster NICE EnginFrame Storage Amazon EBS Amazon EFS Amazon S3 Compute Amazon EC2 instances (Compute and accelerated) Amazon EC2 Spot AWS Auto Scaling Visualization NICE DCV Amazon AppStream 2.0 Networking Enhanced networking Placement groups Elastic Fabric Adapter Amazon FSx for Lustre
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High Performance Computing (HPC) on AWS Virtual Private Cloud on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache On AWS, secure and well- optimized HPC clusters can be automatically created, operated, and torn down in just minutes Encryption everywhere— With your own keys! Amazon S3 and Amazon Glacier On-premises HPC resources Corporate datacenter AWS Snowball AWS Direct Connect Thin or zero client— no local data Third-party IP providers and collaborators Machine learning and analytics
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Broadest and deepest compute platform choice Categories Capabilities Options + + = 175instance types
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Broadest choice of processors and architectures Right compute for the right application and workload Intel® Xeon® Scalable (Skylake) processor AMD EPYC processor AWS Graviton Processor Xilinx UltraScale+ FPGA NVIDIA V100 Tensor Core GPUs
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Hypervisor Management, Security, and Monitoring Storage Compute Instances Network Nitro Architecture  Nearly 100% of resources available for compute  Improved security SERVER NITRO
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Wide range of compute instances for HPC workloads General purpose Dense storage Compute optimized FPGA GPU Compute Storage optimized Graphics intensive Memory optimized High I/O R5 R5d R4 EC2 Bare Metal R5a High I/O General purpose burstable Direct access to physical server resources Virtual Private Servers Amazon Lightsail X1 X1eT2 T2 Unlimited M4 M5 M5d C5 C5n C4 I3I2 I3m G2 G3 P2 P3 P3dn F1 Compute & memory intensive Z1d Z1dmD2 H1 EC2 Elastic GPUs Graphics acceleration for EC2 instances EC2 Fleet • Simplified provisioning • Massive scale • Flexible capacity allocation
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High bandwidth compute instances: C5n HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Massively scalable performance • C5n Instances will offer up to 100 Gbps of network bandwidth • Significant improvements in maximum bandwidth, packet per seconds, and packets processing • Custom designed Nitro network cards • Purpose-built to run network bound workloads including distributed cluster and database workloads, HPC, real-time communications and video streaming Featuring
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High clock speed compute instances: Z1d HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Up to 4 GHz sustained, all-turbo performance • Z1d instances are optimized for memory-intensive, compute- intensive applications • Custom Intel Xeon Scalable processor • Up to 4 GHz sustained, all-turbo performance • Up to 385GiB DDR4 memory • Enhanced networking, up to 25 GB throughput Featuring
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High bandwidth compute instances: P3dn HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Optimized for distributed ML training • One of the most powerful GPU instance available in the cloud • Distributed machine learning training across multiple GPU instances • 100 Gbps of networking throughput • Based on NVIDIA’s latest GPU Tesla V100 with 32GB of memory each • The largest Amazon Elastic Compute Cloud (Amazon EC2) P3 instance size available
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T M5d Ideal for workloads that require a balance compute, memory, and networking. Compute intensive workloads Advanced memory intensive workloads. High compute and memory workloads. Currently the fastest available instance. local NVMe-based SSD block-level , high-speed, low-latency storage options. C5d R5d z1d$ M5d Instances C5d Instances R5d Instances z1d Instances Instances with local storage
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Comprehensive portfolio of high performance storage options Block Storage File Storage Object Storage Amazon EBS Amazon EFS gp2io1 st1 sc1 Amazon S3 HDD SSD FSx for Lustre FSx for Windows File Server S3 Multi- tiering Glacier
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Fully managed high performance shared file system: Amazon FSx for Lustre HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache High performing Parallel distributed file system Tune complex performance parameters Massively scalable performance • 100+ GiB/s throughput • Millions of IOPS • Consistent low latencies
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T High Performance Interconnect for HPC applications: Elastic Fabric Adapter (EFA) HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Scale MPI-based applications on AWS • Super-low latency for message passing • Improves application performance • Enables HPC applications to take advantage of the flexibility and elasticity of AWS Cloud • Support for industry-standard Message Passing Interface (MPI) facilities migration of existing HPC applications to AWS with no code changes
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T HPC software stack in Amazon EC2 Userspace Kernel Without EFA With EFA
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T What can EFA do?
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Easy cluster management: AWS ParallelCluster HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Create and manage HPC clusters • Simplifies deployment of HPC in the cloud, including integrating with popular HPC schedulers • Built on AWS CloudFormation, easy to modify to meet specific application or project requirements • Now integrated with AWS Batch • ParallelCluster now supports FSX Lustre and EFA
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Batch AWS Batch dynamically provisions resources Plans, schedules, and executes No batch software to install Focus on your applications and results!
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Efficient job scheduling: Multi-node parallel job support on AWS Batch HPC stack on AWS 3D graphics virtual workstation License managers and cluster head nodes with job schedulers Cloud-based, auto-scaling HPC clusters Shared file storage Storage cache Simplify your compute clusters and scale jobs across multiple instances with AWS Batch support for Multi-node Parallel (MNP) jobs Container 2 Container 4Container 3 Container 1
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS pricing principles
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Simplifying capacity and cost-optimization Reserved Instances for known/steady-state workloads Scale using On-Demand, Spot, or both AWS services make this easy and efficient Now with hibernate on-demand Amazon Elastic Container Service Amazon EMR AWS CloudFormation AWS BatchAmazon EC2 Auto Scaling Amazon Elastic Container Service for Kubernetes EC2 Fleet AWS Thinkbox
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Broad HPC partner community Application partners Infrastructure partners Technology partners Consulting partners
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Customer innovations enabled by HPC on AWS
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ® 2019 Amazon Web Services Inc. or its Affiliates. All rights reserved. QRT delivers investment strategies faster using AWS • QRT lacked the scalability and agility to use compute capacity on demand and keep pace with data growth, which meant it could not generate new alpha fast enough for its business • QRT chose to run its HPC platform on AWS, enabling the company to analyze large volumes of data, unconstrained by capacity issues and no longer needing to restrict analysis to a subset of historical data due to escalating storage costs • As a result, QRT has been able to drive better investment recommendations, bring new strategies to market in weeks instead of six months, and onboard new researchers faster Qube Research & Technologies (QRT) is a quantitative and systematic investment management firm based in London. The company offers services including wealth management, financial planning, and investment advisory to customers across the globe. We can integrate trading data from more data sources each day by scaling our HPC system on demand on AWS. – Nick Harris, Chief Technology Officer, Qube Research & Technologies “ ”
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Manage 50X the number of securities 4,000 times faster In hours, instead of months Run risk models Helping financial institutions model investment risks
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Large hedge fund: Quantitative Research The challenge A quantitative trading firm wanted to provide additional modeling capacity to its researchers responsible for creating and developing risk and trading algorithms in order to reduce the time required to see the results of their models and allow them to more rapidly improve the efficacy of their trading strategies. The solution The firm piloted an implementation of its existing grid infrastructure, which could utilize Amazon EC2 on-demand instances, significantly reducing analysis time. Reluctant to increase costs, not wanting to lose existing responsiveness to the end users, and not beholden to any regulatory or reporting deadlines, the firm turned to Amazon EC2 spot pricing. The result Today, the firm is able to deploy over 80% of its instances using spot pricing, and has achieved a 75% reduction in costs compared to the use of on-demand instances. This grid is deployed across multiple availability zones in the eastern US and can peak up to 75,000 cores. The availability of accelerated compute instances was also a key benefit as their grid relies primarily on GPU-enabled instances.
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ® 2019 Amazon Web Services Inc. or its Affiliates. All rights reserved. AQR Capital leverages AWS to research new investment signals • AQR makes investment decisions based on numerical models and systematic trading. Their on-premises compute grid could not keep up with demand, and the company needed a scalable solution in both compute and memory that was fast without long queue times, easy to use, and secure…and they didn’t want to manage a job scheduler. • AQR leveraged core application services like Amazon S3, Amazon EC2, and Amazon DynamoDB, added encryption at every level, and used AWS Batch to plan, schedule, and execute computing workflows on Spot instances. • With AWS, AQR developed a solution to provide freedom to their researchers while automating everything with virtually unlimited compute, short start times, and no queues at lower costs. AQR Capital Management is a quantitative asset management firm with $196 billion AUM with over 40 strategies, ranging from traditional to alternative products We can back test 20,000 equities times 20 years worth of data, creating an enormous amount of jobs all at once...using spot, we lowered our costs to $15 for 500 physical cores. That is an incredibly low price and something we cannot match on premises. – Michael Raposa, Head of Platform Engineering, AQR Capital “ ”
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 million core-hours of Amazon EC2 capacity 3.24 billion base pairs Complete sequencing of Saving koalas using genomics research
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T FPGA Acceleration for Genomics Acceleratingtime to science “Orlando, Fla., Oct 19, 2017 – The Children’s Hospital of Philadelphia (CHOP) and Edico Genome today set a new scientific world standard in rapidly processing whole human genomes into data files usable for researchers aiming to bring precision medicine into mainstream clinical practice. Utilizing Edico Genome’s DRAGENTM Genome Pipeline, deployed on 1,000 Amazon EC2 F1 instances on the Amazon Web Services (AWS) Cloud, 1,000 pediatric genomes were processed in two hours and 25 minutes. ” single HPC cluster of 1000 FPGAs
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 600 times faster Engineering simulations Helping to make supersonic flights mainstream
  36. 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Running HPC applications at extreme scale Acceleratingtime to innovation “Storage technology is amazingly complex and we’re constantly pushing the limits of physics and engineering to deliver next- generation capacities and technical innovation. This successful collaboration with AWS shows the extreme scale, power and agility of cloud-based HPC to help us run complex simulations for future storage architecture analysis and materials science explorations. Using AWS to easily shrink simulation time from 20 days to 8 hours allows Western Digital R&D teams to explore new designs and innovations at a pace un-imaginable just a short time ago.” – Steve Phillpott, CIO, Western Digital single HPC cluster of 1 million vCPUs
  37. 37. Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. David Pellerin Head WW Business Development AWS Anh Tran Sr. HPC Solutions Architect AWS
  38. 38. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  39. 39. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×