Scientific Computing With Amazon Web Services
•
1 like•1,722 views
Researchers from around the world are increasingly using AWS for a wide-array of use cases. This presentation describes how AWS facilitates scientific collaboration and powers some of the world's largest scientific efforts, including real-world examples from NASA JPL, the European Space Agency (ESA) and CERN's CMS particle detector.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Scientific Computing With Amazon Web Services
Similar to Scientific Computing With Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Scientific Computing With Amazon Web Services
- 1. Scien&fic Compu&ng on AWS: NASA/JPL, ESA and CERN Jamie Kinney Principal Solutions Architect World Wide Public Sector jkinney@amazon.com @jamiekinney 1
- 2. ? How do researchers use AWS today? Can you run HPC on AWS? Should everything run on the cloud? How does AWS facilitate scientific collaboration? 2
- 3. AmazonWebServices AWS Global Infrastructure Application Services Networking Deployment & Administration DatabaseStorageCompute 3
- 4. Amazon EC2 4
- 6. 6
- 8. 8
- 9. 9
- 10. Elastic 10
- 13. ToThousands 13
- 15. Standard (m1) High Memory (m2,m3) High CPU (c1) 15
- 16. Intel Nehalem (cc1.4xlarge) Nvidia GPUs (cg1.4xlarge) 2TB of SSD 120,000 IOPS (hi1.4xlarge) Intel Sandy Bridge E5-2670 (cc2.8xlarge) Sandy Bridge, NUMA, 240GB RAM (cr1.4xlarge) 48 TB of ephemeral storage (hs1.8xlarge) Cluster Compute 16
- 17. 17
- 19. 10 gig E Placement Group Full Bisection EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2EC2 19
- 21. UseCases •Science-as-a-Service •Large-scale HTC (100,000+ core clusters) •Large-scale MapReduce (Hadoop/Spark/Shark) using EMR or EC2 •Small to medium-scale MPI clusters (hundreds of nodes) •Many small MPI clusters working in parallel to explore parameter space •GPGPU workloads •Dev/test of MPI workloads prior to submitting to supercomputing centers •Collaborative research environments •On-demand academic training/lab environments 21
- 22. Large Input Data Sets 22
- 23. ESAGaiaMissionOverview ESA’s Gaia is an ambitious mission to chart a three-dimensional map of the Milky Way Galaxy in order to reveal the composition, formation and evolution of our Galaxy. Gaia will repeatedly analyze and record the positions and magnitude of approximately one billion stars over the course of several years. 1 billion stars x 80 observations x 10 readouts = ~1 x 10^12 samples. 1ms processing time/sample = more than 30 years of processing 23
- 24. GaiaSolutionOverview • Purchase at the beginning of the mission for the anticipated high-water mark • Pay as you go: Launch what you need, as you need it. Turn instances off when you’re done • Purchase additional systems for redundancy • If an instance fails, turn it off and launch a replacement at no additional charge • Large-scale data reprocessing is constrained to available infrastructure. No way to accelerate jobs without additional CapEx • Need to reprocess the data within a few hours, simply launch more instances. 100 machines running for 1 hour at the same cost as 1 machine running for 100 hours • Performance constrained to processor/disk/memory available at time of procurement...for a multi- year mission • AWS frequently launches new instance types running the latest hardware. Simply restart your instances on a newer instance type and stop paying for less-capable infrastructure. • Data transfer and security policies make it difficult to collaborate with researchers located elsewhere • Easily and securely collaborate with researchers around the world 24
- 25. Many Iterations With Varying Parameters 25
- 27. 27
- 28. JPL Pasadena, CA CDSCC Canberra Deep Space Communication Complex MDSCC Madrid Deep Space Communication Complex GDSCC Goldstone Deep Space Communication Complex ARC CheMin Moffett Field, CA MSSS MARDI, MAHLI, MastCam San Diego, CA KSC IKI DAN Moscow, Russia INTA REMS Madrid, Spain LANL ChemCam Los Alamos, NM UofGuelph APXS Guelph, OntarioSwRI RAD Boulder, CO GSFC SAM Greenbelt, MD Plus hundreds of other sites around the world for Co-Is and Colleagues MSL Distributed Operations 28
- 29. Data Locality Challenges Scientist 1 retrieves data from L.A. Scientist 1 returns data to L.A. Scientist 2 retrieves data from L.A. Scientist 2 returns data to L.A. 29
- 30. AWSGlobalInfrastructure 9 regions 25 availability zones 38 edge locations 30
- 31. AWS Public Data Sets AWS.amazon.com/datasets 31
- 32. Data Locality Challenges Researcher in L.A. uploads data to the cloud Scientist 1 uses cloud resources to process data Scientist 2 retrieves data products from edge network Scientist 2 uses cloud resources to process data Global collaboration 32
- 33. 33
- 36. SpotInstances • Bid $X per hour • If current price <= bid, instance starts • If current price > bid, instance terminates • Customers pay market rate, not bid 36
- 37. U. Wisc.: CMS Particle Detector http://www.hep.wisc.edu/~dan/talks/EC2SpotForCMS.pdf 37
- 39. Amazon VPC AWS Direct Connect EC2 EC2 EC2EC2 Los Angeles Singapore Japan London Sao Paolo New York Sydney 39
- 40. 40
- 42. JPL Data Center Decider File Transfer Workers Data Processing Workers Polyphony Amazon SWF Decider Data Processing Tasks File Transfer Tasks Decision Tasks Create EC2 Instances Upload and Download File Chunks Data Processing Workers EC2 EC2 EC2 EC2 S3 42
- 43. SWF EC2 S3 SimpleDB CloudWatch IAMs ELB 5 Giga-pixels in 5 minutes! 43
- 44. EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 Ames Large, tightly- coupled MPI Large EP, smaller scale tightly-coupled MPI, dev/test, burst capacity Small scale MPI and EP NASA Researcher 44
- 45. 45
- 46. 46
- 49. Route53 49
- 51. CloudFront 51
- 53. MarsScienceLaboratory-LiveVideoStreaming Architecture Availability Zone: us-east-1a Adobe Flash Media Server Availability Zone: us-west-1b Telestream Wirecast CloudFront streaming for museum partners Adobe Flash Media Server Elastic Load Balancer Tier 2 Nginx Cache Tier 1 Nginx Cache Cloud Formation Stack Elastic Load Balancer Tier 2 Nginx Cache Tier 1 Nginx Cache Cloud Formation Stack 53
- 64. Thank You 64