This document discusses using Amazon Web Services for scientific computing. It describes how AWS provides scalable, highly available, and flexible cloud computing resources including compute, storage, databases, and other services. Some key benefits highlighted include the ability to rapidly scale resources up or down as needed, pay only for what is used, and build applications without worrying about managing infrastructure. Examples are given of how AWS is used for data analysis, genomic research, and distributed computing applications.

1. Scien&fic
Compu&ng
with
Amazon
Web
Services
Deepak
Singh
NHGRI
Cloud
Compu&ng
Mee&ng,
Bal&more,
2010

2. AWS + science = win

8. scale has implications

9. data management

10. data processing

11. data sharing

12. Image: Chris Dagdigian

14. amazon web services

15. the cloud

16. has_many :definitions

17. infrastructure as a service

19. Your Custom Applications and Services
Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS Toolkit for .NET Cloud
Payments On-Demand
Parallel Processing Messaging
Content Delivery Amazon Flexible Workforce
Amazon Elastic Amazon Simple
Amazon CloudFront Payments Service Amazon Mechanical
MapReduce Queue Service (SQS)
(FPS) Turk
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

20. • Boot from EBS
• AWS Multi Factor Authentication • US West Region
• Virtual Private Cloud private beta • VPC Unlimited Beta
• Lower Reserved Instance Pricing • ELB Support in Console
• Reserved Instances in EU
• Console Support for CloudWatch • CloudFront streaming
• Elastic MapReduce
• SQS in EU • EC2 Spot Instances
• Windows 2008 Support
• RDS Launched • Lowered Prices
• New SimpleDB Features • AWS Security Center • High Memory Instances • AWS Economics Center
• FPS General Availability • Console support for Cloudfront • Reduced EC2 Pricing
• EMR Apache Hive support
• EC2 Reserved Instances • Elastic MapReduce in EU • SAS 70 Type II Audit
• EC2 with Windows
• AWS SDK for .NET
• EC2 in EU
• CloudFront Private Content
• AWS Toolkit for Eclipse • EBS Shared Snapshots • APAC announced
• SimpleDB in EU
• Monitoring in EU
• AWS Import/Export • Auto Scaling in EU
• Lower pricing tiers for Cloudfront • Elastic Load Balancing in EU
• AWS Management Console • Monitoring, Auto Scaling,
and Elastic Load Balancing • AWS Solutions Provider program
• CloudFront adds access logging

21. elasticity

22. 3000 CPU’s for one firm’s risk management application
3444JJ'
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
344'JJ'
I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1='
>?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

23. scale

24. > 1PB of data in S3

26. highly availability

27. Image: Chris Dagdigian

29. “Everything fails, all the time”
-- Werner Vogels

31. “Things will crash. Deal with it”
-- Jeff Dean

32. 2-4% of servers
will die annually
Source: Jeff Dean, LADIS 2009

33. 1-5% of disk drives
will die every year
Source: Jeff Dean, LADIS 2009

34. human errors

35. human errors
~20% admin issues have unintended consequences
Source: James Hamilton

36. scalable & available

37. assume sw/hw failure

38. design apps to be resilient

39. automation & alarming

40. US East Region !"#$%&'()*+
T T
Availability Availability
Zone A Zone B
Availability Availability T
Zone C Zone D

42. elastic load balancing
CloudWatch
auto scaling
SQS
elastic IP
elastic block store

43. flexibility

45. on-demand instances
reserved instances
spot instances

47. some implications

48. computing platforms

51. sudo gem install cloud-crowd
http://cyclecomputing.com
http://wiki.github.com/documentcloud/cloud-crowd

52. http://www.rightscale.com

53. Amazon Elastic
MapReduce
Amazon EC2 Instances
End
Deploy Application
Hadoop Hadoop Hadoop
Elastic Elastic
MapReduce MapReduce
Hadoop Hadoop Hadoop Notify
Web Console, Command
line tools Input output
dataset results
Input
S3
Output
S3
Get Results
Input Data
bucket bucket
Amazon S3

54. application platforms

55. http://heroku.com

56. http://chempedia.com/

58. Image: O’Reilly Radar

59. software distribution

60. http://www.cloudbiolinux.com/

61. http://bitbucket.org/galaxy/galaxy-central/wiki/Home

62. data distribution

64. http://aws.amazon.com/publicdatasets/

65. to conclude

67. built for scale

68. built for availability

69. shared dataspaces
common namespaces

70. task-based resources

71. new software architectures

72. new computing platforms

73. Data Platform
App Platform

74. available today

75. http://aws.amazon.com/education

76. Thank
you!
deesingh@amazon.com
Twi2er:@mndoci
Presenta7on
ideas
from
James
Hamilton,
@mza,
and
@lessig