My talk from Cloud Futures 2010, organized by MSR

1. Scien&fic
Compu&ng
with
Amazon
Web
Services
Deepak
Singh
Cloud
Futures
2010

4. Via Reavel under a CC-BY-NC-ND license

7. life science industry

8. Credit: Bosco Ho

10. By ~Prescott under a CC-BY-NC license

12. data

13. Image: Wikipedia

15. Image
via
image
editor
under
a
CC-­‐BY
License

16. Image: Matt Wood

17. Image: NOAA

22. gigabytes

23. terabytes

24. petabytes

25. petabytes

26. ex
ab
y tes
petabytes ?

27. really fast

28. Image: http://www.broadinstitute.org/~apleite/photos.html

32. data management

33. data processing

34. data sharing

35. Image: Chris Dagdigian

36. compute & storage limited

38. amazon web services

39. the cloud

40. has_many :definitions

41. infrastructure as a service

43. Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

44. Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

45. Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS SDK for .NET Cloud
Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

46. Your Custom Applications and Services
Tools Isolated Networks
Monitoring Management
AWS Toolkit for Eclipse Amazon Virtual Private
Amazon CloudWatch AWS Management Console
AWS SDK for .NET Cloud
Messaging
Amazon Simple Payments On-Demand
Parallel Processing
Content Delivery Queue Service (SQS) Amazon Flexible Workforce
Amazon Elastic
Amazon CloudFront Amazon Simple Payments Service Amazon Mechanical
MapReduce
Notification Service (FPS) Turk
(SNS)
Compute Storage
Amazon Elastic Compute Database
Amazon Simple Amazon RDS and
Cloud (EC2) Storage Service (S3)
- Elastic Load Balancing SimpleDB
- AWS Import/Export
- Auto Scaling

51. elasticity

52. 3000 CPU’s for one firm’s risk management application
3444JJ'
!"#$%&'()'*+,'-./01.2%/'
344'+567/'(.'
8%%9%.:/'
344'JJ'
I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1='
>?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

53. scalability

54. > 1PB of data in S3

56. highly available

57. Image: Chris Dagdigian

59. “Everything fails, all the time”
-- Werner Vogels

61. “Things will crash. Deal with it”
-- Jeff Dean

62. 2-4% of servers
will die annually
Source: Jeff Dean, LADIS 2009

63. 1-5% of disk drives
will die every year
Source: Jeff Dean, LADIS 2009

64. human errors

65. human errors
~20% admin issues have unintended consequences
Source: James Hamilton

66. scalable & available

67. assume sw/hw failure

68. design apps to be resilient

69. automation & alarming

70. US East Region !"#$%&'()*+
T T
Availability Availability
Zone A Zone B
Availability Availability T
Zone C Zone D

72. elastic load balancing
CloudWatch
auto scaling
SNS
SQS
elastic IP
elastic block store

73. cost effective

74. pa
ya
sy
ou
cost effective go

75. on-demand instances
reserved instances
spot instances

77. A MAZON
V PC
A RCHITECTURE
Customer’s
isolated AWS
resources
10
.32
. 2. 0
/24 Subnets
10 10
.32 .32
.1.0 . 3. 0
/24 /24
VPN Gateway
Amazon
Secure VPN Connection
Web Services
over the Internet
Cloud
External
Your Network Customers

78. AWS + science = win

80. 3.7 million classifications in just over three days
~15 million in less than a month
>2.6 million clicks in 100 hours

82. lots and lots and lots and lots
and lots and lots of data and
lots and lots of lots of data

83. scalability & availability

84. we are data geeks not data center geeks

85. data management

86. Shaq Image: Keith Allison under a CC-BY-SA license

87. Shaq Image: Keith Allison under a CC-BY-SA license

88. Shaq Image: Keith Allison under a CC-BY-SA license

89. Shaq Image: Keith Allison under a CC-BY-SA license

90. Shaq Image: Keith Allison under a CC-BY-SA license

93. Biomarker Warehouse
pre-clinical, clinical, 3rd party data and publications
;<./5'=>?6@' !)*(%"&&'
23,3415'61789:1' !#%&$(%&&&'
+,'-./01' !"#$%"&&'
6178170' 6A.7341' B817-135'
Estimated cost: 10 TB warehouse over 3 years

94. data processing

96. http://cyclecomputing.com

97. http://web.mit.edu/stardev/cluster/

98. sudo gem install cloud-crowd
http://cyclecomputing.com
http://wiki.github.com/documentcloud/cloud-crowd

99. http://www.rightscale.com

100. XAFS
http://leonardo.phys.washington.edu/feff/

101. http://bioteam.net

102. ASSEMBLING GENOMES
140
million
454
reads
Image:
Ma)
Wood

103. BLAT @ U. PENN
Map 100 million, 100 base paired end reads
Quad core with 5 GB of RAM would take 16 days
30 high-memory instances; 32 hours; $195

104. HEAVY-ION COLLISIONS
Problem: Quark matter physics conference
imminent but no compute resources handy
Solution: NIMBUS context broker allowed
researchers to provision 300 nodes and get the
simulations done

105. BELLE MONTE CARLO
Credit: Tom Fifield

106. disk read/writes
slow & expensive

107. data processing
fast & cheap

108. distribute the data
parallel reads

110. data processing for the cloud

111. distributed file system
(HDFS)

112. map/reduce

113. Via Cloudera under a Creative Commons License

114. Via Cloudera under a Creative Commons License

115. http://www.cascading.org/

116. apache pig
http://hadoop.apache.org/pig/

117. apache hive
http://hadoop.apache.org/hive/

118. work by @peteskomoroch

120. High Throughput Sequence Analysis
Mike Schatz, University of Maryland

121. CloudBurst
Catalog k-mers Collect seeds End-to-end alignment
http://cloudburst-bio.sourceforge.net; Bioinformatics 2009 25: 1363-1369

122. Crossbow: Rapid whole
genome SNP analysis
Ben Langmead
http://bowtie-bio.sourceforge.net/crossbow/index.shtml

123. Crossbow: Rapid whole genome SNP analysis
Preprocessed reads
Map: Bowtie
Sort: Bin and partition
Reduce: SoapSNP
Langmead B, Schatz MC, Lin, J, Pop M, Salzberg SL. Genome Biol 10(11): R134.

124. Crossbow
condenses
over
1,000
hours
of
resequencing
computa:on
into
a
few
hours
without
requiring
the
user
to
own
or
operate
a
computer
cluster

125. Scalable Genome Assembly
Assembly of Large Genomes with Cloud Computing.
http://contrail-bio.sourceforge.net Schatz MC, Sommer D, Kelley D, Pop M, et al. In Preparation.

126. Amazon Elastic MapReduce
Amazon EC2 Instances
End
Deploy Application
Hadoop Hadoop Hadoop
Elastic Elastic
MapReduce MapReduce
Hadoop Hadoop Hadoop Notify
Web Console, Command
line tools Input output
dataset results
Input
S3
Output
S3
Get Results
Input Data
bucket bucket
Amazon S3

128. data storage & distribution
public & private

129. http://aws.amazon.com/publicdatasets/

131. sharing and collaboration

132. software distribution

134. http://www.cloudbiolinux.com/

135. http://usegalaxy.org/cloud

136. application platforms

137. http://heroku.com

138. http://chempedia.com/

139. Image: O’Reilly Radar

140. business models

143. to conclude

145. built for scale

146. built for availability

147. shared dataspaces
global namespaces

148. task-based resources

149. new software architectures

150. new computing platforms

151. available today

152. http://aws.amazon.com/education

153. Thank
you!
deesingh@amazon.com
Twi?er:@mndoci
Presenta2on
ideas
from
James
Hamilton,
@mza
and
@lessig