Talk given at "Cloud Computing for Systems Biology" workshop

1. The role of cloud compu.ng in big biology Deepak Singh

3. Via Reavel under a CC-BY-NC-ND license

6. life science industry

7. Credit: Bosco Ho

9. By ~Prescott under a CC-BY-NC license

10. context

11. analysis methods

12. technology

13. ? ? technology ? ?

15. back of the room

19. technology technology technology technology

20. technology tec y hn o log olo hn gy c te technology technology y nolog tech gy nolo technology tech

22. Image: Keith Allison under a CC-BY-SA license

23. inherent characteristics

24. data driven

25. multi-dimensional

26. collaborative

27. distributed

29. <amazon web services>

30. the cloud

31. has_many :definitions

32. infrastructure as a service

33. precursors

34. virtualization

35. service oriented architecure

36. distributed computing

38. Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling

39. Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling

40. Isolated Networks Monitoring Management Tools Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for Eclipse Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling

41. Your Custom Applications and Services Isolated Networks Monitoring Management Tools Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for Eclipse Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling

42. scalable

43. scalable cost effective

44. go o u s y scalable ay a P cost effective

45. scalable cost effective reliable

46. scalable cost effective reliable secure

47. Amazon EC2

48. servers on demand

49. highly scalable

51. 3000 CPU’s for one firm’s risk management application 3444JJ' !"#$%&'()'*+,'-./01.2%/' 344'+567/'(.' 8%%9%.:/' 344'JJ' I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1=' >?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

52. design for failure

54. “Everything fails, all the time” -- Werner Vogels

55. assume failure

56. assume failure design backwards

57. assume failure design backwards nothing fails

58. highly available systems

59. elastic block store

60. elastic IP

61. SQS

62. US East Region Availability Availability Zone A Zone B Availability Availability Zone C Zone D

63. data storage

64. one size does not fit all

66. Amazon S3

67. distributed object store

68. durable

69. available

70. !"#$%&'()*+ T T T

71. scalable

72. fast

73. simple

75. structured data anyone?

76. Amazon SimpleDB

77. zero administration

78. highly available

79. schema less

80. key-value store

81. Amazon Relational Data Service

82. single API call

83. MySQL database

84. automatic backup

85. scale up with API call

86. e s ur t fu

87. e s ur t fu master-slave replication data center failover

89. what do people do?

90. solve problems

91. > 1PB of data in S3

93. provide platforms & services

96. Platform as a Service http://heroku.com

97. Computation as a Service http://cyclecomputing.com

98. Computational Platforms sudo gem install cloud-crowd http://cyclecomputing.com http://wiki.github.com/documentcloud/cloud-crowd

99. http://cyclecomputing.com http://wiki.github.com/documentcloud/cloud-crowd

100. they do science Image: Matt Wood

102. 3.7 million classifications in just over three days ~15 million in less than a month >2.6 million clicks in 100 hours

103. Image via image editor under a CC-‐BY License

104. Protein Docking @ Pﬁzer http://bioteam.net

105. http://aws.amazon.com/publicdatasets/

107. </amazon web services>

108. anecdote

109. collaborative project

110. 800 GB

111. Image: Wikipedia Commons

112. weeks to get started

113. Image: Matt Wood

114. Image: Chris Dagdigian

119. gigabytes

120. terabytes

121. petabytes

122. really fast

123. constant flux

124. Image: Chris Dagdigian

125. data management is not data storage

126. masterclass Big data & Biology: The implications of petascale science Tuesday November 17 1:30PM - 3:00PM Room: PB253-254-257-258

127. “science data platform”

128. deliver data to applications

129. deliver data to people

130. typical informatics workflow

135. Via Christolakis under a CC-BY-NC-ND license

136. Via Argonne National Labs under a CC-BY-SA license

137. p p r a il le k Via Argonne National Labs under a CC-BY-SA license

138. Da ta Ap ps

139. Data Platform App Platform

142. Data Platform data services

143. application services App Platform

144. Scalable Data Platform Services APIs Getters Filters Savers WORK

145. must accommodate change

146. must scale

147. highly available

148. loosely coupled

149. dynamic

150. task-based resources

151. one project one set of resources

152. no waiting

153. Protein Docking @ Pﬁzer http://bioteam.net

154. distributed mindset

155. one approach

156. disk read/writes slow & expensive

157. data processing fast & cheap

158. distribute data parallelize reads

159. map/reduce

161. distributed data processing at scale

162. abstracting away hadoop

163. apache hive http://hadoop.apache.org/hive/

164. apache pig http://hadoop.apache.org/pig/

165. cascading http://www.cascading.org/

167. hosted hadoop service

168. hadoop easy & simple

169. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3

170. developers develop & distribute

171. scientists/analysts consume

172. CloudBurst Catalog k-mers Collect seeds End-to-end alignment

173. Mike Schatz, University of Maryland

175. Scalable Data Platform Services APIs Getters Filters Savers WORK

176. IN CONCLUSION

177. large scale biology

178. complex multidimensional data

179. whole lot of data

180. distributed collaborations

181. new computing and data architectures

182. a solution: cloud services

183. distributed

184. scalable

185. economical

186. here today

187. Thank you! deesingh@amazon.com Twi<er:@mndoci Presenta?on ideas from @mza, James Hamilton, and @lessig

Talk given at "Cloud Computing for Systems Biology" workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Talk given at "Cloud Computing for Systems Biology" workshop

Similar to Talk given at "Cloud Computing for Systems Biology" workshop (20)

More from Deepak Singh

More from Deepak Singh (17)

Recently uploaded

Recently uploaded (20)

Talk given at "Cloud Computing for Systems Biology" workshop