Masterworks talk on Big Data and the implications of petascale science

1. Big Data and Biology: The implica4ons of petascale science Deepak Singh

3. Via Reavel under a CC-BY-NC-ND license

6. life science industry

7. Credit: Bosco Ho

9. By ~Prescott under a CC-BY-NC license

11. data

12. Image: Wikipedia

13. biology

14. big data

15. Source: http://www.nature.com/news/specials/bigdata/index.html

16. Image: Matt Wood

17. Hu ma ng en om e Image: Matt Wood

19. not just sequencing

21. Image: Ricardipus

24. more data

28. all hell breaks loose

29. ~100 TB/Week

30. ~100 TB/Week >2 PB/Year

38. years

39. weeks

40. days

41. days

42. mi nu tes days ?

43. gigabytes

44. terabytes

45. petabytes

46. exabytes?

47. really fast

48. Image: http://www.broadinstitute.org/~apleite/photos.html

49. single lab

50. Image: Chris Dagdigian

55. implications of scale

56. data management

57. data processing

58. data sharing

60. fundamental concepts

61. 1. architecting for scale

63. “Everything fails, all the time” -- Werner Vogels

65. “Things will crash. Deal with it” -- Jeff Dean

67. “Remember everything fails” -- Randy Shoup

68. fun with numbers

69. datacenter availability

70. Source: Uptime Institute

71. Tier I: 28.8 hours annual down4me (99.67% availability) Tier II: 22.0 hrs annual down4me (99.75% availability) Tier III: 1.6 hrs annual down4me (99.98% availability) Tier IV: 0.8 hrs annual down4me (99.99% availability) Source: Uptime Institute

72. cooling systems go down

73. power units fail

74. 2-4% of servers will die annually Source: Jeff Dean, LADIS 2009

75. 1-5% of disk drives will die every year Source: Jeff Dean, LADIS 2009

76. 2.3% AFR in population of 13,250 3.3% AFR in population of 22,400 4.2% AFR in population of 246,000 Source: James Hamilton

77. software breaks

78. human errors

79. human errors ~20% admin issues have unintended consequences Source: James Hamilton

80. achieving scalability and availability

81. partitioning

82. redundancy

83. recovery oriented computing Source: http://perspectives.mvdirona.com/, http://roc.cs.berkeley.edu/

84. assume sw/hw failure

85. design apps to be resilient

86. automation

88. Your Custom Applications and Services Isolated Networks Monitoring Management Tools Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for Eclipse Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling

89. Amazon S3

90. durable

91. available

92. !"#$%&'()*+ T T T

93. Amazon EC2

94. highly scalable

95. 3000 CPU’s for one firm’s risk management application 3444JJ' !"#$%&'()'*+,'-./01.2%/' 344'+567/'(.' 8%%9%.:/' 344'JJ' I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1=' >?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'

96. highly available systems

97. dynamic

98. fault tolerant

99. US East Region Availability Availability Zone A Zone B Availability Availability Zone C Zone D

100. 2. one size does not fit all

101. data 2. one size does not fit all ^

102. many data types

103. structured data

104. using the right data store

105. (a) feature first

106. RDBMS Oracle, SQL Server, DB2, MySQL, Postgres

107. Source: http://www.bioinformaticszen.com/

110. use a bigger computer

111. remove joins

112. scaling limits

113. (b) scale first

114. scale is highest priority

115. single RDBMS incapable

116. solution 1: data sharding

117. 10’s

118. 100’s

119. solution 2: scalable key- value store

120. scale is design point MongoDB, Project Voldermort, Cassandra, HBase, BigTable, Amazon SimpleDB, Dynamo

121. (c) simple structured storage

123. simple fast low ops cost BerkeleyDB, Tokyo Cabinet, Amazon SimpleDB

124. (d) purpose optimized stores

126. data warehousing stream processing Aster Data,Vertica, Netezza, Greenplum,VoltDB, StreamBase

127. what about files?

128. cluster file systems Lustre, GlusterFS

129. distributed file systems HDFS, GFS

130. distributed object store Amazon S3, Dynomite

134. 3. processing big data

135. disk read/writes slow & expensive

136. data processing fast & cheap

137. distribute the data parallel reads

139. data processing for the cloud

140. distributed file system (HDFS)

141. map/reduce

142. Via Cloudera under a Creative Commons License

143. Via Cloudera under a Creative Commons License

144. fault tolerance

145. massive scalability

146. petabyte scale

149. hosted hadoop service

150. hadoop easy and simple

151. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3

152. back to the science

153. basic informatics workflow

158. Via Christolakis under a CC-BY-NC-ND license

159. Via Argonne National Labs under a CC-BY-SA license

160. killer app Via Argonne National Labs under a CC-BY-SA license

161. getting the data

162. Register projects Register samples Sample prep Sequencing Analysis These slides cover work presented by Matt Wood at various conferences

164. constant change

165. flexible data capture

166. virtual fields

167. no schema

169. specify at run time

170. specify at run time (bootstrapping)

171. Sample Name Organism Concentration Source: Matt Wood

172. Source: Matt Wood

173. key value pairs

175. change happens

176. V1 V2 Sample Sample Name Name Organism Organism Concentration Concentration Origin Quality metric Source: Matt Wood

177. Source: Matt Wood

178. high throughput

179. lots of pipelines

180. scaling projects/pipelines?

181. lots of apps

182. loosely coupled

183. automation

184. scale operationally

185. be agile

186. now what?

187. Via asklar under a CC-BY license

189. many data types

190. changing data types

191. Shaq Image: Keith Allison under a CC-BY-SA license

196. ?

198. lots and lots and lots and lots and lots and lots of data and lots and lots of lots of data

199. By bitterlysweet under a CC-BY-NC-ND license

200. Source: http://bit.ly/anderson-bigdata

201. Chris Anderson doesn’t understand science

202. “more is different”

203. few data points

204. elaborate models

205. the unreasonable effectiveness of data Source: “The Unreasonable Effectiveness of Data”, Alon Halevy, Peter Norvig, and Fernando Pereira

206. simple models lots of data

208. information platform

210. information platforms at scale

211. one organization

212. 4 TB daily added (compressed)

213. 135 TB data scanned daily (compressed)

214. 15 PB data total capacity

215. ???

216. Facebook data from Ashish Thusoo’s HadoopWorld 2009 talk

217. not always that big

218. can we learn any lessons? Source: “Information Platforms and the Rise of the Data Scientist”, Jeff Hammerbacher in Beautiful Data

219. analytics platform

220. Data warehouse

221. Data warehouse is a repository of an organization's electronically stored data. Data warehouses are designed to facilitate reporting and analysis

225. ETL

226. extract

227. transform

228. load

230. 1 TB

231. MySQL --> Oracle

232. more data

233. more data types

234. changing data types

235. limit data warehouse

236. too limited

237. how do you scale and adapt?

238. 100’s of TBs

239. 1000’s of jobs

240. back to the science

241. back in the day

242. small data sets

243. flat files

244. ../ ../folder1/ ../folder2/ . . . ../folderN/ file1 file2 . . fileN

245. shared file system

246. RDBMS

247. Image: Wikimedia Commons

250. Image: Chris Dagdigian

251. need to process

252. need to analyze

253. 100’s of TBs

254. 1000’s of jobs

255. Facebook data from Ashish Thusoo’s HadoopWorld 2009 talk

257. ETL

259. data mining & analytics

261. analysts are not programmers

262. not savvy with map/reduce

263. apache hive http://hadoop.apache.org/hive/

264. manage & query data

265. manage & query data on top of Hadoop

266. work by @peteskomoroch

269. cascading http://www.cascading.org/

271. apache pig http://hadoop.apache.org/pig/

272. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3

273. hadoop and bioinformatics

274. High Throughput Sequence Analysis Mike Schatz, University of Maryland

275. Short Read Mapping

276. Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1)

277. Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale

278. Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale

279. Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale Need parallelization framework

280. CloudBurst Catalog k-mers Collect seeds End-to-end alignment

281. http://cloudburst-bio.sourceforge.net; Bioinformatics 2009 25: 1363-1369

283. Bowtie: Ultrafast short read aligner Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efﬁcient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.

284. SOAPSnp: Consensus alignment and SNP calling Ruiqiang Li,Yingrui Li, Xiaodong Fang, et al. (2009) "SNP detection for massively parallel whole-genome resequencing" Genome Res

285. Crossbow: Rapid whole genome SNP analysis Ben Langmead http://bowtie-bio.sourceforge.net/crossbow/index.shtml

287. Preprocessed reads

288. Preprocessed reads Map: Bowtie

289. Preprocessed reads Map: Bowtie Sort: Bin and partition

290. Preprocessed reads Map: Bowtie Sort: Bin and partition Reduce: SoapSNP

291. Crossbow condenses over 1,000 hours of resequencing computa:on into a few hours without requiring the user to own or operate a computer cluster

292. Comparing Genomes

293. Estimating relative evolutionary rates from sequence comparisons: Identiﬁcation of probable orthologs Admissible comparisons: A or B vs. D C vs. E Inadmissible comparisons: A or B vs. E C vs. D A B C D E species tree gene tree S. cerevisiae C. elegans

294. Estimating relative evolutionary rates from sequence comparisons: 1. Orthologs found using the Reciprocal smallest distance algorithm 2. Build alignment between two orthologs >Sequence C MSGRTILASTIAKPFQEEVTKAVKQLNFT-----PKLVGLLSNEDPAAKMYANWTGKTCESLGFKYEL-… >Sequence E MSGRTILASKVAETFNTEIINNVEEYKKTHNGQGPLLVGFLANNDPAAKMYATWTQKTSESMGFRYDL… 3. Estimate distance given a substitution matrix Phe Ala Pro Leu Thr Phe Ala µπ Pro µπ µπ µπ Leu µπ µπ µπ µπ A B C D E species tree gene tree S. cerevisiae C. elegans

295. RSD algorithm summary Genome I Genome J Ib Jc Align sequences & Calculate distances L Orthologs: Align sequences & Calculate distances H ib - jc D = 0.1 c vs. D=1.2 vs. D=0.2 a b a vs. D=0.1 vs. D=0.3 c b b b vs. D=0.9 vs. D=0.1 c c b c

296. Prof. Dennis Wall Harvard Medical School

297. Roundup is a database of orthologs and their evolutionary distances. To get started, click browse. Alternatively, you can read our documentation here. Good luck, researchers!

298. massive computational demand

299. 1000 genomes = 5,994,000 processes = 23,976,000 hours

300. 2737 years

302. compared 50+ genomes

303. trends in data sharing

304. data motion is hard

305. cloud services are a viable dataspace

306. share data

307. share applications

309. share results

310. http://aws.amazon.com/publicdatasets/

312. Data Platform App Platform

313. Data Platform App Platform

314. Scalable Data Platform Services APIs Getters Filters Savers WORK

315. to conclude

316. big data

317. change thinking

318. data management data processing data sharing

319. think distributed

320. new software architectures

321. new computing paradigms

322. cloud services

323. the cloud works

324. Thank you! deesingh@amazon.com Twi2er:@mndoci Presenta4on ideas from @mza, James Hamilton, and @lessig

Masterworks talk on Big Data and the implications of petascale science

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Masterworks talk on Big Data and the implications of petascale science

Similar to Masterworks talk on Big Data and the implications of petascale science (20)

More from Deepak Singh

More from Deepak Singh (17)

Recently uploaded

Recently uploaded (20)

Masterworks talk on Big Data and the implications of petascale science