Keynote presentation at OGF 28. The year 2000 saw the release of "The" human genome, the product of a the combined sequencing effort of the whole planet. In 2010, single institutions are sequencing thousands of genomes a year, producing petabytes of data. Furthermore, many of the large scale sequencing projects are based around international collaboration and consortia. The talk will explore how Grid and Cloud technologies are being used to share genomics data around the planet, revolutionizing life science research.

2. Wellcome Trust Sanger Institute

3. [email_address]

5. Managing Data

6. Sharing Data

7. Adventures in the Cloud

9. ~700 employees.

12. DNA sequencing

14. Big labs can afford lots of them.

16. 23 labs.

18. 1 machine.

19. $10,000.

21. $500 genome is probable within 5 years.

23. The scary graph Instrument upgrades Peak Yearly capillary sequencing

26. Sequencing cost: T d =12 months

27. Sequencing Informatics

28. DNA Sequencing TCTTTATTTTAGCTGGACCAGACCAATTTTGAGGAAAGGATACAGACAGCGCCTG AAGGTATGTTCATGTACATTGTTTAGTTGAAGAGAGAAATTCATATTATTAATTA TGGTGGCTAATGCCTGTAATCCCAACTATTTGGGAGGCCAAGATGAGAGGATTGC ATAAAAAAGTTAGCTGGGAATGGTAGTGCATGCTTGTATTCCCAGCTACTCAGGAGGCTG TGCACTCCAGCTTGGGTGACACAG CAACCCTCTCTCTCTAAAAAAAAAAAAAAAAAGG AAATAATCAGTTTCCTAAGATTTTTTTCCTGAAAAATACACATTTGGTTTCA ATGAAGTAAATCG ATTTGCTTTCAAAACCTTTATATTTGAATACAAATGTACTCC 250 Million * 75-108 Base fragments Human Genome (3GBases)

38. 58 rearrangements

39. 334 copy number segments

41. Initial mutation disrupts the normal DNA repair/replication processes.

43. Clinicians will use this information to tailor therapies.

45. Past Collaborations Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre

46. Future Collaborations Collaborations are short term: 18 months-3 years. Sequencing centre Sequencing centre Sequencing centre Sequencing centre Federated access

47. Genomics Data Unstructured data (flat files) Data size per Genome Structured data (databases) Clinical Researchers, non-infomaticians Sequencing informatics specialists Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)

49. Sharing data.

50. Making our software resources available.

51. Managing Data

52. Bulk Data Structured data (databases) Unstructured data (flat files) Data size per Genome Sequencing informatics specialists Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)

54. Sequencing machines fed into an automated analysis pipeline.

56. Identifiable data -> private/controlled archives.

57. Some data held back until journal publication.

58. Compute farm analysis/QC pipeline Alignment/assembly suckers Data pull ... Final Repository (Oracle) 100TB / yr staging area 500 TB Seq 1 Seq 38

61. ... Data pull ... ? Compute farm analysis/QC pipeline assembly/alignment suckers Final Repository (Oracle) 100TB / yr staging area 500TB Seq 1 Seq 38 Compute Farm Compute farm disk Collaberators / 3 rd party sequencing Unmanged LIMS managed data

62. Accidents waiting to happen... From: <User A> (who left 12 months ago) I find the <project> directory is removed . The original directory is &quot;/scratch/ <User B> (who left 6 months ago) &quot; ..where is it ? If this problem cannot be solved ,I am afriaid that <project> cannot be released.

67. Produced by DICE (Data Intensive Cyber Environments) groups at U. North Carolina, Chapel Hill.

68. Successor to SRB.

69. iRODS ICAT Catalogue database Rule Engine Implements policies Irods Server Data on disk User interface WebDAV, icommands,fuse Irods Server Data in database

72. Fast parallel data transfers across local and wide area network links.

74. Allows user at institute A to seamlessly access data at institute B in a controlled manner.

78. Controlled data is hard:

79. Encrypt files and place on private FTP dropboxes.

81. Sharing data

82. Structured Data Structured data (databases) Unstructured data (flat files) Data size per Genome Clinical Researchers, non-infomaticians Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)

84. Functions of genes.

85. Relationships between genes (homologues, functional groups)

86. Links to the medical/scientific literature

90. Provides web interface to genomic data.

92. Direct SQL access.

93. Bulk data download.

95. Data is free for download.

96. Example annotation

97. Example annotation

98. Example annotation

99. Sharing data with Web Services

102. Co-ordinates (typically genome species/version and position).

103. Stylesheet; (how should the data be displayed, eg histogram, color gradient).

111. BioMART Transform / Import Query Common IDs: federatable Common IDs: federatable Oracle CSV Mysql MART MART MART XML GUI PERL SOAP/REST JAVA

117. Clouds

119. We tested it.

120. It is not a commercial endorsement.

121. Other cloud providers exist.

122. It a short hand; feel free to insert your favourite cloud provider instead.

124. Ports easily to Amazon.

126. Compute pipeline is a harder problem.

129. Gene Finding DNA HMM Prediction Alignment with known proteins Alignment with fragments recovered in vivo Alignment with other genes and other species

131. Core algorithms are C.

135. Very IO intensive.

143. Cambridge -> EC2 Dublin: 25 Mbytes/s (200 Mbits/s)

144. 11 hours to move 1TB to Dublin.

146. IO Architecture VS CPU CPU CPU Fat Network Posix Global filesystem CPU CPU CPU CPU thin network Local storage Local storage Local storage Local storage Batch schedular hadoop/S3

149. Going forward

151. But new apps do not exist in isolation.

156. As more labs get sequencers, our potential collaborators also increase.

157. We need good connectivity to everywhere.

163. Culture shock.

165. Phil Butcher

167. Gen-Tao Chiang

168. Pete Clapham

171. Jens Jensen

173. http://www.sanger.ac.uk/cosmic

174. http://www.biomart.org

175. http://www.biodas.org

176. http://www.icgc.org