Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Next generation genomics: Petascale data in the life sciences


Published on

Keynote presentation at OGF 28.

The year 2000 saw the release of "The" human genome, the product of a the combined sequencing effort of the whole planet. In 2010, single institutions are sequencing thousands of genomes a year, producing petabytes of data. Furthermore, many of the large scale sequencing projects are based around international collaboration and consortia. The talk will explore how Grid and Cloud technologies are being used to share genomics data around the planet, revolutionizing life science research.

Published in: Technology

Next generation genomics: Petascale data in the life sciences

  1. 1. Next Generation Genomics: Petascale data in the life sciences <ul><li>Guy Coates
  2. 2. Wellcome Trust Sanger Institute
  3. 3. [email_address] </li></ul>
  4. 4. Outline <ul><li>DNA Sequencing and Informatics
  5. 5. Managing Data
  6. 6. Sharing Data
  7. 7. Adventures in the Cloud </li></ul>
  8. 8. The Sanger Institute <ul><li>Funded by Wellcome Trust. </li><ul><li>2 nd largest research charity in the world.
  9. 9. ~700 employees.
  10. 10. Based in Hinxton Genome Campus, Cambridge, UK. </li></ul><li>Large scale genomic research. </li><ul><li>Sequenced 1/3 of the human genome. (largest single contributor).
  11. 11. We have active cancer, malaria, pathogen and genomic variation / human health studies. </li></ul><li>All data is made publicly available. </li><ul><li>Websites, ftp, direct database. access, programmatic APIs. </li></ul></ul>
  12. 12. DNA sequencing
  13. 13. Next-generation Sequencing Life sciences is drowning in data from our new sequencing machines. Traditional sequencing: <ul><ul><li>96 sequencing reactions carried out per run. </li></ul></ul>Next-generation: sequencing. <ul><ul><li>52 Million reactions per run. </li></ul></ul>Machines are cheap(ish) and small. <ul><ul><li>Small labs can afford one.
  14. 14. Big labs can afford lots of them. </li></ul></ul>
  15. 15. Economic Trends: <ul><li>As cost of sequencing halves every 12 months. </li><ul><li>cf Moore's Law </li></ul><li>The Human genome project: </li><ul><li>13 years.
  16. 16. 23 labs.
  17. 17. $500 Million. </li></ul><li>A Human genome today: </li><ul><li>3 days.
  18. 18. 1 machine.
  19. 19. $10,000.
  20. 20. Large centres are now doing studies with 1000s and 10,000s of genomes. </li></ul><li>Changes in sequencing technology are going to continue this trend. </li><ul><li>“Next-next” generation sequencers are on their way.
  21. 21. $500 genome is probable within 5 years. </li></ul></ul>
  22. 22. Output Trends <ul><li>Our peak “old generation” sequencing: </li><ul><li>August 2007: 3.5 Gbases/month. </li></ul><li>Current output: </li><ul><li>Jan 2010: 4 Tbases/month. </li></ul><li>1000x increase in our sequencing output. </li><ul><li>In August 2007, total size of genbank was 200 Gbases. </li></ul><li>Improvements in chemistry continue to increase the output of machines. </li></ul>
  23. 23. The scary graph Instrument upgrades Peak Yearly capillary sequencing
  24. 24. Managing Growth <ul><li>We have exponential growth in storage and compute. </li><ul><li>Storage /compute doubles every 12 months. </li><ul><li>2009 ~7 PB raw </li></ul></ul><li>Gigabase of sequence ≠ Gigbyte of storage. </li><ul><li>16 bytes per base for for sequence data.
  25. 25. Intermediate analysis typically need 10x disk space of the raw data. </li></ul><li>Moore's law will not save us. </li><ul><li>Transistor/disk density: T d =18 months
  26. 26. Sequencing cost: T d =12 months </li></ul></ul>
  27. 27. Sequencing Informatics
  29. 29. Alignment <ul><li>Find the best match of fragments to a known genome / genomes. </li><ul><li>“grep” for DNA sequences.
  30. 30. Use more sophisticated algorithms that can do fuzzy matching. </li><ul><li>Real DNA has Insertions, deletions and mutations.
  31. 31. Typical algorithms are maq, bwa, ssaha, blast. </li></ul></ul><li>Look for differences </li><ul><li>Single base pair differences (SNP).
  32. 32. Larger insertions/deletions/mutations. </li></ul><li>Typical experiment: </li><ul><li>Compare cancer cell genomes with healthy ones. </li></ul></ul>Reference: ...TTTGCTGAAACCCAAGTGACGCCATCCAGCGTGACCACTGCATTTTTCTCGGTCATCACCAGCATTCTC.... Query: CAAGTGACGCCATCCAGCGTGACCACTGCATTTTTCT A GGTCATCACCAGCA
  33. 33. Assembly <ul><li>Assemble fragments into a complete genome. </li><ul><li>Typical experiment: collect reference genome for a new species. </li></ul><li>“De-novo” assembly. </li><ul><li>Assemble fragment with no external data.
  34. 34. Harder than it looks. </li><ul><li>Non uniform coverage, low depth, non-unique sequence (repeats). </li></ul></ul><li>Alignment based assembly. </li><ul><li>Align fragments to a related genome.
  35. 35. Starting scaffold which can then be refined. </li><ul><li>Eg H. neanderthal. is being assembled against a H. sapiens sequence. </li></ul></ul></ul>
  36. 36. Cancer Genomes <ul><li>Cancer is a disease caused by abnormalities in a cell's genome. </li></ul>
  37. 37. Mutation Details <ul><li>Lung Carcenoma genome </li><ul><li>Nature 2010 463; 184-90. </li></ul><li>22,910 mutations
  38. 38. 58 rearrangements
  39. 39. 334 copy number segments </li></ul>
  40. 40. Analysing Cancer Genomes <ul>Cancer genomes contains a lot of genetic damage. <ul><li>Many of the mutations in cancer are incidental.
  41. 41. Initial mutation disrupts the normal DNA repair/replication processes.
  42. 42. Corruption spreads through the rest of the genome. </li></ul><li>Today: Find the “driver” mutations amongst the thousands of “passengers. </li><ul><li>Identifying the driver mutations will give us new targets for therapies. </li></ul><li>Tomorrow: Analyse the cancer genome of every patient in the clinic. </li><ul><li>Variations in a patient and cancer genetic makeup play a major role in how effective a particular drugs will be.
  43. 43. Clinicians will use this information to tailor therapies. </li></ul></ul>
  44. 44. International Cancer Genome Project <ul><li>Many cancer mutations are rare. </li><ul><li>Low signal-to-noise ratio. </li></ul><li>How do we find the rare but important mutations? </li><ul><li>Sequence lots of cancer genomes. </li></ul><li>International Cancer Genome Project. </li><ul><li>Consortia of sequencing and cancer research centres in 10 countries. </li></ul><li>Aim of the consortia. </li><ul><li>Complete genomic analysis of 50 different tumor types. (50,000 genomes). </li></ul></ul>
  45. 45. Past Collaborations Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre
  46. 46. Future Collaborations Collaborations are short term: 18 months-3 years. Sequencing centre Sequencing centre Sequencing centre Sequencing centre Federated access
  47. 47. Genomics Data Unstructured data (flat files) Data size per Genome Structured data (databases) Clinical Researchers, non-infomaticians Sequencing informatics specialists Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)
  48. 48. Where can grid technologies help us? <ul><li>Managing data.
  49. 49. Sharing data.
  50. 50. Making our software resources available. </li></ul>
  51. 51. Managing Data
  52. 52. Bulk Data Structured data (databases) Unstructured data (flat files) Data size per Genome Sequencing informatics specialists Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)
  53. 53. Bulk Data Management <ul><li>We though we were really good at it. </li><ul><li>All samples that come through the sequencing lab are bar-coded and tracked (Laboratory Information Systems).
  54. 54. Sequencing machines fed into an automated analysis pipeline.
  55. 55. All the data was tracked, analysed and archived appropriately. </li></ul><li>Strict meta-data controls. </li><ul><li>Experiments do not start in the wet-lab until the investigator has supplied all the required data privacy and archiving requirements. </li><ul><li>Anonymised data -> straight into the archive.
  56. 56. Identifiable data -> private/controlled archives.
  57. 57. Some data held back until journal publication. </li></ul></ul></ul>
  58. 58. Compute farm analysis/QC pipeline Alignment/assembly suckers Data pull ... Final Repository (Oracle) 100TB / yr staging area 500 TB Seq 1 Seq 38
  59. 59. It turn out we were looking in the wrong place <ul><li>We had been focused on the sequencing pipeline. </li><ul><li>For many investigators, data coming off the end of the sequencing pipeline is where they start .
  60. 60. Investigators take the mass of finished sequence data out of the archives, onto our compute farms and “do stuff”. </li></ul><li>Huge explosion of data and disk use all over the institute. </li><ul><li>We had no idea what people were doing with their data. </li></ul></ul>
  61. 61. ... Data pull ... ? Compute farm analysis/QC pipeline assembly/alignment suckers Final Repository (Oracle) 100TB / yr staging area 500TB Seq 1 Seq 38 Compute Farm Compute farm disk Collaberators / 3 rd party sequencing Unmanged LIMS managed data
  62. 62. Accidents waiting to happen... From: <User A> (who left 12 months ago) I find the <project> directory is removed . The original directory is &quot;/scratch/ <User B> (who left 6 months ago) &quot; ..where is it ? If this problem cannot be solved ,I am afriaid that <project> cannot be released.
  63. 63. An idea whose time had come <ul>Forward thinking groups had hacked up a file tracking systems for their unstructured data. <ul><li>They could not keep track of where the results.
  64. 64. Problem exacerbated with student turnover (summer students, PhD students on rotation). </li></ul><li>Big wins with little effort. </li><ul><li>Disk space usage dropped by 2/3. </li><ul><li>Lots of individuals keeping copies of the same data set “so I know where it is”. </li></ul><li>Team leaders are happy that their data is where they thing it is. </li><ul><li>Important stuff is on filesystems that are backed up etc. </li></ul></ul><li>But: </li><ul><li>Systems are ad-hoc, quick hacks.
  65. 65. We want an institute wide, standardised system. </li><ul><li>Invest in people to maintain/develop it. </li></ul></ul></ul>
  66. 66. iRODS <ul><li>iRODS: Integrated Rule-Oriented Data System.
  67. 67. Produced by DICE (Data Intensive Cyber Environments) groups at U. North Carolina, Chapel Hill.
  68. 68. Successor to SRB. </li></ul>
  69. 69. iRODS ICAT Catalogue database Rule Engine Implements policies Irods Server Data on disk User interface WebDAV, icommands,fuse Irods Server Data in database
  70. 70. Basic Features <ul><li>Catalogue: </li><ul><li>Put data on disk and keeps a record of where it it.
  71. 71. Add query-able metadata to files. </li></ul><li>Rules engine. </li><ul><li>“Do things” to files based on file data and metadata. </li><ul><li>Eg move data between fast/archival storage. </li></ul><li>Implement policies. </li><ul><li>Experiment A data should be publicly viewable, but experiment B is restricted to certain users until 6 months after deposition. </li></ul></ul><li>Efficient. </li><ul><li>Copes with PB of data and 100,000M+ files.
  72. 72. Fast parallel data transfers across local and wide area network links. </li></ul></ul>
  73. 73. Advanced Features <ul><li>Extensible </li><ul><li>Link the system out to external services. </li><ul><li>Eg external databases holding metadata, external authentication systems. </li></ul></ul><li>Federated </li><ul><li>Physically and logically separated iRODS installs can be federated.
  74. 74. Allows user at institute A to seamlessly access data at institute B in a controlled manner.
  75. 75. Supports replication. Useful for disaster recovery/backup scenarios. </li></ul><li>Policy enforcements </li><ul><li>Enforces data sharing / data privacy rules. </li></ul></ul>
  76. 76. What are we doing with it? <ul><li>Piloting it for internal use. </li><ul><li>Help groups keep track of their data.
  77. 77. Move files between different storage pools. </li><ul><li>Fast scratch space ↔ warehouse disk ↔ Offsite DR centre. </li></ul><li>Link metadata back to our LIMs/tracking databases. </li></ul><li>We need to share data with other institutions. </li><ul><li>Public data is easy: FTP/http.
  78. 78. Controlled data is hard:
  79. 79. Encrypt files and place on private FTP dropboxes.
  80. 80. Cumbersome to manage and insecure. </li></ul><li>Proof of concept to use iRODS to provide controlled access to datasets. </li><ul><li>Will we get buy in for the community? </li></ul></ul>
  81. 81. Sharing data
  82. 82. Structured Data Structured data (databases) Unstructured data (flat files) Data size per Genome Clinical Researchers, non-infomaticians Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)
  84. 84. Functions of genes.
  85. 85. Relationships between genes (homologues, functional groups)
  86. 86. Links to the medical/scientific literature </li></ul></ul>
  87. 87. Ensembl <ul><li>Ensembl is a system for genome Annotation.
  88. 88. Compute Pipeline. </li><ul><li>Take a raw genome and run it through a compute pipeline to find genes and other features of interest.
  89. 89. Ensembl at Sanger/EBI provides automated analysis for 51 vertebrate genomes. </li></ul><li>Data visualisation. </li><ul><li>
  90. 90. Provides web interface to genomic data.
  91. 91. 10k visitors / 126k page views per day. </li></ul><li>Data access and mining. </li><ul><li>OO Perl / Java APIs.
  92. 92. Direct SQL access.
  93. 93. Bulk data download.
  94. 94. BioMart, DAS </li></ul><li>Software is Open Source (apache license).
  95. 95. Data is free for download. </li></ul>
  96. 96. Example annotation
  97. 97. Example annotation
  98. 98. Example annotation
  99. 99. Sharing data with Web Services
  100. 100. Distributed Annotation Service <ul><li>Labs may have data that they want to view with Ensembl. </li><ul><li>Put data into context with everything else. </li></ul><li>DAS is a web-services protocol that allows sharing of annotation information. </li><ul><li>Developed at Cold Spring Harbor Lab and extended by Sanger Institute and others. </li></ul><li>DAS Information; </li><ul><li>metadata: </li><ul><li>Description of the dataset, features supported.
  101. 101. This can be optionally registered/validated at </li></ul><li>Data: </li><ul><li>Object type.
  102. 102. Co-ordinates (typically genome species/version and position).
  103. 103. Stylesheet; (how should the data be displayed, eg histogram, color gradient). </li></ul></ul></ul>
  104. 104. DAS community <ul><li>Currently ~600 DAS providers spread across 45 institutions and 18 counties. </li></ul>Removal of non-responsive services
  105. 108. BioMART <ul>Provides query based access to structured data. <ul><li>Collaboration between CSHL, European Bioinformatics Institute and Ontario Institute for Cancer Research. </li></ul><li>“Tell me the function of genes that have substitution mutations in breast-cancer samples.”
  106. 109. Query requires queries across multiple databases. </li><ul><li>Mutations are stored in COSMIC, Cancer Genome database.
  107. 110. Gene function is stored in Ensembl. </li></ul><li>BioMart provides a unified entry point to these databases. </li></ul>
  108. 111. BioMART Transform / Import Query Common IDs: federatable Common IDs: federatable Oracle CSV Mysql MART MART MART XML GUI PERL SOAP/REST JAVA
  109. 117. Clouds
  110. 118. Disclaimer <ul><li>This talk will use Amazon/EC2.
  111. 119. We tested it.
  112. 120. It is not a commercial endorsement.
  113. 121. Other cloud providers exist.
  114. 122. It a short hand; feel free to insert your favourite cloud provider instead. </li></ul>
  115. 123. Cloud-ifying Ensembl <ul><li>Website </li><ul><li>LAMP stack.
  116. 124. Ports easily to Amazon.
  117. 125. Provides virtual world-wide co-lo. </li></ul><li>Compute Pipeline </li><ul><li>HPTC workload
  118. 126. Compute pipeline is a harder problem. </li></ul></ul>
  119. 127. Expanding markets <ul><li>There are going to be lots of new genomes that need annotating. </li><ul><li>Sequencers moving into small labs, clinical settings.
  120. 128. Limited informatics / systems experience. </li><ul><li>Typically postdocs/PhD who have a “real” job to do. </li></ul></ul><li>We have already done all the hard work on installing the software and tuning it. </li><ul><li>Can we package up the pipeline, put it in the cloud? </li></ul><li>Goal: End user should simply be able to upload their data, insert their credit-card number, and press “GO” . </li></ul>
  121. 129. Gene Finding DNA HMM Prediction Alignment with known proteins Alignment with fragments recovered in vivo Alignment with other genes and other species
  122. 130. Compute Pipeline <ul><li>Architecture: </li><ul><li>OO perl pipeline manager.
  123. 131. Core algorithms are C.
  124. 132. 200 auxiliary binaries. </li></ul><li>Workflow: </li><ul><li>Investigator describes analysis at high level.
  125. 133. Pipeline manager splits the analysis into parallel chunks. </li><ul><li>Typically 50k-100k jobs. </li></ul><li>Sorts out the dependences and then submits jobs to a DRM. </li><ul><li>Typically LSF or SGE. </li></ul><li>Pipeline state and results are stored in a mysql database. </li></ul><li>Workflow is embarrassingly parallel. </li><ul><li>Integer, not floating point.
  126. 134. 64 bit memory address is nice, but not required. </li><ul><li>64 bit file access is required. </li></ul><li>Single threaded jobs.
  127. 135. Very IO intensive. </li></ul></ul>
  128. 136. Running the pipeline in practice <ul><li>Requires a significant amount of domain knowledge.
  129. 137. Software install is complicated. </li><ul><li>Lots of perl modules and dependencies.
  130. 138. Apache wranging if you want to run a website. </li></ul><li>Need a well tuned compute cluster. </li><ul><li>Pipeline takes ~500 CPU days for a moderate genome. </li><ul><li>Ensembl chewed up 160k CPU days last year. </li></ul><li>Code is IO bound in a number of places.
  131. 139. Typically need a high performance filesystem. </li><ul><li>Lustre, GPFS, Isilon, Ibrix etc. </li></ul><li>Need large mysql database. </li><ul><li>100GB-TB mysql instances, very high query load generated from the cluster. </li></ul></ul></ul>
  132. 140. How does this port to cloud environments? <ul><li>Creating the software stack / machine image. </li><ul><li>Creating images with software is reasonably straightforward.
  133. 141. Getting queuing system etc running requires jumping through some hoops. </li></ul><li>Mysql databases </li><ul><li>Lots of best practice on how to do that on EC2. </li></ul><li>But it took time, even for experienced systems people. </li><ul><li>(You will not be firing your system-administrators just yet!). </li></ul></ul>
  134. 142. Moving data is hard <ul><li>Moving large amounts of data across the public internet is hard. </li><ul><li>Commonly used tools are not suited to wide-area networks. </li><ul><li>There is a reason gridFTP/FDT/Aspera exist. </li></ul></ul><li>Data transfer rates (gridFTP/FDT): </li><ul><li>Cambridge -> EC2 East coast: 12 Mbytes/s (96 Mbits/s)
  135. 143. Cambridge -> EC2 Dublin: 25 Mbytes/s (200 Mbits/s)
  136. 144. 11 hours to move 1TB to Dublin.
  137. 145. 23 hours to move 1 TB to East coast. </li></ul><li>What speed should we get? </li><ul><li>Once we leave JANET (UK academic network) finding out what the connectivity is and what we should expect is almost impossible. </li></ul></ul>
  138. 146. IO Architecture VS CPU CPU CPU Fat Network Posix Global filesystem CPU CPU CPU CPU thin network Local storage Local storage Local storage Local storage Batch schedular hadoop/S3
  139. 147. Storage / IO is hard <ul><li>No viable global filesystems on EC2.
  140. 148. NFS has poor scaling at the best of times. </li><ul><li>EC2 has poor inter-node networking. > 8 NFS clients, everything stops. </li></ul><li>“The cloud way”: store data in S3. </li><ul><li>Web based object store. </li><ul><li>Get, put, delete objects. </li></ul><li>Not POSIX. </li><ul><li>Code needs re-writing / forking. </li></ul><li>Limitations; cannot store objects > 5GB. </li></ul><li>Nasty-hacks: </li><ul><li>Subcloud; commercial product that allows you to run a POSIX filesystem on top of S3. </li><ul><li>Interesting performance, and you are paying by the hour... </li></ul></ul></ul>
  141. 149. Going forward
  142. 150. Cloud vs HPTC <ul><li>Re-writing apps to use S3 or hadoop/HDFS is a real hurdle. </li><ul><li>Not an issue for new apps.
  143. 151. But new apps do not exist in isolation.
  144. 152. Barrier for entry is much lower for file-systems. </li></ul><li>Am I being a reactionary old fart? </li><ul><li>15 years ago clusters of PCs were not real supercomputers.
  145. 153. ...then beowulf took over the world. </li></ul><li>Big difference: porting applications between the two architectures was easy. </li><ul><li>MPI/PVM etc. </li></ul><li>Will the market provide “traditional” compute clusters in the cloud? </li></ul>
  146. 154. Networking <ul><li>How do we improve data transfers across the public internet? </li><ul><li>CERN approach; don't.
  147. 155. Dedicated networking has been put in between CERN and the T1 centres who get all of the CERN data. </li></ul><li>Our collaborations are different. </li><ul><li>We have relatively short lived and fluid collaborations. (1-2 years, many institutions).
  148. 156. As more labs get sequencers, our potential collaborators also increase.
  149. 157. We need good connectivity to everywhere. </li></ul></ul>
  150. 158. Can we turn the problem on its head? <ul>Fixing the internet is not going to be cost effective for us. <li>Amazon fixing the internet may be cost effective for them. </li><ul><li>Core to their business model.
  151. 159. All we need to do is get data into Amazon, and then everyone else can get the data from there. </li></ul><li>Cloud as virtual co-location site. </li><ul><li>Mass datastores.
  152. 160. Host mirror sites for our web services. </li></ul><li>Requires us to invest in a fast links to Amazon. </li><ul><li>It changes the business dynamic.
  153. 161. We have effectively tied ourselves to a single provider. </li></ul><li>Expensive mistake if you change your mind, or your provider goes <pop> . </li></ul>
  154. 162. Identity management <ul><li>Web services for linking databases together are mature. </li><ul><li>They are currently all public. </li></ul><li>There will be demand for restricted services. </li><ul><li>Patient identifiable data. </li></ul><li>Our next big challenge. </li><ul><li>Lots of solutions: </li><ul><li>openID, shibboleth, aspis, globus etc. </li></ul><li>Finding consensus will be hard.
  155. 163. Culture shock. </li></ul></ul>
  156. 164. Acknowledgements <ul><li>Sanger Institute
  157. 165. Phil Butcher
  158. 166. ISG </li><ul><li>James Beal
  159. 167. Gen-Tao Chiang
  160. 168. Pete Clapham
  161. 169. Simon Kelley </li></ul><li>Cancer-genome Project </li><ul><li>Adam Butler
  162. 170. John Teague </li></ul></ul><ul><li>STFC </li><ul><li>David Corney
  163. 171. Jens Jensen </li></ul></ul>
  164. 172. Sites of interest <ul><li>
  165. 173.
  166. 174.
  167. 175.
  168. 176. </li></ul>