Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Clouds: All fluff and no substance?


Published on

Keynote given at BOSC, 2010.

Does the hype surrounding cloud match the reality?
Can we use them to solve the problems in provisioning IT services to support next-generation sequencing?

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Clouds: All fluff and no substance?

  1. 1. Clouds: All fluff and no substance? <ul><li>Guy Coates
  2. 2. Wellcome Trust Sanger Institute
  3. 3. [email_address] </li></ul>
  4. 4. Outline <ul><li>Background
  5. 5. Cloud: Where are we at?
  6. 6. Good Fit: Web services
  7. 7. Bad Fit: HPTC compute
  8. 8. Better fit...?
  9. 9. Data management
  10. 10. Collaboration
  11. 11. Grids </li></ul>
  12. 12. The Sanger Institute <ul><li>Funded by Wellcome Trust. </li><ul><li>2 nd largest research charity in the world.
  13. 13. ~700 employees.
  14. 14. Based in Hinxton Genome Campus, Cambridge, UK. </li></ul><li>Large scale genomic research. </li><ul><li>Sequenced 1/3 of the human genome. (largest single contributor).
  15. 15. We have active cancer, malaria, pathogen and genomic variation / human health studies. </li></ul><li>All data is made publicly available. </li><ul><li>Websites, ftp, direct database. access, programmatic APIs. </li></ul></ul>
  16. 16. DNA sequencing
  17. 17. Economic Trends: <ul><li>As cost of sequencing halves every 12 months. </li><ul><li>cf Moore's Law </li></ul><li>The Human genome project: </li><ul><li>13 years.
  18. 18. 23 labs.
  19. 19. $500 Million. </li></ul><li>A Human genome today: </li><ul><li>3 days.
  20. 20. 1 machine.
  21. 21. $10,000.
  22. 22. Large centres are now doing studies with 1000s and 10,000s of genomes. </li></ul><li>Changes in sequencing technology are going to continue this trend. </li><ul><li>“Next-next” generation sequencers are on their way.
  23. 23. $500 genome is probable within 5 years. </li></ul></ul>
  24. 24. The scary graph Instrument upgrades Peak Yearly capillary sequencing
  25. 25. Managing Growth <ul><li>We have exponential growth in storage and compute. </li><ul><li>Storage /compute doubles every 12 months. </li><ul><li>2009 ~7 PB raw </li></ul></ul><li>Gigabase of sequence ≠ Gigbyte of storage. </li><ul><li>16 bytes per base for for sequence data.
  26. 26. Intermediate analysis typically need 10x disk space of the raw data. </li></ul><li>Moore's law will not save us. </li><ul><li>Transistor/disk density: T d =18 months
  27. 27. Sequencing cost: T d =12 months </li></ul></ul>
  28. 28. Cloud: Where are we at?
  29. 29. What is cloud? <ul>Informatician's view: <ul><li>On demand, virtual machines. </li></ul><ul><li>Root access, total ownership. </li></ul><ul><li>Pay-as-you-go model. </li></ul><li>Upper management view: </li><ul><li>“Free” compute we can use to solve all of the hard problems thrown up by new sequencing. </li><ul><li>(8cents/hour is almost free, right...?) </li></ul><li>Twatter/friendface use it, so it must be good. </li></ul></ul>
  30. 30. Hype Cycle Awesome! Just works...
  31. 31. Lost in the clouds...
  32. 32. Victory!
  33. 33. Where are we? ? ? ?
  34. 34. Where are we? <ul><li>We currently have three areas of activity: </li><ul><li>Web presence </li></ul><ul><li>HPTC workload </li></ul><ul><li>Active Data Warehousing </li></ul></ul>
  35. 35. Ensembl <ul><li>Ensembl is a system for genome Annotation.
  36. 36. Data visualisation (Web Presence) </li><ul><li>
  37. 37. Provides web / programmatic interfaces to genomic data.
  38. 38. 10k visitors / 126k page views per day. </li></ul><li>Compute Pipeline (HPTC Workload) </li><ul><li>Take a raw genome and run it through a compute pipeline to find genes and other features of interest.
  39. 39. Ensembl at Sanger/EBI provides automated analysis for 51 vertebrate genomes. </li></ul><ul><li>Software is Open Source (apache license).
  40. 40. Data is free for download. </li></ul><li>We have done cloud experiments with both the web site and pipeline. </li></ul>
  41. 41. Ensembl Website
  42. 43. Web Presence <ul><li>Ensembl has a worldwide audience.
  43. 44. Historically, web site performance was not great. </li><ul><li>Pages were quite heavyweight.
  44. 45. Not properly cached etc. </li></ul><li>Web team spent along time re-designing the code to make it more streamlined. </li><ul><li>Greatly improved performance. </li></ul><li>Coding can only get you so-far. </li><ul><li>If we want the website to be responsive, we need low latency.
  45. 46. “ A canna' change the laws of physics.” </li><ul><li>240ms round trip time. </li></ul><li>We need a set of geographically dispersed mirrors. </li></ul></ul>
  46. 47. <ul><li>Traditional mirror: Real machines in a co-lo facility in California.
  47. 48. Hardware was initially configured on site. </li><ul><li>16 servers, SAN storage, SAN switches, SAN management appliance, Ethernet switches, firewall, out-of-band management etc etc. </li></ul><li>Shipped to the co-lo for installation. </li><ul><li>Sent a person to California for 3 weeks.
  48. 49. Spent 1 week getting stuff into/out of customs. </li><ul><li>****ing FCC paperwork! </li></ul></ul><li>Additional infrastructure work. </li><ul><li>VPN between UK and US. </li></ul><li>Incredibly time consuming. </li><ul><li>Really don't want to end up having to send someone on a plane to the US to fix things. </li></ul></ul>
  49. 50. Usage <ul><li>Geo-IP database to point people to the nearest mirror:
  50. 51. US-West currently takes ~1/3 rd of total Ensembl web traffic. </li><ul><li>Latency down from XXXMs to XXms. </li></ul></ul>
  51. 52. Usage
  52. 53. What has this got to do with clouds?
  53. 54. <ul><li>We want an east coast US mirror to complement our west coast mirror.
  54. 55. Built the mirror in AWS. </li><ul><li>Initially a proof of concept / test-bed for virtual co-location.
  55. 56. Plan for production “real soon now”. </li></ul></ul>
  56. 57. Building a mirror on AWS <ul><li>No physical hardware. </li><ul><li>Work can start as soon as we enter our credit card numbers... </li></ul><li>Some software development / sysadmin work needed. </li><ul><li>Preparation of OS images, software stack configuration.
  57. 58. West-coast was built as an extension of Sanger internal network via VPN.
  58. 59. AWS images built as standalone systems. </li></ul><li>Significant amount of tuning required. </li><ul><li>Initial mysql performance was pretty bad, especially for the large ensembl databases. (~1TB).
  59. 60. Lots of people doing Apache/mysql on AWS, so there is a good amount of best-practice etc available. </li></ul></ul>
  60. 61. Does it work?
  61. 62. Is it cost effective? <ul><li>Lots of misleading cost statements made about cloud. </li><ul><li>“Our analysis only cost $500.”
  62. 63. “$0.085 / hr”. </li></ul><li>What are we comparing against? </li><ul><li>Doing the analysis once? Continually?
  63. 64. Buying a $2000 server?
  64. 65. Leasing a $2000 server for 3 years?
  65. 66. Using $150 of time at your local supercomputing facility?
  66. 67. Buying a $2000 of server but having to build a $1M datacentre to put it in? </li></ul><li>Requires the dreaded Total Cost of Ownership (TCO) calculation. </li><ul><li>hardware + power + cooling + facilities + admin/developers etc </li><ul><li>Incredibly hard to do. </li></ul></ul></ul>
  67. 68. Lets do it anyway... <ul><li>Comparing costs to the co-lo is simpler. </li><ul><li>power, cooling costs are all included.
  68. 69. Admin costs are the same, so we can ignore them. </li><ul><li>Same people responsible for both. </li></ul></ul><li>Cost for Co-location facility: </li><ul><li>$120,000 hardware + $51,000 /yr colo.
  69. 70. $91,000 per year (3 years hardware lifetime). </li></ul><li>Cost for AWS : </li><ul><li>$77,000 per year. </li></ul><li>Result: Estimated 16% cost saving. </li><ul><li>Good saving.
  70. 71. It is not free! </li></ul></ul>
  71. 72. Additional Benefits <ul><li>No need to deal with “real” hardware. </li><ul><li>Faster implementation.
  72. 73. No need to ship server or deal with US customs. </li></ul><li>“Free” hardware upgrades. </li><ul><li>As faster machines become available we can take advantage of them immediately.
  73. 74. No need to get tin decommissioned /re-installed at Co-lo. </li></ul><li>Website + code is packaged together. </li><ul><li>Can be conveniently given away to end users in a “ready-to-run” config.
  74. 75. Simplifies configuration for other users wanting to run Ensembl sites.
  75. 76. Configuring an ensembl site is non-trivial for non-informaticians. </li><ul><li>Cvs, mysql setup, apache configuration etc. </li></ul></ul></ul>
  76. 77. Added benefits
  77. 78. Downsides <ul><li>Packaging OS images and codes did take longer than expected. </li><ul><li>Most of the web-code refactoring to make it “mirror ready” had been done for the initial real colo. </li></ul><li>This needs to be re-done every ensembl release. </li><ul><li>Now part of the ensembl software release process. </li></ul><li>Management overhead does not necessarily go down. </li><ul><li>But it does change. </li></ul></ul>
  78. 79. Going forward <ul><li>Expect mirror to go live later this year. </li><ul><li>Far-east Amazon availability zone is also of interest. </li><ul><li>No timeframe so far. </li></ul></ul><li>“Virtual” Co-location concept will be useful for a number of other projects. </li><ul><li>Other Sanger websites? </li></ul><li>Disaster recovery. </li><ul><li>Eg replicate critical databases / storage into AWS. </li></ul></ul>
  79. 80. Hype Cycle Web services
  80. 81. Ensembl Pipeline <ul><li>HPTC element of Ensembl. </li><ul><li>Takes raw genomes and lays annotation on top. </li></ul></ul>
  82. 83. Raw Sequence -> Something useful
  83. 84. Example annotation
  84. 85. Gene Finding DNA HMM Prediction Alignment with known proteins Alignment with fragments recovered in vivo Alignment with other genes and other species
  85. 86. Compute Pipeline <ul><li>Architecture: </li><ul><li>OO perl pipeline manager.
  86. 87. Core algorithms are C.
  87. 88. 200 auxiliary binaries. </li></ul><li>Workflow: </li><ul><li>Investigator describes analysis at high level.
  88. 89. Pipeline manager splits the analysis into parallel chunks. </li><ul><li>Typically 50k-100k jobs. </li></ul><li>Sorts out the dependences and then submits jobs to a DRM. </li><ul><li>Typically LSF or SGE. </li></ul><li>Pipeline state and results are stored in a mysql database. </li></ul><li>Workflow is embarrassingly parallel. </li><ul><li>Integer, not floating point.
  89. 90. 64 bit memory address is nice, but not required. </li><ul><li>64 bit file access is required. </li></ul><li>Single threaded jobs.
  90. 91. Very IO intensive. </li></ul></ul>
  91. 92. Running the pipeline in practice <ul><li>Requires a significant amount of domain knowledge.
  92. 93. Software install is complicated. </li><ul><li>Lots of perl modules and dependencies.
  93. 94. Apache wranging if you want to run a website. </li></ul><li>Need a well tuned compute cluster. </li><ul><li>Pipeline takes ~500 CPU days for a moderate genome. </li><ul><li>Ensembl chewed up 160k CPU days last year. </li></ul><li>Code is IO bound in a number of places.
  94. 95. Typically need a high performance filesystem. </li><ul><li>Lustre, GPFS, Isilon, Ibrix etc. </li></ul><li>Need large mysql database. </li><ul><li>100GB-TB mysql instances, very high query load generated from the cluster. </li></ul></ul></ul>
  95. 96. Why Cloud? <ul><li>Provides a good example for testing HPTC capabilities of the cloud. </li></ul>
  96. 97. Why Cloud? <ul>Proof of concept <ul><li>Is HTPC is even possible in Cloud infrastructures? </li></ul><li>Coping with the big increase in data </li><ul><li>Will we be able to provision new machines/datacentre space to keep up?
  97. 98. What happens if we need to “out-source” our compute?
  98. 99. Can we be in a position to shift peaks of demand to cloud facilities? </li></ul></ul>
  99. 100. Expanding markets <ul><li>There are going to be lots of new genomes that need annotating. </li><ul><li>Sequencers moving into small labs, clinical settings.
  100. 101. Limited informatics / systems experience. </li><ul><li>Typically postdocs/PhD who have a “real” job to do. </li></ul><li>They may want to run the genebuild pipeline on their data, but they may not have the expertise to do so. </li></ul><li>We have already done all the hard work on installing the software and tuning it. </li><ul><li>Can we package up the pipeline, put it in the cloud? </li></ul><li>Goal: End user should simply be able to upload their data, insert their credit-card number, and press “GO” . </li></ul>
  101. 102. Porting HPTC code to the cloud <ul><li>Software stack / machine image. </li><ul><li>Creating images with software is reasonably straightforward.
  102. 103. No big surprises </li></ul><li>Queuing system </li><ul><li>Pipeline requires a queueing system: (LSF/SGE)
  103. 104. Getting them to run took a lot of fiddling.
  104. 105. Machines need to find each other one they are inside the cloud.
  105. 106. Building an automated “self discovering” cluster takes </li><ul><li>Hopefully others can re-use it. </li></ul></ul><li>Mysql databases </li><ul><li>Lots of best practice on how to do that on EC2. </li></ul><li>But it took time, even for experienced systems people. </li><ul><li>(You will not be firing your system-administrators just yet!). </li></ul></ul>
  106. 107. The big problem... <ul><li>Data:
  107. 108. Moving data into the cloud is hard
  108. 109. Doing stuff with data once it is in the cloud is also hard
  109. 110. If you look closely, most successful cloud projects have small amounts of data (10-100 Mbytes). </li></ul>
  110. 111. Moving data is hard <ul><li>Tools: </li><ul><li>Commonly used tools (FTP,ssh/rsync) are not suited to wide-area networks.
  111. 112. WAN tools: gridFTP/FDT/Aspera. </li></ul><li>Data transfer rates (gridFTP/FDT via our 2 Gbit/s site link). </li><ul><li>Cambridge -> EC2 East coast: 12 Mbytes/s (96 Mbits/s)
  112. 113. Cambridge -> EC2 Dublin: 25 Mbytes/s (200 Mbits/s)
  113. 114. 11 hours to move 1TB to Dublin.
  114. 115. 23 hours to move 1 TB to East coast. </li></ul><li>What speed should we get? </li><ul><li>Once we leave JANET (UK academic network) finding out what the connectivity is and what we should expect is almost impossible. </li></ul><li>Are our disks fast enough? </li><ul><li>Do you have fast enough disks at each end to keep the network full? </li></ul></ul>
  115. 116. Networking <ul><li>How do we improve data transfers across the public internet? </li><ul><li>CERN approach; don't.
  116. 117. Dedicated networking has been put in between CERN and the T1 centres who get all of the CERN data. </li></ul><li>Our collaborations are different. </li><ul><li>We have relatively short lived and fluid collaborations. (1-2 years, many institutions).
  117. 118. As more labs get sequencers, our potential collaborators also increase.
  118. 119. We need good connectivity to everywhere. </li></ul></ul>
  119. 120. Moving data in the cloud <ul><li>Compute nodes need to be able to see the data.
  120. 121. No viable global filesystems on EC2. </li><ul><li>NFS has poor scaling at the best of times.
  121. 122. EC2 has poor inter-node networking. > 8 NFS clients, everything stops. </li></ul><li>“The cloud way”: store data in S3. </li><ul><li>Web based object store. </li><ul><li>Get, put, delete objects. </li></ul><li>Not POSIX. </li><ul><li>Code needs re-writing / forking. </li></ul><li>Limitations; cannot store objects > 5GB. </li></ul><li>Nasty-hacks: </li><ul><li>Subcloud; commercial product that allows you to run a POSIX filesystem on top of S3. </li><ul><li>Interesting performance, and you are paying by the hour... </li></ul></ul></ul>
  122. 123. Compute architecture VS CPU CPU CPU Fat Network Posix Global filesystem CPU CPU CPU CPU thin network Local storage Local storage Local storage Local storage Batch schedular hadoop/S3 Data-store Data-store
  123. 124. Elephant in the room
  124. 125. Why not use map-reduce? <ul><li>Re-writing apps to use S3 or hadoop/HDFS is a real hurdle. </li><ul><li>Nobody want to re-write existing applications. </li><ul><li>They already work on our compute farm. </li></ul><li>Not an issue for new apps.
  125. 126. But hadoop apps do not exist in isolation.
  126. 127. Barrier for entry seems much lower for file-systems. </li><ul><li>We have a lot of non-expert users (this is a good thing). </li></ul></ul><li>Am I being a reactionary old fart? </li><ul><li>15 years ago clusters of PCs were not real supercomputers.
  127. 128. ...then beowulf took over the world. </li></ul><li>Big difference: porting applications between the two architectures was easy. </li><ul><li>MPI/PVM etc. </li></ul><li>Will the market provide “traditional” compute clusters in the cloud? </li></ul>
  128. 129. Hype cycle HPTC
  129. 130. Where are we? <ul><li>You cannot take an existing data-rich HPTC app and expect it to work. </li><ul><li>IO architectures are too different. </li></ul><li>There is some re-factoring going on for the ensembl pipeline. </li><ul><li>Currently on a case-by-case basis.
  130. 131. For the less-data intensive parts. </li></ul></ul>
  131. 132. Shared data archives
  132. 133. Past Collaborations Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre
  133. 134. Future Collaborations Collaborations are short term: 18 months-3 years. Sequencing Centre 3 Sequencing Centre 1 Sequencing Centre 2A Sequencing Centre 2B Federated access
  134. 135. International Cancer Genome Project <ul><li>Many cancer mutations are rare. </li><ul><li>Low signal-to-noise ratio. </li></ul><li>How do we find the rare but important mutations? </li><ul><li>Sequence lots of cancer genomes. </li></ul><li>International Cancer Genome Project. </li><ul><li>Consortia of sequencing and cancer research centres in 10 countries. </li></ul><li>Aim of the consortia. </li><ul><li>Complete genomic analysis of 50 different tumor types. (50,000 genomes). </li></ul></ul>
  135. 136. Genomics Data Unstructured data (flat files) Data size per Genome Structured data (databases) Clinical Researchers, non-infomaticians Sequencing informatics specialists Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)
  136. 137. Sharing Unstructured data <ul><li>Large data volumes, flat files.
  137. 138. Federated access. </li><ul><li>Data is not going to be in once place.
  138. 139. Single institute will have data distributed for DR / worldwide access. </li><ul><li>Some parts of the data may be on cloud stores. </li></ul></ul><li>Controlled access. </li><ul><li>Many archives will be public.
  139. 140. Some will have patient identifiable data.
  140. 141. Plan for it now. </li></ul></ul>
  141. 142. Dark Archives <ul><li>Storing data in an archive is not particularly useful. </li><ul><li>You need to be able to access the data and do something useful with it. </li></ul><li>Data in current archives is “dark”. </li><ul><li>You can put/get data, but cannot compute across it.
  142. 143. Is data in an inaccessible archive really useful? </li></ul></ul>
  143. 144. Last week's bombshell <ul><li>“We want to run out pipeline across 100TB of data currently in EGA/SRA.”
  144. 145. We will need to de-stage the data to Sanger, and then run the compute. </li><ul><li>Extra 0.5 PB of storage, 1000 cores of compute.
  145. 146. 3 month lead time.
  146. 147. ~$1.5M capex. </li></ul></ul>
  147. 148. Cloud / Computable archives <ul><li>Can we move the compute to the data? </li><ul><li>Upload workload onto VMs.
  148. 149. Put VMs on compute that is “attached” to the data. </li></ul></ul>Data CPU CPU CPU CPU Data CPU CPU CPU CPU VM
  149. 150. Practical Hurdles
  150. 151. Where does it live? <ul><li>Most of us are funded to hold data, not to fund everyone else's compute costs to. </li><ul><li>Now need to budget for raw compute power as well as disk.
  151. 152. Implement virtualisation infrastructure, billing etc. </li><ul><li>Are you legally allowed to charge?
  152. 153. Who underwrites it if nobody actually uses your service? </li></ul></ul><li>Strongly implies data has to be held on a commercial provider. </li><ul><li>Amazon etc already have billing infrastructures; why not use it.
  153. 154. Directly exposed to costs. </li><ul><li>Is the service cost effective? </li></ul></ul></ul>
  154. 155. Identity management <ul><li>Which identity management system to use for controlled access?
  155. 156. Culture shock.
  156. 157. Lots of solutions: </li><ul><ul><li>openID, shibboleth(aspis), globus/x509 etc. </li></ul></ul><li>What features are important? </li><ul><li>How much security?
  157. 158. Single sign on?
  158. 159. Delegated authentication? </li></ul><li>Finding consensus will be hard. </li></ul>
  159. 160. Networking: <ul>We still need to get data in. <ul><li>Fixing the internet is not going to be cost effective for us. </li></ul><li>Fixing the internet may be cost effective for big cloud providers. </li><ul><li>Core to their business model.
  160. 161. All we need to do is get data into Amazon, and then everyone else can get the data from there. </li></ul><li>Do we invest in a fast links to Amazon? </li><ul><li>It changes the business dynamic.
  161. 162. We have effectively tied ourselves to a single provider. </li></ul></ul>
  162. 163. Summary
  163. 164. Acknowledgements <ul><li>Phil Butcher
  164. 165. ISG Team </li><ul><li>James Beal
  165. 166. Gen-Tao Chiang
  166. 167. Pete Clapham
  167. 168. Simon Kelley </li></ul><li>1k Genomes Project </li><ul><li>Thomas Keane
  168. 169. Jim Stalker </li></ul><li>Cancer Genome Project </li><ul><li>Adam Butler
  169. 170. John Teague </li></ul></ul>
  170. 171. Backup