Sanger Institute's experiences with the cloud. Given at Green Datacentre & Cloud Control 2011

2. Wellcome Trust Sanger Institute

3. [email_address]

5. ~700 employees.

8. DNA Sequencing TCTTTATTTTAGCTGGACCAGACCAATTTTGAGGAAAGGATACAGACAGCGCCTG AAGGTATGTTCATGTACATTGTTTAGTTGAAGAGAGAAATTCATATTATTAATTA TGGTGGCTAATGCCTGTAATCCCAACTATTTGGGAGGCCAAGATGAGAGGATTGC ATAAAAAAGTTAGCTGGGAATGGTAGTGCATGCTTGTATTCCCAGCTACTCAGGAGGCTG TGCACTCCAGCTTGGGTGACACAG CAACCCTCTCTCTCTAAAAAAAAAAAAAAAAAGG AAATAATCAGTTTCCTAAGATTTTTTTCCTGAAAAATACACATTTGGTTTCA ATGAAGTAAATCG ATTTGCTTTCAAAACCTTTATATTTGAATACAAATGTACTCC 250 Million * 75-108 Base fragments Human Genome (3GBases)

9. Moore's Law Compute/disk doubles every 18 months Sequencing doubles every 12 months

11. 23 labs.

13. 1 machine.

15. The scary graph Peak Yearly capillary sequencing: 30 Gbase Current weekly sequencing: 6000 Gbase

16. Our Science

18. Will improve the understanding of human genetic variation and disease. Genome Research Limited Wellcome Trust launches study of 10,000 human genomes in UK; 24 June 2010 www.sanger.ac.uk/about/press/2010/100624-uk10k.html

20. Hypertension

21. Bipolar disorder

22. Arthritis

23. Obesity

24. Diabetes (types I and II)

25. Breast cancer

26. Malaria

27. Tuberculosis

31. Malignant melanoma

33. Development of novel therapies

34. Targeting of existing therapeutics Lung Cancer and melanoma laid bare; 16 December 2009 www.sanger.ac.uk/about/press/2009/091216.html

35. IT Challenges

37. 1000$ genome*

38. *Informatics not included

39. Sequencing data flow. Alignments (200GB) Variation data (1GB) Feature (3MB) Raw data (10 TB) Sequence (500GB) Sequencer Processing/ QC Comparative analysis datastore Structured data (databases) Unstructured data (Flat files) Internet

41. 1.8 MW power draw

48. Provides web / programmatic interfaces to genomic data.

51. Data is free for download.

52. Sequencing data flow. Alignments (200GB) Variation data (1GB) Feature (3MB) Raw data (10 TB) Sequence (500GB) Sequencer Processing/ QC Comparative analysis datastore Structured data (databases) Unstructured data (Flat files) Internet HPC Compute Pipeline Web / Database infrastructure

53. TCCTCTCTTTATTTTAGCTGGACCAGACCAATTTTGAGGAAAGGATACAGACAGCGCCTG GAATTGTCAGACATATACCAAATCCCTTCTGTTGATTCTGCTGACAATCTATCTGAAAAA TTGGAAAGGTATGTTCATGTACATTGTTTAGTTGAAGAGAGAAATTCATATTATTAATTA TTTAGAGAAGAGAAAGCAAACATATTATAAGTTTAATTCTTATATTTAAAAATAGGAGCC AAGTATGGTGGCTAATGCCTGTAATCCCAACTATTTGGGAGGCCAAGATGAGAGGATTGC TTGAGACCAGGAGTTTGATACCAGCCTGGGCAACATAGCAAGATGTTATCTCTACACAAA ATAAAAAAGTTAGCTGGGAATGGTAGTGCATGCTTGTATTCCCAGCTACTCAGGAGGCTG AAGCAGGAGGGTTACTTGAGCCCAGGAGTTTGAGGTTGCAGTGAGCTATGATTGTGCCAC TGCACTCCAGCTTGGGTGACACAGCAAAACCCTCTCTCTCTAAAAAAAAAAAAAAAAAGG AACATCTCATTTTCACACTGAAATGTTGACTGAAATCATTAAACAATAAAATCATAAAAG AAAAATAATCAGTTTCCTAAGAAATGATTTTTTTTCCTGAAAAATACACATTTGGTTTCA GAGAATTTGTCTTATTAGAGACCATGAGATGGATTTTGTGAAAACTAAAGTAACACCATT ATGAAGTAAATCGTGTATATTTGCTTTCAAAACCTTTATATTTGAATACAAATGTACTCC

54. Annotation

55. Annotation

56. Why Cloud?

60. We need a set of geographically dispersed mirrors.

65. What we actually did: AWS Sanger Sanger VPN

68. Lots of people doing Apache/mysql on AWS, so there is a good amount of best-practice etc available.

69. Traffic

72. Buying a $2000 server?

73. Leasing a $2000 server for 3 years?

74. Using $150 of time at your local supercomputing facility?

79. It is not free!

84. Ensembl team now has a dedicated person responsible for the cloud.

87. HPC Workloads

90. Longer than the science can wait.

92. Large clusters for a short amount of time.

94. Informatics / analysis tools needs to be commodity too.

96. Life Sciences HPC Workloads Tightly Coupled (MPI) Embarrassingly Parallel CPU Bound IO Bound modelling/ docking Simulation Genomics

98. 10,000s of jobs.

99. Core algorithms in C

100. Perl pipeline manager to generate and manage workflow.

101. Batch schedular to execute jobs on nodes.

103. Life Sciences HPC Workloads Tightly Coupled (MPI) Embarrassingly Parallel CPU Bound IO Bound modelling/ docking Simulation Genomics

104. Different Architectures VS CPU CPU CPU Fat Network POSIX Global filesystem CPU CPU CPU CPU thin network Local storage Local storage Local storage Local storage Batch schedular S3 Hadoop?

105. Life Sciences HPC Workloads Tightly Coupled (MPI) Embarrassingly Parallel CPU Bound IO Bound modelling/ docking Simulation Genomics

107. Vast effort to port code, for little benefit.

110. Cambridge -> EC2 Dublin: 25 Mbytes/s (200 Mbits/s)

111. 11 hours to move 1TB to Dublin.

117. Small data / high CPU applications work really well.

118. Large data applications less well.

119. Data Security

121. “Our data is confidential/important/critical to our business.

122. We must keep our data on our computers.”

123. “Apart from when we outsource it already.”

127. More sequencing experiments are trying to link with phenotype data. (ie personally identifiable medical records).

129. Traditional Collaboration DCC: Sequencing Centre + Archive Sequencing centre Sequencing centre Sequencing centre Sequencing centre IT IT IT IT

131. Is data in an inaccessible archive really useful?

132. Private Cloud Collaborations Sequencing Centre Sequencing centre Sequencing centre Sequencing centre Private Cloud IaaS / SaaS Private Cloud IaaS / SaaS

138. James Beal

139. Pete Clapham

140. Simon Kelley

142. Jan-Hinnerk Vogel

144. Glenn Proctor

145. Steve Keenan