Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Microbial Bioinformatics in the
cloud
Introducing CLIMB
Dr Tom Connor
Cardiff University
Biological Cloud Computing Worksh...
Overview
• Background
• View from a newly
minted academic
Bioinformatician at a
“regular” University
– Bioinformatics need...
Big Data
Wave 1 Wave 2 Wave 3
2005-
09
1989-
97
2003-
07
1992-
2002
1993-98
1975-86
1937-611966-71
1967-89
1969-73
1969-81...
About Me
• 2006-2010 - PhD at
Imperial; Population
Genetics / Molecular
Epidemiology of
bacterial pathogens
• 2010-2012 - ...
Building bioinformatics capacity
• When I arrived at Cardiff, I
had the joy of working
out how I was going to do
my resear...
Key Challenges
• Infrastructure
– Storage
– HTC capacity
• Portability of software
• Portability of datasets
• But, we do ...
Advanced Research Computing @
Cardiff (ARCCA)
• 2048 core HPC cluster
• Second hand ~868
Westmere core “HTC”
partition
• 8...
…. So the first thing I did was to buy
myself some servers
• Large HPC clusters often don’t
meet our needs
• Bioinformatic...
Biologists often end up working in silos
• As a discipline we have probably been
taught to think in terms of ‘labs’,
‘grou...
Key Challenges Overall
• We need systems that allow us to rapidly and easily share
complete systems, from the perspective ...
The cloud
• All of the infrastructure
issues are much easier when
tackled at scale
• This concept led to Amazon,
followed ...
Why not use a commercial cloud?
• We often want lots of RAM – Amazon
max flavour size is ~250GB
• Prices are high ($1200/m...
Needs
• Core infrastructure now to
take advantage of new
technologies
• Systems to easily share data
• Repositories that c...
Introducing the CLoud Infrastructure
for Microbial Bioinformatics (CLIMB)
• MRC funded project to
develop Cloud
Infrastruc...
The CLIMB Consortium Are
• Professor Mark Pallen (Warwick) and Dr Sam
Sheppard (Swansea) – Joint PIs
• Professor Mark Acht...
The CLoud Infrastructure for Microbial
Bioinformatics (climb.ac.uk)
• We are creating A one stop shop
for Microbial Bioinf...
System Outline
• 4 sites
• Connected over Janet
• Different sizes of VM available; personal, standard, large memory, huge ...
CLIMB Overview
• 4 sites, running OpenStack
• Hardware procured in a two
stage process
• IBM/OCF provided
compute, Dell/re...
Hardware (per site)
• 2 router/firewalls (capable of
routing >80Gb each
• 3 Controllers
• 21x 64 vCore, 512GB RAM
nodes
• ...
Overview – 4 sites, (virtually) identical
hardware
Each site is connected to the
others over VPN tunnels.
Sites can be eas...
Flavours
• User configurable, with standard
flavours
• Regular; up to 8 vCPUs, 64GB
RAM
• xlarge; up to 16 vCPUs, 256GB
RA...
Access
• Microbial researchers will be
able to access the system
through one of two ways
– Externally, via federated acces...
Where are we now?
• Computational hardware was procured by March 2015 (~6 month process)
• Ahead of schedule - system is n...
VMs are already up
Users are already using CLIMB to do
research
Challenges
• Future Planning (CLIMB will run for 5 years,
then what?)
• Cross-Cloud Integration
• Not reinventing the whee...
The Sequencing Iceberg
All of the sequencing platforms available
now make producing large genomics
datasets relatively che...
Iceberg breaking with the Cloud?
• It is a mechanism for sharing servers
– Clouds remove the need for hardware
maintenance...
CLIMB Next Steps – and future needs
• New images/analytics tools (GVL!)
• Integration of datasets
• Expanding our userbase...
Cloud Infrastructure for Microbial
Bioinformatics
• A multi site system to provide a one-stop-bioinformatics-shop, designe...
The CLIMB Consortium Are
• Professor Mark Pallen (Warwick) and Dr Sam
Sheppard (Swansea) – Joint PIs
• Professor Mark Acht...
CLoud Infrastructure for Microbial
Bioinformatics (CLIMB)
• MRC funded project to
develop Cloud
Infrastructure for
microbi...
Upcoming SlideShare
Loading in …5
×

Climb stateoftheartintro

862 views

Published on

Presentation/outline introducing the CLIMB system presented at the NERC Cloud Computing Workshop at the CLIMB facility in Warwick, 4th June 2015

Published in: Science
  • Be the first to comment

Climb stateoftheartintro

  1. 1. Microbial Bioinformatics in the cloud Introducing CLIMB Dr Tom Connor Cardiff University Biological Cloud Computing Workshop www.climb.ac.uk @tomrconnor ; @mrcclimb
  2. 2. Overview • Background • View from a newly minted academic Bioinformatician at a “regular” University – Bioinformatics needs and challenges • Introducing CLIMB
  3. 3. Big Data Wave 1 Wave 2 Wave 3 2005- 09 1989- 97 2003- 07 1992- 2002 1993-98 1975-86 1937-611966-71 1967-89 1969-73 1969-81 1981-85 1974 1986-87 1969-73 Adapted from Mutreja, Kim, Thomson, Connor et al, Nature, 2011 Population genomics; using genomics to reconstruct the global spread of pathogens Bioinformatics; developing new approaches to analyse massive datasets From Marttinen, Hanage, Croucher, Connor, Harris, Bentley and Corander, Nucleic Acids Res. 2011 From Cheng, Connor, Sirén, Aanensen, Corander, MBE, 2013 Grand challenges; fighting antimicrobial resistance From Fookes et al, PLoS Pathogens 2011 From Reuter, Connor et al, PNAS 2014 From Okoro, Kingsley, Connor et al. Nature Genetics 2012 From He et al, Nature Genetics 2013 From Dziva, Hauser, Connor et al, I&I, 2013 Pathogen genomics: understanding how pathogens evolve
  4. 4. About Me • 2006-2010 - PhD at Imperial; Population Genetics / Molecular Epidemiology of bacterial pathogens • 2010-2012 - Post-Doc, Wellcome Trust Sanger Institute; Pathogen Genomics • 2012 – present Lecturer then Senior Lecturer, Cardiff University Hanage, Fraser, Tang, Connor, Corander, Science, 2009
  5. 5. Building bioinformatics capacity • When I arrived at Cardiff, I had the joy of working out how I was going to do my research in a new place • How to get scripts/software installed • Where to install scripts/software • And I had to do this mostly on my own • Not an unusual story
  6. 6. Key Challenges • Infrastructure – Storage – HTC capacity • Portability of software • Portability of datasets • But, we do have the University Supercomputer….
  7. 7. Advanced Research Computing @ Cardiff (ARCCA) • 2048 core HPC cluster • Second hand ~868 Westmere core “HTC” partition • 8 ‘large memory’ 128GB nodes • Lustre file system (scratch, nominally unlimited, but 50TB total space initially) • NFS /home mount (50GB maximum quota) • Freely available to University Researchers
  8. 8. …. So the first thing I did was to buy myself some servers • Large HPC clusters often don’t meet our needs • Bioinformaticians aren’t the ideal HPC users – Disruptive software needs – Disruptive usage patterns – Disruptive storage needs • Setting up our “own” system seems the most intuitive way to ensure that you have something that works
  9. 9. Biologists often end up working in silos • As a discipline we have probably been taught to think in terms of ‘labs’, ‘groups’ and ‘experiments’ being wet work • We build capacity and teams locally, and those are the resources that we use every day • For bioinformaticians this means we are likely to develop our solutions locally first, building a local group and local capacity • Our software, data set storage, LIMS etc are usually bespoke • Because our software/data is locally stored/setup – it is often less portable than wet lab methods / approaches • Bioinformaticians should be working differently
  10. 10. Key Challenges Overall • We need systems that allow us to rapidly and easily share complete systems, from the perspective of both novice users and experienced developers • We need to develop systems to share complete datasets, rather than forcing users to install loads of bits of software to reconstitute the development environment we used, or forcing us to become proper software developers • We need to lower the barrier to access for for research scientists with a limited understanding of UNIX/Computer Science • We need a system that allows us to train users on systems that they will then be able to use when they go home • We need to understand the needs of individual fields • We need to integrate activities across these needs, to avoid reinventing the wheel
  11. 11. The cloud • All of the infrastructure issues are much easier when tackled at scale • This concept led to Amazon, followed by others, offering Cloud services • A cloud infrastructure provides a mechanism to share systems/software and data, at scale • Let someone else do the admin etc, and all you have to worry about is running the software
  12. 12. Why not use a commercial cloud? • We often want lots of RAM – Amazon max flavour size is ~250GB • Prices are high ($1200/month for ~250GB RAM flavour) • Storage costs also high – 1TB on Amazon S3 costs $30/month (our current costs are £7/month) • Additionally Amazon isn’t designed to facilitate sharing of data etc between different people who have VMs • There are possible issues around T&C’s, governance etc • Even if we overcome these, often these services are too hard for novice users to make use of
  13. 13. Needs • Core infrastructure now to take advantage of new technologies • Systems to easily share data • Repositories that can make tools/methods/data available rapidly and easily • Better use of the existing RCUK/University server estate • To change the view of Biologists about working in silos
  14. 14. Introducing the CLoud Infrastructure for Microbial Bioinformatics (CLIMB) • MRC funded project to develop Cloud Infrastructure for microbial bioinformatics • ~£4M of hardware, capable of supporting >1000 individual virtual servers • Providing a core, national cyberinfrastructure for Microbial Bioinformatics
  15. 15. The CLIMB Consortium Are • Professor Mark Pallen (Warwick) and Dr Sam Sheppard (Swansea) – Joint PIs • Professor Mark Achtman (Warwick), Professor SteveBusby FRS (Birmingham), Dr Tom Connor (Cardiff)*, Professor Tim Walsh (Cardiff), Dr Robin Howe (Public Health Wales) – Co-Is • Dr Nick Loman (Birmingham)* and Dr Chris Quince (Warwick) ; MRC Research Fellows * Principal bioinformaticians architecting and designing the system
  16. 16. The CLoud Infrastructure for Microbial Bioinformatics (climb.ac.uk) • We are creating A one stop shop for Microbial Bioinformatics – Public/private cloud for use by UK academics – Standardised cloud images that implement key pipelines – Storage repository for data/images that are made available online and within our system, anywhere (‘eduroam for microbial genomics’) • We will provide access to other databases from within the system • As well as providing a place to support orphan databases and tools
  17. 17. System Outline • 4 sites • Connected over Janet • Different sizes of VM available; personal, standard, large memory, huge memory • Able to support >1,000 VMs simultaneously (1:1 vCPUs/vRAM : CPUs/RAM) • 7-8PB of object storage across 4 sites (~2-3PB usable with erasure coding) • 4-500TB of local high performance storage per site • A single system, with common log in, and between site data replication • System has been designed to enable the addition of extra nodes / Universities
  18. 18. CLIMB Overview • 4 sites, running OpenStack • Hardware procured in a two stage process • IBM/OCF provided compute, Dell/redhat provided storage • Networks provided by Brocade • Are defining a reference architecture to enable other sites to trivially be added
  19. 19. Hardware (per site) • 2 router/firewalls (capable of routing >80Gb each • 3 Controllers • 21x 64 vCore, 512GB RAM nodes • 3x 192 vCore, 3TB RAM nodes • ~500TB GPFS (local) – 4 controllers – Infiniband, with 10Gb failover • ~2TB Ceph (shared) – 27x 64TB nodes/site – Cross site replication – 10Gb Backbone
  20. 20. Overview – 4 sites, (virtually) identical hardware Each site is connected to the others over VPN tunnels. Sites can be easily added. System can use free router software and commodity hardware, pay for-software or dedicated router/firewalls Our intention is for the system to be presented to users as a single system, with single login, via Shibboleth. We are currently working on that bit  A single system makes it easy(er) to share methods and data! External clouds External databases External clouds External databases
  21. 21. Flavours • User configurable, with standard flavours • Regular; up to 8 vCPUs, 64GB RAM • xlarge; up to 16 vCPUs, 256GB RAM • Huge; up to 192 vCPUs, 3TB RAM • System also supports a scalable virtual cluster (large embarrassingly parallel projects) – 2+ nodes with 2+ vCPUs, 2-4GB RAM/vCPU • Also provides for Long Term Hosting (for orphan datasets/tools)
  22. 22. Access • Microbial researchers will be able to access the system through one of two ways – Externally, via federated access system, login via .ac.uk user login in first instance, later (hopefully) open to anyone who uses shibboleth – Internally, via user accounts setup by consortium for collaborators • Researchers will be able to provision up a set number of VMs
  23. 23. Where are we now? • Computational hardware was procured by March 2015 (~6 month process) • Ahead of schedule - system is now online and in use for research • Adopting two models for access – Access for registered users to core OpenStack system online now – “version 1.0” system providing universal access to predefined images starting with the GVL – Autumn 2015
  24. 24. VMs are already up
  25. 25. Users are already using CLIMB to do research
  26. 26. Challenges • Future Planning (CLIMB will run for 5 years, then what?) • Cross-Cloud Integration • Not reinventing the wheel • Standardising software stacks for .ac.uk clouds • Being able to embrace new technologies • Meeting cloud development needs
  27. 27. The Sequencing Iceberg All of the sequencing platforms available now make producing large genomics datasets relatively cheap and easy However, the major costs and difficulties do not lie with the generation of data, they lie with the pre-requisites for storing and analysing that data Informatics expertise Storage availability Appropriate HTC capacity These are interlinked, and expensive
  28. 28. Iceberg breaking with the Cloud? • It is a mechanism for sharing servers – Clouds remove the need for hardware maintenance and support – Storage, compute, networking are most expensive when bought one by one; building a large system represents better value for money • Sharing servers means you can have standardised systems, simplifying the process of installing and maintaining software – It provides a mechanism for software/data reuse as well as sharing – Also makes training easier; you can use the system you trained on, once you get back home • Sharing servers also makes training easier; you can use the system you trained on, once you get back home
  29. 29. CLIMB Next Steps – and future needs • New images/analytics tools (GVL!) • Integration of datasets • Expanding our userbase • Collaboration with other cloud services • Integrating with databases • Integrating with other clouds • Developing new sites • Developing the system to meet the needs of our users • Developing policy • Defining and developing security policy • Developing/setting up federated access • Possibly looking at capacity to burst out or accept bursts from other resources • Developing our training programme and outreach
  30. 30. Cloud Infrastructure for Microbial Bioinformatics • A multi site system to provide a one-stop-bioinformatics-shop, designed specifically to support Microbial researchers • For both Bioinformaticians and wet lab scientists • Combines hardware with training • Free, simple interface, easy to use • Common login • Easy data and method sharing • Already have multiple users from across UK academia and healthcare
  31. 31. The CLIMB Consortium Are • Professor Mark Pallen (Warwick) and Dr Sam Sheppard (Swansea) – Joint PIs • Professor Mark Achtman (Warwick), Professor SteveBusby FRS (Birmingham), Dr Tom Connor (Cardiff)*, Professor Tim Walsh (Cardiff), Dr Robin Howe (Public Health Wales) – Co-Is • Dr Nick Loman (Birmingham)* and Dr Chris Quince (Warwick) ; MRC Research Fellows * Principal bioinformaticians architecting and designing the system
  32. 32. CLoud Infrastructure for Microbial Bioinformatics (CLIMB) • MRC funded project to develop Cloud Infrastructure for microbial bioinformatics • ~£4M of hardware, capable of supporting >1000 individual virtual servers • Amazon/Google cloud for Academics

×