Your SlideShare is downloading. ×
0
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Sanger HPC infrastructure  Report (2007)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Sanger HPC infrastructure Report (2007)

974

Published on

Overview of the Sanger Institute HPC infrastructure and mangement tools, given at the Spring 2007 Hepix meeting. …

Overview of the Sanger Institute HPC infrastructure and mangement tools, given at the Spring 2007 Hepix meeting.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
974
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
28
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Sanger Institute Site Report Nov 2007 Guy Coates [email_address]
  • 2. About the Institute <ul><li>Funded by Wellcome Trust. </li></ul><ul><ul><li>2 nd largest research charity in the world. </li></ul></ul><ul><ul><li>~700 employees. </li></ul></ul><ul><li>Large scale genomic research. </li></ul><ul><ul><li>Sequenced 1/3 of the human genome (largest single contributor). </li></ul></ul><ul><ul><li>We have active cancer, malaria, pathogen and genomic variation studies. </li></ul></ul><ul><li>All data is made publicly available. </li></ul><ul><ul><li>Websites, ftp, direct database. access, programmatic APIs. </li></ul></ul>
  • 3. Why are we here? <ul><li>HEPIX Themes: </li></ul><ul><ul><li>“We have particle accelerators which throw out massive amounts of data.” </li></ul></ul><ul><ul><li>“We need lots of storage.” </li></ul></ul><ul><ul><li>“We need lots of compute.” </li></ul></ul><ul><ul><li>“Managing it is hard.” </li></ul></ul><ul><li>Different science, same problems. </li></ul>   Sequencing machines
  • 4. Managing Growth <ul><li>We have exponential growth in storage and compute. </li></ul><ul><ul><li>Storage doubles every 12 months. </li></ul></ul><ul><ul><ul><li>We will have at least 2PB of disk next year. </li></ul></ul></ul><ul><li>New sequencing technologies are a huge challenge. </li></ul><ul><ul><li>~50x increase in data production in the space of 6 months. </li></ul></ul><ul><li>New sequencing tech is still growing. </li></ul><ul><ul><li>Known unknowns: </li></ul></ul><ul><ul><ul><li>Higher data output from our current machines. </li></ul></ul></ul><ul><ul><ul><li>More machines. </li></ul></ul></ul><ul><ul><li>Unknown unknowns: </li></ul></ul><ul><ul><ul><li>New big science projects are just a “good idea” away... </li></ul></ul></ul>
  • 5. Data centre <ul><li>4x250 M 2 Data centres. </li></ul><ul><ul><li>2-4KW / M 2 cooling. </li></ul></ul><ul><ul><li>3.4MW power draw. </li></ul></ul><ul><li>Overhead aircon, power and networking. </li></ul><ul><ul><li>Allows counter-current cooling. </li></ul></ul><ul><ul><li>More efficient. </li></ul></ul><ul><li>Technology Refresh. </li></ul><ul><ul><li>1 data centre is an empty shell. </li></ul></ul><ul><ul><ul><li>Rotate into the empty room every 4 years. </li></ul></ul></ul><ul><ul><ul><li>Refurb one of the in-use rooms with the current state of the art. </li></ul></ul></ul><ul><ul><li>“Fallow Field” principle. </li></ul></ul>rack rack rack rack
  • 6. Storage <ul><li>SAN Fabric. </li></ul><ul><ul><li>Dual Brocade fabric (27 switches per fabric). </li></ul></ul><ul><ul><li>~1PB in production today. </li></ul></ul><ul><li>HP EVA 5000/8000 arrays. </li></ul><ul><ul><li>Holds the bulk of our data (~1PB) </li></ul></ul><ul><ul><li>Dual controller, Fibre channel disks, ~50TB per array. </li></ul></ul><ul><ul><li>virtual raid5 (effectively Raid 6). </li></ul></ul><ul><ul><ul><li>Don't need to worry about raidset size being nice multiples of physical disk size etc. </li></ul></ul></ul><ul><ul><ul><li>Allows rapid allocation of storage to projects as required. </li></ul></ul></ul><ul><ul><li>Storage is either directly attached, or used with cluster filesytems. </li></ul></ul><ul><li>Bluearc Titan. </li></ul><ul><ul><li>NFS serving for home directories and storage which needs concurrent windows / linux access. </li></ul></ul><ul><ul><li>EVA storage at the back end. </li></ul></ul><ul><li>Backup. </li></ul><ul><ul><li>Veritas netbackup to a Storagetek SL8500 library. </li></ul></ul><ul><ul><li>12 drives (LTO2&3), 1500 slots. </li></ul></ul>
  • 7. Compute <ul><li>3800 cores in >1500 blades and rack mount servers. </li></ul><ul><ul><li>Blades preferred due to ease of management, space and power efficiency. </li></ul></ul><ul><ul><li>Mostly x86_64 servers, some older x86 systems. </li></ul></ul><ul><ul><ul><li>Single, dual and quad core. </li></ul></ul></ul><ul><ul><li>Token ia64 for large memory machines. </li></ul></ul><ul><ul><ul><li>(SGI Altix 350, 16 CPUs, 192GB memory). </li></ul></ul></ul><ul><li>We use Debian Linux as primary OS. </li></ul><ul><ul><li>Badly burned by proprietary OS and file-systems. </li></ul></ul><ul><ul><ul><li>We still have legacy Alpha / Tru64 / AdvFS data and apps which require migration to Linux. </li></ul></ul></ul><ul><ul><li>99% of systems run Debian Sarge / Etch. </li></ul></ul><ul><ul><ul><li>Run 64bit on x86_64 CPUs. </li></ul></ul></ul><ul><ul><ul><li>SLES9 on Oracle server to say inside support matrix. </li></ul></ul></ul><ul><li>Complex User-base. </li></ul><ul><ul><li>~300 users, diverse workload. </li></ul></ul><ul><ul><li>Typically IO bound, integer intensive, single threaded. </li></ul></ul><ul><ul><ul><li>Scales well on clusters (Apart from the IO bit). </li></ul></ul></ul>
  • 8. Infrastructure / Management <ul><li>Deployment. </li></ul><ul><ul><li>Debian FAI automated installer. Integrated with blade management systems for “fire-and-forget” deployment. </li></ul></ul><ul><ul><ul><li>~2 minutes for a complete OS install. </li></ul></ul></ul><ul><li>Updates. </li></ul><ul><ul><li>cfengine, dsh. </li></ul></ul><ul><li>Monitoring. </li></ul><ul><ul><li>ganglia, nagios. </li></ul></ul><ul><li>RequestTracker. </li></ul><ul><ul><li>Used by many software development and science teams within the Institute as well as the System team. </li></ul></ul><ul><ul><li>External engineers and collaborators have access. </li></ul></ul><ul><ul><li>30k tickets per year. </li></ul></ul><ul><li>Heartbeat 2. </li></ul><ul><ul><li>2-8 node cluster for high-availability. </li></ul></ul><ul><ul><li>Mostly mysql + apache using SAN for storage failover. </li></ul></ul>
  • 9. Tera-scale Oracle <ul><li>Sequencing Trace archive. </li></ul><ul><ul><li>Hold results from all DNA sequencing experiments, everywhere. </li></ul></ul><ul><ul><li>Mirrored with NCBI trace archive. </li></ul></ul><ul><ul><li>Currently ~60TB / 8 Billion traces. </li></ul></ul><ul><ul><li>Doubles in size every 12 months. </li></ul></ul><ul><li>Originally data was on file-system, meta-data in oracle. </li></ul><ul><ul><li>Billions of small files (20-80k). </li></ul></ul><ul><ul><ul><li>File-system worst-case. </li></ul></ul></ul><ul><ul><li>Hard to backup, hard to manage space. </li></ul></ul><ul><ul><li>All on Tru64 /advfs. (Dead architecture). </li></ul></ul><ul><li>We decided to move everything into oracle. </li></ul><ul><ul><li>How hard can it be? </li></ul></ul><ul><ul><ul><li>Tera-scale databases are common (according to Oracle). </li></ul></ul></ul>
  • 10. Tera-scale Oracle <ul><li>Primary. </li></ul><ul><ul><li>4 node Oracle 10g RAC cluster (4 core x86_64, 16GB RAM). </li></ul></ul><ul><ul><li>60TB of EVA / fibre-channel storage, Oracle ASM clustered “file-system”. </li></ul></ul><ul><li>Backup Strategy. </li></ul><ul><ul><li>Replicate database to a secondary database with Oracle dataguard. </li></ul></ul><ul><ul><li>2 node RAC cluster with 60TB of MSA1000 storage (cheap-n-cheerful fibre-channel). </li></ul></ul><ul><ul><ul><li>15 minute delay in replication to protect against “finger trouble”. </li></ul></ul></ul><ul><ul><li>Secondary database is the primary backup (disk-to-disk, fast). </li></ul></ul><ul><ul><ul><li>We can run off the secondary if we need to. </li></ul></ul></ul><ul><ul><li>Dumps to tape taken from the secondary. </li></ul></ul><ul><li>Big oracle is hard. </li></ul><ul><ul><li>Oracle is not well tested (especially by Oracle!) on this scale. </li></ul></ul><ul><ul><li>How will we cope with future growth of the database? </li></ul></ul>
  • 11. Compute farm <ul><li>Exclusively Blade. </li></ul><ul><ul><li>588 IBM HS20/LS20 (42 chassis), 128 HP BL460c (8 chassis). </li></ul></ul><ul><ul><li>2224 cores (mix of 32 and 64 bit), 2GB memory /core. </li></ul></ul><ul><ul><li>Raid 1 system disks. </li></ul></ul><ul><ul><li>Debian Sarge + custom kernel. </li></ul></ul><ul><ul><li>LSF used for job scheduling. </li></ul></ul><ul><ul><ul><li>Typically 10k-100k jobs in the system. </li></ul></ul></ul><ul><li>Networking. </li></ul><ul><ul><li>Extreme Networks. </li></ul></ul><ul><ul><li>1-2x GigE edge. </li></ul></ul><ul><ul><li>2-4x GigE trunks. </li></ul></ul><ul><ul><li>2x 10GigE core. </li></ul></ul><ul><ul><li>Systems distributed across data centres. </li></ul></ul>
  • 12. Farm lustre storage <ul><li>HP SFS / Lustre v2.1.1. </li></ul><ul><ul><li>Based on CFS lustre 1.4. </li></ul></ul><ul><ul><li>In house client port to Debian. </li></ul></ul><ul><li>Lustre for work / scratch areas. </li></ul><ul><ul><li>10 OSS / 20 OST. </li></ul></ul><ul><ul><ul><li>20 SFS20 arrays. </li></ul></ul></ul><ul><ul><ul><li>Dual tailed SCSI (highly available). </li></ul></ul></ul><ul><ul><ul><li>12x 250GB SATA disks. </li></ul></ul></ul><ul><ul><ul><li>Raid 6 + 1 hot spare. </li></ul></ul></ul><ul><ul><ul><li>35 TB usable storage. </li></ul></ul></ul><ul><ul><li>Reliability sacrificed for performance. </li></ul></ul><ul><li>We have NFS as well. </li></ul><ul><ul><li>Lustre random access / meta-data performance is rubbish. </li></ul></ul><ul><ul><li>NFS for home directories. </li></ul></ul>
  • 13. Lustre performance <ul><li>Sustained 11-12 Gbit/s peak. </li></ul><ul><ul><li>This is real work, not a benchmark. </li></ul></ul>
  • 14. Supporting New Sequencing <ul><li>We have 20 Illumina (n é e Solexa) sequencing machines. </li></ul><ul><ul><li>This will produce 20-30TB per day. </li></ul></ul><ul><ul><li>Machines will run 24x7. </li></ul></ul><ul><ul><li>We need to keep raw data for ~2 weeks for analysis and QC. </li></ul></ul><ul><li>320 TB Lustre staging area. </li></ul><ul><ul><li>8 EVA8000 arrays, 28 OSSs, 160 OSTs. </li></ul></ul><ul><ul><ul><li>(8 luns per OSS limit required more OSS than planned). </li></ul></ul></ul><ul><ul><li>3 x 100TB file-systems for production + 50TB file-system for development. </li></ul></ul><ul><ul><ul><li>Smaller file-systems hedge against EVA failure. </li></ul></ul></ul><ul><li>Compute. </li></ul><ul><ul><li>256 HP BL460c blades. 600 cores, mixture of dual / quad core. </li></ul></ul><ul><ul><li>Extreme networks Black Diamond 8810 switch (360 non-blocking GigE ports). </li></ul></ul><ul><li>Scratch storage. </li></ul><ul><ul><li>25TB SFS20 lustre scratch area for ad-hoc analysis. </li></ul></ul>
  • 15. Data pull ... LSF reconfig allows processing and alignment capacity to be interchanged. Lustre Clients have 2xGigE, 4GigE trunks from chassis to core switch. Datastore 320TB EVA (Lustre) sequencer1 sequencer 20 blade chassis (alignment) blade chassis (processing) blade chassis (processing) blade chassis (alignment) blade chassis (alignment) blade chassis (suckers) scratch area 25TB SFS20 (Lustre) Final Repository (NFS)
  • 16. Acknowledgements <ul><ul><li>Tim Cutts </li></ul></ul><ul><ul><li>Simon Kelley </li></ul></ul><ul><ul><li>Pete Clapham </li></ul></ul><ul><ul><li>Mark Flint </li></ul></ul><ul><ul><li>James Beal </li></ul></ul>HP Life sciences / SFS <ul><ul><li>Jon Nicholson </li></ul></ul><ul><ul><li>Russell Vincent </li></ul></ul><ul><ul><li>Dave Holland </li></ul></ul><ul><ul><li>Martin Burton </li></ul></ul>Sanger Institute <ul><ul><li>Eamonn O'Toole </li></ul></ul><ul><ul><li>Gavin Brebner </li></ul></ul><ul><ul><li>Phil Butcher </li></ul></ul>

×