Sharing large amounts of data is easier said than done. This talk gives an overview of our experiences doing big-data science over wide-area networks.

1. Data Sharing: Sanger Experiences Guy Coates Wellcome Trust Sanger Institute [email_address]

6. First attempt: 14 Mbits/s out of 2Gbit/s link.

14. Constant monitoring / bandwidth tests to ensure it stays working.

15. See HEPIX talks for gory details.

17. Is it the only thing we need to do?

19. Sequencing data flow Sanger Sequencer Analysis/ alignment Internal repository EBI EGA / SRA (EBI)

21. 10Gbit/s (1.2 Gbyte/s) link between EBI and Sanger.

22. We share a data-centre.

23. Physically near, so we do not need to worry about WAN issues.

29. Do we need to move the data?

30. Centralised data Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre

33. 3 month lead time.

34. ~$1.5M capex.

35. Federation: A Better way: Collaborations are short term: 18 months-3 years. Sequencing centre Sequencing centre Sequencing centre Sequencing centre Federated access

36. Federation software: Unstructured data (flat files) Data size per Genome Structured data (databases) BioMart IRODS (data grid software) Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)

38. Put VMs on compute that is “attached” to the data. Data CPU CPU CPU CPU Data CPU CPU CPU CPU VM

40. We need cross site teams who can troubleshoot all potential trouble spots.

41. Teams need application & systems people.

43. Gen-Tao Chiang

44. Pete Clapham

45. Simon Kelley