Genome simulation and applications
•Download as PPTX, PDF•
3 likes•169 views
A Topic i had presented at the Cloud and DevOps stream at TechXLR8 London 2017. Topic covers using Google cloud , Kubernetes , docker and cloud functions to create a managed distributed compute infrastructure to generate synthetic genomic data for simulation and infrastructure testing needs.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Genome simulation and applications
Similar to Genome simulation and applications (20)
Recently uploaded
Recently uploaded (20)
Genome simulation and applications
- 1. Genome (FASTQ and VCF) Simulation & Applications Hariprasad Radhakrishnan AstraZeneca, Technology Labs, UK
- 2. Genome Simulation AstraZeneca We are a global, science-led biopharmaceutical business pushing the boundaries of science to deliver life-changing medicines. 61,500 employees worldwide $23bn 2016 Revenue* 100+ Countries
- 3. Our Experiment – • Quick introduction to DNA - DNA Sequencing • synthetically generated Genome data (FASTQ & VCF)! • How to Scale/ run Distributed Compute using Kubernetes and Docker Hari works as an Associate Architect - Data & Analytics in the UK Tech Incubation Lab of AstraZeneca. Genome Simulation Introduction
- 4. DNA & How it is Sequenced Genome Simulation
- 5. AGCCCCTCAGGAGTCCGGCCACATGGAAACTCCTCATTCCGGAGGTCAGTCAGATTTACCCTGGCTCACCTTGGCGTCGCG TCCGGCGGCAAACTAAGAACACGTCGTCTAAATGACTTCTTAAAGTAGAATAGCGTGTTCTCTCCTTCCAGCCTCCGAAAA ACTCGGACCAAAGATCAGGCTTGTCCGTTCTTCGCTAGTGATGAGACTGCGCCTCTGTTCGTACAACCAATTTAGGTGAGT TCAAACTTCAGGGTCCAGAGGCTGATAATCTACTTACCCAAACATAG Deoxyribonucleic acid (DNA) is the chemical inside the nucleus of all cells that carries the genetic instructions for making living organisms. A DNA molecule consists of two strands that wrap around each other to resemble a twisted ladder. Genome Simulation Human DNA
- 7. In a perfect world (just your 3 billion letters): ~700 megabytes In the real world, right off the genome sequencer: 100~200 gigabytes Genome Simulation Human Genome Sequencing For maybe 100,000 + samples 10~20 petabytes
- 8. Simulation and the need for it Genome Simulation
- 9. • Be able to Simulate high-throughput Genome sequencing. • The Genome generated can be used to test existing pipelines and infrastructure. • If run in a distributed mode can generate sufficient data to create near production line scenarios, the data could be used to test the ingestion, processing through the pipeline and subsequent analytic tools. • Synthetic so no issues with privacy, patient de – identification, transfer across regions. Genome Simulation Why Simulate Genomic Data
- 10. Genome Simulation Human Genome – Alignment – Variant Calling
- 11. Our Little Experiment Genome Simulation
- 12. • VarSim was picked as the tool for genome simulation, it provided ways were variations could be introduced in a random fashion into Genome Simulation and the output FASTQ and VCF files would be unique from each other • Other tools were considered, but were not maintained or did not provide sufficient flexibility. Genome Simulation Tool Selection
- 13. References John C. Mu, Marghoob Mohiyuddin, Jian Li, Narges Bani Asadi, Mark B. Gerstein, Alexej Abyzov, Wing H. Wong, and Hugo Y.K. Lam VarSim: A high-fidelity simulation and validation framework for high-throughput genome sequencing with cancer applications Bioinformatics first published online December 17, 2014doi:10.1093/bioinformatics/btu828 Summary: VarSim is a framework for assessing alignment and variant calling accuracy in high- throughput genome sequencing through simulation or real data. In contrast to simulating a raNdom mutation spectrum, it synthesizes diploid genomes with germline and somatic mutations based on a realistic model. This model leverages information such as previously reported mutations to make the synthetic genomes biologically relevant. Genome Simulation VarSim
- 14. • VarSim is a Python/Java based tool that would simulate one genome per run. • We looked into ways were we could parallelize the to generate more genomes. • Build Docker container/Image for the tool. • Experiment run on Google Cloud – Container engine. • Parameters like Coverage, Unique ID & Seed value were externalized in a Lambda function that the Docker images could talk to and receive arguments before execution. • Output FASTQ files and VCF’s would then be stored in Cloud storage. • Ability to choose to generate FASTQ & VCF or just VCF. Genome Simulation Technicalities
- 15. Start Script Google Cloud Libraries Java Libraries SAM Tools VarSim Python Libraries Ref Genome / Insert Sequences / Annotations ART Simulator 4.8 GB CLOUD FUNCTIONS BETA HTTP DOCKER IMAGE OUTPUT FILES FASTQ VCF Genome Simulation Docker Image
- 16. Using Docker Registry on Google Cloud. Given the size of the Docker Image it made sense to take advantage of the high Network speeds between servers on the cloud for quick deployment to the Container Engine. Genome Simulation Container Registry DOCKER IMAGE4.8 GB
- 17. Using Container Registry on Google Cloud. Genome Simulation Container Clusters Can Reach a MAX cluster size of 1000
- 18. Configure the version of Docker image to be deployed to the cluster. Kubernetes takes care of distributing the Docker image to all the instances in the cluster. Genome Simulation Kubernetes - Container Clusters
- 20. • We have around 1000 unique VCF files generated so far. • We have around 10 Genome FASTQ and VCF’s. • More FASTQ and VCF if we can fund it. Cost • Cost $1000 to generate 1000 unique VCF files. 1$ per VCF. • Cost’s to generate FASTQ files vary based on the coverage required. For a 50X coverage the costs work out around 5$ for the FASTQ and VCF. The costs can be brought down by generating FATSQ files in multiple lanes. Genome Simulation Outcome
- 21. John C. Mu, Marghoob Mohiyuddin, Jian Li, Narges Bani Asadi, Mark B. Gerstein, Alexej Abyzov, Wing H. Wong, and Hugo Y.K. Lam VarSim: A high-fidelity simulation and validation framework for high-throughput genome sequencing with cancer applications Bioinformatics first published online December 17, 2014doi:10.1093/bioinformatics/btu828 To folks from Google - Daniel Bergqvist, Nico Gaviola & Craig Box. Mathew Woodwark, Nick Brown, Rob Hernandez, Sandra Giuliani, Frank Lombardi from AstraZeneca for supporting this work. Genome Simulation References Thanks
Editor's Notes
- The human body has about 100 trillion cells with more than 200 different cell types. Each cell harbors the same genetic information in its nucleus in form of DNA containing chromosomes DNA Deoxyribonucleic acid (DNA) is the chemical inside the nucleus of all cells that carries the genetic instructions for making living organisms. A DNA molecule consists of two strands that wrap around each other to resemble a twisted ladder. The sides are made of sugar and phosphate molecules. The �rungs� are made of nitrogen-containing chemicals called bases. Each strand is composed of one sugar molecule, one phosphate molecule, and a base. Four different bases are present in DNA - adenine (A), thymine (T), cytosine (C), and guanine (G). The particular order of the bases arranged along the sugar - phosphate backbone is called the DNA sequence; the sequence specifies the exact genetic instructions required to create a particular organism with its own unique traits. Each strand of the DNA molecule is held together at its base by a weak bond. The four bases pair in a set manner: Adenine (A) pairs with thymine (T), while cytosine (C) pairs with guanine (G). These pairs of bases are known as Base Pairs (bp). These Base Pairs (bp) are the basis of Y-chromosome testing.
- The human body has about 100 trillion cells with more than 200 different cell types. Each cell harbors the same genetic information in its nucleus in form of DNA containing chromosomes DNA Deoxyribonucleic acid (DNA) is the chemical inside the nucleus of all cells that carries the genetic instructions for making living organisms. A DNA molecule consists of two strands that wrap around each other to resemble a twisted ladder. The sides are made of sugar and phosphate molecules. The �rungs� are made of nitrogen-containing chemicals called bases. Each strand is composed of one sugar molecule, one phosphate molecule, and a base. Four different bases are present in DNA - adenine (A), thymine (T), cytosine (C), and guanine (G). The particular order of the bases arranged along the sugar - phosphate backbone is called the DNA sequence; the sequence specifies the exact genetic instructions required to create a particular organism with its own unique traits. Each strand of the DNA molecule is held together at its base by a weak bond. The four bases pair in a set manner: Adenine (A) pairs with thymine (T), while cytosine (C) pairs with guanine (G). These pairs of bases are known as Base Pairs (bp). These Base Pairs (bp) are the basis of Y-chromosome testing.
- @SEQ_ID GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT + !’’*((((***+))%%%++)(%%%%).1***-+*’’))**55CCF>>>>>>CCCCCCC65
- VarSim is a Python/Java based tool that would simulate one genome per run. We looked into ways were we could parallelize the process and run simultaneously on multiple machines to generate more genomes. We packaged the tool into a Docker container/Image so it can be easily shipped and deployed in a cluster. As we had some experience in using Google Cloud, we decided to execute it in a cluster using a managed service called Container Engine (running Kubernetes). This allows us to spin multiple machines (upto 500) and deploy our Docker Image(Varsim) and execute. Parameters like Coverage, Unique ID & Seed value was externalized in a Lambda function that the Docker images could talk to and receive arguments before execution. Output FASTQ files and VCF’s would then be stored in Cloud storage. Ability to choose to generate FASTQ & VCF or just VCF.