Download to read offline
Uploaded byPaolo Missier
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
The document describes a scalable WES/WGS processing pipeline and variant interpretation tool. The pipeline uses a workflow programming model to process data in parallel across cloud resources for improved scalability and cost efficiency. The variant interpretation tool provides a simple interface for clinicians while maintaining a history of investigations and provenance for accountability and analytics across patient cases.
More Related Content
Similar to Invited cloud-e-Genome project talk at 2015 NGS Data Congress
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
- 1. NGSDataCongress London,June2015 P.Misiser Scalable WES ProcessingAnd Variant Interpretation With Provenance Recording Using Workflow On The Cloud Paolo Missier, Jacek Cała, Yaobo Xu, Eldarina Wijaya, Ryan Kirby School of Computing Science and Institute of Genetic Medicine Newcastle University, Newcastle upon Tyne, UK NGS Data Congress London, June 15th, 2015
- 2. NGSDataCongress London,June2015 P.Misiser The Cloud-e-Genome projectat Newcastle 1. NGS data processing: • Implement a flexible WES/WGS pipeline • Scalable deployment over a public cloud • Cost control • Scalability • Flexibility • Of design • Of maintenance • Ensure accountability through traceability • Enable analytics over past patient cases 2. Traceable variant interpretation: • Design a simple-to-use tool to facilitate clinical diagnosis by clinicians • Maintain history of past investigations for analytical purposes Objectives: With an aim to: • 2 year pilot project: 2013-2015 • Funded by UK’s National Institute for Health Research (NIHR) • Cloud resources from Azure for Research Award
- 3. NGSDataCongress London,June2015 P.Misiser Part I: dataprocessing Objectives: • Design and Implement a flexible WES/WGS pipeline • Using workflow technology high level programming • Providing scalable deployment over a public cloud
- 4. NGSDataCongress London,June2015 P.Misiser Scripted NGS dataprocessing pipeline Recalibration Corrects for system bias on quality scores assigned by sequencer GATK Computes coverage of each read. VCF Subsetting by filtering, eg non-exomic variants Annovar functional annotations (eg MAF, synonimity, SNPs…) followed by in house annotations Aligns sample sequence to HG19 reference genome using BWA aligner Cleaning, duplicate elimination Picard tools Variant calling operates on multiple samples simultaneously Splits samples into chunks. Haplotype caller detects both SNV as well as longer indels Variant recalibration attempts to reduce false positive rate from caller
- 5. NGSDataCongress London,June2015 P.Misiser Scripts to workflow- Design Design Cloud Deployment Execution Analysis • Better abstraction • Easier to understand, share, maintain • Better exploit data parallelism • Extensible by wrapping new tools Theoretical advantages of using a workflow programming model
- 6. NGSDataCongress London,June2015 P.Misiser Workflow Design echo Preparingdirectories $PICARD_OUTDIR and $PICARD_TEMP mkdir -p $PICARD_OUTDIR mkdir -p $PICARD_TEMP echo Starting PICARD to clean BAM files... $Picard_CleanSam INPUT=$SORTED_BAM_FILE OUTPUT=$SORTED_BAM_FILE_CLEANED echo Starting PICARD to remove duplicates... $Picard_NoDups INPUT=$SORTED_BAM_FILE_CLEANED OUTPUT = $SORTED_BAM_FILE_NODUPS_NO_RG METRICS_FILE=$PICARD_LOG REMOVE_DUPLICATES=true ASSUME_SORTED=true echo Adding read group information to bam file... $Picard_AddRG INPUT=$SORTED_BAM_FILE_NODUPS_NO_RG OUTPUT=$SORTED_BAM_FILE_NODUPS RGID=$READ_GROUP_ID RGPL=illumina RGSM=$SAMPLE_ID RGLB="${SAMPLE_ID}_${READ_GROUP_ID}” RGPU="platform_Unit_${SAMPLE_ID}_${READ_GROUP_ID}” echo Indexing bam files... samtools index $SORTED_BAM_FILE_NODUPS “Wrapper” blocksUtility blocks
- 7. NGSDataCongress London,June2015 P.Misiser Workflow design raw sequences alignclean recalibrate alignments calculate coverage call variants recalibrate variants filter variants annotate coverage information annotated variants raw sequences align clean recalibrate alignments calculate coverage coverage informationraw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants filter variants Conceptual: Actual:
- 8. NGSDataCongress London,June2015 P.Misiser Anatomy of acomplex parallel dataflow eScience Central: simple dataflow model… raw sequences align clean recalibrate alignments calculate coverage call variants recalibrate variants filter variants annotate coverage information annotated variants raw sequences align clean recalibrate alignments calculate coverage coverage informationraw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants filter variants Sample-split: Parallel processing of samples in a batch
- 9. NGSDataCongress London,June2015 P.Misiser Anatomy of acomplex parallel dataflow … with hierarchical structure
- 10. NGSDataCongress London,June2015 P.Misiser Phase II, toplevel raw sequences align clean recalibrate alignments calculate coverage call variants recalibrate variants filter variants annotate coverage information annotated variants raw sequences align clean recalibrate alignments calculate coverage coverage informationraw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants filter variants Chromosome-split: Parallel processing of each chromosome across all samples
- 11. NGSDataCongress London,June2015 P.Misiser Phase III raw sequences alignclean recalibrate alignments calculate coverage call variants recalibrate variants filter variants annotate coverage information annotated variants raw sequences align clean recalibrate alignments calculate coverage coverage informationraw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants filter variants Sample-split: Parallel processing of samples
- 12. NGSDataCongress London,June2015 P.Misiser Implicit parallelism inthe pipeline align-clean- recalibrate-coverage … align-clean- recalibrate-coverage Sample 1 Sample n Variant calling recalibration Variant calling recalibration Variant filtering annotation Variant filtering annotation …… Chromosome split Per-sample Parallel processing Per-chromosome Parallel processing Stage I Stage II Stage III How does the workflow design exploit this parallelism? raw sequences align clean recalibrate alignments calculate coverage call variants recalibrate variants filter variants annotate coverage information annotated variants raw sequences align clean recalibrate alignments calculate coverage coverage informationraw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants filter variants
- 13. NGSDataCongress London,June2015 P.Misiser Parallel processing overa batch of exomes align lane recalibrate sample call variants recalibrate variants align sample haplotype caller recalibrate sample raw sequences align clean calculate coverage coverage information recalibrate alignments annotate annotated variants Stage 1 Stage 2 Stage 3 filter variants align lanealign lane align sample align lane clean sampleclean sampleclean sample align lane align sample align lane recalibrate sample VC with chr-split haplotype callerhaplotype callerhaplotype caller annotates ampleannotates ampleannotates ample filter samplefilter samplefilter sample annotated variantsannotated variants raw sequencesraw sequences coverage informationcoverage information coverage per samplecoverage per samplecoverage per sample recalibrate
- 14. NGSDataCongress London,June2015 P.Misiser Cloud Deployment Design Cloud Deployment Execution Analysis •Scalability • Fewer installation/deployment requirements, staff hours required • Automated dependency management, packaging • Configurable to make most efficient use of a cluster
- 15. NGSDataCongress London,June2015 P.Misiser Workflow on AzureCloud – modular configuration <<Azure VM>> Azure Blob store e-SC db backend <<Azure VM>> e-Science Central main server JMS queue REST APIWeb UI web browser rich client app workflow invocations e-SC control data workflow data <<worker role>> Workflow engine <<worker role>> Workflow engine e-SC blob store <<worker role>> Workflow engine Workflow engines Module configuration: 3 nodes, 24 cores Modular architecture indefinitely scalable!
- 16. NGSDataCongress London,June2015 P.Misiser Scripts to workflow Design Cloud Deployment ExecutionAnalysis 3. Execution • Runtime monitoring • provenance collection
- 17. NGSDataCongress London,June2015 P.Misiser Performance 3 workflow enginesperform better than our HPC benchmark on larger sample sizes Technical configurations for 3VMs experiments: HPC cluster (dedicated nodes): used 3x8-core compute nodes Intel Xeon E5640, 2.67GHz CPU, 48 GiB RAM, 160 GB scratch space Azure workflow engines: D13 VMs with 8-core CPU, 56 GiB of memory and 400 GB SSD, Ubuntu 14.04.
- 18. NGSDataCongress London,June2015 P.Misiser Scalability There is littleincentive to grow the VM pool beyond 6 engines
- 19. NGSDataCongress London,June2015 P.Misiser Cost 0 2 4 6 8 10 12 14 16 18 0 6 1218 24 CostinGBP Number of samples 3 eng (24 cores) 6 eng (48 cores) 12 eng (96 cores) Again, a 6 engine configuration achieves near-optimal cost/sample
- 20. NGSDataCongress London,June2015 P.Misiser Lessons learnt Design Cloud Deployment Execution Analysis Better abstraction • Easier to understand, share, maintain Better exploit data parallelism Extensible by wrapping new tools • Scalability Fewer installation/deployment requirements, staff hours required Automated dependency management, packaging Configurable to make most efficient use of a cluster Runtime monitoring Provenance collection Reproducibility Accountability
- 21. NGSDataCongress London,June2015 P.Misiser Part II: SVI-Simple, traceable variant interpretation Objectives: • Design a simple-to-use tool to facilitate clinical diagnosis by clinicians • Maintain history of past investigations for analytical purposes • Ensure accountability through traceability • Enable analytics over past patient cases MAF threshold - Non-synonymous - stop/gain - frameshift known polymorphisms Homo / Heterozygous Pathogenicity predictors Variant filtering HPO match HPO to OMIM OMIM match OMIM to Gene Gene Union Gene Intersect Genes in scope User-supplied genes list User-supplied disease keywords User-defined preferred genes Variant Scoping Candidate variants Select variants in scope variants in scope ClinVar lookupClinVar Annotated patient variants Variant Classification RED: found, pathogenic AMBER: not found GREEN: found, benign OMIM AMBER/ not found AMBER/ uncertain NGS pipeline
- 22. NGSDataCongress London,June2015 P.Misiser A database ofpatient cases and investigations Cases:
- 23.
- 24. NGSDataCongress London,June2015 P.Misiser Provenance of variantidentification • A provenance graph is generated for each investigation It accounts for the filtering process for each variant listed in the result Enables analytics over provenance graphs across many investigations - “which variants where identified independently on different cases, and how do they correlate with phenotypes?”
- 25. NGSDataCongress London,June2015 P.Misiser Summary 1. WES/WGS dataprocessing to annotated variants • Scalable, Cloud-based • High level • Low cost / sample 2.Variant interpretation: • Simple • Targeted at clinicians • Built-in accountability of genetic diagnosis • Analytics over a database of past investigations What we are delivering to NIHR:
Editor's Notes
- #3 Objective 1: Implement a cloud-based, secure scalable, computing infrastructure that is capable of translating the potential benefits of high throughput sequencing into actual genetic diagnosis to health care professionals. Obj 2: front end tool to facilitate clinical diagnosis 2 year pilot project Funded by UK’s National Institute for Health Research (NIHR) through the Biomedical Research Council (BRC) Nov. 2013: Cloud resources from Azure for Research Award 1 year’s worth of data/network/computing resources
- #4 Objective 1: Implement a cloud-based, secure scalable, computing infrastructure that is capable of translating the potential benefits of high throughput sequencing into actual genetic diagnosis to health care professionals. Obj 2: front end tool to facilitate clinical diagnosis 2 year pilot project Funded by UK’s National Institute for Health Research (NIHR) through the Biomedical Research Council (BRC) Nov. 2013: Cloud resources from Azure for Research Award 1 year’s worth of data/network/computing resources
- #6 Current local implementation: - Scripted pipeline requires expertise to maintain, evolve Deployed on local department cluster Difficult to scale Cost / patient unknown Unable to take advantage of decreasing cost of commodity cloud resources Coverage information translates into confidence on variant call Recalibration: quality score recalibration -- machine produces colour coding for the 4 aminocids, along with a p-value indicating the highest prob call; these are the Q scores different platforms give differnst system bias on Q scores -- and also depending on the lane. Each lane gives a different systematic bias. The point of recalibration is to correct for this type of bias
- #9 Wrapper blocks, such as Picard-CleanSAM and Picard-MarkDuplicates, communicate via files in the local filesystem of the workflow engine, which is explicitly de- noted as a connection between blocks. The workflow includes also utility blocks to import and export files, i.e. to transfer data from/to the shared data space (in this case, the Azure blob store). These were com- plemented by e-SC shared libraries, which provide better efficiency in running the tools, as they are installed only once and cached by the workflow engine for any future use. Libraries also promote reproducibility because they eliminate dependencies on external data and services. For instance, to access the human reference genome we built and stored in the system a shared library that included the genome data in a specific version and flavour (precisely HG19 from UCSC).
- #11 Loops were used in stage (1) and (3), to iterate over samples that the pipeline was configured to process. control blocks can start a number of sub-workflow invocations, one for each element on their input list. Using these two features, we were able to implement a pattern similar to “map” (in the functional sense), where the initial block generates a list of data samples to process, and then for each element in the list the following block starts a sub-workflow (the loop body);
- #18 Sync design: The subworkflows of each step are executed in parallel but synchronously over a number of samples. It means that the top-level workflow submits N subworkflow invocations for a particular step, wait The primary advantage of the discussed, synchronous de- sign is that the structure of the pipeline is modular and clearly represented by the top-level orchestrating workflow whilst the parallelisation is managed by e-SC automatically. The top-level workflow mainly includes blocks to run subworkflows that are independent parts implementing only the actual work done by a particular step. The control blocks take care of the interaction with the system to submit the subworkflows and also suspend the parent invocation until all of them complete.
- #24 Model currently is sync execution
- #26 Each sample included 2-lane, pair-end raw sequence reads (4 files per sample).The average size of compressed files was nearly 15 GiB per sample; file decompression was included in the pipeline as one of the initial tasks.
- #33 A quick overview of the entered phenotype. Shows how many genes found in OMIM, match with genes found in the patients variants. The graph shows a quick summary of any results produces from ClinVar. The phenotypes section in the bottom right shows results from HPO. In the report sections, on the left hows a collection of all the investigations created for the current case (including the one just created).