Accelerating Analytics for the Future of Genomics

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dr. Swaine Chen Genome Institute of Singapore, National University of Singapore 223537 Accelerating Analytics for the Future of Genomics

2. Accelerating Genomics Research with the Cloud Swaine Chen Genome Institute of Singapore National University of Singapore

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DNA sequencing technology Maxam-Gilbert Chemistry and radiation Solexa (Illumina) Higher density, imaging Oxford Nanopore Electric current detection Capillary seq Miniaturization Parallelization 1970 1980 1990 2000 2010

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DNA sequencing technology Maxam-Gilbert Chemistry and radiation Solexa (Illumina) Higher density, imaging Oxford Nanopore Electric current detection Capillary seq Miniaturization Parallelization 1970 1980 1990 2000 2010 Miniaturization, Parallelization, Digitization

11. GIS, Pre-AWS 128 nodes 40-80 CPUs 128-512 GB RAM Head node On-site data centerOffice area User workstations “SMPs” 96 CPUs 1 TB RAM 1Gbps 40-100 Gbps Archival Storage (10 PB) Office, home Storage 3 PB Compute Storage 4 PB

12. Cluster nodes (~500) 4-8 CPUs 64-128GB RAM Head node On-site data centerOffice area User workstations “SMPs” 128 CPUs 1TB RAM 1Gbps 10-100 Gbps Archival Storage (3 PB) Office, home Storage 1PB Compute Storage 100TBChallenges First-time command line users Heterogeneous compute, storage, network No/low experience • Job management • Optimization • Software config/documentation Spiky workloads Self-inflicted denial of service GIS, Pre-AWS

13. How did we first use AWS? Phase 1 • Reimplement “SMPs” • Users can’t DOS each other • Infinite capacity (and potential for waste) • Full complexity Single instance EBS / compute storage S3 / Object storage Individual user AWSGIS

14. How did we first use AWS? Phase 1 • Reimplement “SMPs” • Users can’t DOS each other • Infinite capacity (and potential for waste) • Full complexity

15. Our current efforts on AWS Phase 2 • Nextflow + AWS Batch • Totally new paradigm, enabled by cloud • AWS for elastic provisioning • Cluster is abstracted away • Leverage this for software S3 / Object storage Individual user AWSGIS AWS Batch

16. Phase 2 • Nextflow + AWS Batch • Totally new paradigm, enabled by cloud • AWS for elastic provisioning • Cluster is abstracted away • Leverage this for software S3 / Object storage Individual user AWSGIS Job repo Jobtasks Docker repo (ECR) AWS Batch

17. Why is this complexity needed? GATK Best Practices – a standard workflow in genomics

18. Capacity + Simplicity on AWS

20. GIS Bacterial Projects: 100× in 4 years 1 10 100 1000 10000 1995 2000 2005 2010 2015 2020 Bacterial genomics: # genomes/paper tracks Moore’s Law Year of publication Numberofgenomes GIS

21. GIS Bacterial Projects: 100× in 4 years 1 10 100 1000 10000 1995 2000 2005 2010 2015 2020 Year of publication Numberofgenomes GIS 2013: 10-100 strains

22. GIS Bacterial Projects: 100× in 4 years 1 10 100 1000 10000 1995 2000 2005 2010 2015 2020 Year of publication Numberofgenomes GIS 2017: 10,000 strains Higher resolution, more perspective

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SINGAPORE’S DENGUE MONITORING Since 2006 All nonresidential buildings checked every 3 months 1 million inspections per year INFRASTRUCTURE

26. Preparing for 1 Million Genomic Devices Phase 3 • Serverless, event-driven model • Massive scale • No user intervention • Fundamentally cloud-driven transformation of our problem solving • Enables continuous monitoring

27. Preparing for 1 Million Genomic Devices Reimplement variant calling 6 hours 15 minutes Auto scatter-gather, high parallelism 1,000 genomes, 25 million GB-s, no intervention 12 genomes on Lambda free tier! 1 10 100 1000 10000 100000 Run own servers GIS + Lambda Genomes per unit cost 20×

28. MANY SMART IDEAS ONE SMART NATION ENABLED BY GENOMICS

29. Maggie Leong Vincent Quah Adrian White Julian Lau Liew Jun Xian Andreas Wilm Shih Chih Chuan Ng Huck Hui Pauline Ng Anders Skanderup National Precision Medicine Program

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The players in the trenches • Medical Population Genetics Platform • Tasked with developing tools/ BP pipelines • Scope creep: run workflow for researchers • Workflowing solution: GATK-Queue (scala) GATK dev team Picard / Ops team • Genomics Platform • Initial data processing -> Picard toolkit • Took over workflows in production • Workflowing solution: Zamboni (scala) Cancer Genome Analysis team • Cancer Program • Tasked with developing tools/ BP workflows for somatic analysis • Workflowing solution: Firehose self-service (python?) The drama: low portability, silos, duplication of effort, looming bottlenecks

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sharing (securely) is caring Traditional Way: Bring data to the researchers Problems Data sharing = data copying Requires big infrastructure at each site Largely fixed compute Individual security implementations Cloud Way: Bring researchers to the data Solutions True data sharing Cloud provides the infrastructure Elastic compute and storage Centralized security implementation

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Genome analysis pipeline throughput is “spiky” • Solution: move to Cloud! Advantages over on-premises computing: – No need to pay for compute power when we aren’t using it – Can tolerate spikes without being forced to maintain a backlog of “things to process once everything calms down” Genome processing requests per day over a several month period

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use containers for portability & reproducibility A container encapsulates all the software dependencies associated with running a program Takes the guesswork out of running workflows on different platforms! GATK 2.8 Java 7 R 2.5.0 GATK 3.8 Java 8 R 3.0.1 BWA Picard Modified from https://www.docker.com/what-container

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Meet Cromwell & WDL Execution engine that can • Run on any platform (on-prem and on Cloud) • Seamlessly scale based on workflow needs • Provide maximal flexibility for all use cases • https://github.com/broadinstitute/cromwell Workflow language that humans can read/write • Methods developers and biomedical scientists at large • https://github.com/openwdl/wdl/

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Two main ways to run Cromwell • API endpoints • More scalable • Some devops needs • Appropriate for production environments • Call caching • Simple self-contained command • Appropriate for independent analysts One-off Server mode java -jar cromwell.jar run hello.wdl hello_inputs.json

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use a workflow execution engine that runs anywhere* Cromwell … HPC TESLocal Google Funnel https://github.com/broadinstitute/cromwell AWS* Alicloud *in development

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Enable local development of workflows, run on the cloud S3 data buckets Managed compute environment AWS Persistent Cromwell server REST API Direct CLI

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cromwell will submit jobs to AWS Batch Job Queues Cromwell inputs inputs outputs GATK = gatk.jar RefFasta = hg38.fasta RefIndex = hg38.fai RefDict = hg38.dict sampleName = sample.name inputBAM = sample.bam bamIndex = sample.bai AWS Batch Workflow Cromwell stages the inputs/outputs for your jobs

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Basic WDL plumbing options call stepA call stepB { input: in=stepA.out } call stepC { input: in=stepB.out } LINEAR CHAINING MULTI-IN/OUT call stepC { input : in1=stepB.out1, in2=stepB.out2 } Array[File] inputFiles scatter(oneFile in inputFiles) { call stepA { input: in=oneFile } } call stepB { input: files=stepA.out } SCATTER-GATHER

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. But what about CWL? Randall Munroe, XKCD https://www.xkcd.com/1739/ Thanks to our Workflow Object Model (WOM), Cromwell now supports multiple versions of WDL as well as CWL 1.0!

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cromwell has been busy Cromwell in production at Broad: Processed 47.5 million jobs over the last two years And this is just the tip of the iceberg!

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Want to discuss further? My Email: rmunshi@broadinstitute.org More Information: Docs: http://cromwell.readthedocs.io/en/develop/ Github: https://www.github.com/broadinstitute/cromwell WDL: http://www.openwdl.org

Accelerating Analytics for the Future of Genomics

You are reading a preview.

Check these out next

Accelerating Analytics for the Future of Genomics

More Related Content

Slideshows for you (20)

Similar to Accelerating Analytics for the Future of Genomics (20)

More from Amazon Web Services (20)