Hammer Lab has built several tools for analyzing genomic data on Spark, as well as libraries for more general computations using RDDs; I’ll discuss some of the most interesting applications and algorithms therein: Guacamole (https://github.com/hammerlab/guacamole) is a somatic variant caller built on Spark; it identifies mutations in cancer genomes in a fraction of the time that comparable tools take. Pageant (https://github.com/hammerlab/pageant) contains miscellaneous other genomic analyses and a few interesting and novel algorithms for massively-parallel Burrows-Wheeler-Transform and FM-Index construction. Magic RDDs (https://github.com/hammerlab/magic-rdds) contains some yet more interesting general-purpose algorithms implemented on RDDs.

1. Algorithms & Tools for
Genomic Analysis on
Spark
Ryan Williams
Hammer Lab @ Mt. Sinai School of Medicine
http://bit.ly/sse2017

2. Slides:
http://bit.ly/sse2017

3. Agenda
• Intro
• Genomics crash course
• Genomics applications
• Fun with magic-rdds
• Scala/Spark project mgmt notes
• Questions

4. Intro

5. Hammer Lab
• Est. 2013 @ Mt. Sinai School of Medicine
• Initial focus: high-quality bioinformatics/genomics software
– distributed systems
– OSS
– static-typing + functional idioms
• Present-day: cancer immunotherapy research
– personal-cancer-vaccine clinicaltrials
– post-hoc clinical-data analysis
• This talk: ≈3yrs of genomic-analysis tool-building w/ Spark

6. Spark/Scala
Tooling Overview

7. Genomics
Crash Course

8. blogs.plos.org/dnascience/…

9. http://www.wikiwand.com/en/Shotgun_sequencing

10. www.thunderbolts.info
• Excessive repetitiveness
– 20% retrotransposons
– L1: 7000bp, 100k copies
– Pseudogenes
• Impossible to resolve with “short reads”
Genome structure makes things
difficult

11. www.pnas.org

12. contig.wordpress.com

13. github.com/rrwick/Bandage

14. “Reference Genome”
Basically
everything
matches
“Coverage Depth”

15. Digitizing Human Genomes
• 1 genome ≈ 3B base-pairs
• Theory:
– “2 bits per base-pair” (A, C, G, T)
– ⇒ 1 genome ≈ 750MB
– <1% unique, person to person
– 7BN genomes ≈ 50PB?
• Reality:
– 1BN 100bp “reads”
– ⇒ 100BN sequenced bases
– Cover the genome at average depth 30 (“30x coverage”)
– 2-bit base, 1-byte quality score ⇒ 100GB / genome
– 100-100k genomes ⇒ 10TB-10PB

16. Genomic
Applications on
Spark

17. Spark/Scala
Tooling Overview

18. www.idtdna.com

19. Joint histogram of coverage-depth distribution of two samples

20. Joint histogram of coverage-depth distribution of two samples

22. Fun with
magic-rdds

23. Spark/Scala
Tooling Overview

24. CappedGroupByKeyRDD

25. SlidingRDD

26. ReverseRDD

27. ScanLeftRDD

28. RunLengthRDD

29. Scala/Spark
Project Management