Hadoop for Bioinformatics

Hadoop for Bioinformatics Deepak Singh Amazon Web Services Hadoop World, NYC

Via Reavel under a CC-BY-NC-ND license

By ~Prescott under a CC-BY-NC license

data sets

many data sets

PFAM PDB GENBANK ENSEMBL Many Others

manageable

Image: Matt Wood

Human genom e Image: Matt Wood

Image: Matt Wood

~100 TB/Week Image: Matt Wood

~100 TB/Week >2 PB/Year Image: Matt Wood

gigabytes

terabytes

petabytes

really fast

typical informatics workﬂow

Via Christolakis under a CC-BY-NC-ND license

Via Argonne National Labs under a CC-BY-SA license

killer app Via Argonne National Labs under a CC-BY-SA license

Via asklar under a CC-BY license

Image: Chris Dagdigian

rethink algorithms

rethink computing

rethink data management

rethink data sharing

operational mindset

scalability

we are data geeks not data center geeks

two key trends

develop applications

distribute applications

use applications

some work

ﬁlters some work ^

High Throughput Sequence Analysis Mike Schatz, University of Maryland

• Read Mapping • Mapping & SNP Discovery • De novo Genome Assembly

Short Read Mapping

Asian Individual Genome: 3.3 Billion 35bp, 104 GB (Wang et al., 2008) African Individual Genome: 4.0 Billion 35bp, 144 GB (Bentley et al., 2008)

Alignment > 10000 CPU hrs

Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1)

Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale

Seed & Extend Good alignments must have signiﬁcant exact alignment Minimal exact alignment length = l/(k+1) Expensive to scale Need parallelization framework

CloudBurst Catalog k-mers Collect seeds End-to-end alignment

http://cloudburst-bio.sourceforge.net; Bioinformatics 2009 25: 1363-1369

CloudBurst efﬁciently reports every k-difference alignment of every read

many applications only need the best alignment

Bowtie: Ultrafast short read aligner Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efﬁcient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.

SOAPSnp: Consensus alignment and SNP calling Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efﬁcient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.

Crossbow: Rapid whole genome SNP analysis Ben Langmead Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efﬁcient alignment of short DNA sequences to the human genome. Genome Biol 10 (3): R25.

Preprocessed reads

Preprocessed reads Map: Bowtie

Preprocessed reads Map: Bowtie Sort: Bin and partition

Preprocessed reads Map: Bowtie Sort: Bin and partition Reduce: SoapSNP

Crossbow condenses over 1,000 hours of resequencing computation into a few hours without requiring the user to own or operate a computer cluster

Comparing Genomes

Estimating relative evolutionary rates from sequence comparisons: Identiﬁcation of probable orthologs Admissible comparisons: A or B vs. D C vs. E Inadmissible comparisons: A or B vs. E C vs. D A B C D E species tree gene tree S. cerevisiae C. elegans

Estimating relative evolutionary rates from sequence comparisons: 1. Orthologs found using the Reciprocal smallest distance algorithm 2. Build alignment between two orthologs >Sequence C MSGRTILASTIAKPFQEEVTKAVKQLNFT-----PKLVGLLSNEDPAAKMYANWTGKTCESLGFKYEL-… >Sequence E MSGRTILASKVAETFNTEIINNVEEYKKTHNGQGPLLVGFLANNDPAAKMYATWTQKTSESMGFRYDL… 3. Estimate distance given a substitution matrix Phe Ala Pro Leu Thr Phe Ala µπ Pro µπ µπ µπ Leu µπ µπ µπ µπ A B C D E species tree gene tree S. cerevisiae C. elegans

RSD algorithm summary Genome I Genome J Ib Jc Align sequences & Calculate distances L Orthologs: Align sequences & Calculate distances H ib - jc D = 0.1 c vs. D=1.2 vs. D=0.2 a b a vs. D=0.1 vs. D=0.3 c b b b vs. D=0.9 vs. D=0.1 c c b c

Prof. Dennis Wall Harvard Medical School

Roundup is a database of orthologs and their evolutionary distances. To get started, click browse. Alternatively, you can read our documentation here. Good luck, researchers!

massive computational demand

1000 genomes = 5,994,000 processes = 23,976,000 hours

2737 years

periodic task

must scale up

not scalability gurus

hadoop streaming

compared 50+ genomes

what’s next?

de novo assembly

machine learning and statistics

protein structure prediction

docking

trajectory analysis

key driving factors?

the ecosystem

Cascading

domain speciﬁc libraries and tools

http://aws.amazon.com/publicdatasets/

http://aws.amazon.com/education/

Thank you! deesingh@amazon.com; Twitter:@mndoci Presentation ideas from @mza, @simon and @lessig

http://hongiiv.tistory.com	856
http://www.genomeweb.com	304
http://mndoci.com	141
http://blogs.scientifik.info	115
http://socmaster.homelinux.org	76
http://www.slideshare.net	52
http://paper.li	23
http://www.linkedin.com	12
http://deepaksingh.net	12
http://www.hanrss.com	6
http://www.mndoci.com	5
http://feeds2.feedburner.com	3
http://www.iweb34.com	3
http://2429-genomeweb.voxcdn.com	2
http://cdnwww.genomeweb.com	2
http://translate.googleusercontent.com	1
http://webcache.googleusercontent.com	1
http://www.jokimproduction.com	1
http://theoldreader.com	1

Hadoop for Bioinformatics

by Deepak Singh

Accessibility

Categories

Upload Details

Usage Rights

19 Embeds 1,616

Statistics

Hadoop for Bioinformatics Presentation Transcript