2. What Ancestry uses from the Hadoop ecosystem
● Hadoop, HDFS, and MapReduce
● HBase
● Columnar, NoSQL data store, unlimited rows and columns
● Azkaban
● Workflow
2
3. What will this presentation cover?
● Describe the problem
● Discoveries with DNA
● Three key steps in the pipeline process
● Measure everything principle
● Three steps with Hadoop
● Hadoop as a job scheduler for the ethnicity step
● Scaling matching step
● MapReduce implementation of phasing
● Performance
● What comes next?
3
4. Discoveries with DNA
● Autosomal DNA test that analyzes 700,000 SNPs
● Over 400,000 DNA samples in our database
● Identified 30 million relationships that connect the genotyped members
through shared ancestors
4
-
5,000,000
10,000,000
15,000,000
20,000,000
25,000,000
30,000,000
-
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
Jan-12 Apr-12 Jul-12 Oct-12 Jan-13 Apr-13 Jul-13 Oct-13 Jan-14 Apr-14
Cousinmatches(4thorcloser)
DatabaseSize(#ofsamples)
DNA DB Size Cousin Matches
5. Three key steps in the pipeline
● What is a pipeline?
1. Ethnicity (AdMixture)
2. Matching (GERMLINE and Jermline)
3. Phasing (Beagle and Underdog)
● First pipeline executed on a single, beefy box.
● Only option is to scale vertically
5
Init
Results
Processing
Finalize
“Beefy Box”
24 cores,
256G mem
6. Measure everything principle
● Start time, end time, duration in seconds, and sample count for every step
in the pipeline. Also the full end-to-end processing time.
● Put the data in pivot tables and graphed each step
● Normalize the data (sample size was changing)
● Use the data collected to predict future performance
6
7. Challenges and pain points
Performance degrades when DNA pool grows
● Static (by batch size)
● Linear (by DNA pool size)
● Quadratic (matching related steps) – time bomb
7
8. Ethnicity step on Hadoop
Using Hadoop as a job scheduler to scale AdMixture
8
9. First step with Hadoop
● What was in place?
● Smart engineers with no Hadoop experience
● Pipeline running on a single computer that would not scale
● New business that needed a scalable solution to grow
First step using Hadoop
● Run AdMixture step in parallel on Hadoop
● Self contained program with set inputs and outputs
● Simple MapReduce implementation
● Experience running jobs on Hadoop
● Freed up CPU and memory on the single computer for the other steps 9
10. What did we do? (Don’t cringe…)
1. Mapper -> Key: User ID, Ethnicity Result
2. Reducer -> Key: User ID, Array [Ethnicity Result]
10
DNA Hadoop Cluster
(40 nodes)
Pipeline Beefy Box
(Process Step Control)
For a batch of 1000
Submit 40 jobs with
25 samples per job
AdMixture Job #1
AdMixture Job #2
AdMixture Job #40
...
Mapper that forks
AdMixture
sequentually for
each sample Reducer #1
Reducer #2
Reducer #3
...
Results go to a
simple reducer that
merges them into a
single results file
(Shuffle)
11. Performance results
● Went from processing 500 samples in 20 hours
to processing 1,000 samples in 2 ½ hours
● Reduced Beagle phasing step by 4 hours
11
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
AdMixture Time (sec)
Sum of Run Size Admixture Time
Provided valuable experience and bought us time
H1 Release
AdMixture on
Hadoop
13. Introducing … GERMLINE!
● GERMLINE is an algorithm that finds hidden relationships within a pool of DNA
● Also refers to the reference implementation of that algorithm written in C++. You
can find it here:
So what’s the problem?
● GERMLINE (the implementation) was not meant to be used in an industrial setting
● Stateless, single threaded, prone to swapping (heavy memory usage)
● GERMLINE performs poorly on large data sets
● Our metrics predicted exactly where the process would slow to a crawl
● Put simply: GERMLINE couldn't scale
13
http://www1.cs.columbia.edu/~gusev/germline/
17. DNA matching : How it works
17
Cersei : ACTGACCTAGTTGAC
Joffrey : TTAAGCCTAGTTGAC
The Input
Cersei Baratheon
• Former queen of
Westeros
• Machiavellian
manipulator
• Mostly evil, but
occasionally
sympathetic
Joffrey Baratheon
• Pretty much the human
embodiment of evil
• Needlessly cruel
• Kinda looks like Justin
Bieber
18. DNA matching : How it works
18
0 1 2
Cersei : ACTGA CCTAG TTGAC
Joffrey : TTAAG CCTAG TTGAC
Separate into words
19. DNA matching : How it works
19
0 1 2
Cersei : ACTGA CCTAG TTGAC
Joffrey : TTAAG CCTAG TTGAC
ACTGA_0 : Cersei
TTAAG_0 : Joffrey
CCTAG_1 : Cersei, Joffrey
TTGAC_2 : Cersei, Joffrey
Build the hash table
20. DNA matching : How it works
20
0 1 2
Cersei : ACTGA CCTAG TTGAC
Joffrey : TTAAG CCTAG TTGAC
ACTGA_0 : Cersei
TTAAG_0 : Joffrey
CCTAG_1 : Cersei, Joffrey
TTGAC_2 : Cersei, Joffrey
Iterate through genome and find matches
Cersei and Joffrey match from position 1 to position 2
22. IBD to relationship estimation
● We use the total length of all shared
segments to estimate the relationship
between two genetic relatives
● This is basically a classification problem
22
5 10 20 50 100 200 500 1000 5000
0.000.010.020.030.040.05
ERSA
total_IBD(cM)
probability
m1
m2
m3
m4
m5
m6
m7
m8
m9
m10
m11
23. But wait...what about Jaime?
23
Jaime : TTAAGCCTAGGGGCG
Jaime Lannister
• Kind of a has-been
• Killed the Mad King
• Has the hots for his
sister, Cersei
24. The way
24
Step one: Update the hash table
Cersei Joffrey
2_ACTGA_0 1
2_TTAAG_0 1
2_CCTAG_1 1 1
2_TTGAC_2 1 1
Already stored in HBase
Jaime : TTAAG CCTAG GGGCG New sample to add
Key : [CHROMOSOME]_[WORD]_[POSITION]
Qualifier : [USER ID]
Cell value : A byte set to 1, denoting that the user has that word at that position on that
chromosome
25. The way
25
Step two: Find matches, update the results table
Already stored in HBase
Jaime and Joffrey match from position 0 to position 1
Jaime and Cersei match at position 1 New matches to add
Key : [CHROMOSOME]_[USER ID]
Qualifier : [CHROMOSOME]_[USER ID]
Cell value : A list of ranges where the two users match on a chromosome
2_Cersei 2_Joffrey
2_Cersei { (1, 2), ...}
2_Joffrey { (1, 2), ...}
29. Parallelism with Hadoop
● Batches are usually about a thousand people
● Each mapper takes a single chromosome for a single person
● MapReduce jobs:
● Job #1: Match words
- Updates the hash table
● Job #2: Match segments
- Identifies areas where the samples match
29
30. Matching steps on Hadoop
1. Hash Table Mapper -> Breaks input into words and fills the hash table (HBase Table #1)
2. Hash Table Reducer -> default reducer (does nothing)
3. Results Mapper -> For each new user, read hash table, fill in the results table (HBase
Table #2)
4. Results Reducer -> Key: Object(User ID #1, User ID #2), Array[Object(chrom + pos,
Matching DNA Segment)] 30
DNA Hadoop Cluster
(40 nodes)
Pipeline Beefy Box
(Process Step Control)
Hash Table
Mapper #1
Hash Table
Mapper #2
Hash Table
Mapper #N
...
Hash Table
Reducer #1
Hash Table
Reducer #2
Hash Table
Reducer #N
...
Results
Mapper #1
Results
Mapper #2
Results
Mapper #N
...
Results
Reducer #1
Results
Reducer #2
Results
Reducer #N
...1 2 3 4
Input:
User ID, Phased DNA
31. Run times for matching with
31
Hours
Samples
A 1700% performance improvement
over GERMLINE!
0
5
10
15
20
25
2,500
7,500
12,500
17,500
22,500
27,500
32,500
37,500
42,500
47,500
52,500
57,500
62,500
67,500
72,500
77,500
82,500
87,500
92,500
97,500
102,500
107,500
112,500
117,500
32. Run times for matching (in hours)
32
Hours
Samples
0
20
40
60
80
100
120
140
160
180
2,500
7,500
12,500
17,500
22,500
27,500
32,500
37,500
42,500
47,500
52,500
57,500
62,500
67,500
72,500
77,500
82,500
87,500
92,500
97,500
102,500
107,500
112,500
117,500
GERMLINE run
times
Jermline run
times
Projected
GERMLINE run
times
33. performance
● Science team is sure the Jermline algorithm is linear
● Improving the accuracy
● Found a bug in original C++ reference code
● Balancing false positives and false negatives
● Binary version of Jermline
● Use less memory and improve speed
● Paper submitted describing the implementation
● Releasing as an Open Source project soon 33
35. Phasing goes to the dogs
● Beagle
● Open source, freely available program
● Multi-threaded process that runs on one computer
● More accurate with a large sample set
● Underdog
● Does the same statistical calculations with a larger reference set, which increases accuracy
● Carefully split into a MapReduce implementation that allows parallel processing
● Collaboration between the DNA Science and Pipeline Developer Teams
35
37. Underdog performance
● Went from 12 hours to process 1,000 samples
to under 25 minutes with a MapReduce implementation
37With improved accuracy!
Underdog
replaces Beagle
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
Total Run Size Total Beagle-Underdog Duration
39. Pipeline steps and incremental change…
● Incremental change over time
● Supporting the business in a “just in time” Agile way
39
0
50000
100000
150000
200000
250000
500
5618
9615
14446
19522
24820
31172
38307
45252
52232
61675
73407
84337
93937
104684
115758
127194
138601
149988
161616
173719
185701
197853
209575
221290
232673
243550
255111
267266
279210
291017
302658
314662
326704
338813
350854
362954
375161
387334
399512
Beagle-Underdog Phasing
Pipeline Finalize
Relationship Processing
Germline-Jermline Results Processing
Germline-Jermline Processing
Beagle Post Phasing
Admixture
Plink Prep
Pipeline Initialization
Jermline replaces
Germline
Ethnicity V2
Release
Underdog Replaces
Beagle
AdMixture on
Hadoop
40. …while the business continues to grow rapidly
40
-
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
450,000
Jan-12 Apr-12 Jul-12 Oct-12 Jan-13 Apr-13 Jul-13 Oct-13 Jan-14 Apr-14
#ofprocessedsamples)
DNA Database Size
41. What’s next? Building different pipelines
● Azkaban
● Allows us to easily tie together steps on Hadoop
● Drop different steps in/out and create different pipelines
● Significant improvement over a hand coded pipeline
● Cloud
● New algorithm changes will force a complete re-run of the entire DNA pool
● Best example: New matching or ethnicity algorithm will force us to reprocess 400K+
samples
● Solution: Use the cloud for this processing while the current pipeline keeps chugging along
41
42. What’s next? Other areas for improvement
● Admixture as a MapReduce implementation
● Last major algorithm that needs to be addressed
● Expect to get performance improvements similar to Underdog
● Matching growth will cause problems
● Matches per run increasing
● Change the handoff
42Keep measuring everything and adjust
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
8,000,000
9,000,000
84337
92130
101143
109887
119669
129170
138601
148166
157710
167551
177654
187745
197853
207799
217516
226958
236516
245548
255111
265284
275335
285149
295020
312617
322655
332777
342854
352904
362954
373109
383312
393394
Jermline Processing Duration Batch Run Size Total Matches Per Run
43. Questions?
Ancestry is hiring for the DNA Pipeline Team!
Tech Roots Blog: http://blogs.ancestry.com/techroots
byetman@ancestry.com
Special thanks to the DNA Science and Pipeline Development Teams at
Ancestry
Editor's Notes
Critically important. In software development you must measure your performance at every step. Does not matter what you are doing, if you are not measuring your performance, you can’t improve. The last point is critical. We could determine the formula for performance of key phases (correlate this) and used that formula to predict future performance at particular DNA pool sizes. We could see the problems coming and knew when we were going to have performance issues.
Story #1: Our first step that was going out of control (going quadratic), was the first implementation of the relationship calculation – happens just after matching. This step was basically two nested for loops that walked over the entire DNA pool for each input sample. Simple code, it worked with small numbers, fell over fast. Time was approaching 5 hours to run. Two of my developers rewrote this in PERL and got it down to 2 minutes 30 seconds. They were ecstatic. One of our DNA Scientists (PhD in Bioinformatics, MS in Computer Science – he knows how to code) wrote an AWK command (nasty regular expressions) that ran in less than 10 seconds. My devs were humbled. For the next week, whenever they ran into Keith, they formally bowed to his skills. (All in good nature, all fun.)
Static by batch size (Phasing). Some steps took a long time but were very consistent. A worry but not critical to fix up front.
Linear by DNA Pool size (Pipeline Initialization). Looked at ways to streamline and improve performance of these steps.
Quadratic – those are the time bombs (Germline, Relationship processing, Germline results processing)
The only way we knew this was coming was because we measured each step in the pipeline.
‘G’ Germline would have to rebuild the hash table for all samples and then re-run all comparisons. An all by all comparison
This is where HBase shines. It is easy to add columns and rows, very efficient with empty cells (sparse matrix). Hammer HBase with multiple processes doing this at the same time.