Download to read offline
![The
Starbuck
2_ACTGA_0
way
Adama
Step one : Update the hash table.
1
2_TTAAG_0
1
2_CCTAG_1
1
1
2_TTGAC_2
1
Already stored in HBase
1
Baltar : TTAAG CCTAG GGGCG
New sample to add
Add a column for every new sample for each user
Key : [CHROMOSOME]_[WORD]_[POSITION]
Qualifier : [USER ID]
Cell value : A byte set to 1, denoting that the user has that word at that
position on that chromosome](https://image.slidesharecdn.com/utahbigmountain-ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02-131112114437-phpapp02/85/Utahbigmountain-ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02-37-320.jpg)
![The
2_Starbuck
2_Starbuck
2_Adama
way
2_Adama
Step two : Find matches.
{ (1, 2), ...}
{ (1, 2), ...}
Baltar and Adama match from position 0 to position 1
Baltar and Starbuck match at position 1
Already
stored in
HBase
New
matches to
add
“Fuzzy Match” the consecutive words. Worst case: Identical twins
Key : [CHROMOSOME]_[USER ID]
Qualifier : [CHROMOSOME]_[USER ID]
Cell value : A list of ranges where the two users match on a chromosome](https://image.slidesharecdn.com/utahbigmountain-ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02-131112114437-phpapp02/85/Utahbigmountain-ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02-38-320.jpg)
More Related Content
Similar to Utahbigmountain ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02
Utahbigmountain ancestrydnahbasehadoop9-7-2013billyetman-130928100600-phpapp02
- 1. Ancestry DNA atScale Using Hadoop and HBase September 7, 2013 1
- 2. What does thistalk cover? What does Ancestry do? How did our journey with Hadoop start? Using Hadoop as a Job Processor DNA Matching with Hadoop and HBase What’s next? 2
- 3.
- 4. Discoveries Are theKey We are the world's largest online family history resource. • Over 30,000 historical content collections • 11 billion records and images • Records dating back to 16th century • 4 petabytes
- 5. Discoveries In Detail The“eureka” moment drives our business
- 6. Discoveries With DNA Spitin a tube, pay $99, learn your past Autosomal DNA tests Over 120,000 DNA samples 700,000 SNPs for each sample 6,000,000+ 4th cousin matches 150,000 100,000 Genotyped samples 50,000 - 6 DNA molecule 1 differs from DNA molecule 2 at a single base-pair location (a C/T polymorphism). (http://en.wikipedia.org/wiki/Singlenucleiotide_polymorphism)
- 7. What does thecustomer see? 7
- 8. Network Effect –Cousin Matches 3,500,000 Cousin Matches 3,000,000 2,500,000 2,000,000 1,500,000 1,000,000 500,000 2,000 10,053 21,205 40,201 Database Size 8 60,240 80,405 115,756
- 9. Where Did WeStart? The process before Hadoop 9
- 10. What’s the Story? Castof Characters (Scientists and Software Engineers) Scientists Software Engineers Think they can code: Think they are Scientists: • Linux • Biology in HS and College • MySQL • Math/Statistics • PERL and/or Python • Read science papers Pressures of a startup business – Release a product, learn, and then scale Sr. Manager and 5 developers and 4 member Science Team 10
- 11. DNA Input Raw Data(A,C,T,G,0): 3 123456789_RZZZZ2_XXXXXXH3Q7U7Q2B_YYYY84598-DNA 0 0 0 -9 C C G G G G G G A A A A C C G G A AAACCGGGGAAGGGAAAGGAGAACCAAAAGGAAAGGGGGCCGGAAGGGGGG G A A A A C G A A A A G A G A A A A G G G G G G A G G G G G G G … (continues for 700,000+ snips) Map File: 0 0 0 0 0 0 0 0 0 0 0 11 rs10005853 rs10015934 rs1004236 rs10059646 rs10085382 rs10123921 rs10127827 rs10155688 rs10162780 rs1017484 rs10188129 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
- 12. What Did “GetSomething Running” Look Like? Old Version Run Watch Dog B Init Rakesh Results Processing 3) Poll status Finalize Creates run Reruns Heart beat 2) Enqueuer (dna validation) Pipeline Control Monitor Monitor 4) Disc Management (V2) Runs on AdMixture (Ethnicity) Beagle (Phasing) and GermLine (Matching) runs here “Beefy Box” Single Beefy Box – Only option is to scale Vertically 12
- 13. Measure Everything Principle •Start time, end time, duration in seconds, and sample count for every step in the pipeline. Also the full end-toend processing time • Put the data in pivot tables and graphed each step • Normalize the data (sample size was changing) #1 • Use the data collected to predict future performance 13
- 14. Challenges and PainPoints Performance degrades when DNA pool grows • Static (by batch size) • Linear (by DNA pool size) • Quadratic (Matching related steps) – Time bomb (Courtesy from Keith’s Potting) 14
- 15. Parallel Ethnicity Jobs UseHadoop as a job processor 15
- 16. Why Attack EthnicityFirst? • Smart developers, little Hadoop experience – Using Hadoop as a job scheduler and scaling the ethnicity step was easier than redesigning the matching step • AdMixture is a self-contained application – Reference panel, the users DNA, and a seed value for inputs – CPU intensive job that writes to stdout • Easy to split up the input • Looked hard enough at the matching problem to realize a HBase, MapReduce solution was realistic 16
- 17. Parallel Ethnicity Jobs Typicalrun of 1000 samples. Queue up one Hadoop job with 40 tasks, 25 samples per task Hadoop Cluster (20 x 4 slots x 96g) Server Server Server Server Server Server Server Server Server Server 1) Map Reduce Admixture Admixture Admixture Admixture Admixture Admixture 17 Admixture Admixture Admixture #2
- 18. 2012-03-01T21:18:03 2012-03-31T16:27:50 2012-04-17T07:31:45 2012-05-17T18:36:08 2012-06-16T15:23:27 2012-06-29T19:42:18 2012-07-11T11:29:56 2012-07-22T07:48:32 2012-07-30T06:56:26 2012-08-08T20:42:30 2012-08-17T20:58:55 2012-09-01T01:51:54 2012-09-11T21:53:05 2012-09-23T21:46:15 2012-10-02T14:28:50 2012-10-14T17:45:53 2012-11-04T02:43:36 2012-11-24T11:12:19 2012-12-12T17:35:15 2012-12-25T04:36:45 2013-01-14T15:18:38 2013-01-29T12:29:56 2013-02-11T10:22:02 2013-03-02T16:03:16 2013-03-29T00:19:36 2013-04-21T02:02:51 2013-05-17T01:34:00 2013-05-29T07:08:04 2013-06-13T13:50:45 2013-06-25T21:06:04 2013-07-17T15:15:27 2013-08-06T07:57:41 Results 1000 sample runsunder 3 hours (one interesting bug) AdMixture Time (sec) 100000 90000 80000 70000 60000 50000 40000 30000 18 Sum of Run Size 20000 Admixture Time 10000 0
- 19. Freed up the“Beefy Box” • Moving AdMixture off left an additional 10 threads for phasing and matching • Memory was freed up for phasing and matching • Just moving AdMixture off, saved over 6 hours of processing on the single box – Bought us time 19
- 20. New Matching Algorithm Hadoopand HBase 20
- 21. What is GERMLINE? • • • GERMLINEis an algorithm that finds hidden relationships within a pool of DNA GERMLINE also refers to the reference implementation of that algorithm written in C++ You can find it here : http://www1.cs.columbia.edu/~gusev/germline/
- 22. So what's theproblem? • • • • GERMLINE (the implementation) was not meant to be used in an industrial setting • Stateless • Single threaded • Prone to swapping (heavy memory usage) • Generic • Used for any DNA (fish, fruit fly, human, …) GERMLINE performs poorly on large data sets Our metrics predicted exactly where the process would slow to a crawl Put simply : GERMLINE couldn't scale
- 23. Hours GERMLINE Run Times(in hours) 25 20 15 10 5 0 60000 57500 55000 52500 50000 47500 45000 42500 40000 37500 35000 32500 30000 27500 25000 22500 20000 17500 15000 12500 10000 7500 5000 2500 Number of samples
- 24. Hours Projected GERMLINE RunTimes (in hours) 700 600 500 400 300 200 GERMLINE run times 100 Projected GERMLINE run times 0 122500 120000 117500 115000 112500 110000 107500 105000 102500 100000 97500 95000 92500 90000 87500 85000 82500 80000 77500 75000 72500 70000 67500 65000 62500 60000 57500 55000 52500 50000 47500 45000 42500 40000 37500 35000 32500 30000 27500 25000 22500 20000 17500 15000 12500 10000 7500 5000 2500 Number of samples
- 25. The Mission :Create a Scalable Matching Engine ... and thus was born (aka "Jermline with a J")
- 26. DNA Matching :How it Works The Input Starbuck : ACTGACCTAGTTGAC Adama : TTAAGCCTAGTTGAC Kara Thrace, aka Starbuck • • • Ace viper pilot Has a special destiny Not to be trifled with Admiral Adama • • Admiral of the Colonial Fleet Routinely saves humanity from destruction
- 27. DNA Matching :How it Works Separate into words 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC
- 28. DNA Matching :How it Works Build the hash table 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC ACTGA_0 : Starbuck TTAAG_0 : Adama CCTAG_1 : Starbuck, Adama TTGAC_2 : Starbuck, Adama
- 29. DNA Matching :How it Works Iterate through genome and find matches 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC ACTGA_0 : Starbuck TTAAG_0 : Adama CCTAG_1 : Starbuck, Adama TTGAC_2 : Starbuck, Adama Starbuck and Adama match from position 1 to position 2
- 30. Does that meanthey're related? ...maybe
- 31. But wait... whatabout Baltar? Baltar : TTAAGCCTAGGGGCG Gaius Baltar • • • Handsome Genius Kinda evil
- 32. Adding a newsample, the GERMLINE way
- 33. The GERMLINE Way Stepone : Rebuild the entire hash table from scratch, including the new sample 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar
- 34. The GERMLINE Way Steptwo : Find everybody's matches all over again, including the new sample. (n x n comparisons) 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar Starbuck and Adama match from position 1 to position 2 Adama and Baltar match from position 0 to position 1 Starbuck and Baltar match at position 1
- 35. The GERMLINE Way Stepthree : Now, throw away the evidence! 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar Starbuck and Adama match from position 1 to position 2 Adama and Baltar match from position 0 to position 1 Starbuck and Baltar match at position 1 You have done this before, and you will have to do it ALL OVER AGAIN.
- 36. Not so good,right? Now let's take a look at the way.
- 37. The Starbuck 2_ACTGA_0 way Adama Step one :Update the hash table. 1 2_TTAAG_0 1 2_CCTAG_1 1 1 2_TTGAC_2 1 Already stored in HBase 1 Baltar : TTAAG CCTAG GGGCG New sample to add Add a column for every new sample for each user Key : [CHROMOSOME]_[WORD]_[POSITION] Qualifier : [USER ID] Cell value : A byte set to 1, denoting that the user has that word at that position on that chromosome
- 38. The 2_Starbuck 2_Starbuck 2_Adama way 2_Adama Step two :Find matches. { (1, 2), ...} { (1, 2), ...} Baltar and Adama match from position 0 to position 1 Baltar and Starbuck match at position 1 Already stored in HBase New matches to add “Fuzzy Match” the consecutive words. Worst case: Identical twins Key : [CHROMOSOME]_[USER ID] Qualifier : [CHROMOSOME]_[USER ID] Cell value : A list of ranges where the two users match on a chromosome
- 39. The Starbuck 2_ACTGA_0 way Adama Baltar 1 1 1 1 2_TTAAG_0 2_CCTAG_1 1 1 2_TTGAC_2 1 1 2_GGGCG_2 1 2_Starbuck 2_Adama { (1), ...} {(1), ...} { (1, 2), ...} 2_Baltar 2_Baltar { (1, 2), ...} 2_Starbuck 2_Adama { (0,1), ...} { (0,1), ...} These are the updated tables after adding Baltar’s information Only looking at 3 samples, chromosome #2, positions 0, 1, and 2 Very simple example of how the matching process works
- 40. But wait ...what about Zarek, Roslin, Hera, and Helo?
- 41. Run them inparallel with Hadoop! Photo by Benh Lieu Song
- 42. Parallelism with Hadoop • Batchesare usually about a thousand people. • Each mapper takes a single chromosome for a single person. o • Three samples per task means 22 jobs with 334 tasks (1000/3) each MapReduce Jobs : Job #1 : Match Words • Updates the hash table Job #2 : Match Segments • Identifies areas where the samples match
- 43. How does Jermlineperform? A 1700% improvement over GERMLINE! Along with more accurate results #3
- 44. Hours Run Times ForMatching (in hours) 25 20 15 10 5 0 120000 117500 115000 112500 110000 107500 105000 102500 100000 97500 95000 92500 90000 87500 85000 82500 80000 77500 75000 72500 70000 67500 65000 62500 60000 57500 55000 52500 50000 47500 45000 42500 40000 37500 35000 32500 30000 27500 25000 22500 20000 17500 15000 12500 10000 7500 5000 2500 Number of samples
- 45. 2500 5000 7500 10000 12500 15000 17500 20000 22500 25000 27500 30000 32500 35000 37500 40000 42500 45000 47500 50000 52500 55000 57500 60000 62500 65000 67500 70000 72500 75000 77500 80000 82500 85000 87500 90000 92500 95000 97500 100000 102500 105000 107500 110000 112500 115000 117500 120000 Hours Run Times ForMatching (in hours) 180 160 140 120 100 GERMLINE run times 80 60 Jermline run times 40 Projected GERMLINE run times 20 0 Number of samples
- 46. Incremental Changes OverTime • Support the business, move incrementally and adjust • After H2, pipeline speed stays flat • 46 (Courtesy from Bill’s plotting)
- 47. Dramatically Increased ourCapacity Bottom line : Without Hadoop and HBase, this would have been expensive and difficult. • Previously, we ran GERMLINE on a single "beefy box". • 12-core 2.2GHZ Opteron 6174 with 256GB of RAM • We had upgraded this machine until it couldn't be upgraded any more. • Processing time was unacceptable, growth was unsustainable. • To continue running GERMLINE on a single box, we would have required a vastly more powerful machine, probably at the supercomputer level – at considerable cost! • Now, we run Jermline on a cluster. • 20 X 12-core 2GHZ Xeon E5-2620 with 96GB of RAM • We can now run 16 batches per day, whereas before we could only run one. • Most importantly, growth is sustainable. To add capacity, we need only add more nodes.
- 48. What’s Next? Hadoop andHBase 48
- 49. Continue to Evolvethe Software • Azkaban for job control – Nearly complete • Phasing – Still runs on the “Beefy Box”, 1000 samples take over 11 hours – Total run time for 1000 samples is about 14 hours. – Re-implement with HBase, MapReduce, Hadoop • Version Updates – New algorithms require us to re-run the entire DNA pool – Burst capacity to the cloud • Machine Learning – Matching (V2) and Ethnicity (V3) both would benefit from a Machine Learning approach 49
- 50. End of theJourney (for now) - Questions? 50
Editor's Notes
- #3 Job Processor: As you will see, we started our Hadoop/DNA journey with something that was fairly basic and then we moved to the matching problemDNA Matching: We will walk through and example of how matching works, discuss how GERMLINE implemented the matching, and contrast that with the Hadoop/HBase implementation we created.
- #4 At Ancestry.com our mission is to help people discover, preserve and share their family history.
- #5 Everything from birth certificates, obituaries, immigration records, census records, voter registration, old phone books, everything.
- #6 Typically, the way it works is this :You search through our records to find one of your relatives. Once you've found enough records that you're satisfied you've found your relative, you attach them to your family tree. After that, Ancestry goes to work for you. Our search engine takes a look at your whole tree to find relatives that you may not know about yet, and presents these to you as hints. (shaky leaf) You can then examine these hints and see if they are, in fact, related to you. It's pretty cool! And the beauty of it is that, say you've found a relative who's researched their family tree pretty extensively? Well, you get to piggyback on all that research by simply adding their family tree to yours. A fine example of crowdsourcing.
- #7 Spit in a tube, pay $99 and learn about your past. That is how Derrick Harris of GigaOm described what we do. DNA is found in every living cell – it is the genetic material that encodes all of the information required to create and maintain life. DNA is passed down from parent to child and is like breadcrumbs left by our Ancestors. And changes in DNA across generations give us a view into history. We can take those breadcrumbs and determine with a large degree of accuracy what your ethnicity is and who else in our database might be your cousin. If we determine that you have a 4th cousin then you likely share a common ancestor with that person between 7 and 10 generations ago or 150 – 300 years ago. We have a team of data scientists and bioinformatics PhD’s working on this effort and have very quickly acquired over 120,000 DNA samples for people that have family trees on our site. Each DNA sample is composed of over 700,000 SNPs or location markers. In order to compare the 700,000 SNPs from each new sample with the 700,000 SNPs from each existing sample that is already in our database we have a sophisticated pipeline of algorithms that run using Hadoop, Hbase and MapReduce for parallel distributed processing.What is our confidence rate of a 4th cousin match?The average customer has close to 30 fourth cousin matches
- #8 Top left our ethnicity chart. To the right, Tree view with cousin hints and surnames in another member’s public tree. Maps pinpointing birth locations. List of surnames that appear in both trees.
- #9 The bottom red line is the size of our DNA pool (i.e. each unique sample in our database). The black line is the number of cousin matches we’ve calculated at the particular DNA pool size. As you can see, the matches start to compound and grow quadratically as the pool size increases. This is a good thing. It means we can find genetic relatives for most customers who take the DNA test.The cousin matches are actually a Big Data problem for our Front End. We are looking at different ways to handle the transfer, storage, and growth of the cousin match data as the DNA pool size increases.
- #11 Every scientist thinks they can code – because they have been doing it for a long time on their own or in an academic environment. But they don’t know what it means to build, deploy, support “production” code. Software engineers understand production code. They just think they understand the math and statistics – after all they are computer scientists. They can understand the science behind DNA, after all, they took Biology in high school. Nowhere near the education of a Bioinformatics or Population Geneticist PhD. The Science Team are the domain experts and the engineers are required to build a production system to meet the domain expert’s needs.Really started light 3 developers and 2 scientists. In fact, for the first 3 months we “borrowed” engineers from other projects to get this started.
- #12 5 possible values – not 4. A C T G and zero. Zero indicates a “read” failure at that position. No sample is perfect, extraction could be off, each run on the same sample will come up with zeros in different spots.QC checks on the sample. If there are too many “zeros” we have the lab try the extraction again. If that fails 2 more times, we issue a recollect (send another kit to the customer and ask them to submit their DNA again).Map file tells you where each value is on a particular chromosome
- #13 Ran AdMixture on 10 threads, Phasing and Germline on 10 threads. AdMixture would usually finish before Beagle (Phasing) and that freed up more memory and threads for Germline. In all, a 500 sample run took about 24 hours to complete (pool size < 25K)IF WE STAYED IN THIS CONFIGURATION (WHICH MATCHED MANY ACEDEMIC ENVIRONMENTS) THE ONLY OPTION WAS TO INCREASE THE HARDWARE. MORE CPUS, MORE MEMORY. SCALING VERTICALLY JUST PLAIN SUCKS!
- #14 Critically important. In software development you must measure your performance at every step. Does not matter what you are doing, if you are not measuring your performance, you can’t improve. The last point is critical. We could determine the formula for performance of key phases (correlate this) and used that formula to predict future performance at particular DNA pool sizes. We could see the problems coming and knew when we were going to have performance issues.Story #1: Our first step that was going out of control (going quadratic), was the first implementation of the relationship calculation – happens just after matching. This step was basically two nested for loops that walked over the entire DNA pool for each input sample. Simple code, it worked with small numbers, fell over fast. Time was approaching 5 hours to run. Two of my developers rewrote this in PERL and got it down to 2 minutes 30 seconds. They were ecstatic. One of our DNA Scientists (PhD in Bioinformatics, MS in Computer Science – he knows how to code) wrote an AWK command (nasty regular expressions) that ran in less than 10 seconds. My devs were humbled. For the next week, whenever they ran into Keith, they formally bowed to his skills. (All in good nature, all fun.)
- #15 Static by batch size (Phasing). Some steps took a long time but were very consistent. A worry but not critical to fix up front.Linear by DNA Pool size (Pipeline Initialization). Looked at ways to streamline and improve performance of these steps.Quadratic – those are the time bombs (Germline, Relationship processing, Germline results processing)The only way we knew this was coming was because we measured each step in the pipeline.
- #17 KEY POINT: We knew we wanted to move this to Hadoop to solve matching. With that end goal in mind, we attacked the AdMixture/Ethnicity step. Without that initial investigation and discovery step, we could have used an MPI Linux environment or some other way to scale AdMixture.
- #18 Story 2: First job we put through was a single job with 500 tasks (sample size of 500). AdMixture is a C++, multithreaded App. When we kicked up all the tasks, it did not leave enough CPU time for the “task health check” to run in the background on the Hadoop node. So the Job Controller would reach out and kill some jobs because they were “misbehaving” – when in fact they were running just fine. Remember Hadoop is intimately aware of the JVM and how it is running. Hadoop does not have a good view into other applications you choose to run. Since AdMixture was C++, Hadoop had no idea how much memory, threads, or CPU was being used per “slot”. We had to back things off, so there was enough room for the Job Controller to get an “ACK” indicating the jobs were running fine.THE ONLY WAY TO UNDERSTAND HADOOP’S CAPABILITIES AND LIMITATIONS IS TO USE IT! BE READY FOR SOME SURPRISES.
- #19 Really happy with this performance. 1000 samples usually run in 2 hours 30 minutes to 2 hours 45 minutes. Two spikes to explain (this is the bug):There is a bug in AdMixture (remember, created by someone who wanted to finish their CS Masters for an academic situation) that showed up occasionally. The program would literally “get lost” and never complete. It would not GPF or throw an error, it just swallowed up the CPU and never completed. Even worse, it usually happened on chromosome 1 or 2, the biggest chromosomes we process. We put a timeout on our tasks. If a task did not finish in 2 hours, we killed it, changed the seed value and resubmitted a new task. This fixes the problem. That explains the spikes.
- #20 This was a great first step. We got valuable Hadoop, MapReduce, job control experience and this first step BOUGHT US TIME!It gave us the confidence to start working on the GERMLINE matching problem.
- #22 Very smart people at Columbia University came up with GERMLINE.
- #23 Remember, for an academic, running a 1000 sample set through GERMLINE was “large”. I’ve talked to people who kept re-running the same 50 fish DNA samples through GERMLINE to clean up the variations between sample extractions (think of it as eliminating all the zeros).In a lot of ways, we were using GERMLINE in a way that it was not built for.
- #24 Mention how we kept upgrading and tightening things up
- #25 Our projections showed how bad the execution time would get. As we approached 120K for the DNA pool size, each additional 500 sample set would require 700 hours to complete – over 4 weeks.
- #26 Germline with a “J” (lead engineer’s first name is Jeremy)This was a “clean room” implementation of the algorithm. Read the reference paper, don’t look at the C++ reference implementation. Work off the original (brilliant) paper.
- #27 Using BattlestarGallactica for the matching example.
- #31 For each person-to-person comparison, we add up the total length of their shared DNA and run that through a statistical model to see how closely they're related. This is the “Relationship Calculation” step that works on the GERMLINE output.
- #34 Remind people that GERMLINE was stateless
- #35 Anytime you see an N-by-N comparison in a computer problem you are working on it should send up huge red flags.
- #38 HBase holds the data. (Mix between a spread sheet and a hash table.) Adding columns is easy. Having a very sparse matrix is fine. Key is the chromosome, the word value, and position (which word). Each new sample adds a column to the table. A value of 1 in the cell indicates this user has this value at this location. A row holds all the samples with that same value in our DNA poolsize.This is really a pretty simple implementation. Remember: SIMPLE SCALES.
- #39 There is a second table for the fuzzy matching phase. It holds the list of ranges where two users match on a chromosome. This is used to create the output of the matching phase. Exactly where two individuals match on each chromosome.
- #41 There were a whole bunch of characters on BattlestarGallactica!
- #43 First run we kicked off one job with (500 samples x 22 chromosomes) 11,000 tasks using Hbase 0.92 and we panicked the HBase region server. That’s where we came up with 22 jobs (one for each chromosome) with about 334 tasks per job. (Moved to HBase 0.94 was much more stable)
- #44 Story #3: We would run samples through the old GERMLINE and the new HadoopJermline. For the most part, they always matched. We finally found a few runs where there were discrepancies. We had to pull in the Science Team to check – we had actually found a bug in the original GERMLINE implementation for an edge case. The clean room implementation of the Hadoop code was “more correct” than the original C++ GERMLINE reference code. Very gratifying to see – but the truth is it had us concerned and confused for about 3 days.Made the natural assumption that the base implementation GERMLINE (with a ‘G’) was 100% correct. That assumption was wrong.
- #45 This slide is a huge relief. We’ve been released and steady for a while. One note, the curve for H2 is not totally flat. It is going up ever so slightly. No worries. We can always add more nodes to the cluster and reduce the time.
- #47 This is an “Agile” development story.Point out a few colors: Dark Green, Orange, Light Green at the top, and the PurpleThe darker green is AdMixture, you can see when we moved to Hadoop (our H1 Release)Orange is the Matching Step (Germline to Jermline, our H2 Release)The lighter green is the pipeline finalization step. We eliminated most of this step when we released H2. We had a failsafe way to fallback to the completed steps of the previous run. We never wanted to fail in the middle of a run, destroy everything, and then have to rerun the entire pool from scratch. Key part of finalization pre-JermlineThe Purple is Phasing (Beagle). Static based on input size, very stable. On our hit list.
- #48 The “Beefy Box” would be a good candidate for a large database server or a single node on a heavily used distributed cache (Memcache-D or Redis)