![Step one : Update the hash table.
Starbuck Adama
2_ACTGA_0 1
2_TTAAG_0 1
2_CCTAG_1 1 1
2_TTGAC_2 1 1
Already stored in HBase
Baltar : TTAAG CCTAG GGGCG New sample to add
Key : [CHROMOSOME]_[WORD]_[POSITION]
Qualifier : [USER ID]
Cell value : A byte set to 1, denoting that the user has that word at that position on
that chromosome
The way](https://image.slidesharecdn.com/casestudies-session7b-130708184631-phpapp02/85/HBaseCon-2013-Apache-HBase-Apache-Hadoop-DNA-and-YOU-22-320.jpg)
![Baltar and Adama match from position 0 to position 1
Baltar and Starbuck match at position 1
Already stored
in HBase
2_Starbuck 2_Adama
2_Starbuck { (1, 2), ...}
2_Adama { (1, 2), ... }
New matches
to add
Key : [CHROMOSOME]_[USER ID]
Qualifier : [CHROMOSOME]_[USER ID]
Cell value : A list of ranges where the two users match on a chromosome
The way
Step two : Find matches.](https://image.slidesharecdn.com/casestudies-session7b-130708184631-phpapp02/85/HBaseCon-2013-Apache-HBase-Apache-Hadoop-DNA-and-YOU-23-320.jpg)
Uploaded byCloudera, Inc.
HBaseCon 2013: Apache HBase, Apache Hadoop, DNA and YOU!
Ancestry.com is the world's largest online family history resource with over 30,000 historical collections, 11 billion records, and 4 petabytes of data. It allows users to spit in a tube, pay $99, and learn about their family origins and find long-lost relatives through Ancestry DNA, which has over 120,000 samples in its database. However, the original DNA matching algorithm called GERMLINE did not scale well to large data sets. Ancestry.com developed its own algorithm called Jermline that uses Hadoop and HBase to match DNA samples in parallel across a cluster of servers, providing a 1700% performance improvement over GERMLINE.
More Related Content
Similar to HBaseCon 2013: Apache HBase, Apache Hadoop, DNA and YOU!
More from Cloudera, Inc.
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
HBaseCon 2013: Apache HBase, Apache Hadoop, DNA and YOU!
- 1. DNA, HBase, Hadoop,and YOU! by Jeremy Pollack
- 2. What does Ancestry.comdo? • Over 30,000 historical content collections • 11 billion records and images • Records dating back to 16th century • 4 petabytes We are the world's largest online family history resource.
- 3. It’s the “eureka”moment of discovery that drives our business!
- 4. DNA molecule 1differs from DNA molecule 2 at a single base-pair location (a C/T polymorphism). (http://en.wikipedia.org/wiki/Single- nucleiotide_polymorphism) What does Ancestry DNA do? "Spit in a tube, pay $99, learn about your past" • Decodes your family origins (ethnicity) • Finds your long-lost relatives • We have identified over four million fourth cousins. • The average customer has close to 30 fourth cousin matches. • By examining these matches, we can connect your family tree to those of your distant relatives. • Ancestry DNA has 120K+ samples, one of the largest DNA databases in the world. • About 690GB of data (uncompressed), or about 6.2 MB per sample
- 5. What is GERMLINE? •GERMLINE is an algorithm that finds hidden relationships within a pool of DNA. • GERMLINE also refers to the reference implementation of that algorithm. • You can find it here : http://www1.cs.columbia.edu/~gusev/germline/
- 6. So what's theproblem? • GERMLINE (the implementation) was not meant to be used in an industrial setting. • Stateless • Single threaded • Prone to swapping • GERMLINE performs poorly on large data sets. • We were running up against its limitations. • Put simply : GERMLINE couldn't scale.
- 7.
- 8. Projected GERMLINE RunTimes (in hours) 0 100 200 300 400 500 600 700 2500 5000 7500 10000 12500 15000 17500 20000 22500 25000 27500 30000 32500 35000 37500 40000 42500 45000 47500 50000 52500 55000 57500 60000 62500 65000 67500 70000 72500 75000 77500 80000 82500 85000 87500 90000 92500 95000 97500 100000 102500 105000 107500 110000 112500 115000 117500 120000 122500 Hours Number of samples GERMLINE run times Projected GERMLINE run times
- 9. The Mission :Create a Scalable Matching Engine ... and thus was born (aka "Jermline with a J")
- 10. DNA Matching :How it Works
- 11. Starbuck : ACTGACCTAGTTGAC Adama: TTAAGCCTAGTTGAC The Input Kara Thrace, aka Starbuck • Ace viper pilot • Has a special destiny • Not to be trifled with Admiral Adama • Admiral of the Colonial Fleet • Routinely saves humanity from destruction • Not so great with model ships
- 12. 0 1 2 Starbuck: ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Separate into words
- 13. 0 1 2 Starbuck: ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC ACTGA_0 : Starbuck TTAAG_0 : Adama CCTAG_1 : Starbuck, Adama TTGAC_2 : Starbuck, Adama Build the hash table
- 14. Iterate through genomeand find matches Starbuck and Adama match from position 1 to position 2 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC ACTGA_0 : Starbuck TTAAG_0 : Adama CCTAG_1 : Starbuck, Adama TTGAC_2 : Starbuck, Adama
- 15. Does that meanthey're related? ...maybe...
- 16. Baltar : TTAAGCCTAGGGGCG Butwait... what about Baltar? Gaius Baltar • Handsome • Genius • Kinda evil
- 17. Adding a newsample, the GERMLINE way
- 18. 0 1 2 Starbuck: ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar Step one : Rebuild the entire hash table from scratch, including the new sample The GERMLINE Way
- 19. Starbuck and Adamamatch from position 1 to position 2 Adama and Baltar match from position 0 to position 1 Starbuck and Baltar match at position 1 Step two : Find everybody's matches all over again, including the new sample. (n x n comparisons) 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar The GERMLINE Way
- 20. Starbuck and Adamamatch from position 1 to position 2 Adama and Baltar match from position 0 to position 1 Starbuck and Baltar match at position 1 Step three : Now, throw away the evidence! 0 1 2 Starbuck : ACTGA CCTAG TTGAC Adama : TTAAG CCTAG TTGAC Baltar : TTAAG CCTAG GGGCG ACTGA_0 : Starbuck TTAAG_0 : Adama, Baltar CCTAG_1 : Starbuck, Adama, Baltar TTGAC_2 : Starbuck, Adama GGGCG_2 : Baltar You have done this before, and you will have to do it ALL OVER AGAIN. The GERMLINE Way
- 21. Not so good,right? Now let's take a look at the way.
- 22. Step one :Update the hash table. Starbuck Adama 2_ACTGA_0 1 2_TTAAG_0 1 2_CCTAG_1 1 1 2_TTGAC_2 1 1 Already stored in HBase Baltar : TTAAG CCTAG GGGCG New sample to add Key : [CHROMOSOME]_[WORD]_[POSITION] Qualifier : [USER ID] Cell value : A byte set to 1, denoting that the user has that word at that position on that chromosome The way
- 23. Baltar and Adamamatch from position 0 to position 1 Baltar and Starbuck match at position 1 Already stored in HBase 2_Starbuck 2_Adama 2_Starbuck { (1, 2), ...} 2_Adama { (1, 2), ... } New matches to add Key : [CHROMOSOME]_[USER ID] Qualifier : [CHROMOSOME]_[USER ID] Cell value : A list of ranges where the two users match on a chromosome The way Step two : Find matches.
- 24. But wait ...what about Zarek, Roslin, Hera, and Helo?
- 25. Photo by BenhLieu Song Run them in parallel with Hadoop!
- 26. • Batches areusually about a thousand people. • Each mapper takes a single chromosome for a single person. • MapReduce Jobs : • Job #1 : Match Words • Updates the hash table • Job #2 : Match Segments • Identifies areas where the samples match Parallelism with Hadoop
- 27. Okay, but howdoes Jermline perform?
- 28. Okay, but howdoes Jermline perform? A 1700% improvement over GERMLINE!
- 29.
- 30. Run Times ForMatching (in hours) 0 20 40 60 80 100 120 140 160 180 2500 5000 7500 10000 12500 15000 17500 20000 22500 25000 27500 30000 32500 35000 37500 40000 42500 45000 47500 50000 52500 55000 57500 60000 62500 65000 67500 70000 72500 75000 77500 80000 82500 85000 87500 90000 92500 95000 97500 100000 102500 105000 107500 110000 112500 115000 117500 120000 Hours Number of samples GERMLINE run times Jermline run times Projected GERMLINE run times
- 31. Bottom line :By leveraging Hadoop and HBase, we dramatically increased our processing capacity. Without Hadoop and HBase, this would have been hideously expensive and difficult. • Previously, we ran GERMLINE on a single "beefy box". • 12-core 2.2GHZ Opteron 6174 with 256GB of RAM • We had upgraded this machine until it couldn't be upgraded any more. • Processing time was unacceptable, growth was unsustainable. • To continue running GERMLINE on a single box, we would have required a vastly more powerful machine, probably at the supercomputer level. • Now, we run Jermline on a cluster. • 20 X 12-core 2GHZ Xeon E5-2620 with 96GB of RAM • We can now run 16 batches per day, whereas before we could only run one. • Most importantly, growth is sustainable. To add capacity, we need only add more nodes.
- 32.
Editor's Notes
- #3 Everything from birth certificates, obituaries, immigration records, census records, voter registration, old phone books, everything.
- #4 Typically, the way it works is this :You search through our records to find one of your relatives. Once you've found enough records that you're satisfied you've found your relative, you attach them to your family tree. After that, Ancestry goes to work for you. Our search engine takes a look at your whole tree to find relatives that you may not know about yet, and presents these to you as hints. (shaky leaf) You can then examine these hints and see if they are, in fact, related to you. I, myself, found all my great-grandparents as well as a few aunts and uncles like that. It's pretty cool! And the beauty of it is that, say you've found a relative who's researched their family tree pretty extensively? Well, you get to piggyback on all that research by simply adding their family tree to yours. A fine example of crowdsourcing.
- #5 However, this has its limitations. What if you don't know your extended family that well? What if your ancestors came to the country as slaves? What if your ancestors came into the country illegally?DNA to the rescue.
- #8 Mention how we kept upgrading and tightening things up
- #16 For each person-to-person comparison, we add up the total length of their shared DNA and run that through a statistical model to see how closely they're related.
- #19 Remind people that GERMLINE was stateless