Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Data Products using Hadoop at Linkedin - Mitul Tiwari

3,779 views

Published on

Hadoop and other big data tools such as Voldemort, Azkaban, and Kafka, drive many data driven products at LinkedIn such as “People You MayKnow” and various recommendation products such as “Jobs You May Be Interested In”. Each of these products can be viewed as a large scale social recommendation problems, which analyzes billions of possible options, and suggest appropriate recommendation.

Since these products analyzes billions of edges and terabytes of data daily, it can be built only using a large scale distributed compute infrastructure. Kafka publish-subscribe messaging system is used to get the data in Hadoop file system. Hadoop MapReduce is used as the basic building block to analyze billions of potential options, and predict recommendation. Over a hundred MapReduce tasks are combined together in a work-flow uising Azkaban, a Hadoop work-flow management tool. The output of Hadoop jobs is finally stored in Voldemort key-value store to serve the data at run-time for efficiency.

During this talk audience will get a basic understanding of link prediction problem behind “ People You May Know” feature, which is a large scale social recommendation problem. Overview of the solution of this problem using Hadoop MapReduce, Azkaban workflow management tool, and Voldemort key-value store will be presented. I will also describe how to efficiently compute the number of common connections (triangle closing) using Hadoop Mapreduce, which is one of the many signals in link prediction.

Overall, people interested in building interesting applications using Hadoop MapReduce will hugely benefit from this talk.

Published in: Technology, Business
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/2ZDZFYj ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ❶❶❶ http://bit.ly/2ZDZFYj ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Data Products using Hadoop at Linkedin - Mitul Tiwari

  1. 1. Building Data Products using Hadoop at Linkedin <ul><li>Mitul Tiwari </li></ul><ul><li>Search, Network, and Analytics (SNA) </li></ul><ul><li>LinkedIn </li></ul>
  2. 2. Who am I?
  3. 3. <ul><li>What do I mean by Data Products? </li></ul>
  4. 4. People You May Know
  5. 5. Profile Stats: WVMP
  6. 6. Viewers of this profile also ...
  7. 7. Skills
  8. 8. InMaps
  9. 9. Data Products: Key Ideas <ul><li>Recommendations </li></ul><ul><ul><li>People You May Know, Viewers of this profile ... </li></ul></ul><ul><li>Analytics and Insight </li></ul><ul><ul><li>Profile Stats: Who Viewed My Profile, Skills </li></ul></ul><ul><li>Visualization </li></ul><ul><ul><li>InMaps </li></ul></ul>
  10. 10. Data Products: Challenges <ul><li>LinkedIn: 2nd largest social network </li></ul><ul><li>120 million members on LinkedIn </li></ul><ul><li>Billions of connections </li></ul><ul><li>Billions of pageviews </li></ul><ul><li>Terabytes of data to process </li></ul>
  11. 11. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  12. 12. Systems and Tools <ul><li>Kafka (LinkedIn) </li></ul><ul><li>Hadoop (Apache) </li></ul><ul><li>Azkaban (LinkedIn) </li></ul><ul><li>Voldemort (LinkedIn) </li></ul>
  13. 13. Systems and Tools <ul><li>Kafka </li></ul><ul><ul><li>publish-subscribe messaging system </li></ul></ul><ul><ul><li>transfer data from production to HDFS </li></ul></ul><ul><li>Hadoop </li></ul><ul><li>Azkaban </li></ul><ul><li>Voldemort </li></ul>
  14. 14. Systems and Tools <ul><li>Kafka </li></ul><ul><li>Hadoop </li></ul><ul><ul><li>Java MapReduce and Pig </li></ul></ul><ul><ul><li>process data </li></ul></ul><ul><li>Azkaban </li></ul><ul><li>Voldemort </li></ul>
  15. 15. Systems and Tools <ul><li>Kafka </li></ul><ul><li>Hadoop </li></ul><ul><li>Azkaban </li></ul><ul><ul><li>Hadoop workflow management tool </li></ul></ul><ul><ul><li>to manage hundreds of Hadoop jobs </li></ul></ul><ul><li>Voldemort </li></ul>
  16. 16. Systems and Tools <ul><li>Kafka </li></ul><ul><li>Hadoop </li></ul><ul><li>Azkaban </li></ul><ul><li>Voldemort </li></ul><ul><ul><li>Key-value store </li></ul></ul><ul><ul><li>store output of Hadoop jobs and serve in production </li></ul></ul>
  17. 17. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  18. 18. People You May Know Alice Bob Carol How do people know each other?
  19. 19. People You May Know Alice Bob Carol How do people know each other?
  20. 20. People You May Know Alice Bob Carol Triangle closing How do people know each other?
  21. 21. People You May Know Alice Bob Carol Triangle closing Prob(Bob knows Carol) ~ the # of common connections How do people know each other?
  22. 22. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  23. 23. Pig Overview <ul><li>Load: load data, specify format </li></ul><ul><li>Store: store data, specify format </li></ul><ul><li>Foreach, Generate: Projections, similar to select </li></ul><ul><li>Group by: group by column(s) </li></ul><ul><li>Join, Filter, Limit, Order, ... </li></ul><ul><li>User Defined Functions (UDFs) </li></ul>
  24. 24. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  25. 25. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  26. 26. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  27. 27. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  28. 28. Triangle Closing in Pig -- connections in (source_id, dest_id) format in both directions connections = LOAD `connections` USING PigStorage(); group_conn = GROUP connections BY source_id; pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2); common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections; STORE common_conn INTO `common_conn` USING PigStorage();
  29. 29. Triangle Closing Example Alice Bob Carol <ul><li>(A,B),(B,A),(A,C),(C,A) </li></ul><ul><li>(A,{B,C}),(B,{A}),(C,{A}) </li></ul><ul><li>(A,{B,C}),(A,{C,B}) </li></ul><ul><li>(B,C,1), (C,B,1) </li></ul>connections = LOAD `connections` USING PigStorage();
  30. 30. Triangle Closing Example Alice Bob Carol <ul><li>(A,B),(B,A),(A,C),(C,A) </li></ul><ul><li>(A,{B,C}),(B,{A}),(C,{A}) </li></ul><ul><li>(A,{B,C}),(A,{C,B}) </li></ul><ul><li>(B,C,1), (C,B,1) </li></ul>group_conn = GROUP connections BY source_id;
  31. 31. Triangle Closing Example Alice Bob Carol <ul><li>(A,B),(B,A),(A,C),(C,A) </li></ul><ul><li>(A,{B,C}),(B,{A}),(C,{A}) </li></ul><ul><li>(A,{B,C}),(A,{C,B}) </li></ul><ul><li>(B,C,1), (C,B,1) </li></ul>pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2);
  32. 32. Triangle Closing Example Alice Bob Carol <ul><li>(A,B),(B,A),(A,C),(C,A) </li></ul><ul><li>(A,{B,C}),(B,{A}),(C,{A}) </li></ul><ul><li>(A,{B,C}),(A,{C,B}) </li></ul><ul><li>(B,C,1), (C,B,1) </li></ul>common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn GENERATE flatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections;
  33. 33. Our Workflow triangle-closing
  34. 34. Our Workflow triangle-closing top-n
  35. 35. Our Workflow triangle-closing top-n push-to-prod
  36. 36. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  37. 37. Our Workflow triangle-closing top-n push-to-prod
  38. 38. Our Workflow triangle-closing top-n push-to-prod remove connections
  39. 39. Our Workflow triangle-closing top-n push-to-prod remove connections push-to-qa
  40. 40. PYMK Workflow
  41. 41. Workflow Requirements <ul><li>Dependency management </li></ul><ul><li>Regular Scheduling </li></ul><ul><li>Monitoring </li></ul><ul><li>Diverse jobs: Java, Pig, Clojure </li></ul><ul><li>Configuration/Parameters </li></ul><ul><li>Resource control/locking </li></ul><ul><li>Restart/Stop/Retry </li></ul><ul><li>Visualization </li></ul><ul><li>History </li></ul><ul><li>Logs </li></ul>
  42. 42. Workflow Requirements <ul><li>Dependency management </li></ul><ul><li>Regular Scheduling </li></ul><ul><li>Monitoring </li></ul><ul><li>Diverse jobs: Java, Pig, Clojure </li></ul><ul><li>Configuration/Parameters </li></ul><ul><li>Resource control/locking </li></ul><ul><li>Restart/Stop/Retry </li></ul><ul><li>Visualization </li></ul><ul><li>History </li></ul><ul><li>Logs </li></ul>Azkaban
  43. 43. Sample Azkaban Job Spec <ul><li>type=pig </li></ul><ul><li>pig.script=top-n.pig </li></ul><ul><li>dependencies=remove-connections </li></ul><ul><li>top.n.size=100 </li></ul>
  44. 44. Azkaban Workflow
  45. 45. Azkaban Workflow
  46. 46. Azkaban Workflow
  47. 47. Our Workflow triangle-closing top-n push-to-prod remove connections
  48. 48. Our Workflow triangle-closing top-n push-to-prod remove connections
  49. 49. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  50. 50. Production Storage <ul><li>Requirements </li></ul><ul><ul><li>Large amount of data/Scalable </li></ul></ul><ul><ul><li>Quick lookup/low latency </li></ul></ul><ul><ul><li>Versioning and Rollback </li></ul></ul><ul><ul><li>Fault tolerance </li></ul></ul><ul><ul><li>Offline index building </li></ul></ul>
  51. 51. Voldemort Storage <ul><ul><li>Large amount of data/Scalable </li></ul></ul><ul><ul><li>Quick lookup/low latency </li></ul></ul><ul><ul><li>Versioning and Rollback </li></ul></ul><ul><ul><li>Fault tolerance through replication </li></ul></ul><ul><ul><li>Read only </li></ul></ul><ul><ul><li>Offline index building </li></ul></ul>
  52. 52. Data Cycle
  53. 53. Voldemort RO Store
  54. 54. Our Workflow triangle-closing top-n push-to-prod remove connections
  55. 55. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  56. 56. Data Quality <ul><li>Verification </li></ul><ul><li>QA store with viewer </li></ul><ul><li>Explain </li></ul><ul><li>Versioning/Rollback </li></ul><ul><li>Unit tests </li></ul>
  57. 57. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  58. 58. Performance <ul><li>Symmetry </li></ul><ul><ul><li>Bob knows Carol then Carol knows Bob </li></ul></ul><ul><li>Limit </li></ul><ul><ul><li>Ignore members with > k connections </li></ul></ul><ul><li>Sampling </li></ul><ul><ul><li>Sample k-connections </li></ul></ul>
  59. 59. Things Covered <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  60. 60. SNA Team <ul><li>Thanks to SNA Team at LinkedIn </li></ul><ul><li>http://sna-projects.com </li></ul><ul><li>We are hiring! </li></ul>
  61. 61. Questions?
  62. 62. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>
  63. 63. Outline <ul><li>What do I mean by Data Products? </li></ul><ul><li>Systems and Tools we use </li></ul><ul><li>Let’s build “People You May Know” </li></ul><ul><li>Managing workflow </li></ul><ul><li>Serving data in production </li></ul><ul><li>Data Quality </li></ul><ul><li>Performance </li></ul>

×