Building Data Driven Products at Linkedin

Building Data Productsusing Hadoop at Linkedin Mitul Tiwari Search, Network, and Analytics (SNA) LinkedIn 1 1

Who am I? 2 2

What do I mean by Data Products? 3 3

People You May Know 4 4

Proﬁle Stats: WVMP 5 5

Viewers of this proﬁle also ... 6 6

Skills 7 7

InMaps 8 8

Data Products: Key IdeasRecommendations People You May Know, Viewers of this profile ...Analytics and Insight Profile Stats: Who Viewed My Profile, SkillsVisualization InMaps 9 9

Data Products: Challenges LinkedIn: 2nd largest social network 120 million members on LinkedIn Billions of connections Billions of pageviews Terabytes of data to process 10 10

OutlineWhat do I mean by Data Products?Systems and Tools we useLet’s build “People You May Know”Managing workﬂowServing data in productionData QualityPerformance 11 11

Systems and ToolsKafka (LinkedIn)Hadoop (Apache)Azkaban (LinkedIn)Voldemort (LinkedIn) 12 12

Systems and ToolsKafka publish-subscribe messaging system transfer data from production to HDFSHadoopAzkabanVoldemort 13 13

Systems and ToolsKafkaHadoop Java MapReduce and Pig process dataAzkabanVoldemort 14 14

Systems and ToolsKafkaHadoopAzkaban Hadoop workﬂow management tool to manage hundreds of Hadoop jobsVoldemort 15 15

Systems and ToolsKafkaHadoopAzkabanVoldemort Key-value store store output of Hadoop jobs and serve in production 16 16

People You May Know How do people Aliceknow each other? Bob Carol 18 18

People You May Know How do people Aliceknow each other? Bob Carol 19 19

People You May Know How do people Aliceknow each other? Bob Carol Triangle closing 20 20

People You May Know How do people Aliceknow each other? Bob Carol Triangle closingProb(Bob knows Carol) ~ the # of common connections 21 21

Triangle Closing in Pig-- connections in (source_id, dest_id) format in both directionsconnections = LOAD `connections` USING PigStorage();group_conn = GROUP connections BY source_id;pairs = FOREACH group_conn GENERATE generatePair(connections.dest_id) as (id1, id2);common_conn = GROUP pairs BY (id1, id2);common_conn = FOREACH common_conn GENERATE ﬂatten(group) as (source_id, dest_id), COUNT(pairs) as common_connections;STORE common_conn INTO `common_conn` USING PigStorage(); 22 22

Pig OverviewLoad: load data, specify formatStore: store data, specify formatForeach, Generate: Projections, similar to selectGroup by: group by column(s)Join, Filter, Limit, Order, ...User Deﬁned Functions (UDFs) 23 23

Triangle Closing Example Alice Bob Carol connections = LOAD `connections` USING1.(A,B),(B,A),(A,C),(C,A) PigStorage();2.(A,{B,C}),(B,{A}),(C,{A})3.(A,{B,C}),(A,{C,B})4.(B,C,1), (C,B,1) 29 29

Triangle Closing Example Alice Bob Carol1.(A,B),(B,A),(A,C),(C,A) group_conn = GROUP connections BY2.(A,{B,C}),(B,{A}),(C,{A}) source_id;3.(A,{B,C}),(A,{C,B})4.(B,C,1), (C,B,1) 30 30

Triangle Closing Example Alice Bob Carol1.(A,B),(B,A),(A,C),(C,A)2.(A,{B,C}),(B,{A}),(C,{A}) pairs = FOREACH group_conn GENERATE3.(A,{B,C}),(A,{C,B}) generatePair(connections.dest_id) as (id1, id2);4.(B,C,1), (C,B,1) 31 31

Triangle Closing Example Alice Bob Carol1.(A,B),(B,A),(A,C),(C,A)2.(A,{B,C}),(B,{A}),(C,{A}) common_conn = GROUP pairs BY (id1, id2); common_conn = FOREACH common_conn3.(A,{B,C}),(A,{C,B}) GENERATE ﬂatten(group) as (source_id, dest_id),4.(B,C,1), (C,B,1) COUNT(pairs) as common_connections; 32 32

Our Workﬂow triangle-closing 33 33

Our Workﬂow triangle-closing top-n 34 34

Our Workﬂow triangle-closing top-n push-to-prod 35 35

Our Workﬂow triangle-closing top-n push-to-prod 37 37

Our Workﬂow triangle-closing remove connections top-n push-to-prod 38 38

Our Workﬂow triangle-closing remove connections top-npush-to-qa push-to-prod 39 39

PYMK Workﬂow 40 40

Workﬂow RequirementsDependency managementRegular SchedulingMonitoringDiverse jobs: Java, Pig, ClojureConﬁguration/ParametersResource control/lockingRestart/Stop/RetryVisualizationHistoryLogs 41 41

Workﬂow RequirementsDependency managementRegular SchedulingMonitoringDiverse jobs: Java, Pig, ClojureConﬁguration/ParametersResource control/lockingRestart/Stop/RetryVisualizationHistory AzkabanLogs 42 42

Sample Azkaban Job Spectype=pigpig.script=top-n.pigdependencies=remove-connectionstop.n.size=100 43 43

Azkaban Workﬂow 44 44

Production StorageRequirements Large amount of data/Scalable Quick lookup/low latency Versioning and Rollback Fault tolerance Ofﬂine index building 50 50

Voldemort StorageLarge amount of data/ScalableQuick lookup/low latencyVersioning and RollbackFault tolerance through replicationRead onlyOfﬂine index building 51 51

Data Cycle 52 52

Voldemort RO Store 53 53

Data QualityVeriﬁcationQA store with viewerExplainVersioning/RollbackUnit tests 56 56

Performance 58 58

PerformanceSymmetry Bob knows Carol then Carol knows Bob 58 58

PerformanceSymmetry Bob knows Carol then Carol knows BobLimit Ignore members with > k connections 58 58

PerformanceSymmetry Bob knows Carol then Carol knows BobLimit Ignore members with > k connectionsSampling Sample k-connections 58 58

Things CoveredWhat do I mean by Data Products?Systems and Tools we useLet’s build “People You May Know”Managing workﬂowServing data in productionData QualityPerformance 59 59

SNA TeamThanks to SNA Team at LinkedInhttp://sna-projects.comWe are hiring! 60 60

Questions? 61 61

Building Data Driven Products at Linkedin

by mitultiwari

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 12

Statistics

Building Data Driven Products at Linkedin Presentation Transcript