PPTX, PDF2,618 views

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

The document provides an overview of the machine learning pipeline, emphasizing data gathering, feature engineering, and model evaluation. Key points include the importance of context in tasks, the challenges of binary tasks, and the significance of quality data. It concludes with best practices for working with data and tips for improving classification models.

Technology◦News & Politics◦

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

1.
Hacking Data Science
19.
Using results toevaluate new features ©2013 LinkedIn Corporation. All Rights Reserved. 19 Model ΔP ΔR ΔPRC Baseline - - - Variation 1 + - + Variation 2 - + + Variation 3 - ++ - - Variation 4 - +++ ++ Variation 5 - +++ ++ Variation 6 - +++ ++ Variation 7 - ++++ +++ Variation 8 - ++++ +++ Variation 9 - ++++ +++ Variation 10 - ++++ +++
24.
Evaluate suggested article-topicpairs  Using results to evaluate new implementations of spam classifier – Improve Prec without drop in Rec  18k comments labeled in 54 hrs for $180 ©2013 LinkedIn Corporation. All Rights Reserved. 24
42.
What makes thebest data mining expert?  Education?  Industry experience?  Amount of publications?  Communication skills?  Hacking skills?  Knowledge of statistics?  Number of endorsements? ©2013 LinkedIn Corporation. All Rights Reserved. 42
44.
Summary ©2013 LinkedIn Corporation.All Rights Reserved. 44 1. Use the data you already have 2. Keep it simple, but not too simple 3. Pick a solvable problem 4. Help your helpers 5. Sample intelligently 6. More (bad) data != better data

Editor's Notes

#4 Supervised (gold, agreement) & unsupervised (behavioral)
#5 Supervised (gold, agreement) & unsupervised (behavioral)
#6 Supervised (gold, agreement) & unsupervised (behavioral)
#7 Supervised (gold, agreement) & unsupervised (behavioral)
#8 Supervised (gold, agreement) & unsupervised (behavioral)
#9 Supervised (gold, agreement) & unsupervised (behavioral)
#10 Supervised (gold, agreement) & unsupervised (behavioral)
#11 Supervised (gold, agreement) & unsupervised (behavioral)
#12 Supervised (gold, agreement) & unsupervised (behavioral)
#13 Supervised (gold, agreement) & unsupervised (behavioral)
#15 Context: why it mattersOff-topic comments lower the perceived value of Influencer content, LI network, etc.Legit members may leave low-quality topics -> no hell-banning
#16 Especially if you only guess on the hard ones+ Gold and wawa don’t work as well with binary tasks
#17 + references to article, other comments, etc.
#18 Sampling: took clusters where at least one item scored poorly with existing classifierStill a biased dataset -> skew gold to catch positive cases (80% of Golds have at least one comment flagged)Treat any comment that got at least 1 vote as “suspect”NEXT TIME: set minimum agreement thresholds and collect more labels dynamically
#19 Sampling: took clusters where at least one item scored poorly with existing classifierStill a biased dataset -> skew gold to catch positive cases (80% of Golds have at least one comment flagged)Treat any comment that got at least 1 vote as “suspect”NEXT TIME: set minimum agreement thresholds and collect more labels dynamically
#20 + Using results to evaluate new implementations of spam classifierImprove Prec without drop in Rec+ 18k comments labeled in 54 hrs for $180
#21 + simple as possible, but not any simpler
#22 need to find timely, relevant content for many subjects
#23 Free-text tagging = standardization pain, plus hard to manage quality+ double-pass -> annoyingStandardized taxonomy: 1,200 topics selected as representative linkedin members interests + random guessing: 1200 topics is still a lot
#24 Pick “likely” labels for evaluation:+ weak classifier to identify skills in an article -> expand to related skills+ weak classifier to identify industry of article -> expand to related skills+ pick labels based on source of article (e.g., forbes -> economy, marketing, etc.)+ 100 candidate labels for each article
#25 + 400k article-topic pairs+ e.g., 60k pairs in ~1 week @ 7c each+ 4 labels for each item, take the average value (rather than looking for consensus)+ bootstrap additional gold from items completed with high agreementLessons+ difference between very & somewhat relevant: “is this the primary topic”+ some non-english articles, some garbled articles
#26 + 400k article-topic pairs+ e.g., 60k pairs in ~1 week @ 7c each+ 4 labels for each item, take the average value (rather than looking for consensus)+ bootstrap additional gold from items completed with high agreementLessons+ difference between very & somewhat relevant: “is this the primary topic”+ some non-english articles, some garbled articles
#27 Working towards a “less” supervised way to create new channels
#28 Preprocessing the data to select likely matches greatly reduced the number of labels needed
#29 search: + helps members find and be found+ People, Jobs, Groups and more
#30 LI search is personalized: + tuple of (user, query, document)Too much to ask a random person to label for training+ “imagine that you’re X and see Y” has its limits+ train from logs
#31 Indirect measures: + CTR@1, CTR@P1, Session Abandonment, etc.Explicit measures:+ what about non-personalized search (such as for recruiters)?+ what about identifying items that are off-topic for all members?
#32 1000 query-result pairs+ retrieve all queries where result@1 didn’t get a click+ remove any queries tagged as {firstname, lastname} where the name in the query matched the name in the profile (we know these perform well}Binary tasks bad – added a second set of questions+ allows us to audit query tagger at the same timeUsing results to triage queries for additional manual review+ also adds an explicit relevance metric to track over time (wtf@1)
#33 Other behavioral stuff:+ individual judgment duration, scrolls, clicks, mouse movement+ jQuery is your friend
#34 Picking the right problem gets you a long way thereSkillRank example----- Meeting Notes (8/15/13 16:55) -----+ name queries really aren't that useful so we excluded those+ ran it internally first, then with turkers++ nearly identical, arguably it was better
#35 Supervised (gold, agreement) & unsupervised (behavioral)
#36 Supervised (gold, agreement) & unsupervised (behavioral)
#37 Supervised (gold, agreement) & unsupervised (behavioral)
#43 Supervised (gold, agreement) & unsupervised (behavioral)
#44 Picking the right problem gets you a long way thereSkillRank example----- Meeting Notes (8/15/13 16:55) -----+ name queries really aren't that useful so we excluded those+ ran it internally first, then with turkers++ nearly identical, arguably it was better
#45 Other fun lessons5. Not by gold alone

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

More Related Content

Viewers also liked

Similar to Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

More from Hakka Labs

Recently uploaded

Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.

Editor's Notes