SlideShare a Scribd company logo
Recruiting SolutionsRecruiting SolutionsRecruiting Solutions
Machine Learning for Search @
Viet Ha-Thuc
Search Quality - LinkedIn
1
2
• 200+ countries and
territories
• 2+ new members per
second
3
● Dual Roles of Search
○ Enable talent discover opportunity
○ Help companies to search for the right talent
4
FLAGSHIP SEARCH
RECRUITER SEARCH
SALES NAVIGATOR
Unique Nature of LinkedIn Search
▪ Heterogeneous sources
People, jobs, companies,
slideshares, members’ posts,
groups
▪ Scale
▪ Deep Personalization
▪ Support many use-cases
Hiring, connecting, job seeking,
research, sales, etc.
5
Overview
6
Query
Federated Search
Spell Correction
Query Tagging
People Companies
Federated Search
Blending
Name Title Skill
Jobs
Overview
7
Query
Federated Search
Spell Correction
Query Tagging
People Companies
Federated Search
Blending
Name Title Skill
Jobs
Agenda
▪ Introduction
▪ Vertical Ranking
–People Search by Skills [BigData’15,SIGIR’16]
–Job Search [KDD’16]
▪ Federation [CIKM’15]
▪ Lessons
8
Introduction
▪ Skills
– 40K+ standardized skills
– Members get endorsed on
skills
– Represent professional
expertise
9
Introduction
▪ Unique challenges to LinkedIn expertise Search
– Scale: 400M members x 40K standardized skills
– Sparsity of skills in profiles
– Personalization
10
…
Reputation
Information a decision maker uses to make a
judgment on an entity with a record (*)
11
(*) “Building web reputation systems”, Glass and Farmer, 2010
Skill Reputation Scores [BigData’15]
12
▪ Decision Maker: searcher
▪ Record: Professional
career
▪ Skill reputation: member
expertise on a skill
▪ Judgment: Hire?
Estimating Skill Reputation
13
Endorse
profile
browsemap
? .85 .45
? ? .35
? .42 ?
? ? .05
Members
Skills
P(expert| member, skill)
Supervised
Learning
algorithm
Estimating Skill Reputation
14
Endorse
profile
browsemap
? .85 .45
? ? .35
? .42 ?
? ? .05
Members
Skills
0.5 1
0.7 0
0 0.6
0.1 0
0.2 0.3 0.5
0.5 0.7 0.2
Members
Skills
Each row is a representation of a
member in latent space
Each column
represents a skill in
latent space
Matrix Factorization
Estimating Skill Reputation
15
Endorse
profile
browsemap
? .85 .45
? ? .35
? .42 ?
.02 ? ?
Members
Skills
0.5 1
0.7 0
0 0.6
0.1 0
0.2 0.3 0.5
0.5 0.7 0.2
Members
Skills
.6 .85 .45
.14 .21 .35
.3 .42 .12
.02 .03 .05
Members
Skills
Fill in unknown cells in
the original matrix
Features
▪ Reputation feature
▪ Social Connection
▪ Homophily
– Geo
– Industry
▪ Textual Features
16
Learning to Rank
▪ Listwise
– Consider relevance is relative to every query
– Allow optimizing quality metric directly
▪ Objective function
– Normalized Discounted Cumulative Gain (NDCG@K)
– Graded relevance labels
17
Labeling Strategy
18
▪ Logs + Top-K randomization
Uncertain (removed)
Bad: label = 0
Good: label = 1click
InMail Perfect: label = 3
Experiments
CTR@10 # Messages
per Search
Flagship +11% +20%
Premium +18% +37%
19
▪ Query Tagging
▪ Target Segment: skill and no-name
▪ Baseline
– No skill reputation feature
– Hand-tuned
Agenda
▪ Introduction
▪ Vertical Ranking
–People Search by Skills [BigData’15, SIGIR’16]
–Job Search [KDD’16]
▪ Federation [CIKM’15]
▪ Lessons
20
Challenges of Job Search
▪ “Hidden” structures
▪ Query only represents a small fraction of information need
–“San Francisco”, “software engineer”, “java”“Hidden” structures
▪ Job attractiveness varies on many aspects
–“Hot” titles: “data scientist”
–Top companies: Google, Facebook, etc.
–Trending skills: machine learning, big data, etc.,
–Location
21
Entity-Aware Matching
22
Expertise Homophily
▪ “Classic” homophily in social networks
–People tend to interact with similar ones
▪ Expertise homophily in job search
–Searcher tends to apply for jobs with similar expertise
–Apply rate of job results with overlapping skills is 2x higher
▪ Expertise: skill reputation scores
23
Entity-faceted CTRs
▪ Job attractiveness
– Historical CTRs for individual jobs
– Challenge: job lifetime is short -> unreliable estimation
▪ Entity-faceted historical CTRs
– CTRs of jobs with standardized tile “data scientist”
– CTRs of jobs from company IBM
– CTRs of jobs requiring trending skill: machine learning, big data, etc.
▪ Advantages
– Alleviate data sparseness by grouping jobs by facets
– Resolve cold start problem
24
Experiment Results
▪ Baseline
▪ All of the existing features except entity-aware ones
▪ Machine learned
▪ Optimized for the same objective function
25
CTR Apply Rate
Improvement +11.3% +5.3%
Agenda
▪ Introduction
▪ Vertical Ranking
–People Search by Skills [BigData’15, SIGIR’16]
–Job Search [KDD’16]
▪ Federation [CIKM’15]
▪ Lessons
26
Personalized Blending
Personalized Blending
▪ Why do we need this?
– Not to overwhelm the user with too much information
– Make results personally relevant
Blending Flow
Learning Model
▪ Training data: click logs
▪ Features
– Relevance scores from base rankers
– Searcher intent
– Query intent
– Prior scores
Calibrate Scores across Verticals
▪ Relevance scores from vertical rankers are incomparable
Calibrate Scores across Verticals
▪ Relevance scores from vertical rankers are incomparable
▪ Construct composite features
People relevance score of searcher if result is People
f 1= ⎨0, otherwise
Searcher Intent
Searcher’s job seeking intent if result is job vertical cluster
Searcher’s job seeking intent if result is individual job
Searcher’s recruiting intent if result is people vertical cluster
Searcher’s recruiting intent if result is individual people
...
Take-Aways
▪ Text match is still important but not enough
▪ Advanced features based on semi-structured
data
– People search: skill reputation scores
– Job Search: expertise homophily
▪ Personalized Learning-to-Rank is crucial
34
35
Email: vhathuc@linkedin.com
References
▪“Personalized Expertise Search at LinkedIn”, Ha-Thuc,
Venkataraman, Rodriguez, Sinha, Sundaram and Guo,
BigData, 2015
▪“Personalized Federated Search at LinkedIn”, Arya, Ha-
Thuc and Sinha, CIKM, 2015
▪“Learning to Rank Personalized Search Results in
Professional Networks”, Ha-Thuc and Sinha, SIGIR, 2016
▪“How to Get Them a Dream Job?”, Li, Arya, Ha-Thuc,
Sinha, KDD, 2016
36

More Related Content

Viewers also liked

Concept search for e commerce with solr
Concept search for e commerce with solrConcept search for e commerce with solr
Concept search for e commerce with solr
lucenerevolution
 
Part 2 (machine learning overview) all machine learning is pattern search
Part 2 (machine learning overview)   all machine learning is pattern searchPart 2 (machine learning overview)   all machine learning is pattern search
Part 2 (machine learning overview) all machine learning is pattern search
International School of Engineering
 
Applied machine learning for search engine relevance 3
Applied machine learning for search engine relevance 3Applied machine learning for search engine relevance 3
Applied machine learning for search engine relevance 3
Charles Martin
 
Learning To Rank User Queries to Detect Search Tasks
Learning To Rank User Queries to Detect Search TasksLearning To Rank User Queries to Detect Search Tasks
Learning To Rank User Queries to Detect Search Tasks
Franco Maria Nardini
 
Learning to Rank: An Introduction to LambdaMART
Learning to Rank: An Introduction to LambdaMARTLearning to Rank: An Introduction to LambdaMART
Learning to Rank: An Introduction to LambdaMART
Julian Qian
 
Learning to rank fulltext results from clicks
Learning to rank fulltext results from clicksLearning to rank fulltext results from clicks
Learning to rank fulltext results from clicks
tkramar
 
eCommerce for Everyone: What to Expect in 2017 - State of Search
eCommerce for Everyone: What to Expect in 2017 - State of SearcheCommerce for Everyone: What to Expect in 2017 - State of Search
eCommerce for Everyone: What to Expect in 2017 - State of Search
Elizabeth Marsten
 
Machine Learning Search and SEO - Zenith; Duluth, MN.
Machine Learning Search and SEO - Zenith; Duluth, MN. Machine Learning Search and SEO - Zenith; Duluth, MN.
Machine Learning Search and SEO - Zenith; Duluth, MN.
Eric Enge
 
Penguin 4.0 - State of Search 2016
Penguin 4.0 - State of Search 2016 Penguin 4.0 - State of Search 2016
Penguin 4.0 - State of Search 2016
Eric Enge
 
Machine Learning and Search -State of Search 2016
Machine Learning and Search -State of Search 2016 Machine Learning and Search -State of Search 2016
Machine Learning and Search -State of Search 2016
Eric Enge
 
Architecture of a search engine
Architecture of a search engineArchitecture of a search engine
Architecture of a search engine
Sylvain Utard
 
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
recsysfr
 
Machine Learning Real Life Applications By Examples
Machine Learning Real Life Applications By ExamplesMachine Learning Real Life Applications By Examples
Machine Learning Real Life Applications By Examples
Mario Cartia
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised Learning
Lukas Tencer
 
AWS Elastic Beanstalk - Running Microservices and Docker
AWS Elastic Beanstalk - Running Microservices and DockerAWS Elastic Beanstalk - Running Microservices and Docker
AWS Elastic Beanstalk - Running Microservices and Docker
Amazon Web Services
 
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough dataВладимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
Mail.ru Group
 
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
Lucidworks
 
Markov model for the online multichannel attribution problem
Markov model for the online multichannel attribution problemMarkov model for the online multichannel attribution problem
Markov model for the online multichannel attribution problem
adavide1982
 
Apriori Algorithm
Apriori AlgorithmApriori Algorithm
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems -  ACM RecSys 2013 tutorialLearning to Rank for Recommender Systems -  ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
Alexandros Karatzoglou
 

Viewers also liked (20)

Concept search for e commerce with solr
Concept search for e commerce with solrConcept search for e commerce with solr
Concept search for e commerce with solr
 
Part 2 (machine learning overview) all machine learning is pattern search
Part 2 (machine learning overview)   all machine learning is pattern searchPart 2 (machine learning overview)   all machine learning is pattern search
Part 2 (machine learning overview) all machine learning is pattern search
 
Applied machine learning for search engine relevance 3
Applied machine learning for search engine relevance 3Applied machine learning for search engine relevance 3
Applied machine learning for search engine relevance 3
 
Learning To Rank User Queries to Detect Search Tasks
Learning To Rank User Queries to Detect Search TasksLearning To Rank User Queries to Detect Search Tasks
Learning To Rank User Queries to Detect Search Tasks
 
Learning to Rank: An Introduction to LambdaMART
Learning to Rank: An Introduction to LambdaMARTLearning to Rank: An Introduction to LambdaMART
Learning to Rank: An Introduction to LambdaMART
 
Learning to rank fulltext results from clicks
Learning to rank fulltext results from clicksLearning to rank fulltext results from clicks
Learning to rank fulltext results from clicks
 
eCommerce for Everyone: What to Expect in 2017 - State of Search
eCommerce for Everyone: What to Expect in 2017 - State of SearcheCommerce for Everyone: What to Expect in 2017 - State of Search
eCommerce for Everyone: What to Expect in 2017 - State of Search
 
Machine Learning Search and SEO - Zenith; Duluth, MN.
Machine Learning Search and SEO - Zenith; Duluth, MN. Machine Learning Search and SEO - Zenith; Duluth, MN.
Machine Learning Search and SEO - Zenith; Duluth, MN.
 
Penguin 4.0 - State of Search 2016
Penguin 4.0 - State of Search 2016 Penguin 4.0 - State of Search 2016
Penguin 4.0 - State of Search 2016
 
Machine Learning and Search -State of Search 2016
Machine Learning and Search -State of Search 2016 Machine Learning and Search -State of Search 2016
Machine Learning and Search -State of Search 2016
 
Architecture of a search engine
Architecture of a search engineArchitecture of a search engine
Architecture of a search engine
 
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
CONTENT2VEC: a Joint Architecture to use Product Image and Text for the task ...
 
Machine Learning Real Life Applications By Examples
Machine Learning Real Life Applications By ExamplesMachine Learning Real Life Applications By Examples
Machine Learning Real Life Applications By Examples
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised Learning
 
AWS Elastic Beanstalk - Running Microservices and Docker
AWS Elastic Beanstalk - Running Microservices and DockerAWS Elastic Beanstalk - Running Microservices and Docker
AWS Elastic Beanstalk - Running Microservices and Docker
 
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough dataВладимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
Владимир Гулин, Mail.Ru Group, Learning to rank using clickthrough data
 
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
Implementing Conceptual Search in Solr using LSA and Word2Vec: Presented by S...
 
Markov model for the online multichannel attribution problem
Markov model for the online multichannel attribution problemMarkov model for the online multichannel attribution problem
Markov model for the online multichannel attribution problem
 
Apriori Algorithm
Apriori AlgorithmApriori Algorithm
Apriori Algorithm
 
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems -  ACM RecSys 2013 tutorialLearning to Rank for Recommender Systems -  ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
 

Similar to Machine Learning for Search at LinkedIn

From complexity to clarity in one week with Enterprise Design Sprints
From complexity to clarity in one week with Enterprise Design SprintsFrom complexity to clarity in one week with Enterprise Design Sprints
From complexity to clarity in one week with Enterprise Design Sprints
Lisa Schlecht
 
Find and be Found: Information Retrieval at LinkedIn
Find and be Found: Information Retrieval at LinkedInFind and be Found: Information Retrieval at LinkedIn
Find and be Found: Information Retrieval at LinkedIn
Daniel Tunkelang
 
Job Search Strategies
Job Search StrategiesJob Search Strategies
Job Search Strategies
Katie Mantooth
 
Large scale social recommender systems and their evaluation
Large scale social recommender systems and their evaluationLarge scale social recommender systems and their evaluation
Large scale social recommender systems and their evaluation
Mitul Tiwari
 
Toronto | ConnectIn 2013
Toronto | ConnectIn 2013Toronto | ConnectIn 2013
Toronto | ConnectIn 2013
LinkedIn Talent Solutions
 
How to Build Your Personal Brand on LinkedIn
How to Build Your Personal Brand on LinkedIn How to Build Your Personal Brand on LinkedIn
How to Build Your Personal Brand on LinkedIn
Doreen Olson
 
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
HireClix
 
Recruiting Passive Candidates HireClix - Social Recruiting Seminar - Target...
Recruiting Passive Candidates   HireClix - Social Recruiting Seminar - Target...Recruiting Passive Candidates   HireClix - Social Recruiting Seminar - Target...
Recruiting Passive Candidates HireClix - Social Recruiting Seminar - Target...
HireClix
 
Seo training syllabus
Seo training syllabusSeo training syllabus
Seo training syllabus
nals0407
 
LinkedIn Optimization Branding and SEO
LinkedIn Optimization Branding and SEOLinkedIn Optimization Branding and SEO
LinkedIn Optimization Branding and SEO
Boulder SEO Marketing
 
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
SmashFly Technologies
 
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Qi Guo
 
RallyFwd Dec 2022 - Lori Sylvia.pdf
RallyFwd Dec 2022 - Lori Sylvia.pdfRallyFwd Dec 2022 - Lori Sylvia.pdf
RallyFwd Dec 2022 - Lori Sylvia.pdf
Rally Recruitment Marketing
 
Ai next con_seattle_18
Ai next con_seattle_18Ai next con_seattle_18
Ai next con_seattle_18
Liang Zhang
 
Recruitment March 2014
Recruitment March 2014Recruitment March 2014
Recruitment March 2014
Timothy Holden
 
Contact Tree Team3
Contact Tree Team3Contact Tree Team3
Contact Tree Team3
Stanford University
 
LeanLaunchpad_finaldemo_ContactTree
LeanLaunchpad_finaldemo_ContactTreeLeanLaunchpad_finaldemo_ContactTree
LeanLaunchpad_finaldemo_ContactTree
Ana
 
Social2Business - LinkedIn Masterclass
Social2Business - LinkedIn MasterclassSocial2Business - LinkedIn Masterclass
Social2Business - LinkedIn Masterclass
Roger Christie
 
Webinar: Maximizing Your Potential with Data Leadership
Webinar: Maximizing Your Potential with Data LeadershipWebinar: Maximizing Your Potential with Data Leadership
Webinar: Maximizing Your Potential with Data Leadership
DATAVERSITY
 
Developer career
Developer careerDeveloper career
Developer career
Michał Gruca
 

Similar to Machine Learning for Search at LinkedIn (20)

From complexity to clarity in one week with Enterprise Design Sprints
From complexity to clarity in one week with Enterprise Design SprintsFrom complexity to clarity in one week with Enterprise Design Sprints
From complexity to clarity in one week with Enterprise Design Sprints
 
Find and be Found: Information Retrieval at LinkedIn
Find and be Found: Information Retrieval at LinkedInFind and be Found: Information Retrieval at LinkedIn
Find and be Found: Information Retrieval at LinkedIn
 
Job Search Strategies
Job Search StrategiesJob Search Strategies
Job Search Strategies
 
Large scale social recommender systems and their evaluation
Large scale social recommender systems and their evaluationLarge scale social recommender systems and their evaluation
Large scale social recommender systems and their evaluation
 
Toronto | ConnectIn 2013
Toronto | ConnectIn 2013Toronto | ConnectIn 2013
Toronto | ConnectIn 2013
 
How to Build Your Personal Brand on LinkedIn
How to Build Your Personal Brand on LinkedIn How to Build Your Personal Brand on LinkedIn
How to Build Your Personal Brand on LinkedIn
 
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
Recruitment Marketing - HireClix - Recruitment Advertising - Anatomy of ppc r...
 
Recruiting Passive Candidates HireClix - Social Recruiting Seminar - Target...
Recruiting Passive Candidates   HireClix - Social Recruiting Seminar - Target...Recruiting Passive Candidates   HireClix - Social Recruiting Seminar - Target...
Recruiting Passive Candidates HireClix - Social Recruiting Seminar - Target...
 
Seo training syllabus
Seo training syllabusSeo training syllabus
Seo training syllabus
 
LinkedIn Optimization Branding and SEO
LinkedIn Optimization Branding and SEOLinkedIn Optimization Branding and SEO
LinkedIn Optimization Branding and SEO
 
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
How CDW connected the dots with a Recruitment Marketing Platform by @TroyHein...
 
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
Talent Search and Recommendation Systems at LinkedIn: Practical Challenges an...
 
RallyFwd Dec 2022 - Lori Sylvia.pdf
RallyFwd Dec 2022 - Lori Sylvia.pdfRallyFwd Dec 2022 - Lori Sylvia.pdf
RallyFwd Dec 2022 - Lori Sylvia.pdf
 
Ai next con_seattle_18
Ai next con_seattle_18Ai next con_seattle_18
Ai next con_seattle_18
 
Recruitment March 2014
Recruitment March 2014Recruitment March 2014
Recruitment March 2014
 
Contact Tree Team3
Contact Tree Team3Contact Tree Team3
Contact Tree Team3
 
LeanLaunchpad_finaldemo_ContactTree
LeanLaunchpad_finaldemo_ContactTreeLeanLaunchpad_finaldemo_ContactTree
LeanLaunchpad_finaldemo_ContactTree
 
Social2Business - LinkedIn Masterclass
Social2Business - LinkedIn MasterclassSocial2Business - LinkedIn Masterclass
Social2Business - LinkedIn Masterclass
 
Webinar: Maximizing Your Potential with Data Leadership
Webinar: Maximizing Your Potential with Data LeadershipWebinar: Maximizing Your Potential with Data Leadership
Webinar: Maximizing Your Potential with Data Leadership
 
Developer career
Developer careerDeveloper career
Developer career
 

Recently uploaded

Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
Paul Walk
 
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
3a0sd7z3
 
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
k4ncd0z
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
thezot
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
APNIC
 
Bengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal BrandingBengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal Branding
Tarandeep Singh
 
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
xjq03c34
 
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
rtunex8r
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
davidjhones387
 
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
APNIC
 
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
3a0sd7z3
 
HijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process HollowingHijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process Hollowing
Donato Onofri
 

Recently uploaded (12)

Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?Should Repositories Participate in the Fediverse?
Should Repositories Participate in the Fediverse?
 
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
快速办理(Vic毕业证书)惠灵顿维多利亚大学毕业证完成信一模一样
 
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理一比一原版(USYD毕业证)悉尼大学毕业证如何办理
一比一原版(USYD毕业证)悉尼大学毕业证如何办理
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
 
Bengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal BrandingBengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal Branding
 
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
办理新西兰奥克兰大学毕业证学位证书范本原版一模一样
 
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
怎么办理(umiami毕业证书)美国迈阿密大学毕业证文凭证书实拍图原版一模一样
 
Discover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to IndiaDiscover the benefits of outsourcing SEO to India
Discover the benefits of outsourcing SEO to India
 
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
 
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
快速办理(新加坡SMU毕业证书)新加坡管理大学毕业证文凭证书一模一样
 
HijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process HollowingHijackLoader Evolution: Interactive Process Hollowing
HijackLoader Evolution: Interactive Process Hollowing
 

Machine Learning for Search at LinkedIn

  • 1. Recruiting SolutionsRecruiting SolutionsRecruiting Solutions Machine Learning for Search @ Viet Ha-Thuc Search Quality - LinkedIn 1
  • 2. 2 • 200+ countries and territories • 2+ new members per second
  • 3. 3 ● Dual Roles of Search ○ Enable talent discover opportunity ○ Help companies to search for the right talent
  • 5. Unique Nature of LinkedIn Search ▪ Heterogeneous sources People, jobs, companies, slideshares, members’ posts, groups ▪ Scale ▪ Deep Personalization ▪ Support many use-cases Hiring, connecting, job seeking, research, sales, etc. 5
  • 6. Overview 6 Query Federated Search Spell Correction Query Tagging People Companies Federated Search Blending Name Title Skill Jobs
  • 7. Overview 7 Query Federated Search Spell Correction Query Tagging People Companies Federated Search Blending Name Title Skill Jobs
  • 8. Agenda ▪ Introduction ▪ Vertical Ranking –People Search by Skills [BigData’15,SIGIR’16] –Job Search [KDD’16] ▪ Federation [CIKM’15] ▪ Lessons 8
  • 9. Introduction ▪ Skills – 40K+ standardized skills – Members get endorsed on skills – Represent professional expertise 9
  • 10. Introduction ▪ Unique challenges to LinkedIn expertise Search – Scale: 400M members x 40K standardized skills – Sparsity of skills in profiles – Personalization 10 …
  • 11. Reputation Information a decision maker uses to make a judgment on an entity with a record (*) 11 (*) “Building web reputation systems”, Glass and Farmer, 2010
  • 12. Skill Reputation Scores [BigData’15] 12 ▪ Decision Maker: searcher ▪ Record: Professional career ▪ Skill reputation: member expertise on a skill ▪ Judgment: Hire?
  • 13. Estimating Skill Reputation 13 Endorse profile browsemap ? .85 .45 ? ? .35 ? .42 ? ? ? .05 Members Skills P(expert| member, skill) Supervised Learning algorithm
  • 14. Estimating Skill Reputation 14 Endorse profile browsemap ? .85 .45 ? ? .35 ? .42 ? ? ? .05 Members Skills 0.5 1 0.7 0 0 0.6 0.1 0 0.2 0.3 0.5 0.5 0.7 0.2 Members Skills Each row is a representation of a member in latent space Each column represents a skill in latent space Matrix Factorization
  • 15. Estimating Skill Reputation 15 Endorse profile browsemap ? .85 .45 ? ? .35 ? .42 ? .02 ? ? Members Skills 0.5 1 0.7 0 0 0.6 0.1 0 0.2 0.3 0.5 0.5 0.7 0.2 Members Skills .6 .85 .45 .14 .21 .35 .3 .42 .12 .02 .03 .05 Members Skills Fill in unknown cells in the original matrix
  • 16. Features ▪ Reputation feature ▪ Social Connection ▪ Homophily – Geo – Industry ▪ Textual Features 16
  • 17. Learning to Rank ▪ Listwise – Consider relevance is relative to every query – Allow optimizing quality metric directly ▪ Objective function – Normalized Discounted Cumulative Gain (NDCG@K) – Graded relevance labels 17
  • 18. Labeling Strategy 18 ▪ Logs + Top-K randomization Uncertain (removed) Bad: label = 0 Good: label = 1click InMail Perfect: label = 3
  • 19. Experiments CTR@10 # Messages per Search Flagship +11% +20% Premium +18% +37% 19 ▪ Query Tagging ▪ Target Segment: skill and no-name ▪ Baseline – No skill reputation feature – Hand-tuned
  • 20. Agenda ▪ Introduction ▪ Vertical Ranking –People Search by Skills [BigData’15, SIGIR’16] –Job Search [KDD’16] ▪ Federation [CIKM’15] ▪ Lessons 20
  • 21. Challenges of Job Search ▪ “Hidden” structures ▪ Query only represents a small fraction of information need –“San Francisco”, “software engineer”, “java”“Hidden” structures ▪ Job attractiveness varies on many aspects –“Hot” titles: “data scientist” –Top companies: Google, Facebook, etc. –Trending skills: machine learning, big data, etc., –Location 21
  • 23. Expertise Homophily ▪ “Classic” homophily in social networks –People tend to interact with similar ones ▪ Expertise homophily in job search –Searcher tends to apply for jobs with similar expertise –Apply rate of job results with overlapping skills is 2x higher ▪ Expertise: skill reputation scores 23
  • 24. Entity-faceted CTRs ▪ Job attractiveness – Historical CTRs for individual jobs – Challenge: job lifetime is short -> unreliable estimation ▪ Entity-faceted historical CTRs – CTRs of jobs with standardized tile “data scientist” – CTRs of jobs from company IBM – CTRs of jobs requiring trending skill: machine learning, big data, etc. ▪ Advantages – Alleviate data sparseness by grouping jobs by facets – Resolve cold start problem 24
  • 25. Experiment Results ▪ Baseline ▪ All of the existing features except entity-aware ones ▪ Machine learned ▪ Optimized for the same objective function 25 CTR Apply Rate Improvement +11.3% +5.3%
  • 26. Agenda ▪ Introduction ▪ Vertical Ranking –People Search by Skills [BigData’15, SIGIR’16] –Job Search [KDD’16] ▪ Federation [CIKM’15] ▪ Lessons 26
  • 28. Personalized Blending ▪ Why do we need this? – Not to overwhelm the user with too much information – Make results personally relevant
  • 30. Learning Model ▪ Training data: click logs ▪ Features – Relevance scores from base rankers – Searcher intent – Query intent – Prior scores
  • 31. Calibrate Scores across Verticals ▪ Relevance scores from vertical rankers are incomparable
  • 32. Calibrate Scores across Verticals ▪ Relevance scores from vertical rankers are incomparable ▪ Construct composite features People relevance score of searcher if result is People f 1= ⎨0, otherwise
  • 33. Searcher Intent Searcher’s job seeking intent if result is job vertical cluster Searcher’s job seeking intent if result is individual job Searcher’s recruiting intent if result is people vertical cluster Searcher’s recruiting intent if result is individual people ...
  • 34. Take-Aways ▪ Text match is still important but not enough ▪ Advanced features based on semi-structured data – People search: skill reputation scores – Job Search: expertise homophily ▪ Personalized Learning-to-Rank is crucial 34
  • 36. References ▪“Personalized Expertise Search at LinkedIn”, Ha-Thuc, Venkataraman, Rodriguez, Sinha, Sundaram and Guo, BigData, 2015 ▪“Personalized Federated Search at LinkedIn”, Arya, Ha- Thuc and Sinha, CIKM, 2015 ▪“Learning to Rank Personalized Search Results in Professional Networks”, Ha-Thuc and Sinha, SIGIR, 2016 ▪“How to Get Them a Dream Job?”, Li, Arya, Ha-Thuc, Sinha, KDD, 2016 36