Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
What's wrong with you,
Recruiter-John? A non-
trivial recommender
challenge.
Budapest, June 2016
@fabianabel
http://recsys...
Challenge
Given a user, the goal is to recommend job postings…
1. that the user may be interested in and
2. for which the ...
Job recommendations
Job recommendations
5
Title
Company
Employment type
and career level
Full-text
description
Key properties of a job posting
6
Key sources for understanding user demands
Social Network
explicit and
implicit
connections
Profile
Fabian Abel
Data Sci...
7
Relevance Estimation
Social Network
explicit and
implicit
connections
Profile
Fabian Abel
Data Scientist
Haves:
Interest...
8
Relevance Estimation + Additional Filters
Content-
based
features
Collaborative
features
Social
features
Usage
behavior
...
Challenges
Issues that we have to fight with…
9
What John writes…
10
And what he means…
Recruiter-John
International Sales Manager Call Center Agent
(10 EUR per hour)
Sal...
What Paul says he is…
11
And what he means…
Paul, the Candidate
CEO Network Engineer
(currently unemployed)
BI Engineer
(s...
12
Understanding the meaning of things that people write
in job postings and in their profiles is not trivial…
13
Profiles vs. People’s wishes for
their future
past
past
Profile describes a
user‘s past/current
position(s), not future...
14
Career path patterns: locations
How far away are the jobs that the users bookmark?
0-50 km
35%
51-200 km
22%
>200 km
43%
15
Career path patterns: locations
Climbing up the ladder?
junior
junior
senior
manger
senior manger
Today
Nextstep
53%
se...
CTR of job recommender over time
16
2014 2015 2016
CTR
New year,
new job
 bad CTR
No Love for
job RecSys
Intensified
love...
RecSys Challenge
http://recsyschallenge.com
17
Challenge
Given a user, the goal is to recommend job postings…
1. that the user may be interested in and
2. for which the ...
RecSys Challenge
Given a user, the goal is to predict those job postings that the
user will interact with.
19
?
Scala Dev,...
Datasets
1. Training data:
• User demographics (jobtitle, discipline, industry, career level, # CV entries,
country, regio...
Evaluation Measure
Mixture of…
- Precision@k (k = 2, 4, 6, 20)
= fraction of relevant items in the top k
- Recall@30 = fra...
Current Status
22
http://recsyschallenge.com
Current Status (ordered by rank)
23
http://recsyschallenge.com
Join the challenge!
• Deadline for submissions: June 26th 2016
 Current leaders: >600k points (ca. 20% of max. possible p...
Thank you
@fabianabel
http://recsyschallenge.com
www.xing.com
Upcoming SlideShare
Loading in …5
×

What's wrong with Recruiter-John? A non-trivial recommender challenge.

347 views

Published on

Some background about http://recsyschallenge.com

Abstract: Maria from Illusion Inc. recently told her colleague Recruiter-John that she needs an additional team member: "I'm looking for a new team member that can support us in building up our new search engine. Some experience with Lucene or other search technologies and some background in information retrieval would be cool!". Recruiter-John then did his work and redirected her message to the world: "We are searching for a Data Scientist who has a PhD in Computer Science or some related field, 5 years industry experience in building search engines with Lucene at large scale, is highly skilled in Elixir and Erlang and fits into our young, dynamic team…". Paul is currently searching for a job: "I am a Senior Software Engineer with more than 10 years of industry experience in Java, Scala, C++ and 15 other programming languages, familiar with technologies such as Solr, Hadoop, Pig, Hive, Hbase, Cassandra, Riak, … I'm currently looking for a job where I can bring my dog to work." Bringing together the demand and offer on the job market is a non-trivial problem.

In this talk, we will share some of our frustration and will describe how we tackle this problem with recommender systems on XING, a career-oriented social network. We will give insights on the RecSys challenge (http://recsyschallenge.com) which is our attempt to let the bright minds of the RecSys community solve the recommendation problem that we failed to solve properly over the past four years. :-)

Published in: Data & Analytics
  • Be the first to comment

What's wrong with Recruiter-John? A non-trivial recommender challenge.

  1. 1. What's wrong with you, Recruiter-John? A non- trivial recommender challenge. Budapest, June 2016 @fabianabel http://recsyschallenge.com
  2. 2. Challenge Given a user, the goal is to recommend job postings… 1. that the user may be interested in and 2. for which the user is an appropriate candidate. 2 Scala Dev (m/w) Scala Engineer Scala Dev, Hamburg user job postings Job recommende r companie s
  3. 3. Job recommendations
  4. 4. Job recommendations
  5. 5. 5 Title Company Employment type and career level Full-text description Key properties of a job posting
  6. 6. 6 Key sources for understanding user demands Social Network explicit and implicit connections Profile Fabian Abel Data Scientist Haves: Interests: web science big data, hadoop skills & co. Interactions data web social media clicks, shares, ratings big data kununu Interactions of similar users similar usershadoop scala
  7. 7. 7 Relevance Estimation Social Network explicit and implicit connections Profile Fabian Abel Data Scientist Haves: Interests: web science big data, hadoop skills & co. Interactions data web social media clicks, shares, ratings big data kununu Interactions of similar users similar usershadoop scala Content- based features Collaborative features Social features Usage behavior features Relevance Estimation (regression model) Logistic Regression P(relevant | x) = 1 1 + e-(b0 + bi xi)i n feature vector impact of feature xi
  8. 8. 8 Relevance Estimation + Additional Filters Content- based features Collaborative features Social features Usage behavior features Relevance Estimation (regression model) Location- based filtering Content- based diversification Monetary- based diversification Career Level filtering Filtering & Diversification 0.92 0.8 0.76 …
  9. 9. Challenges Issues that we have to fight with… 9
  10. 10. What John writes… 10 And what he means… Recruiter-John International Sales Manager Call Center Agent (10 EUR per hour) Sales Manager Sales Manager for B2B customers (80K EUR per year) Data Scientist skilled in Hadoop, Scala, Elasticsearch, … with PhD in … Data Analyst (skilled in SAS or Excel)
  11. 11. What Paul says he is… 11 And what he means… Paul, the Candidate CEO Network Engineer (currently unemployed) BI Engineer (skilled in old-school ETL) Shopman (in a kiosk) Data Scientist with 100+ skills Sales Manager
  12. 12. 12 Understanding the meaning of things that people write in job postings and in their profiles is not trivial…
  13. 13. 13 Profiles vs. People’s wishes for their future past past Profile describes a user‘s past/current position(s), not future wishes
  14. 14. 14 Career path patterns: locations How far away are the jobs that the users bookmark? 0-50 km 35% 51-200 km 22% >200 km 43%
  15. 15. 15 Career path patterns: locations Climbing up the ladder? junior junior senior manger senior manger Today Nextstep 53% senior 72% manger 54% senior manger 52%
  16. 16. CTR of job recommender over time 16 2014 2015 2016 CTR New year, new job  bad CTR No Love for job RecSys Intensified love for job RecSys B/A test: are the changes in the job RecSys really responsible for increased CTRs? Increase of job inventory (from 100k to ca. 750k) Using algorithms from RecSys Challenge - Feedback App - LSI for MoreLikeThis component - Entity resolution - Explanations - … Running out of ideas :-)  recsyschallenge.com
  17. 17. RecSys Challenge http://recsyschallenge.com 17
  18. 18. Challenge Given a user, the goal is to recommend job postings… 1. that the user may be interested in and 2. for which the user is an appropriate candidate. 18 Scala Dev (m/w) Scala Engineer Scala Dev, Hamburg user job postings Job recommende r companie s
  19. 19. RecSys Challenge Given a user, the goal is to predict those job postings that the user will interact with. 19 ? Scala Dev, Hamburg job postings Scala Engineer 2 months of impressions & interactions click bookmark
  20. 20. Datasets 1. Training data: • User demographics (jobtitle, discipline, industry, career level, # CV entries, country, region) [1M] • Job postings (title, discipline, industry, career level, country region) [1M] • Interactions (user_id, item_id, interaction_type, timestamp) [10M, 2 months] • Impressions (user_id, item_id, week) [30M, 2 months] 2. Task files: • Users (= User IDs for whom recommendations should be computed) [150k] • Candidate items (= item IDs that are allowed to be recommended) [300k] 3. Solution (secret) • Interactions (user_id, item_id) [1M, 1 week] Anonymization (Strings  IDs; users and interactions are enriched with artitificial noise) 20 http://recsyschallenge.com
  21. 21. Evaluation Measure Mixture of… - Precision@k (k = 2, 4, 6, 20) = fraction of relevant items in the top k - Recall@30 = fraction of relevant items in the top k - Success@30 = probability that at least one relevant item was recommended in the top 30 21 http://recsyschallenge.com
  22. 22. Current Status 22 http://recsyschallenge.com
  23. 23. Current Status (ordered by rank) 23 http://recsyschallenge.com
  24. 24. Join the challenge! • Deadline for submissions: June 26th 2016  Current leaders: >600k points (ca. 20% of max. possible points)  Prizes: 1st = 3,000 EUR; 2nd = 1,500 EUR, 3rd = 500 EUR • Workshop at RecSys 2016 in Boston: Sep 15th • RecSys Challenge 2017:  Dream = online evaluation 24 http://recsyschallenge.com
  25. 25. Thank you @fabianabel http://recsyschallenge.com www.xing.com

×