This document discusses recommending job ads to people based on their profile and interests. It describes a job recommendation framework that uses features like a user's career path, social connections, interests and interactions to estimate the relevance of job postings. A regression model is trained on past user interactions to combine these feature scores. Additional filters may then be applied to further refine recommendations. Career path graphs are mined from user profiles to infer appropriate job roles and industries based on their experience and education. The system aims to identify job postings that closely match a user's demands and skills.

1. Recommending Job Ads
to People
Fabian Abel, http://xing.com
December 2014 @wisdelft & @RecSysNL

3. Job recommendations

4. Recommender REST Service
Returns a ranked list of items
Request::
GET /rest/recommendations/jobs/user/4320245
4
User ID
Response:
[
{“item”: “urn:x-xing:job:5463323”, “score”: 0.87, “reasons”: […]},
{“item”: “urn:x-xing:job:5463267”, “score”: 0.87, “reasons”: […]},
{“item”: “urn:x-xing:job:5464812”, “score”: 0.87, “reasons”: […]},
{“item”: “urn:x-xing:job:5462781”, “score”: 0.87, “reasons”: […]},
]

5. Deployment infrastructure
Infrastructure on which we deploy recommender services
5
Search
indices
XING
Sources
/
XING
services
MySQL
NoSQL
live
updates
Batch
processing
batch
updates
Infrastructure
for
recommenders
Recommender REST service
XING
Products

6. RecSys Infrastructure
Infrastructure on which we deploy recommender services
6
Search
indices
XING
Sources
/
XING
services
MySQL
NoSQL
live
updates
Batch
processing
batch
updates
Infrastructure
for
recommenders
Recommender REST service
XING
Products

7. Pointers
7
Technologies
• Scala, Play, Akka: https://playframework.com/ - https://typesafe.com/ (à
start e.g. with some Activator template: https://typesafe.com/activator)
• Elasticsearch: http://www.elasticsearch.org/ (à start e.g. with “Getting
started” on https://github.com/elasticsearch/elasticsearch)
• Hadoop & Co., e.g. Hortonworks distributionà e.g.
https://github.com/hortonworks/hadoop-tutorials):
• Hive: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF
• Spark: https://spark.apache.org/
• Deployment, e.g.: https://www.docker.com/

8. Building/enhancing a recommender service
Typical steps within the development cycle
8
1. Analyze (click) data to identify potential features.
Technologies: Hive, R, Excel
2. Implement batch jobs for creating “indexes” that build the basis for the chosen
features.
Technologies: Hive, MapReduce, Oozie, Elasticsearch, Cassandra, MySQL
3. Implement service logic that queries the “indexes” and implements the actual
recommender interface (à def getRecos(userId: Long): Future[Seq[Reco[T]]]).
Technologies: Scala, Play framework
4. Analysis and offline evaluation of recommender: quantitative statistics, manual
assesment, leave-n-out-evaluation.
Technologies: R, Scala
5. Deployment and A/B test phase with in-depth analyses.
Technologies: R, Hive, Scala, Excel

9. Job Recommendation
Framework

10. Challenge
Identifying job postings that match the demands of the user and company
10
… User
0.92 0.8 0.76
Job
Recommender
Companies Job postings

11. Key properties of a job posting
Job postings on XING
11
Title
Company
Employment type
and career level
Full-text
description

12. Key properties of a job posting
Job postings on XING
12
Employees of the
company
Social benefits
Location

13. Key sources for understanding user demands
Exploiting patterns that are found in the data(graph)
13
Social Network
explicit and
implicit
connections
Senior Data
Scientist
Data
Strategist Manager
Data Scientist
Engineer
Researcher
Careerpath
next career
options
Profile
Fabian Abel
Data Scientist
Haves:
Interests:
web science
big data, hadoop skills & co.
data
web
social media
Interactions
clicks, shares,
ratings
big data
kununu

14. Relevance Estimation
Final relevance score of an item is obtained by combining the
scores coming from the “sub-recommenders” (= features)
14
Social Network
explicit and
implicit
connections
Senior Data
Scientist
Data
Strategist Manager
Data Scientist
Engineer
Researcher
Careerpath
next career
options
Profile
Fabian Abel
Data Scientist
Haves:
Interests:
web science
big data, hadoop skills & co.
data
web
social media
Interactions
clicks, shares,
ratings
big data
kununu
Content-based
features
Knowledge
graph
features
Collaborative
features
Usage
behavior
features
Relevance
Estimation
(regression model)
Logistic Regression
P(relevant | x) =
1
1 + e - (b0 + bi xi) i n
feature vector impact of feature xi

15. Learning the model for
relevance estimation
1 + e - (b0 + bi xi) i n
15
user
i
(was
confronted
with
item
x
at
Jme
t)
x1:
profile
match
x2:
interestg
graph
match
P(relevant | x) =
x3:
cf
score
x4:
LocaJon
match
x5:
career
level
match
1
...
relevant?
u1
(t1)
0.87
0
0.81
1.0
0.75
...
0
u2
(t2)
0.0
0.76
0.61
0.15
1.0
...
1
u3
(t3)
0.13
0.42
0
0
1.0
...
0
u4
(t4)
0.5
0
0
0.0
0.0
...
1
...
...
...
...
...
...
...
...
Model:
b1
=
0.12
b2
=
0.05
b2=
0.5
b3
=
0.8
b4
=
0.65
..
b0
=1.43
...
...
...
...
...
...
...
u2
(t5)
0.0
0.76
0.61
0.15
1.0
...
0
u3
(t6)
0.15
0.67
0.9
1.0
1.0
...
1
...
...
...
...
...
...
...
...

16. Learning the model for
relevance estimation
1 + e - (b0 + bi xi) i n
correct?
16
user
i
(was
confronted
with
item
x
at
Jme
t)
x1:
profile
match
x2:
interestg
graph
match
P(relevant | x) =
x3:
cf
score
x4:
LocaJon
match
x5:
career
level
match
1
...
relevant?
u1
(t1)
0.87
0
0.81
1.0
0.75
...
0
u2
(t2)
Training
0.0
data
0.76
0.61
0.15
1.0
...
1
u3
(t3)
0.13
0.42
0
0
1.0
...
0
u4
(t4)
0.5
0
0
0.0
0.0
...
1
...
...
...
...
...
...
...
...
Model:
b1
=
0.12
b2
=
0.05
b2=
0.5
b3
=
0.8
b4
=
0.65
..
b0
=1.43
...
...
...
...
...
...
...
u2
(t5)
0.0
0.76
0.61
0.15
1.0
...
0
u3
(t6)
Test
&
valida0.15
Jon
data
0.67
0.9
1.0
1.0
...
1
...
...
...
...
...
...
...
...
!
Model
with
highest
predic4on
accuracy
wins
the
game

17. Relevance Estimation
Learnt model (= parameters of the function) are used to compute the
combined relevance scores
17
Social Network
… User
0.92 0.8 0.76
explicit and
implicit
connections
Senior Data
Scientist
Data
Strategist Manager
Data Scientist
Engineer
Researcher
Careerpath
next career
options
Profile
Fabian Abel
Data Scientist
Haves:
Interests:
web science
big data, hadoop skills & co.
data
web
social media
Interactions
clicks, shares,
ratings
big data
kununu
Content-based
features
Knowledge
graph
features
Collaborative
features
Usage
behavior
features
Relevance
Estimation
(regression model)
Job postings

18. Relevance Estimation + Additional Filters
Filtering (rules) may dampen the relevance scores or filter out items
18
Content-based
features
Knowledge
graph
features
Collaborative
features
Usage
behavior
features
Job Reco.
Service
Relevance
Estimation
(regression model)
Location-based
filtering
Social
filtering
Filter out
declined/
known
Career Level
filtering
Filter

19. Mining for features
Example: career path mining

20. Properties of career path steps
Analyzing the CV (= professional experience + education)
20
Professional experience
Job title
Time range
Company
Career level Industry

21. Properties of career path steps
Analyzing the CV (= professional experience + education)
21
Professional experience
Job title
Time range
Company
Career level Industry
Educational Background
Time range University
Study Degree
program

22. Career path transitions
Understanding transitions in the career path graph
22
Data Scientist
Machine Learning Expert
J2EE Developer
Web Developer
MSc Computer Science
CV:
Web Developer
MSc Computer
Science
Data Scientist
J2EE
Developer
Machine
Learning
Expert

23. Career path transitions
Understanding transitions in the career path graph
23
Data Scientist
Data Scientist
Machine Learning Expert
Machine Learning Expert
J2EE Developer
Web Developer
MSc Computer Science
CV:
Web Developer
MSc Computer
Science
Data Scientist
J2EE
Developer
Machine
Learning
Expert
Data Scientist
Machine Learning Expert
PhD Data Mining
CV:
J2EE Developer
MSc Computer Science
CV:
PhD Data
Mining

24. Career path graph
Weighted directed graph with different types of nodes (job roles, education)
Association rule mining for constructing
24
the career path graph:
• Association rules (= edges):
Job role A à Job role B
Education X à Job Role Y
...
• Minimum support (e.g. at least k
transitions with A and B have to occur in
the data)
• Minimum confidence (= probability(B | A)
= weights of edges)
Web Developer
MSc Computer
Science
Data Scientist
J2EE
Developer
Machine
Learning
Expert
Similarly,
graphs
are
constructed
for:
PhD Data
Mining
Industry
X
à
Industry
Y
Job
role
Aà
Company
C
...
Thresholds
for
min-­‐support
and
min-­‐
confidence
need
to
be
learned
(ideally
individually
per
“industry
segment”)

25. Career path hypergraph
Mining association rules with multiple premises
Association rule mining for constructung the
25
career path graph:
• Association rules with more than one
premise:
Job role A, Job role B à Job role C
Education X, Job role A à Job Role B
• Minimum support (e.g. at least k transitions
with A and B and C have to occur in the data)
• Minimum confidence (= probability(C | A,B))
Data Scientist
Machine
Learning
Expert
PhD Data
Mining
J2EE
Developer

26. Inferring Features from Career path graph(s)
Probabilities that the job role is appropriate for the user
26
Data
Scientist
Job posting
Machine
Learning
Expert
PhD Data
Mining
User
Features:
P( D at a | Machine
Learning
, )
Scientist
Expert
F2: PhD Data = 0.79
Mining
P( D at a | Machine
Learning
)
Scientist
Expert
F1: = 0.52
P( D at a | Machine
Learning
, )
Scientist
Expert
F3: 5 years = 0.6
experience
…
Career
path
graph

27. 27
futureme.xing.com
spin-futureme.off project which xing.allows com
for
browsing the career-path graph

28. futureme.xing.com
28

29. Evaluation of recommender
systems

30. Evaluation of Recommender Systems
Metrics & Methods
Compare
30
Key Metrics: Precision@k, “Success-Rates@k” (e.g. CTR)
Evaluation methods:
• Quantitative statistics: count total number of recommendations per user,
measure overlap between old and new recommender, …
• Ad-hoc assessments: UI for assessing the quality of individual
recommendations and comparing two lists
• Leave-n-out cross validation: hide parts of the historic data, run the
recommender and try to predict the “hidden data”
• A/B testing: x% of users are served with strategy A, 100-x% with strategy
Leave-­‐n-­‐out
cross
valida3on
Track
history
of
interacJons:
User
X
performs
AcJon
Y:
Training
Data
ValidaJon
Data
Strategy X
Strategy Y
Measure
Prediction X
Measure
Prediction Y
A/B
Tes3ng
Strategy X
Strategy Y
KPIs X
KPIs Y
Users
Group
A
Group
B

31. Is Strategy A better than strategy B?
Based
on
1
Million
samples
31
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
A
B
Success@4
Success@20
Based
on
1000
samples

32. Significance tests
Example: t-tests
Null Hypothesis H0: performance
of strategy A = performance of
strategy B
Alternative H1: performance of A
> performance of B
32
t-­‐test
computes
p-­‐value:
probability
that
the
staJsJcal
result
is
–
under
the
given
null
hypothesis
–
at
least
as
extreme
as
the
one
that
was
observed
!
reject
H0
with
p-­‐value
<
α
(=
significance
level)

33. Significance tests
Example: t-tests
Null Hypothesis H0: performance
of strategy A = performance of
strategy B
Alternative H1: performance of A
> performance of B
33
Beware
of
p-­‐score
hacking,
e.g.:
“I
found
a
metric
where
we
have
a
significant
improvement
(with
p
<
0.05)”
t-­‐test
computes
p-­‐value:
probability
that
the
staJsJcal
result
is
–
under
the
given
null
hypothesis
–
at
least
as
extreme
as
the
one
that
was
observed
!
reject
H0
with
p-­‐value
<
α
(=
significance
level)

34. Challenges regarding the evaluation
Understanding the performance of a recommender system is not easy
34
Challenges:
• Tracking clicks/interactions, e.g. not all recommendation clicks/interactions can
easily be tracked (e.g. email, third party apps are not properly tracked)
• Changes on the platform, e.g. A/B tests in the UI, marketing campaigns, etc.
may impact the perfrormance of the recommender
• “Novelty” effect: if recommendations change strongly then users are curious and
interact with recommendations, but after a while the curiosity may drop again
which leads again to decreasing click-through-rates
• “Position bias”: top-ranked recommendations are more likely to be clicked “by
defaut”
• “p-score hacking”: http://en.wikipedia.org/wiki/P-hacking

35. The professional network
www.xing.com
Thank you
@fabianabel
xing.com
futureme.xing.com