The right model for the job: vacancy recommendations at Randstad

to our
data science
meetup
welcome
hosted by
randstad group the netherlands IT
and BigData Republic

a story about
data driven
randstad group.
randstad group meets
data scientists.
falco vermeer

© randstad group the netherlands IT | 3
our purpose is to
support people and
organizations in
realizing their true
potential.
randstad group
human forward.

randstad in numbers
#1 HR services provider worldwide.
€ 23.8
billion in revenue
key figures 2018
262,500
permanent placements
38,820
avg. corporate employees
4,826
offices in 39 countries
48%
women in leadership positions
670,900
people we help to work every day

|| 5
back in the good old days...
information has always been an important asset for Randstad.

| 6
labour market
predicting trends and scarcity
on the labour market
candidates
tailored insights on job
& career at every moment
customers
predict the most suitable
candidate at the right time

||
data driven randstad
purpose - what & how
7© randstad group the netherlands IT
”With data & insights we help our customers & talents moving forward in the world of work,
and by building solutions for our own consultants we help them in organising their work &
making decisions in a more effective way.”
For decisions at strategic & tactical level we develop dashboards,
a self-service BI environment and ad-hoc analytics to help making
the right decision at the right time.
For decisions on operational level we automate the decision
making process and help the user making smarter choices.
Therefore we build models and integrate them in our processes.

|| 8
our data driven journey.
redesigning the core of our company
2014
2018
datatechnology
skills & peopleorganisation

||
Data Scientist
meet our data & analytics team!
9
(Advanced) Analyst
Machine Learning
Engineer
Analytics Translator
Data Engineer BI Engineer
© randstad group the netherlands IT

||
our core process since 60 years.
a lot of potential for predictive data use cases

vacancy recommender
the use case
hugo valk
randstad group the Netherlands IT

the business process of randstad.

team SmartMatch.
SmartMatch has the ambition to improve the
end user experience by providing actionable
insights in the needs and wishes of talents as
well as clients.
SmartMatch complements Randstad’s
expertise using Artificial Intelligence and
Machine Learning.
SmartMatch develops algorithms and exposes
these via web services.

main challenges.
data quality database utilizationpersonalized advice
| 14

data quality.
With our taxonomy we can for example:
Provide clients with suggestions on necessary
skills, certificates and educations you might
need for this job
Present candidates skills they might have
based on their work experience.

database utilization.
We provide:
a talent search engine based on ElasticSearch that
replaced the old engine.
the talent recommender which is an algorithm to find the
most promising candidates for a vacancy in our database.

personalized advice for candidates.
The vacancy recommender instantly offers the
best matching vacancies for any talent.
The vacancy recommender will increase the
quality of applications and improves the
utilization of talents in a more efficient way.

primary impact of smartmatch solutions.

“working software is the
primary measure of progress.”
— Agile Manifesto

vacancy recommender - recruiter perspective.

|
vacancy recommender
talent perspective.
When you are known to us, we
provide you with personalized
vacancy recommendations.
The same component is shown on
the job board, where yuu search
through our vacancies.

results vacancy recommender.
Results on the Tempo-Team website are
looking great!
27% improvement in application rate
compared to self-found vacancies
Still waiting for results from the
Randstad website and the effect for our
recruiters.
27%

“All we have to decide is what
to do with the time that is
given us.”
— J.R.R. Tolkien

project lifecycle vacancy recommender.
typical
adoption
curve
time
adoption

inception PoC MVP Scale-up Mature Phase-out
ideas &
research
business concept
plan
develop model
test model with
historical data
test model with
selected users
deploy to
production
API
minimal
engineering effort
but not losing
quality
larger user groups
more channels
model
improvements
more use cases
iterate on product
maintenance

implementation frontend recruiter
inception PoC
implementation Randstad jobboard
MVP Scale-up Mature Phase-out
idea
implementation tempo-team jobboard
evaluation
1st version algorithm
test on google analytics data
implementation tempo-team jobboard improved version
evaluation

project learnings - what went well.
In the past we had problems to scale:
+ engineering involvement too late
leading to code quality issues
+ many features in model from the beginning
high complexity made maintenance more problematic
First time we went through this lifecycle, which did help to address these issues.
Takeaways:
→ minimum viable product must be as simple as possible but no simpler!
→ minimum viable product must be of good quality from the start!

project learnings - what could be better.
frontends are developed and maintained in another team.
- difficult to get good measurements
- difficulties to develop in sync because of different priorities
Takeaway:
→ Agree on the priorities before you start your MVP!

thank you!
any questions?

|
vacancy recommender
the science.
robbert van der gugten

|
matching from multiple perspectives.
32
match
recruiter provide all open vacancies
talent
wants job of the highest
possible aspiration
client
wants best possible
talent available

|
online journey
matching by showing relevant vacancies to talent.
33
talent match
kpi
search find apply
model
touchpoint
rank
data
feedback

| 34
vacancy recommender.
more effective and efficient process.
no time to
read
no need to
read
vacancy recommender

|
the reciprocal recommendation problem.
35
“capability match”
“preferred vacancy predictor”
talent vacancy

|
talent vacancy
preferred vacancy predictor.

|
we can use interaction data.
37
1 ? ? 5 1
? 1 1 ? ?
? ? 5 1 ?
? 5 ? ? 1
1 ? 1 ? ?
talent vacancy
talent-vacancy interaction matrix

|
vacancies
talent
×
≈ predicted
preferences
=
predicted preference of
talent t for vacancy v
preferences
38
matrix factorization.
talent factors
vacancy factors

|
from scipy.sparse import csr_matrix
from sklearn.decomposition import NMF
talent_ids = df['talent_id'].unique()
vacancies = df['vacancy_id'].unique()
data = df['action'].tolist()
row = df['talent_id'].astype('category', categories=talent_ids).cat.codes
col = df['vacancy_id'].astype('category', categories=vacancies).cat.codes
factorizer = NMF(n_components=20, random_state=42)
model = factorizer.fit(sparse_matrix)
sparse_matrix = csr_matrix((data, (row, col)),
shape=(len(talent_ids), len(vacancies))
talent_factors = model.transform(sparse_matrix)
vacancy_factors = model.components_

|
problem: vacancies are timely.
time
{v1
,v2
,v3
} {v1
,v2
,v3
,v4
} {v2
,v4
}

|
available data.
41
profile online interactions
preferred job title
location
preferred salary
hours per week
night shift
...
job detail page
job applications
soon: applied filters
soon: search queries

|
modelling.
42
matrix factorization “item” additional features

|
scoring.
[x1
, x2
, x3
]x
=
x4
pred(t, job title)
-
[v1
, v2
, v3
]v
[t1
, t2
, t3
]t

|
offline evaluation set up.
calculate the average rank of historical matches
44
historical matches
use the model to score vacancies
for different talents
rank all vacancy scores for
each talent
1
2
3
calculate average rank for
the historical matches
1
0.993
0.945

|
offline evaluation results.
45
top 10
percentage
average rank
0.83 0.50
30.2 1.6
average
rank
top 10%

|
metric
matching by showing relevant vacancies to talent.
46
talent match
kpi
search find apply
touchpoint
rank datamodel
feedback

|
live results: ideal experiment setup.
47
recommendations no recommendations random recommendations
compare application rate between three groups

|
live results: used setup.
48
compare application rate for logged in users
recommendations no recommendations

|
live results.
application rate improvement: 9%

|
talent vacancy
capability match.
© randstad group the netherlands IT 50

|
same goal, different perspective, different data.
51
profile work experience
education
skills
driving licences
languages
...
job title
sector
company
...

|
beyond matrix factorization.
?1 ? ? 1 1
? 1 1 ? ?
? ? 1 1 ?
? 1 ? ? 1
1 ? 1 ? ?

|
StarSpace model.

| 54
+ ?
... ... ...

StarSpace model.

| 55
talent_model = Model(inputs=talent_inputs,
outputs=talent_embedding)
vacancy_model = Model(inputs=vacancy_inputs,
outputs=vacancy_embedding)
talent_embedding = talent_model(talent_input_features)
pos_vacancy_embedding = vacancy_model(vacancy_pos_input_features)
random_vacancy_embedding = vacancy_model(vacancy_random_input_features)
pos_similarity = Dot(axes=1, normalize=True)(
[talent_embedding, pos_vacancy_embedding])
random_similarity = Dot(axes=1, normalize=True)(
[talent_embedding, random_vacancy_embedding])
similarities = Concatenate()([pos_similarity, random_similarity])

| 56
model = Model(
inputs=[talent_input_features, vacancy_pos_input_features,
vacancy_random_input_features],
outputs=similarities)
similarities = Concatenate()([pos_similarity, random_similarity])

| 57
scoring.
57
talent embedding
cache
realtime
similarity
all vacancy embeddings

|
offline evaluation results.
58
top 10
percentage
average rank
0.78 0.50
33.7 1.6
average
rank
top 10%

|
bringing it together.

|
live results.
application rate improvement
preferred vacancy predictor: 9%
preferred vacancy predictor + capability match: 27%

|
key takeaways.
matrix factorization provides a good baseline method to
recommend vacancies to talents
1
2
3
use the StarSpace algorithm to generate comparable
embeddings of different entities
the reciprocal recommendation problem can be solved by
combining approaches

|
one StarSpace to rule them all.
+ ?
... ... ...

|
thank you!
questions?

vacancy recommender
engineering the solution
hugo valk

“This is a story of how a
[team] had an adventure, and
found [themselves] doing and
saying things altogether
unexpected..”
— J.R.R. Tolkien

so, how do we bring this to the user?

• providing computing power for ML jobs
• scheduling jobs
• ingesting data and training
• serving model scores via an API
• integration & final API
• moving to ‘near real time’
the steps to put the model in production.

setting the stage
69
the context in 2016.

november 2016.
Data center AWS
Data Lake
(RedShift)
backoffice
service
websites
frontoffice ...
...
...
... ...
Sales application
... ...
AWS was new territory for
Randstad. Most of the IT
was hosted on-premise.
Just a data lake.
No standards and
out-of-the-box ML services.
SageMaker came one year
later.
Some team members
had experience with
earlier AWS efforts.

the solution
what we put in production.

“Little by little,
one travels far.”
— J.R.R. Tolkien

• scheduling jobs

|
We use Docker on AWS
Elastic Container Services
• ECS Cluster with 1 big EC2
instance assigned to it
• We run ECS Services with a task
definition
• An ECS task runs a number of
Docker containers
ECS Cluster
computing power with AWS ECS.
EC2 Instance
ECS Service ECS Service
ECS
Task
ECS
Task

Since we have several Docker containers
running, we will omit the Service/Task
structure and the actual EC2 instances
for clarity.
we will summarize this.
ECS Cluster
ECS Cluster
EC2 Instance
ECS Service ECS Service
ECS
Task
ECS
Task

• scheduling jobs

|
SmartMatch Compute
Cluster
Apache Airflow is commonly used for
scheduling tasks.
• Distributed workers
• Execute a directed acyclic graph
(DAG) of tasks
• Web interface to view and control
the DAGs
Deployed as Docker containers.
scheduling with Apache Airflow.
Airflow
core
Airflow
webserver
Airflow
workers
Dag
syncer
ELB
PostGreSQL
RDS
Redis
ElastiCache
DAGs in
S3

scheduling with Apache Airflow.

|
SmartMatch Compute
Cluster
The training DAGs are run on the Airflow
worker. We omit the compute cluster, the
scheduler, the UI, the Airflow storage
components, etc.
In the next slides we will show a DAG running
in one of the Airflow workers like this:
we summarize again.
Airflow
core
Airflow
webserver
Airflow
workers
Dag
syncer
ELB
PostGreSQL
RDS
Redis
ElastiCache
DAGs in
S3
Training job

• scheduling jobs

ingesting data and training.
Shared
feature
store
Preferred
Vacancy Predictor
trainer
Shared
features
ingestion
Capability Match
trainer
Data Lake
(RedShift)
Google Analytics
Capability Match
model in S3
Preferred
Vacancy Predictor
model in S3

• scheduling jobs

SmartMatch
Production
Cluster
Preferred Vacancy
Predictor scoring engine
Capability Match scoring
engine
serving model predictions with Flask.
• A separate ECS cluster for serving production
APIs
• Internal model APIs with Flask
• These serve the scores for the individual model
components that make up the total prediction
ELB
ELB

SmartMatch
Production
Cluster
Preferred Vacancy
Predictor scoring engine
Capability Match scoring
engine
yet another summary.
We will omit the ELB and the fact that the scoring
engines run as a Docker container.
Preferred
Vacancy Predictor
scorer
Capability Match
scorer
ELB
ELB

serving the model scores.
Shared
feature
store
Preferred Vacancy
Predictor trainer
Shared
features
ingestion
Capability Match
trainer
Data Lake
(RedShift)
Google Analytics
Capability Match
model & data in
S3
Preferred Vacancy
Predictor model &
data in S3
Capability Match
current data ingest
Preferred Vacancy Predictor
current data ingest
Preferred Vacancy
Predictor scorer
Capability Match
scorer
Daily restart lambda

• scheduling jobs

integration and final API.
Preferred Vacancy
Predictor scorer
Capability Match
scorer
Vacancy
Recommender
API
Frontend
system
Firehose
register
API definition
API
manager

• scheduling jobs

we only refresh data in batch.
Shared
feature
store
Preferred Vacancy
Predictor trainer
Shared
features
ingestion
Capability Match
trainer
Data Lake
(RedShift)
Google Analytics
Capability Match
model & data in
S3
Preferred Vacancy
Predictor model &
data in S3
Capability Match
current data ingest
current data ingest
Preferred Vacancy
Predictor scorer
Capability Match
scorer

‘near real time’ updates on vacancies.
Capability Match
model & data in
S3
Preferred Vacancy
Predictor model &
data in S3
Preferred Vacancy
Predictor scorer
Capability Match
scorer
Vacancy
Management
System
Business
Event
Service
PostGreSQL
persistent
Vacancy
cache
SQS
API manager
Lambda reads vacancy
and performs
preprocessing

time to recap
WHAT have
we just done?

Shared feature
store
Preferred Vacancy
Predictor trainer
Shared features
ingestion
Capability Match trainer
Data Lake
(RedShift)
Google Analytics
Capability Match model
& data in S3
Preferred Vacancy
Predictor model & data
in S3
Capability Match current
data ingest
Preferred Vacancy Predictor current
data ingest
Preferred Vacancy
Predictor scorer
Capability Match
scorer Frontend
system
Vacancy
Management
System
Business
Event
Service

having done all this
we learned a lot.

we now know many AWS components.

we have great experience with fully scripted environments.

Shared
feature store
Preferred Vacancy
Predictor trainer
Shared
features
ingestion Capability Match
trainer
Data Lake
(RedShift)
Google
Analytics
Capability Match
model & data in S3
Preferred Vacancy
Predictor model &
data in S3
Capability Match
current data ingest
current data ingest
Preferred Vacancy
Predictor trainer
Capability Match
trainer Frontend
system
Vacancy
Management
System
Business
Event
Service
we have seen the different components of a ML solution.
data
ingestion
handling live events
training
models
expose scoring
results with API

we have been through the journey of a data scientist.
access to
the data
exploration
of the data
modeling
versioning &
source
control
scheduling
model
training
scoring in
production

“real artists ship!”
98
rinse & repeat all this for,
let’s say,
another 50 algorithms...

“Don’t adventures ever have
an end? I suppose not.
Someone else always has to
carry on the story.”
— J.R.R. Tolkien

moving forward
the DataHub.

problems the data scientist encounters.
access to
the data
exploration
of the data
modeling
versioning
build
deploy
scheduling
model
training
scoring in
production
laptop
or
shared RStudio
Server
no standards
lack of
knowledge
no shared
self-service
scheduling
infrastructure
lack of skills
building
production APIs

self-service DataHub.
data lake
project
space
ECS task
SageMaker
training job
model
endpoint
SageMaker
notebooks
DAG
data
subscriptions
Airflow is used to
schedule
everything
DataHub provides
roles, policies,
naming, tagging, etc.
SageMaker notebook
instances contain
JupyterLab for
research
Standardizing data
unloads/subscriptions
will later allow for
notifications and
governance

project template with cookiecutter.
• provides default project layout
• folder for notebooks - store your research
• DAG - write the DAG to run your scheduled jobs
• operator for data ingest
• operators for AWS SageMaker
• operator for ECS tasks
• Docker files - for SageMaker models or custom tasks
• standard Airflow operators to perform tasks on the DataHub
• Jenkins webhook provides standard deploy scripts

10
4
data subscriptions.
Get a periodic data extract from the
DataHub in your project space.
• no involvement data engineering
• central management of all data
subscriptions

10
5
self-service JupyterLab
SageMaker instances.
Get a JupyterLab instance with access
to the data lake and your own project
space.
• no involvement data engineering
• comes with Anaconda installed
• comes with working GPU support

revisiting the data scientist journey.
access to
the data
exploration
of the data
modeling
versioning
build
deploy
scheduling
model
training
scoring in
production
personal
computing power
with SageMaker
notebook
instances
standard build &
deploy pipeline
maintained by
data engineering
Airflow cluster
with fixed user
management
project template
with all the
necessary
components
standard way of working supported by data engineering

“A single dream
is more powerful
than a thousand realities.”
— J.R.R. Tolkien

thank you!
any questions?

The right model for the job: vacancy recommendations at Randstad

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The right model for the job: vacancy recommendations at Randstad

Similar to The right model for the job: vacancy recommendations at Randstad (20)

Recently uploaded

Recently uploaded (20)

The right model for the job: vacancy recommendations at Randstad