Big Data Meetup Argentina Presentation About an implementation of Learning to Rank @OLXGroup Argentina

1. How we learned to rank
search results
Mouloud Lounaci & Andres Pipicello
Argentina Big Data Meetup
Meetup #6: Long time, no see OLX
October, 2018

2. 2
mouloud.lounaci@olx.com
@mlounaci
https://www.linkedin.com/in/mlounaci
andres.pipicello@olx.com
https://www.linkedin.com/in/andrespipicello

3. Plan
OLX Group
Personalization and Relevance (PnR)
Learning To Rank (LTR)
The ranking journey
Building Dataset
Modeling
Serving the model
Results
3

4. 4
Preface What do you need to know
about OLX Group ?
OLX Group

5. Scale of Data at OLX Group
5
35B
Monthly
Page Views
350M
Monthly
Users
60M
Monthly
Listings
4B
Daily
Events
Every minute...
2.5M events captured
500 houses listed
500 cars listed
1000 phones listed

6. Classifieds ?
6
Two-sided Marketplace
● Buyers looking for goods or services
● Sellers offering goods or services
OLX’s Mission
● Match buyers with sellers using
○ large scale data
○ state-of-the-art technology

7. Today it’s ALL about “search”
7
▪ Retrieval of relevant listings
– Query understanding
– Query-listing matching
▪ Ranking of relevant listings
– Learning to rank (LTR) using
query, user and listing
features
Query
Ranked relevant
Items

8. Actually it’s about “Ranking”
8
▪ Retrieval of relevant listings
– Query understanding
– Query-listing matching
▪ Ranking of relevant listings
– Learning to rank (LTR) using
query, user and listing
features
Query
Ranked Items

9. 9
Chapter 1
Who are “we” ?
What do we do ?
How do we do it ?Personalization and
Relevance

10. 10
PnR - The Team

11. PnR - Architecture
11

12. PnR - Architecture
12
Data Sources
Indexing
Retrieval +
Ranking

13. Ad Retrieval
13
Phoenix
Constructs the feed to send to the user
Manages all the different spells (algorithms)
used in the feed
Splitter for A|B Testing

14. Ad Retrieval
14
Loki
Executes the spell (algorithm) from Phoenix
Interacts with all the different data sources
Caches items for fast Page 2 retrieval

15. 15
Chapter 2
Ragnarok ?
Why LTR ?
What is LTR for usLearning To Rank

16. What we want to do ?
“Learn from the data how to rank a resultset
for a search query”
aka.
RAGNAROK
AD RERANKER

17. Manual models become hard to tune with a very large number of
features.
1
Leverages large volume of user behaviour (Clicks/replies) data in
an automated way
Create a personalized ranking by including user features (social
search)
2
3
Why Learning To Rank ?

18. Top r e
do n
Top Rer d
do n
User
query
(Re)ranked
results
Spell
Returned
Documents
User
behaviour
If I click/reply,
then it’s
relevant for me
RAGNAROK
AD RERANKER
Overview

19. 19
Chapter 3
Before we start, we need
Tools
The search for gold
Mining the gold (Spark),
Funnel ?
Modeling, or transforming
the gold
Serving the model
The ranking journey

20. The ranking journey
Step 1
Building Infra
● Access large
history Data
(Reservoir)
● Build Infra to
process it
(EMR)
Building Dataset
● Process Label
(Judgement
score proxy
with
Clicks/Replies).
● Process
Features.
Step 2
Analysing Dataset
● Analyse Click
and reply
behaviour
● Build “Gold
standard”
dataset for
ranking
Step 3
Building Model
● Iterating on
models
● Evaluating
models
● Selecting a
model
Step 4
Serving Model
● Design Service
Architecture
● Define Service
Requirement
● Create Ranking
Endpoint
Step 5
Integration with PnR
Architecture
● Integrate the
ranking in the
ad retrieval flow
● Define the
interaction with
the PnR
components .
Step 6

21. 21
Step 1
Building Infra

22. Step 1
User browsing
(parquet - 1h delay)
RELEVANCE
RESERVOIR
Ads
(json - 5 min delay)
Labeled Dataset
Features
Building Infra
STORAGEPROCESSING
Building Infra

23. Big Data ?
23
1 year android history for South Africa
data...
Step 1
Building Infra
5B
User events
800M
Search
Impressions
40M
Individual
searches

24. Scalability is key
24
1 year history for South Africa data...
Step 1
Building Infra
5B
User events
800M
Search
Impressions
40M
Individual
searches

25. 25
Step 2
Building Dataset

26. Step 2
Gold looks like this for us...
query_id query
features
Item
position
item_id Item
features
Label (Relevance
Judgement)
1 ... 1 item1 ... 0
1 ... 2 item2 ... 3
1 ... 3 item3 ... 1
1 ... 4 item4 ... 2
Building Dataset
The search for the “gold” standard dataset

27. Step 2
Gold looks like this for us...
query_id query
features
Item
position
item_id Item
features
Label (Relevance
Judgement)
1 ... 1 item1 ... 0
1 ... 2 item2 ... 3
1 ... 3 item3 ... 1
1 ... 4 item4 ... 2
Building Dataset
The search for the “gold” standard dataset

28. Step 2
We used spark (EMR) to build the dataset from user browsing data.
Building Dataset
Hydra
(Trackings)
Labeled
searches
(funnel)
Labeling
(apply funnel)
Let’s “spark” it off

29. Proxy label, the “funnel”
Step 2
Building Dataset
?

30. 30
Step 3
Analyzing Dataset

31. Step 3
● Considering searches with at least one reply for training
(to improve quality)
● Include searches with more than 3(4) impressions (user
behaviour affected by smaller resultset)
● Inside each search consider impression up to 30-50-60th
position
● Metric that gives more importance to top position(NDCG with
customized decay)
Analyzing Dataset
Analysing Dataset

32. 32
Step 4
Building Model

33. Q1
D1,1
D1,2
…
D1,m
Q2 Qn
...
D2,1
D2,2
...
D2,m
Dn,1
Dn,2
...
Dn,m
Pointwise Pairwise Listwise
f(Qi, Di,j) = s o F(Qi, Di,j > Di,k) = s o n {o,1} f(Qi, {Di,j,...,Di,m}) = {Di,j,...,Di,m})
ra d
Baseline
Q1,D1
Q1,D2
Q1,D3
Q1,D1 0.85
Q1,D3 0.65
Q1,D2 0.30
Q1,D1>D2
Q1,D2>D3
Q1,D3>D4
Q1,D1>D2 1
Q1,D2>D3
0
Q1,D3>D4
1
Q1,D1
Q1,D2
Q1,D3
D1
D3
D2
Step 4
Building Model
Start with a simple approach

34. McRank from classification to ranking
● Pointwise approach
● Train a classifier to predict the
relevance judgment k i {0, 1, 2}
● Use the class probabilities P(Y=k)
Ran g re =∑ P(Y=k) ∗T(k), w e n o se T(k)=k
Inspired by :
https://papers.nips.cc/paper/3270-mcrank-learning-to-rank-using-multiple-classification-and-gradient-boosting.pdf
Step 4
Building Model

35. 35
Combined Model
Item Features Buyer Features Seller Features
Static Features
Interaction
Features
(Browsing)
Three Classes of Features
Step 4
Building Model

36. Search String
Search Location
Search Time
Ad Title
Ad Description
Ad Location
Ad Creation Time
Ad Price
Ad Private or Business
Ad Image Count
Ad Category
Textual Similarity (BM25)
Length of the Title
Length of the Description
Freshness
Proximity
Price
Is the Seller a Private Business
Image Count
Category
Raw attributes Features
Static Item/query features
Step 4
Building Model

37. Item Interaction Features - Example
Interactions
Impressions
Ad Views(Clicks)
Replies
Data source: ods.fact_listing_activity
Time Interval
30 days
7 days
Last day
Item Interaction Features
num_impressions_30days
num_adviews_30days
num_replies_30days
num_impressions_7days
num_adviews_7days
num_replies_7days
num_impressions_lastday
num_adviews_lastday
num_replies_lastday
Step 4
Building Model

38. 38
Step 5
Serving Model

39. We met “Mleap“ on the way
39
Step 5
Serving Model

40. 40
Step 5
Serving Model
The Service
Aws Data Pipeline For
training
Scala Akka Htttp with
mleap service on
Openshift for
prediction
Training every 7 days Serving
RAGNAROK
AD RERANKER
Ranked Items
ReRanked Items

41. 41
Chapter 4
Does this work ?
Offline ?
Online ?Results

42. Preliminary results (Offline)
Feature Weight
Proximity 13
bm25 8.7
Freshness 4.6
Price 0
Title Length -4.3
Description Length -7.3
+14% nDCG

43. Preliminary results (Online)
Feature Weight
Proximity 13
bm25 8.7
Freshness 4.6
Price 0
Title Length -4.3
Description Length -7.3
+14% nDCG
+8%
Replies/DAU

44. 44
Final results (Offline)
Feature Weight
Item Replies Received - 30 days 21.6
Preference for Cars - 30 days 15
Proximity 9.1
bm25 8.2
Preference for Car Parts - 30 days 8.2
Freshness 5.7
+71% nDCG
Item Performance
Buyer preference
Basic features

45. 45
Final results (Online)
Feature Weight
Item Replies Received - 30 days 21.6
Preference for Cars - 30 days 15
Proximity 9.1
bm25 8.2
Preference for Car Parts - 30 days 8.2
Freshness 5.7
+71% nDCG
Coming soon...
Item Performance
Buyer preference
Basic features

46. 46
The end Thank you
Any questions ?