This document summarizes an approach to the CIKM Cup 2016 Track 1 competition on cross-device linking. The goal was to link devices to restore a graph of user-device connections using click log data. The approach involved first using information retrieval techniques to optimize recall and select candidate device pairs, then using machine learning to optimize precision and rank true pairs higher. Features were created to profile devices and measure similarities. An XGBoost model was trained on the candidate pairs and evaluated through cross-validation. The top K pairs by probability were selected to balance precision and recall. Post-competition improvements included stacking and Markov clustering.

1. CIKM CUP 2016: Track 1
Cross-Device Linking
Alexey Grigorev
Berlin Machine Learning
2016.12.05

2. About Me
Software Developer BI Masters @ TU Berlin Data Scientist

3. CIKM Cup 2016: Cross-Device Linking
user advertisements ad providers

4. Goal: Restore the Graph
?
training data:
know the links
new unseen devices:
no links

5. Data
240k train “users” (devices), 100k test users
500k train device-device pairs, 215k test pairs
Denormalized: 2.5 Gb click logs + 1 Gb URLs & titles
67m clicks in total, 197 clicks per user on average

6. How to Approach?
● Machine Learning?
● First, optimize Recall
○ IR, unsupervised
○ Select “candidate” device-device pairs
○ Build a design matrix
● Then, optimize Precision
○ ML, supervised
○ Push the true pairs up the list
● Select top K pairs s.t. F1 is max
Train Test

7. Optimizing Recall
● Recall: fraction of all true device pairs we discover
● Information Retrieval problem!
● For each device need to find the most similar ones
● Device == Document with
○ Tokens from all visited URL + Tokens from all titles
○ Put them together into a one single document
● Then use standard IR methods like TF-IDF

8. Optimizing Recall
most similar
least similar
IR
ES MLT query Top 70 candidatesDevice
(240k + 100k) * 70 = 24m
1
1
0

9. Optimizing Precision
1
1
0
0
0
1
● Now have high Recall, but low Precision
○ Recall: fraction of all positive pairs we discover
○ Precision: fraction of positive pairs within our results
● Use Supervised Machine Learning for improving it
Next steps:
● Create features for each device pair
● Train a ranking ML model
● Take top most reliable predictions

10. Features: Profiling
● Create a profile for each user
● Profile from sessions (30 minutes inactivity cut):
○ Session duration
○ Number of visits per session
○ Number of sessions with only one visit
○ Duration of breaks between and within sessions
○ Number of consecutive requests with a ≤ 1ms delay
○ Starts and ends of sessions
○ Number of unique domains per session
○ Similarity of domains/urls/titles within each session
○ For all features: min, mean, max and std

11. Features: Device-Device Similarities
● |profile1.feature - profile2.feature|
● TF-IDF similarity of
○ Domains
○ Titles
○ URLs
● LSA similarity of the same
● 54 features in total

12. Optimizing Precision: Ranking Model
1
1
0
1
0
1
0.90
0.87
0.2
0.3
0.7
Train 240k * 70 = 17m
100k-200k
Top K
Test 100k * 70 = 7m

13. Features: Importance
pair
features
profile
difference
features
XGB feature importance:
# times used in split

14. Cross-Validation
FOLD 1 FOLD 2vs
information leak!

15. Cross-Validation
● Split the graph into non-overlapping
regions
● For each region separately
○ Build ES index (i.e. apply filter)
○ Build a model
● Evaluation (AUC + F1):
○ Apply F1 model to F2 data
○ And vise versa
F1 F2
0.90
0.87
0.7
0.90
0.87
0.7

16. Evaluation
Public/private test split: 50/50
● During the competition:
○ Evaluation on 1st half of data
● After the competition: 2nd half
● P = 0.5 of real P
Test
normal F1
“real” evaluation
function

17. Choosing K
● Order the pairs by the probability
● For each K calculate P, R and F1
● Select best K such that F1 is max
● 8th position

18. Post-Competition
What did others do?
● Using several candidate selection methods
● Stacking with rank features (by D. Dremov)
● Markov Clustering (by I. Bendyna)

19. Rank Features
source: http://gh.mltrainings.ru/presentations/Dremov_CIKMCup2016_DCA.pdf slide 9
● Relative position of a node within a group
● Motivation: “local” within-group effect instead of global
● df_train.groupby('user_1')[feature].rank()

20. Stacking (post competition)
all features
XGBoost
ET
best features
XGBoost
rank features
8th → 5th position

21. Markov Clustering
source: http://gh.mltrainings.ru/presentations/Bendyna_CIKMCup2016_DCA.pdf
● take a connected component
● add loops
● put into a Markov Matrix M
○ also called “Stochastic Matrix”
○ values in cols sum up to 1
● calculate M ** n
○ ~ n Random Walk steps
● for each element M.v = M.v ** p
○ makes weak links weaker
● re-normalize and repeat
Animation http://micans.org/mcl/ani/mcl-animation.html

22. Links & Further Info
● Competition website: http://cikmcup.org/
● Competition platform: https://competitions.codalab.org/competitions/11171
● My solution: https://github.com/alexeygrigorev/cikm-cup-2016-cross-device
● Reports: http://cikmcup.org/workshop.html
Self-promotion:
● http://alexeygrigorev.com/
● contact@alexeygrigorev.com

23. Thank you. Questions?