Shallow learning @ Criteo
Nicolas Le Roux
Scientific Program Manager – R&D
I N T R O D U C T I O N
Copyright © 2013 Criteo
WHAT DOES CRITEO DO?
3
• We buy advertising space on websites
• We display ads for our partners
• We get paid if the user ...
HOW DO WE BUY ADVERTISING SPACE?
4
• Real-time bidding (RTB): it is an auction
• Second-price
• Optimal strategy: bid the ...
WHAT TO DO ONCE WE WIN THE DISPLAY?
5
• Choose the best products
• Choose the color, the font, the layout…
• Generate the ...
WHERE IS THE ML AT CRITEO?
6
• Click prediction model
• Product recommendation
• Testing, …
Copyright © 2013 Criteo
WHERE IS THE ML AT CRITEO?
7
• Click prediction model
• Product recommendation
• Testing, …
Copyright © 2013 Criteo
INFRASTRUCTURE AT CRITEO
8
• 5 data centers on 3 continents
• More than 3 500 servers
• More than 30B HTTP requests per da...
C L I C K P R E D I C T I O N
M O D E L
Copyright © 2013 Criteo
CTR PREDICTION MODEL
10Copyright © 2013 Criteo
CTR PREDICTION MODEL
11Copyright © 2013 Criteo
CTR PREDICTION MODEL
12Copyright © 2013 Criteo
CTR PREDICTION MODEL
13Copyright © 2013 Criteo
CTR PREDICTION MODEL
14Copyright © 2013 Criteo
15-SECOND ML CRASH COURSE
15Copyright © 2013 Criteo
NEW HIRE REACTION
16Copyright © 2013 Criteo
NEW HIRE REACTION
17Copyright © 2013 Criteo
NEW HIRE REACTION
18Copyright © 2013 Criteo
NEW HIRE REACTION
19Copyright © 2013 Criteo
20Copyright © 2013 Criteo
21Copyright © 2013 Criteo
22
• Merchant ID is too raw
• We can learn fancy features for each merchant:
demographics, size, ...
• Hard to not penaliz...
OPTIMIZING THE MODEL
23Copyright © 2013 Criteo
DEALING WITH MANY VARIABLES
24Copyright © 2013 Criteo
DEALING WITH MANY VARIABLES
25Copyright © 2013 Criteo
OTHER CHALLENGES
26
• How long do you wait for sales after a click?
• How much in the past do you go to learn the model?
•...
P R O D U C T
R E C O M M E N D A T I O N
Copyright © 2013 Criteo
THE CHALLENGES OF PRODUCT RECOMMENDATION
28Copyright © 2013 Criteo
THE CHALLENGES OF PRODUCT RECOMMENDATION
29Copyright © 2013 Criteo
NEW HIRE REACTION
30
• « You should use [his PhD topic], it’s awesome! »
Copyright © 2013 Criteo
NEW HIRE REACTION
31
• « You should use [his PhD topic], it’s awesome! »
• « It just takes a week to train. »
Copyright © ...
NEW HIRE REACTION
32
• « You should use [his PhD topic], it’s awesome! »
• « It just takes a week to train. »
• « In real-...
MAJOR HURDLES
33
• Products come and go
• There might be a sale (Black Friday)
• The « panties syndrome »
• Complementary ...
OTHER CHALLENGES
34
• Multiple products in a banner
• Interaction between products and layout
• Different timeframes for d...
A FIRST RECIPE FOR SUCCESS
35
• There are many sources of success/failure
• It is often suboptimal to focus on one
• The f...
CONCLUSION
36
• Prediction is at the core of our business
• Huge engineering constraints
• Different bottlenecks than in a...
Nicolas: n.leroux@criteo.com
Lisa: l.sarny@criteo.com
Clem: c.daines@criteo.com
QUESTIONS ?
Upcoming SlideShare
Loading in …5
×

Machine learning at Criteo - Paris Datageeks

17,566 views

Published on

Machine learning at Criteo - How to scale machine learning algorithms to perform real-time prediction of CTR and recommendation of products.

Published in: Technology
  • Be the first to comment

Machine learning at Criteo - Paris Datageeks

  1. 1. Shallow learning @ Criteo Nicolas Le Roux Scientific Program Manager – R&D
  2. 2. I N T R O D U C T I O N Copyright © 2013 Criteo
  3. 3. WHAT DOES CRITEO DO? 3 • We buy advertising space on websites • We display ads for our partners • We get paid if the user clicks on the ad Copyright © 2013 Criteo
  4. 4. HOW DO WE BUY ADVERTISING SPACE? 4 • Real-time bidding (RTB): it is an auction • Second-price • Optimal strategy: bid the expected gain • Expected gain = CPC * CTR Copyright © 2013 Criteo
  5. 5. WHAT TO DO ONCE WE WIN THE DISPLAY? 5 • Choose the best products • Choose the color, the font, the layout… • Generate the banner Copyright © 2013 Criteo
  6. 6. WHERE IS THE ML AT CRITEO? 6 • Click prediction model • Product recommendation • Testing, … Copyright © 2013 Criteo
  7. 7. WHERE IS THE ML AT CRITEO? 7 • Click prediction model • Product recommendation • Testing, … Copyright © 2013 Criteo
  8. 8. INFRASTRUCTURE AT CRITEO 8 • 5 data centers on 3 continents • More than 3 500 servers • More than 30B HTTP requests per day • More than 2B banners displayed per day • Peak traffic: 500K HTTP requests per second Copyright © 2013 Criteo
  9. 9. C L I C K P R E D I C T I O N M O D E L Copyright © 2013 Criteo
  10. 10. CTR PREDICTION MODEL 10Copyright © 2013 Criteo
  11. 11. CTR PREDICTION MODEL 11Copyright © 2013 Criteo
  12. 12. CTR PREDICTION MODEL 12Copyright © 2013 Criteo
  13. 13. CTR PREDICTION MODEL 13Copyright © 2013 Criteo
  14. 14. CTR PREDICTION MODEL 14Copyright © 2013 Criteo
  15. 15. 15-SECOND ML CRASH COURSE 15Copyright © 2013 Criteo
  16. 16. NEW HIRE REACTION 16Copyright © 2013 Criteo
  17. 17. NEW HIRE REACTION 17Copyright © 2013 Criteo
  18. 18. NEW HIRE REACTION 18Copyright © 2013 Criteo
  19. 19. NEW HIRE REACTION 19Copyright © 2013 Criteo
  20. 20. 20Copyright © 2013 Criteo
  21. 21. 21Copyright © 2013 Criteo
  22. 22. 22 • Merchant ID is too raw • We can learn fancy features for each merchant: demographics, size, ... • Hard to not penalize large merchants Copyright © 2013 Criteo
  23. 23. OPTIMIZING THE MODEL 23Copyright © 2013 Criteo
  24. 24. DEALING WITH MANY VARIABLES 24Copyright © 2013 Criteo
  25. 25. DEALING WITH MANY VARIABLES 25Copyright © 2013 Criteo
  26. 26. OTHER CHALLENGES 26 • How long do you wait for sales after a click? • How much in the past do you go to learn the model? • Do you value more long term or short term variables? • How to choose the best variables? Copyright © 2013 Criteo
  27. 27. P R O D U C T R E C O M M E N D A T I O N Copyright © 2013 Criteo
  28. 28. THE CHALLENGES OF PRODUCT RECOMMENDATION 28Copyright © 2013 Criteo
  29. 29. THE CHALLENGES OF PRODUCT RECOMMENDATION 29Copyright © 2013 Criteo
  30. 30. NEW HIRE REACTION 30 • « You should use [his PhD topic], it’s awesome! » Copyright © 2013 Criteo
  31. 31. NEW HIRE REACTION 31 • « You should use [his PhD topic], it’s awesome! » • « It just takes a week to train. » Copyright © 2013 Criteo
  32. 32. NEW HIRE REACTION 32 • « You should use [his PhD topic], it’s awesome! » • « It just takes a week to train. » • « In real-time, you just have to score every product. » Copyright © 2013 Criteo
  33. 33. MAJOR HURDLES 33 • Products come and go • There might be a sale (Black Friday) • The « panties syndrome » • Complementary vs. similar products Copyright © 2013 Criteo
  34. 34. OTHER CHALLENGES 34 • Multiple products in a banner • Interaction between products and layout • Different timeframes for different products Copyright © 2013 Criteo
  35. 35. A FIRST RECIPE FOR SUCCESS 35 • There are many sources of success/failure • It is often suboptimal to focus on one • The first step to address each source is often manual Copyright © 2013 Criteo
  36. 36. CONCLUSION 36 • Prediction is at the core of our business • Huge engineering constraints • Different bottlenecks than in academia • Build from the ground up, not the other way around Copyright © 2013 Criteo
  37. 37. Nicolas: n.leroux@criteo.com Lisa: l.sarny@criteo.com Clem: c.daines@criteo.com QUESTIONS ?

×