Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Shallow learning @ Criteo
Nicolas Le Roux
Scientific Program Manager – R&D
I N T R O D U C T I O N
Copyright © 2013 Criteo
WHAT DOES CRITEO DO?
3
• We buy advertising space on websites
• We display ads for our partners
• We get paid if the user ...
HOW DO WE BUY ADVERTISING SPACE?
4
• Real-time bidding (RTB): it is an auction
• Second-price
• Optimal strategy: bid the ...
WHAT TO DO ONCE WE WIN THE DISPLAY?
5
• Choose the best products
• Choose the color, the font, the layout…
• Generate the ...
WHERE IS THE ML AT CRITEO?
6
• Click prediction model
• Product recommendation
• Testing, …
Copyright © 2013 Criteo
WHERE IS THE ML AT CRITEO?
7
• Click prediction model
• Product recommendation
• Testing, …
Copyright © 2013 Criteo
INFRASTRUCTURE AT CRITEO
8
• 5 data centers on 3 continents
• More than 3 500 servers
• More than 30B HTTP requests per da...
C L I C K P R E D I C T I O N
M O D E L
Copyright © 2013 Criteo
CTR PREDICTION MODEL
10Copyright © 2013 Criteo
CTR PREDICTION MODEL
11Copyright © 2013 Criteo
CTR PREDICTION MODEL
12Copyright © 2013 Criteo
CTR PREDICTION MODEL
13Copyright © 2013 Criteo
CTR PREDICTION MODEL
14Copyright © 2013 Criteo
15-SECOND ML CRASH COURSE
15Copyright © 2013 Criteo
NEW HIRE REACTION
16Copyright © 2013 Criteo
NEW HIRE REACTION
17Copyright © 2013 Criteo
NEW HIRE REACTION
18Copyright © 2013 Criteo
NEW HIRE REACTION
19Copyright © 2013 Criteo
20Copyright © 2013 Criteo
21Copyright © 2013 Criteo
22
• Merchant ID is too raw
• We can learn fancy features for each merchant:
demographics, size, ...
• Hard to not penaliz...
OPTIMIZING THE MODEL
23Copyright © 2013 Criteo
DEALING WITH MANY VARIABLES
24Copyright © 2013 Criteo
DEALING WITH MANY VARIABLES
25Copyright © 2013 Criteo
OTHER CHALLENGES
26
• How long do you wait for sales after a click?
• How much in the past do you go to learn the model?
•...
P R O D U C T
R E C O M M E N D A T I O N
Copyright © 2013 Criteo
THE CHALLENGES OF PRODUCT RECOMMENDATION
28Copyright © 2013 Criteo
THE CHALLENGES OF PRODUCT RECOMMENDATION
29Copyright © 2013 Criteo
NEW HIRE REACTION
30
• « You should use [his PhD topic], it’s awesome! »
Copyright © 2013 Criteo
NEW HIRE REACTION
31
• « You should use [his PhD topic], it’s awesome! »
• « It just takes a week to train. »
Copyright © ...
NEW HIRE REACTION
32
• « You should use [his PhD topic], it’s awesome! »
• « It just takes a week to train. »
• « In real-...
MAJOR HURDLES
33
• Products come and go
• There might be a sale (Black Friday)
• The « panties syndrome »
• Complementary ...
OTHER CHALLENGES
34
• Multiple products in a banner
• Interaction between products and layout
• Different timeframes for d...
A FIRST RECIPE FOR SUCCESS
35
• There are many sources of success/failure
• It is often suboptimal to focus on one
• The f...
CONCLUSION
36
• Prediction is at the core of our business
• Huge engineering constraints
• Different bottlenecks than in a...
Nicolas: n.leroux@criteo.com
Lisa: l.sarny@criteo.com
Clem: c.daines@criteo.com
QUESTIONS ?
Upcoming SlideShare
Loading in …5
×

Machine learning at Criteo - Paris Datageeks

18,043 views

Published on

Machine learning at Criteo - How to scale machine learning algorithms to perform real-time prediction of CTR and recommendation of products.

Published in: Technology
  • Be the first to comment

Machine learning at Criteo - Paris Datageeks

  1. 1. Shallow learning @ Criteo Nicolas Le Roux Scientific Program Manager – R&D
  2. 2. I N T R O D U C T I O N Copyright © 2013 Criteo
  3. 3. WHAT DOES CRITEO DO? 3 • We buy advertising space on websites • We display ads for our partners • We get paid if the user clicks on the ad Copyright © 2013 Criteo
  4. 4. HOW DO WE BUY ADVERTISING SPACE? 4 • Real-time bidding (RTB): it is an auction • Second-price • Optimal strategy: bid the expected gain • Expected gain = CPC * CTR Copyright © 2013 Criteo
  5. 5. WHAT TO DO ONCE WE WIN THE DISPLAY? 5 • Choose the best products • Choose the color, the font, the layout… • Generate the banner Copyright © 2013 Criteo
  6. 6. WHERE IS THE ML AT CRITEO? 6 • Click prediction model • Product recommendation • Testing, … Copyright © 2013 Criteo
  7. 7. WHERE IS THE ML AT CRITEO? 7 • Click prediction model • Product recommendation • Testing, … Copyright © 2013 Criteo
  8. 8. INFRASTRUCTURE AT CRITEO 8 • 5 data centers on 3 continents • More than 3 500 servers • More than 30B HTTP requests per day • More than 2B banners displayed per day • Peak traffic: 500K HTTP requests per second Copyright © 2013 Criteo
  9. 9. C L I C K P R E D I C T I O N M O D E L Copyright © 2013 Criteo
  10. 10. CTR PREDICTION MODEL 10Copyright © 2013 Criteo
  11. 11. CTR PREDICTION MODEL 11Copyright © 2013 Criteo
  12. 12. CTR PREDICTION MODEL 12Copyright © 2013 Criteo
  13. 13. CTR PREDICTION MODEL 13Copyright © 2013 Criteo
  14. 14. CTR PREDICTION MODEL 14Copyright © 2013 Criteo
  15. 15. 15-SECOND ML CRASH COURSE 15Copyright © 2013 Criteo
  16. 16. NEW HIRE REACTION 16Copyright © 2013 Criteo
  17. 17. NEW HIRE REACTION 17Copyright © 2013 Criteo
  18. 18. NEW HIRE REACTION 18Copyright © 2013 Criteo
  19. 19. NEW HIRE REACTION 19Copyright © 2013 Criteo
  20. 20. 20Copyright © 2013 Criteo
  21. 21. 21Copyright © 2013 Criteo
  22. 22. 22 • Merchant ID is too raw • We can learn fancy features for each merchant: demographics, size, ... • Hard to not penalize large merchants Copyright © 2013 Criteo
  23. 23. OPTIMIZING THE MODEL 23Copyright © 2013 Criteo
  24. 24. DEALING WITH MANY VARIABLES 24Copyright © 2013 Criteo
  25. 25. DEALING WITH MANY VARIABLES 25Copyright © 2013 Criteo
  26. 26. OTHER CHALLENGES 26 • How long do you wait for sales after a click? • How much in the past do you go to learn the model? • Do you value more long term or short term variables? • How to choose the best variables? Copyright © 2013 Criteo
  27. 27. P R O D U C T R E C O M M E N D A T I O N Copyright © 2013 Criteo
  28. 28. THE CHALLENGES OF PRODUCT RECOMMENDATION 28Copyright © 2013 Criteo
  29. 29. THE CHALLENGES OF PRODUCT RECOMMENDATION 29Copyright © 2013 Criteo
  30. 30. NEW HIRE REACTION 30 • « You should use [his PhD topic], it’s awesome! » Copyright © 2013 Criteo
  31. 31. NEW HIRE REACTION 31 • « You should use [his PhD topic], it’s awesome! » • « It just takes a week to train. » Copyright © 2013 Criteo
  32. 32. NEW HIRE REACTION 32 • « You should use [his PhD topic], it’s awesome! » • « It just takes a week to train. » • « In real-time, you just have to score every product. » Copyright © 2013 Criteo
  33. 33. MAJOR HURDLES 33 • Products come and go • There might be a sale (Black Friday) • The « panties syndrome » • Complementary vs. similar products Copyright © 2013 Criteo
  34. 34. OTHER CHALLENGES 34 • Multiple products in a banner • Interaction between products and layout • Different timeframes for different products Copyright © 2013 Criteo
  35. 35. A FIRST RECIPE FOR SUCCESS 35 • There are many sources of success/failure • It is often suboptimal to focus on one • The first step to address each source is often manual Copyright © 2013 Criteo
  36. 36. CONCLUSION 36 • Prediction is at the core of our business • Huge engineering constraints • Different bottlenecks than in academia • Build from the ground up, not the other way around Copyright © 2013 Criteo
  37. 37. Nicolas: n.leroux@criteo.com Lisa: l.sarny@criteo.com Clem: c.daines@criteo.com QUESTIONS ?

×