1) Criteo is an advertising technology company that uses machine learning to target personalized ads to users across devices. 2) It has over 2000 employees, works with 16,000 publishers and 11,000 advertisers worldwide, and generated $11 billion for its clients in 2015. 3) Criteo's machine learning challenges include selecting the best products and ads to show users from huge catalogs in milliseconds, building a graph connecting billions of devices to identify users, and extensively testing new models and systems.

1. RecSys FR: 4th Session
6th October 2016
Etienne Sanson – manager R&D Engine

2. About Criteo
1

3. 3 | Copyright © 2016 Criteo
Our mission
TARGET THE
RIGHT USER
AT THE
RIGHT TIME
WITH THE RIGHT
MESSAGE

4. 4 | Copyright © 2016 Criteo
Key Figures
16 000
PUBLISHERS
90%
RETENTION RATE2
+130
COUNTRIES
LISTED ON THE
NASDAQ
SINCE
OCTOBER 2013
R&D REPRESENTS 21%
OF THE WORKFORCE
2000
EMPLOYEES
21
BILLIONS $3
11 000
ADVERTISERS
1.19 bn€1
31
OFFICES
1: REVENUE IN 2015
2: ANNUAL RATE 2015
3: $ OF TURNOVER GENERATED TO OUR CLIENTS - TURNOVER POST-CLICK WW FROM JANUARY TO DECEMBER
2015

5. 5 | Copyright © 2016 Criteo
Revenue Growth
2009 2010 2011 2012 2013
22M$
86M$
199M$
349M$
589M$
988M$
2014
1,3MM$
2015

6. 6 | Copyright © 2016 Criteo
GENERAL CONCEPT
Users visit an
advertiser’s website
1
Criteo identifies the users
(via cookies)
2
Users leave the advertiser’s website
& browse publisher on the Internet
3
Criteo identifies users on
these pages
(via cookie)
4
Criteo displays an advertising
banner, personalized for
each user
5
Click through directly
to the advertiser’s
page
6
@
Retargeting principles

7. 7 | Copyright © 2016 Criteo
Infrastructure Key Figures
Sunnyvale
2 PoP
500 kVA
1 559 Servers
New York
2 PoP
930 kVA
2 625 Servers
Hong Kong
2 PoP
472 kVA
2185 Servers
Paris
4 Pop
1 800 kVA
3 625 Servers
Amsterdam
2 PoP
+2 500 kVA
3 609 Servers
Tokyo
2 PoP
455 kVA
2 564 Servers
Shanghai
1 PoP
200 kVA
931 Servers
World Wide
15 PoP
6,8 MVA
17 098 Servers
> 55Gbps
+ 2.5M req/s
Hosting Global Partners :

8. About ML@Criteo
2

9. 9 | Copyright © 2016 Criteo
Our challenges – Product recommendation
• Select the best ~10 products to show to a user
• >1B users
• Product catalog contains ~1M items, up to 1B
• Time constraints: 20ms
• Combination of offline/online processing steps
• CF
• Product embeddings (word2vec -> prod2vec)
• CNN for image features
What products should
we recommend?

10. 10 | Copyright © 2016 Criteo
Our challenges - Bidding
How much should we
bid for this display?
What is the best
campaign to display?
My company
BUY! BUY! BUY!
BUY!
• Select the best campaign to display and evaluate its
value in a few ms
• Large scale regression models
• >1B daily displays (but few positive examples!)
• >1M parameters
• Distributed optimization (SGD, L-BFGS)
• Feature Engineering
• Transfer learning, FFM, Policy learning
• Marketplace, game theory, auction theory

11. 11 | Copyright © 2016 Criteo
Our challenges – X-device
• Build a huge graph (Billions of nodes/edges):
• Nodes = devices
• Edge = the 2 devices belong to the same user
• How to connect 2 devices?
• How to know the ground truth?
• How to keep it stable?
• At scale & taking care about privacy
Who is the user
behind the device?

12. 12 | Copyright © 2016 Criteo
Our challenges – Testing
• We test everything!
• Offline tests / AB Tests
• Infrastructure to perform large-scale tests
• >100K offline tests / year
• >1K AB Tests / year
• Dedicated teams
• Technical / Business Metrics
• Randomization
• Counterfactual evaluation

13. Thank you!
…and we’re hiring!