Presentation from the Data Products Meetup 7 - "How did I learn how not to run a Data Science project" by Florian Bonnet.

1. Florian Bonnet - HelloFresh
August 20th 2019

2. 1. Starters - HelloFresh – who we are/what we do?
2. Entrée – Referral Marketing - A channel prone to optimization
3. Main Dish – Back to the Future – Dissecting a Failure
4. Dessert – Summary – What did I learn

3. HelloFresh
Who we are? What we do?

4. Spicies
... ... ... ... ...
Oct.
11
Oct.
18
Oct.
25
Nov.
1
Nov.
8
Nov.
15
Freshly sourced ingredients
in the right amount
Delivered to your home
on a weekly basis
Subscription model but you can pause anytime

5. TYPICAL FOOD SUPPLY CHAIN
10 days and 5 parties
OUR SUPPLY CHAIN REVOLUTION
3 days and 3 parties
Consumer Home Day 10
23% waste
Producer Day
1
Wholesaler Day
2
2% waste
Warehouse
Day 4
3% waste
Supermarket
Day 6
11% waste
 STARTING WITH
CONSUMER
 FASTER AND
FRESHER
 NO FOOD
WASTE
Better for Customers
More Margin for
HelloFresh
Consumer
Home
Day 3
Producer
Day 1
Consume
r Day 2
Source: Company information; United States Department of Agriculture; Canaccord Genuity estimates
Note: 5% “Farm to Retail” waste data split to wholesale and warehouse.

6. 1. Source: Recent company filings
2.5m
active users
Best
Meal
Delivery
Service
#1
Industrial
Excellenc
e Award
DE&EU

7. Referral Marketing
An Important marketing channel prone to
optimization

8. Growth of Customer Base
Paid Marketing
Customer Base Boxes Shipped
Marketing Manager
CCV/CAC = Low
Unhappy Manager

9. Growth of Customer Base
Paid Marketing
Customer Base Boxes Shipped
Marketing Manager
What‘s up!
Check out
HelloFresh it is
super cool!
Good Thx :)
Awesome!!!
Customer
Friend
More Customers
High
+
Happy Manager

10. Customer Friend
Free
... ...

11. Customer Friend
Free
... ...
Profitability threshold
1
#boxes
CCV/CAC

12. Customer Friend
Free
... ...
Profitability threshold
1
#boxes
CCV/CAC

13. Customer Friend
Free
... ...
Profitability threshold
1
#boxes
CCV/CAC

14. Alles gut
Customer Friend
Free
... ...
Profitability threshold
1
#boxes
CCV/CAC

15. Customer Friend
Free
Free
Free
Free
Free
Free
Free
Find a way for customer to order more
Identify bad referrer
upfront and prevent
them from referring
1
2

16. Friend
checkout

17. Friend
checkout
Surprised 2nd
box discount at
checkout

18. Friend
checkout
Surprised 2nd
box discount at
checkout Discount $10 $20 $30 $40
Uplift 52W LTV -2.3% +2.1% +1.4% +8.2%
Phase1
Randomized AB Test
 POC
 Data Gathering

19. Friend
checkout
Surprised 2nd
box discount at
checkout Discount $10 $20 $30 $40
Uplift 52W LTV -2.3% +2.1% +1.4% +8.2%
Phase1Phase2
Randomized AB Test
 POC
 Data Gathering
Data Science Modeling
 Finding best discount
for each friend
 Apply in production

20. Friend
checkout
Surprised 2nd
box discount at
checkout Discount $10 $20 $30 $40
Uplift 52W LTV -2.3% +2.1% +1.4% +8.2%
Phase1Phase2
Randomized AB Test
 POC
 Data Gathering
Data Science Modeling
 Finding best discount
for each friend
 Apply in production
Customer
Friends
Behavioral
Data
Revenue/costs
Data

21. Friend
checkout
Surprised 2nd
box discount at
checkout Discount $10 $20 $30 $40
Uplift 52W LTV -2.3% +2.1% +1.4% +8.2%
Phase1Phase2
Randomized AB Test
 POC
 Data Gathering
Data Science Modeling
 Finding best discount
for each friend
 Apply in production
Customer
Friends
Behavioral
Data
Revenue/costs
Data
Customer
Free
time
#1
Free
#2
Free
#3
Free
#4
Try it!Really
good!
Changed
my life
Best
Recipes!

22. Friend
checkout
Surprised 2nd
box discount at
checkout Discount $10 $20 $30 $40
Uplift 52W LTV -2.3% +2.1% +1.4% +8.2%
Phase1Phase2
Randomized AB Test
 POC
 Data Gathering
Data Science Modeling
 Finding best discount
for each friend
 Apply in production
Customer
Friends
Behavioral
Data
Revenue/costs
Data
Customer
Free
time
#1
Free
#2
Free
#3
Free
#4
Try it!Really
good!
Changed
my life
Best
Recipes!

23. Friend
checkout
Surprised 2nd
box discount at
checkout
Phase1Phase2
Randomized AB Test
(Q1)
 POC
 Data Gathering
Data Science Modeling
 Finding best discount
for each friend
 Apply in production
Customer
Friends
Behavioral
Data
Revenue/costs
Data
Customer
Free
time
Friend#1
Free
Friend#2
Free
Friend#3
Free
Firend#4
Try it!Really
good!
Changed
my life
Best
Recipes!
Which discount for which friend ot maximize cohort ROI?
CustomerCustomerCustomerCustomer

24. 1
2
3
Display the right discount at checkout
Store the discount on Friend‘s account
Track Redemptions

25.  1 year of work
 Multiple fails in execution and process
 Lots of rework
 No tangible results
 Frustrated team
 Frustrated stakeholders
$0 in the bank
No knowledge
aquired

26. Back to the Future
Dissecting a failure

27. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2

28. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
 Too confident in results
 No deep dive in data and User Experience to see if results
reflected the experience proposed:
‒ Not all test users exposed to the popup
‒ Unbalanced test and control groups

29. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
 Too confident
 No deep dive
 Requirements not clear enough
‒ Experience not well thought through leading to several
changes in scope
 No clear ownership / project manager
‒ Each function thinking the other ones do the necessary
 holes in the net resulting in last minute rework or
missing pieces
 No proper tracking
‒ Experience and key elements not tracked properly 
dirty data

30. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
 Too confident
 No deep dive
 Requirements not clear enough
 No clear ownership / project
manager
 No proper tracking
 Confusions Data Science / Data Analysts slow down
process
 Analysis not prepared enough ahead of time  loss of time
 No clear expectations as of the results between DS, PO,
stakeholders  doubt spreading around

31. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
 Too confident
 No deep dive
 Requirements not clear enough
 No clear ownership / project
manager
 No proper tracking
 Confusions Data
Science / Data Analyst
 Analysis not prepared
enough ahead of time
 No clear expectations
as of the results
 Creation of heavy machinery to solve all the issues 
huge amount of resources mobilized & NSMMVP
 Changes in requirements from Data Science to make it a
success  frustration, complexification, lack of trust, lost
time
 No clear expectations on results and next steps

32. A/B test
And decision
Phase 1 of test Analysis and Learnings Rework & Phase 2 Test 2
 Too confident
 No deep dive
 Requirements not clear enough
 No clear ownership / project
manager
 No proper tracking
 Confusions Data
Science / Data Analyst
 Analysis not prepared
enough ahead of time
 No clear expectations
as of the results
 Creation of heavy machinery to solve
all the issues
 Changes in requirements from Data
Science to make it a success
 No clear expectations on results and
next steps

33. Summary
What did I learn?

34. BE FE QA
PO DS PAML
Involve everyone from day one
BE
PO
PA DS QA ML
1

35. Clear PMO (does not have to be the PO)
Accountable for success
2

36. Customer
Friends
Behavioral
Data
Revenue/costs
Data
DSPA
Actionable
A/B Test
Quick
iteration to
validate model
and process
Simple Model
first vs. Nobel
prize
machinery
3

37. A/B
test
QA, QA, QA, QA a.k.a
monitor the experience
continuously
4

38. Don’t be afraid to cut your
losses
5

39. DSPA
Actionable A/B
Test
Quick iteration
to validate
model and
process
Simple Model
first vs. Nobel
prizemachinery
Clear PMO accountable for success
Involve everyone from day one
BE FE QA
PO DS PAML
1
2
3
A/B
test
4
5
Agile all the way
Don’t be afraid to cut your losses
QA,QA,QA,QA