Emerce GAUC 2020 - Lars Harmsen - Beerwulf - Brewing a culture of data-driven decisions

2. Brewing a culture of
data-driven decisions
Lars Harmsen
Product Owner Experimentation
lars.harmsen@beerwulf.com

3. About Beerwulf
Disruption2: Craft x Digital
x
1955
1957
1959
1961
1963
1965
1967
1969
1971
1973
1975
1977
1979
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
2013
2015
September 2015
4,000+ active breweries
according to the
Brewers Association
1987
H.R. 1337 signed by
Jimmy Carter
legalizes homebrewing
1995
Boston Beer Co.
IPO (Sam Adams)
2005
The last annual drop
in operating breweries
was a decade ago
Publicadmin.
Chemicals
Agriculture
Energy
Construction
Education
Food
Manufacturing
Healthcare
Professionalsvcs
Home&Garden
Insurance
Banking
Logistics
FMCG
Automotive
Mobility
Fashion
Consumer electronics
Media
Travel
Stateofdisruption
Point of disruption journey

4. About Beerwulf
Beerwulf is set-up as a Corporate Venture

5. About Beerwulf
Bringing brewers and drinkers together, online

6. What’s the best thing, the biggest
advantage, of being an e-
commerce business?

7. What’s the best thing, the biggest
advantage, of being an e-
commerce business?

8. Why are these companies so successful?

9. Why are these companies so successful?

10. How can we create a culture
where all decisions are driven by
data?

11. Data-driven decision making
Data-driven decisions across all teams & 75 individuals

14. • The data is not accessible
• It takes too much time
• It doesn’t tell me what I want
• I am not sure what I’m looking at
• I don’t think it is possible to get the data I want
• I don’t need data
• Not aware of potential damage if wrong decision is made
• And many, many more…
Data-driven decision making
Why we weren’t making data-driven decisions
“
”

15. Data-driven decision making
Game plan – The ‘Simple’ Version
1 42 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform

16. How do we make data accessible
and insightful?

17. Step 1 - Make data easily accessible & insightful
Start with the basics
2017
2017
2017

18. Step 1 - Make data easily accessible & insightful
Start with the basics
Why
What & Where
2017
2017
2017

19. Step 1 - Make data easily accessible & insightful
Your tooling landscape evolves over time
Why
What & Where
202020202019
2019
2019
2018 2017

20. Step 1 - Make data easily accessible & insightful
Your tooling landscape evolves over time
2020
Easy to use &
easy to get insights
Hard to use &
hard to get insights
20202019
2019
2019
2018 2017

21. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Accessible data

22. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Rainbow
effect
Accessible data

23. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
Rainbow
effect
Accessible data

25. Step 1 - Make data easily accessible & insightful
Don’t look “closlier”, look smarter!
Accessible

26. Step 1 - Make data easily accessible & insightful
Don’t look “closlier”, look smarter!
Insightful
Rainbow
effect
Valuable
element
Exposure
rate
Conversion
rate
Accessible

27. Step 1 - Make data easily accessible & insightful
Money makes the world go ‘round
Revenue per click

28. Test
variant
Control Experiment
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior

29. Test
variant
Control Experiment
Inconclusive
- No increase in subscriptions
- No impact on conversion rate
- Higher exit rate
- More sessions per user
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
N = 145.889

30. Step 1 - Make data easily accessible & insightful
Often, the data doesn’t really tell you that much…
- Is this a high click rate?

31. Step 1 - Make data easily accessible & insightful
Data becomes insightful in the right context
Event action % of sessions
Click color selector 7,32%
Click ‘Explore’ 3,20%
Click ‘Add to cart’ 2,31%

32. Step 1 - Make data easily accessible & insightful
Combining insightful data with insightful data

33. Control Experiment
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
=

34. Control Experiment
Step 1 - Make data easily accessible & insightful
Experiment to see the impact on behavior
+ 3,3%
Add to cart rate
=
Winner!
N = 103.112

35. Having insightful data doesn’t
mean you will always win

36. Having insightful data doesn’t
mean you will always win
But you sure as hell
will win more often

37. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 42 3
Make data easily
accessible & insightful
Coach, teach,
train, & challenge
to use data
Everyone used data
to make decisions
Beerwulf becomes
#1 beer platform

39. insights help
use data, give
insights
Lars’ cheesy proverb

40. Step 2 - Coach, teach, train, & challenge
What we tell people to look for in data

41. Use data to find
the best solution
for the user
Step 2 - Coach, teach, train, & challenge
What we tell people to look for in data

42. Step 2 - Coach, teach, train, & challenge
How we teach to find the best solutions
Workshops & training
Self study Explain and teach
Share examples (and the data behind it)
Learningbydoing

43. Workshops & training
Contentsquare University Explain and teach
Share examples (and the data behind it)
Learningbydoing
Step 2 - Coach, teach, train, & challenge
How we teach to find the best solutions
Self study

44. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..

45. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..

46. Step 2 - Coach, teach, train, & challenge
How to find better solutions for the user..
• Rage click rate of 2,35% (3 clicks in <2 seconds)
• Scroll is never ending
• Only 1 or 2 additional images in the scroll
• We are hiding the images from our users

47. Control Experiment
Step 2 - Coach, teach, train, & challenge
Experiment to see the impact on behavior
=

48. Control Experiment
Step 2 - Coach, teach, train, & challenge
Experiment to see the impact on behavior
=
Inconclusive
- No impact on add to cart &
conversion rate
- More thumbnail images viewed
per visitor
- Higher time on page
N = 93.410

49. Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time
Journey: Step 1  Step 2  Step 1

50. Control Experiment
Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time

51. Decrease of 122% in Step 1 > 2 > 1 behavior
Step 2 - Coach, teach, train, & challenge
Providing the right information at the right time
Journey Group % Δ
Step 1 > Step 2 > Step 1
Control 6,34%
-122%
Variant 2,86%
Control Experiment
+ 4,02%
conversion rate
N = 64.603

52. Step 2 - Coach, teach, train, & challenge
What we tell people to be cautious ofUse data to find
the wrong solution
for the user

53. Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”
Control Experiment

54. Control Experiment
Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”

55. Control Experiment
The use of the word blog
decreased clicks on the menu
item by 51%
& decreased conversion rate by
15% for those who clicked
Step 2 - Coach, teach, train, & challenge
“We think that blog is clearer and better!”
N = 47.199

56. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 4✓ 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform

57. Data-driven decision making
Game plan – The ‘Simple’ Version
✓ 4✓ 3
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform

58. Step 3 - Everyone uses data to make decisions
Understanding behavior where it matters most

59. Step 3 - Everyone uses data to make decisions
Understanding behavior where it matters most

60. Step 3 - Everyone uses data to make decisions
Understanding behavior
Step 1

61. Step 3 - Everyone uses data to make decisions
Understanding behavior
Step 1 Step 2 Step 3

62. Step 3 - Everyone uses data to make decisions
Understanding behavior

63. Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Control Experiment

64. Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Page Group Pageviews Δ
Step 1
Control 5,01%
-156%
Variant 1,96%
Step 2
Control 2,15%
-357%
Variant 0,47%
Step 3
Control 3,32%
-121%
Variant 1,50%
Control Experiment
View homepage directly after checkout page…

65. Winner
• Effect on behavior visible in all steps
• Decreased ‘Checkout > Homepage’
behavior by 169% (from 10,5% to 3,9%)
• Conversion rate increased by 0,75% (from
69,5% to 70,1%)
Step 3 - Everyone uses data to make decisions
Experiment to see the impact on behavior
Control Experiment
+ 0,75%
Conversion rate
N = 84.764

66. Step 3 - Everyone uses data to make decisions
Data helps you to solve your business challenges

67. Step 3 - Everyone uses data to make decisions
When it all comes together
Category
manager
UX
analyst
UX designer Experiment
lead
Company
wide
“14,6% of users
order full boxes
of one beer”
“12,8% of users
click add to cart
several times”
“15% of users
wants to order
in bulk”
“2 experiments
showed positive
impact on CTA”
Free delivery on
full box, saves
time and money

68. Step 3 - Everyone uses data to make decisions
Experiment with different solutions

69. Orders
(96,5% probability)
+5,72%
9,23% ➝ 9,75%
Add to cart
(99,2% probability)
+5,88%
30,83% ➝ 32,65%
Avg. order
quantity
+ 6,4%
2,04 ➝ 2,17
Add box to
cart button
7,25%
Revenue
per visitor
+ 6,7%
↧
Avg. order
value
+ 2,6%
Step 3 - Everyone uses data to make decisions
Improve for both Beerwulf and the user
Cost of running
the test
€ 1890
ROI on test
33656%

70. Will everyone make
the right decisions?

71. NO, of course not!

72. • Have an amazing team of talented people
• Have insightful data guiding us
• Know how to make the right decisions
• Know how to prevent the wrong decisions
• Learn from our mistakes and take action
• Focus on making our customers happy!
Step 3 - Everyone uses data to make decisions
Doing the right thing

74. ✓ ✓✓ ✓
Make data easily
accessible & insightful
Coach, teach, train,
& challenge
to use data
Everyone uses data
to make decisions
Beerwulf becomes
#1 beer platform
Step 4
Beerwulf will become THE #1 beer platform

75. Cheers!
Lars Harmsen
Product Owner Experimentation
lars.harmsen@beerwulf.com

76. Bedankt voor je aandacht!
Ga naar de Emerce GAUC-event app
of naar glsr.live/GAUC2020
en laat weten wat je van de sessie vond