This document outlines the process and components for running rapid experiments to drive growth. It discusses establishing a growth culture, experiment methodology including goal setting, hypothesis testing, metrics, design, results, and learnings. Examples provided include optimizing features, channels, and online spend. The overall message is that an experimentation mindset and frequent testing of ideas is needed to guide continuous improvement and growth.
19. Our success at Amazon is a function of
how many experiments we do per year,
per month, per week, per day.
Being wrong might hurt you a bit..
But being slow will kill you.
20. Guiding Growth
A. WHAT IS A GROWTH CULTURE?
B. WHAT DOES YOUR TEAM LOOK LIKE?
C. MINDSET
22. >> CONDITIONS FOR A GROWTH CULTURE?
RULE #1: BUILD A CULTURE FOR GROWTH
BUY-IN OF TEAM & MANAGEMENT
TOLERANCE FOR INSECURITY
ACCESS TO DATA
FORTUNE FAVOURS THE BOLD
OPEN TO TOOLS & SYSTEMS
30. Guiding Growth
>> HIGH INVESTMENT (MANY RESOURCES)
>> RISK THAT TEAM DON’T SHARE EACH
OTHERS LEARNINGS
>> TUNNEL VS FUNNEL VISION
>> DON’T BE A BLOCKER FOR OTHER
VERTICALS
CON’S
34. Guiding Growth
>> GROWTH PEOPLE THINK DIFFERENT
THEN EXECUTION
>> RISK TAKERS (CONTROLLED BY DATA)
>> AUTONOMOUS
>> DEDICATION AND FOCUS MOST
RESULTS
PRO’S
45. Guiding Growth
Why this specific challenge or goal?
Ideation
Why this specific experiment?
What part of the AAARRR funnel is the experiment
geared towards?
What ICES Scores apply?
How much time and resources will it take to run this
experiment?
47. Guiding Growth
We believe that:
[this change has the effect]
“We know we have it right/wrong
When we see the following in the market:
[qualitative feedback] en/of [quantitive feedback].
48. Guiding Growth
Why do you think this hypothesis makes sense?
Is this assumption based on data or on gut feeling?
When has the experiment failed?
What span of results would be acceptable to keep on
testing?
50. Guiding Growth
Example
A conversion percentage < 5%, we consider failed.
Conversions between 5-10%, repeat experiment with
changes.
Results above 10% are considered a succes and will
be implemented
52. Guiding Growth
What relevant experiments have been done before?
Are there any reasons for the current situation?
What are the metrics to beat?
What customer problems will this experiment solve?
56. Guiding Growth
>> Step-by-step plan
How will you execute the experiment?
How will you track results?
Will you have enough traffic to say something about
the results?
57. Guiding Growth
>> Tools
What tools do you need to put the experiment live?
Are there any tools that cab ensure more results or
make your life a little easier?
58. Guiding Growth
>> Timebox
Limit the time spent per new experiment to 20 hours.
Are there any other smaller researches you can do
that will answer the same learning goals?
59. Guiding Growth
>> State..
How much and which variations you have for the
experiment.
Deadlines for designing, starting, result gathering
and learning from experiments.
64. Guiding Growth
Make a summary of the results.
Think of the metrics you want to measure
Breakdown the different variations and your control
variable.
65. Guiding Growth
How does monitoring work?
Controle your experiment > hygiene check
minimal 2 weeks (no influence of external factors)
Be patient ! Give experiment time to validate or
invalidate
67. Guiding Growth
Why was the experiment a success?
Or why not?
What can we learn from this experiment?
How can we re-use this information?
How does this experiment impact the clients deeper
down the funnel?
69. Guiding Growth
Is there a follow up needed?
Should we run additional experiments?
Succesfull experiment > extra budget?
Who will be in charge of implementing these
changes?
How can we make this experiment repeatable?