1) How to design powerful experiments provides tips for setting up successful A/B tests and experiments to make data-driven decisions and reduce risks. 2) Key tips include having the proper experimentation infrastructure in place, following an iterative process of developing hypotheses, designing experiments, analyzing results, and executing. 3) Case studies show that A/B tests at Google and REA Group led to increased annual revenue and conversion rates, demonstrating the value of experimentation.

1. How to Design Powerful
Experiments
Ying Zhang, REA Group | Nov 2021

2. Product Management

3. How do we know
we re building the right thing?
The fundamental question:

4. A/B Experiment
50% Traffic 50% Traffic

5. A/B Experiment
+$200m
In annual revenue
Google – Test 41 shades of blue

6. Multi-Page Experiment
Control
Variant
2 3 4 5
1
2 3 4 5
1
+5.6%
In completion rate
REA Group – Calculator Experiment, 2021

7. Split-Url Experiment
REA Group – Broker Page Experiment, 2021
Control Variant
+46%
In conversion rate

8. #1
Make data-driven decisions.

9. #2
Reduce risks upfront.

10. #3
Refining our understanding of user
behavior and customer problems.

11. 75%
of experiments fail in practice,
i.e., not being able to move
success metrics
67% of experiments fail
85% of experiments fail
1 in 5,000 experiments managed to move
success metrics
92% of experiments fail

12. Tips to Set You Up for Success

13. #1
Make sure you have the
experimentation infrastructure
in place.

14. Credits: Reforge

15. #2
Follow a structured, iterative
experimentation process.

16. Develop hypotheses Design experiments Analyze results
Execute
• Use behavioral science
to strengthen your ideas
• What’s the problem
you’re trying to solve?
• What assumptions
have you made about
users?
• What’s the minimum viable
test you can run to test your
hypothesis?
• How many people do you
need? How long should the
experiment run? How do you
define success?
• If traffic is < 1000 UVs per day,
conversion < 50%, probably
not worthy of running an
experiment.
• Typical duration is 2 to 8
• weeks
• Do you have the right
infrastructure to help you carry
out the experiment as designed?
• Double check your success
metrics are clearly defined,
tagged up and configured
correctly in your
experimentation tool
❗
• Was the experiment a
success or failure? Why
is it a success or failure?
• How different was the
result from what you
expected?
• What’s the next step?
❗
• Be self-disciplined and take
the time to review, discuss and
communicate results.
Typical Experiment Lifecycle

17. “ Almost any question can be answered cheaply,
quickly, and finally by a test campaign. And that’s the
way to answer them – not by arguments around a
table. Go to the court of last resort – the buyers of
your product.”
Claude Hopkins
Author of Scientific Advertising