SlideShare a Scribd company logo
1 of 33
Download to read offline
Learn from the 18+ mistakes that I’ve made while running experimentation programs & how to overcome them.
Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.
Martijn Scheijbeler
Growth & SEO at Postmates
Former Marketing Director at The Next Web
martijn.scheijbeler@postmates.com
@MartijnSch
Page
Page
You finally realized:

“We want to do more A/B Testing for our
- Every Manager Ever
Page 5
What I’m seeing:
Most Experimentation Programs Are Based on Guesswork
Page 6
@MARTIJNSCH
#LAUNCHSCALE
450%900%1,350%1,800%
Janaury February March April May June July August September October November December
Page
Or in other words:
Given the options,

I prefer to learn from success.
7
@MARTIJNSCH
#LAUNCHSCALE
Page
Ideas
Quality
Assurance
Design &
Engineering
Hypothesis
Run & Analysis
Repeat
A/B Testing
Repeat

this process…
8
@MARTIJNSCH
#LAUNCHSCALE
Page
Or in other words:
Test. Eat. Sleep. Repeat.
9
@MARTIJNSCH
#LAUNCHSCALE
Page 10
WHAT CAN YOU TEST?
You want to know what might improve the bottomline.
Ideas

Generate a
backlog of
ideas
You want to generate a lot of ideas to make sure that
there are always new ways to keep on testing with the
current resources that you have. By having a longer
backlog you have the opportunity to test and prioritize at
the same time.
PROCESSES ARE ESSENTIAL
Ideas
Quality
Assurance
Design &
Engineering
Hypothesis
Run &
Analysis
Repeat
A/B Testing
Repeat

this process…
@MARTIJNSCH
#LAUNCHSCALE
They just launch, they don’t test.
MISTAKE #1
By not testing what you’re launching you’re setting yourself up
for failure.
MISTAKE #2
“We’re losing money by testing”, it’s the opposite. The
companies that aren’t testing are losing money because they’re
putting losers online without looking at the numbers.
Companies that believe
they’re wasting money.
@MARTIJNSCH
#LAUNCHSCALE
Expecting Big Wins
MISTAKE #3
They’re expecting to win the lottery by setting up their testing
program. Truth is, most of the tests that you’ll be running will be
a loser. But these are the biggest wins that you’ll get as they’ll
teach you something.
MISTAKE #4
Looking at your competitors might be useful to come up with
new ideas for your backlog. But 99% of the time your
competitors also won’t have a clue on what they’re doing.
My Competitor is Doing X, so that’s
why we’re testing X.
@MARTIJNSCH
#LAUNCHSCALE
Page
Running tests
when you don’t
have traffic.
MISTAKE #5
When you don't have enough traffic
you won’t reach the right
significance (%95+) to make sure
that you can say that you have a
winner/loser or not.
13
@MARTIJNSCH
#LAUNCHSCALE
Page
WHAT CAN YOU TEST?
You want to know what might improve the bottomline.
Hypothesis

Do you really
know what you
will be testing?
Write a hypothesis, you want to know why you’re testing
what you’re testing. Otherwise you cannot actively make
sure that you’re test are having the outcome that you’re
expecting. It will also help you in making sure that your
tests will provide an active learning instead of just trying
to hunt down some results.
PROCESSES ARE ESSENTIAL
Ideas
Quality
Assurance
Design &
Engineering
HypothesisRepeat
A/B Testing
Repeat

this process…
Run &
Analysis
14
@MARTIJNSCH
#LAUNCHSCALE
Page
They don’t create
a hypothesis.
MISTAKE #6
When you start testing you want to
make sure that you know what kind
of impact you’re expecting from an
experiment.
15
@MARTIJNSCH
#LAUNCHSCALE
Testing multiple variables at
the same time, making 3
changes basically requires 3
tests.
MISTAKE #7
“Let’s test the button, oh and let’s also change the header + CTA”.
MISTAKE #8
“Let’s run a test because I like the colors green and yellow more
and I heard on BART that they work better”.
Use numbers as the basis of your
research, not your gut feeling.
@MARTIJNSCH
#LAUNCHSCALE
Page
WHAT CAN YOU TEST?
You want to know what might improve the bottomline.
Design &
Engineering

Have designers
& engineers work
together to
create
treatments
When you’re designing and engineering your tests,
because in all honesty who’s using the WYSIWYG editors
after a few tests you want to make sure you’re in
constant sync with your designers to make sure that the
tests that you’re running are within the guidelines of your
design (or not) and that the code is written as it’s
supposed to be.
PROCESSES ARE ESSENTIAL
Ideas
Quality
Assurance
Design &
Engineering
HypothesisRepeat
A/B Testing
Repeat

this process…
Run &
Analysis
17
@MARTIJNSCH
#LAUNCHSCALE
Before and After is not an A/B
test. We launched, let’s see what
the impact is.
MISTAKE #9
A/B Testing doesn’t work when you’re just launching a feature and
trying to compare the results: before/after. It doesn’t take into
account all the different variables that could be impacting your
results.
MISTAKE #10
Many times I’ve seen companies or designers worry too much
about the designs that they’re testing. Make sure that you can
test with multiple variants if that’s possible so it’s not up to the
designer to decide what is working.
They go over 71616 revisions for
the design.
@MARTIJNSCH
#LAUNCHSCALE
Page
WHAT CAN YOU TEST?
You want to know what might improve the
bottomline.
Quality
Assurance

Are you sure your
test is working?
You have the test ‘working’ as your designers and
engineers have created the deliverables for the test. But
are you really sure that your test is working in all the
browsers that your visitors are using? Make sure to test
extensively before every test that you’re activating.
PROCESSES ARE ESSENTIAL
Quality
Assurance
Design &
Engineering
HypothesisRepeat
A/B Testing
Repeat

this process…
Ideas
Run &
Analysis
19
@MARTIJNSCH
#LAUNCHSCALE
Page
They don’t Q&A
their tests. Even
your mother can
have an opinion
this time.
MISTAKE #11
Make sure that you’re QA’ing your
experiment before you activate it. It
will prevent you from stopping a test
halfway when your results are
skewed.
#lovemoms
20
@MARTIJNSCH
#LAUNCHSCALE
Page
WHAT CAN YOU TEST?
You want to know what might improve the
bottomline.
Run &
Analysis

Time to have
users
experience your
treatments
It’s time to start running your test, make sure that you’re
running. Essential is that you’re running your tests for
the right duration. After this comest the best part,
analysis.
PROCESSES ARE ESSENTIAL
Quality
Assurance
Design &
Engineering
Hypothesis
Run &
Analysis
Repeat
A/B Testing
Repeat

this process…
Ideas
21
@MARTIJNSCH
#LAUNCHSCALE
Page
Running your
tests not long
enough, calling
the results early.
MISTAKE #12
You want to make sure that you
reach significance, but at the same
time you also need to make sure
that you’re dealing with your
business cycles.
22
@MARTIJNSCH
#LAUNCHSCALE
Page
Running multiple
tests with
overlap.. it’s
possible, but
segment the sh*t
out of your tests.
MISTAKE #13
You have multiple tests running?
Great, but make sure that you also
segment based on that data and
plot what the impact is of the
overlap in between tests.
23
@MARTIJNSCH
#LAUNCHSCALE
Page 24
@MARTIJNSCH
#LAUNCHSCALE
Data is not sent to your main
analytics tool, or you’re
comparing your a/b testing tool
to analytics, good luck.
MISTAKE #14
MISTAKE #15
Basic fail #1, as long as you don’t have the right significance
you’re basically running your testing program like flipping a
coin: you have a 50/50 chance of finding a winner.
Going with your results without
significance.
@MARTIJNSCH
#LAUNCHSCALE
Page
WHAT CAN YOU TEST?
You want to know what might improve the
bottomline.
Repeat

Start all over
again: eat,
test, sleep,
repeat
You’ve been running your first experiments that hopefully
have been successful. Now comes the next step:
repeating the process.
PROCESSES ARE ESSENTIAL
Quality
Assurance
Design &
Engineering
Hypothesis
Run &
Analysis
Repeat
A/B Testing
Repeat

this process…
Ideas
26
@MARTIJNSCH
#LAUNCHSCALE
Not deploying your winner fast
enough, takes 2 months to
launch
MISTAKE #16
MISTAKE #17
Keep track of what you’re testing. Once you’re exceeding 50
experiments it’s unlikely that you still know what you’ve tested
before and what the results were.
They’re not keeping track of their
tests. No documentation.
@MARTIJNSCH
#LAUNCHSCALE
Speed is one of the most essential components of a Growth
process. So not launching your winning tests or waiting for it for
a while is not helping you or your company.
Page
They give up.
MISTAKE #18
NEVER EVER GIVE UP ON TESTING.
28
@MARTIJNSCH
#LAUNCHSCALE
If you double the number of
experiments you’re going to
double your inventiveness
- Jeff Bezos, Amazon
Page
Ideas:
1. They just launch, they don’t test.
2. Companies that believe they’re wasting money.
3. Expecting Big Wins.
4. My Competitor is Doing X, so that’s why we’re testing X.
5. Running tests when you don’t have traffic.
Hypothesis:
1. They don’t create a hypothesis.
2. Testing multiple variables at the same time, making 3 changes basically requires 3 tests.
3. Use numbers as the basis of your research, not your gut feeling.
Design & Engineering:
1. Before and After is not an A/B test. We launched, let’s see what the impact is.
2. They go over 71616 revisions for the design.
30
Summary
@MARTIJNSCH
#LAUNCHSCALE
Page
Quality Assurance:
1. They don’t Q&A their tests. Even your mother can have an opinion this time.
Run & Analysis:
1. Running your tests not long enough, calling the results early.
2. Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests.
3. Data is not sent to your main analytics tool, or you’re comparing your a/b testing tool to analytics, good
luck.
4. Going with your results without significance
5. You run your tests for too long… more than 4 weeks is not to be advise, cookie deletion. Multiple issues.
Repeat:
1. Not deploying your winner fast enough, takes 2 months to launch
2. They’re not keeping track of their tests. No documentation.
3. They Give Up!
31
Summary
@MARTIJNSCH
#LAUNCHSCALE
Page
Thank you, have fun!
32
@MartijnSch
#LAUNCHSCALE
@MARTIJNSCH
Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.

More Related Content

What's hot

[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli
[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli
[CXL Live 16] When, Why and How to Do Innovative Testing by Marie PolliCXL
 
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...CXL
 
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...CXL
 
3 Reasons to Use Data for Successful CRO
3 Reasons to Use Data for Successful CRO3 Reasons to Use Data for Successful CRO
3 Reasons to Use Data for Successful CROJason Wright
 
Conversion Rate Optimization 101 - Kick-Start Your Growth Engine
Conversion Rate Optimization 101 - Kick-Start Your Growth EngineConversion Rate Optimization 101 - Kick-Start Your Growth Engine
Conversion Rate Optimization 101 - Kick-Start Your Growth EngineKissmetrics on SlideShare
 
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...CXL
 
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...CXL
 
Technical Marketing and Growth Hacking Low Hanging Fruit
Technical Marketing and Growth Hacking Low Hanging FruitTechnical Marketing and Growth Hacking Low Hanging Fruit
Technical Marketing and Growth Hacking Low Hanging FruitDavid Arnoux . Growth
 
Methods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenMethods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenChris Green
 
DevOps Traction - Continuous Lifecycle 2016 London
DevOps Traction - Continuous Lifecycle 2016 LondonDevOps Traction - Continuous Lifecycle 2016 London
DevOps Traction - Continuous Lifecycle 2016 LondonDevOps Ltd.
 
A/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryA/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryUseItBetter
 
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummit
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummitSEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummit
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummitSearch Engine Journal
 
How Changing Your Mind can increase Successs #BrightonSEO 2016
How Changing Your Mind can increase Successs #BrightonSEO 2016 How Changing Your Mind can increase Successs #BrightonSEO 2016
How Changing Your Mind can increase Successs #BrightonSEO 2016 Lisa Myers
 
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...Search Engine Journal
 
Web Research Tools & Training
Web Research Tools & TrainingWeb Research Tools & Training
Web Research Tools & TrainingAstek Consulting
 
Data Science: The Product Manager's Primer
Data Science: The Product Manager's PrimerData Science: The Product Manager's Primer
Data Science: The Product Manager's PrimerProduct School
 
Technical Marketing: The New Normal by Michael King - #SEJSummit Atlanta
Technical Marketing: The New Normal by Michael King - #SEJSummit AtlantaTechnical Marketing: The New Normal by Michael King - #SEJSummit Atlanta
Technical Marketing: The New Normal by Michael King - #SEJSummit AtlantaSearch Engine Journal
 
Essential Technical SEO learnings from 120+ site migrations
Essential Technical SEO learnings from 120+ site migrationsEssential Technical SEO learnings from 120+ site migrations
Essential Technical SEO learnings from 120+ site migrationsChris Green
 
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynoteEcommerce Conversion World, London March 23 2017 - Ton Wesseling keynote
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynoteOnline Dialogue
 
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & Customers
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & CustomersPeep Laja, CEO, ConversionXL - How to Turn Data into Insights & Customers
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & CustomersTraction Conf
 

What's hot (20)

[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli
[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli
[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli
 
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...
[CXL Live 16] SaaS Optimization - Effective Metrics, Process and Hacks by Ste...
 
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...
[Elite Camp 2016] Paul Rouke - Tackling the Global Pandemic That is Bullshit ...
 
3 Reasons to Use Data for Successful CRO
3 Reasons to Use Data for Successful CRO3 Reasons to Use Data for Successful CRO
3 Reasons to Use Data for Successful CRO
 
Conversion Rate Optimization 101 - Kick-Start Your Growth Engine
Conversion Rate Optimization 101 - Kick-Start Your Growth EngineConversion Rate Optimization 101 - Kick-Start Your Growth Engine
Conversion Rate Optimization 101 - Kick-Start Your Growth Engine
 
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
 
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
[Elite Camp 2016] Amy Harrison - The Science of Screenwriting for Online Mark...
 
Technical Marketing and Growth Hacking Low Hanging Fruit
Technical Marketing and Growth Hacking Low Hanging FruitTechnical Marketing and Growth Hacking Low Hanging Fruit
Technical Marketing and Growth Hacking Low Hanging Fruit
 
Methods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenMethods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris green
 
DevOps Traction - Continuous Lifecycle 2016 London
DevOps Traction - Continuous Lifecycle 2016 LondonDevOps Traction - Continuous Lifecycle 2016 London
DevOps Traction - Continuous Lifecycle 2016 London
 
A/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryA/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey Theory
 
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummit
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummitSEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummit
SEJ Summit 2015: The Content Evolution by Marcus Tober #SEJSummit
 
How Changing Your Mind can increase Successs #BrightonSEO 2016
How Changing Your Mind can increase Successs #BrightonSEO 2016 How Changing Your Mind can increase Successs #BrightonSEO 2016
How Changing Your Mind can increase Successs #BrightonSEO 2016
 
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...
SEJ Summit 2015: From Associates to the C-Suite: How Social Should Integrate ...
 
Web Research Tools & Training
Web Research Tools & TrainingWeb Research Tools & Training
Web Research Tools & Training
 
Data Science: The Product Manager's Primer
Data Science: The Product Manager's PrimerData Science: The Product Manager's Primer
Data Science: The Product Manager's Primer
 
Technical Marketing: The New Normal by Michael King - #SEJSummit Atlanta
Technical Marketing: The New Normal by Michael King - #SEJSummit AtlantaTechnical Marketing: The New Normal by Michael King - #SEJSummit Atlanta
Technical Marketing: The New Normal by Michael King - #SEJSummit Atlanta
 
Essential Technical SEO learnings from 120+ site migrations
Essential Technical SEO learnings from 120+ site migrationsEssential Technical SEO learnings from 120+ site migrations
Essential Technical SEO learnings from 120+ site migrations
 
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynoteEcommerce Conversion World, London March 23 2017 - Ton Wesseling keynote
Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote
 
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & Customers
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & CustomersPeep Laja, CEO, ConversionXL - How to Turn Data into Insights & Customers
Peep Laja, CEO, ConversionXL - How to Turn Data into Insights & Customers
 

Similar to Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.

A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsPirate Skills
 
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013TEST Huddle
 
Master the Essentials of Conversion Optimization
Master the Essentials of Conversion OptimizationMaster the Essentials of Conversion Optimization
Master the Essentials of Conversion Optimizationjoshuapaulharper
 
Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar   what you're doing wrong in your cro program (sharable version)Cro webinar   what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)VWO
 
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...How to Increase Your Testing Success by Combining Qualitative and Quantitativ...
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...Optimizely
 
Culture of Optimization
Culture of OptimizationCulture of Optimization
Culture of OptimizationNikki Johnson
 
REV UP 360: Rev up your ROI with Conversion Testing - Andy Batten
REV UP 360: Rev up your ROI with Conversion Testing - Andy BattenREV UP 360: Rev up your ROI with Conversion Testing - Andy Batten
REV UP 360: Rev up your ROI with Conversion Testing - Andy BattenAMA Colorado
 
iSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doingiSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doingBridgeline Digital
 
How to fortify UA strategy with creatives testing
How to fortify UA strategy with creatives testingHow to fortify UA strategy with creatives testing
How to fortify UA strategy with creatives testingSplitMetrics
 
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)Trigger
 
Growing Your Business Through Experimentation
Growing Your Business Through ExperimentationGrowing Your Business Through Experimentation
Growing Your Business Through ExperimentationHiten Shah
 
Growth Hacking - High Tempo Testing
Growth Hacking - High Tempo TestingGrowth Hacking - High Tempo Testing
Growth Hacking - High Tempo TestingTomek Duda
 
Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Always Be Testing - Learn from Every A/B Test (Hiten Shah)Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Always Be Testing - Learn from Every A/B Test (Hiten Shah)Future Insights
 
Driving Product Success through Experimentation
Driving Product Success through ExperimentationDriving Product Success through Experimentation
Driving Product Success through ExperimentationNUS-ISS
 
Still A/B testing your buttons? You need to think (much) bigger...
Still A/B testing your buttons? You need to think (much) bigger...Still A/B testing your buttons? You need to think (much) bigger...
Still A/B testing your buttons? You need to think (much) bigger...Guido X Jansen
 
Master the essentials of conversion optimization
Master the essentials of conversion optimizationMaster the essentials of conversion optimization
Master the essentials of conversion optimizationArnas Rackauskas
 
Text Ad Testing Master's Guide - A ClickEquations White Paper
Text Ad Testing Master's Guide - A ClickEquations White PaperText Ad Testing Master's Guide - A ClickEquations White Paper
Text Ad Testing Master's Guide - A ClickEquations White PaperClickEquations
 
Andreas Reiffen - SMX London Slidedeck
Andreas Reiffen - SMX London SlidedeckAndreas Reiffen - SMX London Slidedeck
Andreas Reiffen - SMX London SlidedeckCrealytics
 

Similar to Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong. (20)

A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate Skills
 
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
 
Master the Essentials of Conversion Optimization
Master the Essentials of Conversion OptimizationMaster the Essentials of Conversion Optimization
Master the Essentials of Conversion Optimization
 
Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar   what you're doing wrong in your cro program (sharable version)Cro webinar   what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)
 
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...How to Increase Your Testing Success by Combining Qualitative and Quantitativ...
How to Increase Your Testing Success by Combining Qualitative and Quantitativ...
 
Culture of Optimization
Culture of OptimizationCulture of Optimization
Culture of Optimization
 
REV UP 360: Rev up your ROI with Conversion Testing - Andy Batten
REV UP 360: Rev up your ROI with Conversion Testing - Andy BattenREV UP 360: Rev up your ROI with Conversion Testing - Andy Batten
REV UP 360: Rev up your ROI with Conversion Testing - Andy Batten
 
iSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doingiSG Webinar – AB Testing: The most important thing you’re NOT doing
iSG Webinar – AB Testing: The most important thing you’re NOT doing
 
How to fortify UA strategy with creatives testing
How to fortify UA strategy with creatives testingHow to fortify UA strategy with creatives testing
How to fortify UA strategy with creatives testing
 
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)
12 Steps to Effective Growth Hacking (www.wepullthetrigger.com)
 
Growing Your Business Through Experimentation
Growing Your Business Through ExperimentationGrowing Your Business Through Experimentation
Growing Your Business Through Experimentation
 
CRO Cheat Sheet
CRO Cheat SheetCRO Cheat Sheet
CRO Cheat Sheet
 
Growth Hacking - High Tempo Testing
Growth Hacking - High Tempo TestingGrowth Hacking - High Tempo Testing
Growth Hacking - High Tempo Testing
 
Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Always Be Testing - Learn from Every A/B Test (Hiten Shah)Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Always Be Testing - Learn from Every A/B Test (Hiten Shah)
 
Driving Product Success through Experimentation
Driving Product Success through ExperimentationDriving Product Success through Experimentation
Driving Product Success through Experimentation
 
Still A/B testing your buttons? You need to think (much) bigger...
Still A/B testing your buttons? You need to think (much) bigger...Still A/B testing your buttons? You need to think (much) bigger...
Still A/B testing your buttons? You need to think (much) bigger...
 
Master the essentials of conversion optimization
Master the essentials of conversion optimizationMaster the essentials of conversion optimization
Master the essentials of conversion optimization
 
Text ad testing masters guide click equations white paper v2
Text ad testing masters guide   click equations white paper v2Text ad testing masters guide   click equations white paper v2
Text ad testing masters guide click equations white paper v2
 
Text Ad Testing Master's Guide - A ClickEquations White Paper
Text Ad Testing Master's Guide - A ClickEquations White PaperText Ad Testing Master's Guide - A ClickEquations White Paper
Text Ad Testing Master's Guide - A ClickEquations White Paper
 
Andreas Reiffen - SMX London Slidedeck
Andreas Reiffen - SMX London SlidedeckAndreas Reiffen - SMX London Slidedeck
Andreas Reiffen - SMX London Slidedeck
 

More from Martijn Scheijbeler

BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...Martijn Scheijbeler
 
Not Provided - Search Marketing Thursday 7 November 2013
Not Provided - Search Marketing Thursday 7 November 2013Not Provided - Search Marketing Thursday 7 November 2013
Not Provided - Search Marketing Thursday 7 November 2013Martijn Scheijbeler
 

More from Martijn Scheijbeler (6)

Digital Analytics Summit 2022
Digital Analytics Summit 2022Digital Analytics Summit 2022
Digital Analytics Summit 2022
 
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
BrightonSEO October 2022 - Martijn Scheybeler - SEO Testing: Find Out What Wo...
 
Brighton SEO - September 2016
Brighton SEO - September 2016Brighton SEO - September 2016
Brighton SEO - September 2016
 
Friends of Search 2016
Friends of Search 2016Friends of Search 2016
Friends of Search 2016
 
SEO Meetup Utrecht
SEO Meetup UtrechtSEO Meetup Utrecht
SEO Meetup Utrecht
 
Not Provided - Search Marketing Thursday 7 November 2013
Not Provided - Search Marketing Thursday 7 November 2013Not Provided - Search Marketing Thursday 7 November 2013
Not Provided - Search Marketing Thursday 7 November 2013
 

Recently uploaded

Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...VereigenMedia1
 
History of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfHistory of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfwilliam charnock
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Nugget Global
 
The Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineThe Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineglobalbrandmagazines1
 
Digital Marketing complete introduction.
Digital Marketing complete introduction.Digital Marketing complete introduction.
Digital Marketing complete introduction.Kashish Bindra
 
AMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo EvangelistaAMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo Evangelistacrevangelista
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Associazione Digital Days
 
The Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfThe Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfFinance Advertising Network
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaEsteller
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logonelaohaimbodi
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...Social Samosa
 
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024Welcome to the Future: New Digital Marketing Trends Shaking Up 2024
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024rabboniseo
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Valters Lauzums
 
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Associazione Digital Days
 
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...Valters Lauzums
 
TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23Social Samosa
 
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxv100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxjeromecuyos1
 
Gen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfGen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfMedia Logic
 
HAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewHAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewEsteller
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxSahil Srivastava
 

Recently uploaded (20)

Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...
 
History of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfHistory of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdf
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
 
The Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazineThe Fall of Social Media Marketing :- Global brand magazine
The Fall of Social Media Marketing :- Global brand magazine
 
Digital Marketing complete introduction.
Digital Marketing complete introduction.Digital Marketing complete introduction.
Digital Marketing complete introduction.
 
AMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo EvangelistaAMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo Evangelista
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
 
The Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfThe Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdf
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compelta
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logo
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
 
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024Welcome to the Future: New Digital Marketing Trends Shaking Up 2024
Welcome to the Future: New Digital Marketing Trends Shaking Up 2024
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
 
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
Richard van der Velde, Technical Support Lead for Cookiebot @CMP – “Artificia...
 
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...
Social Media Marketing Lecture for Advanced Digital & Social Media Strategy a...
 
TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23TAM Sports IPL 17 Advertising Report- M01 - M23
TAM Sports IPL 17 Advertising Report- M01 - M23
 
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptxv100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
v100 20-Year HyperMarketing Plan by Jerome Cuyos.pptx
 
Gen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdfGen Z and Millennial Debit Card Use Survey.pdf
Gen Z and Millennial Debit Card Use Survey.pdf
 
HAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewHAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_Preview
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptx
 

Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.

  • 1. Learn from the 18+ mistakes that I’ve made while running experimentation programs & how to overcome them. Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.
  • 2. Martijn Scheijbeler Growth & SEO at Postmates Former Marketing Director at The Next Web martijn.scheijbeler@postmates.com @MartijnSch
  • 4. Page You finally realized:
 “We want to do more A/B Testing for our - Every Manager Ever
  • 5. Page 5 What I’m seeing: Most Experimentation Programs Are Based on Guesswork
  • 6. Page 6 @MARTIJNSCH #LAUNCHSCALE 450%900%1,350%1,800% Janaury February March April May June July August September October November December
  • 7. Page Or in other words: Given the options,
 I prefer to learn from success. 7 @MARTIJNSCH #LAUNCHSCALE
  • 8. Page Ideas Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat
 this process… 8 @MARTIJNSCH #LAUNCHSCALE
  • 9. Page Or in other words: Test. Eat. Sleep. Repeat. 9 @MARTIJNSCH #LAUNCHSCALE
  • 10. Page 10 WHAT CAN YOU TEST? You want to know what might improve the bottomline. Ideas
 Generate a backlog of ideas You want to generate a lot of ideas to make sure that there are always new ways to keep on testing with the current resources that you have. By having a longer backlog you have the opportunity to test and prioritize at the same time. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat
 this process… @MARTIJNSCH #LAUNCHSCALE
  • 11. They just launch, they don’t test. MISTAKE #1 By not testing what you’re launching you’re setting yourself up for failure. MISTAKE #2 “We’re losing money by testing”, it’s the opposite. The companies that aren’t testing are losing money because they’re putting losers online without looking at the numbers. Companies that believe they’re wasting money. @MARTIJNSCH #LAUNCHSCALE
  • 12. Expecting Big Wins MISTAKE #3 They’re expecting to win the lottery by setting up their testing program. Truth is, most of the tests that you’ll be running will be a loser. But these are the biggest wins that you’ll get as they’ll teach you something. MISTAKE #4 Looking at your competitors might be useful to come up with new ideas for your backlog. But 99% of the time your competitors also won’t have a clue on what they’re doing. My Competitor is Doing X, so that’s why we’re testing X. @MARTIJNSCH #LAUNCHSCALE
  • 13. Page Running tests when you don’t have traffic. MISTAKE #5 When you don't have enough traffic you won’t reach the right significance (%95+) to make sure that you can say that you have a winner/loser or not. 13 @MARTIJNSCH #LAUNCHSCALE
  • 14. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Hypothesis
 Do you really know what you will be testing? Write a hypothesis, you want to know why you’re testing what you’re testing. Otherwise you cannot actively make sure that you’re test are having the outcome that you’re expecting. It will also help you in making sure that your tests will provide an active learning instead of just trying to hunt down some results. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat
 this process… Run & Analysis 14 @MARTIJNSCH #LAUNCHSCALE
  • 15. Page They don’t create a hypothesis. MISTAKE #6 When you start testing you want to make sure that you know what kind of impact you’re expecting from an experiment. 15 @MARTIJNSCH #LAUNCHSCALE
  • 16. Testing multiple variables at the same time, making 3 changes basically requires 3 tests. MISTAKE #7 “Let’s test the button, oh and let’s also change the header + CTA”. MISTAKE #8 “Let’s run a test because I like the colors green and yellow more and I heard on BART that they work better”. Use numbers as the basis of your research, not your gut feeling. @MARTIJNSCH #LAUNCHSCALE
  • 17. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Design & Engineering
 Have designers & engineers work together to create treatments When you’re designing and engineering your tests, because in all honesty who’s using the WYSIWYG editors after a few tests you want to make sure you’re in constant sync with your designers to make sure that the tests that you’re running are within the guidelines of your design (or not) and that the code is written as it’s supposed to be. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat
 this process… Run & Analysis 17 @MARTIJNSCH #LAUNCHSCALE
  • 18. Before and After is not an A/B test. We launched, let’s see what the impact is. MISTAKE #9 A/B Testing doesn’t work when you’re just launching a feature and trying to compare the results: before/after. It doesn’t take into account all the different variables that could be impacting your results. MISTAKE #10 Many times I’ve seen companies or designers worry too much about the designs that they’re testing. Make sure that you can test with multiple variants if that’s possible so it’s not up to the designer to decide what is working. They go over 71616 revisions for the design. @MARTIJNSCH #LAUNCHSCALE
  • 19. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Quality Assurance
 Are you sure your test is working? You have the test ‘working’ as your designers and engineers have created the deliverables for the test. But are you really sure that your test is working in all the browsers that your visitors are using? Make sure to test extensively before every test that you’re activating. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat
 this process… Ideas Run & Analysis 19 @MARTIJNSCH #LAUNCHSCALE
  • 20. Page They don’t Q&A their tests. Even your mother can have an opinion this time. MISTAKE #11 Make sure that you’re QA’ing your experiment before you activate it. It will prevent you from stopping a test halfway when your results are skewed. #lovemoms 20 @MARTIJNSCH #LAUNCHSCALE
  • 21. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Run & Analysis
 Time to have users experience your treatments It’s time to start running your test, make sure that you’re running. Essential is that you’re running your tests for the right duration. After this comest the best part, analysis. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat
 this process… Ideas 21 @MARTIJNSCH #LAUNCHSCALE
  • 22. Page Running your tests not long enough, calling the results early. MISTAKE #12 You want to make sure that you reach significance, but at the same time you also need to make sure that you’re dealing with your business cycles. 22 @MARTIJNSCH #LAUNCHSCALE
  • 23. Page Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests. MISTAKE #13 You have multiple tests running? Great, but make sure that you also segment based on that data and plot what the impact is of the overlap in between tests. 23 @MARTIJNSCH #LAUNCHSCALE
  • 25. Data is not sent to your main analytics tool, or you’re comparing your a/b testing tool to analytics, good luck. MISTAKE #14 MISTAKE #15 Basic fail #1, as long as you don’t have the right significance you’re basically running your testing program like flipping a coin: you have a 50/50 chance of finding a winner. Going with your results without significance. @MARTIJNSCH #LAUNCHSCALE
  • 26. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Repeat
 Start all over again: eat, test, sleep, repeat You’ve been running your first experiments that hopefully have been successful. Now comes the next step: repeating the process. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat
 this process… Ideas 26 @MARTIJNSCH #LAUNCHSCALE
  • 27. Not deploying your winner fast enough, takes 2 months to launch MISTAKE #16 MISTAKE #17 Keep track of what you’re testing. Once you’re exceeding 50 experiments it’s unlikely that you still know what you’ve tested before and what the results were. They’re not keeping track of their tests. No documentation. @MARTIJNSCH #LAUNCHSCALE Speed is one of the most essential components of a Growth process. So not launching your winning tests or waiting for it for a while is not helping you or your company.
  • 28. Page They give up. MISTAKE #18 NEVER EVER GIVE UP ON TESTING. 28 @MARTIJNSCH #LAUNCHSCALE
  • 29. If you double the number of experiments you’re going to double your inventiveness - Jeff Bezos, Amazon
  • 30. Page Ideas: 1. They just launch, they don’t test. 2. Companies that believe they’re wasting money. 3. Expecting Big Wins. 4. My Competitor is Doing X, so that’s why we’re testing X. 5. Running tests when you don’t have traffic. Hypothesis: 1. They don’t create a hypothesis. 2. Testing multiple variables at the same time, making 3 changes basically requires 3 tests. 3. Use numbers as the basis of your research, not your gut feeling. Design & Engineering: 1. Before and After is not an A/B test. We launched, let’s see what the impact is. 2. They go over 71616 revisions for the design. 30 Summary @MARTIJNSCH #LAUNCHSCALE
  • 31. Page Quality Assurance: 1. They don’t Q&A their tests. Even your mother can have an opinion this time. Run & Analysis: 1. Running your tests not long enough, calling the results early. 2. Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests. 3. Data is not sent to your main analytics tool, or you’re comparing your a/b testing tool to analytics, good luck. 4. Going with your results without significance 5. You run your tests for too long… more than 4 weeks is not to be advise, cookie deletion. Multiple issues. Repeat: 1. Not deploying your winner fast enough, takes 2 months to launch 2. They’re not keeping track of their tests. No documentation. 3. They Give Up! 31 Summary @MARTIJNSCH #LAUNCHSCALE
  • 32. Page Thank you, have fun! 32 @MartijnSch #LAUNCHSCALE @MARTIJNSCH