SlideShare a Scribd company logo
1 of 40
Download to read offline
Many thanks to:
Growth Hacking Meetup
- Episode 4 -

9 A/B Testing Use Cases
But first… some A/B Testing
mistakes startups make ALL the time
1
1

A/B tests are ended too early

Statistical significance (large sample size, etc) is what
tells you version A is actually better than version B
As an optimizer, your job 

is to figure out the truth. 

You have to put your ego aside.
2
1

Tests are not run for full weeks

Conversion rates can vary greatly depending on the
day of the week. Always test a full week at a time.
3
1

Always pay attention to external factors

Is it Christmas? External factors definitely affect your
test results. When in doubt, run a follow-up test.
4
1

Testing random ideas without a hypothesis
You’re wasting precious time and traffic.
Never do that.
By the way, if you want to test button colors

“green vs orange” is not the essence of A/B testing. 

It’s about understanding the target audience.
Use your traffic on high-impact stuff. 

Test data-driven hypotheses.
5
1

They give up after the first test fails

Run a follow-up test, learn from it, and
improve your hypotheses.
“I have not failed 10,000 times. 

I have successfully found 10,000 ways that will not work.”
- Thomas Alva Edison, inventor
6
1

They’re ignoring small gains 

Only 1 out of 8 A/B tests drive 

significant change.
7
1

They’re not running tests at all times

Every single day without a test is a wasted day.
Testing is learning.
Now, for the second part… 

12 Surprising A/B Test Results to
Stop You Making Assumptions
Understand A/B Testing in 9 use cases & 7 mistakes
Test 1: Which Copy Increased Trial Sign-Ups?
Result
In this test, Version B increased
sign-ups by 38% – a big rise.
However, you would think version
A was the better design.
Why does Version B work? 

Simply because the copy is better.
Lesson: landing pages don’t have to
be pretty to be effective.
Test 2: Which Landing Page Got 24% More
Lead Generation Form Submissions?

Picture vs No picture
Result
Surprisingly, Version A was the
page that got the 24% increase in
submissions, simply by removing
the image from the page.
This is a great example of why you should confirm
your assumptions with quantitative testing.
Test 3: Which Radically Redesigned Form
Increased B2B Leads By 368.5%?
Result
Version A is an obvious winner, but it’s not just
the big red button that makes the difference.
Version A keeps things really tight and uses
grouping to visually shrink the impact of the form
When designing your landing page,
don’t overestimate your user’s
tolerance, goodwill, and patience.
Test 4: Does Matching Headline & Body Copy
to Your Initial Ad Copy Really Matter?
Result
Version B looks as if it should be better: the
headline copy is snappier, the sub-head clearer,
but in tests version A increased leads by 115%.
Why? Version A was designed to tie in with the ads
that drive users to the page. 

Lesson: making the elements of the sales funnel
work together increases efficacy.
Test 5: Which Landing Page Increased
Mortgage Application Leads by 64%?
Result
Video can be a very effective tool in
communicating lots of information in a compact
form. But the presence of video couldn’t save
Version B; Version A increased leads by 64%.
Test 6: Does adding testimonials to your
Homepage increase Sales?
Result
It would seem that this would have very little
effect, but in practice this small change
increased sales by 34% – a big margin.
Why is this? Having ‘social proof’, even in this
basic form, humanizes the conversion experience
Test 7: Does Social Proof Increase Conversions?
‘Big Brand Prices’ vs. Consumer Ratings
Result
‘Social proof’ is a powerful tool that can have a
demonstrable effect on conversion outcomes.
But in this case, customers were just looking for the
cheapest offer. Asking them to consider an additional
piece of information – customer satisfaction – made
them back away from the conversion.
Test 8: Does an Email Security Seal Help or
Hinder Lead Generation Form Completions?
Result
Actually, users were put off by seeing it
in this context, assuming that they were
about to pay for something
Test 9: Which Page Got an 89.8% Lift in
Free Account Sign Ups?
Result
Version B wins: It has three bullet points,
as opposed to words in speech bubbles.
Removing a distraction from the page and
reducing the risk of users navigating away
from the page worked better
Whether you are right or wrong: 

First, make sure you get the basics right before you
start testing, and second, always be testing, because
unless you test, you can never be absolutely sure.
You are Welcome to Join the Adventure
hey@thefamily.co

More Related Content

What's hot

Email Strategy, Design and User Experience
Email Strategy, Design and User ExperienceEmail Strategy, Design and User Experience
Email Strategy, Design and User ExperienceLitmus
 
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing Win
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing WinQualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing Win
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing WinOptimizely
 
How to Reduce the Pain of Prospecting [Webinar Slide Deck]
How to Reduce the Pain of Prospecting [Webinar Slide Deck]How to Reduce the Pain of Prospecting [Webinar Slide Deck]
How to Reduce the Pain of Prospecting [Webinar Slide Deck]CustomerCentric Selling
 
[CXL Live 16] Fight Back Against Back by Rand Fishkin
[CXL Live 16] Fight Back Against Back by Rand Fishkin[CXL Live 16] Fight Back Against Back by Rand Fishkin
[CXL Live 16] Fight Back Against Back by Rand FishkinCXL
 
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to Ask
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to AskEverything You Wanted to Know About LinkedIn, but Were Too Embarrassed to Ask
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to AskJonathan Rick
 
Google Analytics Fundamentals: Set Up and Basics for Measurement
Google Analytics Fundamentals: Set Up and Basics for MeasurementGoogle Analytics Fundamentals: Set Up and Basics for Measurement
Google Analytics Fundamentals: Set Up and Basics for MeasurementOrbit Media Studios
 
Growth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-YoungGrowth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-YoungMattan Griffel
 
Why It's Time to Kill the Cold Call Once and For All
Why It's Time to Kill the Cold Call Once and For AllWhy It's Time to Kill the Cold Call Once and For All
Why It's Time to Kill the Cold Call Once and For AllHubSpot
 
How to Run a Lean Startup
How to Run a Lean StartupHow to Run a Lean Startup
How to Run a Lean StartupAmancio Bouza
 
ACH Marketing Presentation
ACH Marketing PresentationACH Marketing Presentation
ACH Marketing PresentationPlatoonDaddy
 
Partnering With Others
Partnering With Others Partnering With Others
Partnering With Others agencyside
 
Content Marketing Lessons From 10 Years and 41,525,103 Website Visits
Content Marketing Lessons From 10 Years and 41,525,103 Website VisitsContent Marketing Lessons From 10 Years and 41,525,103 Website Visits
Content Marketing Lessons From 10 Years and 41,525,103 Website VisitsHiten Shah
 
Driving Growth Through Email Marketing
Driving Growth Through Email MarketingDriving Growth Through Email Marketing
Driving Growth Through Email MarketingGrowth Hacking Asia
 
Growing Moz: 8 Lessons Learned
Growing Moz: 8 Lessons LearnedGrowing Moz: 8 Lessons Learned
Growing Moz: 8 Lessons LearnedRand Fishkin
 
What is email list building. 100 fast list building methods and techniques
What is email list building. 100 fast list building methods and techniquesWhat is email list building. 100 fast list building methods and techniques
What is email list building. 100 fast list building methods and techniquesIsabella
 

What's hot (20)

Email Strategy, Design and User Experience
Email Strategy, Design and User ExperienceEmail Strategy, Design and User Experience
Email Strategy, Design and User Experience
 
Paid Search: How to Hit a Moving Target Without Losing Your Head (or Your Wal...
Paid Search: How to Hit a Moving Target Without Losing Your Head (or Your Wal...Paid Search: How to Hit a Moving Target Without Losing Your Head (or Your Wal...
Paid Search: How to Hit a Moving Target Without Losing Your Head (or Your Wal...
 
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing Win
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing WinQualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing Win
Qualaroo & Optimizely Webinar - 5 Steps to Your First A/B Testing Win
 
How to Reduce the Pain of Prospecting [Webinar Slide Deck]
How to Reduce the Pain of Prospecting [Webinar Slide Deck]How to Reduce the Pain of Prospecting [Webinar Slide Deck]
How to Reduce the Pain of Prospecting [Webinar Slide Deck]
 
[CXL Live 16] Fight Back Against Back by Rand Fishkin
[CXL Live 16] Fight Back Against Back by Rand Fishkin[CXL Live 16] Fight Back Against Back by Rand Fishkin
[CXL Live 16] Fight Back Against Back by Rand Fishkin
 
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to Ask
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to AskEverything You Wanted to Know About LinkedIn, but Were Too Embarrassed to Ask
Everything You Wanted to Know About LinkedIn, but Were Too Embarrassed to Ask
 
Google Analytics Fundamentals: Set Up and Basics for Measurement
Google Analytics Fundamentals: Set Up and Basics for MeasurementGoogle Analytics Fundamentals: Set Up and Basics for Measurement
Google Analytics Fundamentals: Set Up and Basics for Measurement
 
Digital content best_practices
Digital content best_practicesDigital content best_practices
Digital content best_practices
 
Growth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-YoungGrowth Hacking with Cassie Lancellotti-Young
Growth Hacking with Cassie Lancellotti-Young
 
Why It's Time to Kill the Cold Call Once and For All
Why It's Time to Kill the Cold Call Once and For AllWhy It's Time to Kill the Cold Call Once and For All
Why It's Time to Kill the Cold Call Once and For All
 
How to Run a Lean Startup
How to Run a Lean StartupHow to Run a Lean Startup
How to Run a Lean Startup
 
ACH Marketing Presentation
ACH Marketing PresentationACH Marketing Presentation
ACH Marketing Presentation
 
Turning interviews into content
Turning interviews into contentTurning interviews into content
Turning interviews into content
 
Partnering With Others
Partnering With Others Partnering With Others
Partnering With Others
 
Content Marketing Lessons From 10 Years and 41,525,103 Website Visits
Content Marketing Lessons From 10 Years and 41,525,103 Website VisitsContent Marketing Lessons From 10 Years and 41,525,103 Website Visits
Content Marketing Lessons From 10 Years and 41,525,103 Website Visits
 
The Journey from Stranger to Customer
The Journey from Stranger to CustomerThe Journey from Stranger to Customer
The Journey from Stranger to Customer
 
Driving Growth Through Email Marketing
Driving Growth Through Email MarketingDriving Growth Through Email Marketing
Driving Growth Through Email Marketing
 
Growing Moz: 8 Lessons Learned
Growing Moz: 8 Lessons LearnedGrowing Moz: 8 Lessons Learned
Growing Moz: 8 Lessons Learned
 
What is email list building. 100 fast list building methods and techniques
What is email list building. 100 fast list building methods and techniquesWhat is email list building. 100 fast list building methods and techniques
What is email list building. 100 fast list building methods and techniques
 
article
articlearticle
article
 

Viewers also liked

No excuses user research
No excuses user researchNo excuses user research
No excuses user researchLily Dart
 
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...GeekWire
 
Wireframes - a brief overview
Wireframes - a brief overviewWireframes - a brief overview
Wireframes - a brief overviewJenni Leder
 
SQL Tutorial for Marketers
SQL Tutorial for MarketersSQL Tutorial for Marketers
SQL Tutorial for MarketersJustin Mares
 
Using Your Growth Model to Drive Smarter High Tempo Testing
Using Your Growth Model to Drive Smarter High Tempo TestingUsing Your Growth Model to Drive Smarter High Tempo Testing
Using Your Growth Model to Drive Smarter High Tempo TestingSean Ellis
 
10 Ways You're Using AdWords Wrong and How to Correct Those Practices
10 Ways You're Using AdWords Wrong and How to Correct Those Practices 10 Ways You're Using AdWords Wrong and How to Correct Those Practices
10 Ways You're Using AdWords Wrong and How to Correct Those Practices Kissmetrics on SlideShare
 
A Guide to User Research (for People Who Don't Like Talking to Other People)
A Guide to User Research (for People Who Don't Like Talking to Other People)A Guide to User Research (for People Who Don't Like Talking to Other People)
A Guide to User Research (for People Who Don't Like Talking to Other People)Stephanie Wills
 
10 Mobile Marketing Campaigns That Went Viral and Made Millions
10 Mobile Marketing Campaigns That Went Viral and Made Millions10 Mobile Marketing Campaigns That Went Viral and Made Millions
10 Mobile Marketing Campaigns That Went Viral and Made MillionsMark Fidelman
 
Mastering Google Adwords In 30 Minutes
Mastering Google Adwords In 30 MinutesMastering Google Adwords In 30 Minutes
Mastering Google Adwords In 30 MinutesNik Cree
 
Lean Community Building: Getting the Most Bang for Your Time & Money
Lean Community Building: Getting the Most Bang for  Your Time & MoneyLean Community Building: Getting the Most Bang for  Your Time & Money
Lean Community Building: Getting the Most Bang for Your Time & MoneyJennifer Lopez
 
The Science behind Viral marketing
The Science behind Viral marketingThe Science behind Viral marketing
The Science behind Viral marketingDavid Skok
 
Biz Dev 101 - An Interactive Workshop on How Deals Get Done
Biz Dev 101 - An Interactive Workshop on How Deals Get DoneBiz Dev 101 - An Interactive Workshop on How Deals Get Done
Biz Dev 101 - An Interactive Workshop on How Deals Get DoneScott Pollack
 
The Essentials of Community Building by Mack Fogelson
The Essentials of Community Building by Mack FogelsonThe Essentials of Community Building by Mack Fogelson
The Essentials of Community Building by Mack FogelsonMackenzie Fogelson
 
LinkedIn Ads Platform Master Class
LinkedIn Ads Platform Master ClassLinkedIn Ads Platform Master Class
LinkedIn Ads Platform Master ClassLinkedIn
 
Optimize Your Sales & Marketing Funnel
Optimize Your Sales & Marketing FunnelOptimize Your Sales & Marketing Funnel
Optimize Your Sales & Marketing FunnelHubSpot
 
How to: Viral Marketing + Brand Storytelling
How to: Viral Marketing + Brand Storytelling How to: Viral Marketing + Brand Storytelling
How to: Viral Marketing + Brand Storytelling Elle Shelley
 
User experience doesn't happen on a screen: It happens in the mind.
User experience doesn't happen on a screen: It happens in the mind.User experience doesn't happen on a screen: It happens in the mind.
User experience doesn't happen on a screen: It happens in the mind.John Whalen
 
How Top Brands Use Referral Programs to Drive Customer Acquisition
How Top Brands Use Referral Programs to Drive Customer AcquisitionHow Top Brands Use Referral Programs to Drive Customer Acquisition
How Top Brands Use Referral Programs to Drive Customer AcquisitionKissmetrics on SlideShare
 

Viewers also liked (20)

No excuses user research
No excuses user researchNo excuses user research
No excuses user research
 
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...
Brenda Spoonemore - A biz dev playbook for startups: Why, when and how to do ...
 
Wireframes - a brief overview
Wireframes - a brief overviewWireframes - a brief overview
Wireframes - a brief overview
 
SQL Tutorial for Marketers
SQL Tutorial for MarketersSQL Tutorial for Marketers
SQL Tutorial for Marketers
 
Using Your Growth Model to Drive Smarter High Tempo Testing
Using Your Growth Model to Drive Smarter High Tempo TestingUsing Your Growth Model to Drive Smarter High Tempo Testing
Using Your Growth Model to Drive Smarter High Tempo Testing
 
HTML & CSS Masterclass
HTML & CSS MasterclassHTML & CSS Masterclass
HTML & CSS Masterclass
 
10 Ways You're Using AdWords Wrong and How to Correct Those Practices
10 Ways You're Using AdWords Wrong and How to Correct Those Practices 10 Ways You're Using AdWords Wrong and How to Correct Those Practices
10 Ways You're Using AdWords Wrong and How to Correct Those Practices
 
A Guide to User Research (for People Who Don't Like Talking to Other People)
A Guide to User Research (for People Who Don't Like Talking to Other People)A Guide to User Research (for People Who Don't Like Talking to Other People)
A Guide to User Research (for People Who Don't Like Talking to Other People)
 
10 Mobile Marketing Campaigns That Went Viral and Made Millions
10 Mobile Marketing Campaigns That Went Viral and Made Millions10 Mobile Marketing Campaigns That Went Viral and Made Millions
10 Mobile Marketing Campaigns That Went Viral and Made Millions
 
Mastering Google Adwords In 30 Minutes
Mastering Google Adwords In 30 MinutesMastering Google Adwords In 30 Minutes
Mastering Google Adwords In 30 Minutes
 
Intro to Facebook Ads
Intro to Facebook AdsIntro to Facebook Ads
Intro to Facebook Ads
 
Lean Community Building: Getting the Most Bang for Your Time & Money
Lean Community Building: Getting the Most Bang for  Your Time & MoneyLean Community Building: Getting the Most Bang for  Your Time & Money
Lean Community Building: Getting the Most Bang for Your Time & Money
 
The Science behind Viral marketing
The Science behind Viral marketingThe Science behind Viral marketing
The Science behind Viral marketing
 
Biz Dev 101 - An Interactive Workshop on How Deals Get Done
Biz Dev 101 - An Interactive Workshop on How Deals Get DoneBiz Dev 101 - An Interactive Workshop on How Deals Get Done
Biz Dev 101 - An Interactive Workshop on How Deals Get Done
 
The Essentials of Community Building by Mack Fogelson
The Essentials of Community Building by Mack FogelsonThe Essentials of Community Building by Mack Fogelson
The Essentials of Community Building by Mack Fogelson
 
LinkedIn Ads Platform Master Class
LinkedIn Ads Platform Master ClassLinkedIn Ads Platform Master Class
LinkedIn Ads Platform Master Class
 
Optimize Your Sales & Marketing Funnel
Optimize Your Sales & Marketing FunnelOptimize Your Sales & Marketing Funnel
Optimize Your Sales & Marketing Funnel
 
How to: Viral Marketing + Brand Storytelling
How to: Viral Marketing + Brand Storytelling How to: Viral Marketing + Brand Storytelling
How to: Viral Marketing + Brand Storytelling
 
User experience doesn't happen on a screen: It happens in the mind.
User experience doesn't happen on a screen: It happens in the mind.User experience doesn't happen on a screen: It happens in the mind.
User experience doesn't happen on a screen: It happens in the mind.
 
How Top Brands Use Referral Programs to Drive Customer Acquisition
How Top Brands Use Referral Programs to Drive Customer AcquisitionHow Top Brands Use Referral Programs to Drive Customer Acquisition
How Top Brands Use Referral Programs to Drive Customer Acquisition
 

Similar to Understand A/B Testing in 9 use cases & 7 mistakes

Taming The HiPPO
Taming The HiPPOTaming The HiPPO
Taming The HiPPOGoogle A/NZ
 
A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsPirate Skills
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experimentsDeven
 
9 CRO Hacks to Accelerate the B2B Funnel
9 CRO Hacks to Accelerate the B2B Funnel9 CRO Hacks to Accelerate the B2B Funnel
9 CRO Hacks to Accelerate the B2B FunnelDemandWave
 
How to build an effective conversion program on a budget
 How to build an effective conversion program on a budget  How to build an effective conversion program on a budget
How to build an effective conversion program on a budget Joni Lindgren
 
Web analytics 101: Optimization
Web analytics 101: OptimizationWeb analytics 101: Optimization
Web analytics 101: OptimizationSociety_Consulting
 
Milion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B TestingMilion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B TestingAzhar Bandeali
 
Data-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMData-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMProduct School
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingJack Nguyen (Hung Tien)
 
DM 201: Testing Creative - March 29, 2017
DM 201: Testing Creative - March 29, 2017DM 201: Testing Creative - March 29, 2017
DM 201: Testing Creative - March 29, 2017Avalon Consulting
 
Ab testing for_beginners
Ab testing for_beginnersAb testing for_beginners
Ab testing for_beginnersLiberteks
 
Getting Started with A/B Testing
Getting Started with A/B TestingGetting Started with A/B Testing
Getting Started with A/B TestingAmy
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementBen Dressler
 

Similar to Understand A/B Testing in 9 use cases & 7 mistakes (20)

Taming The HiPPO
Taming The HiPPOTaming The HiPPO
Taming The HiPPO
 
A/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate SkillsA/B Testing Blueprint | Pirate Skills
A/B Testing Blueprint | Pirate Skills
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experiments
 
The Power of A/B Testing
The Power of A/B TestingThe Power of A/B Testing
The Power of A/B Testing
 
Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10Hong Kong Web Analytics Wednesday #10
Hong Kong Web Analytics Wednesday #10
 
9 CRO Hacks to Accelerate the B2B Funnel
9 CRO Hacks to Accelerate the B2B Funnel9 CRO Hacks to Accelerate the B2B Funnel
9 CRO Hacks to Accelerate the B2B Funnel
 
How to build an effective conversion program on a budget
 How to build an effective conversion program on a budget  How to build an effective conversion program on a budget
How to build an effective conversion program on a budget
 
Realise the Benefits of Holistic Testing
Realise the Benefits of Holistic TestingRealise the Benefits of Holistic Testing
Realise the Benefits of Holistic Testing
 
Web analytics 101: Optimization
Web analytics 101: OptimizationWeb analytics 101: Optimization
Web analytics 101: Optimization
 
Milion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B TestingMilion Dollar Impact Through Metrics, Analytics & A/B Testing
Milion Dollar Impact Through Metrics, Analytics & A/B Testing
 
Data-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PMData-Driven Decision Making by Expedia Sr PM
Data-Driven Decision Making by Expedia Sr PM
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
 
DM 201: Testing Creative - March 29, 2017
DM 201: Testing Creative - March 29, 2017DM 201: Testing Creative - March 29, 2017
DM 201: Testing Creative - March 29, 2017
 
Ab testing
Ab testingAb testing
Ab testing
 
Ab testing for_beginners_
Ab testing for_beginners_Ab testing for_beginners_
Ab testing for_beginners_
 
Ab testing for_beginners
Ab testing for_beginnersAb testing for_beginners
Ab testing for_beginners
 
Getting Started with A/B Testing
Getting Started with A/B TestingGetting Started with A/B Testing
Getting Started with A/B Testing
 
Ab Testing Guide for Beginners
Ab Testing Guide for Beginners Ab Testing Guide for Beginners
Ab Testing Guide for Beginners
 
Ab testing for_beginners
Ab testing for_beginnersAb testing for_beginners
Ab testing for_beginners
 
Creating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvementCreating a culture that provokes failure and boosts improvement
Creating a culture that provokes failure and boosts improvement
 

More from TheFamily

Building a design culture from day one
Building a design culture from day oneBuilding a design culture from day one
Building a design culture from day oneTheFamily
 
Individual Contributors vs Managers
Individual Contributors vs ManagersIndividual Contributors vs Managers
Individual Contributors vs ManagersTheFamily
 
Build the decentralized team you ever dreamed of
Build the decentralized team you ever dreamed ofBuild the decentralized team you ever dreamed of
Build the decentralized team you ever dreamed ofTheFamily
 
CEOs best practices to win time back & focus on what matters
CEOs best practices to win time back & focus on what mattersCEOs best practices to win time back & focus on what matters
CEOs best practices to win time back & focus on what mattersTheFamily
 
Managing fully remote teams
Managing fully remote teamsManaging fully remote teams
Managing fully remote teamsTheFamily
 
State of European Tech by Atomico
State of European Tech by AtomicoState of European Tech by Atomico
State of European Tech by AtomicoTheFamily
 
Building a real estate startup
Building a real estate startupBuilding a real estate startup
Building a real estate startupTheFamily
 
A VC view on Enterprise Sales
A VC view on Enterprise SalesA VC view on Enterprise Sales
A VC view on Enterprise SalesTheFamily
 
Find your style and create emotions
Find your style and create emotionsFind your style and create emotions
Find your style and create emotionsTheFamily
 
From product to ecosystem
From product to ecosystemFrom product to ecosystem
From product to ecosystemTheFamily
 
Demystifying the product black box
Demystifying the product black boxDemystifying the product black box
Demystifying the product black boxTheFamily
 
The secrets to create bank brand love
The secrets to create bank brand loveThe secrets to create bank brand love
The secrets to create bank brand loveTheFamily
 
Building an insurance startup with Alan, Luko, Coverd & Balderton
Building an insurance startup with Alan, Luko, Coverd & BaldertonBuilding an insurance startup with Alan, Luko, Coverd & Balderton
Building an insurance startup with Alan, Luko, Coverd & BaldertonTheFamily
 
Mixing Product & Tech by Jean Lebrument, CTO & CPO at Brigad
Mixing Product & Tech by Jean Lebrument, CTO & CPO at BrigadMixing Product & Tech by Jean Lebrument, CTO & CPO at Brigad
Mixing Product & Tech by Jean Lebrument, CTO & CPO at BrigadTheFamily
 
A new breed of CTO - Philippe Vimard, CTO & COO at Doctolib
A new breed of CTO - Philippe Vimard, CTO & COO at DoctolibA new breed of CTO - Philippe Vimard, CTO & COO at Doctolib
A new breed of CTO - Philippe Vimard, CTO & COO at DoctolibTheFamily
 
Building a logistics startup  with Trusk, Totem & SpaceFill
Building a logistics startup  with Trusk, Totem & SpaceFillBuilding a logistics startup  with Trusk, Totem & SpaceFill
Building a logistics startup  with Trusk, Totem & SpaceFillTheFamily
 
Building an accounting startup with Fred de la compta, Acasi & Chaintrust
Building an accounting startup with Fred de la compta, Acasi & ChaintrustBuilding an accounting startup with Fred de la compta, Acasi & Chaintrust
Building an accounting startup with Fred de la compta, Acasi & ChaintrustTheFamily
 
Scale your tech team from 0 to Series A
Scale your tech team from 0 to Series A Scale your tech team from 0 to Series A
Scale your tech team from 0 to Series A TheFamily
 
Onboarding developers and setting them up for success
Onboarding developers and setting them up for successOnboarding developers and setting them up for success
Onboarding developers and setting them up for successTheFamily
 
Apprendre à penser comme un journaliste
Apprendre à penser comme un journalisteApprendre à penser comme un journaliste
Apprendre à penser comme un journalisteTheFamily
 

More from TheFamily (20)

Building a design culture from day one
Building a design culture from day oneBuilding a design culture from day one
Building a design culture from day one
 
Individual Contributors vs Managers
Individual Contributors vs ManagersIndividual Contributors vs Managers
Individual Contributors vs Managers
 
Build the decentralized team you ever dreamed of
Build the decentralized team you ever dreamed ofBuild the decentralized team you ever dreamed of
Build the decentralized team you ever dreamed of
 
CEOs best practices to win time back & focus on what matters
CEOs best practices to win time back & focus on what mattersCEOs best practices to win time back & focus on what matters
CEOs best practices to win time back & focus on what matters
 
Managing fully remote teams
Managing fully remote teamsManaging fully remote teams
Managing fully remote teams
 
State of European Tech by Atomico
State of European Tech by AtomicoState of European Tech by Atomico
State of European Tech by Atomico
 
Building a real estate startup
Building a real estate startupBuilding a real estate startup
Building a real estate startup
 
A VC view on Enterprise Sales
A VC view on Enterprise SalesA VC view on Enterprise Sales
A VC view on Enterprise Sales
 
Find your style and create emotions
Find your style and create emotionsFind your style and create emotions
Find your style and create emotions
 
From product to ecosystem
From product to ecosystemFrom product to ecosystem
From product to ecosystem
 
Demystifying the product black box
Demystifying the product black boxDemystifying the product black box
Demystifying the product black box
 
The secrets to create bank brand love
The secrets to create bank brand loveThe secrets to create bank brand love
The secrets to create bank brand love
 
Building an insurance startup with Alan, Luko, Coverd & Balderton
Building an insurance startup with Alan, Luko, Coverd & BaldertonBuilding an insurance startup with Alan, Luko, Coverd & Balderton
Building an insurance startup with Alan, Luko, Coverd & Balderton
 
Mixing Product & Tech by Jean Lebrument, CTO & CPO at Brigad
Mixing Product & Tech by Jean Lebrument, CTO & CPO at BrigadMixing Product & Tech by Jean Lebrument, CTO & CPO at Brigad
Mixing Product & Tech by Jean Lebrument, CTO & CPO at Brigad
 
A new breed of CTO - Philippe Vimard, CTO & COO at Doctolib
A new breed of CTO - Philippe Vimard, CTO & COO at DoctolibA new breed of CTO - Philippe Vimard, CTO & COO at Doctolib
A new breed of CTO - Philippe Vimard, CTO & COO at Doctolib
 
Building a logistics startup  with Trusk, Totem & SpaceFill
Building a logistics startup  with Trusk, Totem & SpaceFillBuilding a logistics startup  with Trusk, Totem & SpaceFill
Building a logistics startup  with Trusk, Totem & SpaceFill
 
Building an accounting startup with Fred de la compta, Acasi & Chaintrust
Building an accounting startup with Fred de la compta, Acasi & ChaintrustBuilding an accounting startup with Fred de la compta, Acasi & Chaintrust
Building an accounting startup with Fred de la compta, Acasi & Chaintrust
 
Scale your tech team from 0 to Series A
Scale your tech team from 0 to Series A Scale your tech team from 0 to Series A
Scale your tech team from 0 to Series A
 
Onboarding developers and setting them up for success
Onboarding developers and setting them up for successOnboarding developers and setting them up for success
Onboarding developers and setting them up for success
 
Apprendre à penser comme un journaliste
Apprendre à penser comme un journalisteApprendre à penser comme un journaliste
Apprendre à penser comme un journaliste
 

Recently uploaded

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 

Recently uploaded (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 

Understand A/B Testing in 9 use cases & 7 mistakes

  • 2. Growth Hacking Meetup - Episode 4 - 9 A/B Testing Use Cases
  • 3. But first… some A/B Testing mistakes startups make ALL the time
  • 4. 1 1 A/B tests are ended too early
 Statistical significance (large sample size, etc) is what tells you version A is actually better than version B
  • 5. As an optimizer, your job 
 is to figure out the truth. 
 You have to put your ego aside.
  • 6. 2 1 Tests are not run for full weeks
 Conversion rates can vary greatly depending on the day of the week. Always test a full week at a time.
  • 7. 3 1 Always pay attention to external factors
 Is it Christmas? External factors definitely affect your test results. When in doubt, run a follow-up test.
  • 8. 4 1 Testing random ideas without a hypothesis You’re wasting precious time and traffic. Never do that.
  • 9. By the way, if you want to test button colors “green vs orange” is not the essence of A/B testing. 
 It’s about understanding the target audience.
  • 10. Use your traffic on high-impact stuff. 
 Test data-driven hypotheses.
  • 11. 5 1 They give up after the first test fails
 Run a follow-up test, learn from it, and improve your hypotheses.
  • 12. “I have not failed 10,000 times. 
 I have successfully found 10,000 ways that will not work.” - Thomas Alva Edison, inventor
  • 13. 6 1 They’re ignoring small gains 
 Only 1 out of 8 A/B tests drive 
 significant change.
  • 14. 7 1 They’re not running tests at all times
 Every single day without a test is a wasted day. Testing is learning.
  • 15. Now, for the second part… 
 12 Surprising A/B Test Results to Stop You Making Assumptions
  • 17. Test 1: Which Copy Increased Trial Sign-Ups?
  • 18. Result In this test, Version B increased sign-ups by 38% – a big rise. However, you would think version A was the better design.
  • 19. Why does Version B work? 
 Simply because the copy is better. Lesson: landing pages don’t have to be pretty to be effective.
  • 20. Test 2: Which Landing Page Got 24% More Lead Generation Form Submissions? Picture vs No picture
  • 21. Result Surprisingly, Version A was the page that got the 24% increase in submissions, simply by removing the image from the page.
  • 22. This is a great example of why you should confirm your assumptions with quantitative testing.
  • 23. Test 3: Which Radically Redesigned Form Increased B2B Leads By 368.5%?
  • 24. Result Version A is an obvious winner, but it’s not just the big red button that makes the difference. Version A keeps things really tight and uses grouping to visually shrink the impact of the form
  • 25. When designing your landing page, don’t overestimate your user’s tolerance, goodwill, and patience.
  • 26. Test 4: Does Matching Headline & Body Copy to Your Initial Ad Copy Really Matter?
  • 27. Result Version B looks as if it should be better: the headline copy is snappier, the sub-head clearer, but in tests version A increased leads by 115%.
  • 28. Why? Version A was designed to tie in with the ads that drive users to the page. 
 Lesson: making the elements of the sales funnel work together increases efficacy.
  • 29. Test 5: Which Landing Page Increased Mortgage Application Leads by 64%?
  • 30. Result Video can be a very effective tool in communicating lots of information in a compact form. But the presence of video couldn’t save Version B; Version A increased leads by 64%.
  • 31. Test 6: Does adding testimonials to your Homepage increase Sales?
  • 32. Result It would seem that this would have very little effect, but in practice this small change increased sales by 34% – a big margin. Why is this? Having ‘social proof’, even in this basic form, humanizes the conversion experience
  • 33. Test 7: Does Social Proof Increase Conversions? ‘Big Brand Prices’ vs. Consumer Ratings
  • 34. Result ‘Social proof’ is a powerful tool that can have a demonstrable effect on conversion outcomes. But in this case, customers were just looking for the cheapest offer. Asking them to consider an additional piece of information – customer satisfaction – made them back away from the conversion.
  • 35. Test 8: Does an Email Security Seal Help or Hinder Lead Generation Form Completions?
  • 36. Result Actually, users were put off by seeing it in this context, assuming that they were about to pay for something
  • 37. Test 9: Which Page Got an 89.8% Lift in Free Account Sign Ups?
  • 38. Result Version B wins: It has three bullet points, as opposed to words in speech bubbles. Removing a distraction from the page and reducing the risk of users navigating away from the page worked better
  • 39. Whether you are right or wrong: 
 First, make sure you get the basics right before you start testing, and second, always be testing, because unless you test, you can never be absolutely sure.
  • 40. You are Welcome to Join the Adventure hey@thefamily.co