This document discusses common mistakes companies make with A/B testing and provides the results of 9 A/B tests. Some key mistakes with A/B testing are ending tests too early, not running tests for full weeks, and testing random ideas without hypotheses. The 9 A/B test results showed that assumptions should not be made without testing, and things like removing images, simplifying forms, and adding social proof can significantly increase conversions contrary to initial expectations. The document emphasizes that testing is important for learning what actually works best rather than making assumptions.

1. Many thanks to:

2. Growth Hacking Meetup
- Episode 4 -
9 A/B Testing Use Cases

3. But first… some A/B Testing
mistakes startups make ALL the time

4. 1
1
A/B tests are ended too early
Statistical significance (large sample size, etc) is what
tells you version A is actually better than version B

5. As an optimizer, your job
is to figure out the truth.
You have to put your ego aside.

6. 2
1
Tests are not run for full weeks
Conversion rates can vary greatly depending on the
day of the week. Always test a full week at a time.

7. 3
1
Always pay attention to external factors
Is it Christmas? External factors definitely affect your
test results. When in doubt, run a follow-up test.

8. 4
1
Testing random ideas without a hypothesis
You’re wasting precious time and traffic.
Never do that.

9. By the way, if you want to test button colors
“green vs orange” is not the essence of A/B testing.
It’s about understanding the target audience.

10. Use your traffic on high-impact stuff.
Test data-driven hypotheses.

11. 5
1
They give up after the first test fails
Run a follow-up test, learn from it, and
improve your hypotheses.

12. “I have not failed 10,000 times.
I have successfully found 10,000 ways that will not work.”
- Thomas Alva Edison, inventor

13. 6
1
They’re ignoring small gains
Only 1 out of 8 A/B tests drive
significant change.

14. 7
1
They’re not running tests at all times
Every single day without a test is a wasted day.
Testing is learning.

15. Now, for the second part…
12 Surprising A/B Test Results to
Stop You Making Assumptions

17. Test 1: Which Copy Increased Trial Sign-Ups?

18. Result
In this test, Version B increased
sign-ups by 38% – a big rise.
However, you would think version
A was the better design.

19. Why does Version B work?
Simply because the copy is better.
Lesson: landing pages don’t have to
be pretty to be effective.

20. Test 2: Which Landing Page Got 24% More
Lead Generation Form Submissions?
Picture vs No picture

21. Result
Surprisingly, Version A was the
page that got the 24% increase in
submissions, simply by removing
the image from the page.

22. This is a great example of why you should confirm
your assumptions with quantitative testing.

23. Test 3: Which Radically Redesigned Form
Increased B2B Leads By 368.5%?

24. Result
Version A is an obvious winner, but it’s not just
the big red button that makes the difference.
Version A keeps things really tight and uses
grouping to visually shrink the impact of the form

25. When designing your landing page,
don’t overestimate your user’s
tolerance, goodwill, and patience.

26. Test 4: Does Matching Headline & Body Copy
to Your Initial Ad Copy Really Matter?

27. Result
Version B looks as if it should be better: the
headline copy is snappier, the sub-head clearer,
but in tests version A increased leads by 115%.

28. Why? Version A was designed to tie in with the ads
that drive users to the page.
Lesson: making the elements of the sales funnel
work together increases efficacy.

29. Test 5: Which Landing Page Increased
Mortgage Application Leads by 64%?

30. Result
Video can be a very effective tool in
communicating lots of information in a compact
form. But the presence of video couldn’t save
Version B; Version A increased leads by 64%.

31. Test 6: Does adding testimonials to your
Homepage increase Sales?

32. Result
It would seem that this would have very little
effect, but in practice this small change
increased sales by 34% – a big margin.
Why is this? Having ‘social proof’, even in this
basic form, humanizes the conversion experience

33. Test 7: Does Social Proof Increase Conversions?
‘Big Brand Prices’ vs. Consumer Ratings

34. Result
‘Social proof’ is a powerful tool that can have a
demonstrable effect on conversion outcomes.
But in this case, customers were just looking for the
cheapest offer. Asking them to consider an additional
piece of information – customer satisfaction – made
them back away from the conversion.

35. Test 8: Does an Email Security Seal Help or
Hinder Lead Generation Form Completions?

36. Result
Actually, users were put off by seeing it
in this context, assuming that they were
about to pay for something

37. Test 9: Which Page Got an 89.8% Lift in
Free Account Sign Ups?

38. Result
Version B wins: It has three bullet points,
as opposed to words in speech bubbles.
Removing a distraction from the page and
reducing the risk of users navigating away
from the page worked better

39. Whether you are right or wrong:
First, make sure you get the basics right before you
start testing, and second, always be testing, because
unless you test, you can never be absolutely sure.

40. You are Welcome to Join the Adventure
hey@thefamily.co