This document discusses A/B testing and its importance for lean startups. It provides examples of companies that conducted A/B tests, including Tropicana's redesign of its juice box packaging that resulted in a 20% drop in sales and cost $33 million. The presentation emphasizes that lean startups should be data-driven, listen to customers, and take a scientific approach using the scientific method of hypothesis testing. It also provides tips on designing effective A/B tests and names several A/B testing tools.

1. A/B Testing for Lean Startups
Everything you ever Secrets from the
wanted to know experts revealed
about split testing.
Watch Presentation Learn More
Pete Mauro @mauronic
Mucker Labs, February 27, 2012

3. In 2009 Tropicana wanted a new juice box

4. So they redesigned the package
New box launched in
thousands of stores
nationwide

5. How did it go?
• “Ugly, stupid, cheap and
generic looking”
• Ad agency defends it
• Sales plunge 20%
• Agency CEO fired
• Total cost: $33 Million

6. 8 Months +
$33 Million Dollars =
This Package Sucks

7. Very un-lean, dude

8. Lean Startups are Data Driven

9. Lean Startups Listen to their Customers

10. Lean Startups are Scientific

11. The Scientific Method
Ask Question
Do research
Hypothesis
Analyze Data
Conclusion

12. Believe it or not, some people leave Facebook

13. Let’s pretend we work at Facebook
Ask Question How can we stop more people from deactivating their
Facebook account?
Do research The reason why people use Facebook is because their friends
are on Facebook.
Hypothesis If you remind people about their friends who are being left
behind, we will lower the deactivation rate.

14. Facebook’s Actual A/B Test
Control
Experiment
Source: Julie Zhou, Facebook

15. The guilt trip worked
Control
Experiment
✔
Reduced Deactivation Rate by 7%
Source: Julie Zhou, Facebook

16. Quiz: Guess the Winner
A B
Source: http://www.abtests.com/

17. 5
Quiz: Guess the Winner Points
A B
✔
80% improvement on sign ups
Source: http://www.abtests.com/

18. Quiz: Guess the Winner
A
B
Source: http://www.abtests.com/

19. 10
Quiz: Guess the Winner Points
A
B
Source: http://www.abtests.com/
✔ +60% signups

20. Designing a Test
• Anyone at the company should contribute
• Ideas can come from user testing
• Use best practices
• Be creative, try crazy ideas
• Use common sense: don’t piss off your users
for the sake of metrics

21. The 41 Shades of Blue Test at Google
Link Color

22. The Local Maxima
Source: Joshua Porter

23. A/B Testing Tools
• Kissmetrics
• Visual Website Optimizer
• Optimizely
• Unbounce
• Mailchimp (email campaigns)
• Google Website Optimizer
• And more…

24. Mailchimp Split Test

25. Further Reading
• Metrics Driven Design (See Joshua Porter on
Slideshare)
• The New Science of Product Development
(See James Birchler on Slideshare)
• Quora: “What are the most unexpected things
people have learned from A/B tests?”
• Abtests.com & whichtestwon.com

26. Thanks!
Follow me I’m on Twitter
@mauronic @mauronic