Downloaded 106 times
Uploaded byPete Mauro
A/B Testing for Lean Startups
This document discusses A/B testing and its importance for lean startups. It provides examples of companies that conducted A/B tests, including Tropicana's redesign of its juice box packaging that resulted in a 20% drop in sales and cost $33 million. The presentation emphasizes that lean startups should be data-driven, listen to customers, and take a scientific approach using the scientific method of hypothesis testing. It also provides tips on designing effective A/B tests and names several A/B testing tools.
More Related Content
![[CXL Live 16] Using Urgency to Boost E-commerce Conversions by Viljo Vabrit](https://cdn.slidesharecdn.com/ss_thumbnails/idrzpyvjsocrxbmdxh20-signature-42db88fde2139044dd6963018b134da7e709739587431e34ce8d4285185c8547-poli-160403200011-thumbnail.jpg?width=640&height=640&fit=bounds)
![[500DISTRO] Dissecting UX: Funnel Fixes & Other CRO Tips for Growth That Pays](https://cdn.slidesharecdn.com/ss_thumbnails/05neilpatelkiss-final-140806211641-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to A/B Testing for Lean Startups
A/B Testing for Lean Startups
- 1. A/B Testing forLean Startups Everything you ever Secrets from the wanted to know experts revealed about split testing. Watch Presentation Learn More Pete Mauro @mauronic Mucker Labs, February 27, 2012
- 3. In 2009 Tropicanawanted a new juice box
- 4. So they redesignedthe package New box launched in thousands of stores nationwide
- 5. How did itgo? • “Ugly, stupid, cheap and generic looking” • Ad agency defends it • Sales plunge 20% • Agency CEO fired • Total cost: $33 Million
- 6. 8 Months + $33Million Dollars = This Package Sucks
- 7.
- 8. Lean Startups areData Driven
- 9. Lean Startups Listento their Customers
- 10. Lean Startups areScientific
- 11. The Scientific Method Ask Question Do research Hypothesis Analyze Data Conclusion
- 12. Believe it ornot, some people leave Facebook
- 13. Let’s pretend wework at Facebook Ask Question How can we stop more people from deactivating their Facebook account? Do research The reason why people use Facebook is because their friends are on Facebook. Hypothesis If you remind people about their friends who are being left behind, we will lower the deactivation rate.
- 14. Facebook’s Actual A/BTest Control Experiment Source: Julie Zhou, Facebook
- 15. The guilt tripworked Control Experiment ✔ Reduced Deactivation Rate by 7% Source: Julie Zhou, Facebook
- 16. Quiz: Guess theWinner A B Source: http://www.abtests.com/
- 17. 5 Quiz: Guess the Winner Points A B ✔ 80% improvement on sign ups Source: http://www.abtests.com/
- 18. Quiz: Guess theWinner A B Source: http://www.abtests.com/
- 19. 10 Quiz: Guess the Winner Points A B Source: http://www.abtests.com/ ✔ +60% signups
- 20. Designing a Test • Anyone at the company should contribute • Ideas can come from user testing • Use best practices • Be creative, try crazy ideas • Use common sense: don’t piss off your users for the sake of metrics
- 21. The 41 Shadesof Blue Test at Google Link Color
- 22. The Local Maxima Source:Joshua Porter
- 23. A/B Testing Tools • Kissmetrics • Visual Website Optimizer • Optimizely • Unbounce • Mailchimp (email campaigns) • Google Website Optimizer • And more…
- 24.
- 25. Further Reading • MetricsDriven Design (See Joshua Porter on Slideshare) • The New Science of Product Development (See James Birchler on Slideshare) • Quora: “What are the most unexpected things people have learned from A/B tests?” • Abtests.com & whichtestwon.com
- 26. Thanks! Follow me I’m on Twitter @mauronic @mauronic
Editor's Notes
- #4 The marketing people felt that they needed to modernize their imageTropicana hired Arnell Group, a subsidiary of Omnicom Group, to reimagine the packaging
- #5 His team spent 6 months and tens of thousands of dollars to develop this new package
- #6 Customers hated itAd agency CEO defends design in a press conference saying “emotionally it is very difficult for people to grasp this bold new change”Two moths after introduction its pulled from the shelves
- #9 Eric Ries’ big ideaProgress is not measured by what you make, but what you learn.Goal: spin through this loop as fast as possible
- #10 Steve blank says get out of the buildingYour opinion doesn’t matter, your customer’s does
- #12 Just because we use this method does not be that we lack vision or creativity
- #13 Lets look at a case study
- #14 Your hypothesis must be measurable
- #16 Millions of users retainedFacebook likes to say “the code wins”. What this really means is let’s stop arguing about what we think is right and test it with our users.
- #18 Kiss metrics video anecdote
- #19 Can you even tell what the difference is?
- #20 Just about every conversion goal that tracked showed a dramatic improvement. Signups for free trial: +60%Engagement rate (inverse of bounce rate): +20%Visit to signup page: +30%Visit to homepage: +13%Visit to blog main page: +78%
- #21 Tests are powerful because opinions don’t matter anymore Don’t piss off users: Appsumo pop-up, signup on blog post, ipad app download interstitial on
- #22 This is testing gone astray~ 2.5% of users each got a shade for 2 weeksDoug Bowman cited this test as epitomizing the reason why he left as head designer at google
- #23 Don’t get caught in a micro-optimization loopBig gains come from bold changes
- #24 Nowadays there is no excuse not to A/B testEach of these sites have great resources on things like landing page best practices
- #25 Web based testing software is ridiculously easy to use now