Seriously Advanced A/B Testing by Wyatt Jenkins


Published on

Seriously Advanced A/B Testing by Wyatt Jenkins

The Lean Startup Conference 2013

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Growth = ExperimentationI’m going to make you better at testing – whether you are at a big company or a small, I’ve got some information about testing that can help.Testing & Experimentation is important – we take it very seriouslyHere’s a few things that can help
  • Your product is made up of a bunch of interactions with real people who value the service you provide. Your product is data. The more of this data you have, the faster you achieve confidence on tests. ** Different companies in the audience will be of different sizes, but for Shutterstock in a highly commoditized industry, data is everything.
  • 1TB per day3 downloads a second50
  • Edimology of smoke tests – you plug in rado
  • Why does this matter? Sometimes it’s fun to put things on a scale to show differences in opinion. Of course none of these brands would appreciate me doing this, but let’s check it out to prove a point. Problem to solve – Unclear lines between marketing and productMarketing design not yet good at conversionMarketing assets look different than websiteHow do we create a testing culture at Shutterstock
  • Leadership team isn’t involved in tactics of testing.
  • This comes into effect if a test is strategic. Because you might be testing strategy – than a flat to win is good. If you are running a revenue test, than only win is what we go on. Because there is a cost to switching to a new design.
  • 1-2mSmall changes matter. Spot the difference, it’s worth 2million dollars.
  • Got our ass kicked on this – lost potentially 3m Browse account modal- existing features don’t always work
  • It’s important to test both iterative changes and big swings in some cases, or you may only make small changes.
  • Everthing
  • There’s an “ideas” list sitting on on our internal wiki with 200+ test ideas waiting to go.
  • Define Highest paid person opinion
  • Don’t’ overbuild or conduct full redesigns when all you need are optimizations. It’s too much work to take a swing like that.
  • Here’s a slide from Joshua Porter – really bright guy Why do I drive this point home – because I’ve seen testing cultures where the teams literally can’t launch a new feature. They are so entrenched in the mindset of optimization that a new product is out of the question.
  • Seriously Advanced A/B Testing by Wyatt Jenkins

    1. 1. GROWTH Is simply a series of experiments Wyatt Jenkins – VP Product @wyatt_earp_ @shutterstock
    2. 2. Lean Testing How Shutterstock Tests Everything and Stays Nimble
    3. 3. Wyatt Jenkins VP, Product Development Shutterstock Computer science, Interaction Design, Product Strategy M.I.T. & University of Colorado Partner / Board Member of Beatport @wyatt_earp_
    4. 4. Data is a competitive Advantage
    5. 5. Data is your product The more Data you have, the better your product can be. The more usage you have, the faster you can achieve confidence on tests
    6. 6. Data is a competitive advantage 3 Downloads / second 1 TB per day
    7. 7. Tips & Types
    8. 8. A/B Testing
    9. 9. Smoke Tests
    10. 10. Multi-Variate
    11. 11. Brand -vs- Optimization
    12. 12. Brand -vs- Optimization
    13. 13. Teams • • • • • • Keep „em small – 3 ppl. for testing No hand-offs Let go of “Strategy / Process” Metrics driven – choose one Data highly accessible Decision making de-centralized
    14. 14. Qualitative –vs- Quantitative Idea Generation Validation Qualitative Quantitative
    15. 15. Testing Lessons
    16. 16. Results – Run significance on win / flat / lose Cell Keystone Redesign Content Types: Visual Content Types: Text-Only Browse % XX% XX% XX% Flat 0.02% 48.63% 96.10% Win 0.01% 38.88% 2.02% Lose 99.98% 12.49% 1.87%
    17. 17. Volume of Tests • Testing win $ amount unreliable metric • 100s to 1000s in future • 30% win rate
    18. 18. Testing Lessons • Leave design principles at the door • Test small changes • Don‟t forget performance • Hard to test strategy • Re-Test ideas.
    19. 19. Metrics Choose one, view many
    20. 20. Tools
    21. 21. Devent
    22. 22. Lil‟ Brother
    23. 23. Absinthe
    24. 24. DataViz
    25. 25. Test Examples
    26. 26. Pricing Headlines
    27. 27. Browse Modal
    28. 28. Iteration & Big Swing
    29. 29. What else do we test? • • • • • Linguistics (Language detection) Relevance / Popular Localized experiences SEM landing pages Contributor sign up conversion
    30. 30. Search Grid
    31. 31. Search Grid 175px Thumbnails
    32. 32. Search Grid 300px Thumbnails
    33. 33. “Experimentation” Culture
    34. 34. Positive: Enabled People
    35. 35. Positive: Hippo Killers
    36. 36. Related Searches Positive: More Ideas See Daylight
    37. 37. Positive: Hackathons
    38. 38. Positive: Learn What to Design
    39. 39. Challenges: Can‟t Optimize Big Swings * Josh Porter:
    40. 40. Testing = Growth & Data = Competitive Advantage Wyatt Jenkins @wyatt_earp_ @shutterstock
    41. 41. Thank You Wyatt Jenkins @wyatt_earp_ @shutterstock