Sept 16-17<br />Vancouver, CANADA<br />Fast & Furious – 10 Principles for Rapid Testing<br />Presenting Lessons In Testing...
Intuit Small Business – Web Business<br />How much: On order of ~$100 M annual revenue transacted online<br />What: Deskto...
QuickBooks.com – Various Timepoints<br />3<br />Jared Waxman<br />
I’ve tested…<br />I’ve run 1 or MORE tests against <br />each of these hypotheses:<br />How much does a Free product canni...
…and tested…<br />Do videos sell better than static images?<br />Do expanded versus collapsible product specs perform bett...
…and tested…<br />Where is the optimal place in the purchase funnel for customer to configure their product?<br />Where is...
…and tested…<br />Where is the right step in the purchase funnel to cross-sell other offerings?<br />How many cross-sell o...
…and tested…<br />Are bullet points or prose styles more effective on landing pages?<br />Are PPC ad group specific links ...
…and tested.<br />Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/p...
What has been the benefit of testing?<br />10<br />Half our growthin the online business comes from testing the pure site ...
Today’s Focus: Organization Design<br />11<br />These gears must all work together. <br /> Each deserves separate treatmen...
Fast & Furious<br />My Testing Team throughput in just 12 business days:<br /><ul><li>5 tests (3 of which shown on next pa...
11 Factors Tested
26 Recipes In All</li></ul>1 Big Winner: 20% increase on the success formula<br />Annual impact ~ $10M<br />Plus…<br />1 L...
Then Narrow <br />(8 Recipes)<br />3/4/09<br />Control<br />Product A: Index 1.60<br />Formula: Index 1.20 WE/WD<br />3<br...
Top 10 “Go Fast” Testing Principles<br />10. Testing “Strike Force” with no outside dependencies<br />9. Exactly one “Appr...
Process & Traps<br />15<br />Traps:<br /><ul><li>Perfection</li></ul>Traps:<br /><ul><li>Debate</li></ul>Traps:<br /><ul><...
Re-tests, small tests</li></ul>Traps:<br /><ul><li>The “Evil And”
Management not aligned – competing goals</li></ul>Jared Waxman<br />
In the org, you’ll encounter…<br />Digging Deeper into Organization Design<br />Jared Waxman<br />
HiPOs<br />17<br />Jared Waxman<br />
HiPOs: Tips<br />Get Management to understand why testing success probability means it’s in their interest to have a healt...
Targets<br />19<br />Jared Waxman<br />
Targets: Tips<br />Set by Management, not testing team.<br />Testing Team Lead responsible to push back if needs more cons...
Design by Committee<br />Jared Waxman<br />
Design By Committee: Tips<br />Hire a Test Lead whose business instinct and creative eye you know and trust.  <br />Keep h...
Sacred Cows<br />Jared Waxman<br />
Sacred Cows: Tips<br />24<br />Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over ...
Upcoming SlideShare
Loading in …5
×

Rapid A/B Testing 10 Principles (Jared Waxman)

3,842 views
3,683 views

Published on

Smart web and ecommerce companies companies are turning to Website Optimization (A/B and MVT Testing) as the next big area for growth. When done right, optimizing your website can be the most highly leveraged function within web marketing. Most companies fail at the first step - organization design - and never get their A/B testing program off the ground. Jared Waxman discusses how Intuit changed up the organization process to achieve one-day A/B testing sprints, and saw multi-million dollar payback.

Published in: Business, Technology
2 Comments
13 Likes
Statistics
Notes
No Downloads
Views
Total views
3,842
On SlideShare
0
From Embeds
0
Number of Embeds
158
Actions
Shares
0
Downloads
95
Comments
2
Likes
13
Embeds 0
No embeds

No notes for slide
  • Big picture: A platypus. Widely ridiculed as being a sorry excuse for an animal, Tail of beaver, beak of a duck. Perhaps the result of design by committee. G-d wanted to cut out early for the weekend, and assigned his helpers to figure it out. No one was the clear leader, and this is what they came up with. Anybody want their site looking like this?Why it’s important:*The more opinions into the test concepts, the more conservative or closer to status quo it will be. Then you’ll get less of a signal, harder to read the test. More likely to waste your effort.*Also, the design of experiment is more likely to be compromised with a lot of opinions going in, and therefore what learning you get tends to be less clear, true, or extensible to future decisions
  • Big Picture: At a large organization, testing and analytics folks often feel like this little scared lion cub about to be run over by a stampede. Once Management realizes how valuable testing & analytics really is, everyone comes at you with their project. Needs it now. And doesn’t care much that it doesn’t fit with the priorities you’ve been given, or is beyond the bandwidth of your resources or traffic.Why It’s Important?*You don’t want the squeakiest wheel to get the grease. You want to move the most important metrics in your business.*You want to attract and retain talent, and don’t want them overloaded, underappreciated, and unwilling to stay*You want to have a plan for executing through peaks
  • Rapid A/B Testing 10 Principles (Jared Waxman)

    1. 1. Sept 16-17<br />Vancouver, CANADA<br />Fast & Furious – 10 Principles for Rapid Testing<br />Presenting Lessons In Testing From<br />Makers of Quicken, QuickTax and Quickbooks<br />Jared Waxman<br />jared@benchmark-analytics.com<br />
    2. 2. Intuit Small Business – Web Business<br />How much: On order of ~$100 M annual revenue transacted online<br />What: Desktop software, Services, Software-as-a-Service (subscription model)<br />Where: 45 Websites, 10 Shopping Carts<br />Traffic: QuickBooks.com ~ 1 M monthly unique cookies<br />2<br />Jared Waxman<br />
    3. 3. QuickBooks.com – Various Timepoints<br />3<br />Jared Waxman<br />
    4. 4. I’ve tested…<br />I’ve run 1 or MORE tests against <br />each of these hypotheses:<br />How much does a Free product cannibalize Paid products?<br />What’s the optimal way to position a Free product in the lineup to maximize all units while minimizing paid cannibalization?<br />Does one Free offering cannibalize less than another Free offering?<br />What is the most effective “voice” used in describing the product?<br />What category of images sells the product the best?<br />4<br />Jared Waxman<br />
    5. 5. …and tested…<br />Do videos sell better than static images?<br />Do expanded versus collapsible product specs perform better?<br />Does having a special price for upgraders yield higher overall sales than a flat pricing model? Does it improve customer sentiment?<br />Does exposing retail options significantly shrink direct online sales?<br />Which discount % level of promotional price discounts yields highest revenue?<br />Does adding cart CTA on home and line-up pages increase conversion or change pathing?<br />What is the optimal color, size, placement, and copy on CTAs?<br />5<br />Jared Waxman<br />
    6. 6. …and tested…<br />Where is the optimal place in the purchase funnel for customer to configure their product?<br />Where is the optimal place in the purchase funnel to ask for credit card info?<br />How many pages in the checkout flow is optimal?<br />What home page weighting of offerings moves us to the desired platform mix while continuing to grow revenue?<br />How effective are certain product chooser tools, demo, or guides in moving conversion rates?<br />How effective is a certain cart abandonment tactic in moving conversion rates?<br />6<br />Jared Waxman<br />
    7. 7. …and tested…<br />Where is the right step in the purchase funnel to cross-sell other offerings?<br />How many cross-sell offerings are optimal for a given funnel step?<br />Should cross-sells be passive or intrusive?<br />How effective and efficient is proactive chat in boosting over conversion?<br />How many CTAs and/or featured offerings are optimal on the home page?<br />How many offerings is optimal within the line-up?<br />7<br />Jared Waxman<br />
    8. 8. …and tested…<br />Are bullet points or prose styles more effective on landing pages?<br />Are PPC ad group specific links on landing pages more effective than generic links?<br />What’s the likelihood of successful for a new product offering if added to the online store?<br />How would a New User vs Existing User focused site perform against overall online revenue?<br />Can lift conversion rates by guessing at New vs Existing and then sending them down different paths?<br />How effective would a simple User Contribution System be at engaging and converting prospects?<br />8<br />Jared Waxman<br />
    9. 9. …and tested.<br />Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/product offerings reduce the net traffic that gets sent?<br />Do menu titles perform better if they are job-oriented or product-oriented?<br />Do promos perform better as %s or as $s off?<br />Do promos perform better as %/$ off vs Free Shipping?<br />Does a certain messaging approach consistently outperform another regardless of specifics?<br />9<br />Jared Waxman<br />
    10. 10. What has been the benefit of testing?<br />10<br />Half our growthin the online business comes from testing the pure site experience. The other half from acquisition programs.<br />Testing is highly leveraged growth – a handful of employees. Produces 3-digit ROI.<br />Acquisition programs requires serious cash and overall produces 2-digit ROI.<br />Jared Waxman<br />
    11. 11. Today’s Focus: Organization Design<br />11<br />These gears must all work together. <br /> Each deserves separate treatment.<br />Focus of this presentation<br />Jared Waxman<br />
    12. 12. Fast & Furious<br />My Testing Team throughput in just 12 business days:<br /><ul><li>5 tests (3 of which shown on next page) including 1 microsite
    13. 13. 11 Factors Tested
    14. 14. 26 Recipes In All</li></ul>1 Big Winner: 20% increase on the success formula<br />Annual impact ~ $10M<br />Plus…<br />1 Loser (but killed quickly)<br />Some breakevens<br />Jared Waxman<br />
    15. 15. Then Narrow <br />(8 Recipes)<br />3/4/09<br />Control<br />Product A: Index 1.60<br />Formula: Index 1.20 WE/WD<br />3<br />Narrow Again <br />(4 Recipes)<br />4/9/09<br />Go Broad <br />(8 recipes)<br />2/20/09<br />1<br />2<br />4<br />Product A: Index 2.00<br />And Broad, Again!<br />3/9/09<br />Formula: Index 1.29 Weekend, .91 Weekday<br />Rapid Testing Team<br />Formula: Index 0.50<br />Jared Waxman<br />
    16. 16. Top 10 “Go Fast” Testing Principles<br />10. Testing “Strike Force” with no outside dependencies<br />9. Exactly one “Approver”<br />8. One straight-forward formula for success<br />7. Inform at go-live, then again with results<br />6. Fully-baked ideas as the starter, then testing team tightens it. After phase 1, iterate with true MVT and refine as appropriate.<br />5. Test ideas based in theory and customer data. Then go test, not talk.<br />4. Manage to deadlinewith quality. Reduce scope as needed!<br />3. Take in ideas from anywhere and everywhere. Don’t take in orders.<br />2. Open, flexible test calendar basing next tests on data from existing ones. No pre-slotted commitments<br />and finally…<br />If you follow the prior 9 principles you’ll be able to deliver…<br />1. One-Day Test Sprints! From schedule-it to go-live in 8 hrs.<br />Jared Waxman<br />14<br />
    17. 17. Process & Traps<br />15<br />Traps:<br /><ul><li>Perfection</li></ul>Traps:<br /><ul><li>Debate</li></ul>Traps:<br /><ul><li>Ownership issues
    18. 18. Re-tests, small tests</li></ul>Traps:<br /><ul><li>The “Evil And”
    19. 19. Management not aligned – competing goals</li></ul>Jared Waxman<br />
    20. 20. In the org, you’ll encounter…<br />Digging Deeper into Organization Design<br />Jared Waxman<br />
    21. 21. HiPOs<br />17<br />Jared Waxman<br />
    22. 22. HiPOs: Tips<br />Get Management to understand why testing success probability means it’s in their interest to have a healthy distance from individual test designs. It’s in your interest too.<br />Get Management engaged at the right level – setting hierarchy of goals and resourcing for highly leveraged testing.<br />18<br />Jared Waxman<br />
    23. 23. Targets<br />19<br />Jared Waxman<br />
    24. 24. Targets: Tips<br />Set by Management, not testing team.<br />Testing Team Lead responsible to push back if needs more constraints.<br />Put in place a hierarchy of success metrics.<br />Should be consistent across seasons, managers, businesses.<br />Revenue per visitor (RPV) typically at top of hierarchy.<br />Can assign each success metric it’s own team.<br />20<br />Jared Waxman<br />
    25. 25. Design by Committee<br />Jared Waxman<br />
    26. 26. Design By Committee: Tips<br />Hire a Test Lead whose business instinct and creative eye you know and trust. <br />Keep him/her in the kitchen on business strategy. <br />Then let him/her be the one and only approver on what goes live.<br />Ideas get thrown “over the wall” to the testing team.<br />Testing Teams set up as neutral, with no built-in biases towards specific businesses, ideas, or managers.<br />Idea submitter and/or related business unit have no official role during the execution phase.<br />22<br />Jared Waxman<br />
    27. 27. Sacred Cows<br />Jared Waxman<br />
    28. 28. Sacred Cows: Tips<br />24<br />Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over the wall”. Assign the Test Lead the job of selecting and executing.<br />Separate Segment Marketers from Testing.<br />Get a clear list of constraints, a consistent success metric, and then challenge everything.<br />Outside viewpoint. Much easier to get people think outside the box when they know some crazy-sounded idea worked somewhere else.<br />Test big. And if you can’t, test small. Remember you can also pull if it’s losing money.<br />Influence through your framing. Testing can be framed not as a challenge, but a way to quantify a “known” lever so it can be expanded or further optimized.<br />Jared Waxman<br />
    29. 29. Stampedes<br />Jared Waxman<br />
    30. 30. Stampedes: Tips<br />Cordon off your Testing Teams to focus your highly leveragable resources<br />Set clear, long-term goals for the Teams.<br />Empower the Team Leads to decide priorities, timing, workload.<br />Leverage outside agencies to rapidly expand capabilities and expertise as needed.<br />26<br />Jared Waxman<br />
    31. 31. Fast & Furious – 10 Principles for Rapid Testing<br />Sept 16-17<br />Vancouver, CANADA<br />Presenting Lessons In Testing From<br />Jared Waxman<br />jared@benchmark-analytics.com<br />

    ×