Sept 16-17Vancouver, CANADAFast & Furious – 10 Principles for Rapid TestingPresenting Lessons In Testing FromMakers of Quicken, QuickTax and QuickbooksJared Waxmanjared@benchmark-analytics.com

Intuit Small Business – Web BusinessHow much: On order of ~$100 M annual revenue transacted onlineWhat: Desktop software, Services, Software-as-a-Service (subscription model)Where: 45 Websites, 10 Shopping CartsTraffic: QuickBooks.com ~ 1 M monthly unique cookies2Jared Waxman

QuickBooks.com – Various Timepoints3Jared Waxman

I’ve tested…I’ve run 1 or MORE tests against each of these hypotheses:How much does a Free product cannibalize Paid products?What’s the optimal way to position a Free product in the lineup to maximize all units while minimizing paid cannibalization?Does one Free offering cannibalize less than another Free offering?What is the most effective “voice” used in describing the product?What category of images sells the product the best?4Jared Waxman

…and tested…Do videos sell better than static images?Do expanded versus collapsible product specs perform better?Does having a special price for upgraders yield higher overall sales than a flat pricing model? Does it improve customer sentiment?Does exposing retail options significantly shrink direct online sales?Which discount % level of promotional price discounts yields highest revenue?Does adding cart CTA on home and line-up pages increase conversion or change pathing?What is the optimal color, size, placement, and copy on CTAs?5Jared Waxman

…and tested…Where is the optimal place in the purchase funnel for customer to configure their product?Where is the optimal place in the purchase funnel to ask for credit card info?How many pages in the checkout flow is optimal?What home page weighting of offerings moves us to the desired platform mix while continuing to grow revenue?How effective are certain product chooser tools, demo, or guides in moving conversion rates?How effective is a certain cart abandonment tactic in moving conversion rates?6Jared Waxman

…and tested…Where is the right step in the purchase funnel to cross-sell other offerings?How many cross-sell offerings are optimal for a given funnel step?Should cross-sells be passive or intrusive?How effective and efficient is proactive chat in boosting over conversion?How many CTAs and/or featured offerings are optimal on the home page?How many offerings is optimal within the line-up?7Jared Waxman

…and tested…Are bullet points or prose styles more effective on landing pages?Are PPC ad group specific links on landing pages more effective than generic links?What’s the likelihood of successful for a new product offering if added to the online store?How would a New User vs Existing User focused site perform against overall online revenue?Can lift conversion rates by guessing at New vs Existing and then sending them down different paths?How effective would a simple User Contribution System be at engaging and converting prospects?8Jared Waxman

…and tested.Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/product offerings reduce the net traffic that gets sent?Do menu titles perform better if they are job-oriented or product-oriented?Do promos perform better as %s or as $s off?Do promos perform better as %/$ off vs Free Shipping?Does a certain messaging approach consistently outperform another regardless of specifics?9Jared Waxman

What has been the benefit of testing?10Half our growthin the online business comes from testing the pure site experience. The other half from acquisition programs.Testing is highly leveraged growth – a handful of employees. Produces 3-digit ROI.Acquisition programs requires serious cash and overall produces 2-digit ROI.Jared Waxman

Today’s Focus: Organization Design11These gears must all work together. Each deserves separate treatment.Focus of this presentationJared Waxman

Fast & FuriousMy Testing Team throughput in just 12 business days:5 tests (3 of which shown on next page) including 1 microsite11 Factors Tested26 Recipes In All1 Big Winner: 20% increase on the success formulaAnnual impact ~ $10MPlus…1 Loser (but killed quickly)Some breakevensJared Waxman

5 tests (3 of which shown on next page) including 1 microsite

11 Factors Tested

26 Recipes In All

Then Narrow (8 Recipes)3/4/09ControlProduct A: Index 1.60Formula: Index 1.20 WE/WD3Narrow Again (4 Recipes)4/9/09Go Broad (8 recipes)2/20/09124Product A: Index 2.00And Broad, Again!3/9/09Formula: Index 1.29 Weekend, .91 WeekdayRapid Testing TeamFormula: Index 0.50Jared Waxman

Top 10 “Go Fast” Testing Principles10. Testing “Strike Force” with no outside dependencies9. Exactly one “Approver”8. One straight-forward formula for success7. Inform at go-live, then again with results6. Fully-baked ideas as the starter, then testing team tightens it. After phase 1, iterate with true MVT and refine as appropriate.5. Test ideas based in theory and customer data. Then go test, not talk.4. Manage to deadlinewith quality. Reduce scope as needed!3. Take in ideas from anywhere and everywhere. Don’t take in orders.2. Open, flexible test calendar basing next tests on data from existing ones. No pre-slotted commitmentsand finally…If you follow the prior 9 principles you’ll be able to deliver…1. One-Day Test Sprints! From schedule-it to go-live in 8 hrs.Jared Waxman14

Process & Traps15Traps:PerfectionTraps:DebateTraps:Ownership issuesRe-tests, small testsTraps:The “Evil And”Management not aligned – competing goalsJared Waxman

Perfection

Debate

Ownership issues

Re-tests, small tests

The “Evil And”

Management not aligned – competing goals

In the org, you’ll encounter…Digging Deeper into Organization DesignJared Waxman

HiPOs17Jared Waxman

HiPOs: TipsGet Management to understand why testing success probability means it’s in their interest to have a healthy distance from individual test designs. It’s in your interest too.Get Management engaged at the right level – setting hierarchy of goals and resourcing for highly leveraged testing.18Jared Waxman

Targets19Jared Waxman

Targets: TipsSet by Management, not testing team.Testing Team Lead responsible to push back if needs more constraints.Put in place a hierarchy of success metrics.Should be consistent across seasons, managers, businesses.Revenue per visitor (RPV) typically at top of hierarchy.Can assign each success metric it’s own team.20Jared Waxman

Design by CommitteeJared Waxman

Design By Committee: TipsHire a Test Lead whose business instinct and creative eye you know and trust. Keep him/her in the kitchen on business strategy. Then let him/her be the one and only approver on what goes live.Ideas get thrown “over the wall” to the testing team.Testing Teams set up as neutral, with no built-in biases towards specific businesses, ideas, or managers.Idea submitter and/or related business unit have no official role during the execution phase.22Jared Waxman

Sacred CowsJared Waxman

Sacred Cows: Tips24Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over the wall”. Assign the Test Lead the job of selecting and executing.Separate Segment Marketers from Testing.Get a clear list of constraints, a consistent success metric, and then challenge everything.Outside viewpoint. Much easier to get people think outside the box when they know some crazy-sounded idea worked somewhere else.Test big. And if you can’t, test small. Remember you can also pull if it’s losing money.Influence through your framing. Testing can be framed not as a challenge, but a way to quantify a “known” lever so it can be expanded or further optimized.Jared Waxman

StampedesJared Waxman

Stampedes: TipsCordon off your Testing Teams to focus your highly leveragable resourcesSet clear, long-term goals for the Teams.Empower the Team Leads to decide priorities, timing, workload.Leverage outside agencies to rapidly expand capabilities and expertise as needed.26Jared Waxman

Fast & Furious – 10 Principles for Rapid TestingSept 16-17Vancouver, CANADAPresenting Lessons In Testing FromJared Waxmanjared@benchmark-analytics.com