Downloaded 33 times
![Thank You!
Email me sullivac@gmail.com
Slides [slideshare]
Linkedin linkd.in/pvrg14](https://image.slidesharecdn.com/survivingthehypecycle-shortcutstosplittestingsuccess-16sep2015-v3-150916093103-lva1-app6892/85/Surviving-the-hype-cycle-Shortcuts-to-split-testing-success-64-320.jpg)
Uploaded byCraig Sullivan
Surviving the hype cycle Shortcuts to split testing success
The document discusses best practices for split testing success, emphasizing the importance of analytics health checks, proper test placement, and thorough research. It identifies common mistakes that can lead to poor results, such as testing without a solid hypothesis or prioritizing incorrectly. The author also encourages collaboration across teams to eliminate silos and improve testing outcomes.
More Related Content
Similar to Surviving the hype cycle Shortcuts to split testing success
![[Srijan Wednesday Webinars] Why Adopt Analytics Driven Testing](https://cdn.slidesharecdn.com/ss_thumbnails/srijanwednesdaywebinarsanalyticsdriventesting2-170615131224-thumbnail.jpg?width=640&height=640&fit=bounds)
Surviving the hype cycle Shortcuts to split testing success
- 1. Surviving the HypeCycle Shortcuts to Split Testing Success @OptimiseOrDie
- 2. @OptimiseOrDi e • Split Testing,Analytics, UX • Lean, Agile, Growth Optimisation • 100M+ visitors tested • 200+ sites, 26 languages • Starting Testing? Bad ROI? • Not learning enough? • My job – to transfer skills! @OptimiseOrDiecolinski.tumblr.com
- 4. 1989-98 : PRICEWATERHOUSE @OptimiseOrDie 30 Years Later?….
- 5. @OptimiseOrDie #badfinance RBS EmergencyCash Source : rbs.co.uk To confirm your withdrawal: Please enter the amount in Euros you requested:
- 6.
- 7. Surviving the HypeCycle Shortcuts to Split Testing Success @OptimiseOrDie
- 8.
- 9. 1 Tool Installed 2Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
- 10.
- 11. 1. Get AnalyticsHealth Checked 2. Test in the right place 3. Understand Cross Device 4. Do your Research 5. Prioritise your testing @OptimiseOrDie 6. Perform Pre Flight Checks 7. Know how long to test 8. Have a good reason to test 9. Learn from your tests 10. Burn down the silos 10 Shortcuts to Testing Success
- 12. @OptimiseOrDie 1. Your AnalyticsSetup is Broken
- 13. • Nearly 100Sites in 3 years • 95% were broken, often badly • Trust in data was missing • Management made bad calls • Nobody checked the tills • Calibrate from the basics up! @OptimiseOrDie • What sales do we capture? • What categories? • What about refunds, lunch money, gift certificates? • How do we monitor fraud? • Do we check it adds up? • Where does this data go? 1. What about my clients?
- 14. @OptimiseOrDie 1. Bulls*** flowsupwards! Cool BS Dashboard BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS reports BS Reports TILLS DEPT STORE DIVISIO N
- 15. @OptimiseOrDie • Review takes1-3 days • Prioritise the issues • Fix directly with developers • Integrate with the Testing Tool1 Get an Analytics Health Check
- 16. @OptimiseOrDie 2. You Testin the Wrong Place
- 17. 2. Let’s DoRandom Testing Let’s try the homepage I’ve got targets to hit! I hate this job Let’s test button colours! Has lots of opinions but no data Spends too much time on Twitter Driven by Ego and Competitors Wishes he cared about testing
- 18. @OptimiseOrDie “STOP copying yourcompetitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL 2. Let’s Do Random Testing
- 19. @OptimiseOrDie 2. Modelling -Intent All traffic HearingSight StoreOther Step 1 Step 2 Step 3 Goal Page PageHearing
- 20. @OptimiseOrDie 2. Modelling –Multiple Funnel Endings All traffic InfluenceIntent Influence Step 1 Step 2 Step 3 Goal Page Page Entry Page 1 2 3 4 1 2 3 4 1 2 3 4
- 21. Bounce Search or Category Product Page Add to basket View basket Checkout @OptimiseOrDie 2.Modelling – Ring Models
- 22. 2. Modelling –Horizontal Funnels
- 23. @OptimiseOrDie • Do someAnalytics modelling • Understand the shedding of layers • Narrow your focus and scope • Bank better gains earlier in time 2 Test in the Right Places
- 24. @OptimiseOrDie • Do youreally know your mix? • What iPhone models visit? • How big is tablet traffic? • What screen sizes do they have? • Find out BEFORE you design tests • Check BEFORE you launch tests • Use Google Analytics to find out • 3 reports to rule them all https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA 3. Our customers use iPhones, right?
- 25. @OptimiseOrDie 3. iPhone Modelsin Google Analytics Screen Resolution 320 x 480 = iPhone 4/4S 320 x 568 = iPhone 5/5S 375 x 667 = iPhone 6 414 x 736 = iPhone 6+ https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA
- 26. @OptimiseOrDie • Desktop Browsers& versions • Tablet Models • Mobile Device Models • Screen Resolutions3 Figure Out the Device Mix for Testing
- 27. @OptimiseOrDie 4. You Don’tDo Your Research Time Budget Buyin
- 28. @OptimiseOrDie 4. 1 HourPage Analytics Influence Pages Entry Points Landing Pages Device Mix Customer Mix Traffic Mix Flow Intent Marketing -> Site flow Page or Process Next Steps Abandonment Exits Mix of abandonment Flow
- 29. @OptimiseOrDie 4. 1 HourLanding Page Analytics • How old are the visitors? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What are the key metrics like (e.g. bounce rate, conversion)? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What is the goal or ecommerce conversion through this page? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What channel traffic comes to the page? https://www.google.com/analytics/web/template?uid=Kjb9q8M4QN-fsPe8dOGaig • What is the mix of tablet / mobile / desktop to the page? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • What are the resolutions of devices? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • How slow are the landing pages? https://www.google.com/analytics/web/template?uid=AavFsgMoRkucYYKnxlB76Q • What are the pages right after the landing page? (Use a landing page report and choose the ‘Entrance Paths’ to show next pages.) • What is the flow like from this page? (Use the Behaviour Flow Report) • What does it look like on the top devices? (Use real devices + Appthwack.com, Crossbrowsertesting.com or Deviceanywhere.com)
- 30. @OptimiseOrDie 4. If youhave 2 hours • Check the: – Marketing Creatives – Search Engine Results Page – Ad Copy, Display Advert or Email – Look at Landing page ZERO! • Make a horizontal funnel from the landing page • Friends, Family, New Employee testing • 5 Second Test
- 31. @OptimiseOrDie 4. If youhave 4 hours PLUS • Snap interviews (Sales, Customer Services, Tech Support) • Run a quick poll or survey (See my tools slides)
- 32. @OptimiseOrDie • Lean Analytics •UX Research • Interviewing • Surveys and Polls4 No Excuses – Do Your Research
- 33. @OptimiseOrDie 5. You don’tPrioritise your Tests Scoring can be cost, time to market, resource, risk, political complexity • Cost 1-10 Higher is cheaper • Time 1-10 Higher is shorter • Opportunity 1-10 Higher is greater Score = Cost * Time * Opportunity • For financial institutions, risk should be a factor • Want to build your own? – ask me!
- 34. @OptimiseOrDie 5. Opportunity vs.Cost 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Cost (high is better) Opportunity MONEY!
- 35. @OptimiseOrDie 5. Make aMoney Model Test Description Metric 2% lift 5% lift 10% lift Estimate Product page Simplification Basket adds 200,000 500,000 1,000,000 500,000 Register new Improve onboarding New register funnel ratio 25,000 62,500 125,000 250,000 IE8 bugs in cart Fix key broken stuff IE8 Conversion 80,000 200,000 400,000 200,000 Category list page Get product higher User Category -> Product 500,000 1,250,000 2,500,000 1,250,000 Payment Page New card handling User Payment -> Thank you 60,000 150,000 300,000 300,000
- 36. @OptimiseOrDie • Score allTest Targets • Use Cost vs. Opportunity minimum • Check it works! • Make a Money Model 5 Prioritise your Testing Targets
- 37. @OptimiseOrDie 6. You Don’tTest Before Launch • Dirty secret of AB testing? • People break their tests all the time! • Most people don’t notice Why? • Because developers can break them very easily • What if your AB test was broken on iPhones? • If you didn’t know, would your results be valid? • About 40% of my tests fail basic QA
- 38. Browser Checks www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com www.multibrowserviewer.com Mobile& Tablet www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article & Info bit.ly/devicetesting 6. Here is my £80M testing rig
- 39. @OptimiseOrDie 6. Always UseReal Devices
- 40. @OptimiseOrDie • Check EveryTest Works • Browser and Devices • Check Analytics records correctly 6 Perform Pre Flight Checks
- 41. • TWO BUSINESSCYCLES minimum (week/mo) • 1 PURCHASE CYCLE minimum • 250 CONVERSIONS minimum per creative (e.g. checkouts) • 350, 500, more if response is very similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie 6. Know How Long to Test for…
- 42. @OptimiseOrDie • Most criticalmistake • Use a test calculator • Full business cycles, 2 minimum • Don’t waste time hoping7 Know How Long to Test for
- 43. Insight - Inputs #FAIL Competitor copying Guessing Dicerolling An article the CEO read Competitor change Panic Ego Opinion Cherished notions Marketing whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature blindness Knee jerk reactons @OptimiseOrDie 8. So you think you have a Hypothesis?
- 44. @OptimiseOrDie 1. Because wesaw (data/feedback) 2. We expect that (change) will cause (impact) 3. We’ll measure this using (data metric) bit.ly/hyp_kit 8. Use this to deflect stupid testing!
- 45. @OptimiseOrDie • Don’t doEgo driven testing • Use the Hypothesis Kit! 8 Get a Proper Hypothesis Going
- 46. @OptimiseOrDie 9. Our Testingteaches us Nothing! • Either your research or hypothesis is weak • Work back from the outcome! What if A won – what would that tell us? What if A failed – what would that tell us? • What is the value to the business in finding out the answer? • Is the finding actionable widely and deeply? • Testing isn’t about lifts – it’s about learning
- 47. @OptimiseOrDie • Do yourresearch • Form a solid hypothesis • Work back from the outcomes • Learning useful stuff = huge lifts9 Design Tests for Maximum Learning
- 48. @OptimiseOrDie 10. Burn Downthe Silos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
- 49. @OptimiseOrDie 10. FT Example •Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
- 50. @OptimiseOrDie 10. FT Example •100s of releases a day! • MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
- 51. @OptimiseOrDie 10. Positive Attributes •Iterative & Agile Design • Small agile teams • Rapid and iterative UX testing • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing Tools
- 52. @OptimiseOrDie • Agile, Lean,Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features 10 Burn Down the Silos!
- 53. @OptimiseOrDie #goodfinance Simple &Moven Banking Apps • Solving real user problems • Budgeting (safe to spend, daily limit) • Transfers (free) • Cheque deposits (camera) • Analyses patterns, oddities • Prevents bad stuff happening • Helps you save money • Multiple account handling www.simple.com/banking www.moven.com www.tink.se
- 54. @OptimiseOrDie #goodfinance iZettle &Square • Wireless card payments • Offline mode • Next day payment • Truly mobile POS • Real problems, new solutions!
- 55. @OptimiseOrDie #goodfinance Acorns • Investyour spare change • Rounds up spend • Invests the change • Tweak the portfolio • Student or < 24 years old? • Free of fees!
- 56. 56 “If you thinkof technology as something that’s spreading like a sort of fractal stain, almost every point on the edge represents an interesting problem.” Paul Graham
- 57. @OptimiseOrDie Threats and Opportunities- Crypto TCP/IP is to… as Crypto is to… Network Transport Currency Transport Web Browsers Wrappers & Primitives Websites New financial services Services & Experience Transformational products
- 58.
- 59.
- 60. Rumsfeldian Space• What ifwe changed our prices? • What if we gave away less for free? • What if we took this away? • What about 3 packages, not 5? • What are these potential futures I can take? • How can I know before I spend money? • UPS left hand turning -10 Million Gallons saved in fuel: http://compass.ups.com/UPS-driver-avoid-left-turns/ • McDonalds Hipster Test Store bit.ly/1TiURi7
- 61. Congratulations! Today you’re thelucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy! Innovation Testing
- 62.
- 63. 1 Tool Installed 2Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
- 64. Thank You! Email mesullivac@gmail.com Slides [slideshare] Linkedin linkd.in/pvrg14
Editor's Notes
- #3 So – a boring intro bit about me – don’t worry – it’ll be over quickly. You can have a look at your phones for a couple of minutes or just watch David Lynch’s finger instead.
- #4 And here are some of the clients I’ve been working for recently. Dull bit is now officially over. And I’ll now talk about the big problem with split testing – it’s just not working for a lot of companies.
- #6 Examples of bad finance Céad míle mallacht ort!
- #7 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #8 So what is the subject of my talk today? It’s about the glue between the worlds of measurement and the tools, culture and methodologies – the scientific approach to testing things – that drives measurable innovation within companies. It’s about taking things you observe or discover and turning them into ‘what if?’ scenarios. And all the tech you’re using today came from this scientific approach – and many millions of hours of testing things and failing (Pilkington) Bet let’s go back in time a bit – Back to the Eighties!
- #9 I’ve been working on this presentation and thinking about it for over two months now. And this is one of the first graphs I wanted to include, because it represents the hype cycle in AB testing. This is what a lot of companies go through with new technology adoption, so I wanted to show you what the AB testing hype cycle would look like [CLICK]
- #11 So if this is what ramping up your AB testing feels like at your company, welcome to the club! You are not alone! I know what’s going wrong because I’ve made all these mistakes myself. Today is about giving you shortcuts away from stupidity and getting you from burning cash to productive testing. My pain is your gain!
- #12 And here are the top 10 reasons – there are about 70 odd ways people manage to break their AB testing but these are the most common mistakes, particularly for companies just starting or scaling up. Let’s run through each one – I’ve included a summary slide of each one, so you’ve got a nice handy checklist to take back to the office.
- #17 So where do you start testing? Where do you focus your efforts? Over here? This bit over here?
- #54 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #55 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #56 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #58 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #59 And here is the NatWest Complaints form – the pinnacle of user experience in financial technology? So I can only make a complaint if I actually remove all the punctuation from what I write! And the weird thing was, once you got an error, it changed the rules on you. See this ‘etc.’ bit on the right – they should put that in bold, because I had no idea how to work out their rules. It took me 15 minutes to make the complaint but it got funnier. “Please let us know what you would like us to do to put things right” – Oh, let me think – how about you take the person who made this and publicly torture them in front of a cheering crowd – that might help. And “We will be in touch after 5 working days” – is that because it will take me that long to get past the form or because you really care?
- #61 And now a bit about something I call Rumsfeldian Space – exploring the unknowns. This is vital if you want to make your testing bold enough to get great results.
- #63 Hopefully now you’ve joined Guessaholics Anonymous from today, you can move on to a data driven and more productive future. Enjoy less wastage in your products, greater efficiency in marketing spend and grow faster than you imagined. All it takes is a bit of love of your customers, your business and some solid ground rules.
- #65 I hope you enjoyed my talk as much I did writing it. All my details are here including the slides, for you to download. Go forth and Optimise!