Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AB Testing and UX - a love story with numbers and people (by Craig Sullivan at #NUX6)

517 views

Published on

AB Testing and UX - a love story with numbers and people

Slides from the NUX6 talk by Craig Sullivan, Friday 27th October 2017.
2017.nuxconf.uk / nuxuk.org

Synopsis:

What’s wrong with the web these days? The mobile experience sucks. The customer experience sucks. It doesn’t work. It’s too hard to use. The text is too small. Nobody measures this happening. The interaction patterns suck. Nobody ever calls up to complain but nobody does anything anyway. Millions of people lose countless days to friction, poor design and frustrating moments on their devices.

There may be thousands of things you can fix that look promising – but how do you know where to start? What if you could measure what sucked, where it sucked and how big the problem was? Using lightweight research methods and tools, you can stop making excuses and start knowing exactly what to do. Life becomes much simpler and easier with a scientific method of optimising growth or delight within your product.

Craig has trained over 500 people on how to measure and optimise their product experience, finding 100M of ‘lost revenue’ using just one of the techniques you will learn. With reports, checklists, downloadable templates and toolkits for every budget and stage of growth – you can stop guessing tomorrow.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

AB Testing and UX - a love story with numbers and people (by Craig Sullivan at #NUX6)

  1. 1. CRAIG SULLIVAN Senior Optimisation Consultant @OptimiseOrDie 24TH OCTOBER 2013 EMETRICS LONDON A LOVE STORY WITH NUMBERS AND PEOPLE 27T H OCT 2017 NUX MANCHESTER Optimiser Of Everything AB TESTING AND UX
  2. 2. Digital Product Design is And we’re all going to die… @OptimiseOrDie bit.ly/ProductPoo
  3. 3. BACK TO 1999 @OptimiseOrDie •  Now I’m known as ‘that testing guy’ •  But I started life as a UX practitioner •  >600 hours moderating in 3 years •  Everything I did started here •  Where did it take me?
  4. 4. •  UX, Analytics, Tools, blended. •  I coach people and teams on how to experiment profitably •  I’m a plumber looking for ‘ leaks’ in the delight or revenue for a business •  Most companies don’t know •  This is my job @OptimiseOrDie TODAY
  5. 5. Why do I get so angry? Because We are F***ing Stupid! @OptimiseOrDie
  6. 6. @OptimiseOrDieAnnual Sales in DE/FR/UK/IT/ES = £4.5Bn (2015) After Millions of Years of Evolution…
  7. 7. @OptimiseOrDie
  8. 8. @OptimiseOrDie 3 minutes later…
  9. 9. Reddit Explains… “If it comes off in one, it's gonna be banging! If you’re picking plastic out of your dinner for ten minutes, it's probably gonna taste like shit.” @OptimiseOrDie
  10. 10. Quick Calculation: •  £4.5Bn Sales @ £3.5 average price •  Assume 60% are failures •  Costs 90 seconds of wasted time •  72514285714 seconds wasted •  1208571428 minutes •  20142857 hours •  839,285 days of life - wasted on friction! •  What about injuries? Food wastage? Ennui? @OptimiseOrDie
  11. 11. @OptimiseOrDie How hard is this to fix?
  12. 12. A Packaging Manifesto: 1.  Packaging should never require a tool or knife to open 2.  Packaging should not require finger control or strength which discriminates against population groups 3.  Any packaging must be capable of being returned to its original state to preserve any unused contents @OptimiseOrDie
  13. 13. SECTION TITLE SWINGING IN SCOTLAND @OptimiseOrDie
  14. 14. SECTION TITLE “AB TESTING FACE”
  15. 15. “Burning Rubber Doughnuts in the Car Park of User Experience ” “Testing Debt, man”
  16. 16. @OptimiseOrDie 1. Change Growth TrajectoryWould you do it in retail?
  17. 17. Awful terrible, unbelievably bad shit 80% Passable 15% OK 4% <1% 5 YEARS AND 130 COMPANIES Good
  18. 18. @OptimiseOrDie “We are a data driven and customer centric organisation…” (but none of us have watched, read or listened to customer feedback in a year)
  19. 19. @OptimiseOrDie “We have Session Replay, Heatmaps, Form Analytics and Voice of Customer Feedback” (we just don’t look or act on what they tell us)
  20. 20. @OptimiseOrDie “We have agile product development teams” (but we haven’t reformed anywhere else - signoff takes a month)
  21. 21. @OptimiseOrDie “We have a huge Conversion Optimisation Team of 6 people” (but they’re working in a Silo, tinkering with button testing or trivial insights)
  22. 22. @OptimiseOrDie “We apply lean thinking at all levels of the organisation” (but we haven’t changed the way teams and managers actually work)
  23. 23. @OptimiseOrDie “We have the best tools, methodologies and team around” (but absolutely nothing useful happens)
  24. 24. @OptimiseOrDie Current State of the Internets
  25. 25. @OptimiseOrDie Is this a “Landing Page Experience”?
  26. 26. www.ba.com/gatwick
  27. 27. www.ba.com/gatwick
  28. 28. <2-3mm Last night I saw an advert…
  29. 29. Last night I saw an advert…
  30. 30. Are you f***ing kidding me?
  31. 31. And it gets worse!
  32. 32. And it gets worse!
  33. 33. Screwing up Single Fields! Spell Checker Enabled Wrong Keyboard Chosen @LukeW
  34. 34. The Pinnacle of Optimisation?
  35. 35. Point Optimised Persuasion Ninjas
  36. 36. The Ikea of Psychological Ninja Messaging? •  Like a ‘Franchise’ store •  Every space competing directly •  Maxed out on persuasion •  Point optimised •  Works ‘on average’ •  Holistic Experience? •  How do I actually FEEL?
  37. 37. The Ikea of Psychological Ninja Messaging? CMA to probe Hotels: bbc.in/2hfh70r
  38. 38. @OptimiseOrDie Persuasion Motivation Usability Broken shit How to Shi8 the Needle 101
  39. 39. @OptimiseOrDie •  Clothing 1M per month, 2 bugs •  Travel 2M per month, 13 bugs •  Telecoms 1.4M per month, 3 bugs •  Travel 0.6M per month, 1 bug (iPad) •  Event Site 2.5M per month, 11 bugs –  These were only the first bugs we found –  < 5 days time to fix – an ROI of 91,150% –  Larger than the annual IT budget for some! How much do product defects cost?
  40. 40. When’s the last @me you checked these? @OptimiseOrDie Marke-ng Whiffle Can people read it? Can people use it? Does it load and work quickly? Does it work on customer devices?
  41. 41. @OptimiseOrDie Just add AB tes@ng for instant sparkle!
  42. 42. Why is this stuff important? “Because an experience is only as good as its shittiest part.” @OptimiseOrDie
  43. 43. •  We’re swamped with data •  The underlying data is often flawed or unreliable •  We don’t know what’s wrong with our products •  We change stuff without evidence or observation •  We build stuff nobody wants or needs •  Product changes are not measured or tested •  Nobody takes stuff away from the website •  We build what internal people want, not customers •  We don’t remove costly defects and wastage •  Lean & Agile have left us with new types of debt •  Ego is the enemy of all good work •  Much Internet design is infused and driven by Ego @OptimiseOrDie What is going wrong?
  44. 44. Thx @sdelbecqui @ericries @ycaseau
  45. 45. @OptimiseOrDie Lean Organisations: Pinterest Twitter Linkedin Dropbox BitTorrent Airbnb FT ING Spotify Booking.com
  46. 46. @OptimiseOrDie •  Copywriter •  Data scien-st •  Product Lead •  UX Researcher •  Developer •  Designer Booking.com - Corporate Models
  47. 47. @OptimiseOrDie FT.COM - Decoupling
  48. 48. @OptimiseOrDie •  MVP approach •  Launch as alpha, beta, pilot, phased rollout •  This approach includes split testing to tune, optimize or decide! •  100s of releases a day! •  Like getting in a shower •  Read more at labs.ft.com FT.COM – Release Methodology
  49. 49. @OptimiseOrDie •  Small teams with direct access to publish •  Ability to set and get metrics data directly •  Tools, Autonomy, Lack of interference •  No Project Managers or Business Analysts •  Stakeholders decoupled from work FT.COM - Corporate Models bit.ly/2mZAxH7
  50. 50. @OptimiseOrDie •  Business wants: “15% increase in 12 months” •  Business defines ‘outcomes’ – teams deliver •  No long signoff chain •  No pesky meddling during execution •  Less time wasting meetings with morons! •  Outcome not feature driven optimization FT.COM - Decoupling bit.ly/2mZAxH7
  51. 51. @OptimiseOrDie •  Most product changes we make are a waste of time! •  The Financial Times – 50-70% of product developer time wasted in a siloed organisational structure •  Minimum project time was “At least 18 months” for anything •  Projects were always late, underperformed, didn’t shift metrics as expected •  Post changes, every IT project was under time & budget •  Products were iterated to hack their way to success FT.COM – Advantages
  52. 52. @OptimiseOrDie “Product changes and AB tests are just the same thing to us. Where we’re moving to is measuring and testing everything we do. This approach tells us what works, what doesn’t, what to keep, what to remove and what we need to iterate.” Financial Times – NextFT project FT.COM – Advantages
  53. 53. •  Reform the Organisation, not the tools •  Invest in analytics and people skills •  Set people free to learn new disciplines and skills •  Stop delivering ‘features’ and start doing ‘outcomes’ •  Validate everything - changes, tests, personalisation •  Test everything that *can* be tested •  Stop Ego from driving product – use insights & data •  Continuous improvement rather than large releases •  Use multiple product ‘horizons’ for strategy •  Blend Lean, Agile, Data, UX and AB testing •  Let’s give you some tips to take away and use! @OptimiseOrDie What can we do about it?
  54. 54. @OptimiseOrDie Conversion Research in One Hour: bit.ly/batmantesting Session Replay Tools for Research: bit.ly/decibelWAW & bit.ly/ReplayTools The List of UX Tools to Rule Them All: bit.ly/UxTools & bit.ly/uxresourcepack Lean Research – Further Reading:
  55. 55. @OptimiseOrDie Lean Organisations – resources: How Booking.com tests & organises teams: http://bit.ly/SFrisby Growth Team Models & Explanations: http://bit.ly/GHmodel ING video on Lean Corporate: http://bit.ly/ingvideo
  56. 56. 1.  Find out your Testing Bandwidth - Can you run any tests at all? 2.  Stop copying stuff blindly - Test or validate changes, don’t assume 3.  Get with Statistics - Doing testing? You need stats skillz 4.  Run a test Calculator - How long it takes and when to stop 5.  Prioritise All the Testing - Do the right tests in the right order 6.  Get a Decent Hypothesis - Forget this, forget your AB testing 7.  Record the device experience - Store the screen & recordings 8.  Simplify your Test Decisions - Stop overcomplicating test analysis 9.  Avoid Toxic AB Testing - Defects ruin your work and decisions 10.  Test Across 3 Horizons of Product - How to test ahead of yourself @OptimiseOrDie 10 AB Testing Power-ups
  57. 57. >1200 tests a year >1000 tests live 99% chance you’re in 10 tests 7000 tests in 2011 100,000 tests last year @BartS 1: Find Out your Testing Bandwidth
  58. 58. 1: Do you have testing bandwidth? •  This is a little Amoeba •  It hasn’t evolved to ride a bike yet •  Do you have enough traffic? •  < 500 ‘conversions’ a month? •  Work on product/market fit! •  Work that user research! @OptimiseOrDie
  59. 59. 2: Stop Copying Stuff Blindly… •  Your customers are not the same •  Your site is not the same •  Your platform is not the same •  Your advertising isn’t the same •  The traffic isn’t the same •  The UX is not the same •  Your Device Mix is not the same •  Your mistakes are different bit.ly/BestPUnicorns bit.ly/BPSuccess bit.ly/BestpFAIL @OptimiseOrDie
  60. 60. @OptimiseOrDie “STOP copying your competitors. They may not know what the f*** they are doing either” Peep Laja, ConversionXL
  61. 61. @OptimiseOrDie “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP “Statistics for Dummies” http://bit.ly/stats4dummies 3: Get With Statistics
  62. 62. @OptimiseOrDie •  Run a test calculator first •  If the answer is 9 Million years, do something else instead! Online Dialogue Suite: bit.ly/CaLC1 VWO : bit.ly/CaLC2 Optimizely : bit.ly/CaLC3 Evan Miller : bit.ly/CaLC5 •  Testing is trading time and effort vs. potential reward 4: Run a Test Calculator
  63. 63. @OptimiseOrDie §  Quadrants §  PIE §  ICE §  ICE redux §  TIR §  Money Model §  Hotwire §  PXL 5: Prioritise Everything!
  64. 64. @OptimiseOrDie Target Users per week CR % AOV Each 5% 5% li8 detected sample Min Weeks 10% li8 detected sample Min Weeks 15% li8 detected sample Min weeks Basket 100,000 5% £140 £35,000 48098 1 12301 1 5590 1 Landing 1 60,000 3% £190 £17,100 25566 3 6531 1 9528 1 Landing 2 18,000 0.6% £100 £540 419876 47 107513 12 48913 6 •  How long will it take to see a difference of 5, 10 and 15%? •  How much sample are you going to need to ‘detect’ an effect you’re hoping for •  S-ck to tests that have decent sta-s-cal power, to save was-ng -me! •  Read these two ar-cles: https://inbound.org/discuss/conversionxl-live-notes-slides-and-q-a- slideshare-s-now-included#paulinemarol https://conversionxl.com/better-way-prioritize-ab-tests/ 5: Statistical Power & Money Model
  65. 65. @OptimiseOrDie bit.ly/hyp_kit 6: Get a Proper Test Hypothesis
  66. 66. @OptimiseOrDie Problem Statement Hypothesis Valida-on Rejected Hypothesis Hypothesis Valida-on AB TEST Build/Measure/Learn Hypothesis Valida-on MVP Build/Measure/Learn 6: Get a Hypothesis Hierarchy bit.ly/2mZAxH7
  67. 67. @OptimiseOrDie •  Store the design layout in Google Analytics or other package •  Use this to segment the results by the size of the design seen! 7: Record the Design people ‘see’
  68. 68. @OptimiseOrDie bit.ly/Zoltar1 8: Simplify your Test Decisions
  69. 69. bit.ly/devicetesting 9: Avoid Toxic Testing using QA
  70. 70. Current Product Op-misa-on MVP, Next Product Feature op-misa-on Bedding in Unknown Product Discovery & Explora-on 10: Test Across 3 Horizons bit.ly/3horiz
  71. 71. 1.  Get an Analytics Audit - Broken tills mean broken company 2.  Make a Device List - Prove you know this please 3.  Test and Fix Broken shit - Customers don’t self report problems 4.  Do Rapid Usability testing - Low cost and vital for great ideas 5.  Add Session Recording - See broken stuff, get evidence, fix it 6.  Run a Customer Survey - Regular surveys provide long term insight 7.  Install Voice of Customer tools - This asking people shit will never catch on 8.  Hire a Copywriter - Best money you’ll ever spend 9.  Invest in Analytics - Google Analytics is NOT free 10.  Add Form Analytics - Stop guessing and know what happens 11.  Check your Web Performance - Performance makes £££ @OptimiseOrDie 11 Optimisation Power-ups
  72. 72. @OptimiseOrDie 1: Get an Analytics Audit, NOW! “Check the tills are working AND recording correctly!” @OptimiseOrDie @CharlesMeaden
  73. 73. •  What’s YOUR testing list? •  Anyone checked if it’s true? •  Use Google Analytics to tell you! •  Tip – do these as a minimum: •  iPhone 5/6 •  iPad late model •  Samsung late model •  Desktop (Chrome, Edge, Safari) bit.ly/UltimateXDO @OptimiseOrDie 2: WHAT do they ACTUALLY use
  74. 74. @OptimiseOrDie End Device Discrimination TODAY! •  If your shop doors aren’t wide enough? •  Is it your fault or the customer?
  75. 75. @OptimiseOrDie Desktop Testing Crossbrowsertesting.com Browserstack.com Mobile & Tablet Deviceanywhere.com Aws.amazon.com/device-farm bit.ly/MobileDeviceLab 3: Test Real Customer Handsets – NOW!
  76. 76. Som, feedback Concepts: Usability Hub www.usabilityhub.com Five second test www.fivesecondtest.com No Lab Testing: Usertesting www.usertesting.com Userlytics www.userlytics.com Userzoom www.userzoom.com Ethnio www.ethnio.com What Users Do www.whatusersdo.com Loop11 www.loop11.com 77 4: Validation & Usability tools
  77. 77. Som, feedback DESKTOP & LAPTOP CamStudio (free) www.camstudio.org Mediacam AV (cheap) www.netu2.co.uk Silverback (Mac) www.silverbackapp.com Screenflow (Mac) www.telestream.net MOBILE UX Recorder (iOS) www.uxrecorder.com Skype Hugging bit.ly/tesTfm Airplay Mirroring bit.ly/JnwtMo DIARY STUDIES Get rich Context bit.ly/multichannels 78 4: Guerrilla & Diary Research:
  78. 78. @OptimiseOrDie 4: Shitty User Research – Stop It! What the f*** is going on here: •  Biased samples from recruitment & screening •  Ineffably bad or biased moderator scripts •  Focus on quantitative questions in qual research •  Biased online panels (serial testers, poor screening) •  Low cost does not mean good fidelity •  Moderation as a skill to me seems ‘less valued’
  79. 79. 1.5 Interviewing Tips •  Doing this stuff properly is hard •  Interviewing and testing tips: “Don’t Make Me Think” amzn.to/1gIZEJn “Rocket Surgery Made Easy” amzn.to/1e0hnUL “Talking to Customers” bit.ly/1e0hT58 “Talking with Participants” bit.ly/1kKL3LE “Don’t listen to Users” bit.ly/1cQpiIE “Interviewing Tips” bit.ly/1fKqu03 “More interviewing Tips” bit.ly/1bmvGT 80
  80. 80. @OptimiseOrDie 5: Session Recording for “User Struggle” •  Helping you find out WHY! Decibel Insight www.decibelinsight.com Hotjar www.hotjar.com Inspectlet www.inspectlet.com SessionCam www.sessioncam.com Clicktale www.clicktale.com Mouseflow www.mouseflow.com Ghostrec www.ghostrec.com Yandex WebVisor (free) metrica.yandex.com/promo/webvisor
  81. 81. @OptimiseOrDie SurveyGizmo www.surveygizmo.com Google Forms www.google.com/forms/about/ Surveymonkey www.surveymonkey.co.uk Typeform www.typeform.com Qualtrics www.qualtrics.com 6: Run a Customer Survey
  82. 82. @OptimiseOrDie 7: Install Voice of Customer or Polling Tools HotJar www.hotjar.com Kampyle www.kampyle.com Qualaroo www.qualaroo.com Webengage www.webengage.com
  83. 83. @OptimiseOrDie 8: Pay for a Copywriter FFS
  84. 84. @OptimiseOrDie 8: The Economics of Copywriting “Stop asking people to fill-in by writing copy. Especially don’t ask developers to do this! Hire staff or train them. Bad copy on your site costs more money than hiring someone. Good copy earns tens or hundreds of times that cost..”
  85. 85. @OptimiseOrDie 9: Invest in Analytics •  Google Analytics is NOT free •  Your Car will not run without Oil •  Your Analytics will not run without Love •  Put budget into optimising the analytics! It’s like writing a love note to your future self! Do it today. Start writing it.
  86. 86. @OptimiseOrDie 10: Add Form Analytics Formisimo www.formisimo.com Heap analytics www.heapanalytics.com Hotjar www.hotjar.com Google Tag Manager & Analytics bit.ly/SimoGA “This is so we can call you in case there’s a problem delivering your order.”
  87. 87. 11: Check your Web Performance •  Turn off Wi-Fi at all times! •  Use these performance tools: Mobile & Desktop Mobile Speed test Google Pagespeed Insights: •  Google analytics TELLS you what’s slow: •  Google Statistics on Site Performance: (< 2 seconds good, 2-3 just OK, 3+ bad) @OptimiseOrDie
  88. 88. 1.  Get an Analytics Audit - Broken tills mean broken company 2.  Make a Device List - Prove you know this please 3.  Test and Fix Broken shit - Customers don’t self report problems 4.  Do Rapid Usability testing - Low cost and vital for great ideas 5.  Add Session Recording - See broken stuff, get evidence, fix it 6.  Run a Customer Survey - Regular surveys provide long term insight 7.  Install Voice of Customer tools - This asking people shit will never catch on 8.  Hire a Copywriter - Best money you’ll ever spend 9.  Invest in Analytics - Google Analytics is NOT free 10.  Add Form Analytics - Stop guessing and know what happens 11.  Check your Web Performance - Performance makes £££ @OptimiseOrDie 11 Optimisation Power-ups
  89. 89. SO WHAT’S THE MESSAGE?
  90. 90. SECTION TITLE 1 MOST AB TESTING IS DOG POO 2 CROSS DEVICE EXPERIENCES SUCK 3 DEFECTS AND WASTAGE ARE HUGE 4 EONS ARE WASTED BUILDING USELESS STUFF 5 PEOPLE & CULTURE MAKE PROBLEMS, NOT TOOLS 6 DATA DRIVES NOTHING, PEOPLE ARE THE LENS 7 UX, ANALYTICS, TESTING & TOOLS = SUCCESS 8 TEAM PRODUCTIVITY & QUALITY STILL SUCK 9 REFORM THE COMPANY, NOT THE TECHNOLOGY 10 QUAL WITH QUANT TRIANGULATES & FOCUSES SUMMARY
  91. 91. 92
  92. 92. @OptimiseOrDie The Experience Optimisation Engine (1) Pre Model Checklist Groundwork and common sense checks before any modelling or research begins (2) Rapid Modelling Basic sketching & modelling to prioritise focal areas for research, reduce scope (3) Research or Heuristic Use upfront & rapid analytics insights to prioritise focal areas for research time (4) Model Post research – model the site around priorities from research, analytics & more (5) Streaming This is where the big pile of issues gets turned into different project backlogs (6) Triage & Triangulate Where you cluster themes or issues, triangulate clues from multiple tools and run sizing against test targets (7) Prioritisation Where you bubble the most promising test ideas to the top of the stack bit.ly/CRO4Agencies
  93. 93. s zn+1 = zn 2 + c Your are Optimising a Business: @OptimiseOrDie
  94. 94. CRAIG SULLIVAN Senior Optimisation Consultant @OptimiseOrDie 24TH OCTOBER 2013 EMETRICS LONDONOptimiser Of Everything THANK YOU! Manchester Workshop Next Week: bit.ly/MancClass 20 hours of training for £380: bit.ly/CRO4Agencies Get this Deck: http://bit.ly/ProductPoo Read my stuff: medium.com/@optimiseordie @OptimiseOrDie

×