Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AB Testing and UX - a love story with numbers and people (by Craig Sullivan at #NUX6)


Published on

AB Testing and UX - a love story with numbers and people

Slides from the NUX6 talk by Craig Sullivan, Friday 27th October 2017. /


What’s wrong with the web these days? The mobile experience sucks. The customer experience sucks. It doesn’t work. It’s too hard to use. The text is too small. Nobody measures this happening. The interaction patterns suck. Nobody ever calls up to complain but nobody does anything anyway. Millions of people lose countless days to friction, poor design and frustrating moments on their devices.

There may be thousands of things you can fix that look promising – but how do you know where to start? What if you could measure what sucked, where it sucked and how big the problem was? Using lightweight research methods and tools, you can stop making excuses and start knowing exactly what to do. Life becomes much simpler and easier with a scientific method of optimising growth or delight within your product.

Craig has trained over 500 people on how to measure and optimise their product experience, finding 100M of ‘lost revenue’ using just one of the techniques you will learn. With reports, checklists, downloadable templates and toolkits for every budget and stage of growth – you can stop guessing tomorrow.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

AB Testing and UX - a love story with numbers and people (by Craig Sullivan at #NUX6)

  2. 2. Digital Product Design is And we’re all going to die… @OptimiseOrDie
  3. 3. BACK TO 1999 @OptimiseOrDie •  Now I’m known as ‘that testing guy’ •  But I started life as a UX practitioner •  >600 hours moderating in 3 years •  Everything I did started here •  Where did it take me?
  4. 4. •  UX, Analytics, Tools, blended. •  I coach people and teams on how to experiment profitably •  I’m a plumber looking for ‘ leaks’ in the delight or revenue for a business •  Most companies don’t know •  This is my job @OptimiseOrDie TODAY
  5. 5. Why do I get so angry? Because We are F***ing Stupid! @OptimiseOrDie
  6. 6. @OptimiseOrDieAnnual Sales in DE/FR/UK/IT/ES = £4.5Bn (2015) After Millions of Years of Evolution…
  7. 7. @OptimiseOrDie
  8. 8. @OptimiseOrDie 3 minutes later…
  9. 9. Reddit Explains… “If it comes off in one, it's gonna be banging! If you’re picking plastic out of your dinner for ten minutes, it's probably gonna taste like shit.” @OptimiseOrDie
  10. 10. Quick Calculation: •  £4.5Bn Sales @ £3.5 average price •  Assume 60% are failures •  Costs 90 seconds of wasted time •  72514285714 seconds wasted •  1208571428 minutes •  20142857 hours •  839,285 days of life - wasted on friction! •  What about injuries? Food wastage? Ennui? @OptimiseOrDie
  11. 11. @OptimiseOrDie How hard is this to fix?
  12. 12. A Packaging Manifesto: 1.  Packaging should never require a tool or knife to open 2.  Packaging should not require finger control or strength which discriminates against population groups 3.  Any packaging must be capable of being returned to its original state to preserve any unused contents @OptimiseOrDie
  15. 15. “Burning Rubber Doughnuts in the Car Park of User Experience ” “Testing Debt, man”
  16. 16. @OptimiseOrDie 1. Change Growth TrajectoryWould you do it in retail?
  17. 17. Awful terrible, unbelievably bad shit 80% Passable 15% OK 4% <1% 5 YEARS AND 130 COMPANIES Good
  18. 18. @OptimiseOrDie “We are a data driven and customer centric organisation…” (but none of us have watched, read or listened to customer feedback in a year)
  19. 19. @OptimiseOrDie “We have Session Replay, Heatmaps, Form Analytics and Voice of Customer Feedback” (we just don’t look or act on what they tell us)
  20. 20. @OptimiseOrDie “We have agile product development teams” (but we haven’t reformed anywhere else - signoff takes a month)
  21. 21. @OptimiseOrDie “We have a huge Conversion Optimisation Team of 6 people” (but they’re working in a Silo, tinkering with button testing or trivial insights)
  22. 22. @OptimiseOrDie “We apply lean thinking at all levels of the organisation” (but we haven’t changed the way teams and managers actually work)
  23. 23. @OptimiseOrDie “We have the best tools, methodologies and team around” (but absolutely nothing useful happens)
  24. 24. @OptimiseOrDie Current State of the Internets
  25. 25. @OptimiseOrDie Is this a “Landing Page Experience”?
  26. 26.
  27. 27.
  28. 28. <2-3mm Last night I saw an advert…
  29. 29. Last night I saw an advert…
  30. 30. Are you f***ing kidding me?
  31. 31. And it gets worse!
  32. 32. And it gets worse!
  33. 33. Screwing up Single Fields! Spell Checker Enabled Wrong Keyboard Chosen @LukeW
  34. 34. The Pinnacle of Optimisation?
  35. 35. Point Optimised Persuasion Ninjas
  36. 36. The Ikea of Psychological Ninja Messaging? •  Like a ‘Franchise’ store •  Every space competing directly •  Maxed out on persuasion •  Point optimised •  Works ‘on average’ •  Holistic Experience? •  How do I actually FEEL?
  37. 37. The Ikea of Psychological Ninja Messaging? CMA to probe Hotels:
  38. 38. @OptimiseOrDie Persuasion Motivation Usability Broken shit How to Shi8 the Needle 101
  39. 39. @OptimiseOrDie •  Clothing 1M per month, 2 bugs •  Travel 2M per month, 13 bugs •  Telecoms 1.4M per month, 3 bugs •  Travel 0.6M per month, 1 bug (iPad) •  Event Site 2.5M per month, 11 bugs –  These were only the first bugs we found –  < 5 days time to fix – an ROI of 91,150% –  Larger than the annual IT budget for some! How much do product defects cost?
  40. 40. When’s the last @me you checked these? @OptimiseOrDie Marke-ng Whiffle Can people read it? Can people use it? Does it load and work quickly? Does it work on customer devices?
  41. 41. @OptimiseOrDie Just add AB tes@ng for instant sparkle!
  42. 42. Why is this stuff important? “Because an experience is only as good as its shittiest part.” @OptimiseOrDie
  43. 43. •  We’re swamped with data •  The underlying data is often flawed or unreliable •  We don’t know what’s wrong with our products •  We change stuff without evidence or observation •  We build stuff nobody wants or needs •  Product changes are not measured or tested •  Nobody takes stuff away from the website •  We build what internal people want, not customers •  We don’t remove costly defects and wastage •  Lean & Agile have left us with new types of debt •  Ego is the enemy of all good work •  Much Internet design is infused and driven by Ego @OptimiseOrDie What is going wrong?
  44. 44. Thx @sdelbecqui @ericries @ycaseau
  45. 45. @OptimiseOrDie Lean Organisations: Pinterest Twitter Linkedin Dropbox BitTorrent Airbnb FT ING Spotify
  46. 46. @OptimiseOrDie •  Copywriter •  Data scien-st •  Product Lead •  UX Researcher •  Developer •  Designer - Corporate Models
  47. 47. @OptimiseOrDie FT.COM - Decoupling
  48. 48. @OptimiseOrDie •  MVP approach •  Launch as alpha, beta, pilot, phased rollout •  This approach includes split testing to tune, optimize or decide! •  100s of releases a day! •  Like getting in a shower •  Read more at FT.COM – Release Methodology
  49. 49. @OptimiseOrDie •  Small teams with direct access to publish •  Ability to set and get metrics data directly •  Tools, Autonomy, Lack of interference •  No Project Managers or Business Analysts •  Stakeholders decoupled from work FT.COM - Corporate Models
  50. 50. @OptimiseOrDie •  Business wants: “15% increase in 12 months” •  Business defines ‘outcomes’ – teams deliver •  No long signoff chain •  No pesky meddling during execution •  Less time wasting meetings with morons! •  Outcome not feature driven optimization FT.COM - Decoupling
  51. 51. @OptimiseOrDie •  Most product changes we make are a waste of time! •  The Financial Times – 50-70% of product developer time wasted in a siloed organisational structure •  Minimum project time was “At least 18 months” for anything •  Projects were always late, underperformed, didn’t shift metrics as expected •  Post changes, every IT project was under time & budget •  Products were iterated to hack their way to success FT.COM – Advantages
  52. 52. @OptimiseOrDie “Product changes and AB tests are just the same thing to us. Where we’re moving to is measuring and testing everything we do. This approach tells us what works, what doesn’t, what to keep, what to remove and what we need to iterate.” Financial Times – NextFT project FT.COM – Advantages
  53. 53. •  Reform the Organisation, not the tools •  Invest in analytics and people skills •  Set people free to learn new disciplines and skills •  Stop delivering ‘features’ and start doing ‘outcomes’ •  Validate everything - changes, tests, personalisation •  Test everything that *can* be tested •  Stop Ego from driving product – use insights & data •  Continuous improvement rather than large releases •  Use multiple product ‘horizons’ for strategy •  Blend Lean, Agile, Data, UX and AB testing •  Let’s give you some tips to take away and use! @OptimiseOrDie What can we do about it?
  54. 54. @OptimiseOrDie Conversion Research in One Hour: Session Replay Tools for Research: & The List of UX Tools to Rule Them All: & Lean Research – Further Reading:
  55. 55. @OptimiseOrDie Lean Organisations – resources: How tests & organises teams: Growth Team Models & Explanations: ING video on Lean Corporate:
  56. 56. 1.  Find out your Testing Bandwidth - Can you run any tests at all? 2.  Stop copying stuff blindly - Test or validate changes, don’t assume 3.  Get with Statistics - Doing testing? You need stats skillz 4.  Run a test Calculator - How long it takes and when to stop 5.  Prioritise All the Testing - Do the right tests in the right order 6.  Get a Decent Hypothesis - Forget this, forget your AB testing 7.  Record the device experience - Store the screen & recordings 8.  Simplify your Test Decisions - Stop overcomplicating test analysis 9.  Avoid Toxic AB Testing - Defects ruin your work and decisions 10.  Test Across 3 Horizons of Product - How to test ahead of yourself @OptimiseOrDie 10 AB Testing Power-ups
  57. 57. >1200 tests a year >1000 tests live 99% chance you’re in 10 tests 7000 tests in 2011 100,000 tests last year @BartS 1: Find Out your Testing Bandwidth
  58. 58. 1: Do you have testing bandwidth? •  This is a little Amoeba •  It hasn’t evolved to ride a bike yet •  Do you have enough traffic? •  < 500 ‘conversions’ a month? •  Work on product/market fit! •  Work that user research! @OptimiseOrDie
  59. 59. 2: Stop Copying Stuff Blindly… •  Your customers are not the same •  Your site is not the same •  Your platform is not the same •  Your advertising isn’t the same •  The traffic isn’t the same •  The UX is not the same •  Your Device Mix is not the same •  Your mistakes are different @OptimiseOrDie
  60. 60. @OptimiseOrDie “STOP copying your competitors. They may not know what the f*** they are doing either” Peep Laja, ConversionXL
  61. 61. @OptimiseOrDie “Why every Internet Marketer should be a Statistician” “Statistical Significance does not equal Validity” “Understanding the Cycles in your site” “Statistics for Dummies” 3: Get With Statistics
  62. 62. @OptimiseOrDie •  Run a test calculator first •  If the answer is 9 Million years, do something else instead! Online Dialogue Suite: VWO : Optimizely : Evan Miller : •  Testing is trading time and effort vs. potential reward 4: Run a Test Calculator
  63. 63. @OptimiseOrDie §  Quadrants §  PIE §  ICE §  ICE redux §  TIR §  Money Model §  Hotwire §  PXL 5: Prioritise Everything!
  64. 64. @OptimiseOrDie Target Users per week CR % AOV Each 5% 5% li8 detected sample Min Weeks 10% li8 detected sample Min Weeks 15% li8 detected sample Min weeks Basket 100,000 5% £140 £35,000 48098 1 12301 1 5590 1 Landing 1 60,000 3% £190 £17,100 25566 3 6531 1 9528 1 Landing 2 18,000 0.6% £100 £540 419876 47 107513 12 48913 6 •  How long will it take to see a difference of 5, 10 and 15%? •  How much sample are you going to need to ‘detect’ an effect you’re hoping for •  S-ck to tests that have decent sta-s-cal power, to save was-ng -me! •  Read these two ar-cles: slideshare-s-now-included#paulinemarol 5: Statistical Power & Money Model
  65. 65. @OptimiseOrDie 6: Get a Proper Test Hypothesis
  66. 66. @OptimiseOrDie Problem Statement Hypothesis Valida-on Rejected Hypothesis Hypothesis Valida-on AB TEST Build/Measure/Learn Hypothesis Valida-on MVP Build/Measure/Learn 6: Get a Hypothesis Hierarchy
  67. 67. @OptimiseOrDie •  Store the design layout in Google Analytics or other package •  Use this to segment the results by the size of the design seen! 7: Record the Design people ‘see’
  68. 68. @OptimiseOrDie 8: Simplify your Test Decisions
  69. 69. 9: Avoid Toxic Testing using QA
  70. 70. Current Product Op-misa-on MVP, Next Product Feature op-misa-on Bedding in Unknown Product Discovery & Explora-on 10: Test Across 3 Horizons
  71. 71. 1.  Get an Analytics Audit - Broken tills mean broken company 2.  Make a Device List - Prove you know this please 3.  Test and Fix Broken shit - Customers don’t self report problems 4.  Do Rapid Usability testing - Low cost and vital for great ideas 5.  Add Session Recording - See broken stuff, get evidence, fix it 6.  Run a Customer Survey - Regular surveys provide long term insight 7.  Install Voice of Customer tools - This asking people shit will never catch on 8.  Hire a Copywriter - Best money you’ll ever spend 9.  Invest in Analytics - Google Analytics is NOT free 10.  Add Form Analytics - Stop guessing and know what happens 11.  Check your Web Performance - Performance makes £££ @OptimiseOrDie 11 Optimisation Power-ups
  72. 72. @OptimiseOrDie 1: Get an Analytics Audit, NOW! “Check the tills are working AND recording correctly!” @OptimiseOrDie @CharlesMeaden
  73. 73. •  What’s YOUR testing list? •  Anyone checked if it’s true? •  Use Google Analytics to tell you! •  Tip – do these as a minimum: •  iPhone 5/6 •  iPad late model •  Samsung late model •  Desktop (Chrome, Edge, Safari) @OptimiseOrDie 2: WHAT do they ACTUALLY use
  74. 74. @OptimiseOrDie End Device Discrimination TODAY! •  If your shop doors aren’t wide enough? •  Is it your fault or the customer?
  75. 75. @OptimiseOrDie Desktop Testing Mobile & Tablet 3: Test Real Customer Handsets – NOW!
  76. 76. Som, feedback Concepts: Usability Hub Five second test No Lab Testing: Usertesting Userlytics Userzoom Ethnio What Users Do Loop11 77 4: Validation & Usability tools
  77. 77. Som, feedback DESKTOP & LAPTOP CamStudio (free) Mediacam AV (cheap) Silverback (Mac) Screenflow (Mac) MOBILE UX Recorder (iOS) Skype Hugging Airplay Mirroring DIARY STUDIES Get rich Context 78 4: Guerrilla & Diary Research:
  78. 78. @OptimiseOrDie 4: Shitty User Research – Stop It! What the f*** is going on here: •  Biased samples from recruitment & screening •  Ineffably bad or biased moderator scripts •  Focus on quantitative questions in qual research •  Biased online panels (serial testers, poor screening) •  Low cost does not mean good fidelity •  Moderation as a skill to me seems ‘less valued’
  79. 79. 1.5 Interviewing Tips •  Doing this stuff properly is hard •  Interviewing and testing tips: “Don’t Make Me Think” “Rocket Surgery Made Easy” “Talking to Customers” “Talking with Participants” “Don’t listen to Users” “Interviewing Tips” “More interviewing Tips” 80
  80. 80. @OptimiseOrDie 5: Session Recording for “User Struggle” •  Helping you find out WHY! Decibel Insight Hotjar Inspectlet SessionCam Clicktale Mouseflow Ghostrec Yandex WebVisor (free)
  81. 81. @OptimiseOrDie SurveyGizmo Google Forms Surveymonkey Typeform Qualtrics 6: Run a Customer Survey
  82. 82. @OptimiseOrDie 7: Install Voice of Customer or Polling Tools HotJar Kampyle Qualaroo Webengage
  83. 83. @OptimiseOrDie 8: Pay for a Copywriter FFS
  84. 84. @OptimiseOrDie 8: The Economics of Copywriting “Stop asking people to fill-in by writing copy. Especially don’t ask developers to do this! Hire staff or train them. Bad copy on your site costs more money than hiring someone. Good copy earns tens or hundreds of times that cost..”
  85. 85. @OptimiseOrDie 9: Invest in Analytics •  Google Analytics is NOT free •  Your Car will not run without Oil •  Your Analytics will not run without Love •  Put budget into optimising the analytics! It’s like writing a love note to your future self! Do it today. Start writing it.
  86. 86. @OptimiseOrDie 10: Add Form Analytics Formisimo Heap analytics Hotjar Google Tag Manager & Analytics “This is so we can call you in case there’s a problem delivering your order.”
  87. 87. 11: Check your Web Performance •  Turn off Wi-Fi at all times! •  Use these performance tools: Mobile & Desktop Mobile Speed test Google Pagespeed Insights: •  Google analytics TELLS you what’s slow: •  Google Statistics on Site Performance: (< 2 seconds good, 2-3 just OK, 3+ bad) @OptimiseOrDie
  88. 88. 1.  Get an Analytics Audit - Broken tills mean broken company 2.  Make a Device List - Prove you know this please 3.  Test and Fix Broken shit - Customers don’t self report problems 4.  Do Rapid Usability testing - Low cost and vital for great ideas 5.  Add Session Recording - See broken stuff, get evidence, fix it 6.  Run a Customer Survey - Regular surveys provide long term insight 7.  Install Voice of Customer tools - This asking people shit will never catch on 8.  Hire a Copywriter - Best money you’ll ever spend 9.  Invest in Analytics - Google Analytics is NOT free 10.  Add Form Analytics - Stop guessing and know what happens 11.  Check your Web Performance - Performance makes £££ @OptimiseOrDie 11 Optimisation Power-ups
  91. 91. 92
  92. 92. @OptimiseOrDie The Experience Optimisation Engine (1) Pre Model Checklist Groundwork and common sense checks before any modelling or research begins (2) Rapid Modelling Basic sketching & modelling to prioritise focal areas for research, reduce scope (3) Research or Heuristic Use upfront & rapid analytics insights to prioritise focal areas for research time (4) Model Post research – model the site around priorities from research, analytics & more (5) Streaming This is where the big pile of issues gets turned into different project backlogs (6) Triage & Triangulate Where you cluster themes or issues, triangulate clues from multiple tools and run sizing against test targets (7) Prioritisation Where you bubble the most promising test ideas to the top of the stack
  93. 93. s zn+1 = zn 2 + c Your are Optimising a Business: @OptimiseOrDie
  94. 94. CRAIG SULLIVAN Senior Optimisation Consultant @OptimiseOrDie 24TH OCTOBER 2013 EMETRICS LONDONOptimiser Of Everything THANK YOU! Manchester Workshop Next Week: 20 hours of training for £380: Get this Deck: Read my stuff: @OptimiseOrDie