Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jared Spool (Founder, UIE) - Design Metrics That Matter

297 views

Published on

This video for this talk from Business of Software Conference USA 2018 will be published here soon: http://businessofsoftware.org/videos

Published in: Business
  • Be the first to comment

Jared Spool (Founder, UIE) - Design Metrics That Matter

  1. 1. @jmspool Jared M. Spool Is Design 
 Metrically Opposed?
  2. 2. @jmspool Jared M. Spool Is Design 
 Metrically Opposed?
  3. 3. 1Are we measuring the right thing?
  4. 4. @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ 33 Email Addresses @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ 77 Email Addresses @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @
  5. 5. @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ 33 Email Addresses 77 Email Addresses More email addresses are better @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ All email addresses are equal
  6. 6. @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ More email addresses are better All email addresses are equal Inferences Observations
  7. 7. Inferences Observations
  8. 8. Design DecisionsInferencesObservations Design Decisions Inferences Observations
  9. 9. Design DecisionsInferencesObservations More email addresses are better 2nd variant had more email addresses Use 2nd variant
  10. 10. Design DecisionsInferencesObservations Why we think it happened? What did we see? How will we improve the design?
  11. 11. Log file filled with “ ” (blanks)
  12. 12. Enter button focus was broken Didn’t enter any search text Was looking for advanced search Log file is broken
  13. 13. Design DecisionsInferencesObservations Broken focus Log file contained blanks Fix Focus No search text Advanced search Log file bug Add label to search box Build advanced
 search feature Fix bug
  14. 14. The best designers never stop at the first inference.
  15. 15. Log file contained blanks Design DecisionsInferencesObservations Broken focus Fix Focus No search text Advanced search Log file bug Add label to search box Build advanced
 search feature Fix bug Users didn’t know to type query into box Log file contained blanks
  16. 16. Research turns inferences into observations.
  17. 17. 2Useless measures & silly metrics
  18. 18. SnapChat AirBNB Uber Facebook MarketValuation InstancesofletterEonhomepage
  19. 19. Counting the letter E is a stupid metric.
  20. 20. Measure: Metric: Analytic: Something we can count. A measure we track. A measure software tracks.
  21. 21. Time on page
  22. 22. Time on page Bounce rate % Exit Pageviews Sessions w/search Time after search % Search Refinements
  23. 23. Bounce rate What should we do differently?
  24. 24. “ R.H. Coase, British economist If you torture data long enough, it will confess to anything you’d like.”
  25. 25. What Google Analytics can’t tell you: ‣What content was useful? ‣What people found confusing? ‣Who is your site’s biggest spender? ‣What do big spenders do that others don’t? ‣What should you do to improve your content? ‣Why did someone click?
  26. 26. What Google Analytics can’t tell you: Why?
  27. 27. Conversion Rate
  28. 28. M MATHEMATICAL CONTENT CONTAINS SOME FORMULAIC MATERIAL
 MAY BE INAPPROPRIATE FOR DESIGNERS THIS PRESENTATIONHAS

  29. 29. Conversion Rate Conversion Rate # of people who purchase # of people who visited = 1.0 % 10,000 1,000,000 = Conversion Rate # of people who purchase # of people who visited =
  30. 30. Conversion Rate 1.0 % 10,000 1,000,000 = 2.0 % 20,000 1,000,000 = 2.0 % 10,000 500,000 = 10,000
  31. 31. Conversion Rate 1,000,000 = 1,000,000 = 500,000 = 10,000 = x $100 $1,000,000 20,000 x $100 $2,000,000 = = 10,000 x $100 $1,000,000 1.0 % 2.0 % 2.0 % 10,00010,000 20,00020,000 10,00010,000
  32. 32. Conversion Rate 20,000 1,000,000 = 10,000 500,000 = $2,000,000= = $1,000,000 20,000 10,000 2.0 % 2.0 %
  33. 33. What should we design for? $2,000,000 $1,000,000 2.0 % 2.0 % Conversion Rate? Money?
  34. 34. Conversion Rate Conversion Rate # of people who purchase # of people who visited = Visit Visit Visit Purchase Conversion Rate = 25 % ? Conversion Rate = 100 % ?
  35. 35. What conversion rate can’t tell you: Why?
  36. 36. Delightful Amazing Awesome Excellent Remarkable Incredible Satisfactory
  37. 37. Satisfactory Edible
  38. 38. Achieving satisfaction is too low a bar to set for our designs. We can do better.
  39. 39. Satisfied Satisfied Neutral Dissatisfied Dissatisfied Extremely Somewhat Somewhat Extremely SatisfactionSurveyScale
  40. 40. DelightedExtremely DelightedSomewhat FrustratedSomewhat FrustratedExtremely Satisfied Delight/FrustrationSurveyScale
  41. 41. Pro tip: Nothing says “we don’t care” like a page of 10-point satisfaction scales.
  42. 42. 10-point scales make noise feel like science.
  43. 43. What satisfaction surveys and Net Promoter Score can’t tell you: Why?
  44. 44. We need metrics that help us improve our users’ experience.
  45. 45. 3A measured experience.
  46. 46. Customerjourneymap ☺ ☹ Searchingforahotel Reviewingtheproperty Reviewingroomrates Startbooking Guestinformation Billinginformation Paymentinformation Confirmbooking Done!
  47. 47. ☹ Thingsusersfindfrustrating Content that is confusing ☹ Incomplete information ☹ Can’t remember password ☹ Hidden features ☹ Hidden navigation ☹ Confusing navigation ☹ Error messages
  48. 48. Thingsthatusersfindfrustrating Phone numbers can’t have dashes or spaces. ☹ Error messages The credit card security code is required (again). Username and password do not match.
  49. 49. Pro tip: Customize analytics by counting error message deliveries.
  50. 50. Case study: Improve checkout process for a 
 major e-commerce site.
  51. 51. Shippinginformation Billinginformation Paymentinformation Confirmpurchase Done! 100% 50% Pageviews
  52. 52. Shippinginformation Billinginformation Paymentinformation Confirmpurchase 50% Pageviews Shopforproduct ReviewShoppingCart100% Done!
  53. 53. Shippinginformation Billinginformation Paymentinformation Confirmpurchase 50% Pageviews Shopforproduct ReviewShoppingCart Logintoaccount Requestpasswordreset Clickonlinkinemailtoreset Putinnewpassword 100% Done!
  54. 54. Shippinginformation Billinginformation Paymentinformation Confirmpurchase 50% Pageviews Shopforproduct ReviewShoppingCart Logintoaccount Requestpasswordreset Clickonlinkinemailtoreset Putinnewpassword 100% Done!
  55. 55. Lost revenue from account sign-in issues: $300,000,000per year The team built a guest checkout (no sign-in required). Recovered the $300,000,000 within 1 year.
  56. 56. Qualitative usability research $300,000,000 Quantitative custom metrics + Mostimportantcustommetric: Unrealized shopping cart value from password issues.
  57. 57. Qualitative findings must drive our quantitative research agenda.
  58. 58. The team’s initial inferences ‣The checkout steps were where we’d find the biggest improvement. ‣It was “normal” to lose customers before checkout. ‣Cart review went straight to checkout without additional steps. ‣All of the screens were instrumented.
  59. 59. Shippinginformation Billinginformation Paymentinformation Confirmpurchase 50% Pageviews Shopforproduct ReviewShoppingCart Logintoaccount Requestpasswordreset Clickonlinkinemailtoreset Putinnewpassword 100% Done!
  60. 60. Shippinginformation Billinginformation Paymentinformation Confirmpurchase 50% Pageviews Shopforproduct ReviewShoppingCart Logintoaccount Requestpasswordreset Clickonlinkinemailtoreset Putinnewpassword 100% Done! ☺ ☹ Focus quantitative research on frustration.
  61. 61. Observations trump inferences.
  62. 62. “Analytics are controlled by a different group.” Nolongeracceptable:
  63. 63. Owned by the UXTeam Owned by theAnalyticsTeam Under new management: the UX Team
  64. 64. DJ Patil, Former US Chief Data Scientist
  65. 65. “Analytics are controlled by a different group.” Data science is now an essential skill for every UX team. Nolongeracceptable:
  66. 66. Continually question what the metric is trying to tell you. Nolongeracceptable: “I don’t understand what the metrics mean.”
  67. 67. Keep the numbers simple,so you can focus on the behaviors. Nolongeracceptable: “I’m not good with numbers.”
  68. 68. Applying targeted metrics to qualitative research: a powerful addition to the UX designer’s toolkit.
  69. 69. 4Who collects the metrics?
  70. 70. Design: The Rendering of Intent
  71. 71. Myspace MonthlyActiveUsers Market Valuation Pinterest Whatsapp Snapchat LinkedIn Twitter YouTube Tumblr Instagram 0 1,000,000,000 $0 $40,000,000,000
  72. 72. “ Robert Fabricant The medium of design is behavior”
  73. 73. Design must drive metric collection, not the other way around.
  74. 74. Is Design Metrically Opposed? ‣We need metrics that help us improve the experience. ‣Avoid jumping from observations to inference too soon. ‣ Make sure you’re testing alternative inferences. ‣Customize metrics to match experience objectives. ‣Data science is now an essential UX skill.
  75. 75. @jmspool Go ahead! Follow me on the Twitters. jspool@uie.com Find me at: Copyright © 2015 User Interface Engineering Don’t forget to connect to me on the LinkedIn. uie.com

×