How to Quantitatively Measure Your User Experience


Published on

How do you know if the user experience your team is slaving over is succeeding or failing? How can your team use data to make better decisions? Quantitative measures can help answer these questions and can complement more traditional qualitative research methods like usability testing.

Published in: Design, Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Quantitatively Measure Your User Experience

  1. 1. Measures of Success: How to QuantitativelyMeasure Your User ExperienceRichard Dalton, @mauvyrusset 1
  2. 2. 2
  3. 3. #1 Evaluate your experience against something you care about – is it meeting it’s objectives? @mauvyrusset #WVpdx 3
  4. 4. 4
  5. 5. #2 The objectives of your experience will likely differ from those of the person sitting next to you @mauvyrusset #WVpdx 5
  6. 6. Users have goals Goals … and the business has goalsAre realized through Users do things to try to meet their goalsTasks + Emotions … and the business wants users to do things so it can meet its goalsAre enabled & encouraged by Capabilities help users to do the tasks they want to do Capabilities … and encourage users to do the tasks the business wants them to doAre created & changed by Projects create new capabilities Projects Projects change existing capabilities 6
  7. 7. #3 Measure how well tasks are satisfied by capabilities, not projects – otherwise you have no baseline @mauvyrusset #WVpdx 7
  8. 8. 8
  9. 9. User-driven tasksFind an investment company Capabilities Business-driven tasksMake good investment decisions Web Follow Vanguard’s investing principlesMonitor my investments Phone Learn why Vanguard is greatAct on my investments Paper Bring assets to VanguardStay current on news and commentary E-mail Use Vanguards products & servicesDeal with taxes Mobile devicesHelp other people be successful investors Radio/TV Self-provision on the webHelp my heirs be successful at Vanguard Spread the word about Vanguard Trust Vanguard 90 tasks grouped 635 capabilities and 45 tasks grouped into 8 categories counting … into 7 categories 9
  10. 10. 10
  11. 11. User-driven tasksResearch an item Capabilities Business-driven tasks Get information on the item Find out how much the item costs Close the sale Item profile Compare the item to others like it (web) Buy the item See if other people like the item Save the item to look at later Cross sell Print details about the item See related items See related items Trust usReady to buy Believe that the site is safe and secure Buy the item Spread the word Find out shipping costs and times Tell other people about the item Find out how to payTell others Tell a friend about the item 11
  12. 12. Buy the item 1 Buy the item Emotional considerations Shoppers are commonly fearful or unsure that they have chosen the 2 Get information on the item best product for their needs and want to “comparison shop” the price 3 Find out how much the item costs and/or features of several products. 4 See if other people like the item 5 Find out shipping costs and times High-level design & content approach Show recently viewed items and provide access to a “compare to 6 Compare the item to others like it similar items” tool. 7 Believe that the site is safe and secure 8 Find out how to pay Success criteria & measures – “what is good?” 9 Save the item to look at later The ratio of users looking at recently viewed items via the “compare See related items to similar items” tool vs. pogo-sticking back to the gallery page should10 be 20:1 or better. See related items11 Print details about the item Tell a friend about the item12 Tell other people about the item 12
  13. 13. Measures 13
  14. 14. Measures vs. Success Criteria Measure Success Criteria 14
  15. 15. S3.01 Tell a friend about the item S2.03 Find out how to pay S1.01 Get information on the item S2.02 Find out S1.02 Find out S1.03 Compare the shipping costs how much the item to others like it and times item costs S1.04 See if other people like the item S2.01 Buy the item P1.01 Buy the item P4.02 Tell other P3.01 Believe P2.01 See related items people about that the site is the item safe and secure Drivers OutcomeIshikawa (fishbone) diagram 15
  16. 16. #4 Measuring outcomes can tell you if a capability is failing. Measuring drivers can tell you why @mauvyrusset #WVpdx 16
  17. 17. What’s being evaluated? MeasureA. General effectiveness in satisfying the task 1. Completion or conversion rates “I’ve done it” or “Now I understand” 2. Did future user behavior change as a result of this interaction?B. Findability of a known item Speed to find the first task “I know where I’m going, don’t get in my way”C. Were the user expectations met? Link bounce rate “Oh, that wasn’t what I wanted, let me go back …”D. Client satisfaction 1. “Was this useful/helpful” surveys “Huh, that information was useless” 2. “Loss of sale” surveysE. Was enough information provided? Usage of a second source of information “Don’t leave me wanting more info”F. Are users travelling the paths we expected them to? Relative usage of one path vs. another to “I can’t find the link, I’ll go back to the homepage first” identical destination 17
  18. 18. Success Criteria Enduring Temporary Use for on-going monitoring Use for point-in-time improvement Measure a single solution Compare two or more solutions against a predefined criteria against one another using A/B testing Criteria set by past user Point-in-time “winners” behavior or future expectations. identified by test results. Example Example Task: Get information on the item Task: Print details about the item Loss of sale surveys should show that A/B test two versions of the problems with item information printer friendly entry point (link vs. account for less than 5% of lost sales button) 18
  19. 19. #5 Ask; how would the user behave if we nailed the design? How would they behave if we screwed it up? @mauvyrusset #WVpdx 19
  20. 20. Cultural Challenges 20
  21. 21. Sometimes people aren’t ready to listen 21
  22. 22. #6 Be open about the uses & limitations of data & involve people early – it helps gain buy-in @mauvyrusset #WVpdx 22
  23. 23. Beware good intentions 23
  24. 24. #7 Avoid misleading measures - temptation to use the data is too strong. Ask “what if the result is X?” @mauvyrusset #WVpdx 24
  25. 25. The only thing we have to fear is fear itself Ostridge 25
  26. 26. #8 Be unbiased. Don’t be afraid to measure things that might contradict your own opinion @mauvyrusset #WVpdx 26
  27. 27. I’m sorry Dave. I’m afraid I can’t do that 27
  28. 28. #9 Don’t lose your perspective about how data fits into your decision-making process @mauvyrusset #WVpdx 28
  29. 29. But we have over 635 capabilities! 29
  30. 30. #10 Start small. Pick a capability, identify objectives, define measures and watch what happens @mauvyrusset #WVpdx 30
  31. 31. Thank you Richard Dalton @mauvyrusset 31
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.