0
1<br />A Practical Guide to Measuring User Experience<br />Richard Dalton, @mauvyrusset, #measuringux<br />http://www.flic...
#1 	To learn from failure, you need to know if you’re failing and why – measurement can tell you that <br />@mauvyrusset  ...
THERE IS NOTHING FUNNY ABOUT R.O.I.<br />3<br />
4<br />
#2 	Evaluate your experience against something you care about – is it meeting it’s objectives?<br />@mauvyrusset   #measur...
6<br />
#3 	The objectives of your experience will likely differ from those of the person sitting next to you<br />@mauvyrusset   ...
Users have goals<br />… and the business has goals<br />Users do things to try to meet their goals<br />… and the business...
#4 	Measure how well tasks are satisfied by capabilities, not projects – otherwise you have no baseline<br />@mauvyrusset ...
User-driven tasks<br />Business-driven tasks<br />Capabilities<br />Find an investment company<br />Find an investment com...
11<br />http://filmfanatic.org/reviews/wp-content/uploads/2008/01/anfscd-parrot.png<br />
User-driven tasks<br />Business-driven tasks<br />Capabilities<br />Research an item<br />Close the sale<br />Item profile...
13<br />Buy the item<br />1<br />Emotional considerations<br />Buy the item<br />Shoppers are commonly fearful or unsure t...
14<br />
Measures vs. Success Criteria<br />Measure<br />Success Criteria<br />15<br />http://3.bp.blogspot.com/_iUXX3rF1Axo/TC0fGh...
S1.01 Get information on the item<br />S2.03 Find out how to pay<br />S3.01 Tell a friend about the item<br />S2.02 Find o...
#5 	Measuring outcomes can tell you if a capability is failing. Measuring drivers can tell you why<br />@mauvyrusset   #me...
What’s being evaluated?<br />Measure<br />Completion or conversion rates<br />Did future user behavior change as a result ...
Success Criteria<br />Enduring<br />Temporary<br />Use for on-going monitoring<br />Use for point-in-time improvement<br /...
#6	Ask; how would the user behave if we nailed the design? How would they behave if we screwed it up?<br />@mauvyrusset   ...
21<br />
Sometimes people aren’t ready to listen<br />22<br />
#7 	Be open about the uses & limitations of data & involve people early – it helps gain buy-in<br />@mauvyrusset   #measur...
Beware good intentions<br />24<br />
#8 	Avoid misleading measures - temptation to use the data is too strong. Ask “what if the result is X?”<br />@mauvyrusset...
The only thing we have to fear is fear itself<br />Ostridge<br />26<br />http://fc05.deviantart.net/fs36/f/2008/285/4/f/Yo...
#9 	Be unbiased. Don’t be afraid to measure things that might contradict your own opinion<br />@mauvyrusset   #measuringux...
I’m sorry Dave. I’m afraid I can’t do that<br />28<br />http://img395.imageshack.us/img395/3899/hal90001600en6.jpg<br />
#10 	Don’t lose your perspective about how data fits into your decision-making process<br />@mauvyrusset   #measuringux<br...
But we have over 635 capabilities!<br />30<br />http://www.flickr.com/photos/tammra/279392432/<br />
#11 	Start small. Pick a capability, identify objectives, define measures and watch what happens<br />@mauvyrusset   #meas...
Thank you<br />Richard Dalton<br />richard@mauvyrusset.com<br />mauvyrusset.com<br />@mauvyrusset<br />32<br />
Upcoming SlideShare
Loading in...5
×

A Practical Guide to Measuring User Experience

22,022

Published on

Measuring the success (or failure) of an experience can be a daunting and confusing endeavor. In this presentation Richard shares 11 guidelines to help you quantitatively measure your user experience.

Richard provides techniques and tips for each phase and illustrates their use with real examples from his team’s work at Vanguard. To conclude, he describes some of the cultural and change management challenges involved when an organization uses data to inform design decisions.

Published in: Design
2 Comments
51 Likes
Statistics
Notes
No Downloads
Views
Total Views
22,022
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
384
Comments
2
Likes
51
Embeds 0
No embeds

No notes for slide

Transcript of "A Practical Guide to Measuring User Experience"

  1. 1. 1<br />A Practical Guide to Measuring User Experience<br />Richard Dalton, @mauvyrusset, #measuringux<br />http://www.flickr.com/photos/kwl/5282931021<br />
  2. 2. #1 To learn from failure, you need to know if you’re failing and why – measurement can tell you that <br />@mauvyrusset #measuringux<br />2<br />
  3. 3. THERE IS NOTHING FUNNY ABOUT R.O.I.<br />3<br />
  4. 4. 4<br />
  5. 5. #2 Evaluate your experience against something you care about – is it meeting it’s objectives?<br />@mauvyrusset #measuringux<br />5<br />
  6. 6. 6<br />
  7. 7. #3 The objectives of your experience will likely differ from those of the person sitting next to you<br />@mauvyrusset #measuringux<br />7<br />
  8. 8. Users have goals<br />… and the business has goals<br />Users do things to try to meet their goals<br />… and the business wants users to do things so it can meet its goals <br />Users do things to try to meet their goals<br />… and the business wants users to do things so it can meet its goals <br />Users also have feelings about their goals & tasks<br />… and the business wants to encourage certain feelings<br />Capabilities help users to do the tasks they want to do<br />… and encourage users to do the tasks the business wants them to do<br />Projects create new capabilities<br />Projects change existing capabilities<br />Goals<br />Are realized through<br />Tasks + Emotions<br />Tasks<br />+ Emotions<br />Are enabled & encouraged by<br />Capabilities<br />Are created & changed by<br />Projects<br />8<br />
  9. 9. #4 Measure how well tasks are satisfied by capabilities, not projects – otherwise you have no baseline<br />@mauvyrusset #measuringux<br />9<br />
  10. 10. User-driven tasks<br />Business-driven tasks<br />Capabilities<br />Find an investment company<br />Find an investment company<br />Make good investment decisions<br />Follow Vanguard’s investing principles<br />Web<br />Phone<br />Paper<br />E-mail<br />Mobile devices<br />Radio/TV<br />Monitor my investments<br />Learn why Vanguard is great<br />Act on my investments<br />Bring assets to Vanguard<br />Stay current on news and commentary<br />Use Vanguard's products & services<br />Deal with taxes<br />Self-provision on the web<br />Help other people be successful investors<br />Spread the word about Vanguard<br />Help my heirs be successful at Vanguard<br />Trust Vanguard<br />90 tasks grouped<br />into 8 categories<br />45 tasks grouped<br />into 7 categories<br />635 capabilities and counting …<br />10<br />
  11. 11. 11<br />http://filmfanatic.org/reviews/wp-content/uploads/2008/01/anfscd-parrot.png<br />
  12. 12. User-driven tasks<br />Business-driven tasks<br />Capabilities<br />Research an item<br />Close the sale<br />Item profile<br />(web)<br />12<br />Get information on the item<br />Find out how much the item costs<br />Compare the item to others like it<br />Buy the item<br />See if other people like the item<br />Cross sell<br />Save the item to look at later<br />See related items<br />Print details about the item<br />See related items<br />Trust us<br />Ready to buy<br />Believe that the site is safe and secure<br />Buy the item<br />Spread the word<br />Find out shipping costs and times<br />Tell other people about the item<br />Find out how to pay<br />Tell others<br />Tell a friend about the item<br />
  13. 13. 13<br />Buy the item<br />1<br />Emotional considerations<br />Buy the item<br />Shoppers are commonly fearful or unsure that they have chosen the best product for their needs and want to “comparison shop” the price and/or features of several products.<br />Get information on the item<br />2<br />Find out how much the item costs<br />3<br />See if other people like the item<br />4<br />High-level design & content approach<br />Find out shipping costs and times<br />5<br />Show recently viewed items and provide access to a “compare to similar items” tool.<br />Compare the item to others like it<br />6<br />Believe that the site is safe and secure<br />7<br />Find out how to pay<br />8<br />Success criteria – “what is good?”<br />Save the item to look at later<br />9<br />The ratio of users looking at recently viewed items via the “compare to similar items” tool vs. pogo-sticking back to the gallery page should be 20:1 or better.<br />See related items<br />10<br />12<br />See related items<br />11<br />Print details about the item<br />Tell a friend about the item<br />Tell other people about the item<br />
  14. 14. 14<br />
  15. 15. Measures vs. Success Criteria<br />Measure<br />Success Criteria<br />15<br />http://3.bp.blogspot.com/_iUXX3rF1Axo/TC0fGhmlV0I/AAAAAAAAAMw/atKJuTja0AM/s1600/butterfly+growth+chart2.jpg<br />http://www.clarklings.com/uploaded_images/IMG_1100-732092.JPG<br />
  16. 16. S1.01 Get information on the item<br />S2.03 Find out how to pay<br />S3.01 Tell a friend about the item<br />S2.02 Find out shipping costs and times<br />S1.03 Compare the item to others like it<br />S1.02 Find out how much the item costs<br />S1.04 See if other people like the item<br />Research an item<br />Tell others<br />Ready to buy<br />S2.01 Buy the item<br />P1.01 Buy the item<br />Spread the word<br />Trust us<br />Cross sell<br />P2.01 See related items<br />P3.01 Believe that the site is safe and secure<br />P4.02 Tell other people about the item<br />Outcome<br />Drivers<br />Ishikawa (fishbone) diagram<br />http://en.wikipedia.org/wiki/Ishikawa_diagram<br />16<br />
  17. 17. #5 Measuring outcomes can tell you if a capability is failing. Measuring drivers can tell you why<br />@mauvyrusset #measuringux<br />17<br />
  18. 18. What’s being evaluated?<br />Measure<br />Completion or conversion rates<br />Did future user behavior change as a result of this interaction?<br />General effectiveness in satisfying the task<br />“I’ve done it” or “Now I understand”<br />Findability of a known item<br />“I know where I’m going, don’t get in my way”<br />Speed to find the first task<br />Were the user expectations met?<br />“Oh, that wasn’t what I wanted, let me go back …” <br />Link bounce rate<br />Client satisfaction<br />“Huh, that information was useless”<br />“Was this useful/helpful” surveys<br />“Loss of sale” surveys<br />Was enough information provided?<br />“Don’t leave me wanting more info”<br />Usage of a second source of information<br />Are users travelling the paths we expected them to?<br />“I can’t find the link, I’ll go back to the homepage first”<br />Relative usage of one path vs. another to identical destination<br />18<br />
  19. 19. Success Criteria<br />Enduring<br />Temporary<br />Use for on-going monitoring<br />Use for point-in-time improvement<br />Measure a single solution against a predefined criteria<br />Compare two or more solutions against one another using A/B testing<br />Criteria set by past user behavior or future expectations.<br />Point-in-time “winners” identified by test results.<br />Example<br />Task: Get information on the item<br />Loss of sale surveys should show that problems with item information account for less than 5% of lost sales<br />Example<br />Task: Print details about the item <br />A/B test two versions of the printer friendly entry point (link vs. button)<br />19<br />
  20. 20. #6 Ask; how would the user behave if we nailed the design? How would they behave if we screwed it up?<br />@mauvyrusset #measuringux<br />20<br />
  21. 21. 21<br />
  22. 22. Sometimes people aren’t ready to listen<br />22<br />
  23. 23. #7 Be open about the uses & limitations of data & involve people early – it helps gain buy-in<br />@mauvyrusset #measuringux<br />23<br />
  24. 24. Beware good intentions<br />24<br />
  25. 25. #8 Avoid misleading measures - temptation to use the data is too strong. Ask “what if the result is X?”<br />@mauvyrusset #measuringux<br />25<br />
  26. 26. The only thing we have to fear is fear itself<br />Ostridge<br />26<br />http://fc05.deviantart.net/fs36/f/2008/285/4/f/You_make_kitty_scared_by_GreenLabRat.jpg<br />
  27. 27. #9 Be unbiased. Don’t be afraid to measure things that might contradict your own opinion<br />@mauvyrusset #measuringux<br />27<br />
  28. 28. I’m sorry Dave. I’m afraid I can’t do that<br />28<br />http://img395.imageshack.us/img395/3899/hal90001600en6.jpg<br />
  29. 29. #10 Don’t lose your perspective about how data fits into your decision-making process<br />@mauvyrusset #measuringux<br />29<br />
  30. 30. But we have over 635 capabilities!<br />30<br />http://www.flickr.com/photos/tammra/279392432/<br />
  31. 31. #11 Start small. Pick a capability, identify objectives, define measures and watch what happens<br />@mauvyrusset #measuringux<br />31<br />
  32. 32. Thank you<br />Richard Dalton<br />richard@mauvyrusset.com<br />mauvyrusset.com<br />@mauvyrusset<br />32<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×