0
Beyond Just Usability:
Desirability & Usefulness Testing
Susan Mercer, Senior Experience Researcher
@susanamercer
Usability Testing
Usability = (Efficiency + Effectiveness)
In a given
context of use
Usable
Cute
Acerbic
Usability	
  
Usefulness	
  Desirability	
  
Usability	
  
Consistent	
  
Organized	
  
Understandable	
  
Simple	
  
Efficient	
  
Usefulness	
  
Effec=ve	
  
Essen=al	
  
Relevant	
  
Helpful	
  
Suppor=ng	
  
Desirability	
  
Exci=ng	
  
Fresh	
  
Friendly	
  
Encouraging	
  
Meaningful	
  
Usability	
  
Usefulness	
  Desirability	
  
DESIRABILITY
TESTING METHODS
Open-­‐Ended  Ques,oning
•  What  do  you  think  of…?
•  What  is  your  reac,on  to…?
•  How  would  you  describe…?
Sentence  Comple,on
•  This  app  makes  me  feel…
•  If  I  used  this  all  day,  I  would…
•  Reading  on  this  websit...
Product  Reac,on  Cards
•  118  words
•  58%  posi,ve
•  36%  nega,ve
•  6%  unclear
•  21%  synonyms  with  1+  other  wo...
Product  Reac,on  Cards  –  Alternate  List
•  Removed  synonyms
•  Added  opposites  to  create  Posi,ve  /  Nega,ve  Pai...
Annoying	
   Costly	
   Expected	
   Interes=ng	
   Poor	
  quality	
  	
   Trivial	
  
Apathe=c	
   Crea=ve	
   Familiar	...
AJrac=ve	
   UnaJrac=ve	
   Excep=onal	
   Common	
   Organized	
   Disorganized	
  
Calm	
   Busy	
   Exci=ng	
   Dull	
 ...
Reliable	
   Accessible	
   Appealing	
   Confusing	
  
Comfortable	
   Organized	
   Professional	
   Busy	
  
Confident	
...
68	
   67	
  
62	
  
42	
  
20	
  
22	
  
17	
  
31	
  
0	
  
10	
  
20	
  
30	
  
40	
  
50	
  
60	
  
70	
  
Exis1ng	
  ...
Exis1ng	
   Design	
  A	
   Design	
  B	
   Design	
  C	
  
Appealing	
   Appealing	
   Appealing	
   Appealing	
  
Clean	...
Product  Reac,on  Dyads
Rate  on  a  5-­‐point  scale  from  Posi,ve  –  Nega,ve  word  pairs
AJrac=ve	
   ¤	
   ¢	
   ¢...
-­‐3.5	
   -­‐2.5	
   -­‐1.5	
   -­‐0.5	
   0.5	
   1.5	
   2.5	
   3.5	
  
Discouraging	
  
Unrefined	
  
Useless	
  
Comp...
Photo  Selec,on
LEM  Tool  
(Susa  Group)
Open-­‐Ended  Ques,oning
Sentence  Comple,on
Product  Reac,on  Cards
Product  Reac,on  Dyads
Photo  Selec,on
LEM  Tool
Usability	
  
Desirability	
   Usefulness	
  
USEFULNESS
TESTING
Pre-­‐Usability  Interview  
•  How  do  you  reconcile  invoices  today…?
•  Who  needs  to  approve  these  and  why…?
•...
During  Usability  Tes,ng
Watch  for  comments  about  usefulness
•  “Wow.  This  would  really  make  it  easier  to  reo...
Post-­‐Usability  Interview  
Show  screens  again  and  ask:
•  “How  would  this  work  for  you  in  your  job?”
•  “Wh...
TESTING	
  METHODS	
  
Desirability  Tes,ng:
•  Open-­‐Ended  Ques,oning
•  Sentence  Comple,on
•  Product  Reac,on  Cards...
Thank You!
Susan Mercer
smercer@madpow.com
@susanamercer
Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
Upcoming SlideShare
Loading in...5
×

Beyond Just Usability: Desirability and Usefulness Testing

2,634

Published on

Much of our work in UX research focuses on usability – evaluating products and interfaces to ensure they are easy-to-use. However, in today’s digital world, they are no longer enough. Consumers also have come to expect entertaining and engaging experiences. Web and mobile applications need to be usable, useful and engaging.

So, how do we evaluate web interfaces to determine how useful and engaging they are? Desirability has been evaluated in recent years by the use of the Product Reaction Card technique, originated by folks at Microsoft. However, there are many other techniques used in market and industrial design research that we can borrow to complement this technique. Likewise, we can use standard usability testing techniques with lines of questioning with a slightly different focus to evaluate the relative usefulness of different solutions for a particular user group.

In this talk, I discuss several techniques that I have used in recent months to evaluate the usefulness and desirability of interfaces The best techniques I have discovered to evaluate usefulness involve open-ended interview questions regarding current processes and pain points, followed by a usability evaluation of the interface and then a reflective interview discussing the benefits and drawbacks of that solution to their personal situation. To evaluate desirability, I will discuss the product reaction card technique and variations using more defined vocabularies for emotional responses and product personalities. In addition I will show results from techniques borrowed from psychology and marketing research - sentence completion, collaging, and the use of dyad rating scales. These techniques offer a variety of both qualitative and quantitative data that can be used to compare different interface options.

Published in: Design, Technology
1 Comment
12 Likes
Statistics
Notes
  • Update. One of the questions I received from the audience is regarding the Product Reaction Dyads I mention on slide 24. The question was 'Should the positive/negative order of the word pairs be alternated to avoid bias?'

    I was aware that Bob Virzi had encountered problems with the System Usability Scale (SUS) survey with it's alternating positive and negative statements. Jeff Sauro and James Lewis also conducted a study and reported that at the CHI conference in 2011.

    http://www.measuringusability.com/papers/sauro_lewisCHI2011.pdf

    With the SUS, participants sometimes lost track of whether a statement was positive or negative and made mistakes in answering. Also, researchers sometimes made errors in miscoding and scoring the answers. The net conclusion is that alternating positive and negative statements caused more problems than just having all positive statements.

    Based on that, I think that having all word pairs aligned the same way should not substantially change the results.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
2,634
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
73
Comments
1
Likes
12
Embeds 0
No embeds

No notes for slide

Transcript of "Beyond Just Usability: Desirability and Usefulness Testing"

  1. 1. Beyond Just Usability: Desirability & Usefulness Testing Susan Mercer, Senior Experience Researcher @susanamercer
  2. 2. Usability Testing
  3. 3. Usability = (Efficiency + Effectiveness) In a given context of use
  4. 4. Usable
  5. 5. Cute
  6. 6. Acerbic
  7. 7. Usability   Usefulness  Desirability  
  8. 8. Usability   Consistent   Organized   Understandable   Simple   Efficient  
  9. 9. Usefulness   Effec=ve   Essen=al   Relevant   Helpful   Suppor=ng  
  10. 10. Desirability   Exci=ng   Fresh   Friendly   Encouraging   Meaningful  
  11. 11. Usability   Usefulness  Desirability  
  12. 12. DESIRABILITY TESTING METHODS
  13. 13. Open-­‐Ended  Ques,oning •  What  do  you  think  of…? •  What  is  your  reac,on  to…? •  How  would  you  describe…?
  14. 14. Sentence  Comple,on •  This  app  makes  me  feel… •  If  I  used  this  all  day,  I  would… •  Reading  on  this  website  is… •  Compared  to  other  apps,  this  app  is… •  I  want  to  use  this  because… •  I  do  not  want  to  use  this  because…
  15. 15. Product  Reac,on  Cards •  118  words •  58%  posi,ve •  36%  nega,ve •  6%  unclear •  21%  synonyms  with  1+  other  words •  Mostly  posi,ve  words
  16. 16. Product  Reac,on  Cards  –  Alternate  List •  Removed  synonyms •  Added  opposites  to  create  Posi,ve  /  Nega,ve  Pairs •  Compared  against  two  other  emo,onal  word  vocabularies  for   Product  Appeal  and  Emo,ons •  Marc  Hassenzhahl,  Darmstadt  University  of  Technology,  Germany •  Geneva  Emo,on  Wheel
  17. 17. Annoying   Costly   Expected   Interes=ng   Poor  quality     Trivial   Apathe=c   Crea=ve   Familiar   Irrelevant   Predictable   UnaJrac=ve   AJrac=ve   Desirable   Fast   Managable   Professional   Uncontrollable   Boring   Difficult   Flexible   Meaningful     Relaxing   Understandable     Busy   Difficult   Fragile   Mo=va=ng     Relevant   Undesirable   Calm   Disappointed   Fresh   Nondescript   Reliable   Unexci=ng   Cheap   Discouaraging   Friendly   Not  Secure   Rigid   Unimagina=ve   Clean   Disorganized   Frustra=ng   Not  Valuable   Sa=sfying     Unimportant   Clear   Dull   Fun   Novel   Secure   Unnecessary   CluJered   Easy  to  use               Hard  to  Use   Obstruc=ng   Simple   Unpredictable   Common   Effec=ve   High  quality   Old   Slow   Unprofessional   Compelling   Efficient   Impersonal   Op=mis=c   Sophis=cated   Unrefined   Complex   Effortless   Impressive   Ordinary   Stable   Unreliable   Comprehensive   Empowering   Incomplete   Organized   Standard   Unusable   Confident   Encouraging   Incomprehensible   Original   Strange   Unwelcoming   Confusing   Enthusias=c   Inconsistent     Overwhelming     Stressful   Usable   Conserva=ve   Essen=al     Ineffec=ve   Patronizing   Suppor=ng   Useful   Consistent   Excep=onal   Inefficient   Personal   Tedious   Useless   Controllable         Exci=ng   Innova=ve   Pessimis=c   Time-­‐consuming   Valuable   Convenient   Exclusive   Inspiring   Pleasant   Time-­‐Saving     Weak  
  18. 18. AJrac=ve   UnaJrac=ve   Excep=onal   Common   Organized   Disorganized   Calm   Busy   Exci=ng   Dull   Original   Ordinary   Clean   CluJered   Exclusive   Standard   Personal   Impersonal   Clear   Confusing   Familiar   Strange   Pleasant   Annoying   Compelling   Unimportant   Fast   Slow   Predictable   Unpredictable   Comprehensive   Incomplete   Flexible   Rigid   Professional   Unprofessional   Confident   Weak   Fresh   Old   Relaxing   Stressful   Consistent   Inconsistent   Friendly   Unwelcoming   Relevant   Irrelevant   Controllable   Uncontrollable   Frustra=ng   Encouraging   Reliable   Unreliable   Convenient   Difficult   Fun   Tedious   Sa=sfying   Disappointed   Costly   Cheap   High  quality   Poor  quality   Secure   Not  Secure   Crea=ve   Unimagina=ve   Impressive   Nondescript   Simple   Complex   Desirable   Undesirable   Innova=ve   Conserva=ve   Sophis=cated   Unrefined   Easy  to  use   Hard  to  use   Inspiring   Unexci=ng   Stable   Fragile   Effec=ve   Ineffec=ve   Interes=ng   Boring   Suppor=ng   Obstruc=ng   Efficient   Inefficient   Manageable   Overwhelming   Time-­‐saving   Time-­‐consuming   Effortless   Difficult   Meaningful   Trivial   Understandable   Incomprehensible   Empowering   Patronizing   Mo=va=ng   Discouraging   Usable   Unusable   Enthusias=c   Apathe=c   Novel   Expected   Useful   Useless   Essen=al   Unnecessary   Op=mis=c   Pessimis=c   Valuable   Not  Valuable  
  19. 19. Reliable   Accessible   Appealing   Confusing   Comfortable   Organized   Professional   Busy   Confident   Straight  Forward   Approachable   Complex   Effec=ve   Understandable   Efficient   Dull   Friendly   Easy  to  Use   Flexible   Hard  to  Use   Invi=ng   Clear   Helpful   Impersonal   Time-­‐saving   Convenient   Overwhelming   Trustworthy   Clean   Boring   Useful   Familiar   Frustra=ng   Simplis=c   In=mida=ng   Valuable   Stressful   Usable   Time-­‐consuming   Customers   Customers  &   Non-­‐Customers   Non-­‐Customers  
  20. 20. 68   67   62   42   20   22   17   31   0   10   20   30   40   50   60   70   Exis1ng   Design  A   Design  B   Design  C   Count   Number  of  Words   Posi=ve     Nega=ve  
  21. 21. Exis1ng   Design  A   Design  B   Design  C   Appealing   Appealing   Appealing   Appealing   Clean   Clean   Clean   Clean   Efficient   Efficient   Efficient   Efficient   Engaging   Engaging   Engaging   Intui=ve   Professional   Professional   Professional   Useful   Useful   Useful   Useful   Total   9   20   16   9  
  22. 22. Product  Reac,on  Dyads Rate  on  a  5-­‐point  scale  from  Posi,ve  –  Nega,ve  word  pairs AJrac=ve   ¤   ¢   ¢   ¢   ¢   UnaJrac=ve   Calm   ¢   ¤   ¢   ¢   ¢   Busy   Clean   ¢   ¤   ¢   ¢   ¢   CluJered   Clear   ¤   ¢   ¢   ¢   ¢   Confusing   Comprehensive   ¢   ¢   ¢   ¤   ¢   Incomplete  
  23. 23. -­‐3.5   -­‐2.5   -­‐1.5   -­‐0.5   0.5   1.5   2.5   3.5   Discouraging   Unrefined   Useless   Complex   Irrelevant   Boring   Expected   Unprofessional   Mo=va=ng   Sophis=cated   Useful   Simple   Relevant   Interes=ng   Novel   Professional  
  24. 24. Photo  Selec,on
  25. 25. LEM  Tool   (Susa  Group)
  26. 26. Open-­‐Ended  Ques,oning Sentence  Comple,on Product  Reac,on  Cards Product  Reac,on  Dyads Photo  Selec,on LEM  Tool
  27. 27. Usability   Desirability   Usefulness  
  28. 28. USEFULNESS TESTING
  29. 29. Pre-­‐Usability  Interview   •  How  do  you  reconcile  invoices  today…? •  Who  needs  to  approve  these  and  why…? •  Where  do  you  track  which  claims  have  been  processed…? •  When  a  new  pa,ent  arrives,  what  do  you  do?
  30. 30. During  Usability  Tes,ng Watch  for  comments  about  usefulness •  “Wow.  This  would  really  make  it  easier  to  reorder  the  same  things.” •  “I  don’t  like  this.  It  would  take  me  too  long  to  do  this  part.” Watch  for  neutral  reac,ons •  “Eh.  This  is  OK.” •  Probe:  “How  would  this  help,  or  not  help,  you  do  your  job?”
  31. 31. Post-­‐Usability  Interview   Show  screens  again  and  ask: •  “How  would  this  work  for  you  in  your  job?” •  “What  about  this  works  for  you?  What  doesn’t  work?”
  32. 32. TESTING  METHODS   Desirability  Tes,ng: •  Open-­‐Ended  Ques,oning •  Sentence  Comple,on •  Product  Reac,on  Cards •  Product  Reac,on  Dyads •  LEM  Tool Usefulness  Tes,ng: •  Pre  &  Post-­‐Interview  Ques,ons •  Listening  &  Asking  Probing   Ques,ons
  33. 33. Thank You! Susan Mercer smercer@madpow.com @susanamercer
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×