2 Studies UX types should know about (Straub UXPA unconference13)

1,682 views
1,524 views

Published on

I described these two studies during the Research in Practice: Studies UXers should know about workshop. I expected them to be drive-bys ... as in, "Yah, yah, .. have heard that ... let's move on." I was surprised to find that the group -- a sharp, engaged and thoughtful group-- didn't know these studies. Instead of a few minutes description, we discussed and debated how these studies might influence UX practice for almost an hour. Based on that, I got nudged (Culprit = @susandra Susan Dray) to presenting these two @ the UXPA unconference.

There are many other studies studies that all UXPros should be familiar with ...

Published in: Design, Technology, Business
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,682
On SlideShare
0
From Embeds
0
Number of Embeds
50
Actions
Shares
0
Downloads
4
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

2 Studies UX types should know about (Straub UXPA unconference13)

  1. 1. UXPA13 unconference 10 minutes. 2 studies. Discuss. Kath Straub PhD  
  2. 2. Truth in advertising I’m not talking about my own work. I’m talking about studies in the (peer-reviewed) literature that you should know about and cite….
  3. 3. The question How many users? Really.
  4. 4. The study people point to: Nielsen & Landauer (1993) JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems. ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213, May1993,Amsterdam,TheNetherlands 4 # problems found: N(1-(1-λ)i) N= Total # of usability problems λ = probablity of finding the average usability problem when running a single, average subject or using a single, average evaluator i= # of participants or evaluators # of test participants or expert evaluators
  5. 5. The study people point to: Nielsen & Landauer (1993) JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems. ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213, May1993,Amsterdam,TheNetherlands 5 Important  insight   If  you  don’t  test  any  users   or  have  anyone  review  the   usability,  you  won’t  learn   about  any  problems.   # of test participants or expert evaluators
  6. 6. The study people point to: Nielsen & Landauer (1993) JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems. ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213, May1993,Amsterdam,TheNetherlands 6 You  hit  diminishing  returns  on   finding  new  problems  at  ~   5par>cipants  or  5  evaluators   (assuming  each  user/evaluator  unearths  1/3  of   the  usability  problems)   # of test participants or expert evaluators
  7. 7. The real data (its in the paper ….) is messier. JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems. ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213, May1993,Amsterdam,TheNetherlands 7 # of test participants or expert evaluators This  shows  what  happens   when  you  play  with  the   assump>on  that  each   par>cipant/reviewer  finds     ~  1/3  of  the  problems  
  8. 8. Why this study should give us pause…. JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems. ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213, May1993,Amsterdam,TheNetherlands 8 •  Its not really answering this particular question. It is answering the question “Is there a mathematical model that describes/predicts the point of diminishing returns with test participants or evaluators based on the post hoc analysis of 11 specific studies? •  The data is a lot messier than is typically cited NB: This study is still worth reading, ….
  9. 9. 9 The study that answers the question: How many users do I need to test to be confident I will find most of the UX problems? Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments & Computers, 35, 3, Psychonomic Society (2003), 379--383.
  10. 10. What did Faulkner do? 10 Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. 1.  Conducted a large scale (100 participant) usability test of a corporate intranet (A real study, btw.) 2.  Analyzed findings by test and by participant EX: 100 participants uncovered 800 problems Participant 1 uncovered 8 problems Participant 2 uncovered 20 problems Participant 3 uncovered 25 problems (NOTE: These are not the real #s) 3. Simulated usability tests by drawing repeated random sets of participants from the pool of 100. Set sizes were 5, 10, 15, 20 …. 50 per test. 4. Plotted the (range of variability) of # problems found for each test by participant groups size
  11. 11. Variability of findings in tests w/ 5 (random) users 11 %age  of  problems  found   Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. Quite  a  few  outliers,  based     on  random  par4cipant  draws  
  12. 12. Variability of findings in tests w/ 10 (random) users 12 Each  dot  is  a   simulated  “test”   %age  of  problems  found   Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. A  few  outliers  here  …  
  13. 13. Variability of findings in tests w/ 15 (random) users 13 %age  of  problems  found   Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. Each  dot  is  a   simulated  “test”   No  outliers  here  …  
  14. 14. The Δ in variability of problems found (Min – Max) 14 Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. Practitioner Takeaway If you test …. •  5, you may observe between 55-85% of the problems, depending on the luck of the participant draw. •  10 you will capture between 82% - 94% of the problems •  15 you get between 90-97% of the problems How comfortable are you (or is your client) about the risk in testing small Ns?
  15. 15. 15 Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting. BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003), 379--383. Practitioner Takeaway If you test …. •  Only 5, statistically, you observe between 55-85% of the problems. How many/few you observe depends on the luck of your participant draw. •  With 10 you will capture between 82% - 94% of the problems •  15 you get between 90-97% of the problems How comfortable are you (or is your client) about the risk of missing problems because you are testing small Ns?
  16. 16. One typical response to Faulkner’s finding But, but…, I’m a UX type I do a heuristic review along the way. Doesn’t that mitigate the risk?
  17. 17. Question #2 Do users care about what UXers rant about?
  18. 18. The Nielsen heuristics …. 18 Do you have a scorecard? Is it (even loosely)   based on this?  
  19. 19. The Nielsen heuristics …. 19 Forget dog years. How many years abo is this in internet time? Note that these are (still!) useful. But at a different level of description.
  20. 20. 20 The study that answers the question: Are the things that trip users up the same as the things that UX types notice in expert review?   Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users   Really  Care  About?  A  Comparison  of  Usability   Problems  Found  by  Users  and  Experts  on   Highly  Interac>ve  Websites  CHI’12,  Aus>n,   Texas,  USA.
  21. 21. What did Petrie, et. al. do? 21 Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Compared Usability Testing Findings (What users focus on) 6 websites - 30 users Expert Review Findings (What UXers focus on) 14 experts* / 3 different ER strategies •  Collabora>ve  heuris>c  evalua>on   •  Group  Usability  Expert  Walkthrough   •  Group  Domain  Expert  Walkthrough  (DEW)   935 problems found
  22. 22. 4 categories of problems 22 Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Only 14% of the identified problems overlap??   Fourbroadcategoriesofproblems χ2 is not significantly different for this table. That means that the distribution of the number problems found in each category by only users, only UXers or both is not different from what would be predicted.
  23. 23. 23 % of problems in that sub-category   Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Problems  that  UXers  and  users  both  no>ce  …  
  24. 24. 24 Users uncovered 1.8 times as many problems, so we would expect this to be 1.8 : 1   Users : Experts   Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Problems  Users  only  reported  more  than  UXers  only  
  25. 25. 25 Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Experts uncovered 1.8 times fewer problems, so we expect this ratio to be is 0.55 : 1   Experts : Users   Problems  UXers  only  reported  more  than  Users  only  
  26. 26. 26 Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   Problems  that  we  agree  about  …   Practitioner Takeaways •  We need to recognize that users care less and are less hindered by some of the things we traditionaly obsess about •  We need to be cautious about guidelines. We need them, but we also need to recognize that things change and guidelines need to be updated. •  Users can learn! As more users have become more experienced with the web, they’ve learned some of the basic patterns … AND some of the basic mistakes designers make. And they work around them. We must update our guidelines periodically, against real use behaviors, in studies such as this one.
  27. 27. What does “Expert” mean for Petrie, et.al.? 27 Petrie,  H.  &  Power,  C.  (2012)  What  Do  Users  Really  Care  About?  A  Comparison  of  Usability  Problems   Found  by  Users  and  Experts  on  Highly  Interac>ve  Websites  CHI’12,  Aus>n,  Texas,  USA   14 usability experts •  5 women / 9 men •  higher education qualifications or courses in HCI. •  5+ years experience in usability •  Worked as professionals in user experience, interaction or software/product design •  Usability responsibilities ranged from 50-100% time •  11 self-described as “experienced”; 3 as “junior”. •  Nearly all had conducted heuristic evaluations
  28. 28. QUESTIONS LATER ….? 28 kstraub     kath@usability.org     @kathstraub  
  29. 29. Usability.org services include •  Customer Research •  User Experience Design/Digital strategy for Web, GUI and mobile •  Content in Plain Language •  Design for sustainable Behavioral Change •  Usability Evaluation (Data-driven continuous improvement) We present hands-on training and mentoring in •  User research methods •  Interaction design for Web, GUI and Mobile •  Usability evaluation methods •  Content strategies and writing in plain language •  Workshop: Design strategies that drive behavioral change/Gamification •  Workshop: Research in Practice: Studies Usability Professionals should to know about * Many thanks to the Norwegian Tax Authority for allowing us to use their RiP workshop photo

×