Single-user problems


Published on

Presentation of the paper "Outliers in usability testing: How to treat usability problems found for only one test participant?" at NordiCHI'12, København, October 16, 2012

1 Like
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Single-user problems

  1. 1. Outliers in usability testing:How to treat usability problems found for onlyone test participant Asbjørn Følstad, SINTEF Effie Lai-Chong Law, University of Leicester Kasper Hornbæk, University of Copenhagen NordiCHI 2012
  2. 2. Content1 Single-user problems2 Yes, they are abundant3 … but how to deal with them?4 Current practices – straight from the horses mouth5 Recommendations 2
  3. 3. Single-user problems The problems backed up with data from only a single participant in usability test 3
  4. 4. Single-user problems Are they relevant? May be infrequent usability problems - Point estimate: .25 (LaPlace) - 95% conf. int.: .01-.58 (Adj. Wald) Are they valid? May be an artefact of the test situation "there is always a risk of being misled by the spurious behavior of a single person" (Nielsen, 2000 – 4
  5. 5. Single-user problems are abundant Office system eval. Content 15 participants Nielsen and management Landauer system eval. 77 of 145 (1993) problems single- 17 participants user problems 41 of 88 problemsLaw and single-userHvannberg problems(2004) Law, E.L.-C. Hvannberg, E.T. Analysis of Combinatorial User Effect in International Usability Tests. In Proc. CHI 04, ACM Press (2004), 9-16. Nielsen, J., Landauer, T.K. A mathematical model of the finding of usability problems. In Proc. CHI 93, ACM Press (1993), 206-213. 5
  6. 6. Advice on how to deal with them is scarce View "unique Short discussion of problems as noise Kjeldskov, Skov single-user problems. rather than real and Stage Recommend to report usability problems" (2004) these as outliers in study of instant data analysis Report single user problems as real problems in a stress- test of problem Woolrych predictions and Cockton (2001) Kjeldskov J., Skov M. B., Stage J. Instant Data Analysis: Evaluating Usability in a Day. In Proc. NordiCHI 04, ACM Press (2004), 233-240. Woolrych, A., Cockton, G. Why and when five test users aren’t enough. In Proc. IHM-HCI 2001, Cépadèus Éditions (2001), 105-108. 6
  7. 7. Asking the practitioners for current practicesOpportunity: Larger survey onanalysis practices in usabilityevaluationIncluded question on single-userproblems89 usability practitioners answeredthis particular questionMedian 6 yrs. work experience17 different countriesUsability tests with median of 8user participants 7
  8. 8. Potential outcomes for single-user problems 8 acceptParticipants as divided as the 4 classify as low prioritylittle advice provided in theliterature 4 record as outlier 6 reject (22 items total on this theme) 8
  9. 9. Relevant conditions when making the call 18 Problem severity 9 Test participants profileA range of conditions 6 Sample sizereported as relevant. Butsome maybe deserving to bereported more often? 6 Artifact of the test situation? 5 Task importance 5 Other (49 items total on this theme) 9
  10. 10. Resources and strategies when making the call 9 Discuss with experts or team members20 reported to rely on own 9 New/extended evaluationsprofessional knowledge and 8 Check against heuristics /experience. guidelines / principlesHowever, several potentialuseful resources and […]strategies were reported 3 Specific process or policy 2 Confirmed hypotheses / previous experiences 2 Debrief with users (63 items total on this theme) 10
  11. 11. When considering the study findings … How did you handle single-user problems in your latest usability test? 11
  12. 12. Recommendations1 Procedure for handling single-user problems2 Pay particular attention to sample size3 Check against knowledge resources – guidelines, heuristics, or previous evaluations4 Seek advice - from experts or team members5 Be alert: Artefact of the test situation? 12
  13. 13. Thank you 13