Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pictorial System Usability Scale (P-SUS) - Developing an Instrument for Measuring Perceived Usability

13 views

Published on

Slides of CHI conference in Glasgow 2019, paper session

Published in: Science
  • Be the first to comment

  • Be the first to like this

Pictorial System Usability Scale (P-SUS) - Developing an Instrument for Measuring Perceived Usability

  1. 1. Pictorial System Usability Scale (P-SUS) Developing an Instrument for Measuring Perceived Usability Baumgartner J. ▪ Frei N. ▪ Kleinke M. ▪ Sauer J. ▪ Sonderegger A.
  2. 2. AGENDA 1. introduction 2. scale development 3. validation study 4. discussion 2
  3. 3. 1. INTRODUCTION 3
  4. 4. USABILITY EVALUATION 1. introduction 4
  5. 5. 1. introduction POTENTIAL PROBLEMS OF VERBAL USABILITY SCALES 5 § They can be tedious after a (long) usability test or survey § Require attention/motivation of the participant § Require comprehension of the specific language § Is normally not a pleasure/experience
  6. 6. 1. introduction PICTORIAL SCALES IN HCI SAM - Self Assessment Manikin (Bradley & Lang, 1994) PREMO - Product Emotion Measurement Tool (Desmet, 2003, 2009) LEMtool – Layered Emotion Measurement Tool (Huisman & Van Hout, 2010) 6 AniSAM - Animated Self Assessment Manikin (Sonderegger et al., 2016, 2019)
  7. 7. PICTORIAL USABILITY SCALES IN HCI PSIUS – Pictorial Single Item Usability Scale (Baumgartner et al., 2019) -3 -1 1 3-4 -2 0 2 4 PSIUS – Pictorial Single Item Usability Scale (Baumgartner et al., in prep.) 7 1. introduction
  8. 8. ADVANTAGES More fun than verbal questionnaires Faster in filling in pictorial questionnaires Culture-free Higher levels of validity than verbal questionnaires Kunin, 1955; Bradley & Lang, 1994; Wissmath et al., 2010 Comprehensible across language borders DE = FR = EN No need to translate words into feelings/thoughts 8 1. introduction
  9. 9. PROJECT‘S AIM 9 1. introduction §Provide an inclusive, motivating and pleasant alternative to the assessment of perceived usability §Gather initial data allowing to determine psychometric properties
  10. 10. 2. SCALE DEVELOPMENT 10
  11. 11. 1. introduction SYSTEM USABILITY SCALE (SUS) 11 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009
  12. 12. 1. introduction SYSTEM USABILITY SCALE (SUS) 12 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009
  13. 13. 1. introduction SYSTEM USABILITY SCALE (SUS) 13 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009
  14. 14. DEVELOPMENT PROCESS Association Elicitation Test Design Meetings Validation Study Think-Aloud Expert Survey Sketching 14 2. scale development
  15. 15. FINAL SCALE Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 1. I think that I would like to use this system frequently vs. 15 2. scale development
  16. 16. FINAL SCALE 16 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 2. I found the system unnecessarily complex. vs. ?!
  17. 17. FINAL SCALE 17 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 3. I thought the system was easy to use vs. = EASY= ARRGH!
  18. 18. FINAL SCALE 18 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 4. I think that I would need the support of a technical person to be able to use this system vs. AHA z zzz ? HELP
  19. 19. FINAL SCALE 19 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 5. I found the various functions in this system were well integrated vs.
  20. 20. FINAL SCALE 20 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 6. I thought there was too much inconsistency in this system vs. ?!
  21. 21. FINAL SCALE 21 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 7. I would imagine that most people would learn to use this system very quickly vs. 10 2 3 4 hours How to use 10 2 3 4 hours ? ? How to use
  22. 22. FINAL SCALE 22 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 8. I found the system very cumbersome to use vs. 1 2 ARRGH! 1 2 4 5 3
  23. 23. FINAL SCALE 23 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 9. I felt very confident using the system vs.
  24. 24. FINAL SCALE 24 2. scale development Tu We Th Fr Sa SuMo Future use Tu We Th Fr Sa SuMo Future use 10. I needed to learn a lot of things before I could get going with this system vs. 21 1 2 3
  25. 25. 3. VALIDATION STUDY 25
  26. 26. §Do P-SUS has similar psychometric properties like SUS? (à Focus on convergent validity) §Do P-SUS is more fun than SUS? §Do participants are faster filling in P-SUS than SUS? RESEARCH QUESTIONS 26 3. validation study
  27. 27. STUDY Sample § N=60 (65% female) § 50 students, 10 employees § 20 - 31 yrs (M=22.88, SD=1.56) Experimental Setup § Smartphone Prototype (Hamborg et al., 2014) § Lab/at home § Portable usability test lab Measures and Instruments § Usability (P-SUS, SUS) § Questionnaire Completion Time § Motivation/Fun (IMI - Intrinsic Motivation Inventory - Wilde et al., 2009) § … Experimental Design § One-factorial, between-subjects § Independent variable: System Usability (low vs. high) 27 3. validation study
  28. 28. Interaction with Smartphone Prototype (low or high usability) PROCEDURE 1 2 Filling in verbal questionnaires 3 Filling in pictorial questionnaires 28 3. validation study
  29. 29. 29 3. validation study
  30. 30. 30 3. validation study
  31. 31. RESULTS – CONVERGENT VALIDITY (SUS vs. P-SUS) 31 3. validation study GLOBAL SCORE 100 50 75 25 0 USABILITYSCORE(0-100)
  32. 32. RESULTS – CONVERGENT VALIDITY (SUS vs. P-SUS) 32 3. validation study GLOBAL SCORE 100 50 75 25 0 USABILITYSCORE(0-100) .624*** .673*** .801*** .378**r .548*** .211 .575*** .646*** .668*** .489*** .865***
  33. 33. RESULTS – MOTIVATION (IMI) 33 3. validation study 7 FUN ** JOY *** INTEREST *** 6 5 4 3 2 1 RATING(1-7) *** GLOBAL SCORE MOTIVATIONSCORE(1-7) verbal SUS * p < .05; ** p < .01; *** p < .001 Error bars: 95% CI pictorial SUS
  34. 34. RESULTS – QUESTIONNAIRE COMPLETION TIME 34 3. validation study *** 150 120 90 60 30 0 SECONDS verbal SUS * p < .05; ** p < .01; *** p < .001 Error bars: 95% CI pictorial SUS
  35. 35. 4. DISCUSSION 35
  36. 36. § 2/3 of P-SUS items have a substantial correlation with corresponding SUS items (r > .500). § Very high correlation between SUS score and P-SUS score (r = .865) § P-SUS is perceived more fun and more interesting § SUS is filled in more quickly FINDINGS 4. discussion 36
  37. 37. § Sample - Small size § Sample – Too homogeneous § Educational background § Cultural background § Not all items are exclusively nonverbal LIMITATIONS 37 4. discussion
  38. 38. FUTURE STUDIES § Refinement of scales (i.e. items 04, 06 and 10) § Validation study with bigger and more heterogeneous sample § Consider animations for more meaningful scales § Need to develop guidelines/standardised approach 38 4. discussion
  39. 39. CONCLUSION § Satisfactory psychometric properties of P-SUS § At least for this pilot study § Increased questionnaire experience § Longer completion time 39 4. discussion
  40. 40. Coauthors ACKNOWLEDGMENT Naomi Frei Mascha Kleinke Juergen Sauer Andreas Sonderegger Institutions University of Fribourg Puzzle ITC We Are Cube Design support Franziska Asenbauer 40 4. discussion Mayra Overney Veronica Solombrino
  41. 41. Thank you for your attention! 41 Contact @ux_padawan juergen.baumgartner@unifr.ch

×