Slides of CHI conference in Glasgow 2019, paper session

1. Pictorial System Usability Scale (P-SUS)
Developing an Instrument for Measuring Perceived Usability
Baumgartner J. ▪ Frei N. ▪ Kleinke M. ▪ Sauer J. ▪ Sonderegger A.

2. AGENDA
1. introduction
2. scale development
3. validation study
4. discussion
2

3. 1. INTRODUCTION
3

4. USABILITY EVALUATION
1. introduction
4

5. 1. introduction
POTENTIAL PROBLEMS OF VERBAL USABILITY SCALES
5
§ They can be tedious after a (long) usability test or survey
§ Require attention/motivation of the participant
§ Require comprehension of the specific language
§ Is normally not a pleasure/experience

6. 1. introduction
PICTORIAL SCALES IN HCI
SAM - Self Assessment Manikin
(Bradley & Lang, 1994)
PREMO - Product Emotion Measurement Tool
(Desmet, 2003, 2009)
LEMtool – Layered Emotion Measurement Tool
(Huisman & Van Hout, 2010)
6
AniSAM - Animated Self Assessment Manikin
(Sonderegger et al., 2016, 2019)

7. PICTORIAL USABILITY SCALES IN HCI
PSIUS – Pictorial Single Item Usability Scale
(Baumgartner et al., 2019)
-3 -1 1 3-4 -2 0 2 4
PSIUS – Pictorial Single Item Usability Scale
(Baumgartner et al., in prep.)
7
1. introduction

8. ADVANTAGES
More fun than verbal
questionnaires
Faster in filling in pictorial
questionnaires
Culture-free
Higher levels of validity
than verbal questionnaires
Kunin, 1955; Bradley & Lang, 1994; Wissmath et al., 2010
Comprehensible across
language borders
DE = FR = EN
No need to translate words
into feelings/thoughts
8
1. introduction

9. PROJECT‘S AIM
9
1. introduction
§Provide an inclusive, motivating and pleasant alternative
to the assessment of perceived usability
§Gather initial data allowing to determine psychometric
properties

10. 2. SCALE DEVELOPMENT
10

11. 1. introduction
SYSTEM USABILITY SCALE (SUS)
11
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009

12. 1. introduction
SYSTEM USABILITY SCALE (SUS)
12
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009

13. 1. introduction
SYSTEM USABILITY SCALE (SUS)
13
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
Brooke, 1986; Lewis & Sauro, 2009; Bangor et al., 2009

14. DEVELOPMENT PROCESS
Association
Elicitation
Test
Design
Meetings
Validation
Study
Think-Aloud
Expert Survey
Sketching
14
2. scale development

15. FINAL SCALE
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
1. I think that I would like to use this system frequently
vs.
15
2. scale development

16. FINAL SCALE
16
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
2. I found the system unnecessarily complex.
vs.
?!

17. FINAL SCALE
17
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
3. I thought the system was easy to use
vs.
= EASY= ARRGH!

18. FINAL SCALE
18
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
4. I think that I would need the support of a technical person
to be able to use this system
vs.
AHA
z
zzz
?
HELP

19. FINAL SCALE
19
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
5. I found the various functions in this system were well
integrated
vs.

20. FINAL SCALE
20
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
6. I thought there was too much inconsistency in this system
vs.
?!

21. FINAL SCALE
21
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
7. I would imagine that most people would learn to use this
system very quickly
vs.
10 2 3 4
hours
How to
use
10 2 3 4
hours
?
?
How to
use

22. FINAL SCALE
22
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
8. I found the system very cumbersome to use
vs.
1 2
ARRGH!
1 2
4
5
3

23. FINAL SCALE
23
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
9. I felt very confident using the system
vs.

24. FINAL SCALE
24
2. scale development
Tu We Th Fr Sa SuMo
Future use
Tu We Th Fr Sa SuMo
Future use
10. I needed to learn a lot of things before I could get going
with this system
vs.
21 1 2 3

25. 3. VALIDATION STUDY
25

26. §Do P-SUS has similar psychometric properties like SUS?
(à Focus on convergent validity)
§Do P-SUS is more fun than SUS?
§Do participants are faster filling in P-SUS than SUS?
RESEARCH QUESTIONS
26
3. validation study

27. STUDY
Sample
§ N=60 (65% female)
§ 50 students, 10 employees
§ 20 - 31 yrs (M=22.88, SD=1.56)
Experimental Setup
§ Smartphone Prototype
(Hamborg et al., 2014)
§ Lab/at home
§ Portable usability test lab
Measures and Instruments
§ Usability (P-SUS, SUS)
§ Questionnaire Completion Time
§ Motivation/Fun (IMI - Intrinsic
Motivation Inventory - Wilde et al.,
2009)
§ …
Experimental Design
§ One-factorial, between-subjects
§ Independent variable:
System Usability (low vs. high)
27
3. validation study

28. Interaction with Smartphone Prototype
(low or high usability)
PROCEDURE
1
2 Filling in verbal questionnaires
3 Filling in pictorial questionnaires
28
3. validation study

29. 29
3. validation study

30. 30
3. validation study

31. RESULTS – CONVERGENT VALIDITY (SUS vs. P-SUS)
31
3. validation study
GLOBAL
SCORE
100
50
75
25
0
USABILITYSCORE(0-100)

32. RESULTS – CONVERGENT VALIDITY (SUS vs. P-SUS)
32
3. validation study
GLOBAL
SCORE
100
50
75
25
0
USABILITYSCORE(0-100)
.624*** .673*** .801*** .378**r .548*** .211 .575*** .646*** .668*** .489*** .865***

33. RESULTS – MOTIVATION (IMI)
33
3. validation study
7
FUN
**
JOY
***
INTEREST
***
6
5
4
3
2
1
RATING(1-7)
***
GLOBAL
SCORE
MOTIVATIONSCORE(1-7)
verbal SUS
* p < .05; ** p < .01; *** p < .001
Error bars: 95% CI
pictorial SUS

34. RESULTS – QUESTIONNAIRE COMPLETION TIME
34
3. validation study
***
150
120
90
60
30
0
SECONDS
verbal SUS
* p < .05; ** p < .01; *** p < .001
Error bars: 95% CI
pictorial SUS

35. 4. DISCUSSION
35

36. § 2/3 of P-SUS items have a substantial correlation with
corresponding SUS items (r > .500).
§ Very high correlation between SUS score and P-SUS score
(r = .865)
§ P-SUS is perceived more fun and more interesting
§ SUS is filled in more quickly
FINDINGS
4. discussion
36

37. § Sample - Small size
§ Sample – Too homogeneous
§ Educational background
§ Cultural background
§ Not all items are exclusively nonverbal
LIMITATIONS
37
4. discussion

38. FUTURE STUDIES
§ Refinement of scales (i.e. items 04, 06 and 10)
§ Validation study with bigger and more heterogeneous sample
§ Consider animations for more meaningful scales
§ Need to develop guidelines/standardised approach
38
4. discussion

39. CONCLUSION
§ Satisfactory psychometric properties of P-SUS
§ At least for this pilot study
§ Increased questionnaire experience
§ Longer completion time
39
4. discussion

40. Coauthors
ACKNOWLEDGMENT
Naomi
Frei
Mascha
Kleinke
Juergen
Sauer
Andreas
Sonderegger
Institutions
University of
Fribourg
Puzzle ITC We Are Cube
Design support
Franziska
Asenbauer
40
4. discussion
Mayra
Overney
Veronica
Solombrino

41. Thank you for your attention!
41
Contact @ux_padawan
juergen.baumgartner@unifr.ch