Subjective questionnaires

Leonardo da Vinci Partnerships Project
GUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Usability Testing
Subjective Questionnaires
Cristina Cachero
This project has been funded with support from the European
Commission under the Lifelong Learning Programme

 Inquiry method: you ask your users about what they do,
but you do not observe it directly.
 There are several proposals that have shown enough
internal and external validity, and that can be applied at
different times during the performance test
 After all the tasks have been finished (test-level questionnaires)
 Before/after each task is performed (task-level questionnaires)
 Also, questionnaires may include open questions that
can be codified and analyzed.
2

 Test-level (post-test) questionnaires (1/2):
 CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences,
all positive. They measure four dimensions: system utility, information quality,
interface quality and general satisfaction. It was devised to be administered off-
line
 PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be
administered in person
 QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five
groups: general reaction, screen, terminology/system info, learning, system
capabilities.
 EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items
representing five dimensions: content, accuracy, format, ease of use and
timeliness. All the items are combined into a global EUCS measure. 5-point
scale (Almost never… almost always)
 USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in
four categories: utility, satisfaction, ease of use and ease of learning. For each
one, seven-point scale items..This project has been funded with support from the European
3

 Test-level (post-test) questionnaires (2/2):
 SUS: Systems Usability Scale [Brooke 1996]: 10 questions, 5
formulated in positive and 5 formulated in negative. It gives a
global measure of the site usability
 Net promoter Scores (NPS) ©: A single score regarding the
fidelity of the user
 SUMI [Kirakowski 1996]
 Questionnaires that rank sites (based on proprietary DB)
 SUPR-Q: UX questionnaire . Includes usability (4 items), credibility (trust, value & comfort, 5 items),
loyalty (2 items, one of which is the Net Promoter Score) and Appearance (2 items). Five-point scale
from strongly disagree to strongly agree except for Item 10 (net promoter question)
 WAMMI, ACSI, Opinion Lab
 Proprietary (beware of reliability and validity!!)
4
Source: ISO 13407

5
Post-test SQ: EUCS [Abdinnour-Helm 2006].

6
Post-test SQ: USE [Lund 2001]
http://hcibib.org/perlman/question.cgi?form=USE

 SUS (Systems Usability Scale) [Brooke 1996] 10
questions, 5 formulated in positive and 5 formulated in
negative. It gives a global measure of the site usability (it
does not distinguish among different components). It is
the most reliable [Tullis and Stetson, 2004]
7
Post-test SQ: SUS [Brooke 1996]
 The mean SUS value is 66%, the 25
percentile is 57% and the 75
percentile is 77%. This means that we
should get a SUS greater than 80%
(taking into account confidence
intervals) to be reasonably sure of the
global satisfaction of our users.

 Activity: Read the original SUS paper, published by
Brooke in 1996. You have the paper available in the
Moodle platform.
8
Post-test SQ: SUS

9
Post-test SQ: SUS: Exercise
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly
Disagree Agree

10
Post-test SQ: SUS: Score?
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
Strongly Strongly
Disagree Agree











11
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
Strongly Strongly
Disagree Agree










Post-test SQ: SUS. Steps
 Reverse items 2,4,6,8,10
 Sum positions for each item (0 to 4)
 Multiply by 2.5
4
1
1
4
1
2
1
1
4
3
TOTAL
SCORE: 22
SUS SCORE:
22*2.5=55

12
Post-test SQ: SUS. Threshold value
Which should be
our SUS objective?

13
Post-test SQ: SUS. Adaptation
 The SUS vocabulary can be adapted to the idyosyncrasy
of the particular system
 Example: OHIM: system->web site

14
Post-test SQ: SUS. Calculator
 Calculator: http://www.measuringux.com/SUS-scores.xls

 Net Promoter Scores: measure obtained through a single
question (loyalty of the user to the app):
 How likely is it that you’ll recommend this product to a friend or
colleague? (0..10)
 Three segments:
 Promoters: Responses from 9 to 10
 Passives: Responses from 7 to 8
 Detractors: Responses from 0 to 6
 Promoter score (-100..+100): %Promoters-%Detractors
15
Post-test SQ: Net Promoter Score

 Questionnaires that provide rankings to compare your
results against the results from other similar apps
 SUPR-Q (Sauro)
 WAMMI (www.wammi.org) (SUMI successor)
 ACSI (www.TheACSI.org): particularly interesting
form government websites
 OpinionLab (www.OpinionLab.com)
16
Post-test SQ: Rankings

17
SQ: Rankings. SUPR-Q (4 sub-scales)
NPS

18
Post-test SQ: Rankings. SUPR-Q.
 SUPR-Q Score: Sum all items + ½ item 10

19
Post-test SQ: Rankings. SUPR-Q.
 Besides the values for each factor, SUPR-Q is backed on a
proprietary DB that allows to generate a ranking of percentiles
 E.g. your app belongs to percentil 75 regarding
usability, what means, it is among the best 25%.
 The SUPR-Q usability factor has a strong correlation with a SUS
score, r = .96. p < .001, meaning just four questions account for 93
percent of the variation in SUS (.96 squared). These questions are a
good substitute for SUS on websites
 Validity and reliability: http://www.suprq.com/

20
Post-test SQ: Rankings. WAMMI
 http://www.wammi.com/samples/index.html
 Results are divided into five areas: attractiveness,
controllability, efficiency, helpfulness and
Learnability, plus an overall usability score.
 The scores are standardized (from comparison to
their reference database), so 50 is medium and
100 is perfect.

21
Post-test SQ: Rankings. WAMMI
 Sample graphic showing how a give website positions in
reference to average scores in each axis

 Circle five words that describe what you think about
this design:

 What are the three things you like best and least
about the Web site?
 _____________________________________________
 ______________________________________________
 ______________________________________________
22
Post-test SQ: proprietary (OHIM 1/2)
Cute
Stable
Responsive
Friendly
Helpful
Reputable
Approachable
Reliable
Cluttered
Good
Confident
Trustworthy
Current and cool
Service oriented
Boring
Easy to use
Confusing
Comfortable
Annoying
Informative
Out-of-date
High-tech
Sensitive
Secure
Straightforward
Amateurish

 If you could make one significant change to this Web site, what
change would you make?
 (Ask the participant if he is using/knows other IP websites) How do
you find the site in comparison to other IP web sites?
 Do you feel this site is current? Why?
 If you were to describe this site to a colleague in a sentence or two,
what would you say?
 Do you use the current OHIM web site? If Yes then  Do you think
that the new OHIM web site clarity of structure is: Better, Same As,
Worse or Don’t know, than the current one?
 Do you have any other questions or comments about the Web site or
your experiences with it?
23
Post-test SQ: proprietary (OHIM 2/2)

 Expectation Rating
 After Scenario Questionnaire (ASQ)
 Awareness-Usefulness gap
24
Task level (Post-Scenario) SQ

 Pre-task: How easy/difficult do you thing that the
following task is going to be?
 Post-task: How easy/difficult has been for you to
carry out this task?
25
Post-Scenario SQ: Expectation Rating

 Three post-task questions:
 I feel satisfied with the easiness with which I have
completed the task.
 I feel satisfied with the time that it has taken me to
complete this task.
 I feel satisfied with the support information (online
help, messages, documentation, and so on) that I
have had available while I was completing this task
(only if your system offers online help)
26
Post-Scenario SQ: ASQ

 Two post-task questions:
 ‘‘Were you aware of this functionality prior to this study?
(yes or no)
 On a 1 to 5 scale, how useful is this functionality to you?
 Calculation:
 Convert the likert item to a binary scale (e.g. 4 or 5 useful,
rest not useful) and draw the graph.
 Differences between awareness and usefulness indicate
which tasks to promote (e.g. by redesigning the visual
hierarchy) in your app.
27
Post-Scenario SQ: Awar-Usefuln Gap

 Graph sample:
28
Post-Scenario SQ: Awar-Usefuln Gap

 More questionnaires are appearing by the day in the
literature. Many of them have been deemed
necessary to cover the idiosyncrasy of new
platforms/domains
 E.g. MPUQ: Mobile Phone Usability Questionnaire
29
SQ: Other questionnaires
Reliability and Validity of the Mobile Phone
Usability Questionnaire (MPUQ)

 Can be post-test or post-task
 Reasons why the users are promoters or detractors of your
product
 Insights from users gathered from field studies
 Complaints about a product sent to the customer service
 Why the task was difficult to complete
 …
 The way of processing this kind of questions is to
turn the open comments into categories, quantify
them and analyze them.
 http://www.measuringusability.com/blog/quantify-comments.php
30
SQ: Open Questions

 All the questionnaires must be validated with respect to
their reliability, validity and utility. Once validated, they
still need to be revalidated when the context of
application varies.
31
SQ: Validation

 Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and
instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.
 [Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2
 [Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user
satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in
Computing Systems. New York: ACM, pp. 213–218.
 [Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and
Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.
 [Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In
Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp.
169–177.
 [Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability
studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.
 [Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction
with a Web Site.
 [Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and
Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman.
32
References

These slides are made available under the license Creative
Commons Attribution-NonCommercial-NoDerivs CC BY-
NC-ND. More information about license:
http://creativecommons.org/licenses/by-nc-nd/3.0/.
These slides were created under Leonardo da Vinci
Partnerships Project 2012-1-PL1-LEO04-28181 GUI
USABILITY AND ACCESSIBILITY: EXCHANGING
KNOWLEDGE AND EXPERIENCES (http://usability-
accessibility.org/).
33
Attributions

Subjective questionnaires

More Related Content

What's hot

Viewers also liked

Similar to Subjective questionnaires

Recently uploaded

Subjective questionnaires

Editor's Notes