Subjective questionnaires

  • 104 views
Uploaded on

 

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
104
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • In order to compute the overall value of the scale, the first step is to make all the items consistent (in this case, making lower values imply more negative feelings). For this purpose, we must reverse the items 2, 4, 6, 8 and 10. If the user strongly agreed (5) to the sentence ‘I found the system unnecessariliy complex’, reversing the item is equivalent to saying that the user strongly disagrees to the opposite of that sentence (the positive feeling). Since she strongly disagrees, she does not add to the SUM score.
  • In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
  • In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
  • In order to be on the safe side, we should aim at getting SUS scores greater than 80% (including confidence intervals).
  • Figura extraida del libro Measuring Web Usability
  • [Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability
    studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.
  • One type of analysis that can be very valuable is to look at the difference between
    participants’ awareness of a specific piece of information or functionality and the
    perceived usefulness of that same piece of information or functionality once they
    are made aware of it. For example, if a vast majority of participants are unaware of
    some specific functionality, but once they notice it they find it very useful, you
    should promote or highlight that functionality in some way.
    To analyze awareness–usefulness gaps, you must have both an awareness and a
    usefulness metric. We typically ask participants about awareness as a yes/no
    question—for example, ‘‘Were you aware of this functionality prior to this study?
    (yes or no).’’ Then we ask: ‘‘On a 1 to 5 scale, how useful is this functionality to you?
    (1 = Not at all useful; 5 = Very useful).’’ This assumes that they have had a couple of
    minutes to explore the functionality. Next, you will need to convert the rating-scale
    data into a top-2-box score so that you have an apples-to-apples comparison. Simply
    plot the percent of participants who are aware of the functionality next to the
    percent of those who found the functionality useful (percent top-2-box). The difference
    between the two bars is called the awareness–usefulness gap (see Figure 6.29).
  • One type of analysis that can be very valuable is to look at the difference between
    participants’ awareness of a specific piece of information or functionality and the
    perceived usefulness of that same piece of information or functionality once they
    are made aware of it. For example, if a vast majority of participants are unaware of
    some specific functionality, but once they notice it they find it very useful, you
    should promote or highlight that functionality in some way.
    To analyze awareness–usefulness gaps, you must have both an awareness and a
    usefulness metric. We typically ask participants about awareness as a yes/no
    question—for example, ‘‘Were you aware of this functionality prior to this study?
    (yes or no).’’ Then we ask: ‘‘On a 1 to 5 scale, how useful is this functionality to you?
    (1 = Not at all useful; 5 = Very useful).’’ This assumes that they have had a couple of
    minutes to explore the functionality. Next, you will need to convert the rating-scale
    data into a top-2-box score so that you have an apples-to-apples comparison. Simply
    plot the percent of participants who are aware of the functionality next to the
    percent of those who found the functionality useful (percent top-2-box). The difference
    between the two bars is called the awareness–usefulness gap (see Figure 6.29).
  •  You can then further
    analyze the data by generating a confidence interval to understand what percent of all users likely
    feel this way (see Chapter 3).
  •  You can then further
    analyze the data by generating a confidence interval to understand what percent of all users likely
    feel this way (see Chapter 3).
  •  You can then further
    analyze the data by generating a confidence interval to understand what percent of all users likely
    feel this way (see Chapter 3).

Transcript

  • 1. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES Usability Testing Subjective Questionnaires Cristina Cachero This project has been funded with support from the European Commission under the Lifelong Learning Programme
  • 2. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Inquiry method: you ask your users about what they do, but you do not observe it directly.  There are several proposals that have shown enough internal and external validity, and that can be applied at different times during the performance test  After all the tasks have been finished (test-level questionnaires)  Before/after each task is performed (task-level questionnaires)  Also, questionnaires may include open questions that can be codified and analyzed. This project has been funded with support from the European Commission under the Lifelong Learning Programme 2 Subjective Questionnaires
  • 3. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Test-level (post-test) questionnaires (1/2):  CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences, all positive. They measure four dimensions: system utility, information quality, interface quality and general satisfaction. It was devised to be administered off- line  PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be administered in person  QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five groups: general reaction, screen, terminology/system info, learning, system capabilities.  EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items representing five dimensions: content, accuracy, format, ease of use and timeliness. All the items are combined into a global EUCS measure. 5-point scale (Almost never… almost always)  USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in four categories: utility, satisfaction, ease of use and ease of learning. For each one, seven-point scale items..This project has been funded with support from the European Commission under the Lifelong Learning Programme 3 Subjective Questionnaires
  • 4. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Test-level (post-test) questionnaires (2/2):  SUS: Systems Usability Scale [Brooke 1996]: 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability  Net promoter Scores (NPS) ©: A single score regarding the fidelity of the user  SUMI [Kirakowski 1996]  Questionnaires that rank sites (based on proprietary DB)  SUPR-Q: UX questionnaire . Includes usability (4 items), credibility (trust, value & comfort, 5 items), loyalty (2 items, one of which is the Net Promoter Score) and Appearance (2 items). Five-point scale from strongly disagree to strongly agree except for Item 10 (net promoter question)  WAMMI, ACSI, Opinion Lab  Proprietary (beware of reliability and validity!!) This project has been funded with support from the European Commission under the Lifelong Learning Programme 4 Subjective Questionnaires Source: ISO 13407
  • 5. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 5 Post-test SQ: EUCS [Abdinnour-Helm 2006].
  • 6. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 6 Post-test SQ: USE [Lund 2001] http://hcibib.org/perlman/question.cgi?form=USE
  • 7. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  SUS (Systems Usability Scale) [Brooke 1996] 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability (it does not distinguish among different components). It is the most reliable [Tullis and Stetson, 2004] This project has been funded with support from the European Commission under the Lifelong Learning Programme 7 Post-test SQ: SUS [Brooke 1996]  The mean SUS value is 66%, the 25 percentile is 57% and the 75 percentile is 77%. This means that we should get a SUS greater than 80% (taking into account confidence intervals) to be reasonably sure of the global satisfaction of our users.
  • 8. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Activity: Read the original SUS paper, published by Brooke in 1996. You have the paper available in the Moodle platform. This project has been funded with support from the European Commission under the Lifelong Learning Programme 8 Post-test SQ: SUS
  • 9. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 9 Post-test SQ: SUS: Exercise 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree
  • 10. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 10 Post-test SQ: SUS: Score? 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree          
  • 11. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 11 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree           Post-test SQ: SUS. Steps  Reverse items 2,4,6,8,10  Sum positions for each item (0 to 4)  Multiply by 2.5 4 1 1 4 1 2 1 1 4 3 TOTAL SCORE: 22 SUS SCORE: 22*2.5=55
  • 12. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 12 Post-test SQ: SUS. Threshold value Which should be our SUS objective?
  • 13. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 13 Post-test SQ: SUS. Adaptation  The SUS vocabulary can be adapted to the idyosyncrasy of the particular system  Example: OHIM: system->web site
  • 14. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 14 Post-test SQ: SUS. Calculator  Calculator: http://www.measuringux.com/SUS-scores.xls
  • 15. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Net Promoter Scores: measure obtained through a single question (loyalty of the user to the app):  How likely is it that you’ll recommend this product to a friend or colleague? (0..10)  Three segments:  Promoters: Responses from 9 to 10  Passives: Responses from 7 to 8  Detractors: Responses from 0 to 6  Promoter score (-100..+100): %Promoters-%Detractors This project has been funded with support from the European Commission under the Lifelong Learning Programme 15 Post-test SQ: Net Promoter Score
  • 16. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Questionnaires that provide rankings to compare your results against the results from other similar apps  SUPR-Q (Sauro)  WAMMI (www.wammi.org) (SUMI successor)  ACSI (www.TheACSI.org): particularly interesting form government websites  OpinionLab (www.OpinionLab.com) This project has been funded with support from the European Commission under the Lifelong Learning Programme 16 Post-test SQ: Rankings
  • 17. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 17 SQ: Rankings. SUPR-Q (4 sub-scales) NPS
  • 18. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 18 Post-test SQ: Rankings. SUPR-Q.  SUPR-Q Score: Sum all items + ½ item 10
  • 19. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 19 Post-test SQ: Rankings. SUPR-Q.  Besides the values for each factor, SUPR-Q is backed on a proprietary DB that allows to generate a ranking of percentiles  E.g. your app belongs to percentil 75 regarding usability, what means, it is among the best 25%.  The SUPR-Q usability factor has a strong correlation with a SUS score, r = .96. p < .001, meaning just four questions account for 93 percent of the variation in SUS (.96 squared). These questions are a good substitute for SUS on websites  Validity and reliability: http://www.suprq.com/
  • 20. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 20 Post-test SQ: Rankings. WAMMI  http://www.wammi.com/samples/index.html  Results are divided into five areas: attractiveness, controllability, efficiency, helpfulness and Learnability, plus an overall usability score.  The scores are standardized (from comparison to their reference database), so 50 is medium and 100 is perfect.
  • 21. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 21 Post-test SQ: Rankings. WAMMI  Sample graphic showing how a give website positions in reference to average scores in each axis
  • 22. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Circle five words that describe what you think about this design:   What are the three things you like best and least about the Web site?  _____________________________________________  ______________________________________________  ______________________________________________ This project has been funded with support from the European Commission under the Lifelong Learning Programme 22 Post-test SQ: proprietary (OHIM 1/2) Cute Stable Responsive Friendly Helpful Reputable Approachable Reliable Cluttered Good Confident Trustworthy Current and cool Service oriented Boring Easy to use Confusing Comfortable Annoying Informative Out-of-date High-tech Sensitive Secure Straightforward Amateurish
  • 23. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  If you could make one significant change to this Web site, what change would you make?  (Ask the participant if he is using/knows other IP websites) How do you find the site in comparison to other IP web sites?  Do you feel this site is current? Why?  If you were to describe this site to a colleague in a sentence or two, what would you say?  Do you use the current OHIM web site? If Yes then  Do you think that the new OHIM web site clarity of structure is: Better, Same As, Worse or Don’t know, than the current one?  Do you have any other questions or comments about the Web site or your experiences with it? This project has been funded with support from the European Commission under the Lifelong Learning Programme 23 Post-test SQ: proprietary (OHIM 2/2)
  • 24. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Expectation Rating  After Scenario Questionnaire (ASQ)  Awareness-Usefulness gap This project has been funded with support from the European Commission under the Lifelong Learning Programme 24 Task level (Post-Scenario) SQ
  • 25. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Pre-task: How easy/difficult do you thing that the following task is going to be?  Post-task: How easy/difficult has been for you to carry out this task? This project has been funded with support from the European Commission under the Lifelong Learning Programme 25 Post-Scenario SQ: Expectation Rating
  • 26. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Three post-task questions:  I feel satisfied with the easiness with which I have completed the task.  I feel satisfied with the time that it has taken me to complete this task.  I feel satisfied with the support information (online help, messages, documentation, and so on) that I have had available while I was completing this task (only if your system offers online help) This project has been funded with support from the European Commission under the Lifelong Learning Programme 26 Post-Scenario SQ: ASQ
  • 27. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Two post-task questions:  ‘‘Were you aware of this functionality prior to this study? (yes or no)  On a 1 to 5 scale, how useful is this functionality to you?  Calculation:  Convert the likert item to a binary scale (e.g. 4 or 5 useful, rest not useful) and draw the graph.  Differences between awareness and usefulness indicate which tasks to promote (e.g. by redesigning the visual hierarchy) in your app. This project has been funded with support from the European Commission under the Lifelong Learning Programme 27 Post-Scenario SQ: Awar-Usefuln Gap
  • 28. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Graph sample: This project has been funded with support from the European Commission under the Lifelong Learning Programme 28 Post-Scenario SQ: Awar-Usefuln Gap
  • 29. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  More questionnaires are appearing by the day in the literature. Many of them have been deemed necessary to cover the idiosyncrasy of new platforms/domains  E.g. MPUQ: Mobile Phone Usability Questionnaire This project has been funded with support from the European Commission under the Lifelong Learning Programme 29 SQ: Other questionnaires Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ)
  • 30. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Can be post-test or post-task  Reasons why the users are promoters or detractors of your product  Insights from users gathered from field studies  Complaints about a product sent to the customer service  Why the task was difficult to complete  …  The way of processing this kind of questions is to turn the open comments into categories, quantify them and analyze them.  http://www.measuringusability.com/blog/quantify-comments.php This project has been funded with support from the European Commission under the Lifelong Learning Programme 30 SQ: Open Questions
  • 31. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  All the questionnaires must be validated with respect to their reliability, validity and utility. Once validated, they still need to be revalidated when the context of application varies. This project has been funded with support from the European Commission under the Lifelong Learning Programme 31 SQ: Validation
  • 32. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.  [Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2  [Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in Computing Systems. New York: ACM, pp. 213–218.  [Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.  [Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 169–177.  [Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.  [Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction with a Web Site.  [Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman. This project has been funded with support from the European Commission under the Lifelong Learning Programme 32 References
  • 33. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES These slides are made available under the license Creative Commons Attribution-NonCommercial-NoDerivs CC BY- NC-ND. More information about license: http://creativecommons.org/licenses/by-nc-nd/3.0/. These slides were created under Leonardo da Vinci Partnerships Project 2012-1-PL1-LEO04-28181 GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES (http://usability- accessibility.org/). This project has been funded with support from the European Commission under the Lifelong Learning Programme 33 Attributions