Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Subjective questionnaires

2,905 views

Published on

Published in: Education
  • Be the first to comment

Subjective questionnaires

  1. 1. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES Usability Testing Subjective Questionnaires Cristina Cachero This project has been funded with support from the European Commission under the Lifelong Learning Programme
  2. 2. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Inquiry method: you ask your users about what they do, but you do not observe it directly.  There are several proposals that have shown enough internal and external validity, and that can be applied at different times during the performance test  After all the tasks have been finished (test-level questionnaires)  Before/after each task is performed (task-level questionnaires)  Also, questionnaires may include open questions that can be codified and analyzed. This project has been funded with support from the European Commission under the Lifelong Learning Programme 2 Subjective Questionnaires
  3. 3. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Test-level (post-test) questionnaires (1/2):  CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences, all positive. They measure four dimensions: system utility, information quality, interface quality and general satisfaction. It was devised to be administered off- line  PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be administered in person  QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five groups: general reaction, screen, terminology/system info, learning, system capabilities.  EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items representing five dimensions: content, accuracy, format, ease of use and timeliness. All the items are combined into a global EUCS measure. 5-point scale (Almost never… almost always)  USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in four categories: utility, satisfaction, ease of use and ease of learning. For each one, seven-point scale items..This project has been funded with support from the European Commission under the Lifelong Learning Programme 3 Subjective Questionnaires
  4. 4. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Test-level (post-test) questionnaires (2/2):  SUS: Systems Usability Scale [Brooke 1996]: 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability  Net promoter Scores (NPS) ©: A single score regarding the fidelity of the user  SUMI [Kirakowski 1996]  Questionnaires that rank sites (based on proprietary DB)  SUPR-Q: UX questionnaire . Includes usability (4 items), credibility (trust, value & comfort, 5 items), loyalty (2 items, one of which is the Net Promoter Score) and Appearance (2 items). Five-point scale from strongly disagree to strongly agree except for Item 10 (net promoter question)  WAMMI, ACSI, Opinion Lab  Proprietary (beware of reliability and validity!!) This project has been funded with support from the European Commission under the Lifelong Learning Programme 4 Subjective Questionnaires Source: ISO 13407
  5. 5. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 5 Post-test SQ: EUCS [Abdinnour-Helm 2006].
  6. 6. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 6 Post-test SQ: USE [Lund 2001] http://hcibib.org/perlman/question.cgi?form=USE
  7. 7. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  SUS (Systems Usability Scale) [Brooke 1996] 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability (it does not distinguish among different components). It is the most reliable [Tullis and Stetson, 2004] This project has been funded with support from the European Commission under the Lifelong Learning Programme 7 Post-test SQ: SUS [Brooke 1996]  The mean SUS value is 66%, the 25 percentile is 57% and the 75 percentile is 77%. This means that we should get a SUS greater than 80% (taking into account confidence intervals) to be reasonably sure of the global satisfaction of our users.
  8. 8. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Activity: Read the original SUS paper, published by Brooke in 1996. You have the paper available in the Moodle platform. This project has been funded with support from the European Commission under the Lifelong Learning Programme 8 Post-test SQ: SUS
  9. 9. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 9 Post-test SQ: SUS: Exercise 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree
  10. 10. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 10 Post-test SQ: SUS: Score? 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree          
  11. 11. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 11 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1. I think that I would like to use this system frequently 2. I found the system unnecessarily complex 3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system 5. I found the various functions in this system were well integrated 6. I thought there was too much inconsistency in this system 7. I would imagine that most people would learn to use this system very quickly 8. I found the system very cumbersome to use 9. I felt very confident using the system 10. I needed to learn a lot of things before I could get going with this system Strongly Strongly Disagree Agree           Post-test SQ: SUS. Steps  Reverse items 2,4,6,8,10  Sum positions for each item (0 to 4)  Multiply by 2.5 4 1 1 4 1 2 1 1 4 3 TOTAL SCORE: 22 SUS SCORE: 22*2.5=55
  12. 12. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 12 Post-test SQ: SUS. Threshold value Which should be our SUS objective?
  13. 13. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 13 Post-test SQ: SUS. Adaptation  The SUS vocabulary can be adapted to the idyosyncrasy of the particular system  Example: OHIM: system->web site
  14. 14. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 14 Post-test SQ: SUS. Calculator  Calculator: http://www.measuringux.com/SUS-scores.xls
  15. 15. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Net Promoter Scores: measure obtained through a single question (loyalty of the user to the app):  How likely is it that you’ll recommend this product to a friend or colleague? (0..10)  Three segments:  Promoters: Responses from 9 to 10  Passives: Responses from 7 to 8  Detractors: Responses from 0 to 6  Promoter score (-100..+100): %Promoters-%Detractors This project has been funded with support from the European Commission under the Lifelong Learning Programme 15 Post-test SQ: Net Promoter Score
  16. 16. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Questionnaires that provide rankings to compare your results against the results from other similar apps  SUPR-Q (Sauro)  WAMMI (www.wammi.org) (SUMI successor)  ACSI (www.TheACSI.org): particularly interesting form government websites  OpinionLab (www.OpinionLab.com) This project has been funded with support from the European Commission under the Lifelong Learning Programme 16 Post-test SQ: Rankings
  17. 17. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 17 SQ: Rankings. SUPR-Q (4 sub-scales) NPS
  18. 18. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 18 Post-test SQ: Rankings. SUPR-Q.  SUPR-Q Score: Sum all items + ½ item 10
  19. 19. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 19 Post-test SQ: Rankings. SUPR-Q.  Besides the values for each factor, SUPR-Q is backed on a proprietary DB that allows to generate a ranking of percentiles  E.g. your app belongs to percentil 75 regarding usability, what means, it is among the best 25%.  The SUPR-Q usability factor has a strong correlation with a SUS score, r = .96. p < .001, meaning just four questions account for 93 percent of the variation in SUS (.96 squared). These questions are a good substitute for SUS on websites  Validity and reliability: http://www.suprq.com/
  20. 20. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 20 Post-test SQ: Rankings. WAMMI  http://www.wammi.com/samples/index.html  Results are divided into five areas: attractiveness, controllability, efficiency, helpfulness and Learnability, plus an overall usability score.  The scores are standardized (from comparison to their reference database), so 50 is medium and 100 is perfect.
  21. 21. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES This project has been funded with support from the European Commission under the Lifelong Learning Programme 21 Post-test SQ: Rankings. WAMMI  Sample graphic showing how a give website positions in reference to average scores in each axis
  22. 22. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Circle five words that describe what you think about this design:   What are the three things you like best and least about the Web site?  _____________________________________________  ______________________________________________  ______________________________________________ This project has been funded with support from the European Commission under the Lifelong Learning Programme 22 Post-test SQ: proprietary (OHIM 1/2) Cute Stable Responsive Friendly Helpful Reputable Approachable Reliable Cluttered Good Confident Trustworthy Current and cool Service oriented Boring Easy to use Confusing Comfortable Annoying Informative Out-of-date High-tech Sensitive Secure Straightforward Amateurish
  23. 23. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  If you could make one significant change to this Web site, what change would you make?  (Ask the participant if he is using/knows other IP websites) How do you find the site in comparison to other IP web sites?  Do you feel this site is current? Why?  If you were to describe this site to a colleague in a sentence or two, what would you say?  Do you use the current OHIM web site? If Yes then  Do you think that the new OHIM web site clarity of structure is: Better, Same As, Worse or Don’t know, than the current one?  Do you have any other questions or comments about the Web site or your experiences with it? This project has been funded with support from the European Commission under the Lifelong Learning Programme 23 Post-test SQ: proprietary (OHIM 2/2)
  24. 24. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Expectation Rating  After Scenario Questionnaire (ASQ)  Awareness-Usefulness gap This project has been funded with support from the European Commission under the Lifelong Learning Programme 24 Task level (Post-Scenario) SQ
  25. 25. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Pre-task: How easy/difficult do you thing that the following task is going to be?  Post-task: How easy/difficult has been for you to carry out this task? This project has been funded with support from the European Commission under the Lifelong Learning Programme 25 Post-Scenario SQ: Expectation Rating
  26. 26. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Three post-task questions:  I feel satisfied with the easiness with which I have completed the task.  I feel satisfied with the time that it has taken me to complete this task.  I feel satisfied with the support information (online help, messages, documentation, and so on) that I have had available while I was completing this task (only if your system offers online help) This project has been funded with support from the European Commission under the Lifelong Learning Programme 26 Post-Scenario SQ: ASQ
  27. 27. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Two post-task questions:  ‘‘Were you aware of this functionality prior to this study? (yes or no)  On a 1 to 5 scale, how useful is this functionality to you?  Calculation:  Convert the likert item to a binary scale (e.g. 4 or 5 useful, rest not useful) and draw the graph.  Differences between awareness and usefulness indicate which tasks to promote (e.g. by redesigning the visual hierarchy) in your app. This project has been funded with support from the European Commission under the Lifelong Learning Programme 27 Post-Scenario SQ: Awar-Usefuln Gap
  28. 28. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Graph sample: This project has been funded with support from the European Commission under the Lifelong Learning Programme 28 Post-Scenario SQ: Awar-Usefuln Gap
  29. 29. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  More questionnaires are appearing by the day in the literature. Many of them have been deemed necessary to cover the idiosyncrasy of new platforms/domains  E.g. MPUQ: Mobile Phone Usability Questionnaire This project has been funded with support from the European Commission under the Lifelong Learning Programme 29 SQ: Other questionnaires Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ)
  30. 30. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Can be post-test or post-task  Reasons why the users are promoters or detractors of your product  Insights from users gathered from field studies  Complaints about a product sent to the customer service  Why the task was difficult to complete  …  The way of processing this kind of questions is to turn the open comments into categories, quantify them and analyze them.  http://www.measuringusability.com/blog/quantify-comments.php This project has been funded with support from the European Commission under the Lifelong Learning Programme 30 SQ: Open Questions
  31. 31. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  All the questionnaires must be validated with respect to their reliability, validity and utility. Once validated, they still need to be revalidated when the context of application varies. This project has been funded with support from the European Commission under the Lifelong Learning Programme 31 SQ: Validation
  32. 32. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES  Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.  [Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2  [Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in Computing Systems. New York: ACM, pp. 213–218.  [Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.  [Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 169–177.  [Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.  [Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction with a Web Site.  [Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman. This project has been funded with support from the European Commission under the Lifelong Learning Programme 32 References
  33. 33. Leonardo da Vinci Partnerships Project GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES These slides are made available under the license Creative Commons Attribution-NonCommercial-NoDerivs CC BY- NC-ND. More information about license: http://creativecommons.org/licenses/by-nc-nd/3.0/. These slides were created under Leonardo da Vinci Partnerships Project 2012-1-PL1-LEO04-28181 GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES (http://usability- accessibility.org/). This project has been funded with support from the European Commission under the Lifelong Learning Programme 33 Attributions

×