Summary of Usability Testing Methods: Subjective Measures Part I and II by Joe Dumas
•Download as PPTX, PDF•
1 like•2,191 views
This document discusses methods for creating effective subjective measures for usability testing. It covers three main topics: 1. Sources of distortion in surveys from participants, interviewers, and the task situation that can influence responses. 2. Common question formats used to measure attitudes and opinions in usability testing, including Likert scales, semantic differentials, and scales of measurement. 3. Considerations for usability practitioners using questions as subjective measures, such as interpreting how questions will be understood and using multiple measures to evaluate usability.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Summary of Usability Testing Methods: Subjective Measures Part I and II by Joe Dumas
Similar to Summary of Usability Testing Methods: Subjective Measures Part I and II by Joe Dumas (20)
Recently uploaded
Recently uploaded (20)
Summary of Usability Testing Methods: Subjective Measures Part I and II by Joe Dumas
- 1. UsabilityTestingMethods:Subjective MeasuresPart Iand II byJoeDumas* Summary presented by Meeyong Rao HCDE 517 -Usability Testing Winter 2014 *From: The Common Ground, The Newsletter of Usability Professionals (1998)
- 2. Part I – Creating Effective Questions and Answers
- 3. Three Sources of Distortion in Surveys 1. Participants • Biases: yea-saying, central tendency, leniency, social desirability
- 4. ThreeSources of Distortion in Surveys Cont. 2. Interviewer • Training • Experience
- 5. 3. Of the task situation itself • Method of administration: Self-administered vs. F2F • Effect of question wording • Memory limitation: forgetting & telescoping ThreeSources of Distortion in Surveys Cont.
- 6. The Choice of Answer Options • Questions about frequency of behavior are influenced by the distribution of pre-coded answer options • Example: if a participant is asked about how frequently s/he watches TV… A low range from ½ hour to 2 and ½ hours reported lower averages of watching TV than a higher range including 2 and ½ hours to 4 and ½ hours
- 7. Part II – Measuring Attitudes and Opinions
- 8. MostUsedQuestionFormatforOpinionsof Usability:LikertScale Q. Common Ground is a quality newsletter. Strongly Agree Strongly Disagree 1 2 3 4 5 • #’s represent equal intervals and how much one is preferred over another • 5, 7, or 9-point scale (provides true midpoint); little value in having more than 9 or 10 • If measuring ONLY “the intensity of conviction” then do not include neutral point on scale
- 9. Scale EndPoint Labels Very Difficult Very Easy 1 2 3 4 5 • Scale doesn’t measure one single dimension • Measures two dimensions: difficulty and easy • Saying a product is “difficult” to use is not the same as saying it is “not at all easy” • These labels activate two different cognitive structures; should activate only one
- 10. FrequentlyUsedFormatforOpinionsof Usability:SemanticDifferential Favorable - - - - - - Unfavorable Unattractive - - - - - - Attractive Pleasant - - - - - - Unpleasant • Respondents check one of the 7 lines between word pairs • Polar opposites • Connotative meaning, Not denotative meaning Ex. School mascot - Husky • Denotative: animal • Connotative: strength/power
- 11. Negative Numbers in Scale Q. Please rate how successful you have been in life. Not at Successful Very Successful -3 -2 -1 0 1 2 3 • Participants give significantly more positive ratings using negative ratings scale compared to positive scale of 1 to 7
- 12. Scales of Measurement: Nominal Q. What is your gender? 1. Male 2. Female • Numbers assigned to answer options do not have meaning except for distinction • Numbers don’t measure anything. Could be letters of the alphabet instead.
- 13. Scales of Measurement: Ordinal Q. When you are considering purchasing a smart phone, please rank the importance of each of the following on your decision. Please use a ranking from 1 to 5, where 1 is most important and 5 is least important. ___ Apps ___ Battery life ___ Quality of camera ___ Operating system ___ Voice control • Numbers represent ordering (or magnitude) of a property of an object • Doesn’t indicate size of the difference
- 14. Scales of Measurement: Interval • Numbers represent equal intervals • Shows how much one is preferred over another Examples: Likert scale and semantic differential
- 15. Scales of Measurement: Ratio Q. How many hours per day do you use an iPhone? ___ Hours • Similar to interval scales except has an absolute zero
- 16. Thoughtsfor Usability Practitioners Using Questions as Subjective Measures 1. Always consider why you are asking a question and how participants might interpret the words/format 2. When you find a question that works, stick with it 3. Use questions to stimulate discussions 4. Never restrict your usability evaluation to one measure, especially a subjective measure
Editor's Notes
- Yea-saying: tendency of participants to want to agree. Central tendency: won’t rate anything extremely high or low. Social desirability: need social approval and will overstate their experience with certain things. i.e., technology, voting, having library cards
- Poorly trained interviewers or inexperienced interviewers can cause participants to distort answers or misinterpret questions.
- Self administeredvs face to face. Advice is for usability testers to find wording that works and stick with it. People forget about events, especially the further away from an event we are. Also telescoping which means if you ask a participant how many times they have used a product over the past 6 weeks, they will include usage that occurred beyond those 6 weeks.
- Measures attitudes. Pronounced lick-ert. Don’t like to have likert scale with more than 9 point scale but NPS is an example of one with 11. Another article this week uses a desirability likert scale. Negative numbers “allow” respondents to give a negative response. Joe says it will result in much more negative feedback than a 1 to 5 scale. Whereas Tullis & Albert say respondents won’t use negative numbers.
- Usually use 3-4 adjective pairs for one dimension. Then average values of all pairs for the dimension. Typically use this evaluative dimension in usability.Want to R/L anchor points so the negative isn’t always the right. To counter L-R bias and to make sure respondent isn’t just always choosing the one on the left because they think it is the “right” one.
- aka categorical, aka discrete. Often times will rotate the order of the response categories to counteract other biases.
- The size of the difference between A and B is not important, only the order of measure/preference. (i.e., one property is greater than another property). Must decide at what point there are too many for a respondent to rank and still care about the ones in the middle. If too many, then use a different methodology (maxdiff).Another example is household income.
- Other examples: if you asked respondents to type in the exact amount of their household income instead of choosing a category of pay like $75,000 to $100,000. You can have a household income of $0.
- You can have a household income