Successfully reported this slideshow.
Your SlideShare is downloading. ×

Advanced usability testing - moderating

Ad

Advanced Usability Testing
Telle Zeiler & Rebecca Destello
Moderating

Ad

Telle Zeiler
Zeiler & Destello | Advanced Usability Testing - Moderating
● Program Management (1999 - 2011)
○ Microsoft ve...

Ad

Rebecca Destello
Zeiler & Destello | Advanced Usability Testing - Moderating
Front-end Web Developer
(1998 - 2011)
Bachelo...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Check these out next

1 of 42 Ad
1 of 42 Ad

Advanced usability testing - moderating

Download to read offline

Now that you know how to plan for and construct bullet-proof usability script, take your experience to the next level - learn how to be an effective moderator!

Now that you know how to plan for and construct bullet-proof usability script, take your experience to the next level - learn how to be an effective moderator!

Advertisement
Advertisement

More Related Content

Advertisement
Advertisement

Advanced usability testing - moderating

  1. 1. Advanced Usability Testing Telle Zeiler & Rebecca Destello Moderating
  2. 2. Telle Zeiler Zeiler & Destello | Advanced Usability Testing - Moderating ● Program Management (1999 - 2011) ○ Microsoft vendor ○ Seattle Times ○ Nordstrom.com ● M.S. in HCDE, concentration in user-centered design (2011) ● UX Researcher, support lecturer, UX mentor (2011 - present)
  3. 3. Rebecca Destello Zeiler & Destello | Advanced Usability Testing - Moderating Front-end Web Developer (1998 - 2011) Bachelor of Arts @ UW Art (2008) Researcher / Mentor / Teacher (2011 - now) Master of Science @ UW HCDE (2011) The Jakob Nielsen Experience
  4. 4. What we’ll cover today Moderating a research study ● What works ● What to avoid ● How to work with the participant during the study ● Taking notes ● Unpacking data Zeiler & Destello | Advanced Usability Testing - Moderating
  5. 5. Zeiler & Destello | Advanced Usability Testing - Moderating Q. What makes a good moderator?
  6. 6. Zeiler & Destello | Advanced Usability Testing - Moderating Q. What makes a great moderator?
  7. 7. What makes a great moderator? ● Uses good timing for prompting ● Knowing when to go deeper ● Knowing how to pull participant back from tangents ● Coming into study with the biz. goals in mind and being able to explore when relevant/have time ● Getting data from difficult participants ● “Knows when to hold ‘em, knows when to fold ‘em” Zeiler & Destello | Advanced Usability Testing - Moderating
  8. 8. On Listening The first rule of interviewing is that if the respondent has something to say, the respondent must be able to say it. If you find yourself talking over the respondent, interruption or holding the floor while the respondent tries to interrupt, something is going wrong in the interview. You might want to withdraw some of your attention from the respondent for a moment or two to figure out why you are competing for the floor. But whether you figure it out or not, you ought to stop doing it. It is easy to intrude in an interview. You can interrupt the respondent. You can finish the respondent’s sentences. You can offer your associations to what the respondent is saying. You can suggest explanations for observations about which the respondent is perplexed; for example, if the respondent shrugs and says, “ I don’t know why he said that,” you could propose, “well maybe he was trying to defend himself.” You can insist on completing your questions even if the respondent has already started to answer. You can hop from issue to issue following your own train of thought rather than the respondent’s. With any and all of these, don’t do it. - Weiss, Learning from Strangers: Interviewing (p78) All this to say… Don’t interrupt the participant! Zeiler & Destello | Advanced Usability Testing - Moderating
  9. 9. The role of the Moderator Zeiler & Destello | Advanced Usability Testing - Moderating
  10. 10. Moderating The most important thing to remember about moderating is this: • Being in a study is stressful for a participant • The participant’s comfort level and state of mind are paramount • When people are stressed, their behavior and memory are impaired Zeiler & Destello | Advanced Usability Testing - Moderating
  11. 11. Why is moderating challenging? • Goal is unbiased feedback, but your presence causes bias • Participants need to be comfortable, but the study situation is often unsettling • You need to know what the participant is thinking, but you can’t read their minds • It’s not like a normal conversation, you don’t • Answer questions (usually) • Interrupt • Take an equal role in conversation Zeiler & Destello | Advanced Usability Testing - Moderating
  12. 12. Golden rule: The newer you are to moderating, the less you should talk. Guidelines for Moderating • Keep yourself “small” in relationship to the participant • Be as neutral as possible • Be professional • Err on the side of taking care of the participant Zeiler & Destello | Advanced Usability Testing - Moderating
  13. 13. How to keep them talking: Prompting Zeiler & Destello | Advanced Usability Testing - Moderating
  14. 14. Prompting guidelines When prompting, you want to get the user to elaborate on something that is unclear without ● Introducing bias ● Influencing the user’s behavior ● Taking over as the expert or primary speaker ● Unnerving the user through implication that she did something wrong Q. What are some ways a moderator could introduce bias? Zeiler & Destello | Advanced Usability Testing - Moderating
  15. 15. Prompting techniques Provide Acknowledgment Tokens at natural intervals • “Mm hmm”, or “Uh huh” Focus on questions • What did you expect to happen? Repeat their word or phrase back to them as a question • They say: That was confusing • You say: “confusing…?” Make the alternative explicit • Was this task easy or difficult? Zeiler & Destello | Advanced Usability Testing - Moderating
  16. 16. Handling questions Participants sometimes ask questions - try to turn their question into a question. How would you respond to the following: ● Is that right? Did I do it right? ● Am I supposed to keep going? I’ve been looking for a long time. ● What should I do now? ● Should I click here? ● What is this? Zeiler & Destello | Advanced Usability Testing - Moderating
  17. 17. When and how to intervene In general, it’s preferable to let participants decide how much time to work on a task. Some exceptions ● If they are getting very frustrated ● If you feel that you’ve got data on the task at hand and you want to keep going and get to other tasks Zeiler & Destello | Advanced Usability Testing - Moderating
  18. 18. Prompting: word of warning Remember, people cannot reliably reflect on their own cognition. Questions like... • “What were you thinking when…?” • “Why do you think you…?” ...often result in bad data. Participants will try to come up with a believable rationale while being unaware of the true cause for their actions. Zeiler & Destello | Advanced Usability Testing - Moderating
  19. 19. Reading non-verbal communication Zeiler & Destello | Advanced Usability Testing - Moderating
  20. 20. Nonverbal communication The process of communication through sending and receiving wordless (mostly visual) cues between people. • Posture. General body structure held in a certain way. • Bearing. Orientation of the body. • Body Movement. Movement over time. • Stance. The way a person stands. • Gestures. Expressions using arms, hands and legs. • Facial Expressions. Expressions using face. • Eyes. Eye movements. • Touch. Touch and personal space. • Voice. Tone and pitch of voice as well as speed of speech. Zeiler & Destello | Advanced Usability Testing - Moderating
  21. 21. Let’s critique Zeiler & Destello | Advanced Usability Testing - Moderating
  22. 22. Zeiler & Destello | Advanced Usability Testing - Moderating Moderator: ● What prompting techniques does she use? ● How does she help the participant feel comfortable? ● How does she make sure she is getting good, unbiased data from the participant? Participant: ● Was anything she said surprising or unexpected? ● What did she say or do that required the moderator to bring her back to the task at hand? Usability session role play critique
  23. 23. Let’s debrief Zeiler & Destello | Advanced Usability Testing - Moderating
  24. 24. Collecting data Zeiler & Destello | Advanced Usability Testing - Moderating
  25. 25. Data collection • Know what you are looking for • Define success criteria • Determine how structured to be up front Zeiler & Destello | Advanced Usability Testing - Moderating
  26. 26. Data collection During the usability study, you’ll need to take good, complete, unbiased notes. A few tips: • The more you capture during the study, the less time you have to spend reviewing tapes • Get organized - use codes and shortcuts • If you’re doing data collection - do a quick “clean up” after each session Zeiler & Destello | Advanced Usability Testing - Moderating
  27. 27. Data collection Do capture ● Observations - what you see the participant do ● Quotes - what you hear the participant say ● Questions - actions or statements that were unclear or that you didn’t capture completely Don’t capture ● Inferences - why you think they are doing what they are doing. ● Broad generalizations that do not contain description (e.g. “She had a lot of trouble finding the link”) Zeiler & Destello | Advanced Usability Testing - Moderating
  28. 28. Data collection tricks Use codes to note issues that you’re tracking. Example: ● F = task failure ● E = error message ● H = asks for help ● P = moderator prompted ● U = gave up Zeiler & Destello | Advanced Usability Testing - Moderating
  29. 29. Data collection tricks Use checkboxes in your notetaking script to check off the things you’re looking for. Example: Zeiler & Destello | Advanced Usability Testing - Moderating
  30. 30. Data analysis Convert data you collect into an easy-to-analyze format Example: Zeiler & Destello | Advanced Usability Testing - Moderating
  31. 31. Data collection tricks Use sticky notes or Trello.com to collect observations, quotes, and questions. Example: Zeiler & Destello | Advanced Usability Testing - Moderating
  32. 32. Putting it all together Collecting data during a study • Setup the script to make it easier to capture data • Use codes to capture data during the study • Convert data into an easy to analyze format Zeiler & Destello | Advanced Usability Testing - Moderating
  33. 33. Unpacking data: exercise Zeiler & Destello | Advanced Usability Testing - Moderating
  34. 34. In groups of 2 • Participant A, Participant B • Practice run through your scenario and note potential issues along the way • Run scenario on your partner – take notes! • Switch • We’ll report out as a class Zeiler & Destello | Advanced Usability Testing - Moderating Let’s try it out!
  35. 35. • Task Success: The ability of participants to successfully complete the task without assistance or indicating they would abandon the task. • Error on Task: Identify the number of participants who performed an action that created unexpected results. • Confusion on Task: Identify the number of participants who either verbalized confusion, or who re-entered information to get proper results Zeiler & Destello | Advanced Usability Testing - Moderating Things to look for
  36. 36. Scenario: You and your significant other are spending a long weekend in San Francisco March 27th – 30th and want to book a city tour one of the days you are there. Task: Using your smartphone book a city tour from Travelocity.com. When you get to the checkout page you have finished the task. At the end ask: On a scale of 1-5 with 1 being very difficult and 5 being very easy, how easy or difficult was it to complete that task? Zeiler & Destello | Advanced Usability Testing - Moderating Participant A
  37. 37. Scenario: You and your significant other are spending a long weekend in San Francisco and you want to rent a compact car to get around the city. You fly into SFO March 23rd at 10am and leave from the same airport on March 31st at 5pm. Task: Use your smartphone to reserve a car on Travelocity.com. When you get to the checkout page you have finished the task. At the end ask: On a scale of 1-5 with 1 being very difficult and 5 being very easy, how easy or difficult was it to complete that task? Zeiler & Destello | Advanced Usability Testing - Moderating Participant B
  38. 38. Let’s unpack the data together • Task Success • Error on Task • Confusion on Task • Quotes • Post task ratings Zeiler & Destello | Advanced Usability Testing - Moderating Analysis!
  39. 39. Some words of advice Zeiler & Destello | Advanced Usability Testing - Moderating
  40. 40. ● Be user focused - as a researcher, your users are your stakeholders - learn their needs ● Be flexible - you win some, you lose some ● Be fast - find ways to speed up your study and report out ● Be mindful - choose your words wisely ● Be nosey - get involved in product discussions and proactively look for ways to inform the design process ● Be a partner - you are the design team’s partner Work well with others Zeiler & Destello | Advanced Usability Testing - Moderating
  41. 41. ● Know your stakeholder’s expectations and particular needs for consuming your research findings ● Combine and hack research methods to fit your needs ● Use a matrix figure out the # of participants you need ● Use a spreadsheet to analyze findings and look for patterns ● Find easy ways to code during your study ● Build buy-in from the start More: Assessing interview techniques, Chauncey Wilson: http://bit.ly/2qkKZxI Research tips Zeiler & Destello | Advanced Usability Testing - Moderating
  42. 42. Questions? TelleZeiler@gmail.com RebeccaDestello@gmail.com Zeiler & Destello | Advanced Usability Testing - Moderating

×