Why keep it to yourself? Teaching everyone on the team to do usability testing


Published on

Techniques for teaching people on your development team how to gather data from users.

Published in: Design

Why keep it to yourself? Teaching everyone on the team to do usability testing

  1. 1. 1 Why keep it to yourself? ? Getting everyone on the team to do usability testing Dana Chisnell IA Summit 2010
  2. 2. 2 Influence = Data + Gut
  3. 3. 3 Influence = Data < Gut
  4. 4. 4 Influence = Data x Gut
  5. 5. 5 Influence = Data^Gut
  6. 6. 6 Throughput is a problem.
  7. 7. 7 Influencing design decisions
  8. 8. 8
  9. 9. 9 ? Dev IA IX Dev D D U
  10. 10. 10 ? U
  11. 11. 11 Dev D IX D IA Dev D
  12. 12. 12 Not either/or
  13. 13. 13 Do it together
  14. 14. 14 U D D E V D DEV U I A D U
  15. 15. 15 Yay! DEV DEV IA D IX D U
  16. 16. 16 Influencing design decisions Give up control (you don’t have it, anyway) Get the team to value data over gut Research by lots of amateurs produces better results Otherwise, you’ll burn out Coach and advise instead Work on more interesting, harder questions
  17. 17. 17 Improving the Gut Yay! DEV DEV IA D IX D U
  18. 18. 18 Tasks Moderating Analysis
  19. 19. 19 Tasks: Seeing through the users’ eyes Help each team member think of a user scenario they care about Act out the scenario as the user
  20. 20. 20 App: Church social network ✤ Roles: ✤ Senior Pastor ✤ Social Networking Pastor ✤ Volunteer ✤ Lay Leader ✤ Member ✤ First time visitor ✤ Administrative Assistant
  21. 21. 21 ✤ App: Organizing students to help homebound people vote ✤ User roles: ✤ Student ✤ Voter ✤ Organizer ✤ Trainer ✤ Scheduler
  22. 22. 22 What happened?
  23. 23. 23 Task scenarios set the scene, give context You want to create realistic and relatable stories
  24. 24. 24 Moderating: Active presence Role play practice (in dry runs) Video of the moderator
  25. 25. 25 Moderating, Impartial, unbiased observing not training No teaching! Listen and watch Ask open-ended questions: Why? How? What? Correct and train at the end
  26. 26. 26 Who should moderate? Quick learner Develops rapport, Excellent memory Good listener empathetic
  27. 27. 27 Flight attendant Ensuring the safety and comfort of the participant Joan Dorsey, American Airlines: http://aviationblog.dallasnews.com/archives/aviation-historytrivia/ Sportscaster Echoing with play-by-play Maximizing flow of information to observers John Madden, http://www.forbes.com/2008/01/28/television-sports-madden-biz-sports-cx_lh_0128broadcasters_slide_3.html? thisSpeed=15000 Scientist Planning, managing data, producing reports Marie Curie, Smithsonian Institution http://photography.si.edu/SearchImage.aspx?t=5&id=3523&q=SIL14-C6-05
  28. 28. 28 Moderators: Run the session Observers: Note behaviors and quotes Participant: Do what you normally would do, try to think aloud
  29. 29. 29 What to look for Hesitating Comments Questions Body language Behaviors
  30. 30. 30 Technique Think aloud - ask: Review at end - ask: “Tell me what you’re “Walk me through what you doing” did” “Tell me what you’re “How’d that go?” Use the thinking” ballot as a guide for the discussion “What was confusing or frustrating?”
  31. 31. 31 Review ✤ Participant: What was it like? ✤ Observers: What did you see? ✤ Moderators: What questions do you have?
  32. 32. 32 Mid-session reminders “You are not being Meanwhile, you track time and tested” tasks remaining in the session “That’s useful Don’t be afraid to move on if feedback, thank something is taking too long you” Decide ahead what is low priority “Please tell me what or optional you’re thinking”
  33. 33. 33 Narration Don’t be afraid to Examples: interact “I see you just clicked on the Zit button.” Say what you are “Sorry, I didn’t catch that. Would you say observing - don’t that again?” interpret “That really, really stinks? Could you say what about it stinks?”
  34. 34. 34 Maximizing data ✤ If the participant says “hmmm” or “oops” or “I wonder...” Say: “What questions do you have right now?” ✤ If the participant is silent for 10 - 20 seconds (count!) Say: “What are you thinking?” ✤ If the participant stops because she thinks she’s done or she’s stuck - Summarize what you saw her do - Ask what she will do next
  35. 35. 35 Analysis: Structured discussion Tell stories KJ - to set priorities Guess the reason - to exercise inference-making Observation-inference-opinion-theory
  36. 36. 36 KJ Analysis reach consensus from subjective data similar to affinity diagramming invented by Jiro Kawakita objective, quick 8 simple steps
  37. 37. 37 1. Focus question What needs to be fixed in Product X to improve the user experience? (observations, data) What obstacles do teams face in implementing UE practices? (opinion)
  38. 38. 38 2. Organize the group Call together everyone concerned For user research, only those who observed Typically takes an hour
  39. 39. 39 3. Put opinions or data on notes For a usability test, ask for observations (not inferences, not opinion) No discussion
  40. 40. 40 4. Put notes on a wall Random Read others’ Add items No discussion
  41. 41. 41 5. Group similar items In another part of the room Start with 2 items that seem like they belong together Place them together, one below the other Move all stickies Review and move among groups Every item in a group No discussion
  42. 42. 42 6. Name each group Use a different color Each person gives each group a name Names must be noun clusters Split groups Join groups Everyone must give every group a name No discussion
  43. 43. 43 7. Vote for the most important groups Democratic sharing of opinions Independent of influence or coercion Each person writes their top 3 Rank the choices from most to least important Record votes on the group sticky No discussion
  44. 44. 44 8. Rank the groups Pull notes with votes Order by the number of votes Read off the groups Discuss combining groups Agreement must be unanimous Add combined groups’ votes together Stop at 3-5 clear top priorities
  45. 45. 45
  46. 46. 46 Observations to direction Quantifying subjective data in 4 easy steps
  47. 47. 47 Observations Sources: What you saw usability testing What you heard user research sales feedback support calls and emails training
  48. 48. 48 Inferences Judgements Conclusions Guesses Intuition
  49. 49. 49 Opinions Review the inferences What are the causes? How likely is this inference to be the cause? How often did the observation happen? Are there any patterns in what kinds of users had issues?
  50. 50. 50 Direction What’s the evidence for a design change? What does the strength of the cause suggest about a solution? Test theories
  51. 51. 51 Resistance from the team I don’t have time It’s not my job I don’t know how What if I screw up
  52. 52. 52 Resistance is futile.
  53. 53. 53 Resistance from the team Exchange time in meetings for time with users It will make you better at your job I’ll help you learn You won’t; it doesn’t matter
  54. 54. 54 Stop pouting. Yep, you’re qualified and they aren’t. Go peddle your fish. Haven’t you wanted more visibility?
  55. 55. 55 Improving the Gut A lot of sloppy data is better than a little excellent data Trust your team to be delegated to Shift your mindset from your butt to their Gut
  56. 56. 56 -+ Desktop X? Users Data ? Y? U Recommendations D Direction This?
  57. 57. 57 Yay! DEV DEV IA D IX D U Or this?
  58. 58. 58 Why keep it to yourself? IX Dev IA D U Dev D
  59. 59. 59 Where to learn more Dana’s blog: http:// usabilitytestinghowto.blogspot.com/ Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting
  60. 60. 60 Dana Chisnell dana@usabilityworks.net www.usabilityworks.net 415.519.1148