UX Smackdown! Usability Testing Techniques in the Ring


Published on

Presentation from 2012 SXSW Interactive

Published in: Technology, Design

UX Smackdown! Usability Testing Techniques in the Ring

  1. 1. UX SMACKDOWN!Usability Testing Techniques in the Ring #UXSmackdown
  2. 2. Aviva Rosenstein Rebecca SherrillSr Mgr, User Research UX LeadSalesforce.com Beaconfire.com@uxresearch @beccasKyle Soucy Michael SummersFounding Principal Global User Experience, PaypalUsableInterface.com UserResearch.com@kylesoucy @ez2useJohn WhalenUX Lead & FounderBrilliant Experience.com@johnwhalen #uxSmackdown #UXSmackdown
  3. 3. ROUND 1 Focus Groups Vs. Fie ld Res earch #UXSmackdown
  4. 4. Focus Groups #UXSmackdown
  5. 5. ?Why are we proponents of focus groups? #UXSmackdown
  6. 6. 1. Focus groups do not have to be this...Photo  copyright  AMC #UXSmackdown
  7. 7. 1. Focus groups do not have to be this...Photo  copyright  AMC #UXSmackdown
  8. 8. 2. You don’t have to dress up like a $#%^&*@$!!#fieldstudyninja to get good data. #UXSmackdown
  9. 9. 3a. Focus groups can be “in situ” #UXSmackdown
  10. 10. 3b. Focus groups can be remote #UXSmackdown
  11. 11. 4. Focus groups can have individual assignments beforegroup discussion to produce better data. #UXSmackdown
  12. 12. Drip Coffee?! No M&M’s? I must make a Totally Starbucks run! unacceptable! I can’t work under these conditions! Diet Pepsi is ...and only 12 no substitute for menu choices?!! Diet Coke! Who has the Zagats!Finally, think of what you’d miss behind the glass... Check out skydeckcartoons.com for more. #UXSmackdown
  13. 13. SummaryReasons for conducting Focus Groups:1. Focus groups can be “small groupworkshops,” not “focus groups”.2. You don’t have to dress up like a fieldstudies ninja.3. Focus groups can be “in situ” orremote.4. Focus groups can haveindividual assignments beforegroup discussion to improve dataquality. #UXSmackdown
  14. 14. Field Research #UXSmackdown
  15. 15. “What  people  say,  what  people  do,  and  what  they  say  they  do  are  entirely  different  things.”   —Margaret Mead #UXSmackdown
  16. 16. Focus groups aren’t enough:If you want to find out: • What users want & need • What they’re doing now • How they feel about it • What you can improve
  17. 17. Immerse yourself in the user’s world #UXSmackdown
  18. 18. ROLE: Business OwnerTASK: Approve visual design directionCONTEXT: Waterfall dev process. Supervises multipleproduct managers. Frequently Approve visual design direction TASK: mobile; uses iPhone.CHARACTERISTICS: Short aention span.Under significant time pressure. PM IDFocuses on visuals and metrics. BOCONTENT CRITERIA: Brief, clear presentation in commonformats consumable on mobile devices Dev Mgr VzD #UXSmackdown
  19. 19. Stakeholder participation in analysis = everyone owns the study findingsPhoto  by  Jane  Mejdahl,    used  under  CC  BY-­‐SA  2.0.   19
  20. 20. Focus group falsehoods: • Human memory fail • Suggestibility • Social norms • Self-enhancement bias • Self serving facilitators #UXSmackdown
  21. 21. Imagine you’re askingabout #UXSmackdown
  22. 22. Oh! How do you Thanklike the gift I you! made you? (Ugh, it’s horrible.) #UXSmackdown
  23. 23. How do you Oh!like the app Great!we built you? (Yuck, it’s useless.) #UXSmackdown
  24. 24. Our study found thatfocus group participantsare more generous,more law abiding, andfloss more often. #UXSmackdown
  25. 25. SummaryField studies let you: • Understand user needs • Uncover tacit knowledge that isn’t consciously available • Find deltas between how users think they do something and their performance • Develop actionable insights • Identify opportunities for innovation #UXSmackdown
  26. 26. ROUND 1Make Some Noise! #UXSmackdown
  27. 27. ROUND 1 Questions #UXSmackdown
  28. 28. ROUND 2 Eye Tr acking Vs. mote T estingUn moderated Re #UXSmackdown
  29. 29. Eye Tracking #UXSmackdown
  30. 30. #UXSmackdown
  31. 31. UnmoderatedRemote Usability Testing: ? #UXSmackdown
  32. 32. #UXSmackdown
  33. 33. Unmoderated Usability Testing #UXSmackdown
  34. 34. Benefits #UXSmackdown
  35. 35. Test hundreds of people simultaneously while keepingthem in their own natural environment. #UXSmackdown
  36. 36. Test multiple websites simultaneously. #UXSmackdown
  37. 37. Reduced costs... depending on the tool you use. #UXSmackdown
  38. 38. Great way to plant the seed of UCD methodologiesand introduce usability testing into a company. UX #UXSmackdown
  39. 39. Fewer logistics to manage and no need to worry aboutno-shows and getting last-minute replacements. #UXSmackdown
  40. 40. Fast Results #UXSmackdown
  41. 41. In-context Feedback “...I’m only offered a search by model number. How ridiculous.” “...I would like to see the products availability and delivery/ shipping information listed.”Image source: Loop11 #UXSmackdown
  42. 42. The danger... I wonderhow a real customerwould actually use this website? Panelist #UXSmackdown
  43. 43. The danger... BLOCKEDBLOCKED #UXSmackdown
  44. 44. There are no shortcutsto conductinggood research. #UXSmackdown
  45. 45. YO U B E THE JU DG E!Good? Evil? #UXSmackdown Photo  copyright  New  Line  Cinema
  46. 46. ROUND 2Make Some Noise! #UXSmackdown
  47. 47. ROUND 2 Questions #UXSmackdown
  48. 48. ROUND 3 elity Proto typesH igh Fid Vs. Itera tive T esting Rapid #UXSmackdown
  49. 49. High Fidelity Prototypes #UXSmackdown
  50. 50. Why #HighFidelityRules... ...and #IterativeTestingFailsPhoto  copyright  Apple #UXSmackdown
  51. 51. Weight 2,061 2,954 Drive Train Rear Rear HP 350 320 Torque 292 273 Top Speed 156 171 0-60 5.3 4.9Why High-Fidelity Prototypes?1. Everything looks great on paper... #UXSmackdown
  52. 52. ...but sometimes the final experience is not the same. #UXSmackdown
  53. 53. ...but sometimes the final experience is not the same. #UXSmackdown
  54. 54. In today’s swipe and swish world you need the full experience. #UXSmackdown
  55. 55. Pause Suppli Perfor HIGHSCORE 0 es mance SCORE 02a. Iterative testing can’t tell you if you’re creating flow. #UXSmackdown
  56. 56. 2a: Iterative testing can’t tell you if you’re creating flow. #UXSmackdown
  57. 57. You should see the front. blue fade yellow fade2b: Iterative testing can’t tell you if you’re creatingpersuasive design. #UXSmackdown
  58. 58. 2b: Iterative testing can’t tell you if you’re creatingpersuasive design. #UXSmackdown
  59. 59. 3. High fidelity prototypes identify the full implications of“stakeholder design” #UXSmackdown
  60. 60. #UXSmackdown
  61. 61. John, couldn’t you iteratively testsomething simple, like a login without a high fidelity prototype?! #UXSmackdown
  62. 62. John, couldn’t you iteratively testsomething simple, like a login without a high fidelity prototype?! YES #UXSmackdown
  63. 63. ...but my argument is that increasingly we need thefull experience to understand a design’s success. #UXSmackdown
  64. 64. Final note: I too find great value in iterative design in theright circumstance! #UXSmackdown
  65. 65. SummaryReasons for High-FidelityPrototypes:1. In today’s swipe and swish worldyou need the full experience.2. You can’t really “iteratively test”,persuasive or multi-channel designs.3. High-fidelity prototypes identify thefull implications of“stakeholder design”. #UXSmackdown
  66. 66. High Fidelity Rapid Iterative Prototype vs. Usability Testing Testing #UXSmackdown
  67. 67. Low-Fidelity Prototyping viously Ad d pre rk N etwo ity Activ #UXSmackdown
  68. 68. RITE Usability Testing Test! Rev ised prob as s lem oon is id as a enti fied > Revise Revise Prototype! Prototype! Test! #UXSmackdown
  69. 69. How do you makedesign decisionsbased on only testing1 person? #UXSmackdown
  70. 70. Modified RITE Process ent ! Revise ! Test! Debrief! Pres ngs type(All users) Findi P roto Test! Debrief! (All users) #UXSmackdown
  71. 71. Benefits of choosing low-fidelity prototypes #UXSmackdown
  72. 72. Getting end user feedbackearlier in the design process. #UXSmackdown
  73. 73. It’s easier to throw awaypaper than code. #UXSmackdown
  74. 74. Provides the opportunity tothink outside the box. #UXSmackdown
  75. 75. It gets the entire team involved. #UXSmackdown
  76. 76. Participants are less apt to givenegative feedback on designs thathave a "finished" appearance. Oh, it’s lovely! I wouldn’t change anything. #UXSmackdown
  77. 77. i FT W! n Lo w-FRIT E o #UXSmackdown
  78. 78. ROUND 3Make Some Noise! #UXSmackdown
  79. 79. ROUND 3 Questions #UXSmackdown
  80. 80. Focus G roups Vs. Field R racking esearc Eye T h Vs. ed Remot e TestingU nmoderat High Fidelity Prototypes Vs. Rapid Iterative Testing #UXSmackdown
  81. 81. #UXSmackdown