UX SMACKDOWN!Usability Testing Techniques in the Ring                      #UXSmackdown
Aviva Rosenstein             Rebecca SherrillSr Mgr, User Research        UX LeadSalesforce.com               Beaconfire.c...
ROUND 1      Focus   Groups             Vs.      Fie ld Res earch                         #UXSmackdown
Focus Groups               #UXSmackdown
?Why are we proponents of focus groups?                                         #UXSmackdown
1. Focus groups do not have to be this...Photo	  copyright	  AMC                   #UXSmackdown
1. Focus groups do not have to be this...Photo	  copyright	  AMC                   #UXSmackdown
2. You don’t have to dress up like a $#%^&*@$!!#fieldstudyninja to get good data.                                         ...
3a. Focus groups can be “in situ”                                    #UXSmackdown
3b. Focus groups can be remote                                 #UXSmackdown
4. Focus groups can have individual assignments beforegroup discussion to produce better data.                            ...
Drip Coffee?!                     No M&M’s?      I must make a                        Totally      Starbucks run!         ...
SummaryReasons for conducting Focus Groups:1. Focus groups can be “small groupworkshops,” not “focus groups”.2. You don’t ...
Field Research                 #UXSmackdown
“What	  people	  say,	  what	  people	  do,	  and	  what	  they	  say	  they	  do	  are	  entirely	  different	  things.”	 ...
Focus groups aren’t enough:If you want to find out: •   What users want & need •   What they’re doing now •   How they fee...
Immerse yourself in the user’s world                               #UXSmackdown
ROLE: Business OwnerTASK: Approve visual design directionCONTEXT: Waterfall dev process. Supervises multipleproduct manage...
Stakeholder participation in analysis =   everyone owns the study findingsPhoto	  by	  Jane	  Mejdahl,	  	  used	  under	 ...
Focus group falsehoods: •   Human memory fail •   Suggestibility •   Social norms •   Self-enhancement bias •   Self servi...
Imagine you’re askingabout                        #UXSmackdown
Oh! How do you                        Thanklike the gift I                         you! made you?                  (Ugh, i...
How do you             Oh!like the app          Great!we built you?                (Yuck, it’s                 useless.)  ...
Our study found thatfocus group participantsare more generous,more law abiding, andfloss more often.                      ...
SummaryField studies let you: •   Understand user needs •   Uncover tacit knowledge that isn’t     consciously available •...
ROUND 1Make Some Noise!             #UXSmackdown
ROUND 1     Questions                 #UXSmackdown
ROUND 2       Eye Tr acking            Vs.                mote T estingUn moderated Re                        #UXSmackdown
Eye Tracking               #UXSmackdown
#UXSmackdown
UnmoderatedRemote Usability Testing:           ?                       #UXSmackdown
#UXSmackdown
Unmoderated Usability Testing                          #UXSmackdown
Benefits           #UXSmackdown
Test hundreds of people simultaneously while keepingthem in their own natural environment.                                ...
Test multiple websites simultaneously.                                         #UXSmackdown
Reduced costs... depending on the tool you use.                                              #UXSmackdown
Great way to plant the seed of UCD methodologiesand introduce usability testing into a company.                      UX   ...
Fewer logistics to manage and no need to worry aboutno-shows and getting last-minute replacements.                        ...
Fast Results               #UXSmackdown
In-context Feedback                                  “...I’m only offered a                               search by model ...
The danger...    I wonderhow a real customerwould actually use this     website?                          Panelist        ...
The danger...          BLOCKEDBLOCKED                                    #UXSmackdown
There are no  shortcutsto conductinggood research.             #UXSmackdown
YO U B E            THE JU                  DG   E!Good?                  Evil?                                   #UXSmack...
ROUND 2Make Some Noise!             #UXSmackdown
ROUND 2     Questions                 #UXSmackdown
ROUND 3          elity Proto  typesH igh Fid              Vs.         Itera tive T esting  Rapid                         #...
High Fidelity Prototypes                #UXSmackdown
Why #HighFidelityRules...   ...and #IterativeTestingFailsPhoto	  copyright	  Apple                                        ...
Weight                2,061       2,954      Drive Train           Rear        Rear      HP                    350        ...
...but sometimes the final experience is not the same.                                                 #UXSmackdown
...but sometimes the final experience is not the same.                                                 #UXSmackdown
In today’s swipe and swish world you need the full experience.                                                            ...
Pause               Suppli   Perfor         HIGHSCORE       0                es      mance                                ...
2a: Iterative testing can’t tell you if you’re creating flow.                                                     #UXSmack...
You should see the front.        blue fade                               yellow fade2b: Iterative testing can’t tell you i...
2b: Iterative testing can’t tell you if you’re creatingpersuasive design.                                                 ...
3. High fidelity prototypes identify the full implications of“stakeholder design”                                         ...
#UXSmackdown
John,     couldn’t you iteratively testsomething simple, like a login without a       high fidelity prototype?!           ...
John,     couldn’t you iteratively testsomething simple, like a login without a       high fidelity prototype?!           ...
...but my argument is that increasingly we need thefull experience to understand a design’s success.                      ...
Final note: I too find great value in iterative design in theright circumstance!                                          ...
SummaryReasons for High-FidelityPrototypes:1. In today’s swipe and swish worldyou need the full experience.2. You can’t re...
High Fidelity         Rapid Iterative Prototype      vs.     Usability   Testing               Testing                    ...
Low-Fidelity Prototyping                    viously           Ad d pre                                      rk            ...
RITE Usability Testing                   Test!     Rev                                  ised                            pr...
How do you makedesign decisionsbased on only testing1 person?                        #UXSmackdown
Modified RITE Process                             ent ! Revise ! Test!        Debrief!   Pres ngs         type(All users) ...
Benefits of choosing    low-fidelity    prototypes                   #UXSmackdown
Getting end user feedbackearlier in the design process.                           #UXSmackdown
It’s easier to throw awaypaper than code.                            #UXSmackdown
Provides the opportunity tothink outside the box.                          #UXSmackdown
It gets the entire team involved.                           #UXSmackdown
Participants are less apt to givenegative feedback on designs thathave a "finished" appearance.                   Oh, it’s...
i FT W!        n Lo w-FRIT E o                       #UXSmackdown
ROUND 3Make Some Noise!             #UXSmackdown
ROUND 3     Questions                 #UXSmackdown
Focus                                     G   roups                                   Vs.                          Field R...
#UXSmackdown
Upcoming SlideShare
Loading in …5
×

UX Smackdown! Usability Testing Techniques in the Ring

5,458 views
5,103 views

Published on

Presentation from 2012 SXSW Interactive

Published in: Technology, Design

UX Smackdown! Usability Testing Techniques in the Ring

  1. UX SMACKDOWN!Usability Testing Techniques in the Ring #UXSmackdown
  2. Aviva Rosenstein Rebecca SherrillSr Mgr, User Research UX LeadSalesforce.com Beaconfire.com@uxresearch @beccasKyle Soucy Michael SummersFounding Principal Global User Experience, PaypalUsableInterface.com UserResearch.com@kylesoucy @ez2useJohn WhalenUX Lead & FounderBrilliant Experience.com@johnwhalen #uxSmackdown #UXSmackdown
  3. ROUND 1 Focus Groups Vs. Fie ld Res earch #UXSmackdown
  4. Focus Groups #UXSmackdown
  5. ?Why are we proponents of focus groups? #UXSmackdown
  6. 1. Focus groups do not have to be this...Photo  copyright  AMC #UXSmackdown
  7. 1. Focus groups do not have to be this...Photo  copyright  AMC #UXSmackdown
  8. 2. You don’t have to dress up like a $#%^&*@$!!#fieldstudyninja to get good data. #UXSmackdown
  9. 3a. Focus groups can be “in situ” #UXSmackdown
  10. 3b. Focus groups can be remote #UXSmackdown
  11. 4. Focus groups can have individual assignments beforegroup discussion to produce better data. #UXSmackdown
  12. Drip Coffee?! No M&M’s? I must make a Totally Starbucks run! unacceptable! I can’t work under these conditions! Diet Pepsi is ...and only 12 no substitute for menu choices?!! Diet Coke! Who has the Zagats!Finally, think of what you’d miss behind the glass... Check out skydeckcartoons.com for more. #UXSmackdown
  13. SummaryReasons for conducting Focus Groups:1. Focus groups can be “small groupworkshops,” not “focus groups”.2. You don’t have to dress up like a fieldstudies ninja.3. Focus groups can be “in situ” orremote.4. Focus groups can haveindividual assignments beforegroup discussion to improve dataquality. #UXSmackdown
  14. Field Research #UXSmackdown
  15. “What  people  say,  what  people  do,  and  what  they  say  they  do  are  entirely  different  things.”   —Margaret Mead #UXSmackdown
  16. Focus groups aren’t enough:If you want to find out: • What users want & need • What they’re doing now • How they feel about it • What you can improve
  17. Immerse yourself in the user’s world #UXSmackdown
  18. ROLE: Business OwnerTASK: Approve visual design directionCONTEXT: Waterfall dev process. Supervises multipleproduct managers. Frequently Approve visual design direction TASK: mobile; uses iPhone.CHARACTERISTICS: Short aention span.Under significant time pressure. PM IDFocuses on visuals and metrics. BOCONTENT CRITERIA: Brief, clear presentation in commonformats consumable on mobile devices Dev Mgr VzD #UXSmackdown
  19. Stakeholder participation in analysis = everyone owns the study findingsPhoto  by  Jane  Mejdahl,    used  under  CC  BY-­‐SA  2.0.   19
  20. Focus group falsehoods: • Human memory fail • Suggestibility • Social norms • Self-enhancement bias • Self serving facilitators #UXSmackdown
  21. Imagine you’re askingabout #UXSmackdown
  22. Oh! How do you Thanklike the gift I you! made you? (Ugh, it’s horrible.) #UXSmackdown
  23. How do you Oh!like the app Great!we built you? (Yuck, it’s useless.) #UXSmackdown
  24. Our study found thatfocus group participantsare more generous,more law abiding, andfloss more often. #UXSmackdown
  25. SummaryField studies let you: • Understand user needs • Uncover tacit knowledge that isn’t consciously available • Find deltas between how users think they do something and their performance • Develop actionable insights • Identify opportunities for innovation #UXSmackdown
  26. ROUND 1Make Some Noise! #UXSmackdown
  27. ROUND 1 Questions #UXSmackdown
  28. ROUND 2 Eye Tr acking Vs. mote T estingUn moderated Re #UXSmackdown
  29. Eye Tracking #UXSmackdown
  30. #UXSmackdown
  31. UnmoderatedRemote Usability Testing: ? #UXSmackdown
  32. #UXSmackdown
  33. Unmoderated Usability Testing #UXSmackdown
  34. Benefits #UXSmackdown
  35. Test hundreds of people simultaneously while keepingthem in their own natural environment. #UXSmackdown
  36. Test multiple websites simultaneously. #UXSmackdown
  37. Reduced costs... depending on the tool you use. #UXSmackdown
  38. Great way to plant the seed of UCD methodologiesand introduce usability testing into a company. UX #UXSmackdown
  39. Fewer logistics to manage and no need to worry aboutno-shows and getting last-minute replacements. #UXSmackdown
  40. Fast Results #UXSmackdown
  41. In-context Feedback “...I’m only offered a search by model number. How ridiculous.” “...I would like to see the products availability and delivery/ shipping information listed.”Image source: Loop11 #UXSmackdown
  42. The danger... I wonderhow a real customerwould actually use this website? Panelist #UXSmackdown
  43. The danger... BLOCKEDBLOCKED #UXSmackdown
  44. There are no shortcutsto conductinggood research. #UXSmackdown
  45. YO U B E THE JU DG E!Good? Evil? #UXSmackdown Photo  copyright  New  Line  Cinema
  46. ROUND 2Make Some Noise! #UXSmackdown
  47. ROUND 2 Questions #UXSmackdown
  48. ROUND 3 elity Proto typesH igh Fid Vs. Itera tive T esting Rapid #UXSmackdown
  49. High Fidelity Prototypes #UXSmackdown
  50. Why #HighFidelityRules... ...and #IterativeTestingFailsPhoto  copyright  Apple #UXSmackdown
  51. Weight 2,061 2,954 Drive Train Rear Rear HP 350 320 Torque 292 273 Top Speed 156 171 0-60 5.3 4.9Why High-Fidelity Prototypes?1. Everything looks great on paper... #UXSmackdown
  52. ...but sometimes the final experience is not the same. #UXSmackdown
  53. ...but sometimes the final experience is not the same. #UXSmackdown
  54. In today’s swipe and swish world you need the full experience. #UXSmackdown
  55. Pause Suppli Perfor HIGHSCORE 0 es mance SCORE 02a. Iterative testing can’t tell you if you’re creating flow. #UXSmackdown
  56. 2a: Iterative testing can’t tell you if you’re creating flow. #UXSmackdown
  57. You should see the front. blue fade yellow fade2b: Iterative testing can’t tell you if you’re creatingpersuasive design. #UXSmackdown
  58. 2b: Iterative testing can’t tell you if you’re creatingpersuasive design. #UXSmackdown
  59. 3. High fidelity prototypes identify the full implications of“stakeholder design” #UXSmackdown
  60. #UXSmackdown
  61. John, couldn’t you iteratively testsomething simple, like a login without a high fidelity prototype?! #UXSmackdown
  62. John, couldn’t you iteratively testsomething simple, like a login without a high fidelity prototype?! YES #UXSmackdown
  63. ...but my argument is that increasingly we need thefull experience to understand a design’s success. #UXSmackdown
  64. Final note: I too find great value in iterative design in theright circumstance! #UXSmackdown
  65. SummaryReasons for High-FidelityPrototypes:1. In today’s swipe and swish worldyou need the full experience.2. You can’t really “iteratively test”,persuasive or multi-channel designs.3. High-fidelity prototypes identify thefull implications of“stakeholder design”. #UXSmackdown
  66. High Fidelity Rapid Iterative Prototype vs. Usability Testing Testing #UXSmackdown
  67. Low-Fidelity Prototyping viously Ad d pre rk N etwo ity Activ #UXSmackdown
  68. RITE Usability Testing Test! Rev ised prob as s lem oon is id as a enti fied > Revise Revise Prototype! Prototype! Test! #UXSmackdown
  69. How do you makedesign decisionsbased on only testing1 person? #UXSmackdown
  70. Modified RITE Process ent ! Revise ! Test! Debrief! Pres ngs type(All users) Findi P roto Test! Debrief! (All users) #UXSmackdown
  71. Benefits of choosing low-fidelity prototypes #UXSmackdown
  72. Getting end user feedbackearlier in the design process. #UXSmackdown
  73. It’s easier to throw awaypaper than code. #UXSmackdown
  74. Provides the opportunity tothink outside the box. #UXSmackdown
  75. It gets the entire team involved. #UXSmackdown
  76. Participants are less apt to givenegative feedback on designs thathave a "finished" appearance. Oh, it’s lovely! I wouldn’t change anything. #UXSmackdown
  77. i FT W! n Lo w-FRIT E o #UXSmackdown
  78. ROUND 3Make Some Noise! #UXSmackdown
  79. ROUND 3 Questions #UXSmackdown
  80. Focus G roups Vs. Field R racking esearc Eye T h Vs. ed Remot e TestingU nmoderat High Fidelity Prototypes Vs. Rapid Iterative Testing #UXSmackdown
  81. #UXSmackdown

×