Presentation from 2012 SXSW Interactive

1. UX SMACKDOWN!
Usability Testing Techniques in the Ring
#UXSmackdown

2. Aviva Rosenstein Rebecca Sherrill
Sr Mgr, User Research UX Lead
Salesforce.com Beaconfire.com
@uxresearch @beccas
Kyle Soucy Michael Summers
Founding Principal Global User Experience, Paypal
UsableInterface.com UserResearch.com
@kylesoucy @ez2use
John Whalen
UX Lead & Founder
Brilliant Experience.com
@johnwhalen
#uxSmackdown
#UXSmackdown

3. ROUND 1
Focus Groups
Vs.
Fie ld Res earch
#UXSmackdown

4. Focus Groups
#UXSmackdown

5. ?
Why are we proponents of focus groups?
#UXSmackdown

6. 1. Focus groups do not have to be this...
Photo
copyright
AMC #UXSmackdown

7. 1. Focus groups do not have to be this...
Photo
copyright
AMC #UXSmackdown

8. 2. You don’t have to dress up like a $#%^&*@$!!
#fieldstudyninja to get good data.
#UXSmackdown

9. 3a. Focus groups can be “in situ”
#UXSmackdown

10. 3b. Focus groups can be remote
#UXSmackdown

11. 4. Focus groups can have individual assignments before
group discussion to produce better data.
#UXSmackdown

12. Drip Coffee?! No M&M’s?
I must make a Totally
Starbucks run! unacceptable!
I can’t work
under these
conditions!
Diet Pepsi is
...and only 12
no substitute for
menu choices?!!
Diet Coke!
Who has the Zagats!
Finally, think of what you’d miss behind the glass...
Check out skydeckcartoons.com for more.
#UXSmackdown

13. Summary
Reasons for conducting Focus Groups:
1. Focus groups can be “small group
workshops,” not “focus groups”.
2. You don’t have to dress up like a field
studies ninja.
3. Focus groups can be “in situ” or
remote.
4. Focus groups can have
individual assignments before
group discussion to improve data
quality.
#UXSmackdown

14. Field Research
#UXSmackdown

15. “What
people
say,
what
people
do,
and
what
they
say
they
do
are
entirely
different
things.”
—Margaret Mead
#UXSmackdown

16. Focus groups aren’t enough:
If you want to find out:
• What users want & need
• What they’re doing now
• How they feel about it
• What you can improve

17. Immerse yourself in the user’s world
#UXSmackdown

18. ROLE: Business Owner
TASK: Approve visual design direction
CONTEXT: Waterfall dev process. Supervises multiple
product managers. Frequently Approve visual design direction
TASK: mobile; uses iPhone.
CHARACTERISTICS: Short aention span.
Under significant time pressure. PM
ID
Focuses on visuals and metrics. BO
CONTENT CRITERIA: Brief, clear presentation in common
formats consumable on mobile devices
Dev Mgr
VzD
#UXSmackdown

19. Stakeholder participation in analysis =
everyone owns the study findings
Photo
by
Jane
Mejdahl,
used
under
CC
BY-­‐SA
2.0.
19

20. Focus group falsehoods:
• Human memory fail
• Suggestibility
• Social norms
• Self-enhancement bias
• Self serving facilitators
#UXSmackdown

21. Imagine you’re asking
about
#UXSmackdown

22. Oh!
How do you
Thank
like the gift I
you!
made you?
(Ugh, it’s
horrible.)
#UXSmackdown

23. How do you Oh!
like the app Great!
we built you?
(Yuck, it’s
useless.)
#UXSmackdown

24. Our study found that
focus group participants
are more generous,
more law abiding, and
floss more often.
#UXSmackdown

25. Summary
Field studies let you:
• Understand user needs
• Uncover tacit knowledge that isn’t
consciously available
• Find deltas between how users
think they do something and their
performance
• Develop actionable insights
• Identify opportunities for
innovation
#UXSmackdown

26. ROUND 1
Make Some Noise!
#UXSmackdown

27. ROUND 1
Questions
#UXSmackdown

28. ROUND 2
Eye Tr acking
Vs.
mote T esting
Un moderated Re
#UXSmackdown

29. Eye Tracking
#UXSmackdown

30. #UXSmackdown

31. Unmoderated
Remote Usability Testing:
?
#UXSmackdown

32. #UXSmackdown

33. Unmoderated Usability Testing
#UXSmackdown

34. Benefits
#UXSmackdown

35. Test hundreds of people simultaneously while keeping
them in their own natural environment.
#UXSmackdown

36. Test multiple websites simultaneously.
#UXSmackdown

37. Reduced costs... depending on the tool you use.
#UXSmackdown

38. Great way to plant the seed of UCD methodologies
and introduce usability testing into a company.
UX
#UXSmackdown

39. Fewer logistics to manage and no need to worry about
no-shows and getting last-minute replacements.
#UXSmackdown

40. Fast Results
#UXSmackdown

41. In-context Feedback
“...I’m only offered a
search by model number.
How ridiculous.”
“...I would like to see the
products availability and delivery/
shipping information listed.”
Image source: Loop11
#UXSmackdown

42. The danger...
I wonder
how a real customer
would actually use this
website?
Panelist
#UXSmackdown

43. The danger...
BLOCKED
BLOCKED
#UXSmackdown

44. There are no
shortcuts
to conducting
good research.
#UXSmackdown

45. YO U B E
THE JU
DG E!
Good? Evil?
#UXSmackdown
Photo
copyright
New
Line
Cinema

46. ROUND 2
Make Some Noise!
#UXSmackdown

47. ROUND 2
Questions
#UXSmackdown

48. ROUND 3
elity Proto types
H igh Fid
Vs.
Itera tive T esting
Rapid
#UXSmackdown

49. High Fidelity
Prototypes
#UXSmackdown

50. Why #HighFidelityRules... ...and #IterativeTestingFails
Photo
copyright
Apple #UXSmackdown

51. Weight 2,061 2,954
Drive Train Rear Rear
HP 350 320
Torque 292 273
Top Speed 156 171
0-60 5.3 4.9
Why High-Fidelity Prototypes?
1. Everything looks great on paper...
#UXSmackdown

52. ...but sometimes the final experience is not the same.
#UXSmackdown

53. ...but sometimes the final experience is not the same.
#UXSmackdown

54. In today’s swipe and swish world you need the full experience.
#UXSmackdown

55. Pause
Suppli Perfor HIGHSCORE 0
es mance
SCORE 0
2a. Iterative testing can’t tell you if you’re creating flow.
#UXSmackdown

56. 2a: Iterative testing can’t tell you if you’re creating flow.
#UXSmackdown

57. You should see the front.
blue fade yellow fade
2b: Iterative testing can’t tell you if you’re creating
persuasive design.
#UXSmackdown

58. 2b: Iterative testing can’t tell you if you’re creating
persuasive design.
#UXSmackdown

59. 3. High fidelity prototypes identify the full implications of
“stakeholder design”
#UXSmackdown

60. #UXSmackdown

61. John,
couldn’t you iteratively test
something simple, like a login without a
high fidelity prototype?!
#UXSmackdown

62. John,
couldn’t you iteratively test
something simple, like a login without a
high fidelity prototype?!
YES
#UXSmackdown

63. ...but my argument is that increasingly we need the
full experience to understand a design’s success.
#UXSmackdown

64. Final note: I too find great value in iterative design in the
right circumstance!
#UXSmackdown

65. Summary
Reasons for High-Fidelity
Prototypes:
1. In today’s swipe and swish world
you need the full experience.
2. You can’t really “iteratively test”,
persuasive or multi-channel designs.
3. High-fidelity prototypes identify the
full implications of
“stakeholder design”.
#UXSmackdown

66. High Fidelity Rapid Iterative
Prototype vs. Usability
Testing Testing
#UXSmackdown

67. Low-Fidelity Prototyping
viously
Ad d pre
rk
N etwo
ity
Activ
#UXSmackdown

68. RITE Usability Testing
Test! Rev
ised
prob as s
lem oon
is id as a
enti
fied
>
Revise Revise
Prototype! Prototype!
Test!
#UXSmackdown

69. How do you make
design decisions
based on only testing
1 person?
#UXSmackdown

70. Modified RITE Process
ent ! Revise !
Test! Debrief! Pres ngs type
(All users) Findi P roto
Test! Debrief!
(All users)
#UXSmackdown

71. Benefits of choosing
low-fidelity
prototypes
#UXSmackdown

72. Getting end user feedback
earlier in the design process.
#UXSmackdown

73. It’s easier to throw away
paper than code.
#UXSmackdown

74. Provides the opportunity to
think outside the box.
#UXSmackdown

75. It gets the entire team involved.
#UXSmackdown

76. Participants are less apt to give
negative feedback on designs that
have a "finished" appearance.
Oh, it’s lovely!
I wouldn’t
change anything.
#UXSmackdown

77. i FT W!
n Lo w-F
RIT E o
#UXSmackdown

78. ROUND 3
Make Some Noise!
#UXSmackdown

79. ROUND 3
Questions
#UXSmackdown

80. Focus
G roups
Vs.
Field R
racking esearc
Eye T h
Vs.
ed Remot e Testing
U nmoderat
High Fidelity Prototypes
Vs.
Rapid Iterative Testing
#UXSmackdown

81. #UXSmackdown