Testing & User Research
Workshop
Photo Credit: kaboodle.com
“The best tool for resolving disputes within a
design team, for making design decisions based
on data rather than opinion,...
Btw I’m Cyd Harrell
I’m UX Evangelist at Code for America
My favorite tools: empathy and duct tape
Today’s Agenda
10:00-10:15: intros
10:15-11:30: core user research technique
11:45-12:30: your research set-up
1:30 - 2:45...
#1 Find out about the people who will use your thing
#2 Align your core team around what you’re doing to do
#3 Find out wh...
The hardest two things about research:
‣ providing the prompt
‣ collecting the response
OversimplificationAlert
The hardest two three things about research:
‣ providing the prompt
‣ collecting the response
‣ getting everyone to buy in...
The hardest three four things:
‣ getting the right participants
‣ providing the prompt
‣ collecting the response
‣ getting...
o.k.soafewthingsarehard
Let’s jump right in
What’s the latest with UX Research?
WHAT’S DIFFERENT IN 2014
Source: Pew Internet and American Life Project, January 2014
WHAT’S DIFFERENT IN 2014
Source: Knight Foundation Digital Access Update, February 2014
TOOLKIT
TOOLKIT
REMEMBER BE
graceful
When All Your
TECHNOLOGY
BREAKS
uximresearch.tumblr.com
Now we do a little experiment
EXERCISE #1
“Giraffe”
‣ 2 groups per table
‣ Pick one moderator and one participant
‣ 10 minutes of observation
MODERATOR Instructions
Your participant has a sheet of paper and that’s it
Goal: find out what kind of physical task instru...
It’s A RACE
You only have 10 minutes to do this max.
Surprises?
Useful phrases 1
the approach
“Hi, would you like to participate in a study
today?”
“I’m working to improve XXXX for citiz...
Useful phrases 2
reassurance
“This isn’t a test of you, it’s a test of the system”
“There are no right or wrong answers”
“...
Useful phrases 3
instructions
“I just need you to be yourself and act as you
naturally would”
“One little thing I’d like y...
Useful phrases 4,5,6,7
keeping the conversation going
“mm-hm” “oh interesting” “sure, I hear you”
“right” “ok” “uh-huh”
(l...
Useful phrases 8
probes
“Can you tell me a little more about that?”
“When you did X a minute ago, can you tell me
how you ...
Useful phrases 9
ratings
“If you had to give this a letter grade, you know
A, B, C, D, F like we used to get in school, wh...
Useful phrases 10
self-explanatory
“Thank you so much, I really appreciate your
taking the time to talk to me.”
DESIGNINGUSER STUDIES
FIND THE RIGHT PARTICIPANTS
uximresearch.tumblr.com Photo Credit: Ed Yourdon
Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
OFFER A TASK
SCRIPT FOR CUSTOMIZATION
“Please look for a pair of pink pumps in your size.”
“Please look for a pair of shoes you’d like ...
MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Thanks & incentive
AUTOMATED must-haves
Expectation setting
Instructions
The beef
Thanks & incentive
Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
STREAM & RECORD WHAT HA...
BREAK
Photo Credit: Sorin Petcu
Testing & User Research
Part 2: your set-up
What’s New with ResearchTools?
Remote Design Research
Definition:
Research where the participant is not
sharing physical space with the
researcher.
% of q...
Screen
Easier
Sharing
Awesome compared to this!
Prototype
So Many
Tools
Flinto
Experience Sampling
Task-driven
iOS & Android
Mobile Client
Simultaneous
Website Testing
WHOSE DEVICE?
Photo Credit: groundsel, inju
Pro:
‣ better context
‣ more comfortable users
Con:
‣ extra installation time
‣ less control
Pro tips for working with users’ own phones:
‣ Have a backup iPhone and backup Android device on hand.
‣ Make a power stri...
Pro tips for offering lab devices:
‣ Have a backup iPhone and backup Android device on hand.
‣ Charge devices between ever...
Physical Setup for MobileTests
Photo Credit: Diane Loviglio, Mozilla
RealTalk About Holding Phones
Source: Steve Hoober in UX Matters, February 2013
TestTargets
PROTOTYPES,WIREFRAMES, SKETCHES?
HECKYES.
Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
PAPER PROTOTYPES, POST-...
Photo from UX Magazine article by Tania Lang
PAPER PROTOTYPES, WITH SCROLLING
Lo-fi
clickable
prototypes
Photo from UX Magazine article by Tania Lang
Prototyping Apps for Higher Fidelity
Proto.io
Fluid
Flinto
PhoneGap
Photo from UX Magazine article by Tania Lang
Cool Hack:
Screenshots in an album or photostream
The hardest two things about research:
‣ providing the prompt
‣ collecting the response
InTheLab
Photo Credit: Hans Makkee
high resolution
portable
streaming camera
Mobile Screen
Recording
Source: OpenDeviceLab.com
Device Labs
uximresearch.tumblr.com
EXERCISE #2
“RollYour Own”
‣ Share out
Taking Notes
Second monitor if possible
Excel Hack
(remoteresear.ch/samples)
Observer Involvement
IM
Campfire (!)
Integrated Chat
Passing Notes
Questions?
Let’s do lunch
Testing & User Research
Part 3: recruiting participants
RECRUITING OPTIONS
‣ Grab someone in the hallway
‣ Grab someone who isn’t involved in your project
‣ IM your outside frien...
TWITTER RECRUITING
ANATOMY OF A RECRUITING TWEET
@person we’re interviewing [job or characteristic] about [topic] on [dates]
(paid research) ...
BIGGER THAN A TWEET
‣ Why are you doing this work?
‣ What exactly are you asking? (How much time, where, when)
‣ Who do yo...
Screener constraints
Keep it short
Always ask an open-ended question
Different from surveys
Who gets paid?
MOBILE FORMS ARE GETTING WAY BETTER!
PANELS
“What happens when you let go of a balloon?”
PRO TIP: HOW TO FIND ACTUAL SPEAKERS OF A LANGUAGE:
“ Qu'est-ce qui se passe ...
WHAT PARTICIPANTS WORRY ABOUT
Who are you and who sent you?
(can I trust you?)
Will participating affect my actual outcome?
Is this going to end up on
YouTube?
remoteresear.ch/samples
“Online Consent”
Paying Incentives
Collect minimal personal info
Deliver quickly
Appropriate for location
Paying Incentives
“Free cookies and
lemonade for your
city feedback”
EXERCISE
‣ Consider format (poster, landing page, online form)
‣ Cover the questions
draft a recruiting appeal
‣ Why are you doing this work?
‣ What exactly are you asking? (How much time, where, when)
‣ Who do you need? (tasks or de...
share out
BREAK
Photo Credit: Mrs Magic
User Testing & Research
Part 4: going beyond usability
WHAT IF IT’S NOT USABILITY?
EXPERIENCE SAMPLING
EXPERIENCE SAMPLING
uximresearch.tumblr.com
Cyd’sResults
Cyd’sResults
CONCEPT & NAVIGATION TESTING:
EXERCISE
UNMODERATED STUDIES
3 parts
Provide test target
Record user behavior
Capture comments (optional)
Prototype
Unmoderated
Testing
Mobile Diary Studies
iOS & Android
Results
Results
Results
CAPTURE WHAT HAPPENS.
Surveys
via SMS
About getting buy-in
AFewWords
How “Testing” Is Perceived
Numbers (& Numbers People)
AreYour Friends
Photo Credit: Search Influence on Flickr
Video Is Another Friend
Photo Credit: Search Influence on Flickr
Try asking for
One Day
$200
PeopleToWatch
Photo Credit: Search Influence on Flickr
“Fix the machine, not the person”
-- Aaron Swartz, 9.25.12
ActualAdvice
There’s always a way
Questions?
cyd@codeforamerica.org
@cydharrell
Thanks for coming.
Go run some awesome research.
Photo Credit: Wade Rockett on Flickr
Open Plans User Testing Workshop
Open Plans User Testing Workshop
Open Plans User Testing Workshop
Open Plans User Testing Workshop
Open Plans User Testing Workshop
Open Plans User Testing Workshop
Upcoming SlideShare
Loading in...5
×

Open Plans User Testing Workshop

707

Published on

Introductory user testing & user research workshop for government staff & civic hackers, delivered at Thoughtworks in NYC on 4/23/14

Published in: Design, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
707
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Open Plans User Testing Workshop

  1. 1. Testing & User Research Workshop
  2. 2. Photo Credit: kaboodle.com
  3. 3. “The best tool for resolving disputes within a design team, for making design decisions based on data rather than opinion, is sitting next to someone who is a real person who wants to accomplish something as they use your design to do it.” --Dana Chisnell, August 2, 2010
  4. 4. Btw I’m Cyd Harrell I’m UX Evangelist at Code for America My favorite tools: empathy and duct tape
  5. 5. Today’s Agenda 10:00-10:15: intros 10:15-11:30: core user research technique 11:45-12:30: your research set-up 1:30 - 2:45: recruiting participants 3:00 - 4:00: going beyond usability
  6. 6. #1 Find out about the people who will use your thing #2 Align your core team around what you’re doing to do #3 Find out what your core team is up against in their work Research
  7. 7. The hardest two things about research: ‣ providing the prompt ‣ collecting the response OversimplificationAlert
  8. 8. The hardest two three things about research: ‣ providing the prompt ‣ collecting the response ‣ getting everyone to buy in OversimplificationAlert
  9. 9. The hardest three four things: ‣ getting the right participants ‣ providing the prompt ‣ collecting the response ‣ getting everyone to buy in OversimplificationAlert
  10. 10. o.k.soafewthingsarehard
  11. 11. Let’s jump right in
  12. 12. What’s the latest with UX Research?
  13. 13. WHAT’S DIFFERENT IN 2014 Source: Pew Internet and American Life Project, January 2014
  14. 14. WHAT’S DIFFERENT IN 2014 Source: Knight Foundation Digital Access Update, February 2014
  15. 15. TOOLKIT
  16. 16. TOOLKIT
  17. 17. REMEMBER BE graceful When All Your TECHNOLOGY BREAKS
  18. 18. uximresearch.tumblr.com
  19. 19. Now we do a little experiment
  20. 20. EXERCISE #1 “Giraffe” ‣ 2 groups per table ‣ Pick one moderator and one participant ‣ 10 minutes of observation
  21. 21. MODERATOR Instructions Your participant has a sheet of paper and that’s it Goal: find out what kind of physical task instructions work Hint: ask them to give you a play by play of what they are doing Try and follow along with your own paper NO helping, even if you could.
  22. 22. It’s A RACE You only have 10 minutes to do this max.
  23. 23. Surprises?
  24. 24. Useful phrases 1 the approach “Hi, would you like to participate in a study today?” “I’m working to improve XXXX for citizens and I’m collecting feedback.”
  25. 25. Useful phrases 2 reassurance “This isn’t a test of you, it’s a test of the system” “There are no right or wrong answers” “Positive and negative feedback is equally valuable” “Nothing you say is going to hurt my feelings”
  26. 26. Useful phrases 3 instructions “I just need you to be yourself and act as you naturally would” “One little thing I’d like you to do differently is talk aloud as you use this today”
  27. 27. Useful phrases 4,5,6,7 keeping the conversation going “mm-hm” “oh interesting” “sure, I hear you” “right” “ok” “uh-huh” (lather, rinse, repeat)
  28. 28. Useful phrases 8 probes “Can you tell me a little more about that?” “When you did X a minute ago, can you tell me how you made that choice?” “Talk me through what happened there.”
  29. 29. Useful phrases 9 ratings “If you had to give this a letter grade, you know A, B, C, D, F like we used to get in school, what would you give it?”
  30. 30. Useful phrases 10 self-explanatory “Thank you so much, I really appreciate your taking the time to talk to me.”
  31. 31. DESIGNINGUSER STUDIES
  32. 32. FIND THE RIGHT PARTICIPANTS uximresearch.tumblr.com Photo Credit: Ed Yourdon
  33. 33. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/ OFFER A TASK
  34. 34. SCRIPT FOR CUSTOMIZATION “Please look for a pair of pink pumps in your size.” “Please look for a pair of shoes you’d like to buy”
  35. 35. MODERATED must-haves Consent Tech set-up The beef De-installation Thanks & incentive
  36. 36. AUTOMATED must-haves Expectation setting Instructions The beef Thanks & incentive
  37. 37. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/ STREAM & RECORD WHAT HAPPENS.
  38. 38. BREAK Photo Credit: Sorin Petcu
  39. 39. Testing & User Research Part 2: your set-up
  40. 40. What’s New with ResearchTools?
  41. 41. Remote Design Research Definition: Research where the participant is not sharing physical space with the researcher. % of qualitative research done remotely* * Source: My wild guess In person Remote (2008) Remote (2012)
  42. 42. Screen Easier Sharing
  43. 43. Awesome compared to this!
  44. 44. Prototype So Many Tools Flinto
  45. 45. Experience Sampling Task-driven iOS & Android
  46. 46. Mobile Client Simultaneous Website Testing
  47. 47. WHOSE DEVICE?
  48. 48. Photo Credit: groundsel, inju Pro: ‣ better context ‣ more comfortable users Con: ‣ extra installation time ‣ less control
  49. 49. Pro tips for working with users’ own phones: ‣ Have a backup iPhone and backup Android device on hand. ‣ Make a power strip part of your testing station ‣ Have chargers for all the phones you’re expecting ‣ Instruct participants to install necessary apps in advance, but leave time in case they don’t ‣ Adjust the screen brightness before you start each session
  50. 50. Pro tips for offering lab devices: ‣ Have a backup iPhone and backup Android device on hand. ‣ Charge devices between every session. ‣ Remove passwords, lock screens, etc. ‣ Dry-run every aspect of the test, since you have the opportunity. uximresearch.tumblr.com
  51. 51. Physical Setup for MobileTests
  52. 52. Photo Credit: Diane Loviglio, Mozilla
  53. 53. RealTalk About Holding Phones Source: Steve Hoober in UX Matters, February 2013
  54. 54. TestTargets
  55. 55. PROTOTYPES,WIREFRAMES, SKETCHES?
  56. 56. HECKYES.
  57. 57. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/ PAPER PROTOTYPES, POST-IT STYLE
  58. 58. Photo from UX Magazine article by Tania Lang PAPER PROTOTYPES, WITH SCROLLING
  59. 59. Lo-fi clickable prototypes
  60. 60. Photo from UX Magazine article by Tania Lang Prototyping Apps for Higher Fidelity Proto.io Fluid Flinto PhoneGap
  61. 61. Photo from UX Magazine article by Tania Lang Cool Hack: Screenshots in an album or photostream
  62. 62. The hardest two things about research: ‣ providing the prompt ‣ collecting the response InTheLab
  63. 63. Photo Credit: Hans Makkee high resolution portable streaming camera
  64. 64. Mobile Screen Recording
  65. 65. Source: OpenDeviceLab.com Device Labs
  66. 66. uximresearch.tumblr.com
  67. 67. EXERCISE #2 “RollYour Own” ‣ Share out
  68. 68. Taking Notes Second monitor if possible Excel Hack (remoteresear.ch/samples)
  69. 69. Observer Involvement IM Campfire (!) Integrated Chat Passing Notes
  70. 70. Questions?
  71. 71. Let’s do lunch
  72. 72. Testing & User Research Part 3: recruiting participants
  73. 73. RECRUITING OPTIONS ‣ Grab someone in the hallway ‣ Grab someone who isn’t involved in your project ‣ IM your outside friends for a quick look ‣ Panels and recruiting agencies ‣ Real citizens from email & consent lists ‣ Physical & web intercepts realness
  74. 74. TWITTER RECRUITING
  75. 75. ANATOMY OF A RECRUITING TWEET @person we’re interviewing [job or characteristic] about [topic] on [dates] (paid research) -interested? [LINK] I’m looking for [job or characteristic(s)] for a [research activity] on [dates] incentive $X [LINK] please RT Pro tip: edit your profile to mention “UX Researcher” and “Current Study: Martian Life”
  76. 76. BIGGER THAN A TWEET ‣ Why are you doing this work? ‣ What exactly are you asking? (How much time, where, when) ‣ Who do you need? (tasks or demographics) ‣ Who is asking? ‣ Who vouches for you? (official sponsor or trusted leader) ‣ Will there be a reward or incentive? ‣ Can this be confidential or anonymous?
  77. 77. Screener constraints Keep it short Always ask an open-ended question Different from surveys Who gets paid?
  78. 78. MOBILE FORMS ARE GETTING WAY BETTER!
  79. 79. PANELS
  80. 80. “What happens when you let go of a balloon?” PRO TIP: HOW TO FIND ACTUAL SPEAKERS OF A LANGUAGE: “ Qu'est-ce qui se passe quand vous lâchez un ballon? ”
  81. 81. WHAT PARTICIPANTS WORRY ABOUT
  82. 82. Who are you and who sent you? (can I trust you?)
  83. 83. Will participating affect my actual outcome?
  84. 84. Is this going to end up on YouTube?
  85. 85. remoteresear.ch/samples “Online Consent”
  86. 86. Paying Incentives Collect minimal personal info Deliver quickly Appropriate for location
  87. 87. Paying Incentives “Free cookies and lemonade for your city feedback”
  88. 88. EXERCISE ‣ Consider format (poster, landing page, online form) ‣ Cover the questions draft a recruiting appeal
  89. 89. ‣ Why are you doing this work? ‣ What exactly are you asking? (How much time, where, when) ‣ Who do you need? (tasks or demographics) ‣ Who is asking? ‣ Who vouches for you? (official sponsor or trusted leader) ‣ Will there be a reward or incentive? ‣ Can this be confidential or anonymous?
  90. 90. share out
  91. 91. BREAK Photo Credit: Mrs Magic
  92. 92. User Testing & Research Part 4: going beyond usability
  93. 93. WHAT IF IT’S NOT USABILITY?
  94. 94. EXPERIENCE SAMPLING
  95. 95. EXPERIENCE SAMPLING uximresearch.tumblr.com
  96. 96. Cyd’sResults
  97. 97. Cyd’sResults
  98. 98. CONCEPT & NAVIGATION TESTING: EXERCISE
  99. 99. UNMODERATED STUDIES
  100. 100. 3 parts Provide test target Record user behavior Capture comments (optional)
  101. 101. Prototype Unmoderated Testing
  102. 102. Mobile Diary Studies iOS & Android
  103. 103. Results
  104. 104. Results
  105. 105. Results
  106. 106. CAPTURE WHAT HAPPENS.
  107. 107. Surveys via SMS
  108. 108. About getting buy-in AFewWords
  109. 109. How “Testing” Is Perceived
  110. 110. Numbers (& Numbers People) AreYour Friends Photo Credit: Search Influence on Flickr
  111. 111. Video Is Another Friend Photo Credit: Search Influence on Flickr
  112. 112. Try asking for One Day $200 PeopleToWatch Photo Credit: Search Influence on Flickr
  113. 113. “Fix the machine, not the person” -- Aaron Swartz, 9.25.12 ActualAdvice
  114. 114. There’s always a way
  115. 115. Questions?
  116. 116. cyd@codeforamerica.org @cydharrell
  117. 117. Thanks for coming. Go run some awesome research. Photo Credit: Wade Rockett on Flickr
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×