1
The quick, the cheap,
and the insightful
Usability testing in the wild
Dana Chisnell
San Francisco STC
November 2008

2
What is a usability test?

3
http://www.sigchi.org/chi97/proceedings/overview/tst.htm

4

5
Quick, cheap, insightful
✤ Minimal steps to focus on
✤ Where the value comes from in doing usability tests
✤ Where it may be risky to go minimalist
✤ How to think about the trade-offs
✤ What’s essential
✤ What might be nice to have

6
Minimal steps

7
Classic, with everything
✤ Chapter 5. Develop a test plan
✤ Chapter 6. Choose a testing environment
✤ Chapter 7. Find and select participants
✤ Chapter 8. Prepare test materials
✤ Chapter 9. Conduct the sessions
✤ Chapter 10. Debrief with participants and observers
✤ Chapter 11. Analyze data and observations
✤ Chapter 12. Create findings and recommendations

8
Photo courtesy of Tom Tullis

9
Classic, with everything
Can take a lot of time: weeks or months.

10
What if you have to do something
quick? and cheap?

11
You don’t have to do it
by the book.

12
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations

13
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations

14
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers

15
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers and decide together

16
Sit next to someone.
Watch them do stuff.

17
Story:

18
California.

19
1 day

20
a ballot and instructions

21
a ballot and instructions

22
random participants

23
no helpers

24
results due: today

25
What to do?

26
Improvise: Ground rules
✤ Say “yes.”
✤ Trust your team and your participants
✤ Don’t let the team or stakeholders block
✤ Work to the top of your collective intelligence

27
How we got value
✤ Used the ballots and instructions at hand
✤ Focused on one thing at a time
✤ Had participants generate and collect data
✤ Drafted observers and debriefed

28
More examples

29
Cafe

30
Company
lobby

31
Trade
show
http://www.flickr.com/photos/dans180/72408664/

32
Listen in
on
customer
service

33
Transit
seat
mates

34
Remote
testing

35
Browser history
Gives a recent snapshot of activity
Bookmarks & favorites
Shows deliberate returns to sites
Email trails
Builds a story
Shows organizing strategies

36
Customer service logs
View on persistent, difficult problems
Sales reps
Proxies for buyers, end users
User forums & wikis
Frames the conversations users are having

37
Value

38
Each phase includes input
from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and
objectives
Supporting usable design

39
Each phase includes input
from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and
objectives
Supporting usable design

40
Where’s the value in testing?
70% watching someone use
the design
20% working with the team
to prepare to test
8% discussing what
happened

41
Getting value in the wild
✤ Use what’s at hand
✤ Narrow the scope of the test
✤ Focus on
✤ what can make the most difference to the
most users
✤ what can be implemented easily with the
resources available

42
✤ If you need
✤ summative
data
When is testing in the ✤ benchmarks
✤ answers to
wild not valuable? hard
problems

43
Risks

44
Risks of testing in the wild
✤ Participant sample may be too small,
biased
✤ Inconsistent approach may net
inconsistent data

45
What do you lose?
✤ Quantitative data
✤ Rigor
✤ Relatively unbiased sample, maybe
http://drb.lifestreamcenter.net/Lessons/process_maps/

46
Trade offs

47
What do you need?
✤ To inform a design:
Testing in the wild Classic usability testing
Qualitative data Quantitative data
Opportunity Planning
Fits the schedule Don’t know how to fit UT into
the schedule
Just in time After the fact
$ $$$$
Something Maybe nothing

48
Where’s the ROI?
✤ Qualitative data
✤ Opportunity
✤ Fitting into a schedule
✤ Timeliness

49
So far.
You don’t have to do it Value of usability Tradeoff: having
by the book testing some data over
having no data
+ observing users
+ working with the
team

50
Coming up.
What’s the bare Steps for testing in What to add if you
minimum the wild have resources

51
Essentials

52
What you need
Somebody
(Human)
✤ Someone who will try the design
Something Some place
✤ Somewhere to test (Activity) (Context)
✤ Something to study

53
1. Plan, minimally
2. Get the team on
board
3. Design the test,
minimally
4. Recruit
participants
5. Conduct
sessions
Follow these steps: 6. Debrief and
decide

54
Plan, minimally
✤ What
✤ Why
✤ Who
✤ When
✤ Where

55
Plan, minimally: Example
✤ What: near-final design
✤ Why: inform user training and support
✤ Who: inexperienced customers
✤ When: the end of the week
✤ Where: trade show at user group meeting

56
Minimalist plan
Find out whether information about admission is
Goals and objectives
easy to find and use
Participant
People who have college-bound kids
characteristics
Description of Sit-by parents attending a high school basketball game, each
method trying three university sites with the same scavenger hunt task
Find out how and when applications are due
List of tasks Determine whether there are fees for applying
Learn when acceptances will be sent

57
Get the team
on board
✤ Visualize the desired user
experience
✤ Share the intellectual property of
observing users

58
Design the
test, minimally
✤ Why are you testing?
✤ What questions are you trying to
answer?
✤ What constrains the design?
✤ What are you going to do with
what you find out?

59
Recruit
participants
✤ Convenience sample
✤ Requirements, not
demographics

60
✤ Staff not on the
design project
✤ Friends and family
✤ Personal,
professional
networks
✤ Online social
networks
✤ Community
organizations
✤ Online classifieds
✤ Association,
society, user
group, union
Sources of participants contacts
✤ Temp agencies

61
Conduct
sessions
✤ Rehearse (P1)
✤ Interview-based tasks (or based
on previous field work)
✤ Explore in short, focused
sessions
✤ Iterate test design and product
design

62
Greet the
participant
Explain the study,
your role, and their
role
Interview (maybe)
Do tasks from
interview
Debrief with
participant
Session outline Debrief with
observers

63
✤ Impartial,
unbiased
observing
✤ No teaching!
✤ Listen and
watch
✤ Ask open-
ended
questions:
Moderating, not training Why? How?
What?

64
Debrief and
decide
✤ Write up issues on sticky notes
and sort them into priority lists
✤ Ask for top 10 items -- base on
data and observations, not
opinion
✤ Take a vote for priorities

65
Nice to have

66
✤ Screened,
scheduled
participants
✤ Official
paperwork
Add these ingredients
✤ Recordings
✤ More observers

67
Participants
What Why
appropriate experience greater confidence in data
scheduled ahead of time easier to get data

68
Paperwork
What Why
script ensure consistent instructions and moderating
consent forms official acknowledgment for taking part
recording waivers permission for recordings
pre-test questionnaire experience, knowledge, value
post-test questionnaire feedback on tasks, UI

69
Recordings
What Why
audio transcripts, verbal protocol analysis
video or photos stories, double-check, highlights
Morae, Camtasia, other digital data, automated collection

70
More observers
✤ Better-informed design recommendations
What Why
developers technology boundaries
management, execs business priorities

71
Nutshell
http://www.flickr.com/photos/stoan/145497333/

72
✤ Value:
observing,
planning
✤ Plan: 4 steps
✤ Recruit:
behavior
✤ Moderating:
Take away: listen, observe

73
Quick. Cheap. Insightful.
You can compensate Low risk, little money Value comes from
for some getting as close as
shortcomings, and just possible to what real
test more people are doing
with your design

74
Where to learn more
Dana’s blog: http://
usabilitytestinghowto.blogspot.com/
Download templates, examples, and
links to other resources from
www.wiley.com/go/usabilitytesting

75
Me
Dana Chisnell
dana@usabilityworks.net
www.usabilityworks.net
415.519.1148