SlideShare a Scribd company logo
1




The quick, the cheap,
and the insightful
Usability testing in the wild
Dana Chisnell
San Francisco STC
November 2008
2




What is a usability test?
3




http://www.sigchi.org/chi97/proceedings/overview/tst.htm
4
5




Quick, cheap, insightful

✤ Minimal steps to focus on

✤ Where the value comes from in doing usability tests

✤ Where it may be risky to go minimalist

✤ How to think about the trade-offs

✤ What’s essential

✤ What might be nice to have
6




Minimal steps
7




Classic, with everything
✤ Chapter 5. Develop a test plan

✤ Chapter 6. Choose a testing environment

✤ Chapter 7. Find and select participants

✤ Chapter 8. Prepare test materials

✤ Chapter 9. Conduct the sessions

✤ Chapter 10. Debrief with participants and observers

✤ Chapter 11. Analyze data and observations

✤ Chapter 12. Create findings and recommendations
8




Photo courtesy of Tom Tullis
9




Classic, with everything
Can take a lot of time: weeks or months.
10




What if you have to do something
quick? and cheap?
11




You don’t have to do it
by the book.
12




What’s essential?

 Develop a test plan

 Choose a testing environment

 Find and select participants

 Prepare test materials

 Conduct the sessions

 Debrief with participants and observers

 Analyze data and observations

 Create findings and recommendations
13




What’s essential?

 Develop a test plan

 Choose a testing environment

 Find and select participants

 Prepare test materials

 Conduct the sessions

 Debrief with participants and observers

 Analyze data and observations

 Create findings and recommendations
14




What’s essential?

 Develop a test plan



 Find participants



 Conduct the sessions

 Debrief with           observers
15




What’s essential?

 Develop a test plan

 Find participants

 Conduct the sessions

 Debrief with observers and decide together
16




Sit next to someone.
Watch them do stuff.
17




Story:
18




California.
19




1 day
20




a ballot and instructions
21




a ballot and instructions
22




random participants
23




no helpers
24




results due: today
25




What to do?
26




Improvise: Ground rules


✤ Say “yes.”

✤ Trust your team and your participants

✤ Don’t let the team or stakeholders block

✤ Work to the top of your collective intelligence
27




How we got value


✤ Used the ballots and instructions at hand

✤ Focused on one thing at a time

✤ Had participants generate and collect data

✤ Drafted observers and debriefed
28




More examples
29




Cafe
30




Company
 lobby
31




Trade
show



        http://www.flickr.com/photos/dans180/72408664/
32




Listen in
   on
customer
 service
33




Transit
 seat
mates
34




Remote
testing
35


Browser history
Gives a recent snapshot of activity




Bookmarks & favorites
Shows deliberate returns to sites




Email trails
Builds a story
Shows organizing strategies
36


Customer service logs
View on persistent, difficult problems




Sales reps
Proxies for buyers, end users




User forums & wikis
Frames the conversations users are having
37




Value
38


            Each phase includes input
            from users

            Multidisciplinary teams

            Enlightened management

            Willingness to learn as you go

            Defined usability goals and
            objectives




Supporting usable design
39


            Each phase includes input
            from users

            Multidisciplinary teams

            Enlightened management

            Willingness to learn as you go

            Defined usability goals and
            objectives




Supporting usable design
40




Where’s the value in testing?

                    70% watching someone use
                    the design

                    20% working with the team
                    to prepare to test

                    8% discussing what
                    happened
41




Getting value in the wild

✤ Use what’s at hand

✤ Narrow the scope of the test

✤ Focus on

  ✤ what can make the most difference to the
    most users

  ✤ what can be implemented easily with the
    resources available
42




                         ✤ If you need

                           ✤ summative
                             data


When is testing in the     ✤ benchmarks

                           ✤ answers to

  wild not valuable?         hard
                             problems
43




Risks
44




Risks of testing in the wild


✤ Participant sample may be too small,
  biased

✤ Inconsistent approach may net
  inconsistent data
45




   What do you lose?
                                                        ✤ Quantitative data
                                                        ✤ Rigor
                                                        ✤ Relatively unbiased sample, maybe




http://drb.lifestreamcenter.net/Lessons/process_maps/
46




Trade offs
47




What do you need?

✤ To inform a design:

   Testing in the wild   Classic usability testing
   Qualitative data      Quantitative data
   Opportunity           Planning
   Fits the schedule     Don’t know how to fit UT into
                         the schedule
   Just in time          After the fact
   $                     $$$$
   Something             Maybe nothing
48




Where’s the ROI?


✤ Qualitative data

✤ Opportunity

✤ Fitting into a schedule

✤ Timeliness
49


So far.
You don’t have to do it   Value of usability   Tradeoff: having
by the book               testing              some data over
                                               having no data
                          + observing users

                          + working with the
                          team
50


Coming up.
What’s the bare   Steps for testing in   What to add if you
minimum           the wild               have resources
51




Essentials
52




What you need


                                             Somebody
                                              (Human)
✤ Someone who will try the design
                                    Something           Some place
✤ Somewhere to test                  (Activity)          (Context)


✤ Something to study
53




                      1. Plan, minimally

                      2. Get the team on
                         board

                      3. Design the test,
                         minimally

                      4. Recruit
                         participants

                      5. Conduct
                         sessions

Follow these steps:   6. Debrief and
                         decide
54




Plan, minimally
✤ What

✤ Why

✤ Who

✤ When

✤ Where
55




Plan, minimally: Example

✤ What: near-final design

✤ Why: inform user training and support

✤ Who: inexperienced customers

✤ When: the end of the week

✤ Where: trade show at user group meeting
56




 Minimalist plan
                       Find out whether information about admission is
Goals and objectives
                       easy to find and use


Participant
                       People who have college-bound kids
characteristics


Description of         Sit-by parents attending a high school basketball game, each
method                 trying three university sites with the same scavenger hunt task

                       Find out how and when applications are due
List of tasks          Determine whether there are fees for applying
                       Learn when acceptances will be sent
57



Get the team
on board
✤ Visualize the desired user
  experience

✤ Share the intellectual property of
  observing users
58



Design the
test, minimally
✤ Why are you testing?

✤ What questions are you trying to
  answer?

✤ What constrains the design?

✤ What are you going to do with
  what you find out?
59



Recruit
participants
✤ Convenience sample

✤ Requirements, not
  demographics
60

                          ✤ Staff not on the
                            design project

                          ✤ Friends and family

                          ✤ Personal,
                            professional
                            networks

                          ✤ Online social
                            networks

                          ✤ Community
                            organizations

                          ✤ Online classifieds

                          ✤ Association,
                            society, user
                            group, union
Sources of participants     contacts

                          ✤ Temp agencies
61



Conduct
sessions
✤ Rehearse (P1)

✤ Interview-based tasks (or based
  on previous field work)

✤ Explore in short, focused
  sessions

✤ Iterate test design and product
  design
62




                  Greet the
                  participant

                  Explain the study,
                  your role, and their
                  role

                  Interview (maybe)

                  Do tasks from
                  interview

                  Debrief with
                  participant

Session outline   Debrief with
                  observers
63




                           ✤ Impartial,
                             unbiased
                             observing

                           ✤ No teaching!

                           ✤ Listen and
                             watch

                           ✤ Ask open-
                             ended
                             questions:
Moderating, not training     Why? How?
                             What?
64



Debrief and
decide
✤ Write up issues on sticky notes
  and sort them into priority lists

✤ Ask for top 10 items -- base on
  data and observations, not
  opinion

✤ Take a vote for priorities
65




Nice to have
66




                        ✤ Screened,
                          scheduled
                          participants

                        ✤ Official
                          paperwork


Add these ingredients
                        ✤ Recordings

                        ✤ More observers
67




Participants

What                      Why




appropriate experience    greater confidence in data




scheduled ahead of time   easier to get data
68




Paperwork
What                      Why


script                    ensure consistent instructions and moderating

consent forms             official acknowledgment for taking part

recording waivers         permission for recordings

pre-test questionnaire    experience, knowledge, value

post-test questionnaire   feedback on tasks, UI
69




Recordings

What                     Why



audio                    transcripts, verbal protocol analysis



video or photos          stories, double-check, highlights



Morae, Camtasia, other   digital data, automated collection
70




More observers
✤ Better-informed design recommendations

  What                  Why


  developers            technology boundaries


  management, execs     business priorities
71




                                               Nutshell




http://www.flickr.com/photos/stoan/145497333/
72




             ✤ Value:
               observing,
               planning

             ✤ Plan: 4 steps

             ✤ Recruit:
               behavior

             ✤ Moderating:
Take away:     listen, observe
73


Quick.                   Cheap.                   Insightful.
You can compensate       Low risk, little money   Value comes from
for some                                          getting as close as
shortcomings, and just                            possible to what real
test more                                         people are doing
                                                  with your design
74




Where to learn more


    Dana’s blog: http://
    usabilitytestinghowto.blogspot.com/

    Download templates, examples, and
    links to other resources from
    www.wiley.com/go/usabilitytesting
75




Me


     Dana Chisnell
     dana@usabilityworks.net

     www.usabilityworks.net
     415.519.1148

More Related Content

What's hot

Removing the Systemic Project Barriers
Removing the Systemic Project BarriersRemoving the Systemic Project Barriers
Removing the Systemic Project Barriers
Jorvig Consulting Inc.
 
Analy probsolv gsw
Analy probsolv gswAnaly probsolv gsw
Analy probsolv gsw
woznite65
 
Discovery Kanban @ LKSE15
Discovery Kanban @ LKSE15Discovery Kanban @ LKSE15
Discovery Kanban @ LKSE15
Patrick Steyaert
 
Wit magazine april_2008
Wit magazine april_2008Wit magazine april_2008
Wit magazine april_2008
Vipul Kocher
 
The Essential Product Owner - Partnering with the team
The Essential Product Owner - Partnering with the teamThe Essential Product Owner - Partnering with the team
The Essential Product Owner - Partnering with the team
Cprime
 
B. den Haak. How to make OKRs Lean Again
B. den Haak. How to make OKRs Lean AgainB. den Haak. How to make OKRs Lean Again
B. den Haak. How to make OKRs Lean Again
Agile Lietuva
 
Real developers-dont-need-unit-tests
Real developers-dont-need-unit-testsReal developers-dont-need-unit-tests
Real developers-dont-need-unit-tests
Skills Matter
 
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
Dubai Quality Group
 
Root cause analysis
Root cause analysisRoot cause analysis
Root cause analysis
Simmy Sharma
 
Woolcock Institute 20 Mar 2012
Woolcock Institute 20 Mar 2012Woolcock Institute 20 Mar 2012
Woolcock Institute 20 Mar 2012
Alex Holcombe
 
Coming Together: integrating industrial design and interaction design
Coming Together: integrating industrial design and interaction designComing Together: integrating industrial design and interaction design
Coming Together: integrating industrial design and interaction design
Joannes Vandermeulen
 
2009 06 01 The Lean Startup Texas Edition
2009 06 01 The Lean Startup Texas Edition2009 06 01 The Lean Startup Texas Edition
2009 06 01 The Lean Startup Texas Edition
Eric Ries
 
About us groupaccelerators_ase
About us groupaccelerators_aseAbout us groupaccelerators_ase
About us groupaccelerators_ase
CapIpad
 
Root cause analysis common problems and solutions
Root cause analysis common problems and solutions Root cause analysis common problems and solutions
Root cause analysis common problems and solutions
ASQ Reliability Division
 
Jens Østergaard on Why Scrum Is So Hard
Jens Østergaard on Why Scrum Is So HardJens Østergaard on Why Scrum Is So Hard
Jens Østergaard on Why Scrum Is So Hard
Marko Gargenta
 
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
Bosnia Agile
 

What's hot (16)

Removing the Systemic Project Barriers
Removing the Systemic Project BarriersRemoving the Systemic Project Barriers
Removing the Systemic Project Barriers
 
Analy probsolv gsw
Analy probsolv gswAnaly probsolv gsw
Analy probsolv gsw
 
Discovery Kanban @ LKSE15
Discovery Kanban @ LKSE15Discovery Kanban @ LKSE15
Discovery Kanban @ LKSE15
 
Wit magazine april_2008
Wit magazine april_2008Wit magazine april_2008
Wit magazine april_2008
 
The Essential Product Owner - Partnering with the team
The Essential Product Owner - Partnering with the teamThe Essential Product Owner - Partnering with the team
The Essential Product Owner - Partnering with the team
 
B. den Haak. How to make OKRs Lean Again
B. den Haak. How to make OKRs Lean AgainB. den Haak. How to make OKRs Lean Again
B. den Haak. How to make OKRs Lean Again
 
Real developers-dont-need-unit-tests
Real developers-dont-need-unit-testsReal developers-dont-need-unit-tests
Real developers-dont-need-unit-tests
 
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
 
Root cause analysis
Root cause analysisRoot cause analysis
Root cause analysis
 
Woolcock Institute 20 Mar 2012
Woolcock Institute 20 Mar 2012Woolcock Institute 20 Mar 2012
Woolcock Institute 20 Mar 2012
 
Coming Together: integrating industrial design and interaction design
Coming Together: integrating industrial design and interaction designComing Together: integrating industrial design and interaction design
Coming Together: integrating industrial design and interaction design
 
2009 06 01 The Lean Startup Texas Edition
2009 06 01 The Lean Startup Texas Edition2009 06 01 The Lean Startup Texas Edition
2009 06 01 The Lean Startup Texas Edition
 
About us groupaccelerators_ase
About us groupaccelerators_aseAbout us groupaccelerators_ase
About us groupaccelerators_ase
 
Root cause analysis common problems and solutions
Root cause analysis common problems and solutions Root cause analysis common problems and solutions
Root cause analysis common problems and solutions
 
Jens Østergaard on Why Scrum Is So Hard
Jens Østergaard on Why Scrum Is So HardJens Østergaard on Why Scrum Is So Hard
Jens Østergaard on Why Scrum Is So Hard
 
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
10 tips how to make your Scrum fail - or succeed if you want by Igor Parac
 

Similar to Quick Cheap Insightful: Usability testing in the wild

User testing presentation
User testing presentationUser testing presentation
User testing presentation
LindaKolker
 
Action research for_librarians_carl2012
Action research for_librarians_carl2012Action research for_librarians_carl2012
Action research for_librarians_carl2012
srosenblatt
 
Action research for_librarians_carl2012
Action research for_librarians_carl2012Action research for_librarians_carl2012
Action research for_librarians_carl2012
srosenblatt
 
Usability Testing Fundamentals
Usability Testing FundamentalsUsability Testing Fundamentals
Usability Testing Fundamentals
debcook
 
Usability Testing Fundamentals
Usability Testing FundamentalsUsability Testing Fundamentals
Usability Testing Fundamentals
debcook
 
Better UX Surveys at UCD2012 by @cjforms
Better UX Surveys at UCD2012 by @cjformsBetter UX Surveys at UCD2012 by @cjforms
Better UX Surveys at UCD2012 by @cjforms
Caroline Jarrett
 
Executing for Every Screen: Build, launch and sustain products for your custo...
Executing for Every Screen: Build, launch and sustain products for your custo...Executing for Every Screen: Build, launch and sustain products for your custo...
Executing for Every Screen: Build, launch and sustain products for your custo...
Steven Hoober
 
Getting Started With User Research, Presented at Agile2010
Getting Started With User Research, Presented at Agile2010Getting Started With User Research, Presented at Agile2010
Getting Started With User Research, Presented at Agile2010
Carol Smith
 
Rapid Project Inception
Rapid Project InceptionRapid Project Inception
Rapid Project Inception
udairaj
 
Process affordances-workshop
Process affordances-workshopProcess affordances-workshop
Process affordances-workshop
drewz lin
 
2012 icse program comprehension
2012 icse program comprehension2012 icse program comprehension
2012 icse program comprehension
Walid Maalej
 
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
Dave Healey
 
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
Dan Arra
 
Process improvement guide
Process improvement guideProcess improvement guide
Process improvement guide
Eng Marzouk
 
Usability testing – Just Do It. Five methods for improving usability in-house
Usability testing – Just Do It. Five methods for improving usability in-houseUsability testing – Just Do It. Five methods for improving usability in-house
Usability testing – Just Do It. Five methods for improving usability in-house
Volkside
 
Prototype and User Test
Prototype and User TestPrototype and User Test
Prototype and User Test
David Gelb
 
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
Login Conference LLC
 
How to increase conversion with usability testing
How to increase conversion with usability testingHow to increase conversion with usability testing
How to increase conversion with usability testing
Danny Setiawan
 
Usability Testing for People w/ Disabilities
Usability Testing for People w/ DisabilitiesUsability Testing for People w/ Disabilities
Usability Testing for People w/ Disabilities
Interactive Accessibility
 
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
Gigi Johnson
 

Similar to Quick Cheap Insightful: Usability testing in the wild (20)

User testing presentation
User testing presentationUser testing presentation
User testing presentation
 
Action research for_librarians_carl2012
Action research for_librarians_carl2012Action research for_librarians_carl2012
Action research for_librarians_carl2012
 
Action research for_librarians_carl2012
Action research for_librarians_carl2012Action research for_librarians_carl2012
Action research for_librarians_carl2012
 
Usability Testing Fundamentals
Usability Testing FundamentalsUsability Testing Fundamentals
Usability Testing Fundamentals
 
Usability Testing Fundamentals
Usability Testing FundamentalsUsability Testing Fundamentals
Usability Testing Fundamentals
 
Better UX Surveys at UCD2012 by @cjforms
Better UX Surveys at UCD2012 by @cjformsBetter UX Surveys at UCD2012 by @cjforms
Better UX Surveys at UCD2012 by @cjforms
 
Executing for Every Screen: Build, launch and sustain products for your custo...
Executing for Every Screen: Build, launch and sustain products for your custo...Executing for Every Screen: Build, launch and sustain products for your custo...
Executing for Every Screen: Build, launch and sustain products for your custo...
 
Getting Started With User Research, Presented at Agile2010
Getting Started With User Research, Presented at Agile2010Getting Started With User Research, Presented at Agile2010
Getting Started With User Research, Presented at Agile2010
 
Rapid Project Inception
Rapid Project InceptionRapid Project Inception
Rapid Project Inception
 
Process affordances-workshop
Process affordances-workshopProcess affordances-workshop
Process affordances-workshop
 
2012 icse program comprehension
2012 icse program comprehension2012 icse program comprehension
2012 icse program comprehension
 
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
SharePoint and Lean Development: Critical Factors for Accelerating Time to Va...
 
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
Macadamian - Product Camp - Top10 researchsecretswebinar dan_arra_march,2013
 
Process improvement guide
Process improvement guideProcess improvement guide
Process improvement guide
 
Usability testing – Just Do It. Five methods for improving usability in-house
Usability testing – Just Do It. Five methods for improving usability in-houseUsability testing – Just Do It. Five methods for improving usability in-house
Usability testing – Just Do It. Five methods for improving usability in-house
 
Prototype and User Test
Prototype and User TestPrototype and User Test
Prototype and User Test
 
Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012Jesse Hull - LOGIN Conference 2012
Jesse Hull - LOGIN Conference 2012
 
How to increase conversion with usability testing
How to increase conversion with usability testingHow to increase conversion with usability testing
How to increase conversion with usability testing
 
Usability Testing for People w/ Disabilities
Usability Testing for People w/ DisabilitiesUsability Testing for People w/ Disabilities
Usability Testing for People w/ Disabilities
 
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
Messy Research: How to Make Qualitative Data Quantifiable and Make Messy Data...
 

More from Dana Chisnell

Next generation of civic designers
Next generation of civic designersNext generation of civic designers
Next generation of civic designers
Dana Chisnell
 
Making Voting Accessible
Making Voting Accessible Making Voting Accessible
Making Voting Accessible
Dana Chisnell
 
Answering voters' questions at county election websites
Answering voters' questions at county election websites Answering voters' questions at county election websites
Answering voters' questions at county election websites
Dana Chisnell
 
Fooled by best practice
Fooled by best practiceFooled by best practice
Fooled by best practice
Dana Chisnell
 
Don't make me read! Less is more in information for voters
Don't make me read! Less is more in information for votersDon't make me read! Less is more in information for voters
Don't make me read! Less is more in information for voters
Dana Chisnell
 
I can't find what i'm looking for
I can't find what i'm looking forI can't find what i'm looking for
I can't find what i'm looking for
Dana Chisnell
 
Usability and security in future voting systems
Usability and security in future voting systemsUsability and security in future voting systems
Usability and security in future voting systems
Dana Chisnell
 
Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants
Dana Chisnell
 
Looking across the voter experience
Looking across the voter experienceLooking across the voter experience
Looking across the voter experience
Dana Chisnell
 
Anywhere Ballot iterations
Anywhere Ballot iterationsAnywhere Ballot iterations
Anywhere Ballot iterations
Dana Chisnell
 
Iterations paper prototype
Iterations   paper prototypeIterations   paper prototype
Iterations paper prototype
Dana Chisnell
 
Rethinking user research for social web
Rethinking user research for social webRethinking user research for social web
Rethinking user research for social web
Dana Chisnell
 
Persona modeler
Persona modelerPersona modeler
Persona modeler
Dana Chisnell
 
Usability of County Election Websites
Usability of County Election WebsitesUsability of County Election Websites
Usability of County Election Websites
Dana Chisnell
 
Usability and security in future voting systems
Usability and security in future voting systemsUsability and security in future voting systems
Usability and security in future voting systems
Dana Chisnell
 
Rethinking user research for social web
Rethinking user research for social webRethinking user research for social web
Rethinking user research for social web
Dana Chisnell
 
Making sense of the data
Making sense of the dataMaking sense of the data
Making sense of the data
Dana Chisnell
 
The users at the end of the mouse - using the 3As to model personas
The users at the end of the mouse - using the 3As to model personasThe users at the end of the mouse - using the 3As to model personas
The users at the end of the mouse - using the 3As to model personas
Dana Chisnell
 
Democracy is a design problem
Democracy is a design problemDemocracy is a design problem
Democracy is a design problem
Dana Chisnell
 
Counter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methodsCounter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methods
Dana Chisnell
 

More from Dana Chisnell (20)

Next generation of civic designers
Next generation of civic designersNext generation of civic designers
Next generation of civic designers
 
Making Voting Accessible
Making Voting Accessible Making Voting Accessible
Making Voting Accessible
 
Answering voters' questions at county election websites
Answering voters' questions at county election websites Answering voters' questions at county election websites
Answering voters' questions at county election websites
 
Fooled by best practice
Fooled by best practiceFooled by best practice
Fooled by best practice
 
Don't make me read! Less is more in information for voters
Don't make me read! Less is more in information for votersDon't make me read! Less is more in information for voters
Don't make me read! Less is more in information for voters
 
I can't find what i'm looking for
I can't find what i'm looking forI can't find what i'm looking for
I can't find what i'm looking for
 
Usability and security in future voting systems
Usability and security in future voting systemsUsability and security in future voting systems
Usability and security in future voting systems
 
Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants
 
Looking across the voter experience
Looking across the voter experienceLooking across the voter experience
Looking across the voter experience
 
Anywhere Ballot iterations
Anywhere Ballot iterationsAnywhere Ballot iterations
Anywhere Ballot iterations
 
Iterations paper prototype
Iterations   paper prototypeIterations   paper prototype
Iterations paper prototype
 
Rethinking user research for social web
Rethinking user research for social webRethinking user research for social web
Rethinking user research for social web
 
Persona modeler
Persona modelerPersona modeler
Persona modeler
 
Usability of County Election Websites
Usability of County Election WebsitesUsability of County Election Websites
Usability of County Election Websites
 
Usability and security in future voting systems
Usability and security in future voting systemsUsability and security in future voting systems
Usability and security in future voting systems
 
Rethinking user research for social web
Rethinking user research for social webRethinking user research for social web
Rethinking user research for social web
 
Making sense of the data
Making sense of the dataMaking sense of the data
Making sense of the data
 
The users at the end of the mouse - using the 3As to model personas
The users at the end of the mouse - using the 3As to model personasThe users at the end of the mouse - using the 3As to model personas
The users at the end of the mouse - using the 3As to model personas
 
Democracy is a design problem
Democracy is a design problemDemocracy is a design problem
Democracy is a design problem
 
Counter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methodsCounter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methods
 

Recently uploaded

一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
02tygie
 
一比一原版美国旧金山大学毕业证(USF学位证)如何办理
一比一原版美国旧金山大学毕业证(USF学位证)如何办理一比一原版美国旧金山大学毕业证(USF学位证)如何办理
一比一原版美国旧金山大学毕业证(USF学位证)如何办理
xnhwr8v
 
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
67n7f53
 
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
t34zod9l
 
一比一原版肯特大学毕业证UKC成绩单一模一样
一比一原版肯特大学毕业证UKC成绩单一模一样一比一原版肯特大学毕业证UKC成绩单一模一样
一比一原版肯特大学毕业证UKC成绩单一模一样
tobbk6s8
 
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
f22b6g9c
 
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
hw2xf1m
 
UXpert_Report (UALR Mapping Renewal 2022).pdf
UXpert_Report (UALR Mapping Renewal 2022).pdfUXpert_Report (UALR Mapping Renewal 2022).pdf
UXpert_Report (UALR Mapping Renewal 2022).pdf
anthonylin333
 
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
3vgr39kx
 
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
k7nm6tk
 
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
bz42w9z0
 
一比一原版布兰登大学毕业证(BU毕业证书)如何办理
一比一原版布兰登大学毕业证(BU毕业证书)如何办理一比一原版布兰登大学毕业证(BU毕业证书)如何办理
一比一原版布兰登大学毕业证(BU毕业证书)如何办理
wkip62b
 
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
zv943dhb
 
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
wkip62b
 
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
21uul8se
 
Practical eLearning Makeovers for Everyone
Practical eLearning Makeovers for EveryonePractical eLearning Makeovers for Everyone
Practical eLearning Makeovers for Everyone
Bianca Woods
 
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
aonx8o5f
 
NHR Engineers Portfolio 2023 2024 NISHANT RATHI
NHR Engineers Portfolio 2023 2024 NISHANT RATHINHR Engineers Portfolio 2023 2024 NISHANT RATHI
NHR Engineers Portfolio 2023 2024 NISHANT RATHI
NishantRathi18
 
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
340qn0m1
 
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
mfria419
 

Recently uploaded (20)

一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
一比一原版美国加州大学欧文分校毕业证(UCI学位证)如何办理
 
一比一原版美国旧金山大学毕业证(USF学位证)如何办理
一比一原版美国旧金山大学毕业证(USF学位证)如何办理一比一原版美国旧金山大学毕业证(USF学位证)如何办理
一比一原版美国旧金山大学毕业证(USF学位证)如何办理
 
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
一比一原版(CSU毕业证书)查尔斯特大学毕业证如何办理
 
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
一比一原版(UWS毕业证)澳洲西悉尼大学毕业证如何办理
 
一比一原版肯特大学毕业证UKC成绩单一模一样
一比一原版肯特大学毕业证UKC成绩单一模一样一比一原版肯特大学毕业证UKC成绩单一模一样
一比一原版肯特大学毕业证UKC成绩单一模一样
 
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
一比一原版(UoN毕业证书)纽卡斯尔大学毕业证如何办理
 
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
原版制作(MDIS毕业证书)新加坡管理发展学院毕业证学位证一模一样
 
UXpert_Report (UALR Mapping Renewal 2022).pdf
UXpert_Report (UALR Mapping Renewal 2022).pdfUXpert_Report (UALR Mapping Renewal 2022).pdf
UXpert_Report (UALR Mapping Renewal 2022).pdf
 
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
一比一原版南安普顿索伦特大学毕业证Southampton成绩单一模一样
 
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
一比一原版(LSBU毕业证书)伦敦南岸大学毕业证如何办理
 
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
一比一原版澳洲科廷科技大学毕业证(Curtin毕业证)如何办理
 
一比一原版布兰登大学毕业证(BU毕业证书)如何办理
一比一原版布兰登大学毕业证(BU毕业证书)如何办理一比一原版布兰登大学毕业证(BU毕业证书)如何办理
一比一原版布兰登大学毕业证(BU毕业证书)如何办理
 
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
一比一原版(UCB毕业证)英国伯明翰大学学院毕业证如何办理
 
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
一比一原版(UC毕业证书)堪培拉大学毕业证如何办理
 
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
一比一原版(NU毕业证书)诺森比亚大学毕业证如何办理
 
Practical eLearning Makeovers for Everyone
Practical eLearning Makeovers for EveryonePractical eLearning Makeovers for Everyone
Practical eLearning Makeovers for Everyone
 
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
一比一原版(Hull毕业证)英国哈珀亚当斯大学毕业证如何办理
 
NHR Engineers Portfolio 2023 2024 NISHANT RATHI
NHR Engineers Portfolio 2023 2024 NISHANT RATHINHR Engineers Portfolio 2023 2024 NISHANT RATHI
NHR Engineers Portfolio 2023 2024 NISHANT RATHI
 
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
一比一原版(LSE毕业证书)伦敦政治经济学院毕业证如何办理
 
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
一比一原版(Teesside毕业证)英国提赛德大学毕业证如何办理
 

Quick Cheap Insightful: Usability testing in the wild

  • 1. 1 The quick, the cheap, and the insightful Usability testing in the wild Dana Chisnell San Francisco STC November 2008
  • 2. 2 What is a usability test?
  • 4. 4
  • 5. 5 Quick, cheap, insightful ✤ Minimal steps to focus on ✤ Where the value comes from in doing usability tests ✤ Where it may be risky to go minimalist ✤ How to think about the trade-offs ✤ What’s essential ✤ What might be nice to have
  • 7. 7 Classic, with everything ✤ Chapter 5. Develop a test plan ✤ Chapter 6. Choose a testing environment ✤ Chapter 7. Find and select participants ✤ Chapter 8. Prepare test materials ✤ Chapter 9. Conduct the sessions ✤ Chapter 10. Debrief with participants and observers ✤ Chapter 11. Analyze data and observations ✤ Chapter 12. Create findings and recommendations
  • 8. 8 Photo courtesy of Tom Tullis
  • 9. 9 Classic, with everything Can take a lot of time: weeks or months.
  • 10. 10 What if you have to do something quick? and cheap?
  • 11. 11 You don’t have to do it by the book.
  • 12. 12 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create findings and recommendations
  • 13. 13 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create findings and recommendations
  • 14. 14 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers
  • 15. 15 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers and decide together
  • 16. 16 Sit next to someone. Watch them do stuff.
  • 20. 20 a ballot and instructions
  • 21. 21 a ballot and instructions
  • 26. 26 Improvise: Ground rules ✤ Say “yes.” ✤ Trust your team and your participants ✤ Don’t let the team or stakeholders block ✤ Work to the top of your collective intelligence
  • 27. 27 How we got value ✤ Used the ballots and instructions at hand ✤ Focused on one thing at a time ✤ Had participants generate and collect data ✤ Drafted observers and debriefed
  • 31. 31 Trade show http://www.flickr.com/photos/dans180/72408664/
  • 32. 32 Listen in on customer service
  • 35. 35 Browser history Gives a recent snapshot of activity Bookmarks & favorites Shows deliberate returns to sites Email trails Builds a story Shows organizing strategies
  • 36. 36 Customer service logs View on persistent, difficult problems Sales reps Proxies for buyers, end users User forums & wikis Frames the conversations users are having
  • 38. 38 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Defined usability goals and objectives Supporting usable design
  • 39. 39 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Defined usability goals and objectives Supporting usable design
  • 40. 40 Where’s the value in testing? 70% watching someone use the design 20% working with the team to prepare to test 8% discussing what happened
  • 41. 41 Getting value in the wild ✤ Use what’s at hand ✤ Narrow the scope of the test ✤ Focus on ✤ what can make the most difference to the most users ✤ what can be implemented easily with the resources available
  • 42. 42 ✤ If you need ✤ summative data When is testing in the ✤ benchmarks ✤ answers to wild not valuable? hard problems
  • 44. 44 Risks of testing in the wild ✤ Participant sample may be too small, biased ✤ Inconsistent approach may net inconsistent data
  • 45. 45 What do you lose? ✤ Quantitative data ✤ Rigor ✤ Relatively unbiased sample, maybe http://drb.lifestreamcenter.net/Lessons/process_maps/
  • 47. 47 What do you need? ✤ To inform a design: Testing in the wild Classic usability testing Qualitative data Quantitative data Opportunity Planning Fits the schedule Don’t know how to fit UT into the schedule Just in time After the fact $ $$$$ Something Maybe nothing
  • 48. 48 Where’s the ROI? ✤ Qualitative data ✤ Opportunity ✤ Fitting into a schedule ✤ Timeliness
  • 49. 49 So far. You don’t have to do it Value of usability Tradeoff: having by the book testing some data over having no data + observing users + working with the team
  • 50. 50 Coming up. What’s the bare Steps for testing in What to add if you minimum the wild have resources
  • 52. 52 What you need Somebody (Human) ✤ Someone who will try the design Something Some place ✤ Somewhere to test (Activity) (Context) ✤ Something to study
  • 53. 53 1. Plan, minimally 2. Get the team on board 3. Design the test, minimally 4. Recruit participants 5. Conduct sessions Follow these steps: 6. Debrief and decide
  • 54. 54 Plan, minimally ✤ What ✤ Why ✤ Who ✤ When ✤ Where
  • 55. 55 Plan, minimally: Example ✤ What: near-final design ✤ Why: inform user training and support ✤ Who: inexperienced customers ✤ When: the end of the week ✤ Where: trade show at user group meeting
  • 56. 56 Minimalist plan Find out whether information about admission is Goals and objectives easy to find and use Participant People who have college-bound kids characteristics Description of Sit-by parents attending a high school basketball game, each method trying three university sites with the same scavenger hunt task Find out how and when applications are due List of tasks Determine whether there are fees for applying Learn when acceptances will be sent
  • 57. 57 Get the team on board ✤ Visualize the desired user experience ✤ Share the intellectual property of observing users
  • 58. 58 Design the test, minimally ✤ Why are you testing? ✤ What questions are you trying to answer? ✤ What constrains the design? ✤ What are you going to do with what you find out?
  • 59. 59 Recruit participants ✤ Convenience sample ✤ Requirements, not demographics
  • 60. 60 ✤ Staff not on the design project ✤ Friends and family ✤ Personal, professional networks ✤ Online social networks ✤ Community organizations ✤ Online classifieds ✤ Association, society, user group, union Sources of participants contacts ✤ Temp agencies
  • 61. 61 Conduct sessions ✤ Rehearse (P1) ✤ Interview-based tasks (or based on previous field work) ✤ Explore in short, focused sessions ✤ Iterate test design and product design
  • 62. 62 Greet the participant Explain the study, your role, and their role Interview (maybe) Do tasks from interview Debrief with participant Session outline Debrief with observers
  • 63. 63 ✤ Impartial, unbiased observing ✤ No teaching! ✤ Listen and watch ✤ Ask open- ended questions: Moderating, not training Why? How? What?
  • 64. 64 Debrief and decide ✤ Write up issues on sticky notes and sort them into priority lists ✤ Ask for top 10 items -- base on data and observations, not opinion ✤ Take a vote for priorities
  • 66. 66 ✤ Screened, scheduled participants ✤ Official paperwork Add these ingredients ✤ Recordings ✤ More observers
  • 67. 67 Participants What Why appropriate experience greater confidence in data scheduled ahead of time easier to get data
  • 68. 68 Paperwork What Why script ensure consistent instructions and moderating consent forms official acknowledgment for taking part recording waivers permission for recordings pre-test questionnaire experience, knowledge, value post-test questionnaire feedback on tasks, UI
  • 69. 69 Recordings What Why audio transcripts, verbal protocol analysis video or photos stories, double-check, highlights Morae, Camtasia, other digital data, automated collection
  • 70. 70 More observers ✤ Better-informed design recommendations What Why developers technology boundaries management, execs business priorities
  • 71. 71 Nutshell http://www.flickr.com/photos/stoan/145497333/
  • 72. 72 ✤ Value: observing, planning ✤ Plan: 4 steps ✤ Recruit: behavior ✤ Moderating: Take away: listen, observe
  • 73. 73 Quick. Cheap. Insightful. You can compensate Low risk, little money Value comes from for some getting as close as shortcomings, and just possible to what real test more people are doing with your design
  • 74. 74 Where to learn more Dana’s blog: http:// usabilitytestinghowto.blogspot.com/ Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting
  • 75. 75 Me Dana Chisnell dana@usabilityworks.net www.usabilityworks.net 415.519.1148