• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Quick Cheap Insightful: Usability testing in the wild
 

Quick Cheap Insightful: Usability testing in the wild

on

  • 12,022 views

How to gather data to made design decisions on almost anywhere almost any time for very little money.

How to gather data to made design decisions on almost anywhere almost any time for very little money.

Statistics

Views

Total Views
12,022
Views on SlideShare
11,995
Embed Views
27

Actions

Likes
30
Downloads
287
Comments
0

4 Embeds 27

http://www.slideshare.net 22
https://twitter.com 3
http://paper.li 1
http://courtneybolton.parasoup.de 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Quick Cheap Insightful: Usability testing in the wild Quick Cheap Insightful: Usability testing in the wild Presentation Transcript

    • 1 The quick, the cheap, and the insightful Usability testing in the wild Dana Chisnell San Francisco STC November 2008
    • 2 What is a usability test?
    • 3 http://www.sigchi.org/chi97/proceedings/overview/tst.htm
    • 4
    • 5 Quick, cheap, insightful ✤ Minimal steps to focus on ✤ Where the value comes from in doing usability tests ✤ Where it may be risky to go minimalist ✤ How to think about the trade-offs ✤ What’s essential ✤ What might be nice to have
    • 6 Minimal steps
    • 7 Classic, with everything ✤ Chapter 5. Develop a test plan ✤ Chapter 6. Choose a testing environment ✤ Chapter 7. Find and select participants ✤ Chapter 8. Prepare test materials ✤ Chapter 9. Conduct the sessions ✤ Chapter 10. Debrief with participants and observers ✤ Chapter 11. Analyze data and observations ✤ Chapter 12. Create findings and recommendations
    • 8 Photo courtesy of Tom Tullis
    • 9 Classic, with everything Can take a lot of time: weeks or months.
    • 10 What if you have to do something quick? and cheap?
    • 11 You don’t have to do it by the book.
    • 12 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create findings and recommendations
    • 13 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create findings and recommendations
    • 14 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers
    • 15 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers and decide together
    • 16 Sit next to someone. Watch them do stuff.
    • 17 Story:
    • 18 California.
    • 19 1 day
    • 20 a ballot and instructions
    • 21 a ballot and instructions
    • 22 random participants
    • 23 no helpers
    • 24 results due: today
    • 25 What to do?
    • 26 Improvise: Ground rules ✤ Say “yes.” ✤ Trust your team and your participants ✤ Don’t let the team or stakeholders block ✤ Work to the top of your collective intelligence
    • 27 How we got value ✤ Used the ballots and instructions at hand ✤ Focused on one thing at a time ✤ Had participants generate and collect data ✤ Drafted observers and debriefed
    • 28 More examples
    • 29 Cafe
    • 30 Company lobby
    • 31 Trade show http://www.flickr.com/photos/dans180/72408664/
    • 32 Listen in on customer service
    • 33 Transit seat mates
    • 34 Remote testing
    • 35 Browser history Gives a recent snapshot of activity Bookmarks & favorites Shows deliberate returns to sites Email trails Builds a story Shows organizing strategies
    • 36 Customer service logs View on persistent, difficult problems Sales reps Proxies for buyers, end users User forums & wikis Frames the conversations users are having
    • 37 Value
    • 38 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Defined usability goals and objectives Supporting usable design
    • 39 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Defined usability goals and objectives Supporting usable design
    • 40 Where’s the value in testing? 70% watching someone use the design 20% working with the team to prepare to test 8% discussing what happened
    • 41 Getting value in the wild ✤ Use what’s at hand ✤ Narrow the scope of the test ✤ Focus on ✤ what can make the most difference to the most users ✤ what can be implemented easily with the resources available
    • 42 ✤ If you need ✤ summative data When is testing in the ✤ benchmarks ✤ answers to wild not valuable? hard problems
    • 43 Risks
    • 44 Risks of testing in the wild ✤ Participant sample may be too small, biased ✤ Inconsistent approach may net inconsistent data
    • 45 What do you lose? ✤ Quantitative data ✤ Rigor ✤ Relatively unbiased sample, maybe http://drb.lifestreamcenter.net/Lessons/process_maps/
    • 46 Trade offs
    • 47 What do you need? ✤ To inform a design: Testing in the wild Classic usability testing Qualitative data Quantitative data Opportunity Planning Fits the schedule Don’t know how to fit UT into the schedule Just in time After the fact $ $$$$ Something Maybe nothing
    • 48 Where’s the ROI? ✤ Qualitative data ✤ Opportunity ✤ Fitting into a schedule ✤ Timeliness
    • 49 So far. You don’t have to do it Value of usability Tradeoff: having by the book testing some data over having no data + observing users + working with the team
    • 50 Coming up. What’s the bare Steps for testing in What to add if you minimum the wild have resources
    • 51 Essentials
    • 52 What you need Somebody (Human) ✤ Someone who will try the design Something Some place ✤ Somewhere to test (Activity) (Context) ✤ Something to study
    • 53 1. Plan, minimally 2. Get the team on board 3. Design the test, minimally 4. Recruit participants 5. Conduct sessions Follow these steps: 6. Debrief and decide
    • 54 Plan, minimally ✤ What ✤ Why ✤ Who ✤ When ✤ Where
    • 55 Plan, minimally: Example ✤ What: near-final design ✤ Why: inform user training and support ✤ Who: inexperienced customers ✤ When: the end of the week ✤ Where: trade show at user group meeting
    • 56 Minimalist plan Find out whether information about admission is Goals and objectives easy to find and use Participant People who have college-bound kids characteristics Description of Sit-by parents attending a high school basketball game, each method trying three university sites with the same scavenger hunt task Find out how and when applications are due List of tasks Determine whether there are fees for applying Learn when acceptances will be sent
    • 57 Get the team on board ✤ Visualize the desired user experience ✤ Share the intellectual property of observing users
    • 58 Design the test, minimally ✤ Why are you testing? ✤ What questions are you trying to answer? ✤ What constrains the design? ✤ What are you going to do with what you find out?
    • 59 Recruit participants ✤ Convenience sample ✤ Requirements, not demographics
    • 60 ✤ Staff not on the design project ✤ Friends and family ✤ Personal, professional networks ✤ Online social networks ✤ Community organizations ✤ Online classifieds ✤ Association, society, user group, union Sources of participants contacts ✤ Temp agencies
    • 61 Conduct sessions ✤ Rehearse (P1) ✤ Interview-based tasks (or based on previous field work) ✤ Explore in short, focused sessions ✤ Iterate test design and product design
    • 62 Greet the participant Explain the study, your role, and their role Interview (maybe) Do tasks from interview Debrief with participant Session outline Debrief with observers
    • 63 ✤ Impartial, unbiased observing ✤ No teaching! ✤ Listen and watch ✤ Ask open- ended questions: Moderating, not training Why? How? What?
    • 64 Debrief and decide ✤ Write up issues on sticky notes and sort them into priority lists ✤ Ask for top 10 items -- base on data and observations, not opinion ✤ Take a vote for priorities
    • 65 Nice to have
    • 66 ✤ Screened, scheduled participants ✤ Official paperwork Add these ingredients ✤ Recordings ✤ More observers
    • 67 Participants What Why appropriate experience greater confidence in data scheduled ahead of time easier to get data
    • 68 Paperwork What Why script ensure consistent instructions and moderating consent forms official acknowledgment for taking part recording waivers permission for recordings pre-test questionnaire experience, knowledge, value post-test questionnaire feedback on tasks, UI
    • 69 Recordings What Why audio transcripts, verbal protocol analysis video or photos stories, double-check, highlights Morae, Camtasia, other digital data, automated collection
    • 70 More observers ✤ Better-informed design recommendations What Why developers technology boundaries management, execs business priorities
    • 71 Nutshell http://www.flickr.com/photos/stoan/145497333/
    • 72 ✤ Value: observing, planning ✤ Plan: 4 steps ✤ Recruit: behavior ✤ Moderating: Take away: listen, observe
    • 73 Quick. Cheap. Insightful. You can compensate Low risk, little money Value comes from for some getting as close as shortcomings, and just possible to what real test more people are doing with your design
    • 74 Where to learn more Dana’s blog: http:// usabilitytestinghowto.blogspot.com/ Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting
    • 75 Me Dana Chisnell dana@usabilityworks.net www.usabilityworks.net 415.519.1148