Quick Cheap Insightful: Usability testing in the wild

1 The quick, the cheap, and the insightful Usability testing in the wild Dana Chisnell San Francisco STC November 2008

2 What is a usability test?

3 http://www.sigchi.org/chi97/proceedings/overview/tst.htm

5 Quick, cheap, insightful ✤ Minimal steps to focus on ✤ Where the value comes from in doing usability tests ✤ Where it may be risky to go minimalist ✤ How to think about the trade-offs ✤ What’s essential ✤ What might be nice to have

6 Minimal steps

7 Classic, with everything ✤ Chapter 5. Develop a test plan ✤ Chapter 6. Choose a testing environment ✤ Chapter 7. Find and select participants ✤ Chapter 8. Prepare test materials ✤ Chapter 9. Conduct the sessions ✤ Chapter 10. Debrief with participants and observers ✤ Chapter 11. Analyze data and observations ✤ Chapter 12. Create ﬁndings and recommendations

8 Photo courtesy of Tom Tullis

9 Classic, with everything Can take a lot of time: weeks or months.

10 What if you have to do something quick? and cheap?

11 You don’t have to do it by the book.

12 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create ﬁndings and recommendations

13 What’s essential? Develop a test plan Choose a testing environment Find and select participants Prepare test materials Conduct the sessions Debrief with participants and observers Analyze data and observations Create ﬁndings and recommendations

14 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers

15 What’s essential? Develop a test plan Find participants Conduct the sessions Debrief with observers and decide together

16 Sit next to someone. Watch them do stuff.

17 Story:

18 California.

19 1 day

20 a ballot and instructions

21 a ballot and instructions

22 random participants

23 no helpers

24 results due: today

25 What to do?

26 Improvise: Ground rules ✤ Say “yes.” ✤ Trust your team and your participants ✤ Don’t let the team or stakeholders block ✤ Work to the top of your collective intelligence

27 How we got value ✤ Used the ballots and instructions at hand ✤ Focused on one thing at a time ✤ Had participants generate and collect data ✤ Drafted observers and debriefed

28 More examples

29 Cafe

30 Company lobby

31 Trade show http://www.ﬂickr.com/photos/dans180/72408664/

32 Listen in on customer service

33 Transit seat mates

34 Remote testing

35 Browser history Gives a recent snapshot of activity Bookmarks & favorites Shows deliberate returns to sites Email trails Builds a story Shows organizing strategies

36 Customer service logs View on persistent, difﬁcult problems Sales reps Proxies for buyers, end users User forums & wikis Frames the conversations users are having

37 Value

38 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Deﬁned usability goals and objectives Supporting usable design

39 Each phase includes input from users Multidisciplinary teams Enlightened management Willingness to learn as you go Deﬁned usability goals and objectives Supporting usable design

40 Where’s the value in testing? 70% watching someone use the design 20% working with the team to prepare to test 8% discussing what happened

41 Getting value in the wild ✤ Use what’s at hand ✤ Narrow the scope of the test ✤ Focus on ✤ what can make the most difference to the most users ✤ what can be implemented easily with the resources available

42 ✤ If you need ✤ summative data When is testing in the ✤ benchmarks ✤ answers to wild not valuable? hard problems

43 Risks

44 Risks of testing in the wild ✤ Participant sample may be too small, biased ✤ Inconsistent approach may net inconsistent data

45 What do you lose? ✤ Quantitative data ✤ Rigor ✤ Relatively unbiased sample, maybe http://drb.lifestreamcenter.net/Lessons/process_maps/

46 Trade offs

47 What do you need? ✤ To inform a design: Testing in the wild Classic usability testing Qualitative data Quantitative data Opportunity Planning Fits the schedule Don’t know how to ﬁt UT into the schedule Just in time After the fact $ $$$$ Something Maybe nothing

48 Where’s the ROI? ✤ Qualitative data ✤ Opportunity ✤ Fitting into a schedule ✤ Timeliness

49 So far. You don’t have to do it Value of usability Tradeoff: having by the book testing some data over having no data + observing users + working with the team

50 Coming up. What’s the bare Steps for testing in What to add if you minimum the wild have resources

51 Essentials

52 What you need Somebody (Human) ✤ Someone who will try the design Something Some place ✤ Somewhere to test (Activity) (Context) ✤ Something to study

53 1. Plan, minimally 2. Get the team on board 3. Design the test, minimally 4. Recruit participants 5. Conduct sessions Follow these steps: 6. Debrief and decide

54 Plan, minimally ✤ What ✤ Why ✤ Who ✤ When ✤ Where

55 Plan, minimally: Example ✤ What: near-ﬁnal design ✤ Why: inform user training and support ✤ Who: inexperienced customers ✤ When: the end of the week ✤ Where: trade show at user group meeting

56 Minimalist plan Find out whether information about admission is Goals and objectives easy to ﬁnd and use Participant People who have college-bound kids characteristics Description of Sit-by parents attending a high school basketball game, each method trying three university sites with the same scavenger hunt task Find out how and when applications are due List of tasks Determine whether there are fees for applying Learn when acceptances will be sent

57 Get the team on board ✤ Visualize the desired user experience ✤ Share the intellectual property of observing users

58 Design the test, minimally ✤ Why are you testing? ✤ What questions are you trying to answer? ✤ What constrains the design? ✤ What are you going to do with what you ﬁnd out?

59 Recruit participants ✤ Convenience sample ✤ Requirements, not demographics

60 ✤ Staff not on the design project ✤ Friends and family ✤ Personal, professional networks ✤ Online social networks ✤ Community organizations ✤ Online classiﬁeds ✤ Association, society, user group, union Sources of participants contacts ✤ Temp agencies

61 Conduct sessions ✤ Rehearse (P1) ✤ Interview-based tasks (or based on previous ﬁeld work) ✤ Explore in short, focused sessions ✤ Iterate test design and product design

62 Greet the participant Explain the study, your role, and their role Interview (maybe) Do tasks from interview Debrief with participant Session outline Debrief with observers

63 ✤ Impartial, unbiased observing ✤ No teaching! ✤ Listen and watch ✤ Ask open- ended questions: Moderating, not training Why? How? What?

64 Debrief and decide ✤ Write up issues on sticky notes and sort them into priority lists ✤ Ask for top 10 items -- base on data and observations, not opinion ✤ Take a vote for priorities

65 Nice to have

66 ✤ Screened, scheduled participants ✤ Ofﬁcial paperwork Add these ingredients ✤ Recordings ✤ More observers

67 Participants What Why appropriate experience greater conﬁdence in data scheduled ahead of time easier to get data

68 Paperwork What Why script ensure consistent instructions and moderating consent forms ofﬁcial acknowledgment for taking part recording waivers permission for recordings pre-test questionnaire experience, knowledge, value post-test questionnaire feedback on tasks, UI

69 Recordings What Why audio transcripts, verbal protocol analysis video or photos stories, double-check, highlights Morae, Camtasia, other digital data, automated collection

70 More observers ✤ Better-informed design recommendations What Why developers technology boundaries management, execs business priorities

71 Nutshell http://www.ﬂickr.com/photos/stoan/145497333/

72 ✤ Value: observing, planning ✤ Plan: 4 steps ✤ Recruit: behavior ✤ Moderating: Take away: listen, observe

73 Quick. Cheap. Insightful. You can compensate Low risk, little money Value comes from for some getting as close as shortcomings, and just possible to what real test more people are doing with your design

74 Where to learn more Dana’s blog: http:// usabilitytestinghowto.blogspot.com/ Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting

75 Me Dana Chisnell dana@usabilityworks.net www.usabilityworks.net 415.519.1148

http://www.slideshare.net	22
http://paper.li	1
http://courtneybolton.parasoup.de	1
https://twitter.com	1

Quick Cheap Insightful: Usability testing in the wild

by Dana Chisnell

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 25

Statistics

Quick Cheap Insightful: Usability testing in the wild Presentation Transcript