Your SlideShare is downloading. ×
Usability Testing for Qualitative Researchers - QRCA NYC Chapter event
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Usability Testing for Qualitative Researchers - QRCA NYC Chapter event


Published on

The goal of this presentation is to give attendees a deeper understanding of usability testing so they can leverage it in their own work. The material will shed light on what is important to the …

The goal of this presentation is to give attendees a deeper understanding of usability testing so they can leverage it in their own work. The material will shed light on what is important to the research buyer and will help the research provider to better understand how to plan, moderate, and report on a usability study. It will also provide information on where they can go to learn more about this very practical qualitative method.

Kay will cover what a usability test is and when to use it, the key planning steps, the language around it, and the unique insights this method produces. She will also discuss the various approaches a market researcher can take when running a usability study at different points in a product’s development (e.g., concept, early prototype, released product).

Published in: Design, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. What qualitative researchers shouldknow about usability testingQRCA - New York City ChapterPresenter: Kay Corr y AubreyJuly 20, 2012
  • 2. Background
  • 3. 333Agenda – what you will learn1. This presentation will be an “appetizer” on usabilitytesting (1 hr 15 minutes)• Overview of how to plan, run, analyze & report ona usability study*• Pointers to where you can learn more2. Hands-on demonstration• We’ll usability test the Android, iPhone, & iPadversions of the new QRCA VIEWS app (45minutes)3You will learn the process of usability testing, its language,and when to use it in a qualitative research project
  • 4. 444What is a usability test?• Qualitative study where typical users try to accomplishtypical tasks on their own with the product• Point is to see how clearly the product “speaks” tothem, meets their expectations, fits into their typicalwork and task flow• Moderator & team watch participant working and keepscore of task success & failure, comments, bodylanguage
  • 5. 555What can be usability tested?• Web sites, software applications• Consumer products (e.g., vacuum cleaners, ovens, mobilephones – examples from projects I’ve done to illustrate therange of where you can apply this technique)• Packaging• Customer service or ordering procedures• Training & documentation• Basically you can usability test any product or servicewhere there is user interaction
  • 6. 666Some Good reasons for running ausability study• Make interaction with a product as fluid & intuitive as possible• Avoid embarrassment – expose usability problems• Test design concepts• Compare design approaches• Challenge assumptions• To compare your product with a competitor’s• Improve ease of use and learning• To better understand users• To understand training and documentation needs• To increase sales, improve your product’s reputation, decrease need fortechnical support• To save money and time (less need to rework the design, fewer calls tocustomer support)
  • 7. 777Where does usability testing fit in with otherqualitative research methods?• Similar to in-depth interviews, focus groups, andethnographies. Listen to participant, note theirbody language, learn their work and task flow,needs, attitudes, and perceptions• Different from other forms of qualitative researchbecause goal is to watch participant perform tasksagainst a product design and keep track of successand failure
  • 8. 888Where does usability testing fit in withother qualitative research methods?Shown with permission
  • 9. Stages of a usability test
  • 10. 101010Major phases of a usability study1. Planning the study2. Running the study3. Analyzing results4. Reporting resultsA typical soup-to-nuts usability study takes about 70 hours
  • 11. Planning a usability study
  • 12. 121212Planning the study• Determine study objectives• Identify target user characteristics, write recruitingscreener, oversee participant recruiting & scheduling• Create usability study plan and task list*• Make sure prototypes & other stimuli are ready• Manage logistics*Bolded items are unique to usability testing
  • 13. 131313Usability objectives should focus on aspecific aspect of the product designAspects of an interactive productdesign that affect usability• Are the features, functions,and content appropriate?• Does the organization andnavigation align with the user’stask and work flow, their“mental model”?• Do users understand labeling& terminology?• Do they know where to start,what to do next, and the rangeof available choices?
  • 14. 141414Need an analysis framework for aninteractive product
  • 15. 151515Sync testing approach to product’sdevelopment stageTest method Project stage FocusPaper prototype Early Overall product concept,terminology, navigationElectronic prototype Design Task flow, visual design, pagelayout, specific features,validate redesignFunctioning product Development & QA Defaults, online help, featureintegration, performanceComparison Post-release Product features, performancebenchmarks, can be withinyour own product or againstcompetitor’s
  • 16. 161616Venue choices for a studyVenue Pros ConsFacility Can invite lots of observers,fewer logistical headaches,can test wider range ofproductsArtificial environment, costs moreConference room Saves money - no travel, cantest wider range of productsMore logistics, artificial environment,observers want to sit in same room astesterLive online More natural (participant isin their environment, savesmoney, no travel, fewerlogistics, easy for team toobserveCan only test Web-based products, hearvoice but don’t see body language,need to recruit more tech savvyparticipants, firewall issuesNative habitat (mobile studies) Person is in context of use -people use mobile phoneswhen they are “on the go”,cheap & quickCannot easily record, only have 1-2observers
  • 17. 171717Tips on recruiting for a usability study• Usability studies typically involve only a few participants (8-10)• Choose confident and articulate participants who care aboutthe product category• Aim to recruit people with varying levels of experience• Define the required foundation skills• Make sure participants understand what it means to be inusability study
  • 18. 181818Task list developmentEXAMPLE TASK LIST“You are the owner of a family-owned business that has just deployed theAcme Web Conferencing product. Today you are working from home but needto hold several meetings with your managers and employees. For the firsttime, you will be using your iPad to participate.”• Task 1 – Your Administrative Assistant, Kathleen, has set up an Acme Webconferencing meeting for you. Go to your e-mail and find the invitation.• Task 2 - Join the conference.• Task 3 – Enter your information: Chris Doe,,987-123-9876.• Task 4 - Enter the meeting room.• Task 5 - Once you are in the meeting room, connect your phone to the Webconference so you can hear what is going on.
  • 19. 191919Planning a study is very involvedCreate project plans and check lists to keep your ducks in a row.
  • 20. 202020Forget a step and you are deadJust kidding
  • 21. Moderating a usability study
  • 22. 222222Running the studySessions follow a structure that is similar to any qualitativeresearch session:• Moderate sessions – steps are the same• Keep track of participants’ task success and failure• Note comments, facial expressions, and body language• Hold debrief discussions with participants and observers• Administer surveys between tasks and after session has endedDuring a usability study you are watching people’s behavior whilelistening closely to what they say
  • 23. 232323What type of data do you collect?• Collect objective results (can they do it? Define up front what“success” means for each task)• Emotional reactions• Practical information (how does this product design fit into theirworld? What kind of training would a person need to beproductive with this technology?)• Typical measures (task success/failure, time on task)• When they struggle, note why• You see patterns after 3-4 sessions but new stuff alwaysemerges (depending on diversity of participant backgrounds)
  • 24. Analyzing results
  • 25. 252525Analyzing the study results• Keep score of task success/failure by participantand task• Identify trends – What was easy? What washard? How did the product design play into theseresults?• Find verbatim comments and video clips thatencapsulate key findings• Debrief continuously with clients – gainingconsensus on issues is critical to your success
  • 26. 262626Use affinity diagramming to help teamreach consensus on issues
  • 27. Reporting results
  • 28. 282828Writing the report• Use a conventional approach to structuring the report(e.g., executive summary, methodology, findings)• Group findings by product features and functions• Where possible, tell the story with numbers (e.g.,number of participants who succeeded at each task)• Layer the information – place summarized informationin the report, and details in a spreadsheet
  • 29. 292929Example of a usability test report
  • 30. 303030Look for opportunities to collectquantitative dataProduct Reaction Cards Participants quickly select 5attributed from among 118 choices The attributes are balanced betweenpositive and negativeSystem Usability Scale Participants answer 10 questions onkey aspects of usability Survey produces a score between 0and 100; a score <60 is consideredpoor
  • 31. 313131Seek results that can be expressed inpicturesSource: Moxie Software
  • 32. 323232Product Reaction Card results addhuman interestText size indicates number of times that attribute was chosen
  • 33. 333333System usability Scale (SUS) scores cutto the chaseProduct ABC Product XYZ0 0 0 0 0 0 0100-10 11-20 21-30 31-40 41-50 51-60 61-70 > 7110101 13 30-10 11-20 21-30 31-40 41-50 51-60 61-70 > 71SUS scores < 60 indicate poor usability. See the study spreadsheet for details.XYZ’s’ SUS scores from all10 participants werebetween 75 and 100
  • 34. 343434Reporting – each discipline expectsspecific insights from the study resultsExecutives want a distilled version, an encapsulation of “the problems” and tounderstand what is perceived as the premium this product offersProduct managers and marketing people seek insights on segmentation,product identity, competitive information, participant reaction to feature setsProduct designers want detailed usability feedback to guide refinements tothe product’s interface and behavior. Is the design intuitive? How does it fitinto the user’s work and task flow?Engineering needs to understand the usability bugs so they can prioritizethem and fix them (often their input is the bottom line)Training and documentation people want to know which content to include intheir work
  • 35. QRCA VIEWSfor Androidand iPhoneHands-on exerciseUsability testing amobile app
  • 36. 363636Hands-on exercise – usability test QRCAVIEWS mobile appSTEPS1. Translate the objectives into a task list2. Break into groups of 2-3 people, one person is the administratoranother the participant, others are observers who will take notes3. Run the study4. Observers will record impressions on colored sticky notes (eachgroup will have its own color)5. We’ll reconvene and do a mini affinity diagramming session totabulate and discuss the results and recommendations forimprovement. We’ll hold a debrief.6. Kay present our results to Laurie and Eddie7. Then we’ll pat ourselves on the back!!!
  • 37. 373737Additional resources• Review a bibliography on user experience plus VIEWS and Quirks articles onhow to run a usability study• Watch Camille Carlin and my QRCA QCast• Sign up for my newsletter “Usability Tidbits”
  • 38. The EndThank You!!
  • 39. 393939About your presenterKay Corry Aubrey - UX researcher and designerKay Corry Aubrey is the owner of Usability Resources, which specializes in user-centeredresearch and design. Kay has over 20 years of experience in applying qualitative researchmethods and usability testing to technology-oriented products and collaborative software.She has led user research, usability, and design efforts for dozens clients including AT&T,the Broad Institute, Affinnova, Constant Contact, Monster Worldwide, the MassachusettsMedical Society, the Mayo Clinic, and iRobot.Kay teaches usability and design for Northeastern University. She is a Managing Editor ofthe QRCA VIEWS magazine, a market research journal. Kay has an MSW from BostonUniversity’s School of Social Work, an MS in information systems from NortheasternUniversity’s Graduate School of Engineering, and a BA from McGill University. She is aRIVA-certified Master Moderator who enjoys doing research with both groups andindividuals.Contact informatione-mail: phone: 781-275-3020