Usability Testing Basics: What's it All About? at Web SIG Cleveland
Upcoming SlideShare
Loading in...5

Usability Testing Basics: What's it All About? at Web SIG Cleveland



Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio....

Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio.

Learn all you need to get started:
- Where you can conduct studies (does it have to be in a lab?)
- Types of studies (RITE, think aloud, etc.)
- Tips for recruiting participants
- Tips for Interacting with participants without biasing the study
- Preparing for the study (materials needed, forms, etc.)
- Guidance for analyzing the study



Total Views
Views on SlideShare
Embed Views



1 Embed 5 5


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Usability Testing Basics: What's it All About? at Web SIG Cleveland Usability Testing Basics: What's it All About? at Web SIG Cleveland Presentation Transcript

    • Web SIG ClevelandMay 21, 2011
      Usability Testing Basics: What's it All About?
    • Talking About Today
    • Agenda
      What is Usability?
      Why we test usability
      Planning studies
      Facilitating sessions
      Analyzing data
    • What is Usability?
    • Usabilityis an important characteristicof what makes a good User Experience
    • Functional Aspects
    • User’s Perspective
      Useful experience
      Feel in control and supported
      Supplements and enhances skills and expertise
      Satisfied Delighted
    • Usability Testing
      Measures users ability to achieve specific goals of effectiveness, efficiency, and satisfaction.
    • What is it?
      Real Users doing real tasks
      Being observed
      Using prototypes or live products
    • Can Test…
      Websites, Mobile, Blenders, Airport service
      Simulations or mockups
      Early prototypes (paper, low-fi)
      Production prototypes (html, hi-fi)
      Help documentation
      Processes (receipt of materials, purchase)
    • It is not…
      Quality testing
      Full accessibility testing
      System testing
      Acceptance testing
    • Do I need a lab?
      Computer / Concept
      Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.
    • Just Do It!
      Anywhere (conference room, remotely)
      Any Stage (earlier in process the better)
      Anytime (un-moderated)
      Realistic test environment
      Photo by Roebot at
    • Why we test Usability
    • "The biggest waste of all is building something no one wants"
      - @ericries #LeanStartupMI via @MelBugai
    • You are not the user
      Designing for someone else
      Need to step back - may miss details
      May be perfect for you, but not the user
      Honest feedback from users
      Validate understanding of tasks and context
      Unforeseen requirements
    • Rationale
      Goals being met
      Content & purpose clear
      Match expectations
      Verify product meets customer needs
      Gather information for future product development
      Comparison testing
    • Save Time & Money
      Up front by testing prototypes
      Reduce maintenance issues
      Reduce Customer Service requests
    • Minimize Human Cost
    • Find Design Problems
      System status available
      Wording choices clear
      Placement of content
      Recognition, Not Recall
    • Benefits of Good UX
      Increased Usefulness
      Increased Efficiency ($$$)
      Improved Productivity
    • Benefits (continued)
      Fewer Errors
      Reduced Training Time
      Improved Acceptance
      Happy Users!
    • Planning studies
    • Define Goals
      What you need to learn about the product and its audience
    • Scope Effort
      Consider budget, resources
      Adding participants increases budget & time
    • Design the Test
      Select Methodology
      Based on goals
      Identify participants
      Tasks to be completed
      Team roles
    • Types of Usability Tests
      Single participant
      Co-discovery (two)
      Group usability testing (3 or more)
      Rapid iterative testing
    • Facilitation Styles
      Talk aloud/Think aloud
      Task focused (limited/no discussion)
      Cooperative usability testing
      Video review with participant after study (Retrospective)
    • Location
      Formal Lab
      Workplace, conference room
      In home
      At a conference
    • Test Roles
      “Computer” (paper prototypes)
      Note taker / data logger / timer
      Software, recording operator
      Product / Technical Expert(s)
    • Use A Script/Guide
      Memory tool for facilitator
      Promote consistency
      Order of questions
      List out scenarios representative of typical tasks
    • Test Guide Includes
      Welcome to participants
      Steps in study(forms, tasks, questions)
      Notes to yourself
      Reset/configuration prompts
      Thank you to participants (incentives if any)
    • Participant Materials
      Provide with
      Text for forms
      Images to upload, etc.
      Tasks in writing if complex (3x5 cards)
    • Task Building
      Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.
    • Questions
      Quality of questions correlates to quality of answers:
      Don’t lead or make assumptions
      Use participant’s words
    • Stretch & Exercise
    • Question 1
      Do you regularly book your travel online to save money?
    • Alternates – Question 1
      How often do you travel?
      What proportion of that do you book online?
      Why do you book travel online?
    • Rationale - Question 1
      Do you regularly book your travel online to save money?
      Address one issue at a time and avoid double-barreled questions.
    • Question 2
      What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel?
    • Alternates – Question 2
      Would you like to correspond with a travel agent while you are booking travel?
      What are some ways that you would like to correspond with a travel agent while you are booking travel?
    • Rationale – Question 2
      What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel?
      This question asked the participant to predict the future.
    • Schedule & Recruit
      Schedule location for pilot and study
      Leave time between sessions
      Sessions no longer than 2 hours
      Invite Observers (key stakeholders, project managers, etc.)
    • Pilot Study – Find Problems
      Tasks are typical
      Concept is on-track
      Time needed to complete
      Practice before going live with participants
      New ideas for follow-on questions or things to observe
      Refine script and tasks
    • How Do I Find Participants?
    • Create a Screener
      List of questions to determines who will participate
      Describe, then get details:
      Computer activities
      Use of product/service
      People who pass screener = user group
    • Which Student?
      Connie via (Christopher Alison Photography) via
    • Representative Users
      Two weeks on average to recruit
      Primary user population
      People with disabilities
      “We are all only temporarily able-bodied. Accessibility is good for us all.”
      Get to spirit of the law (Section 508, WCAG 2.0)
      -@mollydotcom at #stirtrek 2011 via @carologic
    • Hire a Recruiter
      Allows you to focus on activity.
      Can tell if person will be a good participant.
      May already have a list they can start with.
      Good recruiters:
      find right participants.
      give regular updates.
      take care of directions, confirmations, incentives, etc.
    • If You Must Do it Yourself...
      Go where users go and intercept
      Online user groups
      Professional organizations
      Online tools thru your site:
    • Final Recruiting
      Final recruit by phone.
      Ask questions that force them to talk.
      Don’t recruit non-talkers.
      Confirm participant eligibility
      Note unusual issues
      Recruit for pilot test
    • Number of Participants
      As many as possible (rarely statistically significant)
      Usability Testing Research (in 1990’s)
      5 from distinct sub-group of the user population will yield 80% of the findings (Nielsen, Virzi, Lewis)
      Assumes expert has reviewed concept for obvious issues
      Early tests with 8 – 12 users per user group
      Iterative testing (3 per day, iterate, 3 new users)
      Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface. Retrieved: 20080323
    • Honorarium
      Pay them or give product credits
      Amount varies by:
      Amount of time needed
      Their role (doctors need more to persuade them)
      Their interest, devotion to product
    • Facilitating sessions
    • Welcome & Prepare
      Offer beverage
      Express appreciation for help
      Explain purpose of research
      Sign paperwork
      Consent Form
      Non-Disclosure Agreement(s)
    • Participant Reassurance
      Make sure they are comfortable
      Not testing them, rather testing…
      Encourage feedback (positive and negative)
      “I was not involved in the design of this so you can’t hurt my feelings”
    • During Session
      Remain passive (body, face)
      Don’t confirm or reject answers
      Use participant’s words
      Listen for vocalizations
      Watch non-verbal gestures
      Encourage participant to elaborate
      Ask your question and let them talk
    • Silence is GoldenUser’s Time to Think!
    • Get Them Unstuck
      Progressively give assistance
      What are you trying to do right now?
      What do you think the next step is?
      What would you do in this situation if you were at work?
      Do you see anything that might help you? (small hint)
      Have you checked the Help? (medium hint)
      What do you think the xxx button does? (large hint)
    • Debrief Participant
      Post-Study questionnaire
      Thank for participation
      Explain honorarium and delivery
    • Analyzing data
    • Look for Patterns
      Identify repetition
      Continuation of study
      Adds cost
      Delays reporting
      Low probability of many new findings
    • Measurements
      Time on task
      % of tasks completed/not completed
      Number of steps to accomplish task
      Learning time
      Number of errors
      Number of times help consulted
    • Transform Data
      Create Findings and Recommendations
      High level or detailed report
      Think about audience
      How will it be used?
    • Reporting Includes
      Executive summary
      Positive findings
      Provide solutions for negative findings
      Provide level of effort and prioritize
      Examples, screen shots
      Appendix with questionnaires, test materials
    • Do UX Early & Often
      Put it on the Wall as information radiators
      Test findings
      Competitor info
    • Recommended Readings
    • Contact
      Carol J. Smith
    • References
      Cato, John. User-Centered Web Design. Addison Wesley Longman; 2001.
      Hackos, JoAnn T., PhD and Redish, Janice C. User and Task Analysis for Interface Design. Wiley; 1998.
      Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003 UPA Conference. (Activity)
      Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability.
      Kuniavsky, Mike. Observing the User Experience: a Practitioner's Guide to User Research. Morgan Kaufmann, 2003.
      Nielsen, Jakob and Robert L. Mack. Usability Inspection Methods. John Wiley & Sons, Inc. 1994.
      Redish, Janice (Ginny). Letting Go of the Words: Writing Web Content that Works.
      Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc.
    • Tool Considerations
      • In person or remote?
      • Lab or on-site?
      • Prototype limitations (can it be online?, is it a document or a clickable site?)
      • Number of observers, number of participants?
      • Number of facilitators?
      • Logging and video editing needs (time on task, highlight video creation)?
      • Surveys before or after?
      • Eye tracking?
    • Usability Testing Software
      • Morae
      • Ovo
      • SilverBack (Mac only)
      • UserWorks
      • Noldus
      • Tobii (Eye-tracker)
      • SMI (Eye-tracker)
      • SurveyMonkey
    • Screen Sharing Software
      GoToMeeting –
      Lotus SametimeUnyte –
      YuuGuu --
      WebEx –
      Yugma --
      Trouble Shooting: CoPilot -
    • Satisfaction Questionnaires
      Standard Usability Measurement Inventory (SUMI)
      office/desktop software, purchase
      50 questions
      Website Analysis and MeasureMent Inventory (WHAMMI)
      20 questions
      System Usability Scale (SUS)
      10 questions
    • Recommended Sites
      W3C Web Accessibility Initiative
      Accessibility Standards in US (Section 508)
      Jakob Nielsen
      UPA – professional usability organization