Your SlideShare is downloading. ×
Usability Testing Basics: What's it All About? at Web SIG Cleveland
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Usability Testing Basics: What's it All About? at Web SIG Cleveland


Published on

Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio. …

Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio.

Learn all you need to get started:
- Where you can conduct studies (does it have to be in a lab?)
- Types of studies (RITE, think aloud, etc.)
- Tips for recruiting participants
- Tips for Interacting with participants without biasing the study
- Preparing for the study (materials needed, forms, etc.)
- Guidance for analyzing the study

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Web SIG ClevelandMay 21, 2011
    Usability Testing Basics: What's it All About?
  • 2. Talking About Today
  • 3. Agenda
    What is Usability?
    Why we test usability
    Planning studies
    Facilitating sessions
    Analyzing data
  • 4. What is Usability?
  • 5. Usabilityis an important characteristicof what makes a good User Experience
  • 6. Functional Aspects
  • 7. User’s Perspective
    Useful experience
    Feel in control and supported
    Supplements and enhances skills and expertise
    Satisfied Delighted
  • 8. Usability Testing
    Measures users ability to achieve specific goals of effectiveness, efficiency, and satisfaction.
  • 9. What is it?
    Real Users doing real tasks
    Being observed
    Using prototypes or live products
  • 10. Can Test…
    Websites, Mobile, Blenders, Airport service
    Simulations or mockups
    Early prototypes (paper, low-fi)
    Production prototypes (html, hi-fi)
    Help documentation
    Processes (receipt of materials, purchase)
  • 11. It is not…
    Quality testing
    Full accessibility testing
    System testing
    Acceptance testing
  • 12. Do I need a lab?
    Computer / Concept
    Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.
  • 13. Just Do It!
    Anywhere (conference room, remotely)
    Any Stage (earlier in process the better)
    Anytime (un-moderated)
    Realistic test environment
    Photo by Roebot at
  • 14. Why we test Usability
  • 15. "The biggest waste of all is building something no one wants"
    - @ericries #LeanStartupMI via @MelBugai
  • 16. You are not the user
    Designing for someone else
    Need to step back - may miss details
    May be perfect for you, but not the user
    Honest feedback from users
    Validate understanding of tasks and context
    Unforeseen requirements
  • 17. Rationale
    Goals being met
    Content & purpose clear
    Match expectations
    Verify product meets customer needs
    Gather information for future product development
    Comparison testing
  • 18. Save Time & Money
    Up front by testing prototypes
    Reduce maintenance issues
    Reduce Customer Service requests
  • 19. Minimize Human Cost
  • 20. Find Design Problems
    System status available
    Wording choices clear
    Placement of content
    Recognition, Not Recall
  • 21. Benefits of Good UX
    Increased Usefulness
    Increased Efficiency ($$$)
    Improved Productivity
  • 22. Benefits (continued)
    Fewer Errors
    Reduced Training Time
    Improved Acceptance
    Happy Users!
  • 23. Planning studies
  • 24. Define Goals
    What you need to learn about the product and its audience
  • 25. Scope Effort
    Consider budget, resources
    Adding participants increases budget & time
  • 26. Design the Test
    Select Methodology
    Based on goals
    Identify participants
    Tasks to be completed
    Team roles
  • 27. Types of Usability Tests
    Single participant
    Co-discovery (two)
    Group usability testing (3 or more)
    Rapid iterative testing
  • 28. Facilitation Styles
    Talk aloud/Think aloud
    Task focused (limited/no discussion)
    Cooperative usability testing
    Video review with participant after study (Retrospective)
  • 29. Location
    Formal Lab
    Workplace, conference room
    In home
    At a conference
  • 30. Test Roles
    “Computer” (paper prototypes)
    Note taker / data logger / timer
    Software, recording operator
    Product / Technical Expert(s)
  • 31. Use A Script/Guide
    Memory tool for facilitator
    Promote consistency
    Order of questions
    List out scenarios representative of typical tasks
  • 32. Test Guide Includes
    Welcome to participants
    Steps in study(forms, tasks, questions)
    Notes to yourself
    Reset/configuration prompts
    Thank you to participants (incentives if any)
  • 33. Participant Materials
    Provide with
    Text for forms
    Images to upload, etc.
    Tasks in writing if complex (3x5 cards)
  • 34. Task Building
    Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.
  • 35. Questions
    Quality of questions correlates to quality of answers:
    Don’t lead or make assumptions
    Use participant’s words
  • 36. Stretch & Exercise
  • 37. Question 1
    Do you regularly book your travel online to save money?
  • 38. Alternates – Question 1
    How often do you travel?
    What proportion of that do you book online?
    Why do you book travel online?
  • 39. Rationale - Question 1
    Do you regularly book your travel online to save money?
    Address one issue at a time and avoid double-barreled questions.
  • 40. Question 2
    What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel?
  • 41. Alternates – Question 2
    Would you like to correspond with a travel agent while you are booking travel?
    What are some ways that you would like to correspond with a travel agent while you are booking travel?
  • 42. Rationale – Question 2
    What are your thoughts about a new feature that allows you to instant message a travel agent with any questions as you book your travel?
    This question asked the participant to predict the future.
  • 43. Schedule & Recruit
    Schedule location for pilot and study
    Leave time between sessions
    Sessions no longer than 2 hours
    Invite Observers (key stakeholders, project managers, etc.)
  • 44. Pilot Study – Find Problems
    Tasks are typical
    Concept is on-track
    Time needed to complete
    Practice before going live with participants
    New ideas for follow-on questions or things to observe
    Refine script and tasks
  • 45. How Do I Find Participants?
  • 46. Create a Screener
    List of questions to determines who will participate
    Describe, then get details:
    Computer activities
    Use of product/service
    People who pass screener = user group
  • 47. Which Student?
    Connie via (Christopher Alison Photography) via
  • 48. Representative Users
    Two weeks on average to recruit
    Primary user population
    People with disabilities
    “We are all only temporarily able-bodied. Accessibility is good for us all.”
    Get to spirit of the law (Section 508, WCAG 2.0)
    -@mollydotcom at #stirtrek 2011 via @carologic
  • 49. Hire a Recruiter
    Allows you to focus on activity.
    Can tell if person will be a good participant.
    May already have a list they can start with.
    Good recruiters:
    find right participants.
    give regular updates.
    take care of directions, confirmations, incentives, etc.
  • 50. If You Must Do it Yourself...
    Go where users go and intercept
    Online user groups
    Professional organizations
    Online tools thru your site:
  • 51. Final Recruiting
    Final recruit by phone.
    Ask questions that force them to talk.
    Don’t recruit non-talkers.
    Confirm participant eligibility
    Note unusual issues
    Recruit for pilot test
  • 52. Number of Participants
    As many as possible (rarely statistically significant)
    Usability Testing Research (in 1990’s)
    5 from distinct sub-group of the user population will yield 80% of the findings (Nielsen, Virzi, Lewis)
    Assumes expert has reviewed concept for obvious issues
    Early tests with 8 – 12 users per user group
    Iterative testing (3 per day, iterate, 3 new users)
    Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface. Retrieved: 20080323
  • 53. Honorarium
    Pay them or give product credits
    Amount varies by:
    Amount of time needed
    Their role (doctors need more to persuade them)
    Their interest, devotion to product
  • 54. Facilitating sessions
  • 55. Welcome & Prepare
    Offer beverage
    Express appreciation for help
    Explain purpose of research
    Sign paperwork
    Consent Form
    Non-Disclosure Agreement(s)
  • 56. Participant Reassurance
    Make sure they are comfortable
    Not testing them, rather testing…
    Encourage feedback (positive and negative)
    “I was not involved in the design of this so you can’t hurt my feelings”
  • 57. During Session
    Remain passive (body, face)
    Don’t confirm or reject answers
    Use participant’s words
    Listen for vocalizations
    Watch non-verbal gestures
    Encourage participant to elaborate
    Ask your question and let them talk
  • 58. Silence is GoldenUser’s Time to Think!
  • 59. Get Them Unstuck
    Progressively give assistance
    What are you trying to do right now?
    What do you think the next step is?
    What would you do in this situation if you were at work?
    Do you see anything that might help you? (small hint)
    Have you checked the Help? (medium hint)
    What do you think the xxx button does? (large hint)
  • 60. Debrief Participant
    Post-Study questionnaire
    Thank for participation
    Explain honorarium and delivery
  • 61. Analyzing data
  • 62. Look for Patterns
    Identify repetition
    Continuation of study
    Adds cost
    Delays reporting
    Low probability of many new findings
  • 63. Measurements
    Time on task
    % of tasks completed/not completed
    Number of steps to accomplish task
    Learning time
    Number of errors
    Number of times help consulted
  • 64. Transform Data
    Create Findings and Recommendations
    High level or detailed report
    Think about audience
    How will it be used?
  • 65. Reporting Includes
    Executive summary
    Positive findings
    Provide solutions for negative findings
    Provide level of effort and prioritize
    Examples, screen shots
    Appendix with questionnaires, test materials
  • 66. Do UX Early & Often
    Put it on the Wall as information radiators
    Test findings
    Competitor info
  • 67. Recommended Readings
  • 68. Contact
    Carol J. Smith
  • 69. References
    Cato, John. User-Centered Web Design. Addison Wesley Longman; 2001.
    Hackos, JoAnn T., PhD and Redish, Janice C. User and Task Analysis for Interface Design. Wiley; 1998.
    Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003 UPA Conference. (Activity)
    Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability.
    Kuniavsky, Mike. Observing the User Experience: a Practitioner's Guide to User Research. Morgan Kaufmann, 2003.
    Nielsen, Jakob and Robert L. Mack. Usability Inspection Methods. John Wiley & Sons, Inc. 1994.
    Redish, Janice (Ginny). Letting Go of the Words: Writing Web Content that Works.
    Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc.
  • 70. Tool Considerations
    • In person or remote?
    • 71. Lab or on-site?
    • 72. Prototype limitations (can it be online?, is it a document or a clickable site?)
    • 73. Number of observers, number of participants?
    • 74. Number of facilitators?
    • 75. Logging and video editing needs (time on task, highlight video creation)?
    • 76. Surveys before or after?
    • 77. Eye tracking?
  • Usability Testing Software
  • Screen Sharing Software
    GoToMeeting –
    Lotus SametimeUnyte –
    YuuGuu --
    WebEx –
    Yugma --
    Trouble Shooting: CoPilot -
  • 85. Satisfaction Questionnaires
    Standard Usability Measurement Inventory (SUMI)
    office/desktop software, purchase
    50 questions
    Website Analysis and MeasureMent Inventory (WHAMMI)
    20 questions
    System Usability Scale (SUS)
    10 questions
  • 86. Recommended Sites
    W3C Web Accessibility Initiative
    Accessibility Standards in US (Section 508)
    Jakob Nielsen
    UPA – professional usability organization