Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Usability Testing Basics: Remote and In-Person Studies


Published on

Intimidated by conducting your own usability study? This session will give you the tools you need to conduct effective usability tests whether your participants are in the room or in a different country. The session includes practical techniques to successfully plan, prepare, and conduct your test and activities to help you become more confident with the entire process of usability testing. Finally, you’ll get tips on how to get the most useful results from your study.

Participants will also learn about:

Testing protocols
Types of usability testing and required vs. optional resources
Recruiting and scheduling usability tests
Non-disclosure and consent forms and their purposes
Pilot testing
Techniques for interacting with test participants
Current usability testing issues of interest (e.g. testing internationally, moderated vs. un-moderated, etc.)

Published in: Technology
  • Be the first to comment

Usability Testing Basics: Remote and In-Person Studies

  1. 1. Getting Started with UsabilityTesting Instructors: Thyra Rauch, John Schrag, and Carol Smith UXPA 2016
  2. 2. Schedule  05:30 Introductions, Agenda, Materials  05:40 Measurements & Testing  06:05 Activity I  6:25 Planning Test Logistics  6:40 Activity II  7:00 Selecting & Preparing Materials  7:10 Interacting with Test Participants  7:30 Break  8:00 Activity III  8:20 Testing Experiences  8:25 Activity IV and Team Presentations  9:10 Team Presentations  9:25 Final Q&A and Wrap-Up  9:30 Adjourn 2
  3. 3. Introduction Instructor introductions Audience experience 3
  4. 4. Why Test? You are not your user  May miss details; too close to design  Unforeseen requirements Validate understanding of tasks and context More satisfied users  Less training needed  Fewer services calls 4
  5. 5. Myths of Usability Testing Need a lab Testing occurs at the end of the dev cycle Must test everything Need to test 100 people A “good” test means no changes Usability is “system testing”, “user-acceptance testing”, etc. Need a large test team, each playing a specific role/part 5
  6. 6. Just Do It! Make it realistic Any location Any stage of development (earlier is better) Start small  Test an idea (not a feature)  Just a few users Anytime (un-moderated) 6
  7. 7. Getting Started 7
  8. 8. Plan and Prepare for the Test • Scope the effort –Budget, resources, goals, time –Put together a test team • Design the test –Goals, participants, tasks, measures, methodology, team roles etc. –Summative or formative? 8
  9. 9. Testing Plan  Goals  What testing and how to measure success  Test protocol – remote or in person?  User groups and participants  Recruiting/incentives  Scenarios  Share with team  Keep concise 9
  10. 10. Create Test Scenarios Representative of typical tasks Describe what, not how  “You want to buy a shirt for your father. Find a man’s blue dress shirt for under $40.”  “As a systems administrator for your company, you want to install, configure, and test the xyz application.” Avoid leading wording (e.g., actions) 10
  11. 11. Measurement s  Success  Time on task  % of tasks completed/not completed  Number of steps to accomplish task  Learning time  Number of errors  Number of times help consulted  Satisfaction 11Photo By Kenyaboy7Andrew Chipley
  12. 12. Facilitator Role Conducts test and manages time and frustration Communicates with participant (before, during, after) Prompts for more information Ensures legal requirements are met (NDAs, recording permissions) 12
  13. 13. Additional Roles Observer(s)*  Takes careful notes  Observations (notes inferences as well)  Exact quotes  Context “Computer” (paper prototyping, wizard of oz)  Mimics actions of computer 13 *See guidance for Observers in Resources section
  14. 14. Testing Protocols Think-aloud Active intervention Co-discovery Retrospective 14
  15. 15. Activity I 15
  16. 16. Basic Usability Study Select item to test Discover stakeholder’s needs High level goals Measurements to be made High level scenario (sentence or two) 16
  17. 17. Planning Test Logistics 17
  18. 18. Recruit Participants Use a participant screening profile Two weeks to complete recruit (average) Recruiters  Marketing or product marketing department  Professional recruitment agency  Members of your team or yourself 18
  19. 19. Run a Pilot Study Discover problems with study or concept being tested Estimate time needed for the test Refine script and tasks for test Verify if tasks are typical (is it something users actually do?) Practice before going live with participants Get new ideas for follow-on questions or things to observe 19
  20. 20. Do I need a lab? 20 Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994. Computer / Concept Facilitator Participant Observer Timer Logger
  21. 21. Portable Labs Temporary setup  Capture screen, video and audio  Complicates observation Anywhere (conference room, remotely) 21Photo by Roebot at
  22. 22. Remote Testing Moderated and Automated Capture screen, video and audio (not all automated collect this) Moderated: Connect to both computer and phone Technology  Bandwidth, operating system, browser  Test setup and connections ahead of time 22
  23. 23. Activity II 23
  24. 24. Recruiting Profile Key elements  Mimic population using product Critical factors that make a difference  Age  Experience How they will be screened  Phone  Online  Follow-up and scheduling 24
  25. 25. Recruiting Profile Samples  Why decisions made  What going after Create recruiting profile for participants for usability study designed in Activity 1 25
  26. 26. Selecting & Preparing Materials 26
  27. 27. Why Use a Script? Promotes consistency in conducting the test Reminds to welcome and thank people Reminds to reset/configure between participants Ensures NDAs and other legal requirements are not missed Prepare for non-native language speakers  Tell me more… 27
  28. 28. Parts of a Script Welcome to participants Steps in test (forms, tutorials, tasks, questions) Thank you to participants (and incentives if any) Notes to yourself Reset/configuration prompts 28
  29. 29. Test Folders Organize materials per participant Easily access individual results Store as a group Prepare a folder for each participant  All test materials (script, forms, printed tasks etc.)  Don’t use names- label folders: Pilot, P1, P2, P3 29
  30. 30. Informed Consent Your responsibility to inform participant  Every time  Ethical and legal responsibility  Do no harm (physical, emotional) Explain consent form*  Plain language – what, why, stop anytime, etc.  Participant signs (Google form remotely) Rarely lose participants over form 30 *Examples in Appendix
  31. 31. Interacting With Test Participants 31
  32. 32. Welcome Make sure the participant feels welcomed and appreciated. Stress that you are not testing them, rather you are testing… Encourage feedback, positive or negative  “I was not involved in the design of this so you can’t hurt my feelings” 32
  33. 33. Facilitating Follow your script Frame questions in unbiased manner Remain passive in facial expressions and body language as much as possible while participant is working Don’t lead or give clues (unless frustration is clear and evident) Probe don’t demand Gently prompt during think-aloud protocol if necessary 33
  34. 34. Probes Ask open-ended questions Encourage participant to elaborate Use participant’s words Listen for vocalizations and watch non-verbal gestures Have participant tell you when they are done Give them time to think 34
  35. 35. Example Phrases How does this compare with your expectations? What specifically are you referring to? What are you looking for? Tell me more… Can you give me an example? 35
  36. 36. Common Issues Refusal to critique  Ask preference and to compare interfaces  Watch what they DO, not what they SAY Taking control - diverting to their issue  Let them see you are listening and writing it down If no resolution, thank and end session - give them the incentive. 36
  37. 37. Getting Users Unstuck It’s not a question of “if” but “when”  Users might be embarrassed at what they perceive to be their fault  Users may not be able to articulate what they don’t understand Don’t directly answer initial questions 37
  38. 38. Getting Users Unstuck Progressive assistance  What are you trying to do right now?  What do you think the next step is?  What would you do in this situation if you were at work? Hints  Do you see anything that might help you? (small hint)  Have you checked the Help? (medium hint)  What do you think the xxx button does? (large hint) End before frustrated, or take a break 38
  39. 39. Debriefing Verbal and written questions  What did you like best?  What was the worst thing?  What can be improved?  Describe in 3 words  Follow up to specific items noted during testing Ratings  Ease of use  Willingness to use Thank participant 39
  40. 40. Activity III 40
  41. 41. Probes – Group Exercise Are you thinking of using the search feature? Are you trying to find the price of the book? 41
  42. 42. Alternatives Are you thinking of using the search feature? (biased)  Can you tell me what you are thinking right now? (unbiased) Are you trying to find the price of the book? (biased)  What are you trying to do? (unbiased) 42
  43. 43. Rational Can’t know what participant is thinking Keep questions brief and clear Avoid  Asking participants to predict the future  Putting “words in their mouth”  Showing bias or hinting at your opinions 43
  44. 44. Probes – Team Exercises Break into teams Each get a set of biased probes Generate unbiased alternatives Present/discuss with entire group 44
  45. 45. Testing Experiences 45
  46. 46. Low- fidelity/Paper Prototypes Includes wireframes Good time to get initial feedback on concept, word choices and placement Participants  May not be able to understand  May not be act the same as online Always test again later in development 46
  47. 47. High Fidelity Prototypes Ideal time for testing – early enough in development, but participants can “kick the tires” May be difficult to test with (broken links, undeveloped material, etc.) 47
  48. 48. Functioning Products Great for redesign projects – prior to start Get baseline and goals for improvements  Reduce task sequence from 10 clicks to 5  Reduce time to perform task from 10 min to less than 3 min  Increase satisfaction from 4 to 5 on a 5-pt scale Use existing data (Analytics, Help Desk, etc.) to create tasks and inform study 48
  49. 49. Reporting What was tested, how and why Success and Failure and why Recommendations Future testing – benchmarking Just enough to communicate – short & sweet 49
  50. 50. Activity IV 50
  51. 51. Activity 4 Mock up (15 minutes) Round 1 of testing (5 minutes) Revise mock ups (10 minutes) Round 2 of testing (5 minutes) Prepare team presentation (10 minutes) 51
  52. 52. Hotel Reservation Kiosk Design and develop a new hotel check-in kiosk that can be installed in airports Hotel chain wants to make it easy for their customers:  Check into their room at the airport  Request a particular room type, if available  Request transportation to the hotel 52
  53. 53. Ideal Scenario During your search for a local phone book, you notice a hotel chain’s digital Kiosk, and you decide to try it. You need to:  Reserve a Hotel Room for the evening. (A room with 1 bed for 2 people, non-smoking)  Request transportation to the Hotel  Reserve transportation from the hotel back to the airport 53
  54. 54. Alternative Scenario You want to go to the hotel right away to change before going to a fancy restaurant downtown. You need to:  Know if the hotel provides transportation.  Reserve a room with a King-Size bed because you're fat and you want a view of the ocean.  Make sure that your “friend” who is coming tomorrow will have a room available next to yours. 54
  55. 55. Team Presentations 55
  56. 56. Final Q&A Questions? Handouts 56
  57. 57. Survey Please complete the session survey:  57
  58. 58. Contact Us Thyra Rauch IBM  +1 (408) 463-2465 Carol Smith IBM  +1 (412) 212-8737 @carologic 58 John Schrag Autodesk  @jvschrag
  59. 59. Resources 59
  60. 60. Usability by the numbers… 80% of SW life cycle costs occur during maintenance 80% of maintenance costs are due to unmet or unforeseen user requirements Only 20% are bugs or reliability problems Typical software program released with 40 usability design problems Over 70% of CIO's state that one of their biggest problems is communicating with users to understand their needs 60
  61. 61. Number of Participants • Contentious issue • Nielsen - 5-7 participants uncover ~80% of major usability problems • IF representative • IF doing representative tasks. • Testing more = significant diminishing return. • Virzi and Carol Barnum • Small number of users is sufficient. 61 Jakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users. March 19, 2000. Number of Test Participants
  62. 62. Recruitees Could be  Customers  General consumers  Members of organization or company Difficulty of recruiting  Specificity of screener  Availability/interest in study (doctors vs. consumers)  Experience (highly experienced vs. novice) 62
  63. 63. Recruiting Consideration s Accessibility of test location  Include people with disabilities  “We are all only temporarily able-bodied. Accessibility is good for us all.” Language interpreters Remote and on-site studies  Technology availability  Administrative rights  Internet connectivity 63 Quote by @mollydotcom at #stirtrek 2011 via @carologic
  64. 64. Data Collection Forms Customize for each study Space for writing notes – each scenario spelled out. Include space for tickmarks regarding easily measurable activities.  E.g. Participant accesses video clip  E.g. Participant selects advertisement 64
  65. 65. Tool Consideration s In person or remote? Lab or on-site? Prototype limitations (can it be online?, is it a document or a clickable site?) Number of observers, number of participants? Number of facilitators? Logging and video editing needs (time on task, highlight video creation)? Surveys before or after? Eye tracking? 65
  66. 66. Usability Testing Software Morae Ovo SilverBack (Mac only) Loop11 (Remote only) Tobii (Eye-tracker) SMI (Eye-tracker) SurveyMonkey 66
  67. 67. Screen Sharing Software GoToMeeting – Lotus Sametime Unyte – YuuGuu -- WebEx – Yugma -- Trouble Shooting: CoPilot - 67
  68. 68. Satisfaction Questionnaires Standard Usability Measurement Inventory (SUMI)  office/desktop software, purchase  50 questions Website Analysis and MeasureMent Inventory (WHAMMI)  Purchase  20 questions System Usability Scale (SUS)  Free  10 questions 68
  69. 69. Ethics Resources Treatment of human subjects  Web-based course  Gives certificate of completion participant-protections.asp Guidelines for writing informed consent  UXPA Code of Conduct 69
  70. 70. NDA’s Do I need a non-disclosure agreement (NDA)?  Maybe  Legal department (be friendly)  One-way, two-way* AKA  Confidentiality agreement (CA)  Confidential disclosure agreement (CDA)  Proprietary information agreement (PIA)  Secrecy agreement (SA) 70
  71. 71. Activity ______________________________________ Who are you observing? (e.g. SME in soft-goods manufacturing, application admin) What is the situation? ___________________________ (e.g. phone interview, their work at their desk in person, remote usability study, etc.) Date: __________________ Time: __________________ Location: ________________ Note-Taker: ______________ People (P) Who are you observing? Who else is involved in the experience? Objects (O) What objects do they interact with? What apps? Phone, etc.? Environment (E) Characteristics of the setting, number/types of interruptions Messages (M) What is communicated? How is it transmitted? Services (S) What services are available to them? Watch for these types of observations and put additional Questions/Ideas on the back. Content in this document described by Vijay Kumar in his book: 101 Design Methods Physical What do they interact with? Cognitive How associate meanings? How learn? Social Interactions, decision making, scheduling, work? Cultural What are shared norms, habits, values? Emotional What emotions are expressed and how? What are your biggest takeaways? What did you learn? What surprised you? What is the participants need? Interview & Observation Notetaking Form
  72. 72. Observer RulesEveryone who observes a design session is asked to abide by a set of rules.The purpose of these rules is to minimize stress for the participants and to maximize the amount of information we get from the study. Stay for the entire study  Distractions are unhelpful and participants may get the impression that you’re leaving because they’ve done something wrong (e.g. walking out in middle of a movie). If you can attend only part of a study, discuss with the facilitator beforehand to determine whether there is a way to accommodate this. Don’t reveal information about the study to the participant  It is often more useful to explore an area of difficulty in detail rather than try to “get through” all the topics. The facilitator will track time so that we can cover as many of the important areas as possible. Respect participants and the confidentiality of their data  We have promised the participants that their participation is confidential.This means that we should not include their names in any reports or other communication such as email, and we should refrain from discussing them by name outside the test setting.  Do not make negative comments about people—there is always a risk that a derogatory comment could be overheard or otherwise make its way back to the user. Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.
  73. 73. Observer Rules (continued) Remain silent and silence phones  You may notice something so surprising that you are tempted to laugh or exclaim out loud.This is not unusual. Unfortunately, participants might think you are laughing at them. Keep as quiet as possible. You will have opportunities to ask questions at the end. If necessary, pass a note to the facilitator.  Observe only - no questions or support  If reviewing an interface, it’s likely participants will have problems, and it is normal to feel a temptation to help. Please don’t. Instead, try to understand why it was that the user got stuck or went down the wrong path. It’s the facilitator’s role to get users back on track if they get really stuck. And if the facilitator poses a question during the test, he or she is asking the users, not you—please don’t answer unless the facilitator specifically directs a question to you.  Avoid “Design Questions” - Questions that ask the user their opinions about how to design aspects of the application (such as, “Where would you like to see these navigation buttons?”) can take a lot of time to answer and produce only limited results. Instead, focus on trying to understand the problem—we’ll come up with solutions later, outside the test.  In person, be conscious of your body language  Although most studies are interesting, not every moment will be fascinating. If something is happening that isn’t of interest to you but may be to others, sit quietly without fidgeting.Take notes to stay alert. Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.