Usability Testing Kathryn Summers ©2010


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Consistency in labeling, screen layout, and visual coding of information will make the interface much easier for users to learn and use. If users feel confident that similar actions will bring similar results, if they can predict to some degree the behavior of the system, they’ll feel more confident in trying new things, and their exploration will be more productive. Screens that are similar in function or content should be visually consistent. Format of information within screens should be consistent, in order to facilitate recognition and cognitive processing. Use the screen format to support the information architecture of the site.
  • Typically, you want to keep essential information in a single screen, with auxiliary information on separate screens. Extra information slows users down, so you have to find a balance between including useful information and making sure that essential information stays accessible. Too much info or too many features will reduce usability by making desired features harder to find and by p[lacing too high a burden on user memory and user decision-making. With too many features, users are likely to restrict themselves to a tiny subset of available options in self-defense, possibly missing crucial features that would not have been hard to use or understand if they had not been obscured by a forest of possibilities. Does layout effective use gestalt principles of grouping, part/whole relationships, and figure/ground separation.
  • Assessment tests Will involve full prototypes, will focus on tasks, may involve quantitative measurements Validative tests Objectives and goals are typically quantitative Results are used to set standards, performance criteria Test involves reduced interaction between test participant and moderator Test conditions need to be fairly well controlled
  • You can also test how many features the user can remember during a debriefing session. The problem with metrics-based testing is that the individual users may vary widely -- measurements of speed or memory or whatever may depend more on the characteristics of a particular user than of the site or interface you are testing. One solution is to test so many users that you can apply statistical test of validity to the results. This can obviously be expensive. It also requires some statistical expertise.
  • Rubin says testing is not always the best technique to use; he sometimes recommends an expert evaluation instead. This is fine in the very early stages of the project—it’s pointless to bring in users to tell you that you have no global nav, or that you have inconsistent nav, or that you don’t provide any error recovery. But you must never substitute expert evaluation for usability testing in later stages of the project. Nielsen claims 5 usability experts will find 80% of the problems (DR 67), and I don’t believe it.
  • Usability Testing Kathryn Summers ©2010

    1. 1. Usability Testing Kathryn Summers © 2010 Copyright 2003 Kathryn Summers
    2. 2. Usability Testing <ul><li>Preparation </li></ul><ul><ul><li>Recruiting </li></ul></ul><ul><ul><li>Screening </li></ul></ul><ul><ul><li>Tasks and Scenarios </li></ul></ul><ul><li>Testing </li></ul><ul><ul><li>Facilitation </li></ul></ul><ul><ul><li>Observation </li></ul></ul><ul><li>Analysis </li></ul><ul><li>Presentation </li></ul>
    3. 3. Benefits to User-Centered Products (2/09) <ul><li>Users find them more useful (can do more tasks that the user cares about) </li></ul><ul><li>Users can do things more efficiently </li></ul><ul><li>Users learn how to use them more easily and use more often </li></ul><ul><li>Users like them more (less frustrating, more enjoyable </li></ul>
    4. 4. Costs of Poor Design <ul><li>Costs to client </li></ul><ul><ul><li>increased support costs </li></ul></ul><ul><ul><li>reduced sales </li></ul></ul><ul><ul><li>expensive redesign </li></ul></ul><ul><li>Costs to customer/user </li></ul><ul><ul><li>reduced employee productivity </li></ul></ul><ul><ul><li>increased employee frustration </li></ul></ul><ul><ul><li>more frequent mistakes </li></ul></ul>Copyright 2003 Kathryn Summers
    5. 5. Some general usability principles <ul><li>Simple and natural language (user’s language and labels) </li></ul><ul><li>visual consistency (graphics, navigation, layout, labeling) </li></ul><ul><li>appropriate feedback </li></ul><ul><li>error prevention, error recovery </li></ul><ul><li>help information </li></ul>Copyright 2003 Kathryn Summers
    6. 6. And information design principles <ul><li>Information available right when and where it’s needed </li></ul><ul><li>Organization and presentation of information reflects user’s needs and goals </li></ul><ul><li>Visual design reinforces task flows and logical information groupings </li></ul>Copyright 2003 Kathryn Summers
    7. 7. Types of tests: Evaluative Testing <ul><li>Confirm that you’ve met a benchmark for usability before release </li></ul><ul><li>Confirm that product is more usable than a prior version or a competing product </li></ul><ul><li>Evaluative testing will focus on tasks and may often involve quantitative measurements </li></ul>Copyright 2003 Kathryn Summers
    8. 8. Types of tests: Exploratory tests Testing prototypes, or comparing prototypes <ul><li>Focus on exploring relationship between system image and user’s mental model </li></ul><ul><ul><li>Representing classes of objects </li></ul></ul><ul><ul><li>Representing relationships between objects </li></ul></ul><ul><ul><li>Allowing user to manipulate objects </li></ul></ul><ul><li>Test navigation, help access, subject matter organization </li></ul>Copyright 2003 Kathryn Summers
    9. 9. Values of Usability <ul><li>Real world/real user perspective </li></ul><ul><li>Tends to find serious functional issues </li></ul>
    10. 10. Values of Eye-tracking <ul><li>Allows testers to see if users read or not – amount of time spent in area of screen vs scanning </li></ul><ul><li>Has value in decisions on layout/ navigation/labeling </li></ul>
    11. 11. Effective user testing <ul><li>Testing the right people </li></ul><ul><li>Testing the right tasks </li></ul><ul><ul><li>User testing vs. market research vs. “testing the software” </li></ul></ul><ul><li>Watching with attention </li></ul>Copyright 2003 Kathryn Summers
    12. 12. <ul><li>What users expect from a design </li></ul><ul><ul><li>How they respond </li></ul></ul><ul><li>Most understandable ways to present content: How the website should be structured </li></ul><ul><ul><li>Exact function, arrangement of function, and system </li></ul></ul><ul><li>Does the website actually work for the people who are supposedly going to use it? Why or why not? </li></ul><ul><li>Determine who is coming online in your target </li></ul><ul><ul><li>Demographic </li></ul></ul><ul><ul><li>Psychographic </li></ul></ul><ul><ul><li>Attitudes and preferences </li></ul></ul><ul><li>Determine what information will provide value to your target </li></ul><ul><ul><li>Distinct information needs and preferences by target sub-segment </li></ul></ul><ul><ul><li>Information gaps and priorities </li></ul></ul><ul><ul><li>Tools and resources </li></ul></ul>Copyright 2003 Kathryn Summers Market Research User Research
    13. 13. Preparation: Recruiting Participants <ul><li>Identify your target user groups </li></ul><ul><ul><li>(demographics, background knowledge, experience, relationship to site/tool, goals) </li></ul></ul><ul><li>Decide which groups you want, and how many users you’ll recruit </li></ul><ul><ul><li>(decide what characteristics users will share, what characteristics will vary) </li></ul></ul><ul><li>Draft a test participant screener </li></ul><ul><ul><ul><li>(what are make/break characteristics) </li></ul></ul></ul><ul><li>Test the screener </li></ul>
    14. 14. Recruiting Participants <ul><li>Decide on number of participants </li></ul><ul><li>Over recruit (dropout/no show rate) </li></ul><ul><li>Determine compensation </li></ul><ul><ul><li>(amount, cash vs. gift, depends on time spent, organization type, project type) </li></ul></ul><ul><li>Acquire consent </li></ul>
    15. 15. Preparation: Developing the Test <ul><li>Identify test objectives/create list of problems </li></ul><ul><li>Translate objective/problem list into tasks </li></ul><ul><li>Prioritize tasks (frequency, importance), decide which tasks to test, </li></ul><ul><li>Develop scenarios for tasks </li></ul><ul><li>Write test script (scenarios) </li></ul><ul><li>Identify resources participants will need for the tasks </li></ul><ul><ul><li>Hardware, software, data files, instructions, internet connection, time </li></ul></ul><ul><li>Determine what data/measurements to collect (metrics) </li></ul><ul><li>Conduct a pilot test </li></ul>
    16. 16. Tasks vs. Scenarios <ul><li>Tasks are series of actions made to achieve goal – ie., “request article from Interlibrary Loan using online form.” </li></ul><ul><li>Scenarios are tasks put into a short narrative – ie., “your instructor recommends an article and gives you a citation for “Usability Tests made Simple,” from The Journal of Usability, 2 (34), 11-23. You realize that Cook Library doesn’t have that journal, and you would like to request it from Interlibrary Loan”. </li></ul><ul><li>They are meant to take some of the artificiality out of the task. </li></ul>
    17. 17. Tasks vs. Scenarios <ul><li>Scenarios should: </li></ul><ul><li>Allow user to SHOW not TELL actions </li></ul><ul><li>Include one and only one task </li></ul><ul><li>Have defined set of pathways to complete task </li></ul><ul><li>Use natural language (user’s, not system’s) </li></ul><ul><li>Be short – avoid excessive background, irrelevant details </li></ul><ul><li>Be clear and provide enough instruction detail to complete the task (test scenarios with pilot users) </li></ul>
    18. 18. Tasks vs. Scenarios <ul><li>Scenarios should NOT: </li></ul><ul><li>Walk user through process </li></ul><ul><ul><li>(e.g. You have the afore-mentioned article citation. Go to the Interlibrary Loan link located in the list of services on the left. Complete the online form including author, article title, journal title, volume, issue and page number) </li></ul></ul><ul><li>Scenario should test user’s ability to complete the task in a natural way without step-by-step instructions. </li></ul>
    19. 19. Preparation: Writing script <ul><li>Why a script? </li></ul><ul><ul><ul><li>Ensures consistency </li></ul></ul></ul><ul><li>Script should include: </li></ul><ul><ul><li>Explanation of test </li></ul></ul><ul><ul><li>Consent (if not done beforehand) </li></ul></ul><ul><ul><li>Think aloud protocol </li></ul></ul><ul><ul><li>Tasks (description if written, scenarios if verbal) </li></ul></ul><ul><ul><li>Questionnaire/demographic questions </li></ul></ul><ul><ul><li>Follow up if appropriate </li></ul></ul>
    20. 20. Preparation: Establishing roles <ul><li>Facilitator </li></ul><ul><ul><li>Script </li></ul></ul><ul><li>Observer(s) </li></ul><ul><ul><li>Actions </li></ul></ul><ul><ul><li>Metrics </li></ul></ul><ul><ul><li>Comment </li></ul></ul><ul><ul><li>Inferences </li></ul></ul>
    21. 21. Observers can capture metrics in metrics-based testing <ul><li>time on task </li></ul><ul><li>error rate </li></ul><ul><li>successful completion rate </li></ul><ul><li>number of times (or how long) help system is accessed </li></ul><ul><li>time spent recovering from errors </li></ul><ul><li>number of commands/features used </li></ul>Copyright 2003 Kathryn Summers
    22. 22. Preparation: Pilot test <ul><li>Test script to see if scenarios/tasks are working </li></ul><ul><li>Test equipment </li></ul><ul><li>Practice roles for moderator and observers </li></ul>
    23. 23. Preparation: Materials <ul><li>Participant List </li></ul><ul><li>Consent forms </li></ul><ul><li>Compensation/acknowledgements </li></ul><ul><li>Scripts </li></ul><ul><li>Task scenarios (written/part of script) </li></ul><ul><li>Observation forms </li></ul>
    24. 24. Facilitating the test <ul><li>Explain the test </li></ul><ul><li>Help the test participant feel comfortable (emphasis on testing SYSTEM not USER) </li></ul><ul><li>Model/Encourage a think-aloud protocol </li></ul><ul><li>Listen and observe – avoid “rescuing” </li></ul><ul><li>Probe (avoid using leading questions) </li></ul><ul><li>Respect user’s expertise </li></ul>
    25. 25. Becoming a better moderator <ul><li>Become a “people” person </li></ul><ul><li>Be calm and flexible </li></ul><ul><li>Pay close attention </li></ul><ul><li>Watch details but within larger context </li></ul><ul><ul><li>Cohesive picture of each test </li></ul></ul><ul><ul><li>Patterns between tests </li></ul></ul><ul><li>Don’t just rely on memory </li></ul><ul><li>Communicate results effectively </li></ul><ul><li>Stay organized </li></ul><ul><li>Practice!!! </li></ul>Copyright 2003 Kathryn Summers
    26. 26. Watching with attention <ul><li>Try not to test your own forms/designs </li></ul><ul><li>Try to watch with a completely open mind </li></ul><ul><li>Watch carefully—pay attention to body language, facial expressions, sounds, where the user is looking, and what they do </li></ul><ul><li>Pay most attention to behavior, then to explanations, least attention to preferences or predictions about what other users want </li></ul>Copyright 2003 Kathryn Summers
    27. 27. Most of all… <ul><li>Understand that the user never makes “mistakes”—the user only makes efforts to solve the problem </li></ul><ul><li>Your goal is to understand what problem-solving effort the user is making </li></ul>Copyright 2003 Kathryn Summers
    28. 28. Analysis during the test: Keep a problem list <ul><li>Keep a free-hand list during the test—note “critical incidents” that you may ask follow-up questions about at the end of the session </li></ul><ul><li>Use the “highlighter” method </li></ul><ul><ul><li>Have a printout of interface screens on hand; circle problem areas as you observe them during the tests </li></ul></ul><ul><li>Watch the tapes, categorizing related problems after the test </li></ul><ul><li>Capture metrics if desired </li></ul>Copyright 2003 Kathryn Summers
    29. 29. References <ul><li>Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., Wichansky, A. M. (2002). Eye tracking in web search tasks: Design implications. In Proceedings of the 2002 symposium on Eye tracking research & applications (pp. 51-58). Retrieved March 28, 2009 from ACM Digital Library . </li></ul><ul><li>Hackos, J. T. & Redish, J. C. (1998). User and task analysis for interface design . New York: John Wiley & Sons. </li></ul><ul><li>Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991). User interface evaluation in the real world: Comparison of four techniques. In Proceedings of the SIGCHI conference on human factors in computing systems: Reaching through technology (pp. 119-124). Retrieved March 28, 2009 from ACM Digital Library . </li></ul><ul><li>Molich, R., Ede, M. R., Kaasgaard, K., Karyukin, B., (2004). Comparative usability evaluation. Behavior & Information technology, 23 (1), 65. 74. Retrieved March 28 from Academic Search Premier database. </li></ul><ul><li>Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests . New York: John Wiley & Sons. </li></ul><ul><li>Snyder, C. (2003). Paper prototyping: The fast and easy way to design and refine user interfaces . San Francisco: Morgan Kaufman. </li></ul>
    30. 30. Limitations of usability testing <ul><li>Testing situations are always artificial (maybe in a lab, assigned tasks, paid participation) </li></ul><ul><li>Test participants are rarely perfectly representative users </li></ul><ul><li>The test design can never fully duplicate natural user behavior </li></ul><ul><ul><li>Results depend heavily on tasks—we see what we think to look for </li></ul></ul>Copyright 2003 Kathryn Summers
    31. 31. Finding participants with lower literacy skills <ul><li>Try to recruit participants that </li></ul><ul><ul><li>Have a strong motivation to care about the interface (e.g., users with a particular health condition) </li></ul></ul><ul><ul><li>Read at 8 th grade level or below (REALM 60 or below) </li></ul></ul><ul><ul><li>Have been observationally screened for a minimum level of computer ability </li></ul></ul><ul><ul><ul><li>Open browser </li></ul></ul></ul><ul><ul><ul><li>Navigate to website using address bar OR search </li></ul></ul></ul><ul><ul><ul><li>Scroll </li></ul></ul></ul><ul><ul><ul><li>Use links </li></ul></ul></ul><ul><ul><ul><li>Use mouse </li></ul></ul></ul>Copyright 2003 Kathryn Summers
    32. 32. Where to recruit users with lower literacy skills <ul><li>Recruit from </li></ul><ul><ul><li>Health clinics </li></ul></ul><ul><ul><li>Literacy centers </li></ul></ul><ul><ul><li>Public libraries </li></ul></ul><ul><li>Plan for a two-step recruiting approach </li></ul><ul><ul><li>Administer the REALM; observe computer use </li></ul></ul><ul><ul><li>Schedule selected users for testing </li></ul></ul><ul><li>REALM (Rapid Estimate of Adult Literacy in Medicine) </li></ul><ul><ul><li>(Davis, et al., Family Medicine 1993 25.6: 391-5.) </li></ul></ul>Copyright 2003 Kathryn Summers