The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not

  • 374 views
Uploaded on

UXPA Annual Conference 2013 - Wednesday, July 10, 2013; 1:30-2:30pm ET by Kristen Davis; Dick Horst …

UXPA Annual Conference 2013 - Wednesday, July 10, 2013; 1:30-2:30pm ET by Kristen Davis; Dick Horst

This workshop will explore observational note-taking during usability testing, offering guidance and tips for improvement, and collaboratively attempting to formulate best practices. We will conduct interactive exercises where the attendees view usability test session video clips from the perspective of a note-taker. After each clip, we will discuss what was important and what wasn’t important to note regarding the participant’s behavior and comments. Attendees will have the opportunity to share their own opinions and note-taking experiences. We will illustrate the skills and planning necessary for effective note-taking during testing, and discuss the pros and cons of various note-taking approaches.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
374
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
5
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Kristen: Welcome to The Note-Taker's Perspective During Usability Testing: Recognizing What's Important, What’s Not. User experience methodology often focuses on the role and technique of the moderator and may neglect the role of the note-taker, who should be a user experience professional familiar with the project goals and context of use. The note-taker must also be able to identify and efficiently obtain the data you need for actionable reporting. Today, we will be using video clips from actual usability test sessions to facilitate our discussion regarding an effective note-taker’s skillset and collectively we will create a list of note-taking best practices and tips. Copyright UserWorks, Inc. 2013
  • Kristen: Today we will explore ways to become more efficient and effective note-takers by… [read bulleted list] We will share some of our lessons learned, but we are also here to learn from your experiences, approaches and methods for note-taking. This presentation will use video clips to facilitate a discussion to explore note-taking best practices and tips. We will conduct a series of short exercises wherein we will show a series of video clips from usability tests. You will take your own notes while watching these short video clips. After each clip, we will lead a discussion of what was important and what wasn’t important to note about the test participant’s performance and/or comments during that clip. We will show four video clips. Before each clip there will be a brief setup commentary, providing the context of the clip and the test that it illustrates. After each clip, there will be a brief discussion to share ideas and thoughts about what you noted, given the contextual information that was provided. We hope to illustrate the skills and planning necessary for effective note-taking, and discuss the pros and cons of various styles. We hope to decide if it is possible and worthwhile to even strive for developing principles, which could then be codified and taught. We want to make this an interactive discussion because note-taking is a skill better learned by doing rather than being lectured to, similar to moderating. By a show of hands How many have conducted usability tests? How many have been the note-taker? Copyright UserWorks, Inc. 2008
  • Kristen: So why is important to be a good note-taker? [After reading bulleted list] All of these help the report basically write itself. Copyright UserWorks, Inc. 2008
  • Kristen: Copyright UserWorks, Inc. 2008
  • Kristen: Let’s start with the first exercise. Copyright UserWorks, Inc. 2008
  • Kristen: Let’s say you are being pulled into a study at the last minute. I’m going to show you a video clip and I want you to think about what you would make note of, you can write them down if this will help you remember what you would note for our discussion. I’m not going to give you any background to the study shown in the clip or to the situation. Other than this is a clip from a usability test session for NIHSeniorHealth.com, which is a health information site designed for seniors. The task she was asked to complete was, “Why do people with diabetes need to check their feet?” Please write down or think about what is important to note. [Video Clip: NIHSeniorHealth.gov phase 1B P4 13:57-16:59] Copyright UserWorks, Inc. 2008
  • Kristen: Without having the appropriate context it can be difficult to take meaningful notes. What did you note? Did you capture what you thought were the key findings? Did you feel that you missed anything? It’s probably safe to say with a little more context you could improve your notes, and therefore, the report. Copyright UserWorks, Inc. 2008
  • Dick Copyright UserWorks, Inc. 2008
  • Kristen: So let’s try this activity again, this time with some background information on the study. Copyright UserWorks, Inc. 2008
  • Kristen: This is the same study, but a different information retrieval task… [After presenting the bullets] Now, understanding these goals, can anyone think of anything they would have done differently when you were taking notes during the last clip? Copyright UserWorks, Inc. 2008
  • Kristen: Point out left menu on page and mention the pagination appears at the bottom of the page [Video Clip: SeniorHealth.gov 1B P1 30:07-32:29] Copyright UserWorks, Inc. 2008
  • Kristen: Now that you had some context your note-taking should improve. As you might recall, the woman from the first video clip quickly went to the correct section of the site, “Self-monitoring,” but she never interacted with the pagination to locate the information located on the 2 nd page. The man from the second video clip did not immediately click into the correct section of the site, “Causes and Preventions”. He did use the pagination, but was unable to locate information on the 2 nd page of “Causes and Preventions.” He also clicked on the “Symptoms and Diagnosis” link when he was already on that page. Copyright UserWorks, Inc. 2008
  • Kristen: [Incorporate what audience members came up with as well.] Copyright UserWorks, Inc. 2008
  • Dick Copyright UserWorks, Inc. 2008
  • Dick: The stenographer style records verbatim what the participants says in a stream-of-consciousness format. This style works best for studies focused on preference data and is structured more like a cognitive interview. When the evaluation focuses more on user action and behavior, this style may not collect the right type of data by missing what the participant is actually doing. This may include body language and verbal tone in addition to interactions with the product being evaluated. Many professionals say this style should not be used. Recording high level interpretation works best when the team is evaluating prototypes. This approach allows the note-taker to record their interpretation of what the user is doing and saying and why. It combines analysis into data collection. If the project timeline is short, this style saves time. However, such “on the fly” interpretations may prove to be incorrect or inaccurate causing the team to review the recordings to reassess the interpretation. This style works best when the team has established clearly defined objectives for each task. Coding behaviors in addition to free form notes allows the team to collect some of the stream-of-consciousness data along with verbal and nonverbal data. Verbal data includes what the participant is saying, whereas nonverbal data includes their interactions with the product being evaluated, and perhaps facial expressions and body language. Copyright UserWorks, Inc. 2008
  • Kristen: Now, let’s move on to the third exercise. Copyright UserWorks, Inc. 2008
  • Kristen: Purpose of the site is to provide information to the general public and practitioners about the government’s initiative to increase the use of electronic health records or EHRs. Copyright UserWorks, Inc. 2008
  • Kristen: [Video clip: HealthIT.gov] Copyright UserWorks, Inc. 2008
  • Kristen: What did you note? [Ask one person to say what they noted.] By a show of hands, who else noted that? Copyright UserWorks, Inc. 2008
  • Kristen: Which do you think we should have used for this exercise? Why? [A: High-level interpretation.] [Incorporate what audience members came up with as well.] Copyright UserWorks, Inc. 2008
  • Dick Copyright UserWorks, Inc. 2008
  • Dick Copyright UserWorks, Inc. 2008
  • Kristen: Now, let’s move on to the fourth exercise. This time, we’ll be focusing on metrics. It’s important for all members of the research team to come to a consensus and define all metrics prior to data collection. Copyright UserWorks, Inc. 2008
  • Kristen Copyright UserWorks, Inc. 2008
  • Kristen: [Glucose meter study: P1 29:50-32:18] Copyright UserWorks, Inc. 2008
  • Kristen: By a show of hands how many people rated it a … We scored this task with a 2 (minor difficulty) because the participant tried two different menu items before reaching the correct item, and as defined in the rating scale, that would be a 2. By a show of hands, how many also gave this participant a 2… But, as you can see, that task would have received various ratings had this scale not been established before data collection began. Copyright UserWorks, Inc. 2008
  • Kristen: [Incorporate what audience members came up with as well.] Identifying what constitutes an assist during the planning phase also allows the team to distinguish the difference between “completing a task with assistance or help” from “completing the task with difficulty.” An “assist” implies the participants would not have completed the task on their own without intervention from the moderator; whereas “completed with difficulty” implies the participant completed the task on his/her own, but struggled to do so, needing several attempts or did not fully understand how the system was working. Copyright UserWorks, Inc. 2008
  • Dick Copyright UserWorks, Inc. 2008
  • Dick: Do you have any others? Copyright UserWorks, Inc. 2008
  • Dick: Do you have any others? Copyright UserWorks, Inc. 2008
  • Kristen Copyright UserWorks, Inc. 2008

Transcript

  • 1. www.userworks.com (301) 431-0500 Kristen Davis & Dick Horst UXPA International Conference 2013 July 10, 2013 The Note-Taker's PerspectiveThe Note-Taker's Perspective During Usability Testing:During Usability Testing: Recognizing What's Important, What'sRecognizing What's Important, What's NotNot
  • 2. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. OverviewOverview Presentation Objectives: To collectively explore ways to become a more efficient and effective note-taker by: • Developing the skills and planning necessary • Recognizing what’s important to note and what’s not • Understanding the pros and cons of various note-taking styles • Determining what metrics to define and utilize in note-taking • Creating a list of note-taking best practices and tips 2
  • 3. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Why is Effective Note-TakingWhy is Effective Note-Taking Important?Important?  To improve the data analysis process by: • Capturing the appropriate data the first time; less need for reviewing recordings • Making analyses more efficient by categorizing observations on the fly • Increasing accuracy and completeness of notes (so you don’t miss anything important) • Identifying trends across participants 3
  • 4. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Typical ContextTypical Context  One-on-one, moderated sessions (with the participant in-person or remote)  Moderator interacting with the participant, note-taker in the background taking notes  Participant attempting typical task scenarios at the direction of the moderator  Participant thinking aloud Many of the same principles and best practices would also apply to taking notes in focus groups, user interviews, or ethnographic settings, but not all 4
  • 5. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 6
  • 6. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture what you thought were the key findings?  Did you miss anything? 7
  • 7. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Note-Taking MisconceptionsNote-Taking Misconceptions  No preparation needed  Take the notes, then figure out how to analyze and summarize the data  Anyone can do it  Attention to detail, fast typing are the key skills  A one-size-fits-all approach will work for all practitioners and for all products under evaluation 8
  • 8. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. The ContextThe Context  Product: NIHSeniorHealth.gov website  Participant: Seniors who look for health information online  Task: What types of food or drink are related to balance problems?  Study goals: • What navigation path do participants use when looking for specific pieces of information? • Do participants notice and use the pagination? • Does the visual treatment used in the left hand navigation clearly indicate location within a specific health topic?  Task goals:  Are participants able to locate information on the 2nd page?  Are the left hand navigation labels clear? 10
  • 9. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 11
  • 10. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything?  Did you note any of these: • Study goals:  What navigation path do participants use when looking for specific pieces of information?  Do participants notice and use the pagination?  Does the visual design treatment used in the left hand navigation clearly indicate location within the topic? • Task goals:  Are participants able to locate information on the 2nd page?  Are the left hand navigation labels clear? 12
  • 11. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned: Understand theLessons Learned: Understand the GoalsGoals  Important to be familiar with the product and understand the overall goals of the study and the specific goals of the tasks  Goals should be set prior to data collection in order to: • Record appropriate data • Target the type of data that will be useful when summarizing and reporting the findings 13
  • 12. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Different Note-Taking ApproachesDifferent Note-Taking Approaches  Would you try to note: • Everything the participant says? • Everything the participant does?  Pages visited  Links visited  Scrolling, hovering  Participant’s demeanor  Non-verbal signs  Or would you be more selective? • Task scoring (success vs failure) • Errors, wrong paths • Pre-identified issues of interest • Occasional verbatims 14
  • 13. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Different Note-Taking StylesDifferent Note-Taking Styles  Moderator also takes notes  The court stenographer, verbatim  High level interpretation only, not behavior or comment  Coding behaviors in addition to free form notes  Working from a checklist  Multiple note-takers looking for different things  Multiple note-takers working redundantly  Taking notes from playback of a video/audio recording of the session  Using a tool like Morae, Ovo, Silverback, free form typing into a word processing or spreadsheet program, or paper-based (checklist) 15
  • 14. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Practice ExercisePractice Exercise  Product: HealthIT.gov website  Participant: Private physician  Task: Medical professionals and hospitals are encouraged to participate in the Government EHR reimbursement programs, early, to receive the maximum payment. Find out when the Government will start or has started these reimbursement payments.  Study goals: • Are the labels used for navigation clear? • Does the information architecture make sense? • What suggestions do participants have to improve the content?  Task goal: • What navigation path do participants use when looking for specific pieces of information? 17
  • 15. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 18
  • 16. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything? 19
  • 17. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned: RecordLessons Learned: Record Only KeyOnly Key PointsPoints  The method of note-taking impacts the type of data collected during the evaluation  Depending on the evaluation objective, one style or a combination of styles may be more appropriate 20
  • 18. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Challenges for the Note-taker (andChallenges for the Note-taker (and Moderator)Moderator)  Participant says one thing and does another  Participant thinks they have been successful when they really haven’t  Participant doesn’t want to look bad, so bluffs  Participant initially struggles, then succeeds, and speaks highly of the product  Participant blames self and rates product highly despite disastrous task performance  Participant is overly chatty and goes off on tangents 21
  • 19. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. More Challenges for the Note-TakerMore Challenges for the Note-Taker  Being selective in the information you note, without introducing your biases (seeing what you want/expect to see)  Noting participant behavior versus comments  Whether to try to capture participant’s clickpath, menu choices, data entry  Whether to capture timing  Whether to use a shorthand “code” for observations  Whether to capture participant’s non-verbal behavior  Whether to note good clips for a highlights video 22
  • 20. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Practice ExercisePractice Exercise  Product: Blood glucose meter  Participant: Person with diabetes who uses a meter  Task: Change default settings in the meter  Rating scale for scoring task completion: • Completed with ease: Participant easily completed the task (Score: 3) • Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2) • Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1) • Failed to complete: Participant was unable to complete the task (Score: 0) 24
  • 21. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 25
  • 22. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything?  What score did you give the participant? • Completed with ease: Participant easily completed the task (Score: 3) • Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2) • Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1) • Failed to complete: Participant was unable to complete the task (Score: 0) 26
  • 23. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned:Lessons Learned: Define MetricsDefine Metrics  Using a task completion rating scale allows: • Participant performance to be categorized • Note-taker to record the scores in a systematic and consistent way  Carefully defining the performance categories before the fact helps the note-taker • Know what to look for • Efficiently categorize on the fly  Some additional metrics to consider (facilitated by commercial data logging software): • Clickpaths • Time on task • Shorthand codes for behaviors or incidents  Example: Number of times a participant clicked the pagination 27
  • 24. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. "Take-Aways" for Effective Note-"Take-Aways" for Effective Note- TakingTaking  Create a mindset that values the role of the note-taker  Choose someone with appropriate education and experience to appreciate the design issues of interest  Include the note-taker in test planning discussions and debriefs  Establish project-specific note-taking procedures and coding schemes; anticipate analysis needs when planning note-taking  Be sure the note-taker is thoroughly familiar with the product interface, task scenarios, and project goals 29
  • 25. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. And a Few Tips for the EffectiveAnd a Few Tips for the Effective Note-TakerNote-Taker  Focus on what the participant does, not just what they say  Strive to interpret what you are seeing on the fly and take notes about that, not just “raw” observations  Review your notes and fill in any gaps as soon as possible after the session while observations are fresh in your mind  Consult with the team and client mid-course to be sure your notes reflect project goals and team intent  Check available storage space on recording devices before each session  Spot check recordings throughout the study to ensure that everything is working as intended 30
  • 26. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Questions/Comments?Questions/Comments? Kristen Davis UserWorks, Inc. User Experience Specialist/ Note-Taker Extraordinaire kdavis@userworks.com 31 Dick Horst UserWorks, Inc. President/ Principal User Experience Specialist dhorst@userworks.com www.userworks.com (301) 431-0500