UXPA Annual Conference 2013 - Wednesday, July 10, 2013; 1:30-2:30pm ET by Kristen Davis; Dick Horst
This workshop will explore observational note-taking during usability testing, offering guidance and tips for improvement, and collaboratively attempting to formulate best practices. We will conduct interactive exercises where the attendees view usability test session video clips from the perspective of a note-taker. After each clip, we will discuss what was important and what wasn’t important to note regarding the participant’s behavior and comments. Attendees will have the opportunity to share their own opinions and note-taking experiences. We will illustrate the skills and planning necessary for effective note-taking during testing, and discuss the pros and cons of various note-taking approaches.
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not
1. www.userworks.com
(301) 431-0500
Kristen Davis & Dick Horst
UXPA International Conference 2013
July 10, 2013
The Note-Taker's PerspectiveThe Note-Taker's Perspective
During Usability Testing:During Usability Testing:
Recognizing What's Important, What'sRecognizing What's Important, What's
NotNot
2. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
OverviewOverview
Presentation Objectives:
To collectively explore ways to become a more efficient and effective
note-taker by:
• Developing the skills and planning necessary
• Recognizing what’s important to note and what’s not
• Understanding the pros and cons of various note-taking styles
• Determining what metrics to define and utilize in note-taking
• Creating a list of note-taking best practices and tips
2
3. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Why is Effective Note-TakingWhy is Effective Note-Taking
Important?Important?
To improve the data analysis process by:
• Capturing the appropriate data the first time; less need for reviewing
recordings
• Making analyses more efficient by categorizing observations on the fly
• Increasing accuracy and completeness of notes (so you don’t miss
anything important)
• Identifying trends across participants
3
4. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Typical ContextTypical Context
One-on-one, moderated sessions (with the participant in-person or
remote)
Moderator interacting with the participant, note-taker in the
background taking notes
Participant attempting typical task scenarios at the direction of
the moderator
Participant thinking aloud
Many of the same principles and best practices would also apply to
taking notes in focus groups, user interviews, or ethnographic
settings, but not all
4
5.
6. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
ExerciseExercise
Think about and/or record your observations as you watch this
video clip
6
7. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
DiscussionDiscussion
What did you note?
Did you capture what you thought were the key findings?
Did you miss anything?
7
8. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Note-Taking MisconceptionsNote-Taking Misconceptions
No preparation needed
Take the notes, then figure out how to analyze and summarize the
data
Anyone can do it
Attention to detail, fast typing are the key skills
A one-size-fits-all approach will work for all practitioners and for all
products under evaluation
8
9.
10. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
The ContextThe Context
Product: NIHSeniorHealth.gov website
Participant: Seniors who look for health information online
Task: What types of food or drink are related to balance problems?
Study goals:
• What navigation path do participants use when looking for specific
pieces of information?
• Do participants notice and use the pagination?
• Does the visual treatment used in the left hand navigation clearly
indicate location within a specific health topic?
Task goals:
Are participants able to locate information on the 2nd
page?
Are the left hand navigation labels clear?
10
11. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
ExerciseExercise
Think about and/or record your observations as you watch this
video clip
11
12. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
DiscussionDiscussion
What did you note?
Did you capture the key findings?
Did you miss anything?
Did you note any of these:
• Study goals:
What navigation path do participants use when looking for specific
pieces of information?
Do participants notice and use the pagination?
Does the visual design treatment used in the left hand navigation
clearly indicate location within the topic?
• Task goals:
Are participants able to locate information on the 2nd
page?
Are the left hand navigation labels clear?
12
13. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Lessons Learned: Understand theLessons Learned: Understand the
GoalsGoals
Important to be familiar with the product and understand the
overall goals of the study and the specific goals of the tasks
Goals should be set prior to data collection in order to:
• Record appropriate data
• Target the type of data that will be useful when summarizing and
reporting the findings
13
14. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Different Note-Taking ApproachesDifferent Note-Taking Approaches
Would you try to note:
• Everything the participant says?
• Everything the participant does?
Pages visited
Links visited
Scrolling, hovering
Participant’s demeanor
Non-verbal signs
Or would you be more selective?
• Task scoring (success vs failure)
• Errors, wrong paths
• Pre-identified issues of interest
• Occasional verbatims
14
15. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Different Note-Taking StylesDifferent Note-Taking Styles
Moderator also takes notes
The court stenographer, verbatim
High level interpretation only, not behavior or comment
Coding behaviors in addition to free form notes
Working from a checklist
Multiple note-takers looking for different things
Multiple note-takers working redundantly
Taking notes from playback of a video/audio recording of the
session
Using a tool like Morae, Ovo, Silverback, free form typing into a
word processing or spreadsheet program, or paper-based
(checklist)
15
16.
17. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Practice ExercisePractice Exercise
Product: HealthIT.gov website
Participant: Private physician
Task: Medical professionals and hospitals are encouraged to
participate in the Government EHR reimbursement programs, early,
to receive the maximum payment. Find out when the Government will
start or has started these reimbursement payments.
Study goals:
• Are the labels used for navigation clear?
• Does the information architecture make sense?
• What suggestions do participants have to improve the content?
Task goal:
• What navigation path do participants use when looking for specific pieces of
information?
17
18. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
ExerciseExercise
Think about and/or record your observations as you watch this
video clip
18
19. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
DiscussionDiscussion
What did you note?
Did you capture the key findings?
Did you miss anything?
19
20. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Lessons Learned: RecordLessons Learned: Record Only KeyOnly Key
PointsPoints
The method of note-taking impacts the type of data collected
during the evaluation
Depending on the evaluation objective, one style or a combination
of styles may be more appropriate
20
21. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Challenges for the Note-taker (andChallenges for the Note-taker (and
Moderator)Moderator)
Participant says one thing and does another
Participant thinks they have been successful when they really
haven’t
Participant doesn’t want to look bad, so bluffs
Participant initially struggles, then succeeds, and speaks highly of
the product
Participant blames self and rates product highly despite
disastrous task performance
Participant is overly chatty and goes off on tangents
21
22. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
More Challenges for the Note-TakerMore Challenges for the Note-Taker
Being selective in the information you note, without introducing
your biases (seeing what you want/expect to see)
Noting participant behavior versus comments
Whether to try to capture participant’s clickpath, menu choices,
data entry
Whether to capture timing
Whether to use a shorthand “code” for observations
Whether to capture participant’s non-verbal behavior
Whether to note good clips for a highlights video
22
23.
24. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Practice ExercisePractice Exercise
Product: Blood glucose meter
Participant: Person with diabetes who uses a meter
Task: Change default settings in the meter
Rating scale for scoring task completion:
• Completed with ease: Participant easily completed the task (Score: 3)
• Completed with minor difficulty: Participant somewhat struggled to
complete the task (e.g., attempted more than two paths) (Score: 2)
• Completed with major difficulty: Participant significantly struggled
to complete the task (e.g., attempted numerous paths and/or used
customer service) (Score: 1)
• Failed to complete: Participant was unable to complete the task
(Score: 0)
24
25. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
ExerciseExercise
Think about and/or record your observations as you watch this
video clip
25
26. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
DiscussionDiscussion
What did you note?
Did you capture the key findings?
Did you miss anything?
What score did you give the participant?
• Completed with ease: Participant easily completed the task (Score: 3)
• Completed with minor difficulty: Participant somewhat struggled to
complete the task (e.g., attempted more than two paths) (Score: 2)
• Completed with major difficulty: Participant significantly struggled to
complete the task (e.g., attempted numerous paths and/or used customer
service) (Score: 1)
• Failed to complete: Participant was unable to complete the task (Score: 0)
26
27. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Lessons Learned:Lessons Learned: Define MetricsDefine Metrics
Using a task completion rating scale allows:
• Participant performance to be categorized
• Note-taker to record the scores in a systematic and consistent way
Carefully defining the performance categories before the fact
helps the note-taker
• Know what to look for
• Efficiently categorize on the fly
Some additional metrics to consider (facilitated by commercial
data logging software):
• Clickpaths
• Time on task
• Shorthand codes for behaviors or incidents
Example: Number of times a participant clicked the pagination
27
28.
29. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
"Take-Aways" for Effective Note-"Take-Aways" for Effective Note-
TakingTaking
Create a mindset that values the role of the note-taker
Choose someone with appropriate education and experience to
appreciate the design issues of interest
Include the note-taker in test planning discussions and debriefs
Establish project-specific note-taking procedures and coding
schemes; anticipate analysis needs when planning note-taking
Be sure the note-taker is thoroughly familiar with the product
interface, task scenarios, and project goals
29
30. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
And a Few Tips for the EffectiveAnd a Few Tips for the Effective
Note-TakerNote-Taker
Focus on what the participant does, not just what they say
Strive to interpret what you are seeing on the fly and take notes
about that, not just “raw” observations
Review your notes and fill in any gaps as soon as possible after
the session while observations are fresh in your mind
Consult with the team and client mid-course to be sure your notes
reflect project goals and team intent
Check available storage space on recording devices before each
session
Spot check recordings throughout the study to ensure that
everything is working as intended
30
31. The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - UserWorks, Inc.
Questions/Comments?Questions/Comments?
Kristen Davis
UserWorks, Inc.
User Experience Specialist/
Note-Taker Extraordinaire
kdavis@userworks.com
31
Dick Horst
UserWorks, Inc.
President/
Principal User Experience
Specialist
dhorst@userworks.com
www.userworks.com
(301) 431-0500
Editor's Notes
Kristen: Welcome to The Note-Taker's Perspective During Usability Testing: Recognizing What's Important, What’s Not. User experience methodology often focuses on the role and technique of the moderator and may neglect the role of the note-taker, who should be a user experience professional familiar with the project goals and context of use. The note-taker must also be able to identify and efficiently obtain the data you need for actionable reporting. Today, we will be using video clips from actual usability test sessions to facilitate our discussion regarding an effective note-taker’s skillset and collectively we will create a list of note-taking best practices and tips. Copyright UserWorks, Inc. 2013
Kristen: Today we will explore ways to become more efficient and effective note-takers by… [read bulleted list] We will share some of our lessons learned, but we are also here to learn from your experiences, approaches and methods for note-taking. This presentation will use video clips to facilitate a discussion to explore note-taking best practices and tips. We will conduct a series of short exercises wherein we will show a series of video clips from usability tests. You will take your own notes while watching these short video clips. After each clip, we will lead a discussion of what was important and what wasn’t important to note about the test participant’s performance and/or comments during that clip. We will show four video clips. Before each clip there will be a brief setup commentary, providing the context of the clip and the test that it illustrates. After each clip, there will be a brief discussion to share ideas and thoughts about what you noted, given the contextual information that was provided. We hope to illustrate the skills and planning necessary for effective note-taking, and discuss the pros and cons of various styles. We hope to decide if it is possible and worthwhile to even strive for developing principles, which could then be codified and taught. We want to make this an interactive discussion because note-taking is a skill better learned by doing rather than being lectured to, similar to moderating. By a show of hands How many have conducted usability tests? How many have been the note-taker? Copyright UserWorks, Inc. 2008
Kristen: So why is important to be a good note-taker? [After reading bulleted list] All of these help the report basically write itself. Copyright UserWorks, Inc. 2008
Kristen: Copyright UserWorks, Inc. 2008
Kristen: Let’s start with the first exercise. Copyright UserWorks, Inc. 2008
Kristen: Let’s say you are being pulled into a study at the last minute. I’m going to show you a video clip and I want you to think about what you would make note of, you can write them down if this will help you remember what you would note for our discussion. I’m not going to give you any background to the study shown in the clip or to the situation. Other than this is a clip from a usability test session for NIHSeniorHealth.com, which is a health information site designed for seniors. The task she was asked to complete was, “Why do people with diabetes need to check their feet?” Please write down or think about what is important to note. [Video Clip: NIHSeniorHealth.gov phase 1B P4 13:57-16:59] Copyright UserWorks, Inc. 2008
Kristen: Without having the appropriate context it can be difficult to take meaningful notes. What did you note? Did you capture what you thought were the key findings? Did you feel that you missed anything? It’s probably safe to say with a little more context you could improve your notes, and therefore, the report. Copyright UserWorks, Inc. 2008
Dick Copyright UserWorks, Inc. 2008
Kristen: So let’s try this activity again, this time with some background information on the study. Copyright UserWorks, Inc. 2008
Kristen: This is the same study, but a different information retrieval task… [After presenting the bullets] Now, understanding these goals, can anyone think of anything they would have done differently when you were taking notes during the last clip? Copyright UserWorks, Inc. 2008
Kristen: Point out left menu on page and mention the pagination appears at the bottom of the page [Video Clip: SeniorHealth.gov 1B P1 30:07-32:29] Copyright UserWorks, Inc. 2008
Kristen: Now that you had some context your note-taking should improve. As you might recall, the woman from the first video clip quickly went to the correct section of the site, “Self-monitoring,” but she never interacted with the pagination to locate the information located on the 2 nd page. The man from the second video clip did not immediately click into the correct section of the site, “Causes and Preventions”. He did use the pagination, but was unable to locate information on the 2 nd page of “Causes and Preventions.” He also clicked on the “Symptoms and Diagnosis” link when he was already on that page. Copyright UserWorks, Inc. 2008
Kristen: [Incorporate what audience members came up with as well.] Copyright UserWorks, Inc. 2008
Dick Copyright UserWorks, Inc. 2008
Dick: The stenographer style records verbatim what the participants says in a stream-of-consciousness format. This style works best for studies focused on preference data and is structured more like a cognitive interview. When the evaluation focuses more on user action and behavior, this style may not collect the right type of data by missing what the participant is actually doing. This may include body language and verbal tone in addition to interactions with the product being evaluated. Many professionals say this style should not be used. Recording high level interpretation works best when the team is evaluating prototypes. This approach allows the note-taker to record their interpretation of what the user is doing and saying and why. It combines analysis into data collection. If the project timeline is short, this style saves time. However, such “on the fly” interpretations may prove to be incorrect or inaccurate causing the team to review the recordings to reassess the interpretation. This style works best when the team has established clearly defined objectives for each task. Coding behaviors in addition to free form notes allows the team to collect some of the stream-of-consciousness data along with verbal and nonverbal data. Verbal data includes what the participant is saying, whereas nonverbal data includes their interactions with the product being evaluated, and perhaps facial expressions and body language. Copyright UserWorks, Inc. 2008
Kristen: Now, let’s move on to the third exercise. Copyright UserWorks, Inc. 2008
Kristen: Purpose of the site is to provide information to the general public and practitioners about the government’s initiative to increase the use of electronic health records or EHRs. Copyright UserWorks, Inc. 2008
Kristen: [Video clip: HealthIT.gov] Copyright UserWorks, Inc. 2008
Kristen: What did you note? [Ask one person to say what they noted.] By a show of hands, who else noted that? Copyright UserWorks, Inc. 2008
Kristen: Which do you think we should have used for this exercise? Why? [A: High-level interpretation.] [Incorporate what audience members came up with as well.] Copyright UserWorks, Inc. 2008
Dick Copyright UserWorks, Inc. 2008
Dick Copyright UserWorks, Inc. 2008
Kristen: Now, let’s move on to the fourth exercise. This time, we’ll be focusing on metrics. It’s important for all members of the research team to come to a consensus and define all metrics prior to data collection. Copyright UserWorks, Inc. 2008
Kristen Copyright UserWorks, Inc. 2008
Kristen: [Glucose meter study: P1 29:50-32:18] Copyright UserWorks, Inc. 2008
Kristen: By a show of hands how many people rated it a … We scored this task with a 2 (minor difficulty) because the participant tried two different menu items before reaching the correct item, and as defined in the rating scale, that would be a 2. By a show of hands, how many also gave this participant a 2… But, as you can see, that task would have received various ratings had this scale not been established before data collection began. Copyright UserWorks, Inc. 2008
Kristen: [Incorporate what audience members came up with as well.] Identifying what constitutes an assist during the planning phase also allows the team to distinguish the difference between “completing a task with assistance or help” from “completing the task with difficulty.” An “assist” implies the participants would not have completed the task on their own without intervention from the moderator; whereas “completed with difficulty” implies the participant completed the task on his/her own, but struggled to do so, needing several attempts or did not fully understand how the system was working. Copyright UserWorks, Inc. 2008
Dick Copyright UserWorks, Inc. 2008
Dick: Do you have any others? Copyright UserWorks, Inc. 2008
Dick: Do you have any others? Copyright UserWorks, Inc. 2008