www.userworks.com
(301) 431-0500
Kristen Davis & Dick Horst
UXPA International Conference 2013
July 10, 2013
The Note-Take...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
The Note-Taker's Perspective During Usability Testing
UXPA International Conference 2013
Kristen Davis and Dick Horst - Us...
UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective
Upcoming SlideShare
Loading in …5
×

UXPA International 2013 The Note-Taker's Perspective

613 views

Published on

Kristen Davis's and Dick Horst's 2013 UXPA International presentation on The Note-Taker's Perspective During Usability Testing: Recognizing What's Important, What’s Not.

Published in: Technology, Business
  • Be the first to comment

UXPA International 2013 The Note-Taker's Perspective

  1. 1. www.userworks.com (301) 431-0500 Kristen Davis & Dick Horst UXPA International Conference 2013 July 10, 2013 The Note-Taker's PerspectiveThe Note-Taker's Perspective During Usability Testing:During Usability Testing: Recognizing What's Important, What'sRecognizing What's Important, What's NotNot
  2. 2. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. OverviewOverview Presentation Objectives: To collectively explore ways to become a more efficient and effective note-taker by: • Developing the skills and planning necessary • Recognizing what’s important to note and what’s not • Understanding the pros and cons of various note-taking styles • Determining what metrics to define and utilize in note-taking • Creating a list of note-taking best practices and tips 2
  3. 3. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Why is Effective Note-TakingWhy is Effective Note-Taking Important?Important?  To improve the data analysis process by: • Capturing the appropriate data the first time; less need for reviewing recordings • Making analyses more efficient by categorizing observations on the fly • Increasing accuracy and completeness of notes (so you don’t miss anything important) • Identifying trends across participants 3
  4. 4. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Typical ContextTypical Context  One-on-one, moderated sessions (with the participant in-person or remote)  Moderator interacting with the participant, note-taker in the background taking notes  Participant attempting typical task scenarios at the direction of the moderator  Participant thinking aloud Many of the same principles and best practices would also apply to taking notes in focus groups, user interviews, or ethnographic settings, but not all 4
  5. 5. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 6
  6. 6. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture what you thought were the key findings?  Did you miss anything? 7
  7. 7. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Note-Taking MisconceptionsNote-Taking Misconceptions  No preparation needed  Take the notes, then figure out how to analyze and summarize the data  Anyone can do it  Attention to detail, fast typing are the key skills  A one-size-fits-all approach will work for all practitioners and for all products under evaluation 8
  8. 8. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. The ContextThe Context  Product: NIHSeniorHealth.gov website  Participant: Seniors who look for health information online  Task: What types of food or drink are related to balance problems?  Study goals: • What navigation path do participants use when looking for specific pieces of information? • Do participants notice and use the pagination? • Does the visual treatment used in the left hand navigation clearly indicate location within a specific health topic?  Task goals:  Are participants able to locate information on the 2nd page?  Are the left hand navigation labels clear? 10
  9. 9. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 11
  10. 10. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything?  Did you note any of these: • Study goals:  What navigation path do participants use when looking for specific pieces of information?  Do participants notice and use the pagination?  Does the visual design treatment used in the left hand navigation clearly indicate location within the topic? • Task goals:  Are participants able to locate information on the 2nd page?  Are the left hand navigation labels clear? 12
  11. 11. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned: Understand theLessons Learned: Understand the GoalsGoals  Important to be familiar with the product and understand the overall goals of the study and the specific goals of the tasks  Goals should be set prior to data collection in order to: • Record appropriate data • Target the type of data that will be useful when summarizing and reporting the findings 13
  12. 12. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Different Note-Taking ApproachesDifferent Note-Taking Approaches  Would you try to note: • Everything the participant says? • Everything the participant does?  Pages visited  Links visited  Scrolling, hovering  Participant’s demeanor  Non-verbal signs  Or would you be more selective? • Task scoring (success vs failure) • Errors, wrong paths • Pre-identified issues of interest • Occasional verbatims 14
  13. 13. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Different Note-Taking StylesDifferent Note-Taking Styles  Moderator also takes notes  The court stenographer, verbatim  High level interpretation only, not behavior or comment  Coding behaviors in addition to free form notes  Working from a checklist  Multiple note-takers looking for different things  Multiple note-takers working redundantly  Taking notes from playback of a video/audio recording of the session  Using a tool like Morae, Ovo, Silverback, free form typing into a word processing or spreadsheet program, or paper-based (checklist) 15
  14. 14. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Practice ExercisePractice Exercise  Product: HealthIT.gov website  Participant: Private physician  Task: Medical professionals and hospitals are encouraged to participate in the Government EHR reimbursement programs, early, to receive the maximum payment. Find out when the Government will start or has started these reimbursement payments.  Study goals: • Are the labels used for navigation clear? • Does the information architecture make sense? • What suggestions do participants have to improve the content?  Task goal: • What navigation path do participants use when looking for specific pieces of information? 17
  15. 15. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 18
  16. 16. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything? 19
  17. 17. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned: RecordLessons Learned: Record Only KeyOnly Key PointsPoints  The method of note-taking impacts the type of data collected during the evaluation  Depending on the evaluation objective, one style or a combination of styles may be more appropriate 20
  18. 18. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Challenges for the Note-taker (andChallenges for the Note-taker (and Moderator)Moderator)  Participant says one thing and does another  Participant thinks they have been successful when they really haven’t  Participant doesn’t want to look bad, so bluffs  Participant initially struggles, then succeeds, and speaks highly of the product  Participant blames self and rates product highly despite disastrous task performance  Participant is overly chatty and goes off on tangents 21
  19. 19. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. More Challenges for the Note-TakerMore Challenges for the Note-Taker  Being selective in the information you note, without introducing your biases (seeing what you want/expect to see)  Noting participant behavior versus comments  Whether to try to capture participant’s clickpath, menu choices, data entry  Whether to capture timing  Whether to use a shorthand “code” for observations  Whether to capture participant’s non-verbal behavior  Whether to note good clips for a highlights video 22
  20. 20. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Practice ExercisePractice Exercise  Product: Blood glucose meter  Participant: Person with diabetes who uses a meter  Task: Change default settings in the meter  Rating scale for scoring task completion: • Completed with ease: Participant easily completed the task (Score: 3) • Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2) • Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1) • Failed to complete: Participant was unable to complete the task (Score: 0) 24
  21. 21. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. ExerciseExercise  Think about and/or record your observations as you watch this video clip 25
  22. 22. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. DiscussionDiscussion  What did you note?  Did you capture the key findings?  Did you miss anything?  What score did you give the participant? • Completed with ease: Participant easily completed the task (Score: 3) • Completed with minor difficulty: Participant somewhat struggled to complete the task (e.g., attempted more than two paths) (Score: 2) • Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) (Score: 1) • Failed to complete: Participant was unable to complete the task (Score: 0) 26
  23. 23. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Lessons Learned:Lessons Learned: Define MetricsDefine Metrics  Using a task completion rating scale allows: • Participant performance to be categorized • Note-taker to record the scores in a systematic and consistent way  Carefully defining the performance categories before the fact helps the note-taker • Know what to look for • Efficiently categorize on the fly  Some additional metrics to consider (facilitated by commercial data logging software): • Clickpaths • Time on task • Shorthand codes for behaviors or incidents  Example: Number of times a participant clicked the pagination 27
  24. 24. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. "Take-Aways" for Effective Note-"Take-Aways" for Effective Note- TakingTaking  Create a mindset that values the role of the note-taker  Choose someone with appropriate education and experience to appreciate the design issues of interest  Include the note-taker in test planning discussions and debriefs  Establish project-specific note-taking procedures and coding schemes; anticipate analysis needs when planning note-taking  Be sure the note-taker is thoroughly familiar with the product interface, task scenarios, and project goals 29
  25. 25. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. And a Few Tips for the EffectiveAnd a Few Tips for the Effective Note-TakerNote-Taker  Focus on what the participant does, not just what they say  Strive to interpret what you are seeing on the fly and take notes about that, not just “raw” observations  Review your notes and fill in any gaps as soon as possible after the session while observations are fresh in your mind  Consult with the team and client mid-course to be sure your notes reflect project goals and team intent  Check available storage space on recording devices before each session  Spot check recordings throughout the study to ensure that everything is working as intended 30
  26. 26. The Note-Taker's Perspective During Usability Testing UXPA International Conference 2013 Kristen Davis and Dick Horst - UserWorks, Inc. Questions/Comments?Questions/Comments? Kristen Davis UserWorks, Inc. User Experience Specialist/ Note-Taker Extraordinaire kdavis@userworks.com 31 Dick Horst UserWorks, Inc. President/ Principal User Experience Specialist dhorst@userworks.com www.userworks.com (301) 431-0500

×