UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt

516 views

Published on

Kristen Davis and Dick Horst from UserWorks presentation slides on the "Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not" from UXPA-DC Conference Redux 2013

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
516
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Not much focus put on notetaking as with moderating during observed research. Based on our practical experience with have learned some tips for notetaking.
  • Kristen: Today we will explore ways to become more efficient and effective note-takers by…[read bulleted list]We will share some of our lessons learned, but we are also here to learn from your experiences, approaches and methods for note-taking.This presentation will use video clips to facilitate a discussion to explore note-taking best practices and tips. We will conduct a series of short exercises wherein we will show a series of video clips from usability tests. You will take your own notes while watching these short video clips. After each clip, we will lead a discussion of what was important and what wasn’t important to note about the test participant’s performance and/or comments during that clip. We will show four video clips. Before each clip there will be a brief setup commentary, providing the context of the clip and the test that it illustrates. After each clip, there will be a brief discussion to share ideas and thoughts about what you noted, given the contextual information that was provided. We hope to illustrate the skills and planning necessary for effective note-taking, and discuss the pros and cons of various styles. We hope to decide if it is possible and worthwhile to even strive for developing principles, which could then be codified and taught. We want to make this an interactive discussion because note-taking is a skill better learned by doing rather than being lectured to, similar to moderating.By a show of handsHow many have conducted usability tests?How many have been the note-taker?
  • Kristen: So why is important to be a good notetaker?[After reading bulleted list] All of these help the report basically write itself.Second sub-bullet:Making analyses more efficient by categorizing observations on fly
  • Kristen:
  • Kristen: Let’s begin by discussing the planning phase of the project and the necessary steps to play for notetaking.
  • Second bullet: Just take notes, then figure out how to analyze and data summarizeLast bullet: One-size-fits-all approach will work for all practitioners and for all products under evaluation
  • First bullet: Important to be familiar with the product and understand the overall goals of the study and the specific goals of the tasks
  • The method of notetaking impacts the type of data collected during the evaluation and target the type of data that will be useful when summarizing and reporting the findingsDepending on the evaluation objective, one style or a combination of styles may be more appropriate
  • Identifying what constitutes an assist during the planning phase also allows the team to distinguish the difference between “completing a task with assistance or help” from “completing the task with difficulty.” An “assist” implies the participants would not have completed the task on their own without intervention from the moderator; whereas “completed with difficulty” implies the participant completed the task on his/her own, but struggled to do so, needing several attempts or did not fully understand how the system was working.Helps the notetaker:Know what to look forEfficiently categorize on the fly
  • Kristen: So let’s see an example were understand the study context and background would help improve notetaking.
  • Kristen: This is the same study, but a different information retrieval task…[After presenting the bullets] Now, understanding these goals, can anyone think of anything they would have done differently when you were taking notes during the last clip?
  • Kristen:Point out left menu on page and mention the pagination appears at the bottom of the page[Video Clip: SeniorHealth.gov 1B P1 30:07-32:29]The participantdid not immediately click into the correct section of the site, “Causes and Preventions”. He did use the pagination, but was unable to locate information on the 2nd page of “Causes and Preventions.” He also clicked on the “Symptoms and Diagnosis” link when he was already on that page.
  • Kristen: Now, let’s move on to the third exercise.
  • Kristen: Purpose of the site is to provide information to the general public and practitioners about the government’s initiative to increase the use of electronic health records or EHRs.
  • Kristen: [Video clip: HealthIT.gov]For this study we high level interpretation notetaking style. If we had tried to take verbatim notes, we wouldn’t have collected a lot of meaningful data as related to the study goals.
  • Kristen: Now, let’s move on to the fourth exercise.This time, we’ll be focusing on metrics. It’s important for all members of the research team to come to a consensus and define all metrics prior to data collection.
  • Kristen
  • Kristen: [Glucose meter study: P1 29:50-32:18]We scored this task with a 2 (minor difficulty) because the participant tried two different menu items before reaching the correct item, and as defined in the rating scale, that would be a 2. By a show of hands, how many also gave this participant a 2…But, as you can see, that task would have received various ratings had this scale not been established before data collection began.
  • Dick
  • Dick
  • Dick
  • Dick
  • Review your notes and fill in any gaps as soon as possible after sessions while observations are fresh in your mindConsult with the team and client mid-course to be sure your notes reflect project goals and team intentSpot check recordings throughout study to ensure that everything is working as intended
  • Kristen
  • UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt

    1. 1. Notetaker's Perspective During Usability Testing: Recognizing What's Important, What's Not UXPA-DC Conference Redux 2013 October 25, 2013 Kristen Davis ● Dick Horst kdavis@userworks.com ● dhorst@userworks.com 1738 Elton Rd., Suite 138, Silver Spring, MD 20903 (301) 431-0500  www.userworks.com
    2. 2. Presentation Objectives  To explore ways to become a more efficient/effective notetaker by: • • • • • Developing notetaking and test planning skills Recognizing what’s important to note and what’s not Understanding the pros and cons of various styles Determining what metrics to capture and how to define them Discussing a list of best practices and tips UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 2
    3. 3. Why is Effective Notetaking Important?  To improve the data analysis process by: • • • • Capturing the appropriate data the first time Making analyses more efficient Increasing accuracy and completeness of data Identifying trends across participants UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 3
    4. 4. Typical Context  One-on-one, moderated sessions (participant in-person or remote)  Moderator interacting with participant, notetaker elsewhere  Participant attempting typical task scenarios at the direction of the moderator  Participant thinking aloud Some usability test principles and best practices may apply to taking notes in focus groups, user interviews, or ethnographic settings, but not all UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 4
    5. 5. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    6. 6. Misconceptions      No preparation needed Just take notes Anyone can do it Fast typing is only key skill necessary One-size-fits-all approach UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 6
    7. 7. Set Goals     Be familiar with the product under test Understand overall study goals Understand specific task goals Set goals prior to data collection • • Record appropriate data Target type(s) of data UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 7
    8. 8. Notetaking Styles/Approaches    Determine appropriate style • • • Verbatim (court stenographer) – rarely ideal High level interpretation only, not behavior or comments Coding behaviors in addition to free form notes Determine number of notetakers • • • • Moderator is only notetaker – not ideal Multiple notetakers looking for different things Multiple notetakers covering the same things (redundancy) Taking notes from playback of a video/audio recording (not live) Determine data collection tools • • Checklist (word processing or spreadsheet based, or paper-based) Automated tool (e.g., Morae, Ovo, Silverback) UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 8
    9. 9. Data Collection    Define task completion rating scale • • Participant performance to be categorized Systematic and consistent scoring Notetaker can quickly record • • • • Task scoring (success vs failure) Errors, wrong paths Pre-identified issues of interest Occasional verbatims Data logging software records • • • Clickpaths Time on task Shorthand codes for behaviors or incidents UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 9
    10. 10. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    11. 11. Study Context    Product: NIHSeniorHealth.gov website Participants: Seniors who look for health information online Study goals: • • •  What navigation path do participants use when looking for specific pieces of information? Do participants notice and use the pagination? Does the visual treatment used in left hand navigation clearly indicate location within specific health topic? Task goals:  Are participants able to locate information on the 2nd page?  Are the left hand navigation labels clear?  Task: “Why do people with diabetes need to check their feet?” UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 11
    12. 12. Video Example UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 12
    13. 13. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    14. 14. Study Context     Product: HealthIT.gov website Participants: Private physicians Study goals: • • • Are navigation labels clear? Does information architecture make sense? What suggestions do participants have to improve content? Task goal: • What navigation path do participants use when looking for specific pieces of information?  Task: “Medical professionals and hospitals are encouraged to participate in the Government EHR reimbursement programs early to receive the maximum payment. Find out when the Government will start or has started these reimbursement payments.” UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 14
    15. 15. Video Example UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 15
    16. 16. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    17. 17. Study Context    Product: Blood glucose meter Participants: People with diabetes who use glucose meter Rating scale for scoring task completion: • • • •  Completed with ease: Participant easily completed the task Completed with minor difficulty: Participant struggled somewhat to complete the task (e.g., attempted more than two paths) Completed with major difficulty: Participant significantly struggled to complete the task (e.g., attempted numerous paths and/or used customer service) Failed to complete: Participant was unable to complete the task Task: Change meter default settings UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 17
    18. 18. Video Example UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 18
    19. 19. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    20. 20. Pitfalls       Participant says one thing and does another Participant thinks they have been successful when they really haven’t Participant doesn’t want to look bad, so bluffs Participant initially struggles, then succeeds, and speaks highly of product Participant blames self and rates product highly despite disastrous task performance Participant is overly chatty and goes off on tangents UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 20
    21. 21. Considerations        Being selective in information noted, without introducing biases (seeing what you want/expect to see) Noting participant behavior versus comments Whether to try to capture participant’s clickpath, menu choices, data entry Whether to capture timing Whether to use a shorthand “code” for observations Whether to capture participant’s non-verbal behavior Whether to note good clips for a highlights video UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 21
    22. 22. UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013
    23. 23. "Take-Aways" for Effective Notetaking (1 of 2)      Ensure the project team values role of notetaker Choose someone with appropriate education and experience Include notetaker in test planning discussions and project debriefs Establish project-specific notetaking procedures and coding schemes Be sure notetaker is thoroughly familiar with: • • • • Product Project goals Task scenarios Task goals UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 23
    24. 24. "Take-Aways" for Effective Notetaking (2 of 2)        Focus on what participant does, not just what they say Just don’t note observations; interpret what you are seeing Review your notes and fill in any gaps as soon as possible after sessions Consult with the team and client throughout project Check available storage space on recording devices before each session Spot check recordings throughout study in-between sessions Discuss findings with notetaker/moderator after each session, if time allows UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 24
    25. 25. Questions/Comments Kristen Davis Dick Horst UserWorks, Inc. UserWorks, Inc. Senior User Experience Specialist/ Note-Taker Extraordinaire President/ Principal User Experience Specialist kdavis@userworks.com dhorst@userworks.com www.userworks.com (301) 431-0500 UXPA-DC Conference Redux 2013 UserWorks, Inc. October 25, 2013 25

    ×