Experience Research Best Practices


Published on

Presentation given to UX MeetUp Boston on 9/19/13. Covers setting goals for research, choosing the appropriate method, and gathering qualitative data.

Published in: Design, Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Experience Research Best Practices

  1. 1. September 19, 2013 Dan Berlin Experience Research Director dberlin@madpow.net @banderlin Mad*Pow EXPERIENCE RESEARCH BEST PRACTICES
  2. 2. Hi! I’m Dan Berlin @banderlin  BA in psychology from Brandeis U.  Studied visual space perception  Seven years in technical support  Sat as a participant for a usability study for a product I was working on  Realized that user experience (UX) work is the perfect combination of computers and psychology  Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design  Two years at an interactive agency performing usability and neuromarketing research  Then did some freelance UX consulting for about a year  Am now an Experience Research Director at Mad*Pow, an experience design agency based out of Portsmouth, NH  My passion is for research methodology and finding new ways to elicit data from participants 2
  3. 3. About this Presentation  Understanding your research goals  Formative and evaluative studies  How to choose the right method(s)  Methods chart  Gathering qualitative data  Taking notes and organizing findings 3
  5. 5. Understanding Your Research Goals 5 Preparing to prepare for your study Technology User Needs Business Goals  Your overall goal: To uncover actionable business and design insights via user data  Start by understanding the business’ goals  What is the overall goal of the interface?  What are the interactions that drive the business?  What are the important calls to action?  Who are the target audiences?  What do they want to know about their users?  What do they think they already know about their users  How did they learn this?
  6. 6. Understanding Your Research Goals 6 Preparing to prepare for your study  Actionable = the data should indicate what exactly needs to change or be implemented in the interface to align it with users’ expectations and needs  This means you need to start with actionable (and attainable) goals, such as:  How do perform a certain interaction today?  Does the proposed IA align with user expectations?  Are users able to complete a critical transaction?  Do users understand the pricing schedule?  Goals will depend on the type of study  Formative  Performed at the beginning of a project to learn how the project should proceed  Evaluative  Performed during and towards the end of projects to determine if the design aligns with user expectations
  7. 7. Formative  In-Person Usability  Interviews  Focus Groups  Collaging  Ethnography  Surveys  Diary Studies  Card-Sorting 7 Evaluative  In-Person Usability  Remote Usability  Unmoderated Usability  Desirability Testing  Eye-Tracking Understanding Your Research Goals Study Types
  9. 9. How to Choose the Right Method(s)  Make sure your research goals are clear  Know what you want to learn  Know what you will do with the information you gather  Know what decisions the business needs to make  Know your constraints  Timeline?  Resources?  Budget?  Access to Users?  Create a Methods Chart  The answers will become clear 9 Know Goals and Constraints
  10. 10. How to Choose the Right Method(s) 10 Method Pros Cons Candidate Method 1 • List the advantages for this method for this specific project • List the disadvantages for this method for this specific project Yes / No – Would this method be a good candidate for this specific project Method 2 Method 3 Etc… Methods Chart
  11. 11. How to Choose the Right Method(s)  Example Project Goals:  Have 3 concepts for behavior change application. Which one is the best to develop for maximum global appeal?  Client wants answers as quickly as possible  Client wants large numbers to provide confidence behind decision  Client wants data collection in 5 countries around the world  Designers want to know why or why not users chose each concept, to provide additional design direction  User population – adults who own a smart phone and want to get healthier 11 Methods Chart
  12. 12. Methods Chart Example 12 Method Pros Cons Candidate Interviews • Good for capturing motivations for behavior change • Can get good qualitative details on why users prefer each concept or not • Good to explore issues to fine tune survey questions • Can be done via phone (get broad geographic sample) • Can be done quickly • Small numbers – client wants large numbers Y – would be good as a qualitative method to pair with a larger quantitative method Focus Groups • Good for qualitative information gathering • Could generate some interesting conversations about behavior change motivations • Wouldn’t get as much detail as interviews • Concern about group think when evaluating concepts • Concern about not sharing details of personal goals in front of others N – interviews would be better for qualitative Survey • Good for large numbers • Easily replicated across different countries/languages • Can be done online for broad geographic distribution • Can be done quickly • Unclear what exact questions to ask • Doesn’t provide detailed insights into qualitative topics Y – good paired with qualitative method
  13. 13. How to Choose the Right Method(s)  Example Research Plan:  Conduct telephone interviews in US with 12-15 participants  Create online survey in US for 300 participants  Use international research partners to conduct 12-15 interviews in 4 countries  Use international research partners to conduct survey for 300 participants in same 4 countries  Why this approach works:  Interviews provide input into questions for survey  Survey provides large numbers; interviews provide in-depth insight  Both are commonly used methods and easy to replicate in different countries  Both methods can be done quickly 13 Methods Chart
  15. 15. Gathering Qualitative Data 15 Usability Task Creation  Good usability study tasks are:  Non-leading: don’t give away the answer  Single tasks: don’t have users do two things in succession  Realistic: don’t have users do things they would not normally do  Self-explanatory: don’t have multiple sentences explaining the task  Achievable: always document what constitutes a “pass” for the task
  16. 16. Gathering Qualitative Data 16 Moderator’s Guide  A good study guide:  Conveys the study goals and methodology to clients  Serves as a quick reference to the moderator during the study  Provides the moderator with a template to take hand-written notes, with ample space to do so  The typical moderator’s guide contains:  Research method: a short paragraph explaining the usability study  Study goals: a short bulleted list of the study goals  Introduction: the moderator’s opening spiel when explaining the study to the participant  Background questions: Typically demographic or product usage questions  Tasks: The task, pass condition(s), notes for the moderator, and post-task questions  Follow-up questions: The questions to be asked after all the tasks are complete
  17. 17. 17
  18. 18. Gathering Qualitative Data 18 Notes Grid Proper planning for taking notes is very underrated  The organization and thoroughness of your notes will dictate the ease with which you will create the final report  Organized, complete notes = easy reporting  Disorganized, incomplete notes = back to the video you go (ewww!)  Make your notes grid once your study & moderator’s guides are complete  Give each question and subquestion its own row  Put each participant in a new column (or vice-versa, if you like)  DO NOT put each participant/task in a new worksheet  Use data validation for quantitative data  Task ease ratings, task success, multiple choice questions, etc.  Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to the question)
  19. 19. 19 • Do not include the participant name in your notes grid • Include a date/time cell to best align with the videos • Visually separate sections of the study • Fill data validated cells with a light color • Hide columns when you have moved on to the next participant Gathering Qualitative Data Notes Grid
  20. 20. 20 Gathering Qualitative Data Findings Sheet  Scan the notes grid to determine findings  Put these in a spreadsheet and assign and priority and category
  21. 21. In Conclusion  Set actionable and attainable research goals  Know where you are in the project  Use a methods chart to weigh factors and choose a method  Time, participants, goals, budget, etc.  Proper documentation = easier qualitative analysis  Study guide  notes grid  findings spreadsheet  report 21