Experience Research Best Practices
Upcoming SlideShare
Loading in...5

Experience Research Best Practices



Presentation given to UX MeetUp Boston on 9/19/13. Covers setting goals for research, choosing the appropriate method, and gathering qualitative data.

Presentation given to UX MeetUp Boston on 9/19/13. Covers setting goals for research, choosing the appropriate method, and gathering qualitative data.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Experience Research Best Practices Presentation Transcript

  • 1. September 19, 2013 Dan Berlin Experience Research Director dberlin@madpow.net @banderlin Mad*Pow EXPERIENCE RESEARCH BEST PRACTICES
  • 2. Hi! I’m Dan Berlin @banderlin  BA in psychology from Brandeis U.  Studied visual space perception  Seven years in technical support  Sat as a participant for a usability study for a product I was working on  Realized that user experience (UX) work is the perfect combination of computers and psychology  Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design  Two years at an interactive agency performing usability and neuromarketing research  Then did some freelance UX consulting for about a year  Am now an Experience Research Director at Mad*Pow, an experience design agency based out of Portsmouth, NH  My passion is for research methodology and finding new ways to elicit data from participants 2
  • 3. About this Presentation  Understanding your research goals  Formative and evaluative studies  How to choose the right method(s)  Methods chart  Gathering qualitative data  Taking notes and organizing findings 3
  • 5. Understanding Your Research Goals 5 Preparing to prepare for your study Technology User Needs Business Goals  Your overall goal: To uncover actionable business and design insights via user data  Start by understanding the business’ goals  What is the overall goal of the interface?  What are the interactions that drive the business?  What are the important calls to action?  Who are the target audiences?  What do they want to know about their users?  What do they think they already know about their users  How did they learn this?
  • 6. Understanding Your Research Goals 6 Preparing to prepare for your study  Actionable = the data should indicate what exactly needs to change or be implemented in the interface to align it with users’ expectations and needs  This means you need to start with actionable (and attainable) goals, such as:  How do perform a certain interaction today?  Does the proposed IA align with user expectations?  Are users able to complete a critical transaction?  Do users understand the pricing schedule?  Goals will depend on the type of study  Formative  Performed at the beginning of a project to learn how the project should proceed  Evaluative  Performed during and towards the end of projects to determine if the design aligns with user expectations
  • 7. Formative  In-Person Usability  Interviews  Focus Groups  Collaging  Ethnography  Surveys  Diary Studies  Card-Sorting 7 Evaluative  In-Person Usability  Remote Usability  Unmoderated Usability  Desirability Testing  Eye-Tracking Understanding Your Research Goals Study Types
  • 9. How to Choose the Right Method(s)  Make sure your research goals are clear  Know what you want to learn  Know what you will do with the information you gather  Know what decisions the business needs to make  Know your constraints  Timeline?  Resources?  Budget?  Access to Users?  Create a Methods Chart  The answers will become clear 9 Know Goals and Constraints
  • 10. How to Choose the Right Method(s) 10 Method Pros Cons Candidate Method 1 • List the advantages for this method for this specific project • List the disadvantages for this method for this specific project Yes / No – Would this method be a good candidate for this specific project Method 2 Method 3 Etc… Methods Chart
  • 11. How to Choose the Right Method(s)  Example Project Goals:  Have 3 concepts for behavior change application. Which one is the best to develop for maximum global appeal?  Client wants answers as quickly as possible  Client wants large numbers to provide confidence behind decision  Client wants data collection in 5 countries around the world  Designers want to know why or why not users chose each concept, to provide additional design direction  User population – adults who own a smart phone and want to get healthier 11 Methods Chart
  • 12. Methods Chart Example 12 Method Pros Cons Candidate Interviews • Good for capturing motivations for behavior change • Can get good qualitative details on why users prefer each concept or not • Good to explore issues to fine tune survey questions • Can be done via phone (get broad geographic sample) • Can be done quickly • Small numbers – client wants large numbers Y – would be good as a qualitative method to pair with a larger quantitative method Focus Groups • Good for qualitative information gathering • Could generate some interesting conversations about behavior change motivations • Wouldn’t get as much detail as interviews • Concern about group think when evaluating concepts • Concern about not sharing details of personal goals in front of others N – interviews would be better for qualitative Survey • Good for large numbers • Easily replicated across different countries/languages • Can be done online for broad geographic distribution • Can be done quickly • Unclear what exact questions to ask • Doesn’t provide detailed insights into qualitative topics Y – good paired with qualitative method
  • 13. How to Choose the Right Method(s)  Example Research Plan:  Conduct telephone interviews in US with 12-15 participants  Create online survey in US for 300 participants  Use international research partners to conduct 12-15 interviews in 4 countries  Use international research partners to conduct survey for 300 participants in same 4 countries  Why this approach works:  Interviews provide input into questions for survey  Survey provides large numbers; interviews provide in-depth insight  Both are commonly used methods and easy to replicate in different countries  Both methods can be done quickly 13 Methods Chart
  • 15. Gathering Qualitative Data 15 Usability Task Creation  Good usability study tasks are:  Non-leading: don’t give away the answer  Single tasks: don’t have users do two things in succession  Realistic: don’t have users do things they would not normally do  Self-explanatory: don’t have multiple sentences explaining the task  Achievable: always document what constitutes a “pass” for the task
  • 16. Gathering Qualitative Data 16 Moderator’s Guide  A good study guide:  Conveys the study goals and methodology to clients  Serves as a quick reference to the moderator during the study  Provides the moderator with a template to take hand-written notes, with ample space to do so  The typical moderator’s guide contains:  Research method: a short paragraph explaining the usability study  Study goals: a short bulleted list of the study goals  Introduction: the moderator’s opening spiel when explaining the study to the participant  Background questions: Typically demographic or product usage questions  Tasks: The task, pass condition(s), notes for the moderator, and post-task questions  Follow-up questions: The questions to be asked after all the tasks are complete
  • 17. 17
  • 18. Gathering Qualitative Data 18 Notes Grid Proper planning for taking notes is very underrated  The organization and thoroughness of your notes will dictate the ease with which you will create the final report  Organized, complete notes = easy reporting  Disorganized, incomplete notes = back to the video you go (ewww!)  Make your notes grid once your study & moderator’s guides are complete  Give each question and subquestion its own row  Put each participant in a new column (or vice-versa, if you like)  DO NOT put each participant/task in a new worksheet  Use data validation for quantitative data  Task ease ratings, task success, multiple choice questions, etc.  Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to the question)
  • 19. 19 • Do not include the participant name in your notes grid • Include a date/time cell to best align with the videos • Visually separate sections of the study • Fill data validated cells with a light color • Hide columns when you have moved on to the next participant Gathering Qualitative Data Notes Grid
  • 20. 20 Gathering Qualitative Data Findings Sheet  Scan the notes grid to determine findings  Put these in a spreadsheet and assign and priority and category
  • 21. In Conclusion  Set actionable and attainable research goals  Know where you are in the project  Use a methods chart to weigh factors and choose a method  Time, participants, goals, budget, etc.  Proper documentation = easier qualitative analysis  Study guide  notes grid  findings spreadsheet  report 21