Your SlideShare is downloading. ×
Experience Research Best Practices
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Experience Research Best Practices


Published on

Presentation given to UX MeetUp Boston on 9/19/13. Covers setting goals for research, choosing the appropriate method, and gathering qualitative data.

Presentation given to UX MeetUp Boston on 9/19/13. Covers setting goals for research, choosing the appropriate method, and gathering qualitative data.

Published in: Design, Technology, Business

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. September 19, 2013 Dan Berlin Experience Research Director @banderlin Mad*Pow EXPERIENCE RESEARCH BEST PRACTICES
  • 2. Hi! I’m Dan Berlin @banderlin  BA in psychology from Brandeis U.  Studied visual space perception  Seven years in technical support  Sat as a participant for a usability study for a product I was working on  Realized that user experience (UX) work is the perfect combination of computers and psychology  Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design  Two years at an interactive agency performing usability and neuromarketing research  Then did some freelance UX consulting for about a year  Am now an Experience Research Director at Mad*Pow, an experience design agency based out of Portsmouth, NH  My passion is for research methodology and finding new ways to elicit data from participants 2
  • 3. About this Presentation  Understanding your research goals  Formative and evaluative studies  How to choose the right method(s)  Methods chart  Gathering qualitative data  Taking notes and organizing findings 3
  • 5. Understanding Your Research Goals 5 Preparing to prepare for your study Technology User Needs Business Goals  Your overall goal: To uncover actionable business and design insights via user data  Start by understanding the business’ goals  What is the overall goal of the interface?  What are the interactions that drive the business?  What are the important calls to action?  Who are the target audiences?  What do they want to know about their users?  What do they think they already know about their users  How did they learn this?
  • 6. Understanding Your Research Goals 6 Preparing to prepare for your study  Actionable = the data should indicate what exactly needs to change or be implemented in the interface to align it with users’ expectations and needs  This means you need to start with actionable (and attainable) goals, such as:  How do perform a certain interaction today?  Does the proposed IA align with user expectations?  Are users able to complete a critical transaction?  Do users understand the pricing schedule?  Goals will depend on the type of study  Formative  Performed at the beginning of a project to learn how the project should proceed  Evaluative  Performed during and towards the end of projects to determine if the design aligns with user expectations
  • 7. Formative  In-Person Usability  Interviews  Focus Groups  Collaging  Ethnography  Surveys  Diary Studies  Card-Sorting 7 Evaluative  In-Person Usability  Remote Usability  Unmoderated Usability  Desirability Testing  Eye-Tracking Understanding Your Research Goals Study Types
  • 9. How to Choose the Right Method(s)  Make sure your research goals are clear  Know what you want to learn  Know what you will do with the information you gather  Know what decisions the business needs to make  Know your constraints  Timeline?  Resources?  Budget?  Access to Users?  Create a Methods Chart  The answers will become clear 9 Know Goals and Constraints
  • 10. How to Choose the Right Method(s) 10 Method Pros Cons Candidate Method 1 • List the advantages for this method for this specific project • List the disadvantages for this method for this specific project Yes / No – Would this method be a good candidate for this specific project Method 2 Method 3 Etc… Methods Chart
  • 11. How to Choose the Right Method(s)  Example Project Goals:  Have 3 concepts for behavior change application. Which one is the best to develop for maximum global appeal?  Client wants answers as quickly as possible  Client wants large numbers to provide confidence behind decision  Client wants data collection in 5 countries around the world  Designers want to know why or why not users chose each concept, to provide additional design direction  User population – adults who own a smart phone and want to get healthier 11 Methods Chart
  • 12. Methods Chart Example 12 Method Pros Cons Candidate Interviews • Good for capturing motivations for behavior change • Can get good qualitative details on why users prefer each concept or not • Good to explore issues to fine tune survey questions • Can be done via phone (get broad geographic sample) • Can be done quickly • Small numbers – client wants large numbers Y – would be good as a qualitative method to pair with a larger quantitative method Focus Groups • Good for qualitative information gathering • Could generate some interesting conversations about behavior change motivations • Wouldn’t get as much detail as interviews • Concern about group think when evaluating concepts • Concern about not sharing details of personal goals in front of others N – interviews would be better for qualitative Survey • Good for large numbers • Easily replicated across different countries/languages • Can be done online for broad geographic distribution • Can be done quickly • Unclear what exact questions to ask • Doesn’t provide detailed insights into qualitative topics Y – good paired with qualitative method
  • 13. How to Choose the Right Method(s)  Example Research Plan:  Conduct telephone interviews in US with 12-15 participants  Create online survey in US for 300 participants  Use international research partners to conduct 12-15 interviews in 4 countries  Use international research partners to conduct survey for 300 participants in same 4 countries  Why this approach works:  Interviews provide input into questions for survey  Survey provides large numbers; interviews provide in-depth insight  Both are commonly used methods and easy to replicate in different countries  Both methods can be done quickly 13 Methods Chart
  • 15. Gathering Qualitative Data 15 Usability Task Creation  Good usability study tasks are:  Non-leading: don’t give away the answer  Single tasks: don’t have users do two things in succession  Realistic: don’t have users do things they would not normally do  Self-explanatory: don’t have multiple sentences explaining the task  Achievable: always document what constitutes a “pass” for the task
  • 16. Gathering Qualitative Data 16 Moderator’s Guide  A good study guide:  Conveys the study goals and methodology to clients  Serves as a quick reference to the moderator during the study  Provides the moderator with a template to take hand-written notes, with ample space to do so  The typical moderator’s guide contains:  Research method: a short paragraph explaining the usability study  Study goals: a short bulleted list of the study goals  Introduction: the moderator’s opening spiel when explaining the study to the participant  Background questions: Typically demographic or product usage questions  Tasks: The task, pass condition(s), notes for the moderator, and post-task questions  Follow-up questions: The questions to be asked after all the tasks are complete
  • 17. 17
  • 18. Gathering Qualitative Data 18 Notes Grid Proper planning for taking notes is very underrated  The organization and thoroughness of your notes will dictate the ease with which you will create the final report  Organized, complete notes = easy reporting  Disorganized, incomplete notes = back to the video you go (ewww!)  Make your notes grid once your study & moderator’s guides are complete  Give each question and subquestion its own row  Put each participant in a new column (or vice-versa, if you like)  DO NOT put each participant/task in a new worksheet  Use data validation for quantitative data  Task ease ratings, task success, multiple choice questions, etc.  Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to the question)
  • 19. 19 • Do not include the participant name in your notes grid • Include a date/time cell to best align with the videos • Visually separate sections of the study • Fill data validated cells with a light color • Hide columns when you have moved on to the next participant Gathering Qualitative Data Notes Grid
  • 20. 20 Gathering Qualitative Data Findings Sheet  Scan the notes grid to determine findings  Put these in a spreadsheet and assign and priority and category
  • 21. In Conclusion  Set actionable and attainable research goals  Know where you are in the project  Use a methods chart to weigh factors and choose a method  Time, participants, goals, budget, etc.  Proper documentation = easier qualitative analysis  Study guide  notes grid  findings spreadsheet  report 21