Experience Research Best Practices - UX Meet Up Boston 2013 - Dan Berlin

1. September 19, 2013
Dan Berlin
Experience Research Director
dberlin@madpow.net
@banderlin
Mad*Pow
EXPERIENCE RESEARCH BEST PRACTICES

2. Hi! I’m Dan Berlin @banderlin
 BA in psychology from Brandeis U.
 Studied visual space perception
 Seven years in technical support
 Sat as a participant for a usability study for a product I was working on
 Realized that user experience (UX) work is the perfect combination of computers and psychology
 Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design
 Two years at an interactive agency performing usability and neuromarketing research
 Then did some freelance UX consulting for about a year
 Am now an Experience Research Director at Mad*Pow, an experience design agency based out of
Portsmouth, NH
 My passion is for research methodology and finding new ways to elicit data from participants
2

3. About this Presentation
 Understanding your research goals
 Formative and evaluative studies
 How to choose the right method(s)
 Methods chart
 Gathering qualitative data
 Taking notes and organizing findings
3

4. UNDERSTANDING YOUR
RESEARCH GOALS
4

5. Understanding Your Research Goals
5
Preparing to prepare for your study
Technology
User
Needs
Business
Goals
 Your overall goal:
To uncover actionable business and
design insights via user data
 Start by understanding the business’ goals
 What is the overall goal of the interface?
 What are the interactions that drive the business?
 What are the important calls to action?
 Who are the target audiences?
 What do they want to know about their users?
 What do they think they already know about their users
 How did they learn this?

6. Understanding Your Research Goals
6
Preparing to prepare for your study
 Actionable = the data should indicate what exactly needs to change or be implemented in
the interface to align it with users’ expectations and needs
 This means you need to start with actionable (and attainable) goals, such as:
 How do perform a certain interaction today?
 Does the proposed IA align with user expectations?
 Are users able to complete a critical transaction?
 Do users understand the pricing schedule?
 Goals will depend on the type of study
 Formative
 Performed at the beginning of a project to learn how the project should proceed
 Evaluative
 Performed during and towards the end of projects to determine if the design aligns with user expectations

7. Formative
 In-Person Usability
 Interviews
 Focus Groups
 Collaging
 Ethnography
 Surveys
 Diary Studies
 Card-Sorting
7
Evaluative
 In-Person Usability
 Remote Usability
 Unmoderated Usability
 Desirability Testing
 Eye-Tracking
Understanding Your Research Goals
Study Types

8. HOW TO CHOOSE THE
RIGHT METHOD(S)
8

9. How to Choose the Right Method(s)
 Make sure your research goals are clear
 Know what you want to learn
 Know what you will do with the information you gather
 Know what decisions the business needs to make
 Know your constraints
 Timeline?
 Resources?
 Budget?
 Access to Users?
 Create a Methods Chart
 The answers will become clear
9
Know Goals and Constraints

10. How to Choose the Right Method(s)
10
Method Pros Cons Candidate
Method 1 • List the advantages for
this method for this
specific project
• List the disadvantages
for this method for this
specific project
Yes / No –
Would this method be a
good candidate for this
specific project
Method 2
Method 3
Etc…
Methods Chart

11. How to Choose the Right Method(s)
 Example Project Goals:
 Have 3 concepts for behavior change application. Which one is the best to
develop for maximum global appeal?
 Client wants answers as quickly as possible
 Client wants large numbers to provide confidence behind decision
 Client wants data collection in 5 countries around the world
 Designers want to know why or why not users chose each concept, to
provide additional design direction
 User population – adults who own a smart phone and want to get healthier
11
Methods Chart

12. Methods Chart Example
12
Method Pros Cons Candidate
Interviews • Good for capturing
motivations for behavior
change
• Can get good qualitative
details on why users
prefer each concept or
not
• Good to explore issues to
fine tune survey
questions
• Can be done via phone
(get broad geographic
sample)
• Can be done quickly
• Small numbers – client
wants large numbers
Y – would be good as a
qualitative method to pair
with a larger quantitative
method
Focus Groups • Good for qualitative
information gathering
• Could generate some
interesting conversations
about behavior change
motivations
• Wouldn’t get as much
detail as interviews
• Concern about group
think when evaluating
concepts
• Concern about not
sharing details of
personal goals in front of
others
N – interviews would be
better for qualitative
Survey • Good for large numbers
• Easily replicated across
different
countries/languages
• Can be done online for
broad geographic
distribution
• Can be done quickly
• Unclear what exact
questions to ask
• Doesn’t provide detailed
insights into qualitative
topics
Y – good paired with
qualitative method

13. How to Choose the Right Method(s)
 Example Research Plan:
 Conduct telephone interviews in US with 12-15 participants
 Create online survey in US for 300 participants
 Use international research partners to conduct 12-15 interviews in 4
countries
 Use international research partners to conduct survey for 300 participants
in same 4 countries
 Why this approach works:
 Interviews provide input into questions for survey
 Survey provides large numbers; interviews provide in-depth insight
 Both are commonly used methods and easy to replicate in different
countries
 Both methods can be done quickly
13
Methods Chart

14. GATHERING QUALITATIVE DATA
14

15. Gathering Qualitative Data
15
Usability Task Creation
 Good usability study tasks are:
 Non-leading: don’t give away the answer
 Single tasks: don’t have users do two things in succession
 Realistic: don’t have users do things they would not normally do
 Self-explanatory: don’t have multiple sentences explaining the task
 Achievable: always document what constitutes a “pass” for the task

16. Gathering Qualitative Data
16
Moderator’s Guide
 A good study guide:
 Conveys the study goals and methodology to clients
 Serves as a quick reference to the moderator during the study
 Provides the moderator with a template to take hand-written notes, with ample space to
do so
 The typical moderator’s guide contains:
 Research method: a short paragraph explaining the usability study
 Study goals: a short bulleted list of the study goals
 Introduction: the moderator’s opening spiel when explaining the study to the participant
 Background questions: Typically demographic or product usage questions
 Tasks: The task, pass condition(s), notes for the moderator, and post-task questions
 Follow-up questions: The questions to be asked after all the tasks are complete

17. 17

18. Gathering Qualitative Data
18
Notes Grid
Proper planning for taking notes is very underrated
 The organization and thoroughness of your notes will dictate the ease with which you
will create the final report
 Organized, complete notes = easy reporting
 Disorganized, incomplete notes = back to the video you go (ewww!)
 Make your notes grid once your study & moderator’s guides are complete
 Give each question and subquestion its own row
 Put each participant in a new column (or vice-versa, if you like)
 DO NOT put each participant/task in a new worksheet
 Use data validation for quantitative data
 Task ease ratings, task success, multiple choice questions, etc.
 Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to
the question)

19. 19
• Do not include the participant
name in your notes grid
• Include a date/time cell to
best align with the videos
• Visually separate sections of
the study
• Fill data validated cells with a
light color
• Hide columns when you have
moved on to the next
participant
Gathering Qualitative Data
Notes Grid

20. 20
Gathering Qualitative Data
Findings Sheet
 Scan the notes grid to determine findings
 Put these in a spreadsheet and assign and priority and category

21. In Conclusion
 Set actionable and attainable research goals
 Know where you are in the project
 Use a methods chart to weigh factors and choose a method
 Time, participants, goals, budget, etc.
 Proper documentation = easier qualitative analysis
 Study guide  notes grid  findings spreadsheet  report
21