Dan Berlin, an experience research director, emphasizes the importance of understanding research goals in user experience studies. The presentation covers methods for conducting formative and evaluative research, the significance of clear research objectives, and includes a methods chart for selecting appropriate research methodologies. Effective preparation and documentation are key for successful qualitative data analysis and actionable insights.

1. September 19, 2013
Dan Berlin
Experience Research Director
dberlin@madpow.net
@banderlin
Mad*Pow
EXPERIENCE RESEARCH BEST PRACTICES

2. Hi! I’m Dan Berlin @banderlin
 BA in psychology from Brandeis U.
 Studied visual space perception
 Seven years in technical support
 Sat as a participant for a usability study for a product I was working on
 Realized that user experience (UX) work is the perfect combination of computers and psychology
 Went to Bentley U. to earn an MBA and MS in Human Factors in Information Design
 Two years at an interactive agency performing usability and neuromarketing research
 Then did some freelance UX consulting for about a year
 Am now an Experience Research Director at Mad*Pow, an experience design agency based out of
Portsmouth, NH
 My passion is for research methodology and finding new ways to elicit data from participants
2

3. About this Presentation
 Understanding your research goals
 Formative and evaluative studies
 How to choose the right method(s)
 Methods chart
 Gathering qualitative data
 Taking notes and organizing findings
3

4. UNDERSTANDING YOUR
RESEARCH GOALS
4

5. Understanding Your Research Goals
5
Preparing to prepare for your study
Technology
User
Needs
Business
Goals
 Your overall goal:
To uncover actionable business and
design insights via user data
 Start by understanding the business’ goals
 What is the overall goal of the interface?
 What are the interactions that drive the business?
 What are the important calls to action?
 Who are the target audiences?
 What do they want to know about their users?
 What do they think they already know about their users
 How did they learn this?

6. Understanding Your Research Goals
6
Preparing to prepare for your study
 Actionable = the data should indicate what exactly needs to change or be implemented in
the interface to align it with users’ expectations and needs
 This means you need to start with actionable (and attainable) goals, such as:
 How do perform a certain interaction today?
 Does the proposed IA align with user expectations?
 Are users able to complete a critical transaction?
 Do users understand the pricing schedule?
 Goals will depend on the type of study
 Formative
 Performed at the beginning of a project to learn how the project should proceed
 Evaluative
 Performed during and towards the end of projects to determine if the design aligns with user expectations

7. Formative
 In-Person Usability
 Interviews
 Focus Groups
 Collaging
 Ethnography
 Surveys
 Diary Studies
 Card-Sorting
7
Evaluative
 In-Person Usability
 Remote Usability
 Unmoderated Usability
 Desirability Testing
 Eye-Tracking
Understanding Your Research Goals
Study Types

8. HOW TO CHOOSE THE
RIGHT METHOD(S)
8

9. How to Choose the Right Method(s)
 Make sure your research goals are clear
 Know what you want to learn
 Know what you will do with the information you gather
 Know what decisions the business needs to make
 Know your constraints
 Timeline?
 Resources?
 Budget?
 Access to Users?
 Create a Methods Chart
 The answers will become clear
9
Know Goals and Constraints

10. How to Choose the Right Method(s)
10
Method Pros Cons Candidate
Method 1 • List the advantages for
this method for this
specific project
• List the disadvantages
for this method for this
specific project
Yes / No –
Would this method be a
good candidate for this
specific project
Method 2
Method 3
Etc…
Methods Chart

11. How to Choose the Right Method(s)
 Example Project Goals:
 Have 3 concepts for behavior change application. Which one is the best to
develop for maximum global appeal?
 Client wants answers as quickly as possible
 Client wants large numbers to provide confidence behind decision
 Client wants data collection in 5 countries around the world
 Designers want to know why or why not users chose each concept, to
provide additional design direction
 User population – adults who own a smart phone and want to get healthier
11
Methods Chart

12. Methods Chart Example
12
Method Pros Cons Candidate
Interviews • Good for capturing
motivations for behavior
change
• Can get good qualitative
details on why users
prefer each concept or
not
• Good to explore issues to
fine tune survey
questions
• Can be done via phone
(get broad geographic
sample)
• Can be done quickly
• Small numbers – client
wants large numbers
Y – would be good as a
qualitative method to pair
with a larger quantitative
method
Focus Groups • Good for qualitative
information gathering
• Could generate some
interesting conversations
about behavior change
motivations
• Wouldn’t get as much
detail as interviews
• Concern about group
think when evaluating
concepts
• Concern about not
sharing details of
personal goals in front of
others
N – interviews would be
better for qualitative
Survey • Good for large numbers
• Easily replicated across
different
countries/languages
• Can be done online for
broad geographic
distribution
• Can be done quickly
• Unclear what exact
questions to ask
• Doesn’t provide detailed
insights into qualitative
topics
Y – good paired with
qualitative method

13. How to Choose the Right Method(s)
 Example Research Plan:
 Conduct telephone interviews in US with 12-15 participants
 Create online survey in US for 300 participants
 Use international research partners to conduct 12-15 interviews in 4
countries
 Use international research partners to conduct survey for 300 participants
in same 4 countries
 Why this approach works:
 Interviews provide input into questions for survey
 Survey provides large numbers; interviews provide in-depth insight
 Both are commonly used methods and easy to replicate in different
countries
 Both methods can be done quickly
13
Methods Chart

14. GATHERING QUALITATIVE DATA
14

15. Gathering Qualitative Data
15
Usability Task Creation
 Good usability study tasks are:
 Non-leading: don’t give away the answer
 Single tasks: don’t have users do two things in succession
 Realistic: don’t have users do things they would not normally do
 Self-explanatory: don’t have multiple sentences explaining the task
 Achievable: always document what constitutes a “pass” for the task

16. Gathering Qualitative Data
16
Moderator’s Guide
 A good study guide:
 Conveys the study goals and methodology to clients
 Serves as a quick reference to the moderator during the study
 Provides the moderator with a template to take hand-written notes, with ample space to
do so
 The typical moderator’s guide contains:
 Research method: a short paragraph explaining the usability study
 Study goals: a short bulleted list of the study goals
 Introduction: the moderator’s opening spiel when explaining the study to the participant
 Background questions: Typically demographic or product usage questions
 Tasks: The task, pass condition(s), notes for the moderator, and post-task questions
 Follow-up questions: The questions to be asked after all the tasks are complete

17. 17

18. Gathering Qualitative Data
18
Notes Grid
Proper planning for taking notes is very underrated
 The organization and thoroughness of your notes will dictate the ease with which you
will create the final report
 Organized, complete notes = easy reporting
 Disorganized, incomplete notes = back to the video you go (ewww!)
 Make your notes grid once your study & moderator’s guides are complete
 Give each question and subquestion its own row
 Put each participant in a new column (or vice-versa, if you like)
 DO NOT put each participant/task in a new worksheet
 Use data validation for quantitative data
 Task ease ratings, task success, multiple choice questions, etc.
 Always include an extra “Why?” cell for data validated cells (to capture qualitative data related to
the question)

19. 19
• Do not include the participant
name in your notes grid
• Include a date/time cell to
best align with the videos
• Visually separate sections of
the study
• Fill data validated cells with a
light color
• Hide columns when you have
moved on to the next
participant
Gathering Qualitative Data
Notes Grid

20. 20
Gathering Qualitative Data
Findings Sheet
 Scan the notes grid to determine findings
 Put these in a spreadsheet and assign and priority and category

21. In Conclusion
 Set actionable and attainable research goals
 Know where you are in the project
 Use a methods chart to weigh factors and choose a method
 Time, participants, goals, budget, etc.
 Proper documentation = easier qualitative analysis
 Study guide  notes grid  findings spreadsheet  report
21