SlideShare a Scribd company logo
1 of 39
Download to read offline
You Want me to What?
A Practical Guide to Diary Studies
What is a Diary Study?
Definition*:
“Research method that involves providing
participants with the materials and structure
to record daily events, tasks and perceptions
around a given subject in order to gain
insight into their behaviour and needs over
time.”
*http://www.usability.gov/what-and-why/glossary/diary-study.html
Mike’s Study
Purpose: to examine an existing system to prepare for a
global redesign
Why a diary study?
•  To address the difficulty of determining the impact of
known usability issues during onboarding
•  What prevents a motivated user from forming patterns of
use?
•  Examined patterns of use, sticking points, goals and
motivations
Vicky’s Study
Purpose: Understand how different user types work with an Email Marketing System over
time.
Why a diary study:
●  Diary study enabled us to track user actions and satisfaction level OVER TIME – what
changes for the user as they become more skilled?
●  Factors we were gauging: blockers to success, satisfaction points, business goals and
motivations, which features were used most frequently
Liz’s Study
Purpose: Gain an understanding of the experience of current customers when they are
upgraded to a newer version of the product.
Why a diary study:
●  Study conditions recreate the context for transitioning users as closely as possible
●  Can gauge how quickly participants adjust to the new version
●  Participants have the freedom to report any type of issue or concern
●  Longer timeline distinguishes initial learning from persistent issues
What can you learn from a Diary Study?
It can provide a better understanding of usage patterns over time:
•  Learnability
•  Acceptance of a product or product features
•  Issues or challenges that impede usage
Generally: most any type of change in behavior or attitude
Advantages of Diary Studies
•  Authenticity of information collected:
–  User’s environment rather than a lab setting
–  Participants feel safer so they’re more likely to be open
–  Can get past initial learning
•  Benefits for participants:
–  Less intrusive
–  More flexible (e.g., scheduling)
•  Can uncover unexpected issues
Disadvantages & Challenges
•  Overall:
–  Time- and resource-intensive
–  Logistically unwieldy
–  Less control over study environment
•  Participants:
–  Recruitment, Management, & Retention
•  Data:
–  Self-reported, but use the resources available to you
–  Overwhelming!
A Practitioner’s Checklist:
❏  Designing the Study
❏  Recruiting
❏  Managing Participants
❏  Managing the Data
❏  Analyzing the Data
❏  Reporting the Results
Designing the Study
Managing Length of Study
•  Long enough to capture desired
behaviors
•  Stay meaningful for participants and
stakeholders
•  Reflect users’ natural work cadence as
much as possible
Designing the Study
Components
•  Ongoing feedback on user experience
–  Diary entries
•  Interim check-ins
–  Surveys / interviews
•  Final review touch point
–  Follow-up interviews
Designing the Study
Unstructured
•  Allows for more unexpected, possibly more realistic data
•  BUT, may not get desired data, and lots of manual labor
sorting through results
Structured
•  Less time consuming data analysis, higher probability of
getting relevant data
•  BUT, may not reflect user’s natural workflow, and a lot of
dictating may feel too formal and restrictive to users
Example for study design - Diary Questions
•  Name
•  What did you do with the tool today?
•  What did you want to accomplish?
•  Did you accomplish what you wanted?
•  What specific features did you use?
•  What did you like about your experience today?
•  What did you not like about your experience today?
•  What else would you like us to know?
•  Screen shots (optional)
Example for study design - Setup
“Is the tool meeting your
business goals?”
“What features have you
used?”
“Which features do you like
most / least?”
“How has your experience
changed since 3 weeks ago?”
“Have you tried any new
features?”
“Do you feel more skilled than
you did 3 weeks ago?”
Example for study design - Phone Interviews
•  Follow-up on diary entries and
surveys
•  Kept conversation less structured
•  Gave deeper understanding of user
goals, issues, etc.
•  Benefit of hearing user’s tone
(excitement or frustration)
A Practitioner’s Checklist:
✔ Designing the Study
❏  Recruiting
❏  Managing Participants
❏  Managing the Data
❏  Analyzing the Data
❏  Reporting the Results
Recruiting - “Who’s Coming with me?!”
Recruiting
Be realistic about how long this is likely to take
•  It will probably take a lot longer than you think to hit your number
•  Be prepared to be flexible, prioritize screener criteria if necessary
Strategies for minimizing attrition:
•  Graduated payments
–  Have payments tied to something you can control (e.g., interviews)
•  If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it
A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
❏  Managing Participants
❏  Managing the Data
❏  Analyzing the Data
❏  Reporting the Results
Participant management
Managing Participants
•  Make it as easy as possible for them
–  Online data entry
–  Include links in every communication
–  Have some structure (not completely open-ended)
•  Give participants permission to be brief and/or repeat
themselves, and let them be themselves!
–  Remember, it’s about a truer contextual perspective
•  Guard against being too invasive
–  Let participants choose the frequency of reminders
–  Make it explicit that reminders are automated
•  Monitor for compliance and to answer questions or
redirect to support
A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
❏  Managing the Data
❏  Analyzing the Data
❏  Reporting the Results
Managing the Data
There are tons of great online resources -- make use of them!
•  Online form for diary entries
–  Easy for participants to access
–  Researcher has access to data in real time to monitor
•  Online self-scheduler for interviews/checkins
•  Online form for data entry (e.g., for interviews)
•  Survey tools for questionnaires, rating questions etc.
But watch out for:
•  Privacy issues
•  Appropriateness for your participants
A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
❏  Analyzing the Data
❏  Reporting the Results
Analyzing the Data
Analyzing the Data
It’s largely a manual process:
1.  Read through the data looking for issues, patterns etc.
2.  Make a list of tags and cross reference with your data
3.  Repeat
Qualitative data analysis tools:
•  Lots of them available
•  Traditionally have been more helpful for ongoing data collection needs than
for a one-time study
•  Worth keeping an eye out
Analyzing the Data
Strategies:
•  Try to Identify categories and tags in advance as part of defining
the study
•  “Find” is your friend but watch out for:
–  misspellings
–  different word forms
–  different words for the same/similar concepts
–  different meanings for the same words
•  Have a method for collecting or flagging verbatims as you go
•  Use learnings from other touch-points to help identify tags &
categories
Analyzing the Data
Don’t wait until the end!
•  Keep the job manageable
•  Keep analysis timely
•  Allow for mid-course corrections
Include some selected-response measures:
•  Easy to analyze
•  Consistent across participants and over time
•  Provide metrics to supplement open-ended responses
Look for the surprises!
My Approach
Two spreadsheets:
1.  Raw data: diary entries, numbered chronologically
2.  “Processed” data, with columns for:
1.  Tags
2.  Entry numbers
3.  Verbatims
Same 3 Likert questions in all diary entries:
•  “easy” results
My approach: Get your participants to help you!
Two weeks before the final interview:
•  Created individual spreadsheets
•  Asked participants to review and think about:
1.  Their overall experience making the transition
2.  What they liked best
3.  What was the biggest challenge
4.  What we could have done to make the transition easier
DIY vs. DIFM
•  Emerging tools like
dscout can reduce
overhead
–  The trade-off can be
flexibility
–  Decide what you want to
accomplish with your
methodology before you
approach any tools
•  We were curious, so we’re
running a study right now
with thanks to our UXPA
organizers!
A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
✔ Analyzing the Data
❏  Reporting the Results
Reporting the Results
Changes over time
•  Learnability
–  (Qual) Frequency with which issues or problems are reported
–  (Quant) How easy/difficult was it for you to meet your goals in this session? To
what extent were you able to meet your goals in this session?
•  Long-term patterns of use (what features remain important
over time)
–  (Qual) Changes in users’ stated goals
–  (Quant) % of use of different features
•  Changes in user opinions over time
–  (Qual) % of comments classified is positive/negative/neutral
–  (Quant) How satisfied/dissatisfied are you with the product?
Reporting the Results
First Portion Second Portion Third Portion
Reporting the Results
A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
✔ Analyzing the Data
✔ Reporting the Results
Lessons Learned
Mike:
1.  Planned vs. Unplanned learnings (be prepared to dive back in)
2.  It all comes down to Codifying open responses
3.  Visualize learnings for broad adoption
Vicky:
1.  Try to start all participants at the same time
2.  Automate as much as possible
3.  Review the data as you go - don’t wait until the end
Liz:
1.  Structure can be your friend
2.  Control what you can but don’t try to control what you can’t
3.  Embrace the chaos!
Questions?
References
1.  http://www.uxbooth.com/articles/jumpstart-design-research-with-a-diary-study/
2.  http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/
3.  https://uxmag.com/articles/making-the-most-of-ethnographic-research
4.  http://www.spotless.co.uk/insights/6-things-for-ux-diary-study
5.  http://www.uxbooth.com/articles/dear-diary-hard-say-goodbye/
6.  https://blinkux.com/blog/dear-digital-diary-a-powerful-tool-in-user-research/
7.  http://uxpamagazine.org/dear-diary/
8.  http://www.slideshare.net/jaremfan/the-goodness-of-diary-studies

More Related Content

What's hot

Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%UserTesting
 
Advanced usability testing - moderating
Advanced usability testing - moderatingAdvanced usability testing - moderating
Advanced usability testing - moderatingRebecca Destello
 
Guerrilla (or Agile) Evaluation for Learning
Guerrilla (or Agile) Evaluation for LearningGuerrilla (or Agile) Evaluation for Learning
Guerrilla (or Agile) Evaluation for LearningJulie Dirksen
 
The Role of Data in Becoming a High Performing School
The Role of Data in Becoming a High Performing SchoolThe Role of Data in Becoming a High Performing School
The Role of Data in Becoming a High Performing SchoolQualtrics
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative productsShishir Choudhary
 
Interface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and TricksInterface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and TricksJulie Dirksen
 
Evolving The Impact of Usability Testing: Supporting New Roles & Business Me...
Evolving The Impact of Usability Testing:  Supporting New Roles & Business Me...Evolving The Impact of Usability Testing:  Supporting New Roles & Business Me...
Evolving The Impact of Usability Testing: Supporting New Roles & Business Me...UserZoom
 
Perfect Practices and Perils in Research Project Management
Perfect Practices and Perils in Research Project ManagementPerfect Practices and Perils in Research Project Management
Perfect Practices and Perils in Research Project ManagementAMA DocSIG
 
Conducting Usability Research with a Team of One [Revised: October 2009]
Conducting Usability Research with a Team of One [Revised: October 2009]Conducting Usability Research with a Team of One [Revised: October 2009]
Conducting Usability Research with a Team of One [Revised: October 2009]cmk22
 
Beyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceBeyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceAmanda Nance
 
How To Ask The Right Questions
How To Ask The Right QuestionsHow To Ask The Right Questions
How To Ask The Right QuestionsHannah Fiechtner
 
Creating Personas – A guide, not a template
Creating Personas – A guide, not a templateCreating Personas – A guide, not a template
Creating Personas – A guide, not a templateBen Ralph
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative productsShishir Choudhary
 
Cognifide content usabilitytesting-csa2017-v0.1
Cognifide content usabilitytesting-csa2017-v0.1Cognifide content usabilitytesting-csa2017-v0.1
Cognifide content usabilitytesting-csa2017-v0.1Kate Kenyon
 
The laddering technique tina
The laddering technique tinaThe laddering technique tina
The laddering technique tinaJasmin Tabora
 
User Experience (UX) Research in Healthcare
User Experience (UX) Research in HealthcareUser Experience (UX) Research in Healthcare
User Experience (UX) Research in HealthcareDan Berlin
 
UX for Learning Design
UX for Learning DesignUX for Learning Design
UX for Learning DesignJulie Dirksen
 

What's hot (20)

Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%
 
Advanced usability testing - moderating
Advanced usability testing - moderatingAdvanced usability testing - moderating
Advanced usability testing - moderating
 
Don't Fear the User
Don't Fear the UserDon't Fear the User
Don't Fear the User
 
Guerrilla (or Agile) Evaluation for Learning
Guerrilla (or Agile) Evaluation for LearningGuerrilla (or Agile) Evaluation for Learning
Guerrilla (or Agile) Evaluation for Learning
 
The Role of Data in Becoming a High Performing School
The Role of Data in Becoming a High Performing SchoolThe Role of Data in Becoming a High Performing School
The Role of Data in Becoming a High Performing School
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative products
 
Interface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and TricksInterface Design for Elearning - Tips and Tricks
Interface Design for Elearning - Tips and Tricks
 
Evolving The Impact of Usability Testing: Supporting New Roles & Business Me...
Evolving The Impact of Usability Testing:  Supporting New Roles & Business Me...Evolving The Impact of Usability Testing:  Supporting New Roles & Business Me...
Evolving The Impact of Usability Testing: Supporting New Roles & Business Me...
 
Perfect Practices and Perils in Research Project Management
Perfect Practices and Perils in Research Project ManagementPerfect Practices and Perils in Research Project Management
Perfect Practices and Perils in Research Project Management
 
Conducting Usability Research with a Team of One [Revised: October 2009]
Conducting Usability Research with a Team of One [Revised: October 2009]Conducting Usability Research with a Team of One [Revised: October 2009]
Conducting Usability Research with a Team of One [Revised: October 2009]
 
Beyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceBeyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experience
 
How To Ask The Right Questions
How To Ask The Right QuestionsHow To Ask The Right Questions
How To Ask The Right Questions
 
Creating Personas – A guide, not a template
Creating Personas – A guide, not a templateCreating Personas – A guide, not a template
Creating Personas – A guide, not a template
 
User Testing Whirlwind Tour
User Testing Whirlwind TourUser Testing Whirlwind Tour
User Testing Whirlwind Tour
 
Building innovative products
Building innovative productsBuilding innovative products
Building innovative products
 
Cognifide content usabilitytesting-csa2017-v0.1
Cognifide content usabilitytesting-csa2017-v0.1Cognifide content usabilitytesting-csa2017-v0.1
Cognifide content usabilitytesting-csa2017-v0.1
 
The laddering technique tina
The laddering technique tinaThe laddering technique tina
The laddering technique tina
 
User Experience (UX) Research in Healthcare
User Experience (UX) Research in HealthcareUser Experience (UX) Research in Healthcare
User Experience (UX) Research in Healthcare
 
Research project mgt
Research project mgtResearch project mgt
Research project mgt
 
UX for Learning Design
UX for Learning DesignUX for Learning Design
UX for Learning Design
 

Viewers also liked

Правда о ваших будущих клиентах
Правда о ваших будущих клиентахПравда о ваших будущих клиентах
Правда о ваших будущих клиентахWake_up_province
 
Увлеченные делом или как мы строили процессы. Александр Кунташов
Увлеченные делом или как мы строили процессы. Александр КунташовУвлеченные делом или как мы строили процессы. Александр Кунташов
Увлеченные делом или как мы строили процессы. Александр КунташовWake_up_province
 
Estoc del MQN Intercanviat
Estoc del MQN IntercanviatEstoc del MQN Intercanviat
Estoc del MQN Intercanviatmillorquenou
 
Compare & contrast essay
Compare & contrast essayCompare & contrast essay
Compare & contrast essayTung97Michelle
 
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016Ben Chou
 
Import and export payment methods
Import and export payment methodsImport and export payment methods
Import and export payment methodsVertidigm
 
BSD_Tools_AllSlides_CB6 Shorter
BSD_Tools_AllSlides_CB6 ShorterBSD_Tools_AllSlides_CB6 Shorter
BSD_Tools_AllSlides_CB6 ShorterCharlotte E Binns
 
Hunger games vs divergent
Hunger games vs divergent Hunger games vs divergent
Hunger games vs divergent rachaelcheong
 
Геймификациия оценки персонала опыт Veeam
Геймификациия оценки персонала   опыт VeeamГеймификациия оценки персонала   опыт Veeam
Геймификациия оценки персонала опыт VeeamAlexander Shirmanov
 
Les êtres vivants. david, adrián, pablo, facundo y miguel
Les êtres vivants. david, adrián, pablo, facundo y miguelLes êtres vivants. david, adrián, pablo, facundo y miguel
Les êtres vivants. david, adrián, pablo, facundo y migueljlealleon
 
Advantages of SUSE Linux Over Windows
Advantages of SUSE Linux Over WindowsAdvantages of SUSE Linux Over Windows
Advantages of SUSE Linux Over WindowsJeff Reser
 

Viewers also liked (17)

Elevator pitch
Elevator pitchElevator pitch
Elevator pitch
 
Updated JVW
Updated JVWUpdated JVW
Updated JVW
 
AragonBallroomPresentationRev
AragonBallroomPresentationRevAragonBallroomPresentationRev
AragonBallroomPresentationRev
 
Правда о ваших будущих клиентах
Правда о ваших будущих клиентахПравда о ваших будущих клиентах
Правда о ваших будущих клиентах
 
002
002002
002
 
12:04:2015 CV
12:04:2015 CV12:04:2015 CV
12:04:2015 CV
 
Увлеченные делом или как мы строили процессы. Александр Кунташов
Увлеченные делом или как мы строили процессы. Александр КунташовУвлеченные делом или как мы строили процессы. Александр Кунташов
Увлеченные делом или как мы строили процессы. Александр Кунташов
 
Estoc del MQN Intercanviat
Estoc del MQN IntercanviatEstoc del MQN Intercanviat
Estoc del MQN Intercanviat
 
Compare & contrast essay
Compare & contrast essayCompare & contrast essay
Compare & contrast essay
 
M & e indicators
M & e indicatorsM & e indicators
M & e indicators
 
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016
openQA hands on with openSUSE Leap 42.1 - openSUSE.Asia Summit ID 2016
 
Import and export payment methods
Import and export payment methodsImport and export payment methods
Import and export payment methods
 
BSD_Tools_AllSlides_CB6 Shorter
BSD_Tools_AllSlides_CB6 ShorterBSD_Tools_AllSlides_CB6 Shorter
BSD_Tools_AllSlides_CB6 Shorter
 
Hunger games vs divergent
Hunger games vs divergent Hunger games vs divergent
Hunger games vs divergent
 
Геймификациия оценки персонала опыт Veeam
Геймификациия оценки персонала   опыт VeeamГеймификациия оценки персонала   опыт Veeam
Геймификациия оценки персонала опыт Veeam
 
Les êtres vivants. david, adrián, pablo, facundo y miguel
Les êtres vivants. david, adrián, pablo, facundo y miguelLes êtres vivants. david, adrián, pablo, facundo y miguel
Les êtres vivants. david, adrián, pablo, facundo y miguel
 
Advantages of SUSE Linux Over Windows
Advantages of SUSE Linux Over WindowsAdvantages of SUSE Linux Over Windows
Advantages of SUSE Linux Over Windows
 

Similar to A Practical Guide to Diary Studies

Sad lecture 3
Sad lecture 3Sad lecture 3
Sad lecture 3Amin Omi
 
Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Anwal Mirza
 
Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Shayne Botor
 
The basics of customer experience research
The basics of customer experience researchThe basics of customer experience research
The basics of customer experience researchSmaply
 
Needs Assessment
Needs AssessmentNeeds Assessment
Needs AssessmentLeila Zaim
 
Types of data collection tools
Types of data collection tools Types of data collection tools
Types of data collection tools Vipin Kumar
 
WHAT IS RESEARCH REPORT WRITING
WHAT IS RESEARCH REPORT WRITINGWHAT IS RESEARCH REPORT WRITING
WHAT IS RESEARCH REPORT WRITINGFarah Latiff
 
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014ICPSR
 
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...Nebraska Library Commission
 
What is program evaluation lecture 100207 [compatibility mode]
What is program evaluation lecture   100207 [compatibility mode]What is program evaluation lecture   100207 [compatibility mode]
What is program evaluation lecture 100207 [compatibility mode]Jennifer Morrow
 
Analytic emperical Mehods
Analytic emperical MehodsAnalytic emperical Mehods
Analytic emperical MehodsM Surendar
 
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12Barbara O'Neill
 
Research Tools and Techniques
Research Tools and TechniquesResearch Tools and Techniques
Research Tools and TechniquesRonit Kharade
 
Seda emdedding learning technologies evaluating and sustainability3
Seda emdedding learning technologies   evaluating and sustainability3Seda emdedding learning technologies   evaluating and sustainability3
Seda emdedding learning technologies evaluating and sustainability3BrianKilpatrick
 
User Research Tools: Don't just do it, do it well
User Research Tools: Don't just do it, do it wellUser Research Tools: Don't just do it, do it well
User Research Tools: Don't just do it, do it wellZabisco Digital
 
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)Using Surveys to Improve Your Library: Part 1 (Sept. 2018)
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)ALATechSource
 
Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Lynn Connaway
 

Similar to A Practical Guide to Diary Studies (20)

Chap 7
Chap 7Chap 7
Chap 7
 
Sad lecture 3
Sad lecture 3Sad lecture 3
Sad lecture 3
 
Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14Hci evaluationa frame work lec 14
Hci evaluationa frame work lec 14
 
Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016
 
The basics of customer experience research
The basics of customer experience researchThe basics of customer experience research
The basics of customer experience research
 
Needs Assessment
Needs AssessmentNeeds Assessment
Needs Assessment
 
classmar2.ppt
classmar2.pptclassmar2.ppt
classmar2.ppt
 
Types of data collection tools
Types of data collection tools Types of data collection tools
Types of data collection tools
 
WHAT IS RESEARCH REPORT WRITING
WHAT IS RESEARCH REPORT WRITINGWHAT IS RESEARCH REPORT WRITING
WHAT IS RESEARCH REPORT WRITING
 
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014
Instructional Data Sets from Q-step Launch Event (Univ of Exeter) 3-20-2014
 
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
 
What is program evaluation lecture 100207 [compatibility mode]
What is program evaluation lecture   100207 [compatibility mode]What is program evaluation lecture   100207 [compatibility mode]
What is program evaluation lecture 100207 [compatibility mode]
 
Analytic emperical Mehods
Analytic emperical MehodsAnalytic emperical Mehods
Analytic emperical Mehods
 
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12
NEAFCS 2012 Financial Ed eval-assessment-impact-o'neill-09-12
 
Research Tools and Techniques
Research Tools and TechniquesResearch Tools and Techniques
Research Tools and Techniques
 
Seda emdedding learning technologies evaluating and sustainability3
Seda emdedding learning technologies   evaluating and sustainability3Seda emdedding learning technologies   evaluating and sustainability3
Seda emdedding learning technologies evaluating and sustainability3
 
User Research Tools: Don't just do it, do it well
User Research Tools: Don't just do it, do it wellUser Research Tools: Don't just do it, do it well
User Research Tools: Don't just do it, do it well
 
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)Using Surveys to Improve Your Library: Part 1 (Sept. 2018)
Using Surveys to Improve Your Library: Part 1 (Sept. 2018)
 
Research Week 1.pptx
Research Week 1.pptxResearch Week 1.pptx
Research Week 1.pptx
 
Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.
 

Recently uploaded

Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Valters Lauzums
 
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08LloydHelferty
 
Basic overview of nerve conduction studies
Basic overview of nerve conduction studiesBasic overview of nerve conduction studies
Basic overview of nerve conduction studiesDrAbdulAli1
 
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptxerickamwana1
 
Scootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City DeliveryScootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City Deliveryrishi338139
 
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...Sebastiano Panichella
 
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxPrinciples of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxvirginiagaddafi
 
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...soumyapottola
 

Recently uploaded (8)

Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
Personal Branding Lecture for Advanced Digital & Social Media Strategy at UCL...
 
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
Sunlight Spectacle 2024 Practical Action Launch Event 2024-04-08
 
Basic overview of nerve conduction studies
Basic overview of nerve conduction studiesBasic overview of nerve conduction studies
Basic overview of nerve conduction studies
 
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
05.02 MMC - Assignment 4 - Image Attribution Lovepreet.pptx
 
Scootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City DeliveryScootsy Overview Deck - Pan City Delivery
Scootsy Overview Deck - Pan City Delivery
 
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...Testing with Fewer Resources:  Toward Adaptive Approaches for Cost-effective ...
Testing with Fewer Resources: Toward Adaptive Approaches for Cost-effective ...
 
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptxPrinciples of Management Touchstone 4 Template APPLE INC.ppt.pptx
Principles of Management Touchstone 4 Template APPLE INC.ppt.pptx
 
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
Understanding Post Production changes (PPC) in Clinical Data Management (CDM)...
 

A Practical Guide to Diary Studies

  • 1. You Want me to What? A Practical Guide to Diary Studies
  • 2. What is a Diary Study? Definition*: “Research method that involves providing participants with the materials and structure to record daily events, tasks and perceptions around a given subject in order to gain insight into their behaviour and needs over time.” *http://www.usability.gov/what-and-why/glossary/diary-study.html
  • 3. Mike’s Study Purpose: to examine an existing system to prepare for a global redesign Why a diary study? •  To address the difficulty of determining the impact of known usability issues during onboarding •  What prevents a motivated user from forming patterns of use? •  Examined patterns of use, sticking points, goals and motivations
  • 4. Vicky’s Study Purpose: Understand how different user types work with an Email Marketing System over time. Why a diary study: ●  Diary study enabled us to track user actions and satisfaction level OVER TIME – what changes for the user as they become more skilled? ●  Factors we were gauging: blockers to success, satisfaction points, business goals and motivations, which features were used most frequently
  • 5. Liz’s Study Purpose: Gain an understanding of the experience of current customers when they are upgraded to a newer version of the product. Why a diary study: ●  Study conditions recreate the context for transitioning users as closely as possible ●  Can gauge how quickly participants adjust to the new version ●  Participants have the freedom to report any type of issue or concern ●  Longer timeline distinguishes initial learning from persistent issues
  • 6. What can you learn from a Diary Study? It can provide a better understanding of usage patterns over time: •  Learnability •  Acceptance of a product or product features •  Issues or challenges that impede usage Generally: most any type of change in behavior or attitude
  • 7. Advantages of Diary Studies •  Authenticity of information collected: –  User’s environment rather than a lab setting –  Participants feel safer so they’re more likely to be open –  Can get past initial learning •  Benefits for participants: –  Less intrusive –  More flexible (e.g., scheduling) •  Can uncover unexpected issues
  • 8. Disadvantages & Challenges •  Overall: –  Time- and resource-intensive –  Logistically unwieldy –  Less control over study environment •  Participants: –  Recruitment, Management, & Retention •  Data: –  Self-reported, but use the resources available to you –  Overwhelming!
  • 9. A Practitioner’s Checklist: ❏  Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  • 10. Designing the Study Managing Length of Study •  Long enough to capture desired behaviors •  Stay meaningful for participants and stakeholders •  Reflect users’ natural work cadence as much as possible
  • 11. Designing the Study Components •  Ongoing feedback on user experience –  Diary entries •  Interim check-ins –  Surveys / interviews •  Final review touch point –  Follow-up interviews
  • 12. Designing the Study Unstructured •  Allows for more unexpected, possibly more realistic data •  BUT, may not get desired data, and lots of manual labor sorting through results Structured •  Less time consuming data analysis, higher probability of getting relevant data •  BUT, may not reflect user’s natural workflow, and a lot of dictating may feel too formal and restrictive to users
  • 13. Example for study design - Diary Questions •  Name •  What did you do with the tool today? •  What did you want to accomplish? •  Did you accomplish what you wanted? •  What specific features did you use? •  What did you like about your experience today? •  What did you not like about your experience today? •  What else would you like us to know? •  Screen shots (optional)
  • 14. Example for study design - Setup “Is the tool meeting your business goals?” “What features have you used?” “Which features do you like most / least?” “How has your experience changed since 3 weeks ago?” “Have you tried any new features?” “Do you feel more skilled than you did 3 weeks ago?”
  • 15. Example for study design - Phone Interviews •  Follow-up on diary entries and surveys •  Kept conversation less structured •  Gave deeper understanding of user goals, issues, etc. •  Benefit of hearing user’s tone (excitement or frustration)
  • 16. A Practitioner’s Checklist: ✔ Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  • 17. Recruiting - “Who’s Coming with me?!”
  • 18. Recruiting Be realistic about how long this is likely to take •  It will probably take a lot longer than you think to hit your number •  Be prepared to be flexible, prioritize screener criteria if necessary Strategies for minimizing attrition: •  Graduated payments –  Have payments tied to something you can control (e.g., interviews) •  If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it
  • 19. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  • 21. Managing Participants •  Make it as easy as possible for them –  Online data entry –  Include links in every communication –  Have some structure (not completely open-ended) •  Give participants permission to be brief and/or repeat themselves, and let them be themselves! –  Remember, it’s about a truer contextual perspective •  Guard against being too invasive –  Let participants choose the frequency of reminders –  Make it explicit that reminders are automated •  Monitor for compliance and to answer questions or redirect to support
  • 22. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  • 23. Managing the Data There are tons of great online resources -- make use of them! •  Online form for diary entries –  Easy for participants to access –  Researcher has access to data in real time to monitor •  Online self-scheduler for interviews/checkins •  Online form for data entry (e.g., for interviews) •  Survey tools for questionnaires, rating questions etc. But watch out for: •  Privacy issues •  Appropriateness for your participants
  • 24. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  • 26. Analyzing the Data It’s largely a manual process: 1.  Read through the data looking for issues, patterns etc. 2.  Make a list of tags and cross reference with your data 3.  Repeat Qualitative data analysis tools: •  Lots of them available •  Traditionally have been more helpful for ongoing data collection needs than for a one-time study •  Worth keeping an eye out
  • 27. Analyzing the Data Strategies: •  Try to Identify categories and tags in advance as part of defining the study •  “Find” is your friend but watch out for: –  misspellings –  different word forms –  different words for the same/similar concepts –  different meanings for the same words •  Have a method for collecting or flagging verbatims as you go •  Use learnings from other touch-points to help identify tags & categories
  • 28. Analyzing the Data Don’t wait until the end! •  Keep the job manageable •  Keep analysis timely •  Allow for mid-course corrections Include some selected-response measures: •  Easy to analyze •  Consistent across participants and over time •  Provide metrics to supplement open-ended responses Look for the surprises!
  • 29. My Approach Two spreadsheets: 1.  Raw data: diary entries, numbered chronologically 2.  “Processed” data, with columns for: 1.  Tags 2.  Entry numbers 3.  Verbatims Same 3 Likert questions in all diary entries: •  “easy” results
  • 30. My approach: Get your participants to help you! Two weeks before the final interview: •  Created individual spreadsheets •  Asked participants to review and think about: 1.  Their overall experience making the transition 2.  What they liked best 3.  What was the biggest challenge 4.  What we could have done to make the transition easier
  • 31. DIY vs. DIFM •  Emerging tools like dscout can reduce overhead –  The trade-off can be flexibility –  Decide what you want to accomplish with your methodology before you approach any tools •  We were curious, so we’re running a study right now with thanks to our UXPA organizers!
  • 32. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ❏  Reporting the Results
  • 33. Reporting the Results Changes over time •  Learnability –  (Qual) Frequency with which issues or problems are reported –  (Quant) How easy/difficult was it for you to meet your goals in this session? To what extent were you able to meet your goals in this session? •  Long-term patterns of use (what features remain important over time) –  (Qual) Changes in users’ stated goals –  (Quant) % of use of different features •  Changes in user opinions over time –  (Qual) % of comments classified is positive/negative/neutral –  (Quant) How satisfied/dissatisfied are you with the product?
  • 34. Reporting the Results First Portion Second Portion Third Portion
  • 36. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ✔ Reporting the Results
  • 37. Lessons Learned Mike: 1.  Planned vs. Unplanned learnings (be prepared to dive back in) 2.  It all comes down to Codifying open responses 3.  Visualize learnings for broad adoption Vicky: 1.  Try to start all participants at the same time 2.  Automate as much as possible 3.  Review the data as you go - don’t wait until the end Liz: 1.  Structure can be your friend 2.  Control what you can but don’t try to control what you can’t 3.  Embrace the chaos!
  • 39. References 1.  http://www.uxbooth.com/articles/jumpstart-design-research-with-a-diary-study/ 2.  http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/ 3.  https://uxmag.com/articles/making-the-most-of-ethnographic-research 4.  http://www.spotless.co.uk/insights/6-things-for-ux-diary-study 5.  http://www.uxbooth.com/articles/dear-diary-hard-say-goodbye/ 6.  https://blinkux.com/blog/dear-digital-diary-a-powerful-tool-in-user-research/ 7.  http://uxpamagazine.org/dear-diary/ 8.  http://www.slideshare.net/jaremfan/the-goodness-of-diary-studies