Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

You want me to what a practical guide to diary studies

709 views

Published on

Liz Burton, Vicky Morville, Michael Kennedy
Presented at UXPA Boston 2016

You want me to what a practical guide to diary studies

  1. 1. You Want me to What? A Practical Guide to Diary Studies
  2. 2. What is a Diary Study? Definition*: “Research method that involves providing participants with the materials and structure to record daily events, tasks and perceptions around a given subject in order to gain insight into their behaviour and needs over time.” *http://www.usability.gov/what-and-why/glossary/diary-study.html
  3. 3. Mike’s Study Purpose: to examine an existing system to prepare for a global redesign Why a diary study? •  To address the difficulty of determining the impact of known usability issues during onboarding •  What prevents a motivated user from forming patterns of use? •  Examined patterns of use, sticking points, goals and motivations
  4. 4. Vicky’s Study Purpose: Understand how different user types work with an Email Marketing System over time. Why a diary study: ●  Diary study enabled us to track user actions and satisfaction level OVER TIME – what changes for the user as they become more skilled? ●  Factors we were gauging: blockers to success, satisfaction points, business goals and motivations, which features were used most frequently
  5. 5. Liz’s Study Purpose: Gain an understanding of the experience of current customers when they are upgraded to a newer version of the product. Why a diary study: ●  Study conditions recreate the context for transitioning users as closely as possible ●  Can gauge how quickly participants adjust to the new version ●  Participants have the freedom to report any type of issue or concern ●  Longer timeline distinguishes initial learning from persistent issues
  6. 6. What can you learn from a Diary Study? It can provide a better understanding of usage patterns over time: •  Learnability •  Acceptance of a product or product features •  Issues or challenges that impede usage Generally: most any type of change in behavior or attitude
  7. 7. Advantages of Diary Studies •  Authenticity of information collected: –  User’s environment rather than a lab setting –  Participants feel safer so they’re more likely to be open –  Can get past initial learning •  Benefits for participants: –  Less intrusive –  More flexible (e.g., scheduling) •  Can uncover unexpected issues
  8. 8. Disadvantages & Challenges •  Overall: –  Time- and resource-intensive –  Logistically unwieldy –  Less control over study environment •  Participants: –  Recruitment, Management, & Retention •  Data: –  Self-reported, but use the resources available to you –  Overwhelming!
  9. 9. A Practitioner’s Checklist: ❏  Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  10. 10. Designing the Study Managing Length of Study •  Long enough to capture desired behaviors •  Stay meaningful for participants and stakeholders •  Reflect users’ natural work cadence as much as possible
  11. 11. Designing the Study Components •  Ongoing feedback on user experience –  Diary entries •  Interim check-ins –  Surveys / interviews •  Final review touch point –  Follow-up interviews
  12. 12. Designing the Study Unstructured •  Allows for more unexpected, possibly more realistic data •  BUT, may not get desired data, and lots of manual labor sorting through results Structured •  Less time consuming data analysis, higher probability of getting relevant data •  BUT, may not reflect user’s natural workflow, and a lot of dictating may feel too formal and restrictive to users
  13. 13. Example for study design - Diary Questions •  Name •  What did you do with the tool today? •  What did you want to accomplish? •  Did you accomplish what you wanted? •  What specific features did you use? •  What did you like about your experience today? •  What did you not like about your experience today? •  What else would you like us to know? •  Screen shots (optional)
  14. 14. Example for study design - Setup “Is the tool meeting your business goals?” “What features have you used?” “Which features do you like most / least?” “How has your experience changed since 3 weeks ago?” “Have you tried any new features?” “Do you feel more skilled than you did 3 weeks ago?”
  15. 15. Example for study design - Phone Interviews •  Follow-up on diary entries and surveys •  Kept conversation less structured •  Gave deeper understanding of user goals, issues, etc. •  Benefit of hearing user’s tone (excitement or frustration)
  16. 16. A Practitioner’s Checklist: ✔ Designing the Study ❏  Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  17. 17. Recruiting - “Who’s Coming with me?!”
  18. 18. Recruiting Be realistic about how long this is likely to take •  It will probably take a lot longer than you think to hit your number •  Be prepared to be flexible, prioritize screener criteria if necessary Strategies for minimizing attrition: •  Graduated payments –  Have payments tied to something you can control (e.g., interviews) •  If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it
  19. 19. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ❏  Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  20. 20. Participant management
  21. 21. Managing Participants •  Make it as easy as possible for them –  Online data entry –  Include links in every communication –  Have some structure (not completely open-ended) •  Give participants permission to be brief and/or repeat themselves, and let them be themselves! –  Remember, it’s about a truer contextual perspective •  Guard against being too invasive –  Let participants choose the frequency of reminders –  Make it explicit that reminders are automated •  Monitor for compliance and to answer questions or redirect to support
  22. 22. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ❏  Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  23. 23. Managing the Data There are tons of great online resources -- make use of them! •  Online form for diary entries –  Easy for participants to access –  Researcher has access to data in real time to monitor •  Online self-scheduler for interviews/checkins •  Online form for data entry (e.g., for interviews) •  Survey tools for questionnaires, rating questions etc. But watch out for: •  Privacy issues •  Appropriateness for your participants
  24. 24. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ❏  Analyzing the Data ❏  Reporting the Results
  25. 25. Analyzing the Data
  26. 26. Analyzing the Data It’s largely a manual process: 1.  Read through the data looking for issues, patterns etc. 2.  Make a list of tags and cross reference with your data 3.  Repeat Qualitative data analysis tools: •  Lots of them available •  Traditionally have been more helpful for ongoing data collection needs than for a one-time study •  Worth keeping an eye out
  27. 27. Analyzing the Data Strategies: •  Try to Identify categories and tags in advance as part of defining the study •  “Find” is your friend but watch out for: –  misspellings –  different word forms –  different words for the same/similar concepts –  different meanings for the same words •  Have a method for collecting or flagging verbatims as you go •  Use learnings from other touch-points to help identify tags & categories
  28. 28. Analyzing the Data Don’t wait until the end! •  Keep the job manageable •  Keep analysis timely •  Allow for mid-course corrections Include some selected-response measures: •  Easy to analyze •  Consistent across participants and over time •  Provide metrics to supplement open-ended responses Look for the surprises!
  29. 29. My Approach Two spreadsheets: 1.  Raw data: diary entries, numbered chronologically 2.  “Processed” data, with columns for: 1.  Tags 2.  Entry numbers 3.  Verbatims Same 3 Likert questions in all diary entries: •  “easy” results
  30. 30. My approach: Get your participants to help you! Two weeks before the final interview: •  Created individual spreadsheets •  Asked participants to review and think about: 1.  Their overall experience making the transition 2.  What they liked best 3.  What was the biggest challenge 4.  What we could have done to make the transition easier
  31. 31. DIY vs. DIFM •  Emerging tools like dscout can reduce overhead –  The trade-off can be flexibility –  Decide what you want to accomplish with your methodology before you approach any tools •  We were curious, so we’re running a study right now with thanks to our UXPA organizers!
  32. 32. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ❏  Reporting the Results
  33. 33. Reporting the Results Changes over time •  Learnability –  (Qual) Frequency with which issues or problems are reported –  (Quant) How easy/difficult was it for you to meet your goals in this session? To what extent were you able to meet your goals in this session? •  Long-term patterns of use (what features remain important over time) –  (Qual) Changes in users’ stated goals –  (Quant) % of use of different features •  Changes in user opinions over time –  (Qual) % of comments classified is positive/negative/neutral –  (Quant) How satisfied/dissatisfied are you with the product?
  34. 34. Reporting the Results First Portion Second Portion Third Portion
  35. 35. Reporting the Results
  36. 36. A Practitioner’s Checklist: ✔ Designing the Study ✔ Recruiting ✔ Managing Participants ✔ Managing the Data ✔ Analyzing the Data ✔ Reporting the Results
  37. 37. Lessons Learned Mike: 1.  Planned vs. Unplanned learnings (be prepared to dive back in) 2.  It all comes down to Codifying open responses 3.  Visualize learnings for broad adoption Vicky: 1.  Try to start all participants at the same time 2.  Automate as much as possible 3.  Review the data as you go - don’t wait until the end Liz: 1.  Structure can be your friend 2.  Control what you can but don’t try to control what you can’t 3.  Embrace the chaos!
  38. 38. Questions?
  39. 39. References 1.  http://www.uxbooth.com/articles/jumpstart-design-research-with-a-diary-study/ 2.  http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/ 3.  https://uxmag.com/articles/making-the-most-of-ethnographic-research 4.  http://www.spotless.co.uk/insights/6-things-for-ux-diary-study 5.  http://www.uxbooth.com/articles/dear-diary-hard-say-goodbye/ 6.  https://blinkux.com/blog/dear-digital-diary-a-powerful-tool-in-user-research/ 7.  http://uxpamagazine.org/dear-diary/ 8.  http://www.slideshare.net/jaremfan/the-goodness-of-diary-studies

×