Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Effective Digital Diary Studies / UXPA Webinar May 2016

Diary studies are one of the few research methods that allows people to report their in-the-moment experiences, but their usefulness has been limited by the difficulty in getting people to remember to carry them around and fill them out. Mobile phones have breathed new life into diary studies since people are already using them in short spurts throughout their day. However designing a study that collects relevant data can be tricky.

  • Login to see the comments

Effective Digital Diary Studies / UXPA Webinar May 2016

  1. 1. Effective Digital Diary Studies Sara Cambridge UXPA Webinar: May 18, 2016
  2. 2. What do diary studies offer that interviews don’t?
  3. 3. 3 Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget.
  4. 4. Fitness Tracker Case Study 40 people who use fitness trackers were recruited to participate in several digital diary studies over 8 months. The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. 44
  5. 5. 5 Erica, 33 “It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.” Show us how you use your fitness tracker
  6. 6. Digital diary studies excel at: • Gathering contextual information cost-effectively • Capture repeated behaviors in context • Providing instant feedback on products • Uncovering latent needs in general problem space
  7. 7. What This Presentation Will Cover What a Diary Study is Advantages of Diary Studies Platforms for Conducting Digital Diary Studies How to Conduct a Digital Diary Study Highlights for Ensuring an Effective Digital Diary Study 7
  8. 8. What is a Diary Study?
  9. 9. A Diary Study is a Cultural Probe Cultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….
 With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problems and uncover new opportunities. 9 Adam Starkman & Jennifer Chow
  10. 10. Key Elements of Diary Studies Theme-based: the subject and questions are suggested by the researcher. Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided. In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible. Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time. 10
  11. 11. Traditional Diary Study Mozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox. They asked 10 people to record their experiences with bookmarks over 3 days. 11 Brian Groudan, Mozilla research, 2012 / ux/page/24/
  12. 12. Traditional Diary Study People came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme. Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools. 12
  13. 13. Why Use 
 Diary Studies?
  14. 14. What’s so great about ‘in-the-moment’ feedback? Can’t you get the same information 
 in an interview a few days later? 14
  15. 15. 15 Most see it like a video camera…. How Does Memory Work? but really it’s like a slow leak
  16. 16. The Psychology of Habits 16The Power of Habit, Duhigg Behavior trigger Sequence of actions occurs repeatedly Reward / Finish Our brains turn the sequence into one automatic routine; “chunked”
  17. 17. The Psychology of Habits 17The Power of Habit, Duhigg We can’t access the details of habitual actions until we start them Diary studies ask people to report when details are still accessible This also allows for collecting workarounds people are not aware of Behavior trigger Details of actions become accessible Reward / Finish Habit sequence executed
  18. 18. What is a Digital Diary Study?
  19. 19. 19 Getting people to actually do the reporting is a major obstacle for traditional diary studies.
  20. 20. 20 Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day. Average use: 85 times a day* *Study of 23 people aged 18-33:
  21. 21. Tools for Executing Digital Diary Studies Platform (web/app) SMS/Text Email Blog Video 21
  22. 22. Digital Diary Study Platforms 22 Name Platform Cost 20/20 Research Web/app diary study high 24Tru Web/app diary study high CrowdTap Web/app diary study high UX 360 Web diary study high Dscout App diary study medium Dedoose Web diary study low Typeform online survey low
  23. 23. 23 Diary Study Platform Advantages: • Offers analysis; faster to insights • Lots of data collection options • Many have panels for recruiting • Guidance, support, education • Team members can participate • Many participants = quant data Disadvantages: • Cost • New tool • For more tech-savvy participants • Can’t adjust once launched $595 for recruiting $295 for each study
  24. 24. 24 Most diary studies are a mix of multiple choice and open ended questions.
  25. 25. Example Dscout Diary Study:
 New Product Launch Goal: Get feedback on a new digital entertainment product Details: 5 families, 4 months, in-person interviews throughout Platform: Dscout app, asking them to report every time they used the service, answering a series of questions. Key Finding: The service was most robust for mobile viewing, but the families did the bulk of their viewing on TVs. Surprising Finding: The study uncovered significant technical errors that were unknown to our client. 25
  26. 26. 26 SMS/Text Advantages: • Free • Good for less tech savvy participants • Can adjust on the fly • More likely to get immediate response Disadvantages: • All analysis is manual • Limited data input
  27. 27. Example SMS Diary Study: Tracking Diabetes Patients Goal: Learn how often Type 2 Diabetes patients actually check their glucose level, as well as how often they have their glucose monitor with them Details: 6 participants, 4 weeks, in-person interviews at the beginning and the end Daily Text: How are you feeling today? Have you checked your blood glucose today? Where is your meter now? Key Finding: People check their glucose & have meter less often than was assumed. Several aspects were identified that dictate how often people check their glucose Study Insight: Asking how they were feeling made them feel cared for, rather than reminded/judged 27
  28. 28. 28 Email Advantages: • Free • Good for less tech savvy participants • Can adjust on the fly • Likely to get more details than if using SMS Disadvantages: • All analysis is manual • Emails are easy to overlook
  29. 29. Example Email Diary Study:
 New Product Launch Goal: Understand how and why people use different channels when managing their investment account Details: 18 participants, 2 weeks, in-person interviews at the beginning and the end Daily Emails: Questions to understand any activity taken regarding investment accounts (activity taken, channel used, how felt about it, etc), plus occasional tasks for probing deeper on specific aspects Surprising insight: People sometimes conflate how often they think about their money with how often they check their accounts Biggest Challenge: Not having enough time to fully analyze data 29
  30. 30. 30 diaries-a-method-for-understanding-new-usage- patterns.php Video Advantages: • Can collect rich content • Edited video = rich storytelling • Most people send cameras back Disadvantages: • All analysis is manual • Transcribing video • Cost of multiple cameras • Ease of use is critical
  31. 31. 31 studies-to-understand-the-customer-experience/ Blog Advantages: • Free • Can adjust on the fly • Likely to get more input than SMS Disadvantages: • All analysis is manual • Researcher must set up accounts ahead of time
  32. 32. How to Conduct a Digital Diary Study
  33. 33. The 3 Phases of a Diary Studies 33 The goal: a concise, focused study with the right participants. The goal: high participation and high quality data. The goal: to turn data into relevant insights.
  34. 34. • Clarify your research question • Refine and pretest your study • Minimize the work for your participants • Recruit the right participants
  35. 35. Clarify Your Goals & Research Question Before Building the Study Define the goals for the study. What do you want to learn? Turn that into a research question that will guide your efforts as you create your diary study. 36
  36. 36. Fitness Tracker Study Research Question: How are activity trackers used by people attempting to make concrete improvements to their fitness habits? 37
  37. 37. What Type of Diary Study Will 
 You Build? Ideation: Collect ideas/suggestions. 
 “Show me your ideal…” Inventory: Understand key things participants use, own or do. 
 “Show me all of your...” Moments: Capture experiences as they happen in real time. 
 “Show me every time you…” Process: Understand key steps of a process in real time. 
 “Show me your process for…” Reflection: Explore likes, dislikes, memories and motivators. 
 “Tell me how you feel about….” 38
  38. 38. 39 Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue. Help Your Participants Succeed By Reducing the Reporting Burden
  39. 39. 40 Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”.
  40. 40. 41 Another benefit of multiple-choice is you can quickly see trends as the data comes in.
  41. 41. 42 Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly. I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating. I do wish that other activities could be tracked with the Fitbit without having to input it myself. Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me.
  42. 42. 43 Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing.
  43. 43. 45 Diary studies generate a lot of data. It is good to start small. 10 x 10 x 10 =questions participants entries data points 1000
  44. 44. 46 If recruiting through Dscout, you’ll only pay incentives for participants that finish. Expect to pay $5-$20 per entry. We paid $25 for a total of 6 entries. If you’re working with a specialized or busy population, you may need to pay as much as $20/entry.
  45. 45. 47 We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal. Find the Right Participants
  46. 46. • Review & make notes from your data as it’s coming in • Be persistent with participants • Ask them questions for clarity • Do post-study interviews
  47. 47. Watch the data as it comes in. This is the fun part. Are your questions holding up? 
 If not, can you fix it? 
 Either way, learn for next time. 50
  48. 48. 51 Why did you get your first fitness tracker? Larry, 31 “I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.”
  49. 49. 52 Take notes of interesting patterns and questions to ask participants as you go through it.
  50. 50. 54 Minimize your drop off rate. Stay in touch with your participants. Be friendly and persistent.
  51. 51. 55 Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say.
  52. 52. 56 Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their reporting and actual behaviors.
  53. 53. • Use the built-in analysis tools • Start by answering your initial question(s) • Identify patterns and/or exceptions • Share your findings with real examples
  54. 54. 58 The good news is that by simply reviewing and making notes on the data, you have already started on the analysis.
  55. 55. 59 Diary Study platforms offer tools to analyze the multiple choice and rating questions, reducing the work the researcher has to do.
  56. 56. 60 If analyzing manually, data should be entered into a spreadsheet so it can be coded, refined and sorted.
  57. 57. What Further Analysis Should be Done? That depends on many variables, such as: • What your initial question/s were • How surprised you are by the data that has come in • How it fits into other research methods • Your final deliverables 61
  58. 58. Digital Diary Study 
 Key Takeaways
  59. 59. Main Benefits of Using Diary Studies • To collect rich stories about how and why people do things • Get intimate glimpses of how a product fits into people’s lives • Allows for understanding evolving use of a product • Uncover latent needs for new product development • Cost-effective way to collect contextual insights 63
  60. 60. Checklist for a Great Diary Studies Phase 1, Prepare: • Have a clear, focused research question to guide your efforts • Have an appropriate mix of closed/open-ended questions • Pre-testing it to ensure clarity and understanding • Start small & recruit the right participants • Reduce the burden on them Phase 2, Execute: • Watch the data and follow up with participants during the study • Do follow-up interviews with the best participants Phase 3, Analyze: • Answer your initial question first • Give yourself enough time to analyze the data 64
  61. 61. Tips for Building Your Own Diary Study Before you can use it, you may have to sell it. Here are a few things to keep in mind: • It’s an extremely flexible tool; make it work for your needs • Start small, let it prove it’s value to your organization and yourself (no-cost platform, 5-6 participants) • Evangelize the insights and stories you collect to build more support. • You become an expert by doing. So dive in! 65
  62. 62. For a 4-min video about diary studies: Questions about diary studies? Ask! @ maybanks