Tips for conducting long-term diary studies - UPA 2009


Published on

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Pros/cons. I started by recruiting myself: I thought I could identify appropriate customers more accurately than a screener. Also thought customers more likely to participate if contacted directly by Sage. Recruitment continued... Was taking time away from analyzing the data, so I hired a recruiter. Recruiters can also handle some of the admin work. (give examples from bullets) All of this frees up your time for analysis.
  • When deadline passes, let potential participants know they cannot complete the study.
  • Full template is in proceedings. Tell participants how to use the document. This includes: 1. Explain what the diary is 2. How to complete it – Be clear about what you expect. Story here: participant who didn’t use product one week so didn’t fill in diary 3. Remind when and where to send it 4. Contact info – in case they have questions.
  • Makes it easier for participant to fill out Easier to analyze later Can paste it into Excel and rotate it to look across participants
  • With appropriate level of detail Full example in the proceedings
  • This is a quote from one of the diaries. I had no idea what it meant, but she included screenshots. She felt there was a discrepancy between 2 screens showing the same information, but looking at the screenshots, I realized the label on one of them was very confusing.
  • This study: only 1 person included screenshots, probably because template said, “include if it helps you explain the problem”
  • Engages users Their expectations about the product – found out that people wanted a certain feature through these Follow up: by email or phone depending on type of issue Provide support after you understand problem.
  • Because you’re following up w/ participants, it’s important to read entries quickly. It allows you to get clarification while incidents are fresh on their minds. Our product is complex. I didn’t always understand how the system works. Had to understand that to interpret some of the entries.
  • Captured infrequent tasks Use of Help documented, reasons people call support Saw how users resolve issues on their own and how long it takes Gave a sense for overall getting started process
  • “Participants who provide especially helpful information (determined at our discretion) will be given a bonus of $140 for a total of $300.” But didn’t want to affect how often people used the software. I continued 2 participants for 8 weeks
  • Tips for conducting long-term diary studies - UPA 2009

    1. 1. Conducting Longitudinal Diary Studies Amanda Nance Sage June 10, 2009
    2. 2. Early diary study
    3. 3. Goal of this presentation• How to run a better diary study – Create effective template – Understand the data – Keep users engaged• See benefits of the method
    4. 4. Topics for this presentation• Reason for the project• About the diary study• Diary study tips• Unique contributions of the diary method
    5. 5. Reasons for the project• Assess “getting started” experience of our desktop accounting software• Focus on tasks after initial set up• Create requirements for next release
    6. 6. Used multiple methods to get truer pictureSite visits Support calls Survey analysis Researcher calls Diary study Diary from:
    7. 7. About the diary study• 5 participants: small business owners, bookkeepers• Diaries kept for up to 8 weeks beginning at install• Diaries submitted weekly by email• Compensation – at least $40/week
    8. 8. About the diary study• Provided template in Word to record – Daily Work: tools they used, other tasks – Issues and Comments: problems with our product
    9. 9. Tips
    10. 10. Recruit extra participants• Recruited 12• 5 completed• Reasons for drop outs were interesting findings, too
    11. 11. Use a recruiter to save time• Pros and cons• They can recruit and more – Email documents – Remind of deadlines – Collect diaries
    12. 12. Communicate deadlines• Communicate due date for first submission• Disqualify participants who don’t respond• Close the loop
    13. 13. Template tips
    14. 14. Make it easy for participants• Keep it short – This study: 3 questions daily, 6-7 per incident• Separate questions from answers
    15. 15. Separate questions from answers
    16. 16. Show sample entriesWhat did you do for your business today?• I answered several customer phone calls.• Called automated bank system to make sure wehave enough money for payroll.• Called in payroll to ADP (we outsource payroll).• Ordered parts for one of our jobs by phone.
    17. 17. Ask for screenshots“Looking at Aged Receivables…. They donot match. Why are they different?”
    18. 18. Tips for screenshots• If they are important, require them• Include instructions
    19. 19. Structure template for easy analysis• Separate daily entries from critical incidents• Include participant name and week number in footer and file name
    20. 20. Iterate on the template• Prior to launching• As you receive user feedback
    21. 21. Process tips
    22. 22. Require daily entries• Keeps participants engaged• Gives you a reason to pay them• Opportunity to learn about the broader work context
    23. 23. Talk to the participants yourself• Intro call to understand business and product expectations• Follow up on unclear entries• Provide support
    24. 24. Research tips• Read entries as soon as possible• Leave time to research
    25. 25. Data analysis• No different from other methods• Evaluate data – Excel to categorize theme, second theme, week #• Share and act on data – Summarize key findings, provide recommendation for each – Use customer quotes – Share complete diaries
    26. 26. Unique contributions of diary study• Infrequent tasks• Help use• Users resolving issues• Tasks over time
    27. 27. Reasons to run a diary study• No travel required• Non-intrusive, in context• Cheaper than site visits
    28. 28. Engagement and meaningful data• Pay for more complete responses• Encourage useful responses through feedback on recent entries• Continue participants who are useful
    29. 29. Key Points• Daily entries• Explain what you expect• Sample entries• Reasonable number of questions• Personal contact with researcher• Questions?
    30. 30. ContactMore tips in conference proceedingsanance@gmail.comTwitter: @amandaux
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.