Your SlideShare is downloading. ×
Tips for conducting long-term diary studies - UPA 2009
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Tips for conducting long-term diary studies - UPA 2009


Published on

Published in: Technology, Education

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Pros/cons. I started by recruiting myself: I thought I could identify appropriate customers more accurately than a screener. Also thought customers more likely to participate if contacted directly by Sage. Recruitment continued... Was taking time away from analyzing the data, so I hired a recruiter. Recruiters can also handle some of the admin work. (give examples from bullets) All of this frees up your time for analysis.
  • When deadline passes, let potential participants know they cannot complete the study.
  • Full template is in proceedings. Tell participants how to use the document. This includes: 1. Explain what the diary is 2. How to complete it – Be clear about what you expect. Story here: participant who didn’t use product one week so didn’t fill in diary 3. Remind when and where to send it 4. Contact info – in case they have questions.
  • Makes it easier for participant to fill out Easier to analyze later Can paste it into Excel and rotate it to look across participants
  • With appropriate level of detail Full example in the proceedings
  • This is a quote from one of the diaries. I had no idea what it meant, but she included screenshots. She felt there was a discrepancy between 2 screens showing the same information, but looking at the screenshots, I realized the label on one of them was very confusing.
  • This study: only 1 person included screenshots, probably because template said, “include if it helps you explain the problem”
  • Engages users Their expectations about the product – found out that people wanted a certain feature through these Follow up: by email or phone depending on type of issue Provide support after you understand problem.
  • Because you’re following up w/ participants, it’s important to read entries quickly. It allows you to get clarification while incidents are fresh on their minds. Our product is complex. I didn’t always understand how the system works. Had to understand that to interpret some of the entries.
  • Captured infrequent tasks Use of Help documented, reasons people call support Saw how users resolve issues on their own and how long it takes Gave a sense for overall getting started process
  • “Participants who provide especially helpful information (determined at our discretion) will be given a bonus of $140 for a total of $300.” But didn’t want to affect how often people used the software. I continued 2 participants for 8 weeks
  • Transcript

    • 1. Conducting Longitudinal Diary Studies Amanda Nance Sage June 10, 2009
    • 2. Early diary study
    • 3. Goal of this presentation• How to run a better diary study – Create effective template – Understand the data – Keep users engaged• See benefits of the method
    • 4. Topics for this presentation• Reason for the project• About the diary study• Diary study tips• Unique contributions of the diary method
    • 5. Reasons for the project• Assess “getting started” experience of our desktop accounting software• Focus on tasks after initial set up• Create requirements for next release
    • 6. Used multiple methods to get truer pictureSite visits Support calls Survey analysis Researcher calls Diary study Diary from:
    • 7. About the diary study• 5 participants: small business owners, bookkeepers• Diaries kept for up to 8 weeks beginning at install• Diaries submitted weekly by email• Compensation – at least $40/week
    • 8. About the diary study• Provided template in Word to record – Daily Work: tools they used, other tasks – Issues and Comments: problems with our product
    • 9. Tips
    • 10. Recruit extra participants• Recruited 12• 5 completed• Reasons for drop outs were interesting findings, too
    • 11. Use a recruiter to save time• Pros and cons• They can recruit and more – Email documents – Remind of deadlines – Collect diaries
    • 12. Communicate deadlines• Communicate due date for first submission• Disqualify participants who don’t respond• Close the loop
    • 13. Template tips
    • 14. Make it easy for participants• Keep it short – This study: 3 questions daily, 6-7 per incident• Separate questions from answers
    • 15. Separate questions from answers
    • 16. Show sample entriesWhat did you do for your business today?• I answered several customer phone calls.• Called automated bank system to make sure wehave enough money for payroll.• Called in payroll to ADP (we outsource payroll).• Ordered parts for one of our jobs by phone.
    • 17. Ask for screenshots“Looking at Aged Receivables…. They donot match. Why are they different?”
    • 18. Tips for screenshots• If they are important, require them• Include instructions
    • 19. Structure template for easy analysis• Separate daily entries from critical incidents• Include participant name and week number in footer and file name
    • 20. Iterate on the template• Prior to launching• As you receive user feedback
    • 21. Process tips
    • 22. Require daily entries• Keeps participants engaged• Gives you a reason to pay them• Opportunity to learn about the broader work context
    • 23. Talk to the participants yourself• Intro call to understand business and product expectations• Follow up on unclear entries• Provide support
    • 24. Research tips• Read entries as soon as possible• Leave time to research
    • 25. Data analysis• No different from other methods• Evaluate data – Excel to categorize theme, second theme, week #• Share and act on data – Summarize key findings, provide recommendation for each – Use customer quotes – Share complete diaries
    • 26. Unique contributions of diary study• Infrequent tasks• Help use• Users resolving issues• Tasks over time
    • 27. Reasons to run a diary study• No travel required• Non-intrusive, in context• Cheaper than site visits
    • 28. Engagement and meaningful data• Pay for more complete responses• Encourage useful responses through feedback on recent entries• Continue participants who are useful
    • 29. Key Points• Daily entries• Explain what you expect• Sample entries• Reasonable number of questions• Personal contact with researcher• Questions?
    • 30. ContactMore tips in conference proceedingsanance@gmail.comTwitter: @amandaux