Pros/cons. I started by recruiting myself: I thought I could identify appropriate customers more accurately than a screener. Also thought customers more likely to participate if contacted directly by Sage. Recruitment continued... Was taking time away from analyzing the data, so I hired a recruiter. Recruiters can also handle some of the admin work. (give examples from bullets) All of this frees up your time for analysis.
When deadline passes, let potential participants know they cannot complete the study.
Full template is in proceedings. Tell participants how to use the document. This includes: 1. Explain what the diary is 2. How to complete it – Be clear about what you expect. Story here: participant who didn’t use product one week so didn’t fill in diary 3. Remind when and where to send it 4. Contact info – in case they have questions.
Makes it easier for participant to fill out Easier to analyze later Can paste it into Excel and rotate it to look across participants
With appropriate level of detail Full example in the proceedings
This is a quote from one of the diaries. I had no idea what it meant, but she included screenshots. She felt there was a discrepancy between 2 screens showing the same information, but looking at the screenshots, I realized the label on one of them was very confusing.
This study: only 1 person included screenshots, probably because template said, “include if it helps you explain the problem”
Engages users Their expectations about the product – found out that people wanted a certain feature through these Follow up: by email or phone depending on type of issue Provide support after you understand problem.
Because you’re following up w/ participants, it’s important to read entries quickly. It allows you to get clarification while incidents are fresh on their minds. Our product is complex. I didn’t always understand how the system works. Had to understand that to interpret some of the entries.
Captured infrequent tasks Use of Help documented, reasons people call support Saw how users resolve issues on their own and how long it takes Gave a sense for overall getting started process
“Participants who provide especially helpful information (determined at our discretion) will be given a bonus of $140 for a total of $300.” But didn’t want to affect how often people used the software. I continued 2 participants for 8 weeks
Tips for conducting long-term diary studies - UPA 2009
Show sample entriesWhat did you do for your business today?• I answered several customer phone calls.• Called automated bank system to make sure wehave enough money for payroll.• Called in payroll to ADP (we outsource payroll).• Ordered parts for one of our jobs by phone.
Ask for screenshots“Looking at Aged Receivables…. They donot match. Why are they different?”
Tips for screenshots• If they are important, require them• Include instructions
Structure template for easy analysis• Separate daily entries from critical incidents• Include participant name and week number in footer and file name
Iterate on the template• Prior to launching• As you receive user feedback
Require daily entries• Keeps participants engaged• Gives you a reason to pay them• Opportunity to learn about the broader work context
Talk to the participants yourself• Intro call to understand business and product expectations• Follow up on unclear entries• Provide support
Research tips• Read entries as soon as possible• Leave time to research
Data analysis• No different from other methods• Evaluate data – Excel to categorize theme, second theme, week #• Share and act on data – Summarize key findings, provide recommendation for each – Use customer quotes – Share complete diaries
Unique contributions of diary study• Infrequent tasks• Help use• Users resolving issues• Tasks over time
Reasons to run a diary study• No travel required• Non-intrusive, in context• Cheaper than site visits
Engagement and meaningful data• Pay for more complete responses• Encourage useful responses through feedback on recent entries• Continue participants who are useful
Key Points• Daily entries• Explain what you expect• Sample entries• Reasonable number of questions• Personal contact with researcher• Questions?
ContactMore tips in conference firstname.lastname@example.orgTwitter: @amandaux