Diary Studies in HCI & Psychology

Diary Studies in HCI & Psychology: Why They’re Useful and How to Conduct Them Demetrios Karis demetrios.karis@gmail.com UPA Boston’s Tenth Annual Mini UPA Conference May 25, 2011 0

What is a Diary Study? In a diary study, participants go about theirnormal lives except that they report what they’vedone or experienced (via questionnaire or freeformusing an electronic or paper diary) at some interval,signal, or event, and they do this multiple times overdays or weeks. • © 2011 Karis • Diary Studies • 1

I’ll Go Over • How people report • When they report • What they report• Example of what to report – Study on information needs while mobile: ―What time does the post office close?‖ – Questions: 1. Where were you? 2. What were you doing? 3. What was your information need? 4. I addressed the need (At the time, Later, Not at all) 5. If you attempted to address the need, how did you do so? If you didn’t make an attempt, why didn’t you? 6. Could you have addressed your need by looking at your personal data (e.g., email, calendar, web browsing history, chat history, or other) • © 2011 Karis • Diary Studies • 2

Outline• Why diary studies?• History• Example 1: Understanding the mobile phone music experience• Methodology – Three types of diary designs – Paper and pencil versus electronic diaries – The role of media, and the ―snippet‖ technique – Practical considerations: number of participants, duration, data collection, and analysis• Example 2: Information needs while mobile• Example 3: Electronic chronic pain diary• Example 4: A diary study of work-related reading• Related methods – Experience sampling method (ESM) – Day-in-the-life observation• References • © 2011 Karis • Diary Studies • 3

Why Diary Studies?• Diary studies (in HCI): – Provide information on how people are using a device or application in natural everyday situations – and provide information on how this use unfolds over time • When, how, and where they use it • Problems, similar products they’re using, unmet needs – Provide information on user needs in order to provide requirements & design guidance for new devices or applications – Eliminate biases introduced by retrospection – Often more accurate than retrospective techniques• Limitations – Participants’ commitment and dedication required – Training sessions often required • © 2011 Karis • Diary Studies • 4

History • Paper and pencil diary studies began appearing in the psychological literature 70 years ago in the early 1940s • In psychology, used to study – Personality processes – Marital and family interaction – Physical symptoms – Mental health • Example: An investigation of women’s perception of intimacy in their romantic relationships in everyday life, by assessing – The degree of intimacy the typical woman in a committed relationship feels on average – The extent to which the typical woman’s feelings of intimacy vary over time – Whether women differ from one another in their average feelings of intimacy and in the variability of their feelings of intimacy over timeThis example, and much of this presentation, comes from Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life asit is Lived. Annual Review of Psychology, 2003, Volume 54, 579-616.For a review of diary studies in HCI and CSCW prior to 2002, see Palen, L., & Salzman, M. Voice-Mail Diary Studies forNaturalistic Data Capture under Mobile Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACMPress, 87 – 95. • © 2011 Karis • Diary Studies • 5

Example 1: A Diary Study to Help Understand the Cellphone Music ExperienceGoals Understand how using a phone to listen to music fits into a person’s ―musical ecology‖ -- i.e., how do people use their phone and other music devices in their daily life? – More Specific Goals: • Understand the user experience on a particular phone – with respect to music – and whether it’s an improvement over the last version • Recommend changes to improve future music phones• Five Studies 1. Expert review 2. Usability study 3. Diary Study 4. Interviews with iPod users 5. User group study of Rhapsody and online music • © 2011 Karis • Diary Studies • 6

Methodology• Week 1: use your phone as you normally would – Email sent every day asking if they used their phone, and how• Week 2: a task given each day (e.g., rewind song a few seconds) – We receive information on usability and utility, and also whether they had performed the task before• Interviews at the beginning and end of the study Kate Dobroth ran this study. • © 2011 Karis • Diary Studies • 7

Methodology: More Details• Nineteen participants between 18-44 (most between 18 and 24) who used their phone for listening to music (recruited from a list provided by a wireless carrier)• Initial interview – Impressions of the phone, music consumption, and experiences with loading music onto the phone• Daily emails about how phone was used – Did you use your phone today? What did you use it for? – Did you use any of the music features? If so, which ones?• Second week of emails included music-related tasks – Allowed us to explore whether participants had discovered various features, and whether they were useful• Final wrap-up interview – Explore issues and questions from emails • © 2011 Karis • Diary Studies • 8

Lots of Questions• Where are you when you listen to music?• How do you listen to music? (devices)• Why do you listen using each type of device?• How many songs do you have on your phone?• Getting music onto the phone – Where do people get music? – How do they get it onto the phone?• Usage – Which phone features are used most often? – How often do people use the music feature?• Task performance – Performance and impressions of specific tasks completed during the second week• Overall impressions of the phone – Like most and least • © 2011 Karis • Diary Studies • 9

Questions to Consider• Are these results representative of all people who might like to listen to music on their phones?• Would you make design decisions or change requirements based on these results?• What other studies might be worth doing?• Could there be memory issues with a diary study like this? That is, a person may tell you what they did at the end of the day, many hours after they actually did it.• How can diary and quantitative techniques be combined?• Other questions? • © 2011 Karis • Diary Studies • 10

Log Analysis Plus a Diary Study Adding quantitative data from log analyses helped tremendously to answer basic questions.• In the days before Smartphone apps, users could download a version of SuperPages onto their feature phones (a BREW application)• Several usability studies — valuable, but little information on utility• Combining a diary study and log analysis helped to answer a number of questions: – What prevented people from using the service, or finding what they wanted?D – Where were they when they used the service, and what was the context?Both – What are the opportunities for improvement, to make the service fit in with how people actually use the application? – When did people use the service? – What types of searches were most popular?L – How many searches did people conduct in each session? – Which location options were used most often? – Which businesses did people look for? • © 2011 Karis • Diary Studies • 11

Three Types of Diary Designs• Participants can be asked to report on their actions or experiences in three ways 1. At specific intervals • Once a day, or specific times of day (after breakfast, after school, before bed) or specific time intervals (every three waking hours) 2. At a signal from the researcher • Participants prompted to provide diary reports at fixed, random, or a combination of fixed and random intervals 3. When an event occurs • Participants provide reports whenever an event occurs that meets some pre-established criteria• How often should participants report? – Need to consider the temporal patterns of the phenomenon being studied; what’s the expected frequency of the events of interest? – Also, need to consider the type of event or phenomena • Pain: need to report immediate experience, not past • Major events can be reported daily with much less error or bias (did you take mass transit today?) • © 2011 Karis • Diary Studies • 12

Paper and Pencil Diaries• Easy to implement, but several problems – Participants may fail to remember the scheduled response times – Or may fail to have diaries at hand – Potential for retrospection errors: participants rely on reconstruction to complete missed entries – For researcher, the burden of data entry and handling• Suggestions – Make diaries easily portable – Reduce possibility of participant errors by preprinting the dates and times of expected responses onto diary sheets – Reduce demand characteristics by emphasizing the importance of accuracy over the number of responses – Pilot test your diary material on a representative sample of people• Augmented paper diaries – Responses still collected via paper, but these are augmented by signaling devices such as pagers, preprogrammed wristwatches, phone calls, or mobile phones (alarms, email, text messages, etc.) • © 2011 Karis • Diary Studies • 13

Advantages of Electronic Devices• Studies started appearing about 2000• Many advantages – Allow for signaling; can remind you to complete diary entry – Can date and time-stamp entries • Prevents ―back-filling‖ diaries to appear compliant – Can provide location information if device has GPS sensor – Flexibility in presentation of questions, plus the ability to force responses before proceeding – Improved data quality and management, & eliminates data entry • Data doesn’t need to be transcribed and checked • Out-of-range responses prevented • Fewer missing responses – Eliminates risk of skipped questions – Much higher compliance rates using electronic diaries compared to paper (in both children and adults) • More diary entries completed (6.89 days out of 7 for electronic diaries versus 4.97 using paper in Lewandowski et al., 2009)• Early studies: participants provided with PDAs; now, many people carry smartphones with them at all times, and these can be used • © 2011 Karis • Diary Studies • 14

Diary Studies: The Role of Media• Up until now we’ve been assuming that people are just responding to questions by writing their answers or selecting from multiple choices• Photos, video, audio recordings, & objects can help in diary studies – Participants capture events (e.g., via a photo); these can stand alone or can be used to elicit more detail during a later interview • Annotation produced at the same time can be very beneficial (but can add significantly to the effort involved) – Audio can be used in some situations when taking a picture isn’t appropriate (recording an audio memo or calling a number and responding to questions) – In some situations, it may make sense to have participants collect objects Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary Studies. Proceedings of the CHI ‘05 (Portland, OR, April, 2005), ACM Press, 899-908. • © 2011 Karis • Diary Studies • 15

The Snippet Technique• While out and mobile there may not be time to complete a thorough diary entry• The snippet technique: ―The in situ capture of snippets: bits of text, audio, or pictures captured in a matter of seconds…then, at a convenient time, participants access a website to review their snippets and complete thorough, structured diary entries.‖ – Use existing mobile device and send text (SMS), pictures (MMS), voicemail, or a combination• Preliminary results – Participants wrote more using snippet technique – Snippet technique doesn’t seem to decrease accuracy or completeness – Participants typically had a strong preference for one type of media; some used two, no one used all three • Text used most, then audio, then pictures (but some had inadequate cameras) Brandt, J., Weiss, N., & Klemmer, S. R. (2007). Txt4l8r: Lowering the Burden for Diary Studies Under Mobile Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308. • © 2011 Karis • Diary Studies • 16

Practical Considerations Exact values depend on the situation and study goals.• Number of participants – 10 to 20 is typical within CHI – But wide range: 5 to > 50 • BBC diary study on audio listening: 2,000 respondents• Duration – One or two weeks typical – A day to a month or more• Reducing drop-out rates and improving data quality – Recruit ―appropriate‖ participants, interview them if possible, explain requirements – Withhold all or most pay until the study is completed – Keep the time requirements for each diary entry short – Check to make sure responses written when requested, not later – Provide examples so participants know what’s expected – Communicate with participants during study • Personal contact is often more important than monetary incentives in motivating participants • Provide encouragement and feedback on responses (is it detailed enough?) • © 2011 Karis • Diary Studies • 17

Techniques for Collecting Data• Paper diaries – Still a good idea in some situations (e.g., seniors without computers)• Email – Have participants email you their diary entries every day – We’ve found it effective to email a reminder, which can also include questions or rating scales• Mobile phone-based applications/text messages – Participants text their diary entry, or use a smartphone application – ―Snippets‖ (SMS, MMS, voicemail) during day as aid for later recall (see Brandt et al. 2007) • Some now use Twitter as a snippet technique• Voicemail – Participant calls dedicated voicemail line rather than recording event electronically or on paper – Free-form or structured (can be IVR system) – See Palen & Salzman (2002) • © 2011 Karis • Diary Studies • 18

Techniques for Collecting Data (2)• Excel spreadsheet (e.g., worksheets for each day of the week)• Web-based techniques – Google docs or spreadsheets & Google forms (https://docs.google.com) • Free text or preset forms – Blogs (e.g., http://wordpress.org/) • One for each participant; can be password protected – Design a diary as a forum (http://www.phpbb.com/) • Used by Lichtner et al. (2009) – Online surveys • © 2011 Karis • Diary Studies • 19

So you want to run a diary study…• What are you studying, and what’s the expected frequency? – Do you need to collect information multiple times a day, or is once sufficient? – How long should you continue the study? • How many reports of experiences or device use do you need from each participant?• Is reporting event driven? – If not, is reporting at a set interval sufficient, or should you signal the participant to respond?• How will people report on their activities or experiences? – Will they be at a desk or mobile? – Will computers or smartphones be available? – Will they have web access? – Will a snippet technique be useful? (e.g., photo or text message during day, details filled in at night) • © 2011 Karis • Diary Studies • 20

Analysis• In the applied CHI world, the studies are usually qualitative, and simple descriptive statistics are typically sufficient – Exceptions, of course: e.g., Czerwinski et al. (2004) • 10 participants, analysis of task switching, difficulty, length, etc. • Multivariate logistical regression with each user’s task switch entry as input• In academic psychology, multilevel models modified to handle repeated-measures data are recommended (these are also called hierarchical linear models) – Lots of data are often collected; one study on emotion included 2,300 person-days of data • © 2011 Karis • Diary Studies • 21

Example 2: Typical Data in HCI • Keep a diary for two weeks of all your information needs when you’re mobile. Snippet technique used: Text messages sent to email address, which were then posted automatically to the web Later, six questions about each snippet answered on the web 1. Where were you? 2. What were you doing? 3. What was your information need? 4. I addressed the need (At the time, Later, Not at all) 5. If you attempted to address the need, how did you do so? If you didn’t make an attempt, why didn’t you? 6. Could you have addressed your need by looking at your personal data (e.g., email, calendar, web browsing history, chat history, or other)Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. ADiary Study of Mobile Information Needs. InProceedings of CHI ’08 (Florence, Italy, April 2008),ACM Press, 433-442. • © 2011 Karis • Diary Studies • 22

Breakdown of Information NeedsNeed Example % of Total # of # of Part.Category Diary Category Reporting Entries Entries Cat.Trivia ―What did Bob Marley die of, and 18.5% 78 17 when?‖Directions ―Directions to Sammy’s Pizza‖ 13.3% 56 17Point of Interest ―Where is the nearest library or 12.4% 52 17 bookstore?‖Friend Info ―Where are Sam and Trevor?‖ 7.6% 32 8Shopping ―How much does the Pantech phone 7.1% 30 16 cost on the AT&T website?‖Business Hours ―What time does the post office 6.9% 29 15 close?‖Personal Item ―What is my insurance coverage for 6.4% 27 12 cat scans?‖ First 7 of 16 categories from Table 1 of Sohn et al., 2008. • © 2011 Karis • Diary Studies • 23

Example 3: Electronic Chronic Pain Diary Study to record pain by adolescents with juvenile idiopathic arthritis using a Palm Tungsten W PDA (2004)Stinson et al. e-Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents withArthritis. The Clinical Journal of Pain, 2006,• Volume 22 Diary Studies • © 2011 Karis • (3). 24

Example 4: Requirements for a New DeviceIt’s 1997, and you want to gather information so you can create a device forwork-related reading (remember: Sony Reader, 2006; Kindle, 2007) – What kinds of reading do you need to support? – How do people read as part of their jobs? – What are the implications for design?• There’s not much literature on this…what should you do?• One solution: a diary study – Pilot study, to create initial taxonomy of reading activities – 15 participants, diverse occupations (pilot, surgeon, architect, optician) – 5 days – Logged daily document activities during work day • Including: books, journals, electronic documents, PDAs, post-it notes • Used paper log forms, included estimates of duration – Daily interviews with each participant (30 minutes to 3 hours) • Interviewer expanded on description: type of document, collaborative or not, where occurred, etc. • © 2011 Karis • Diary Studies • 25

Work-Related Reading: Results• Taxonomy of reading activities, 10 categories, top three: – Skimming (subcategories: sorting, proofing, browsing) – Reading to search/answer questions – Reading for cross-referencing (to integrate information)• Writing categories – Creation – Note-taking – Annotation – Form-filling – Updating (calendars/schedules)• Reading was accompanied by writing most of the time• Great deal of reading and writing using multiple documents in parallel• About half the time participants used at least two display surfaces concurrently• Shared document viewing common – Different clusters, with high sharers viewing documents with others over 50% of the time • © 2011 Karis • Diary Studies • 26

Design of Digital Reading Devices―The linear, continuous reading of single documents by people on theirown is an unrealistic characterisation of how people read in the courseof their daily work.‖• ―Reading occurs more frequently in conjunction with writing than it does in isolation….[reading devices should] support the marking or writing of documents during the reading process.‖• There is a ―need to consider how single display devices can support the range of cross-document activities people carry out.‖• Flexible search and navigation is important for some common activities.• Collaborative viewing needs to be supported. Do the Kindle and Nook meet work-related needs? Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading: Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April 1998), ACM Press, 241-248. • © 2011 Karis • Diary Studies • 27

Four Examples• Example 1: Understanding the cellphone music experience – The use of a device (mobile phone) for a specific purpose (listening to music)• Example 2: Information needs while mobile – How existing devices and strategies are used to address a need• Example 3: Electronic chronic pain diary – Recording an experience (pain) over time• Example 4: A diary study of work-related reading – Understanding the requirements for a new device • © 2011 Karis • Diary Studies • 28

Two Related Methods• Experience Sampling Method (ESM) – Used to obtain a random selection of everything a person does and experiences in everyday life – Can be considered a type of diary study• Day-in-the-Life Observation – Observation can provide a more accurate record than diary studies (and diary studies are often more accurate than surveys) – But very labor intensive • © 2011 Karis • Diary Studies • 29

Experience Sampling Method (ESM)• Used to obtain a random selection of everything a person does and experiences in everyday life – Participant signaled 8 or more times a day for 1-2 weeks; then questions/ratings presented• Used in psychology for almost 40 years to understand the dynamics of mental health, what and how often people think about things, happiness, moods, etc. ―ESM is a means for collecting information about both the context and content of the daily life of individuals…the unique advantage of ESM is its ability to capture daily life as it is directly perceived from one moment to the next, affording an opportunity to examine fluctuations in the stream of consciousness and the links between the external context and the contents of the mind.‖• An excellent method for determining 1. ―What people do all day, where, and with whom‖ 2. ―How people report experiences, different moments in their lives, along a great variety of dimensions‖ Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method: Measuring the Quality of Everyday Life. Sage Publications, Inc, 2006 • © 2011 Karis • Diary Studies • 30

Day-in-the-Life Observation• Observation can provide a more accurate record than surveys or diary studies – One study found that ―next day telephone interviews understate TV use by 62% and magazine use by 43% when compared with observation; diaries also understate these media consumption (13% and 29%, respectively)‖ (Middletown Media Study 1, Ball State University)• A Day in the Life: An Ethnographic Study of Media Consumption, Middletown Media Study II, Online Publishers Association, July 2006. (Ball State University, Indiana) – 350 adults in Muncie and Indianapolis IN were observed in Spring 2005 – Participants were observed on average nearly 13 hours – about 80% of the waking day (over 5,000 hours in total) – Every 15 seconds, their media consumption and life activities were recorded (15 media and 17 life activities tracked; e.g., eating, child care) – Observations were made at home, work, auto and other locations• Example on next page • © 2011 Karis • Diary Studies • 31

What is a Diary Study? In a diary study, participants go about their normal lives except that they report what they’ve done or experienced (via questionnaire or freeform using an electronic or paper diary) at some interval, signal, or event, and they do this multiple times over days or weeks.• Participants go about their normal lives • A qualitative technique• They report on some activity or experience – Can be combined with surveys, log – Electronic analyses or other quantitative – Paper techniques• They report according to some schedule • Media can be used – Interval – Photos, video, audio recordings – Signal • Snippets can be recorded and – Event expanded upon later• They report repeatedly over days or weeks • Practical considerations – Number, duration, reducing drop-out rates – Ten techniques for collecting data • © 2011 Karis • Diary Studies • 33

References• Adler, A., Gujar, A., Harrison, B.L., O’Hara, K., & Sellen, A. A Diary Study of Work-Related Reading: Design Implications for Digital Reading Devices. Proceedings of CHI ‘98 (Los Angeles, CA, April 1998), ACM Press, 241-248.• Bolger, N., Davis, A., & Rafaeli, E. Diary Methods: Capturing Life as it is Lived. Annual Review of Psychology, 2003, Volume 54, 579-616.• Brandt, J., Weiss, N., & Klemmer, S. R. Txt4l8r: Lowering the Burden for Diary Studies Under Mobile Conditions. Proceedings of CHI ‘07 (San Jose, CA, April 2007) ACM Press, 2303-2308.• Brown, B. A. T., Sellen, A. J., & O’Hara, K. P. A Diary Study of Information Capture in Working Life. Proceedings of CHI ‘2000 (The Hague, Amsterdam, April 2000). ACM Press, 438 – 445.• Carter, S. & Mankoff, J. When Participants Do the Capturing: The Role of Media in Diary Studies. Proceedings of CHI ‘05 (Portland, OR, April 2005), ACM Press, 899-908.• Conducting Commercial Ethnography, GreenBook White Paper: http://www.greenbook.org/marketing- research.cfm/conducting-commercial-ethnography [This paper includes much practical information].• Czerwinski, M., Horvitz, E. & White, S. A Diary Study of Task Switching and Interruptions. Proceedings of CHI ‘04 (Vienna, Austria, April 2004), ACM Press, 175-182.• Hektner, J. H., Schmidt, J. A., & Csikszentmihalyi, M. (Eds.) Experience Sampling Method: Measuring the Quality of Everyday Life. Sage Publications, Inc, 2006. • © 2011 Karis • Diary Studies • 34

References (2)• Lewandowski, A.S., Palermo, T.M., Kirchner, H.L, & Drotar, D. Comparing Diary and Retrospective Reports of Pain and Activity Restriction in Children and Adolescents with Chronic Pain Conditions. The Clinical Journal of Pain, 2009, Volume 25 (4), 299 – 306.• Lichtner, V., Kounkou, A. P., Dotan, A., Kooken, J. P., Maiden, N. A. M. An Online Forum as a User Diary for Remote Workplace Evaluation of a Work-Integrated Learning System. Proceedings of CHI ’09 (Boston, MA, April 2009), ACM Press, 2955-2969.• Palen, L., & Salzman, M. Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile Conditions. Proceedings of CSCW ‘02 (New Orleans, LA, November 2002), ACM Press, 87 – 95.• Papper, R., Holmes, M., & Popovich, M. (2004). Middletown Media Studies 1. Center for Media Design, Ball State University. Available free at http://cms.bsu.edu/Academics/CentersandInstitutes/CMD/InsightandResearch/Capabilities/ProjectGall ery/MiddletownMediaStudies/MiddletownMediaStudiesI.aspx• Papper, R., Holmes, M., Popovich, M., Bloxham, M. (2005). Middletown Media Studies 2 Report Package: Media Day and Concurrent Media Exposure. Center for Media Design, Ball State University. Available for $750 at http://www.bsu.edu/webapps2/cmdreports/product_select.asp?product_id=10• Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. A Diary Study of Mobile Information Needs. Proceedings of CHI ’08 (Florence, Italy, April 2008), ACM Press, 433-442.• Stinson, J. N., Petroz, G. C., Tait, G., Feldman, B. M., Streiner, D., McGrath, P.J., & Stevens, B.J. e- Ouch: Usability Testing of an Electronic Chronic Pain Diary for Adolescents with Arthritis. The Clinical Journal of Pain, 2006, Volume 22 (3). • © 2011 Karis • Diary Studies • 35

Thank you! Questions? Comments? Suggestions? 36

Diary Studies in HCI & Psychology

by UPABoston on Jul 13, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Diary Studies in HCI & Psychology Presentation Transcript