Investigating Self-Reporting Behavior in Long-Term Studies

333 views

Published on

Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects’ forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in or- der to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the sub- jects’ reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly over- estimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and bur- densome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources.
With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data col- lected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommenda- tions for maximizing the reliability of results when conduct- ing large-scale, long-term app usage studies.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
333
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Investigating Self-Reporting Behavior in Long-Term Studies

  1. 1. INVESTIGATINGSELF-REPORTING BEHAVIORIN LONG-TERM STUDIESAndreas Möller ✽, Matthias Kranz ❖,Barbara Schmid ✽, Stefan Diewald ✽, Luis Roalter ✽✽ Technische Universität München, Germany❖ Universität Passau, Germany
  2. 2. LoggingSelf-ReportingDATA COLLECTION
  3. 3. ForgettingAnnoyanceSluggishnessSelf-Perception
  4. 4. RESEARCH QUESTIONSAccuracy? Change over time?Influence ofreporting frequency?Reliability maximization?A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  5. 5. BACKGROUND■ Electronic diaries show highercompliance(Hufford & Shields, 2002)■ Mobile phone as survey tool(Consolvo et al., 2007)Consolvo et al., 2007A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  6. 6. CONTRIBUTIONS■ Empirical quantitative long-term dataon reliability of informationcollected with mobile devices■ Recommendations for maximizingresult reliabilityA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  7. 7. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, ParisMETHODOLOGYEvaluatereportingbehaviorGround truthCan be gainedin automatedwayLimited effortSmartphoneusageGOALREQUIRE-MENTSSOLUTIONFrequently used appsInstalled by everyone
  8. 8. Mail FacebookFrequently used appsInstalled by everyoneA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  9. 9. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, ParisPROCEEDINGWeek 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 6 + 4Pre-Questionnaire Reminder Emails Post-QuestionnairePost-Post-QuestionnaireRequested Self-Reports& Logging
  10. 10. TASK■ Answer questionnaire after Facebook or Mailusage■ Report as accurate as possible1. How long did you use the app?2. How many times did you use the appwithout answering a questionnaire?1 direct reportn indirect reportsA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  11. 11. SELF-REPORTING AND EXPERIENCESAMPLING ASSISTANTA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  12. 12. SERENA■ App usage logging■ Server upload■ Questionnaire triggers□ Event-based□ Time-based□ ManuallyA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  13. 13. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris3 CONDITIONSVoluntary Interval EventNo trigger Daily triggerTrigger afterapp usage30 Participants3,631 Mail usages3,181 Facebook usages
  14. 14. SESSIONSVoluntary Interval EventAmount of reported Facebook usages37.6%63.8%54.3%Indirect reportsDirect reportsA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  15. 15. DURATIONSVoluntaryIntervalEvent1:291:291:222:523:023:35Facebook sessionsRealSelf-ReportedA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  16. 16. OVER TIME■ Self-report ratio decreases■ Actual usage decreases„Answering the questionnairechanged my Facebook usage habits.“Voluntary 2.3Interval 2.2Event 3.55 = strongly agreeA. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, Paris
  17. 17. A. Möller, M. Kranz et al., Investigating Self-Reporting Behavior in Long-Term Studies, CHI 2013, ParisDISCUSSIONSubjectsreported max.70% of usageCommitmentdecreasedReports mayinfluence actualbehaviorSubjectsoverestimatedsession lengthReminder emailspushedcommitment in2nd phaseBehavior changewith increasingburdenLittle control:forgettingHigh control:burdenTrigger influencelower thanhypothesizedApp usagedecreasedMost subjectswould reportmax. 4 weeks
  18. 18. andreas.moeller@tum.dehttps://vmi.lmt.ei.tum.de/serena
  19. 19. Please cite this work as follows:A. Möller, M. Kranz, B. Schmid, L. Roalter, S. DiewaldInvestigating Self-Reporting Behavior In Long-Term StudiesIn: Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems (CHI 2013), pp. 2931-2940, Paris, France, April-May 2013.If you use BibTex, please use the following entry:@inproceedings{chi2013selfreport,author = {Andreas M"{o}ller and Matthias Kranz and Barbara Schmid and LuisRoalter and Stefan Diewald},title = {Investigating Self-Reporting Behavior In Long-Term Studies},booktitle = {Proceedings of the 2013 ACM annual conference on Human Factors inComputing Systems},pages = {2931--2940},series = {CHI 13},year = {2013},isbn = {978-1-4503-1899-0},location = {Paris, France},numpages = {10},publisher = {ACM},address = {New York, NY, USA},}

×