Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using electronic diaries for data collection: An online learning case study


Published on

Presentation at AECT 2016, Las Vegas NV

Published in: Education
  • Be the first to comment

Using electronic diaries for data collection: An online learning case study

  1. 1. V A N E S S A D E N N E N F L O R I D A S T A T E U N I V E R S I T Y V D E N N E N @ F S U . E D U @ V D E N N E N A E C T – O C T O B E R 2 0 1 6 – L A S V E G A S , N V Using electronic diaries for data collection: An online learning case study
  2. 2. Problem Overview: Collecting learner habit data —  Collecting data about online learner habits may prove challenging —  Discussion board posts are a performance, do not illuminate "backstage online" activities (Gibson, 2014) —  Analytic data can be useful, but cannot answer all questions —  Self-report data can fill in gaps —  Common criticism: Self-report data are unreliable
  3. 3. Proposed solution: Electronic diaries for experience sampling —  Why not surveys? ¡  Inability to accurately recall earlier events ¡  Blunt or overly general/summative —  Why diaries? ¡  Track subjective elements of the user experience at multiple intervals, as it occurs in the user's natural setting (Scollon, Kim-Prieto & Diener, 2013; Zirkel, Garcia & Murphy, 2015)
  4. 4. Research Questions —  How do students approach the task of completing electronic diaries? —  Do students accurately represent their course activities in self-report diaries?
  5. 5. Topical Study Context —  An examination of student log-in behaviors and weekly activities in an online course
  6. 6. Method —  Participants (69 consented of approximately 120 in course): ¡  Voluntary ¡  Learners in one intact online class ¡  Undergraduates —  Data: ¡  Diaries ¡  End-of-course survey ¡  LMS data ¡  Survey log data
  7. 7. Diary Procedures —  Diaries conducted twice, during Week 5 & Week 10 —  Day 1: Each participant received: ¡  Unique link to diary (editable online survey) ¡  PDF of diary (offline record keeping) —  Day 7: Reminder —  Day 9: Reminder —  Day 10: Diary closed
  8. 8. Findings: Participation Trends —  Diary 1 ¡  Started by 64 ¡  Completed by 47 —  Diary 2 ¡  Started by 44 ¡  Completed by 44 —  Diary completion/response rate: 63%
  9. 9. Findings: Diary Use/Typicality
  10. 10. Findings: Reliability Check —  18 (20%) diaries randomly selected for a check against LMS data ¡  All items that could be verified were: days logged in; days posted; times posted ¡  Data matching was 98% ¡  Discrepancies occurred in time
  11. 11. Findings: Individual Cases —  Triangulated data necessary for verification —  Some diaries that might have been discarded based on log data were accurate —  Example 1: Diary completed/submitted on Day 2 in 3 minutes ¡  Closer look showed valid data ¡  Student completed full diary ¡  Student said he was going to ignore this class and spend the week working on other classes —  Example 2: Diary completed/submitted on Day 1 in 6 minutes ¡  Closer look showed valid data ¡  Student indicated that she followed a weekly schedule for doing coursework
  12. 12. Discussion —  Two main approaches to diary: ¡  Start on Day 1 or 2 and complete over time ¡  Start at end of week and complete all at once ÷  Presumably keeping records throughout week —  No discrepancies in accuracy, regardless of approach —  Student comments when asked if weeks were typical were extremely frank, supporting the notion of honest responses
  13. 13. Limitations —  Participant bias (non-response bias) ¡  Individuals who are more motivated, conscientious or agreeable may be more likely to complete a diary (Scollon, Kim-Prieto & Diener, 2003) —  Participants in this study may have been the more reliable and active participants in the course ¡  More difficult to capture data from less active students / students with poor study habits ¡  Those who do not participate in discussion may be similarly unlikely to participate in a voluntary study
  14. 14. Conclusion —  University students are able and willing to provide accurate data about course habits using electronic diaries —  Diaries can be triangulated with log data to check reliability —  Other data points may be needed to identify perspectives not included among the diary paticipants
  15. 15. T H A N K Y O U F O R A T T E N D I N G V D E N N E N @ F S U . E D U @ V D E N N E N S L I D E S A V A I L A B L E A T S L I D E S H A R E . N E T / V A N E S S A D E N N E N Questions? R E F E R E N C E S G I L M O R E , D . M . ( 2 0 1 4 ) . G O F F M A N ' S F R O N T S TA G E A N D B A C K S TA G E B E H AV I O R S I N O N L I N E E D U C AT I O N . J O U R N A L O F L E A R N I N G A N A LY T I C S , 1 ( 3 ) , 1 8 7 - 1 9 0 . S C O L L O N , C . N . , K I M - P R I E T O , C . , & D I E N E R , E . ( 2 0 0 3 ) . E X P E R I E N C E S A M P L I N G : P R O M I S E S A N D P I T F A L L S , S T R E N G T H S A N D W E A K N E S S E S . J O U R N A L O F H A P P I N E S S S T U D I E S , 4 ( 1 ) , 5 - 3 4 . Z I R K E L , S . , G A R C I A , J . A . , & M U R P H Y, M . C . ( 2 0 1 5 ) . E X P E R I E N C E - S A M P L I N G R E S E A R C H M E T H O D S A N D T H E I R P O T E N T I A L F O R E D U C AT I O N R E S E A R C H . E D U C AT I O N A L R E S E A R C H E R . D O I : 1 0 3 1 0 2 / 0 0 1 3 1 8 9 X 1 4 5 6 6 8 7 9 .