Presentation at AECT 2016, Las Vegas NV

1. V A N E S S A D E N N E N
F L O R I D A S T A T E U N I V E R S I T Y
V D E N N E N @ F S U . E D U
@ V D E N N E N
A E C T – O C T O B E R 2 0 1 6 – L A S V E G A S , N V
Using electronic diaries for
data collection:
An online learning case study

2. Problem
Overview:
Collecting learner habit
data
— Collecting data about online
learner habits may prove
challenging
— Discussion board posts are a
performance, do not illuminate
"backstage online" activities
(Gibson, 2014)
— Analytic data can be useful, but
cannot answer all questions
— Self-report data can fill in gaps
— Common criticism: Self-report
data are unreliable

3. Proposed
solution:
Electronic diaries for
experience sampling
— Why not surveys?
¡ Inability to accurately recall earlier
events
¡ Blunt or overly general/summative
— Why diaries?
¡ Track subjective elements of the user
experience at multiple intervals, as it
occurs in the user's natural setting
(Scollon, Kim-Prieto & Diener, 2013;
Zirkel, Garcia & Murphy, 2015)

4. Research Questions
— How do students approach the task of completing
electronic diaries?
— Do students accurately represent their course
activities in self-report diaries?

5. Topical Study Context
— An examination of student log-in behaviors and
weekly activities in an online course

6. Method
— Participants (69 consented of approximately 120 in
course):
¡ Voluntary
¡ Learners in one intact online class
¡ Undergraduates
— Data:
¡ Diaries
¡ End-of-course survey
¡ LMS data
¡ Survey log data

7. Diary Procedures
— Diaries conducted twice, during Week 5 & Week 10
— Day 1: Each participant received:
¡ Unique link to diary (editable online survey)
¡ PDF of diary (offline record keeping)
— Day 7: Reminder
— Day 9: Reminder
— Day 10: Diary closed

8. Findings: Participation Trends
— Diary 1
¡ Started by 64
¡ Completed by 47
— Diary 2
¡ Started by 44
¡ Completed by 44
— Diary completion/response rate: 63%

9. Findings: Diary Use/Typicality

10. Findings: Reliability Check
— 18 (20%) diaries randomly selected for a check
against LMS data
¡ All items that could be verified were: days logged in; days
posted; times posted
¡ Data matching was 98%
¡ Discrepancies occurred in time

11. Findings: Individual Cases
— Triangulated data necessary for verification
— Some diaries that might have been discarded based on
log data were accurate
— Example 1: Diary completed/submitted on Day 2 in 3
minutes
¡ Closer look showed valid data
¡ Student completed full diary
¡ Student said he was going to ignore this class and spend the week
working on other classes
— Example 2: Diary completed/submitted on Day 1 in 6
minutes
¡ Closer look showed valid data
¡ Student indicated that she followed a weekly schedule for doing
coursework

12. Discussion
— Two main approaches to diary:
¡ Start on Day 1 or 2 and complete over time
¡ Start at end of week and complete all at once
÷ Presumably keeping records throughout week
— No discrepancies in accuracy, regardless of approach
— Student comments when asked if weeks were typical
were extremely frank, supporting the notion of
honest responses

13. Limitations
— Participant bias (non-response bias)
¡ Individuals who are more motivated, conscientious or
agreeable may be more likely to complete a diary (Scollon,
Kim-Prieto & Diener, 2003)
— Participants in this study may have been the more
reliable and active participants in the course
¡ More difficult to capture data from less active students /
students with poor study habits
¡ Those who do not participate in discussion may be similarly
unlikely to participate in a voluntary study

14. Conclusion
— University students are able and willing to provide
accurate data about course habits using electronic
diaries
— Diaries can be triangulated with log data to check
reliability
— Other data points may be needed to identify
perspectives not included among the diary
paticipants

15. T H A N K Y O U F O R A T T E N D I N G
V D E N N E N @ F S U . E D U
@ V D E N N E N
S L I D E S A V A I L A B L E A T S L I D E S H A R E . N E T /
V A N E S S A D E N N E N
Questions?
R E F E R E N C E S
G I L M O R E , D . M . ( 2 0 1 4 ) . G O F F M A N ' S F R O N T S TA G E A N D B A C K S TA G E B E H AV I O R S I N
O N L I N E E D U C AT I O N . J O U R N A L O F L E A R N I N G A N A LY T I C S , 1 ( 3 ) , 1 8 7 - 1 9 0 .
S C O L L O N , C . N . , K I M - P R I E T O , C . , & D I E N E R , E . ( 2 0 0 3 ) . E X P E R I E N C E S A M P L I N G :
P R O M I S E S A N D P I T F A L L S , S T R E N G T H S A N D W E A K N E S S E S . J O U R N A L O F
H A P P I N E S S S T U D I E S , 4 ( 1 ) , 5 - 3 4 .
Z I R K E L , S . , G A R C I A , J . A . , & M U R P H Y, M . C . ( 2 0 1 5 ) . E X P E R I E N C E - S A M P L I N G
R E S E A R C H M E T H O D S A N D T H E I R P O T E N T I A L F O R E D U C AT I O N R E S E A R C H .
E D U C AT I O N A L R E S E A R C H E R . D O I : 1 0 3 1 0 2 / 0 0 1 3 1 8 9 X 1 4 5 6 6 8 7 9 .