SlideShare a Scribd company logo
Penetrating the Black Box of Time-on-task Estimation
http://bit.do/lak tot
Vitomir Kovanovi´c
School of Informatics,
University of Edinburgh,
Edinburgh, United Kingdom
v.kovanovic@ed.ac.uk
Dragan Gaˇsevi´c
Schools of Education and
Informatics,
University of Edinburgh,
Edinburgh, United Kingdom
dgasevic@acm.org
Shane Dawson
Learning and Teaching Unit,
University of South Australia,
Adelaide, Australia
shane.dawson@unisa.edu.au
Sre´cko Joksimovi´c
School of Interactive Arts
and Technology,
Simon Fraser University,
Burnaby, Canada
sjoksimo@sfu.ca
Ryan S. Baker
Teachers College,
Columbia University,
New York, USA
baker2@exchange.tc.columbia.edu
Marek Hatala
School of Interactive Arts
and Technology,
Simon Fraser University,
Burnaby, Canada
mhatala@sfu.ca
March 19, 2015
Marist College,
Poughkeepsie, NY, USA
Introduction
Time-on-task
“All learning, whether done in school or
elsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
Introduction
Time-on-task
“All learning, whether done in school or
elsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
Introduction
Time-on-task
However, there is a big difference between elapsed time,
and time student actually spent on learning (Carroll,
1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
Introduction
Time-on-task
However, there is a big difference between elapsed time,
and time student actually spent on learning (Carroll,
1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
Introduction
Origins of time-on-task in educational research
• Seminal work by J. Carroll: “A model of school learning” in 1963 put a
direct link between time and learning outcomes.
• Carroll (1963) differentiaded between elapsed time and time spent on
learning.
• How time is spent is what ultimately matters!
• Increase of time-on-task was one of the key principles of effective
education (Chickering and Gamson, 1989).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 3 / 28
Introduction
Challenges with time-on-task measures
• Many different operationalizations and methods of measurement
• Days in school,
• Number of lectures attended,
• Observing students’ behavior every X minutes.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 4 / 28
Introduction
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 5 / 28
Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation
• Typycally simple heuristics are used (limit action duration to X minutes)
• Problems for replications
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?
• Limit all actions to 30 minutes?
• Discard those actions completely?
• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation
• Typycally simple heuristics are used (limit action duration to X minutes)
• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Idea for the study
• Research study that involved clustering using time-on-task measures,
• Adopted one of the heuristics,
• Still, it didn’t feel totally right.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 8 / 28
Introduction Research Questions: Effects of time-on-task measuring on analytics results
Research Questions
Drawing on the Karweit and Slavin (1982) research, we conducted a study to
answer the following questions:
• What effects do different methods for estimation of time on-task-measures
from LMS data have on the results of analytical models?
• Are there differences in their statistical significance and overall conclusions
that can be drawn from them?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 9 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0s
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Methods Dataset
Dataset: Trace data
• 6 offers of 13-week research-intensive fully online masters course at Canadian
public university.
• Moodle used as LMS platform: total of 81 students and 167,261 log records.
Students Actions Messages
Winter 2008 15 33,976 212
Fall 2008 22 49,928 633
Summer 2009 10 21,059 243
Fall 2009 7 11,346 63
Winter 2010 14 31,169 359
Winter 2011 13 19,783 237
Average (SD) 13.5 (5.1) 27,877 (13,561) 291.2 (192.4)
Total 81 167,261 1747
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 11 / 28
Methods Dataset
Time-on-task measures / count measures
• Only used activities that were planned by the course design
• Viewing assignments
• Viewing resources
• Viewing discussions
• Posting to discussions
• Updating discussion messages
• Extracted both count and time-on-task measures
• Count measures used as baseline
• Time-on-task measures extracted in 15 different ways
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 12 / 28
Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Outcome measures
• Grade structure:
• TMA1 (15%): present a research paper
• TMA2 (25%): write literature review paper
• TMA3 (15%): write answers to 6 essay questions
• TMA4 (30%): work in a team on a software project
• Participation (15%): participate productively in course discussions.
• Final grade
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%
1 Triggering Event 308 17.63%
2 Exploration 684 39.17%
3 Integration 508 29.08%
4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)
model.
• CoI model: important dimensions of distance education experience:
1 Social presence: climate within course
2 Teaching presence: role of instructor before and during the course
3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747
messages using cognitive presence
coding scheme
• Excellent agreement, Cohen’s κ = 0.97
(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%
1 Triggering Event 308 17.63%
2 Exploration 684 39.17%
3 Integration 508 29.08%
4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Outcome measures
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
5 CoIHigh: number of integration and resolution messages
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 15 / 28
Methods Dataset
Measures summary
# Name Description
Count measures
1 AsignmentViewCount Number of assignment views.
2 ResourceViewCount Number of resources views.
3 DiscussionViewCount Number of course discussion views.
4 AddPostCount Number of posted messages.
5 UpdatePostCount Number of post updates.
Time-on-task measures
6 AsignmentViewTime Time spent on course assignments.
7 ResourceViewTime Time spent reading course resources.
8 DiscussionViewTime Time spent viewing course discussions.
9 AddPostTime Time spent posting discussion messages.
10 UpdatePostTime Time spent updating discussion messages.
Performance measures
11 TMA2Grade Grade for literature review paper.
12 TMA3Grade Grade for journal papers readings.
13 ParticipationGrade Grade for participation in course discussions.
14 FinalGrade Final grade in the course.
15 CoIHigh Integration and resolution message count.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 16 / 28
Methods Experimental procedure
Time-on-task estimation strategies
# Name Description
Group 1: No outliers processing, different processing of last actions
1 x:x No outliers and last action processing.
2 x:ev No outliers processing, estimation of last action duration.
3 x:rm No outliers processing, removal of last action.
4 x:l60 No outliers processing, 60 min last action duration limit.
5 x:l30 No outliers processing, 30 min last action duration limit.
6 x:l10 No outliers processing, 10 min last action duration limit.
Group 2: Thresholding outliers and last actions
7 l60 60 min duration limit.
8 l30 30 min duration limit.
9 l10 10 min duration limit.
Group 3: Thresholding outliers and estimating last actions
10 l60:ev 60 min duration limit, last actions estimated.
11 l30:ev 30 min duration limit, last actions estimated.
12 l10:ev 30 min duration limit, last actions estimated.
Group 4: Estimating outliers and last actions
13 +60ev Estimate last actions and actions longer than 60 min.
14 +30ev Estimate last actions and actions longer than 30 min.
15 +10ev Estimate last actions and actions longer than 10 min.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 17 / 28
Methods Experimental procedure
Statistical Analysis
• For each of the five outome measures, we constructed 16 multiple regression
models:
• 1 model using count measures
• 15 using diferently extracted time-on-task measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 18 / 28
Results R2 variations of the overall models
Results
R2
Performance Measure Min Max Range Mean SD
TMA2Grade 0.08 0.26 0.18 0.14 0.04
TMA3Grade 0.04 0.17 0.12 0.09 0.04
ParticipationGrade 0.23 0.37 0.13 0.3 0.04
FinalGrade 0.06 0.28 0.23 0.16 0.05
CoIHigh 0.21 0.28 0.07 0.26 0.02
Variation in R2
values across five outcome measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 19 / 28
Results R2 variations of the overall models
Time-on-task extraction configuration
R
2
0.150.250.35
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Higher levels of cognitve presence (Integration + Resolution)
0.050.150.25
Final percentage grade
0.200.300.40
Course participation grade
0.000.100.20
TMA3 grade: journal readings
Group 1:
No outlier processing
Group 2:
Duration limit
Group 3:
Duration limit + estimation
Group 4:
Estimation above limit
0.050.150.25
TMA2 grade: literature review
Counts
Time-on-task
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 20 / 28
Results Significance of individual predictors
Group 1 Group 2 Group 3 Group 4
DV IV x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
TMA2Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
TMA3Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
Part.Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
FinalGrade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
CoIHigh Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Significant model at p < .05
Significant β > 0
Significant β < 0
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 21 / 28
Discussion
Discussion
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 22 / 28
Discussion
History repeats itself
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 23 / 28
Discussion
Slipery road ahead
• We need more caution when using time-on-task measures for building
learning analytics models.
• We need to provide details of how time-on-task has been estimated.
• Supplemetary materials are great for this!
• Source code repositories with time-on-task estimation code.
• Develop plugins for time-on-task extraction from popular platforms.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 24 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,
• too much emphasis on p-values, what goes into the model counts!
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:
• Providing test statistic values and degrees of freedom,
• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),
• Reporting effect sizes (e.g., R2
, η2
, Hedges’ g, Cram´er’s V),
• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,
• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,
• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
Especially for big senstationalistic claims that conflict
existing literature!
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
“Extraordinary claims require extraordinary
proof”(Truzzi, 1978, p. 11)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and their
implications.
• Validate results by adopting several different methods
• Results in loss of test power,
• Too much focus on small effects dependent on particular method being
adopted.
• Conduct replication studies.
• Avoid p-hacking and HARKing (Hypothesizing After the Results are Known)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Limitations
Limitations
• Ca not provide a definitive recommendation for practice,
• One statistical model,
• Despite 160,000 log records, it is still one dataset, and
• There are many more time-on-task estimation strategies.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 27 / 28
Discussion Future work
Future work
• Use DALMOOC data from Prosolo learning platform:
• First six weeks 15 min innactivity logout.
• Second six weeks 60 min innactivity logout.
• How many of students returned (false positive), and how many did not (true
positive)?
• Looking upon ITS research, provide a gold-standard data.
• LMS plugin that through javascript keeps a track of user activity.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 28 / 28
Thank you
Vitomir Kovanovic
vitomir.kovanovic.info
v.kovanovic@ed.ac.uk
References I
Bloom, Benjamin S. (1974). “Time and learning”. In: American Psychologist 29.9, pp. 682–688.
Carroll, Jb (1963). “A Model of School Learning”. English. In: Teachers College Record 64.8.
WOS:A1963CAJ4400010, pp. 723–733.
Chickering, Arthur W and Zelda F Gamson (1989). “Seven principles for good practice in
undergraduate education”. In: Biochemical Education 17.3, pp. 140–141.
Karweit, Nancy and Robert E. Slavin (1982). “Time-on-task: Issues of timing, sampling, and
definition”. In: Journal of Educational Psychology 74.6, pp. 844–851.
Truzzi, Marcello (1978). “On the Extraordinary: An Attempt at Clarification”. In: Zetetic Scholar 1.1.

More Related Content

What's hot

MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
Vitomir Kovanovic
 
Analysis of Metadata and Topic Modeling for
Analysis of Metadata and Topic Modeling forAnalysis of Metadata and Topic Modeling for
Analysis of Metadata and Topic Modeling for
Jigar Mehta
 
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
Eamon Costello
 
Turning teaching innovations into education publications
Turning teaching innovations into education publicationsTurning teaching innovations into education publications
Turning teaching innovations into education publications
Chris Willmott
 
H nathalie gettliffe_sig_eurocall2010lyon[1]
H nathalie gettliffe_sig_eurocall2010lyon[1]H nathalie gettliffe_sig_eurocall2010lyon[1]
H nathalie gettliffe_sig_eurocall2010lyon[1]
nickyjohnson
 
What questions are MOOCs asking? An evidence based investigation
What questions are MOOCs asking? An evidence based investigationWhat questions are MOOCs asking? An evidence based investigation
What questions are MOOCs asking? An evidence based investigation
Eamon Costello
 
The Anatomy of the 21st Century Educator
The Anatomy of the 21st Century Educator The Anatomy of the 21st Century Educator
The Anatomy of the 21st Century Educator
Simon Bates
 
Corpus Analysis of MOOC discussions
Corpus Analysis of MOOC discussionsCorpus Analysis of MOOC discussions
Corpus Analysis of MOOC discussions
Shi Min CHUA
 
The 21st Century Educator - students as partners in teaching and learning
The  21st Century Educator - students as partners in teaching and learning The  21st Century Educator - students as partners in teaching and learning
The 21st Century Educator - students as partners in teaching and learning
Simon Bates
 
Tenc Winterschool09 Davinia Slideshare
Tenc Winterschool09 Davinia SlideshareTenc Winterschool09 Davinia Slideshare
Tenc Winterschool09 Davinia Slideshare
guest94c824
 
Turning teaching initiatives into pedagogic publications
Turning teaching initiatives into pedagogic publicationsTurning teaching initiatives into pedagogic publications
Turning teaching initiatives into pedagogic publications
Chris Willmott
 
An Open Research Approach - 2015 Open Education Week TU Delft
An Open Research Approach - 2015 Open Education Week TU DelftAn Open Research Approach - 2015 Open Education Week TU Delft
An Open Research Approach - 2015 Open Education Week TU Delft
Thieme Hennis
 
«Learning Analytics at the Open University and the UK»
 «Learning Analytics at the Open University and the UK» «Learning Analytics at the Open University and the UK»
«Learning Analytics at the Open University and the UK»
Bart Rienties
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
Dagmar Monett
 
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Denise Whitelock
 
NTU Innovations in Teaching Seminar - students as co creators
NTU Innovations in Teaching Seminar - students as co creatorsNTU Innovations in Teaching Seminar - students as co creators
NTU Innovations in Teaching Seminar - students as co creators
Simon Bates
 
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
Bart Rienties
 
Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning Programming
Peter Brusilovsky
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
Denise Whitelock
 
Exploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
Exploring the Map of Distance and e-Learning Research - a Bibliographic AnalysisExploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
Exploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
EDEN Digital Learning Europe
 

What's hot (20)

MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
 
Analysis of Metadata and Topic Modeling for
Analysis of Metadata and Topic Modeling forAnalysis of Metadata and Topic Modeling for
Analysis of Metadata and Topic Modeling for
 
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
The Future of Online Testing with MOOCs: An Exploratory Analysis of Current P...
 
Turning teaching innovations into education publications
Turning teaching innovations into education publicationsTurning teaching innovations into education publications
Turning teaching innovations into education publications
 
H nathalie gettliffe_sig_eurocall2010lyon[1]
H nathalie gettliffe_sig_eurocall2010lyon[1]H nathalie gettliffe_sig_eurocall2010lyon[1]
H nathalie gettliffe_sig_eurocall2010lyon[1]
 
What questions are MOOCs asking? An evidence based investigation
What questions are MOOCs asking? An evidence based investigationWhat questions are MOOCs asking? An evidence based investigation
What questions are MOOCs asking? An evidence based investigation
 
The Anatomy of the 21st Century Educator
The Anatomy of the 21st Century Educator The Anatomy of the 21st Century Educator
The Anatomy of the 21st Century Educator
 
Corpus Analysis of MOOC discussions
Corpus Analysis of MOOC discussionsCorpus Analysis of MOOC discussions
Corpus Analysis of MOOC discussions
 
The 21st Century Educator - students as partners in teaching and learning
The  21st Century Educator - students as partners in teaching and learning The  21st Century Educator - students as partners in teaching and learning
The 21st Century Educator - students as partners in teaching and learning
 
Tenc Winterschool09 Davinia Slideshare
Tenc Winterschool09 Davinia SlideshareTenc Winterschool09 Davinia Slideshare
Tenc Winterschool09 Davinia Slideshare
 
Turning teaching initiatives into pedagogic publications
Turning teaching initiatives into pedagogic publicationsTurning teaching initiatives into pedagogic publications
Turning teaching initiatives into pedagogic publications
 
An Open Research Approach - 2015 Open Education Week TU Delft
An Open Research Approach - 2015 Open Education Week TU DelftAn Open Research Approach - 2015 Open Education Week TU Delft
An Open Research Approach - 2015 Open Education Week TU Delft
 
«Learning Analytics at the Open University and the UK»
 «Learning Analytics at the Open University and the UK» «Learning Analytics at the Open University and the UK»
«Learning Analytics at the Open University and the UK»
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
 
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
Investigating the Pedagogical Push and Technological Pull of Computer Assiste...
 
NTU Innovations in Teaching Seminar - students as co creators
NTU Innovations in Teaching Seminar - students as co creatorsNTU Innovations in Teaching Seminar - students as co creators
NTU Innovations in Teaching Seminar - students as co creators
 
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
OU/Leverhulme Open World Learning: Knowledge Exchange and Book Launch Event p...
 
Personalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning ProgrammingPersonalized Online Practice Systems for Learning Programming
Personalized Online Practice Systems for Learning Programming
 
Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?Should feedback be at the centre of Personalised Learning?
Should feedback be at the centre of Personalised Learning?
 
Exploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
Exploring the Map of Distance and e-Learning Research - a Bibliographic AnalysisExploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
Exploring the Map of Distance and e-Learning Research - a Bibliographic Analysis
 

Similar to Penetrating the black box of time-on-task estimation

B00242899_MyHonorsProject
B00242899_MyHonorsProjectB00242899_MyHonorsProject
B00242899_MyHonorsProject
Thomas C Nicholson
 
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
Chris Willmott
 
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
IAEME Publication
 
The College Classroom Week 10: Teaching as Research
The College Classroom Week 10: Teaching as ResearchThe College Classroom Week 10: Teaching as Research
The College Classroom Week 10: Teaching as Research
Peter Newbury
 
The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator
Simon Bates
 
The why and what of testa
The why and what of testaThe why and what of testa
The why and what of testa
Tansy Jessop
 
CIRTL Class Meeting 9: Teaching as research
CIRTL Class Meeting 9: Teaching as researchCIRTL Class Meeting 9: Teaching as research
CIRTL Class Meeting 9: Teaching as research
Peter Newbury
 
CIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
CIRTL Spring 2016 The College Classroom Meeting 5 - Active LearningCIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
CIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
Peter Newbury
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
EDEN Digital Learning Europe
 
Class Time Reconsidered
Class Time ReconsideredClass Time Reconsidered
Class Time Reconsidered
Derek Bruff
 
The Search for Truth in Objective & Subject Crowdsourcing
The Search for Truth in Objective & Subject CrowdsourcingThe Search for Truth in Objective & Subject Crowdsourcing
The Search for Truth in Objective & Subject Crowdsourcing
Matthew Lease
 
TESTA Interactive Masterclass
TESTA Interactive MasterclassTESTA Interactive Masterclass
TESTA Interactive Masterclass
Tansy Jessop
 
Plate spinning - Working with Students as Partners
Plate spinning - Working with Students as PartnersPlate spinning - Working with Students as Partners
Plate spinning - Working with Students as Partners
Sam Nolan
 
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning A Quali...
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning  A Quali...Assertion Reason Multiple Choice Testing As A Tool For Deep Learning  A Quali...
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning A Quali...
Pedro Craggett
 
CCC-M F2F meeting_141007
CCC-M F2F meeting_141007CCC-M F2F meeting_141007
CCC-M F2F meeting_141007
McGill_RSB
 
The College Classroom (Wi15) Session 7: Teaching as Research
The College Classroom (Wi15) Session 7: Teaching as ResearchThe College Classroom (Wi15) Session 7: Teaching as Research
The College Classroom (Wi15) Session 7: Teaching as Research
Peter Newbury
 
Assignment Title Conducting Primary ResearchDeveloping the ab.docx
Assignment Title Conducting Primary ResearchDeveloping the ab.docxAssignment Title Conducting Primary ResearchDeveloping the ab.docx
Assignment Title Conducting Primary ResearchDeveloping the ab.docx
ssuser562afc1
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of Inquiry
Vitomir Kovanovic
 
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
Alexandra M. Pickett
 
CIRTL Class Meeting 4: Assessment
CIRTL Class Meeting 4: AssessmentCIRTL Class Meeting 4: Assessment
CIRTL Class Meeting 4: Assessment
Peter Newbury
 

Similar to Penetrating the black box of time-on-task estimation (20)

B00242899_MyHonorsProject
B00242899_MyHonorsProjectB00242899_MyHonorsProject
B00242899_MyHonorsProject
 
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
Pedagogy Involving Capture Technology: Uses of Panopto beyond the recording o...
 
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
A CRITICAL REVIEW ON THE OPTIMIZATION METHODS IN SOLVING EXAM TIMETABLING AND...
 
The College Classroom Week 10: Teaching as Research
The College Classroom Week 10: Teaching as ResearchThe College Classroom Week 10: Teaching as Research
The College Classroom Week 10: Teaching as Research
 
The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator The Anatomy of a 21st Century Educator
The Anatomy of a 21st Century Educator
 
The why and what of testa
The why and what of testaThe why and what of testa
The why and what of testa
 
CIRTL Class Meeting 9: Teaching as research
CIRTL Class Meeting 9: Teaching as researchCIRTL Class Meeting 9: Teaching as research
CIRTL Class Meeting 9: Teaching as research
 
CIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
CIRTL Spring 2016 The College Classroom Meeting 5 - Active LearningCIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
CIRTL Spring 2016 The College Classroom Meeting 5 - Active Learning
 
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
Evaluation in distance education @ VIVES: some tools and trends - Jean-Claude...
 
Class Time Reconsidered
Class Time ReconsideredClass Time Reconsidered
Class Time Reconsidered
 
The Search for Truth in Objective & Subject Crowdsourcing
The Search for Truth in Objective & Subject CrowdsourcingThe Search for Truth in Objective & Subject Crowdsourcing
The Search for Truth in Objective & Subject Crowdsourcing
 
TESTA Interactive Masterclass
TESTA Interactive MasterclassTESTA Interactive Masterclass
TESTA Interactive Masterclass
 
Plate spinning - Working with Students as Partners
Plate spinning - Working with Students as PartnersPlate spinning - Working with Students as Partners
Plate spinning - Working with Students as Partners
 
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning A Quali...
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning  A Quali...Assertion Reason Multiple Choice Testing As A Tool For Deep Learning  A Quali...
Assertion Reason Multiple Choice Testing As A Tool For Deep Learning A Quali...
 
CCC-M F2F meeting_141007
CCC-M F2F meeting_141007CCC-M F2F meeting_141007
CCC-M F2F meeting_141007
 
The College Classroom (Wi15) Session 7: Teaching as Research
The College Classroom (Wi15) Session 7: Teaching as ResearchThe College Classroom (Wi15) Session 7: Teaching as Research
The College Classroom (Wi15) Session 7: Teaching as Research
 
Assignment Title Conducting Primary ResearchDeveloping the ab.docx
Assignment Title Conducting Primary ResearchDeveloping the ab.docxAssignment Title Conducting Primary ResearchDeveloping the ab.docx
Assignment Title Conducting Primary ResearchDeveloping the ab.docx
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of Inquiry
 
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
Student Survey: Open SUNY COTE Online Learning Survey (Spring 2015)
 
CIRTL Class Meeting 4: Assessment
CIRTL Class Meeting 4: AssessmentCIRTL Class Meeting 4: Assessment
CIRTL Class Meeting 4: Assessment
 

More from Vitomir Kovanovic

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
Vitomir Kovanovic
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
Vitomir Kovanovic
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
Vitomir Kovanovic
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Vitomir Kovanovic
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
Vitomir Kovanovic
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Vitomir Kovanovic
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Vitomir Kovanovic
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
Vitomir Kovanovic
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
Vitomir Kovanovic
 
A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...
Vitomir Kovanovic
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
Vitomir Kovanovic
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
Vitomir Kovanovic
 
Understand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsUnderstand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analytics
Vitomir Kovanovic
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
Vitomir Kovanovic
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Vitomir Kovanovic
 
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
Vitomir Kovanovic
 
Automated System for Cognitive Presence Coding
Automated System for Cognitive Presence CodingAutomated System for Cognitive Presence Coding
Automated System for Cognitive Presence Coding
Vitomir Kovanovic
 
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 TutorialTopic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Vitomir Kovanovic
 
What is the source of social capital? The association between social network ...
What is the source of social capital? The association between social network ...What is the source of social capital? The association between social network ...
What is the source of social capital? The association between social network ...
Vitomir Kovanovic
 

More from Vitomir Kovanovic (19)

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
 
A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
 
Understand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsUnderstand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analytics
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
 
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
 
Automated System for Cognitive Presence Coding
Automated System for Cognitive Presence CodingAutomated System for Cognitive Presence Coding
Automated System for Cognitive Presence Coding
 
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 TutorialTopic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
 
What is the source of social capital? The association between social network ...
What is the source of social capital? The association between social network ...What is the source of social capital? The association between social network ...
What is the source of social capital? The association between social network ...
 

Recently uploaded

Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 
Stack Memory Organization of 8086 Microprocessor
Stack Memory Organization of 8086 MicroprocessorStack Memory Organization of 8086 Microprocessor
Stack Memory Organization of 8086 Microprocessor
JomonJoseph58
 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Henry Hollis
 
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdfREASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
giancarloi8888
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
iammrhaywood
 
How to deliver Powerpoint Presentations.pptx
How to deliver Powerpoint  Presentations.pptxHow to deliver Powerpoint  Presentations.pptx
How to deliver Powerpoint Presentations.pptx
HajraNaeem15
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
PsychoTech Services
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
سمير بسيوني
 
Standardized tool for Intelligence test.
Standardized tool for Intelligence test.Standardized tool for Intelligence test.
Standardized tool for Intelligence test.
deepaannamalai16
 
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
zuzanka
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
S. Raj Kumar
 
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
Nguyen Thanh Tu Collection
 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
RAHUL
 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
Jyoti Chand
 
Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"
National Information Standards Organization (NISO)
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
Katrina Pritchard
 
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptxPengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Fajar Baskoro
 
HYPERTENSION - SLIDE SHARE PRESENTATION.
HYPERTENSION - SLIDE SHARE PRESENTATION.HYPERTENSION - SLIDE SHARE PRESENTATION.
HYPERTENSION - SLIDE SHARE PRESENTATION.
deepaannamalai16
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
Nguyen Thanh Tu Collection
 
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDFLifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
Vivekanand Anglo Vedic Academy
 

Recently uploaded (20)

Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 
Stack Memory Organization of 8086 Microprocessor
Stack Memory Organization of 8086 MicroprocessorStack Memory Organization of 8086 Microprocessor
Stack Memory Organization of 8086 Microprocessor
 
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.pptLevel 3 NCEA - NZ: A  Nation In the Making 1872 - 1900 SML.ppt
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
 
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdfREASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
REASIGNACION 2024 UGEL CHUPACA 2024 UGEL CHUPACA.pdf
 
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptxNEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
NEWSPAPERS - QUESTION 1 - REVISION POWERPOINT.pptx
 
How to deliver Powerpoint Presentations.pptx
How to deliver Powerpoint  Presentations.pptxHow to deliver Powerpoint  Presentations.pptx
How to deliver Powerpoint Presentations.pptx
 
Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...Gender and Mental Health - Counselling and Family Therapy Applications and In...
Gender and Mental Health - Counselling and Family Therapy Applications and In...
 
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdfمصحف القراءات العشر   أعد أحرف الخلاف سمير بسيوني.pdf
مصحف القراءات العشر أعد أحرف الخلاف سمير بسيوني.pdf
 
Standardized tool for Intelligence test.
Standardized tool for Intelligence test.Standardized tool for Intelligence test.
Standardized tool for Intelligence test.
 
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptxRESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
RESULTS OF THE EVALUATION QUESTIONNAIRE.pptx
 
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching AptitudeUGC NET Exam Paper 1- Unit 1:Teaching Aptitude
UGC NET Exam Paper 1- Unit 1:Teaching Aptitude
 
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
BÀI TẬP DẠY THÊM TIẾNG ANH LỚP 7 CẢ NĂM FRIENDS PLUS SÁCH CHÂN TRỜI SÁNG TẠO ...
 
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UPLAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
LAND USE LAND COVER AND NDVI OF MIRZAPUR DISTRICT, UP
 
Wound healing PPT
Wound healing PPTWound healing PPT
Wound healing PPT
 
Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"Benner "Expanding Pathways to Publishing Careers"
Benner "Expanding Pathways to Publishing Careers"
 
BBR 2024 Summer Sessions Interview Training
BBR  2024 Summer Sessions Interview TrainingBBR  2024 Summer Sessions Interview Training
BBR 2024 Summer Sessions Interview Training
 
Pengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptxPengantar Penggunaan Flutter - Dart programming language1.pptx
Pengantar Penggunaan Flutter - Dart programming language1.pptx
 
HYPERTENSION - SLIDE SHARE PRESENTATION.
HYPERTENSION - SLIDE SHARE PRESENTATION.HYPERTENSION - SLIDE SHARE PRESENTATION.
HYPERTENSION - SLIDE SHARE PRESENTATION.
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 CẢ NĂM - GLOBAL SUCCESS - NĂM HỌC 2023-2024 (CÓ FI...
 
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDFLifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
Lifelines of National Economy chapter for Class 10 STUDY MATERIAL PDF
 

Penetrating the black box of time-on-task estimation

  • 1. Penetrating the Black Box of Time-on-task Estimation http://bit.do/lak tot Vitomir Kovanovi´c School of Informatics, University of Edinburgh, Edinburgh, United Kingdom v.kovanovic@ed.ac.uk Dragan Gaˇsevi´c Schools of Education and Informatics, University of Edinburgh, Edinburgh, United Kingdom dgasevic@acm.org Shane Dawson Learning and Teaching Unit, University of South Australia, Adelaide, Australia shane.dawson@unisa.edu.au Sre´cko Joksimovi´c School of Interactive Arts and Technology, Simon Fraser University, Burnaby, Canada sjoksimo@sfu.ca Ryan S. Baker Teachers College, Columbia University, New York, USA baker2@exchange.tc.columbia.edu Marek Hatala School of Interactive Arts and Technology, Simon Fraser University, Burnaby, Canada mhatala@sfu.ca March 19, 2015 Marist College, Poughkeepsie, NY, USA
  • 2. Introduction Time-on-task “All learning, whether done in school or elsewhere, requires time.” (Bloom, 1974, p. 682) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
  • 3. Introduction Time-on-task “All learning, whether done in school or elsewhere, requires time.” (Bloom, 1974, p. 682) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
  • 4. Introduction Time-on-task However, there is a big difference between elapsed time, and time student actually spent on learning (Carroll, 1963). Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
  • 5. Introduction Time-on-task However, there is a big difference between elapsed time, and time student actually spent on learning (Carroll, 1963). Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
  • 6. Introduction Origins of time-on-task in educational research • Seminal work by J. Carroll: “A model of school learning” in 1963 put a direct link between time and learning outcomes. • Carroll (1963) differentiaded between elapsed time and time spent on learning. • How time is spent is what ultimately matters! • Increase of time-on-task was one of the key principles of effective education (Chickering and Gamson, 1989). Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 3 / 28
  • 7. Introduction Challenges with time-on-task measures • Many different operationalizations and methods of measurement • Days in school, • Number of lectures attended, • Observing students’ behavior every X minutes. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 4 / 28
  • 8. Introduction Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 5 / 28
  • 9. Introduction Time-on-task in distance education / online leanring • LMS produce a large amount of data that is used for learning analytics • Typically data is stored as a list of events that occured during system use • Many learning analytics studies use time-on-task measures • Time-on-task typically calculated as time difference between recorded events Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
  • 10. Introduction Time-on-task in distance education / online leanring • LMS produce a large amount of data that is used for learning analytics • Typically data is stored as a list of events that occured during system use • Many learning analytics studies use time-on-task measures • Time-on-task typically calculated as time difference between recorded events Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
  • 11. Introduction Time-on-task estimation from LMS trace data • What if action duration is too large? Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
  • 12. Introduction Time-on-task estimation from LMS trace data • What if action duration is too large? • Limit all actions to 10 minutes? • Limit all actions to 30 minutes? • Discard those actions completely? • Estimate their duration based on other available data points? Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
  • 13. Introduction Time-on-task estimation from LMS trace data • What if action duration is too large? • Limit all actions to 10 minutes? • Limit all actions to 30 minutes? • Discard those actions completely? • Estimate their duration based on other available data points? • There are many choices in time-on-task estimation • Only few studies describe the process of time-on-task estimation • Typycally simple heuristics are used (limit action duration to X minutes) • Problems for replications Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
  • 14. Introduction Time-on-task estimation from LMS trace data • What if action duration is too large? • Limit all actions to 10 minutes? • Limit all actions to 30 minutes? • Discard those actions completely? • Estimate their duration based on other available data points? • There are many choices in time-on-task estimation • Only few studies describe the process of time-on-task estimation • Typycally simple heuristics are used (limit action duration to X minutes) • Problems for replications • How those choices affect the final study results? • Can we trust findings based on time-on-task estimates from trace data? Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
  • 15. Introduction Idea for the study • Research study that involved clustering using time-on-task measures, • Adopted one of the heuristics, • Still, it didn’t feel totally right. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 8 / 28
  • 16. Introduction Research Questions: Effects of time-on-task measuring on analytics results Research Questions Drawing on the Karweit and Slavin (1982) research, we conducted a study to answer the following questions: • What effects do different methods for estimation of time on-task-measures from LMS data have on the results of analytical models? • Are there differences in their statistical significance and overall conclusions that can be drawn from them? Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 9 / 28
  • 17. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 18. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 19. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 20. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 T3 - T2 ... ... very long time period T3 Walter Start Viewing Assignment TMA1 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 21. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 T3 - T2 ... ... very long time period T3 Walter Start Viewing Assignment TMA1 T4 - T3 T4 Walter Start Viewing Resource R1 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 22. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 T3 - T2 ... ... very long time period T3 Walter Start Viewing Assignment TMA1 T4 - T3 T4 Walter Start Viewing Resource R1 T5 - T4 ... ... moderately long time period T5 Walter User Login 0s Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 23. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 T3 - T2 ... ... very long time period T3 Walter Start Viewing Assignment TMA1 T4 - T3 T4 Walter Start Viewing Resource R1 T5 - T4 ... ... moderately long time period T5 Walter User Login 0sTwo types of problems • Outlier estimation • Last action estimation Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 24. Time-on-task estimation procedures Time User Action Duration T0 Walter UserLogin 0s T1 Walter Start Viewing Discussion D1 T2 - T1 T2 Walter Start Viewing Discussion D2 T3 - T2 ... ... very long time period T3 Walter Start Viewing Assignment TMA1 T4 - T3 T4 Walter Start Viewing Resource R1 T5 - T4 ... ... moderately long time period T5 Walter User Login 0sTwo types of problems • Outlier estimation • Last action estimation Typical solutions • Ignore problematic actions • Put certain upper limit on action duration • Estimate based on other data points Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
  • 25. Methods Dataset Dataset: Trace data • 6 offers of 13-week research-intensive fully online masters course at Canadian public university. • Moodle used as LMS platform: total of 81 students and 167,261 log records. Students Actions Messages Winter 2008 15 33,976 212 Fall 2008 22 49,928 633 Summer 2009 10 21,059 243 Fall 2009 7 11,346 63 Winter 2010 14 31,169 359 Winter 2011 13 19,783 237 Average (SD) 13.5 (5.1) 27,877 (13,561) 291.2 (192.4) Total 81 167,261 1747 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 11 / 28
  • 26. Methods Dataset Time-on-task measures / count measures • Only used activities that were planned by the course design • Viewing assignments • Viewing resources • Viewing discussions • Posting to discussions • Updating discussion messages • Extracted both count and time-on-task measures • Count measures used as baseline • Time-on-task measures extracted in 15 different ways Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 12 / 28
  • 27. Methods Dataset Outcome measures • Grade structure: • TMA1 (15%): present a research paper • TMA2 (25%): write literature review paper • TMA3 (15%): write answers to 6 essay questions • TMA4 (30%): work in a team on a software project • Participation (15%): participate productively in course discussions. • Final grade Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
  • 28. Methods Dataset Outcome measures • Grade structure: • TMA1 (15%): present a research paper • TMA2 (25%): write literature review paper • TMA3 (15%): write answers to 6 essay questions • TMA4 (30%): work in a team on a software project • Participation (15%): participate productively in course discussions. • Final grade Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
  • 29. Methods Dataset Outcome measures • Grade structure: • TMA1 (15%): present a research paper • TMA2 (25%): write literature review paper • TMA3 (15%): write answers to 6 essay questions • TMA4 (30%): work in a team on a software project • Participation (15%): participate productively in course discussions. • Final grade 1 TMA2 grade 2 TMA3 grade 3 Participation grade 4 Final grade Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
  • 30. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 31. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. • CoI model: important dimensions of distance education experience: 1 Social presence: climate within course 2 Teaching presence: role of instructor before and during the course 3 Cognitive presence: development of critical thinking skills Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 32. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. • CoI model: important dimensions of distance education experience: 1 Social presence: climate within course 2 Teaching presence: role of instructor before and during the course 3 Cognitive presence: development of critical thinking skills • Focus on cognitive presence Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 33. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. • CoI model: important dimensions of distance education experience: 1 Social presence: climate within course 2 Teaching presence: role of instructor before and during the course 3 Cognitive presence: development of critical thinking skills • Focus on cognitive presence • Two coders coded each of 1,747 messages using cognitive presence coding scheme • Excellent agreement, Cohen’s κ = 0.97 (only 32 disagreements) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 34. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. • CoI model: important dimensions of distance education experience: 1 Social presence: climate within course 2 Teaching presence: role of instructor before and during the course 3 Cognitive presence: development of critical thinking skills • Focus on cognitive presence • Two coders coded each of 1,747 messages using cognitive presence coding scheme • Excellent agreement, Cohen’s κ = 0.97 (only 32 disagreements) ID Phase Messages (%) 0 Other 140 8.01% 1 Triggering Event 308 17.63% 2 Exploration 684 39.17% 3 Integration 508 29.08% 4 Resolution 107 6.12% All phases 1747 100% Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 35. Methods Dataset Dataset: discussion messages • Coded discussion messages in accordance with Community of Inquiry (CoI) model. • CoI model: important dimensions of distance education experience: 1 Social presence: climate within course 2 Teaching presence: role of instructor before and during the course 3 Cognitive presence: development of critical thinking skills • Focus on cognitive presence • Two coders coded each of 1,747 messages using cognitive presence coding scheme • Excellent agreement, Cohen’s κ = 0.97 (only 32 disagreements) ID Phase Messages (%) 0 Other 140 8.01% 1 Triggering Event 308 17.63% 2 Exploration 684 39.17% 3 Integration 508 29.08% 4 Resolution 107 6.12% All phases 1747 100% Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
  • 36. Methods Dataset Outcome measures 1 TMA2 grade 2 TMA3 grade 3 Participation grade 4 Final grade 5 CoIHigh: number of integration and resolution messages Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 15 / 28
  • 37. Methods Dataset Measures summary # Name Description Count measures 1 AsignmentViewCount Number of assignment views. 2 ResourceViewCount Number of resources views. 3 DiscussionViewCount Number of course discussion views. 4 AddPostCount Number of posted messages. 5 UpdatePostCount Number of post updates. Time-on-task measures 6 AsignmentViewTime Time spent on course assignments. 7 ResourceViewTime Time spent reading course resources. 8 DiscussionViewTime Time spent viewing course discussions. 9 AddPostTime Time spent posting discussion messages. 10 UpdatePostTime Time spent updating discussion messages. Performance measures 11 TMA2Grade Grade for literature review paper. 12 TMA3Grade Grade for journal papers readings. 13 ParticipationGrade Grade for participation in course discussions. 14 FinalGrade Final grade in the course. 15 CoIHigh Integration and resolution message count. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 16 / 28
  • 38. Methods Experimental procedure Time-on-task estimation strategies # Name Description Group 1: No outliers processing, different processing of last actions 1 x:x No outliers and last action processing. 2 x:ev No outliers processing, estimation of last action duration. 3 x:rm No outliers processing, removal of last action. 4 x:l60 No outliers processing, 60 min last action duration limit. 5 x:l30 No outliers processing, 30 min last action duration limit. 6 x:l10 No outliers processing, 10 min last action duration limit. Group 2: Thresholding outliers and last actions 7 l60 60 min duration limit. 8 l30 30 min duration limit. 9 l10 10 min duration limit. Group 3: Thresholding outliers and estimating last actions 10 l60:ev 60 min duration limit, last actions estimated. 11 l30:ev 30 min duration limit, last actions estimated. 12 l10:ev 30 min duration limit, last actions estimated. Group 4: Estimating outliers and last actions 13 +60ev Estimate last actions and actions longer than 60 min. 14 +30ev Estimate last actions and actions longer than 30 min. 15 +10ev Estimate last actions and actions longer than 10 min. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 17 / 28
  • 39. Methods Experimental procedure Statistical Analysis • For each of the five outome measures, we constructed 16 multiple regression models: • 1 model using count measures • 15 using diferently extracted time-on-task measures Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 18 / 28
  • 40. Results R2 variations of the overall models Results R2 Performance Measure Min Max Range Mean SD TMA2Grade 0.08 0.26 0.18 0.14 0.04 TMA3Grade 0.04 0.17 0.12 0.09 0.04 ParticipationGrade 0.23 0.37 0.13 0.3 0.04 FinalGrade 0.06 0.28 0.23 0.16 0.05 CoIHigh 0.21 0.28 0.07 0.26 0.02 Variation in R2 values across five outcome measures Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 19 / 28
  • 41. Results R2 variations of the overall models Time-on-task extraction configuration R 2 0.150.250.35 x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev Higher levels of cognitve presence (Integration + Resolution) 0.050.150.25 Final percentage grade 0.200.300.40 Course participation grade 0.000.100.20 TMA3 grade: journal readings Group 1: No outlier processing Group 2: Duration limit Group 3: Duration limit + estimation Group 4: Estimation above limit 0.050.150.25 TMA2 grade: literature review Counts Time-on-task Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 20 / 28
  • 42. Results Significance of individual predictors Group 1 Group 2 Group 3 Group 4 DV IV x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev TMA2Grade Assign.ViewTime β coefficients Res.ViewTime Disc.ViewTime AddPostTime UpdatePostTime TMA3Grade Assign.ViewTime β coefficients Res.ViewTime Disc.ViewTime AddPostTime UpdatePostTime Part.Grade Assign.ViewTime β coefficients Res.ViewTime Disc.ViewTime AddPostTime UpdatePostTime FinalGrade Assign.ViewTime β coefficients Res.ViewTime Disc.ViewTime AddPostTime UpdatePostTime CoIHigh Assign.ViewTime β coefficients Res.ViewTime Disc.ViewTime AddPostTime UpdatePostTime x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev Significant model at p < .05 Significant β > 0 Significant β < 0 Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 21 / 28
  • 43. Discussion Discussion Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 22 / 28
  • 44. Discussion History repeats itself Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 23 / 28
  • 45. Discussion Slipery road ahead • We need more caution when using time-on-task measures for building learning analytics models. • We need to provide details of how time-on-task has been estimated. • Supplemetary materials are great for this! • Source code repositories with time-on-task estimation code. • Develop plugins for time-on-task extraction from popular platforms. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 24 / 28
  • 46. Discussion Implications for the Learning Analytics Community Implications for learning anlaytics research Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
  • 47. Discussion Implications for the Learning Analytics Community Implications for learning anlaytics research • Implications on accepted standard of research in learning analytics, Good research practices: • Providing test statistic values and degrees of freedom, • Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α), • Reporting effect sizes (e.g., R2 , η2 , Hedges’ g, Cram´er’s V), • Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
  • 48. Discussion Implications for the Learning Analytics Community Implications for learning anlaytics research • Implications on accepted standard of research in learning analytics, Good research practices: • Providing test statistic values and degrees of freedom, • Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α), • Reporting effect sizes (e.g., R2 , η2 , Hedges’ g, Cram´er’s V), • Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni) • Details of time-on-task estimation should be reported. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
  • 49. Discussion Implications for the Learning Analytics Community Implications for learning anlaytics research • Implications on accepted standard of research in learning analytics, Good research practices: • Providing test statistic values and degrees of freedom, • Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α), • Reporting effect sizes (e.g., R2 , η2 , Hedges’ g, Cram´er’s V), • Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni) • Details of time-on-task estimation should be reported. • Implications on validity of learning analytics findings, • too much emphasis on p-values, what goes into the model counts! Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
  • 50. Discussion Implications for the Learning Analytics Community Implications for learning anlaytics research • Implications on accepted standard of research in learning analytics, Good research practices: • Providing test statistic values and degrees of freedom, • Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α), • Reporting effect sizes (e.g., R2 , η2 , Hedges’ g, Cram´er’s V), • Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni) • Details of time-on-task estimation should be reported. • Implications on validity of learning analytics findings, • too much emphasis on p-values, what goes into the model counts! • Implications on replication potential, • Potential practical impact of learning analytics research. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
  • 51. Discussion Implications for the Learning Analytics Community More general problem • Be more aware of all important methodological decissions and their implications. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
  • 52. Discussion Implications for the Learning Analytics Community More general problem • Be more aware of all important methodological decissions and their implications. Especially for big senstationalistic claims that conflict existing literature! Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
  • 53. Discussion Implications for the Learning Analytics Community More general problem • Be more aware of all important methodological decissions and their implications. “Extraordinary claims require extraordinary proof”(Truzzi, 1978, p. 11) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
  • 54. Discussion Implications for the Learning Analytics Community More general problem • Be more aware of all important methodological decissions and their implications. • Validate results by adopting several different methods • Results in loss of test power, • Too much focus on small effects dependent on particular method being adopted. • Conduct replication studies. • Avoid p-hacking and HARKing (Hypothesizing After the Results are Known) Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
  • 55. Discussion Limitations Limitations • Ca not provide a definitive recommendation for practice, • One statistical model, • Despite 160,000 log records, it is still one dataset, and • There are many more time-on-task estimation strategies. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 27 / 28
  • 56. Discussion Future work Future work • Use DALMOOC data from Prosolo learning platform: • First six weeks 15 min innactivity logout. • Second six weeks 60 min innactivity logout. • How many of students returned (false positive), and how many did not (true positive)? • Looking upon ITS research, provide a gold-standard data. • LMS plugin that through javascript keeps a track of user activity. Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 28 / 28
  • 58. References I Bloom, Benjamin S. (1974). “Time and learning”. In: American Psychologist 29.9, pp. 682–688. Carroll, Jb (1963). “A Model of School Learning”. English. In: Teachers College Record 64.8. WOS:A1963CAJ4400010, pp. 723–733. Chickering, Arthur W and Zelda F Gamson (1989). “Seven principles for good practice in undergraduate education”. In: Biochemical Education 17.3, pp. 140–141. Karweit, Nancy and Robert E. Slavin (1982). “Time-on-task: Issues of timing, sampling, and definition”. In: Journal of Educational Psychology 74.6, pp. 844–851. Truzzi, Marcello (1978). “On the Extraordinary: An Attempt at Clarification”. In: Zetetic Scholar 1.1.