Rethinking time-on-task estimation with outlier detection accounting for individual, time, and task differences Learning analytics conference 2020

1. Quan Nguyen
School of Information
University of Michigan
Rethinking time-on-task estimation with outlier detection
accounting for individual, time, and task differences
quanngu@umich.edu
@QuanNguyen3010

2. Log-data is every where
A typical log file
User ID Content URL Timestamp
111 Section 1.1 https://abc.com/1.1 2018-09-01 09:30:00
111 Section 1.2 https://abc.com/1.2 2018-09-01 09:32:00
111 Section 1.2 https://abc.com/1.2 2018-09-01 09:42:00
111 Section 1.3 https://abc.com/1.3 2018-09-01 09:52:00
112 Section 1.1 https://abc.com/1.1 2018-09-01 09:30:00
112 Section 1.1 https://abc.com/1.1 2018-09-01 23:30:00
…

3. Time-on-task estimation
Time-on-task calculated as the duration between two consecutive clicks
User ID Content URL Timestamp Time-on-task
111 Section 1.1 https://abc.com/1.1 2018-09-01 09:30:00 NA
111 Section 1.2 https://abc.com/1.2 2018-09-01 09:32:00 00:02:00
111 Section 1.2 https://abc.com/1.2 2018-09-01 09:42:00 00:10:00
111 Section 1.3 https://abc.com/1.3 2018-09-01 09:45:00 00:03:00
112 Section 1.1 https://abc.com/1.1 2018-09-01 09:30:00 NA
112 Section 1.1 https://abc.com/1.1 2018-09-01 23:30:00 14:00:00
…

4. Why do we care about time-on-task?
• A proxy of academic/tool ”engagement”
• Highly predictive of academic performance as shown in
many LA studies

5. What are some challenges with time-on-task?
• The time spent might not actually be ”on task”
• What is a “normal” time-on-task?
• How can detect anomalous time-on-task?
Click 1 Click 2

6. How are we handling outliers in time-on-task
• Most studies (including mine) use cut-off threshold (e.g. 15
minutes, 30 minutes)
• Because it’s easy to do
• Because many people do it
• And a round number (15, 30, 60) sounds nice…
But…
• Are all learning activities homogenous?
• Are all students homogenous?
• Are all learning phases homogenous?

7. How are we handling outliers in time-on-task
• Kovanovic et al. (2015) tested different outlier processing
strategies and suggested that the choice of outlier processing
approach could shape the research findings (e.g. predictive power
on academic performance)
• Is there a “correct” way to handle outliers in time-on-task?

8. What do we know about learning behaviors
1. Learning strategies: Given the same instructional condition, each
learner makes different choices about what, how, and when to study
2. Temporal aspects: Learning behavior does not stay constant throughout
a course but varied dynamically at different temporal phases
3. Learning design: The pedagogical design decision will affects how
students engage with each learning task

9. Outliers in time-on-task should be contextualized on
1. Individual: What is normal for each student?
2. Time: What is normal for that point in time (e.g.
beginning vs end of semester)?
3. Task: What is normal for that kind of task (e.g.
watching a 5’ video vs. reading a scientific article)?

10. Outlier detection
Outlier detection has a wide range of techniques and popular
applications in other fields (i.e. fraud detection, weather forecast,
cancer detection, etc.)
However, its application in educational research is still limited

11. Types of outliers
• Global outliers: Data points that fall far
outside of the entirety of the dataset
• Contextual outliers: Data points that deviate
from the rest of the data points within the
same context
• Collective outliers: Refer to a subset of data
points are not anomalous in either a global
or contextual sense but as a collection,
deviates from the entire data set
E.g. A 5ft tall (wo)man in a NBA team
E.g. A 8ft tall (wo)man
E.g. a NBA team

12. Research design
1. Collect log-data of a course in a distance learning institution
2. Compute time-on-task
3. Use different outlier detection methods
4. Aggregate time-on-task per student as a proxy of engagement
5. Correlate engagement as time-on-task with final grade in that course
6. Compare predictive power of different outlier detection methods

13. 1. Collect log-data
• A second-year course in a distance learning institution
• 273 enrolled students
• October 2015 to June 2016

14. 2. Compute time-on-task
• 451,979 time-on-task estimations.

15. 3. Outlier detection methods in the literature
• Simple: Use a cut-off threshold
• Distribution based: Inter-quartile range, +- 2 SD from the mean, z-scores
• A wide range of machine learning methods:
• Distance based:
 Local Outlier Factor (LOF)
 KNN Angle-based Outlier Detection (ABOD)
• Density based
 Robust Kernel Density Estimation (rKDE)
 Ensemble Gaussian Mixture Model (EGMM)
• Decision tree
 Isolation forest

16. 3. Outlier detection methods used in this study
Method Description
60m Cut off at 60 minutes
30m Cut off at 30 minutes
10m Cut of at 10 minutes
IQR Interquartile range
IQR_ITT Interquartile range accounting for individual, time,
and task differences
iForest Isolation forest
iForest_ITT Isolation forest accounting for individual, time, and
task differences

17. Isolation forest explained
• Carry out random partitions (slices)
Normal will take more slices to
be isolated
Outliers will take less slices to be
isolated

18. Account for individual, time, task differences
Xijt is the time-on-task of individual i, in week j, on task t
Compute a threshold S using an outlier detection method
• For each student Si
• For each study week Sj
• For each task St
If a time-on-task duration exceed all three thresholds then it
is an outlier
Xijt is an outlier if
𝑋𝑖𝑗𝑡 > 𝑆𝑖
𝑋𝑖𝑗𝑡 > 𝑆𝑗
𝑋𝑖𝑗𝑡 > 𝑆𝑡

19. Compare performances of outlier detection methods
A unsupervised problem: No ”ground truth” information (i.e.
we could not see what students were doing).
Assumption: If time-on-task is correlated with academic
performance, then a ”good” outlier detection methods might
reduce noises in the predictive model  Higher R2

20. Results
Figure 1: Time-on-task estimation throughout a course timeline

21. Results
Figure 1: Outlier detection based on interquartile range accounting for individual, time, and task differences

22. Results
Academic
performance
Outlier detection methods
60m 30m 10m IQR IQR_ITT iForest iForest_ITT
Time-on-task .003*** .005*** .010*** .022*** .007*** .001*** .002***
(.0004) (.001) (.001) (.002) (.001) (.0001) (.0003)
Constant 39.153*** 38.440*** 37.655*** 37.939*** 37.612*** 45.378*** 44.616***
(2.283) (2.323) (2.410) (2.449) (2.314) (2.124) (2.095)
Observations 186 186 186 186 186 186 186
R2 .307 .313 .308 .294 .333 .197 .225
Adjusted R2 .303 .309 .305 .290 .329 .193 .221
Residual Std.
Error (df = 184)
19.272 19.196 19.255 19.458 18.916 20.742 20.381
F Statistic (df =
1; 184)
81.567*** 83.681*** 82.046*** 76.532*** 91.670*** 45.269*** 53.463***
Note: *p <0.05 **p < 0.01 ***p<0.001
Table 2: A comparison of performance of seven outlier detection methods

23. Conclusions
• A 3-4% increase in model performance measured in R2
when differences in individual, time, and task were
considered.
• There was not much difference in R2 between 10,30,60
minute thresholds
• Isolation forest (surprisingly) has the poorest performance

24. Limitations
• We don’t have a “ground truth” and probably never will
due to privacy issue
• The study was conducted in only one course, in a distance
learning institution
• There are many other outlier detection methods have not
been tested  A future work

25. Take-aways
• LA studies should explicitly report how they process outliers in
time-on-task estimations
• We should use what we know about learning context, learning
behavior, learning theories to contextualize outlier detection
methods
• What is a ”normal” duration depends on each individual
learning pattern, the nature of the task, the learning phase

26. Quan Nguyen
School of Information
University of Michigan
Rethinking time-on-task estimation with outlier detection
accounting for individual, time, and task differences
quanngu@umich.edu
@QuanNguyen3010