Reading and assessment are elementary activities for knowl- edge acquisition in online learning. Assessments represented as quizzes can help learners to identify gaps in their knowledge and understanding, which they can then overcome by reading the corresponding text-based course material. Reversely, quizzes can be used to evaluate reading com- prehension. The predominantly self-regulated interaction of reading and quiz activities in learning systems used in higher education has been little studied. In this paper, we examine this interaction using scroll and log data from an online undergraduate course (N=142). By analyzing pro- cesses and sequential patterns in user sessions, we identified six session clusters for characteristic reading and quiz patterns potentially relevant for adaptive learning support. These clusters showed that individual user sessions included either mainly reading or quizzes, but rarely both.

1. Interactions of reading and assessment activities
Niels Seidel, Dennis Menze
27/07/2022
1

2. Agenda
Introduction
Data & Methods
Results
Discussion
Conclusion and Outlook
2

3. Introduction
3

4. Online Learning
4
• Reading and quizzes are fundamental activities in (online) learning
• Mutually connected:
• Reading for knowledge acquisition to answer quizzes
• Quizzes help to identify knowledge gaps and
measure reading comprehension
• Often embedded in printed textbooks
• ➔ Intelligent textbooks

5. Online Learning
5
• Reading and quizzes are fundamental activities in (online) learning
➢ But often separate in learning systems with modular design like Moodle

6. Motivation
6
• Interaction of reading and quiz activities in learning systems like Moodle
• Students can freely choose activities
• personalize learning paths over semester
• hardly been investigated with a Learning Analytics approach
• Log events
• Sequential pattern mining
• Process mining
(Hassani et al., 2019)

7. Research Question
7
• What sequential patterns can be identified in reading and quiz activities?
• Individual user sessions
• Clusters of frequent learning behaviors
➢ Insights about situations that may require an adaptive learning support
Related Work:
• Sequential patterns of reading and test behaviors → page turns, no scrolling (Sun et al., 2019)
• Transition diagrams of interactions with course material → students not grouped based on
behavioral similarities (Cheng et al., 2017)

8. Data & Methods
8

9. Participants and Design
9
• Compulsory course Operating Systems and Computer Networks of distance learning
B.Sc. Computer Science study program in the winter semester 2020/2021
• Supplementary course set up in Moodle
• Voluntary, additional learning opportunities
• N=142 (of 534 course participants)
• Age: 19-65 (M=37.21, SD=9.03), gender: 52 female, 128 male

10. Material
10
• Moodle course: four units including
• 42 self-assessment questions
and 23 multiple-choice questions
(= quizzes)
• course texts (~ 15.000 words each)
• newsgroup forum
• recordings of live sessions
• 30 assignments corrected by a tutor
• questions for exam preparation
Figure: Presentation of a course unit using the so called Longpage Moodle plugin

11. Longpage: Long reads for long semesters
- course text import from Word and LaTeX
- improved readability
- approximation of reading time
- visualization of individual reading progress
- recommendations of related course units
- individual text highlighting
- bookmarks and personal notes
- threaded discussions
- recommendations for discussion
APLE I APLE II
11
(Seidel et al., 2020; Rieger et al. 2019)

12. Data collection and preprocessing
12
• Sources:
• Moodle standard log store
• Scroll events from web browser (Intersection Observer API)
➢ ~ 240k log entries from reading and quiz activities
Raw data: Scroll events on single page of single user

13. Data collection and preprocessing
13
• Reading sessions categorized
based on tertiles
• Events derived from reading
sessions and timestamps of
quiz-related events
➢ ~ 1.4k individual user sessions
• Time-oriented heuristics:
consecutive reading / quiz events
< 45 minutes time difference

14. Mining processes and sequences
14
• Hypothesis-driven approach: finding study patterns by labeling sub-sequences
• Nominal features:
• Session starts with, ends with and/or mainly consists of reading or quiz activities
• Tertile of the length of the sequence of events per session: [1, 3] < (3, 7] < (7, 84]
• K-means for clustering user sessions by these features
• HeuristicsMiner for visualization of clusters
• PrefixSpan for mining sequential patterns
• Most frequent sequences using support measure

15. Results
15

16. Process mining
16
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

17. Process mining
17
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

18. Process mining
18
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

19. Process mining
19
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

20. Process mining
20
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

21. Process mining
21
Session cluster 1:
find preferred quizzes
Session cluster 3:
cancelling inappropriate quizzes
Session cluster 4:
intensive quiz sessions

22. Process mining
22
Session cluster 2: reading
sessions with single course unit
Session cluster 5: interactions
between reading and quiz activities
Session cluster 6: reading sessions
with multiple course units

23. Sequence mining
23

24. Discussion
24

25. Critical reflection of methods
25
• Other activities like newsgroup discussions, assignments, live sessions,
and self-regulated learning support ignored
• Only 1/3 of sessions and 50% participants with reading activities
• But: print and PDF versions available
• Session identification: time-oriented vs. navigation-oriented heuristics
• semantical interrelations between quizzes and course texts not considered

26. Critical reflection of methods
26
• Maybe relationship between behaviors and difficulty
• Quiz and text difficulty were not measured directly
• Preliminary analysis:
• Correlation between proportion of session clusters per course units and average
correctness of answers
➢ More difficult content → many different quizzes taken, but less mixing of reading and
quizzing in one session

27. Adaptive learning support
27
• prevalence of sessions that are less conducive to learning (e.g., SC1 or SC3)
• low variance in types of sessions (e.g. SC1–SC4, SC6),
thus solely reading or quiz activities
➢ countered by adaptive suggestions for appropriate learning strategies
and reflection of behavior

28. Conclusion and Outlook
28

29. Summary
29
• Six session clusters of reading and quiz activities:
• Mainly quiz (65.5%)
• Mainly reading (22.1%)
• Reading and quiz (12.4%)
• Strong interplay of reading and quizzes could not be confirmed

30. Outlook
30
• Replication of session clusters in following semester of same course
• Study of dependence between behaviors exhibited in session clusters
and difficulty of the material
• Study of correlations of found patterns with other factors like grades,
assignment results, course re-enrollment and dropout rate
• Prediction of session clusters during semester to implement interventions

31. 31
(Menze et al., subm.)
Reading and quiz activities over time
Goal: Observe changes of reading and quiz activities over time.
Data
- N=142, B.Sc. CS course, 1359 user sessions
Method
- periods P1-P6 a’ ~ 1 month
- classified user sessions into mainly reading, quiz, and both
- kmeans clustering
Results
- temporal patterns:
- continuous learning (C1, C4)
- only one (C5) or two periods (C3, C6, C7)
- early dropout (C3, C5, C6)
- activity breaks (C2, C3, C5, C7)
- delayed start (C2, partly C3)
- predominant activities:
- quiz (C4, C5), reading (C6), both (C1)
- transition reading > quiz (C3)
- transition quiz > reading and quiz (C1)
Next steps
- Consider results in learner model to contextualize prompts

32. Thank you for your attention!
32