The document discusses the role of technology and course design in enhancing learning strategies and student engagement, particularly in a flipped classroom setting for a computer systems course. It analyzes various learning patterns and strategies derived from student interactions with course materials, revealing distinctions between effective and ineffective approaches. Key findings emphasize the importance of formative assessments and the dynamic nature of learning strategies as influenced by course structure and instructional methods.

1. Using learning analytics to
uncover learning strategies
Dragan Gašević
@dgasevic
Shape of Educational Data Meeting
April 7, 2016, Fairfax, VA

2. Growing role of technology to
flip classrooms

3. Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active
learning increases student performance in science, engineering, and mathematics. Proceedings of the National
Academy of Sciences, 201319030.

4. How do students study with
technology?

5. BACKGROUND

6. Categorization
Deep and surface approaches to learning
Trigwell, K., & Prosser, M. (1991). Relating approaches to study and quality of learning outcomes at the course
level. British Journal of Educational Psychology, 61(3), 265-275.

7. Poor choices of
learning tactics and strategies
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual review
of psychology, 64, 417-444.

8. Significant role of instructions on
approaches to learning
Trigwell, K., Prosser, M., & Waterhouse, F. (1999). Relations between teachers’ approaches to teaching and students’
approaches to learning. Higher Education, 37(1), 57–70.

9. Role of course design
To prompt active engagement and
challenge higher order thinking
Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learning. Innovations in Education
and Teaching International, 44(4), 349–362.

10. Student profiling
Unsupervised approaches
Lust, G., Elen, J., & Clarebout, G. (2013). Students’ tool-use within a web enhanced course: Explanatory
mechanisms of students’ tool-use pattern. Computers in Human Behavior, 29(5).

11. Sequences of activities
Sequence or process mining, HMMs, etc.
Reimann, P., Markauskaite, L., Bannert, M. (2014). e-Research and learning theory: What do sequence and process
mining methods contribute? British Journal of Educational Technology, 45(3), 528-540.

12. What learning strategies do
students follow
while using technology?

13. Do learning strategies of students
change over time
while using technology?

14. FLIPPED CLASSROOM STUDY

15. Study context
Freshman course in computer systems at USyd
Enrolment: ~300 students
Assessment: midterm + final + project

16. Flipped learning design
Redesigned lecture – an active learning session
requiring students’ preparation

17. Flipped learning design
Videos with multiple-choice questions (MCQs)
Documents with embedded MCQs
Problem (exercise) sequences

18. Exploratory sequence analysis
[1] (CONTENT_ACCESS,3)
[2] (EXE_IN,3)-(EXE_CO,1)-(EXE_IN,1)-(EXE_CO,1)-(EXE_IN,2)
[3] (CONTENT_ACCESS,3)-(EXE_IN,4)
[4] (MC_EVAL,4)
[5] (EXE_IN,5)-(EXE_CO,1)-(EXE_IN,3)-(EXE_CO,1)-(EXE_IN,2)-
(EXE_CO,1)-(EXE_IN,9)-(EXE_CO,4)-(EXE_IN,4)-(EXE_CO,1)-
(EXE_IN,2)-(EXE_CO,2)-(EXE_IN,3)-(EXE_CO,3)-(EXE_IN,1)-
(EXE_CO,2)-(EXE_IN,1)
[6] (CONTENT_ACCESS,2)
Gabadinho, A., Ritschard, G., Müller, N.S. & Studer, M. (2011). Analyzing and visualizing state sequences in R with
TraMineR, Journal of Statistical Software, 40(4), 1-37.
Agglomerative hierarchical clustering of sequences based on
Ward’s algorithm and Levenshtein’s edit distance

19. Clusters of learning sequences
Pattern/strategy 1 (1354, 11.93%): focus on formative assessment, followed by
metacognitive evaluation activities
Pattern/strategy 2 (4736, 41.72%): focus on summative assessment with indicators of
trial-and-error learning

20. Clusters of learning sequences
Pattern/strategy 3 (3228, 28.44%): focus on reading lecture materials with tiny
fraction of formative assessment
Pattern/strategy 4 (2033, 17.91%): focus on the course videos, with not negligible
amount of formative assessment activities; small fraction of metacognitive
evaluation activities at the beginning of the learning sessions

21. Student clustering
based on sequence clusters
All the cluster pairs, except for the 1-2 pair, are significantly different (even after applying the FDR correction for
multiple testing) in terms of both midterm and final exam scores
Intensive/adaptive Strategic/effective Selective/efficiency Minimalist

22. Changes in learning strategy
Feature Feature description
MCQ.TOT.FACT Discretized count of completed formative assessment items (MCQs)
MCQ.PERC.CO.FACT Discretized percentage of correctly solved MCQs
EXC.TOT.FACT Discretized count of completed summative assessment items (exercises)
EXC.PERC.CO Discretized percentage of correctly solved exercises
VID.TOT.FACT Discretized count of play and pause video events
MCQ.SH.TOT.FACT Discretized count of requests for answers on formative MCQs
TG.DENS.FACT Discretized transition graph density
MC.EVAL.FACT Discretized count of dashboard and Hall of Fame views
CONTENT.ACCESS.FACT Discretize count of accesses to the lecture content pages

23. Changes in learning strategy
State Short description Correspondence to
sequence-based
student clusters
1 Low activity level; focus on lecture materials and
summative assessment
Minimalists
2 High activity level; students are engaged with all the
preparation activities and are experimenting with
different learning strategies
Intensive / adaptive
3 Disengaged -
4 Moderate activity level; similar to state 2 in term of
engagement and the diversity of learning strategies, but
with lower activity level
Strategic / effective
5 Focus on summative assessment; low engagement with
lecture materials and very rarely with the course videos;
skipping formative assessment
Selective / efficiency-
oriented

24. Changes in learning strategy

25. CONCLUSION

26. Learning approaches consistent
with the literature

27. Effective vs ineffective
learning strategies

28. Please, don’t confuse this with
learning styles
Pet peeve!

29. Students don’t follow
the same approach all the time

30. Process nature of learning
- beyond coding and counting -
van der Aalst, W. (2012). Process mining: Overview and opportunities. ACM Transactions on Management
Information Systems (TMIS), 3(2), 7.

31. Critical role of course design and
contextual variables

32. Critical role of course design

33. Critical role of course design

34. Trace data about
internal conditions needed

35. Thanks you!