The document summarizes a study that examined how self-regulated learning (SRL) skills relate to students' sense-making of learning analytics dashboards. The study found that two factors supported sense-making: transparency of dashboard design and support for action. SRL skills, particularly goal setting and help-seeking, predicted how relevant students found these sense-making factors. The study concluded dashboards should recognize different SRL levels, scaffold development of SRL skills, have transparent designs, and enable students to take immediate action.

1. Self-regulated learning skills matter in
dashboard sense-making
Ioana Jivet, Marcel Schmitz, Maren Scheffel, Stefan Robbers, Marcus Specht, Hendrik Drachsler
ioana.jivet@ou.nl
@IoanaJJ

2. Source: https://www.flickr.com/photos/bruneluniversity/12926707974/

6. Learning analytics dashboards
… “single displays that aggregate different indicators about learner(s),
learning process(es) and/or learning context(s) into one or multiple
visualisations”
Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg, P. (2017). Perceiving learning
at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41.

7. Shortcomings
Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational
Practice. In E. Lavoué, H. Drachsler, K. Verbert, J. Broisin, & M. Pérez-Sanagustín (Eds.), Data Driven Approaches in Digital Education. EC-TEL
2017. Lecture Notes in Computer Science, vol 10474 (pp. 82-96). Springer, Cham.
Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational
practice. In Proc. of International Conference on Learning Analytics and Knowledge, Sydney, NSW, Australia, March 7–9, 2018 (LAK’18)
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional
conditions in predicting academic success. The Internet and Higher Education, 28, 68-84.

8. Learning analytics process model - Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L.
(2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500-1509.
How to make sense of learning dashboards

9. RQ1: What dashboard features support sense-making?
RQ2: Are learner goals related to the perceived relevance of these
features?
Mastering the topic of the course vs. passing the course
RQ3: Are SRL skills related to the perceived relevance of these features?

10. Methodology

11. Methodology
Qualitative pre-study
23 students in several TLA courses
26 dashboard
design elements

12. “It is also interesting to see the ‘predictors’
which could act as ‘alerts’ to guide me.”

13. “It is also interesting to see the ‘predictors’
which could act as ‘alerts’ to guide me.”
● "predictions of learning behaviour"
● "areas in need of improvement highlighted"

14. Methodology
Qualitative pre-study
23 students in several TLA courses
26 dashboard
design elements
Quantitative main study
179 students
26 dashboard
design elements
SRL skills (OSLQ)
Learner goals:
A. mastering the topic
B. passing the course
L. Barnard, W. Y. Lan, Y. M. To, V. O. Paton, and S.-L. Lai, “Measuring self-regulation in online and
blended learning environments,” Internet High. Educ., vol. 12, no. 1, pp. 1–6, Jan. 2009

15. Main study Priming

16. Results

17. RQ1: What dashboard features are relevant for
sense-making?

18. Sense-making factors - Exploratory factor analysis
25%

19. RQ2 + RQ3: Are learner goals and SRL related to the
perceived relevance of these factors?

20. Learner goals, SRL and sense-making
Results of multiple regression analyses with learner goal and the combined SRL score as predictors and the two sense-making factors as
output variables.

21. Learner goals, SRL and sense-making
Results of multiple regression analyses with learner goal and the combined SRL score as predictors and the two sense-making factors as
output variables.

22. Learner goals, SRL and sense-making
Results of multiple regression analyses with learner goal and the combined SRL score as predictors and the two sense-making factors as
output variables.

23. SRL: self-evaluation
SRL subskills and sense-making
SRL: goal setting
SRL: time management
SRL: help-seeking
SRL: self-evaluation
SRL
MANCOVA (Wilks’ Lambda)
Independent variables: learner goal
Covariates: 4 SRL subscales
Dependent variables: 2 sense-making factors

24. SRL subskills and sense-making
SRL: goal setting
F(2,166) = 3.311, p=.039,
Wilks' Λ = .962
SRL: time management
SRL: help-seeking
F(2,166) = 3.752, p=.025,
Wilks' Λ = .957
SRL: self-evaluation
SRL
MANCOVA (Wilks’ Lambda)
Independent variables: learner goal
Covariates: 4 SRL subscales
Dependent variables: 2 sense-making factors

25. SRL subskills and sense-making
SRL: goal setting
SRL: time
management
SRL: help-seeking
SRL: self-evaluation
SRL
Transparency of design
Support for action
F(1) = 6.078, p = .015
F(1) = 7.260, p = .008

26. Conclusion

27. 2 features that support sense-making:
● Transparency of design
● Support for action
SRL is a predictor for how relevant both sense-making factors are perceived.
Goal setting and help-seeking skills are positively associated with the perceived
relevance of the two sense-making factors.
Results - summary

28. Bodily, R., & Verbert, K. (2017, March). Trends and issues in student-facing learning analytics reporting systems
research. In Proceedings of the seventh international learning analytics & knowledge conference (pp. 309-318).
ACM.
1. Recognise and cater to the different needs of students with different SRL levels.
2. Sense-making of students with high SRL skills could be modelled and used to
scaffold the development of novice SRL learners.
3. Dashboard designs should be transparent.
4. Dashboards should integrate support for action that enables students to take
immediate action.
Implications for dashboard design in HE
17%!

29. Self-regulated learning skills matter in
dashboard sense-making
Ioana Jivet, Marcel Schmitz, Maren Scheffel, Stefan Robbers, Marcus Specht, Hendrik Drachsler
ioana.jivet@ou.nl
@IoanaJJ