1) Learning analytics dashboards aim primarily to foster awareness and reflection in learners, but awareness alone is not enough to affect learning. 2) Dashboards currently provide limited support for self-regulated learning beyond reflection, neglecting goal setting and monitoring. 3) Dashboards commonly use social comparisons as the reference frame, which may be more suitable for performance-oriented learners and neglect mastery-oriented learners. Long term effects of social comparisons are unknown.

1. Awareness is not enough. Pitfalls of learning
analytics dashboards in the educational practice
Ioana Jivet, Maren Scheffel, Hendrik Drachsler, Marcus Specht
Welten Institute
Open University of the Netherlands
ioana.jivet@ou.nl

2. Source: https://www.flickr.com/photos/bruneluniversity/12926707974/

3. Learning analytics dashboards
… “single displays that aggregate different indicators about learner(s),
learning process(es) and/or learning context(s) into one or multiple
visualisations”
Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., ... & Dillenbourg,
P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE
Transactions on Learning Technologies, 10(1), 30-41.

4. Learning and analytics
Learning Analytics

5. Other learning analytics dashboard reviews
None investigated the connection to learning sciences
Rarely suggested using educational concepts in the design of dashboards
What is the theoretical foundation for the development of learning
dashboards aimed at learners?

6. Methodology
“learning analytics” AND (visualization OR visualisation OR dashboard OR widget)
=> 1439 papers
Inclusion criteria
1. In English
2. Fully developed dashboard
3. Dashboard for learners
4. Theoretical concepts
5. Evaluation
95

7. Methodology
“learning analytics” AND (visualization OR visualisation OR dashboard OR widget)
=> 1439 papers
Inclusion criteria
1. In English
2. Fully developed dashboard
3. Dashboard for learners
4. Theoretical concepts
5. Evaluation
26
Papers available at: bit.ly/LADashboards

8. Results – Learning theories and models

9. Results – Dashboard goals and educational concepts

10. Results – Dashboard goals and educational concepts

11. Results – Reference frames

12. Results – Reference frames

13. Results – Reference frames

14. Insights – Dashboard goals
Most common goal: fostering awareness and reflection
Issues:
– being aware = taking action
– How to objectively measure awareness and reflection?
1. LA dashboards should aim to affect cognitive, behavioural and
emotional competencies alongside metacognitive ones.

15. Insights – Support for self-regulated learning
Most common educational concept: self-regulated learning
Issues:
– Rare support for managing learner set goals
– Occasional possibility to track progress
2. Current dashboards are built to support the “reflection and self-
evaluation” phase of SRL and neglect the others.

16. Insights – Reference frames
Most common reference frame: social
Theories that justify the use of social framing:
– Social comparison theory (Festinger, 1954)
– Achievement goal orientation (Elliot & McGregor, 2001)
3. Current dashboards are more suitable for performance oriented
learners, neglecting learners that have mastery preferences.
What happens on the long-term?

17. Conclusions
• The use of educational concepts in dashboard design and evaluation is not
that common (26/95).
• SRL is the most common theory, yet there rarely is support for goal setting
and monitoring.
• Just making learners aware is not enough.
• Currently, there is a strong emphasis on comparison with peers.
Do we have a pedagogical reason for doing so
or are we doing it just because we have the data available?