Learning analytics is hot. But are learning dashboards scalable and sustainable solutions for providing actionable feedback to students? Can learning dashboard be applied for feedback at scale? Is learning analytics applicable in more traditional higher education settings? This talk will share experiences and lessons learned from three European projects (STELA, ABLE, and LALA ) that focuses on scalable applications of learning dashboards and their integration within actual educational practices. Can learning dashboards deployed at scale, create new learning traces? This talk shares experiences of a large scale deployment of learning dashboards with more than 12.000 students. Presented at laffas.eu.

1. Learning Dashboards for feedback at scale
Tinne De Laet
Tinne.DeLaet@kuleuven.be
@TinneDeLaet

2. largest university in Belgium, founded 1425
16 faculties → general university
 55 000 students
no national exam
secondary schools organize
own independent exams
low registration fees
€922,3
typically regular full-time
students, 1 year
no selection allowed
have to except all students with
secondary education diploma
(except Medicine, Dentistry & Performing Arts)

3. dropoutongoing
study duration
How to improve student success??

4. Focus of our research in
Learning Analytics
4
actionable feedback
student-centered
program level
inclusive
first-year experience
institution-wide
Learning Analytics
actual implementation
dashboards

5. positioning test
started in 2013
multiple-choice test on
mathematical problem solving
non-obligatory and non-binding
research
skills and competencies underlying
engineering student success
not only math matters
Pinxten, M., Van Soom, C., Peeters, C. et al.; At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders - any incremental validity of
study strategies?; Eur J Psychol Educ (2019) 34: 45. https://doi.org/10.1007/s10212-017-0361-x
European readySTEMgo project, Early identification of at-risk students in STEM; https://iiw.kuleuven.be/english/readystemgo
prior academic achievement
advice of teacher board of secondary
education
mathematical skills
learning and studying skills
gender
socio-economic status

6. 6
[!] Feedback must be “actionable”.
Warning!
Male are 10% less
likely to be successful.
You are male.
Warning!
Your online activity is
lagging behind.
action?
?
action?
?


7. 7
[!] Start with the available data.
Lots of data may eventually become
available in the future …
…. already start with what is available
(*)
(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.
British Journal of Educational Technology, 44(4), 616-628.

8. 8
[!] Not all data is usable.
example data from a traditional course with “VLE as a file system”
test scores
activity/week (#days)
weeks of the year

9. 9
[!] Not all data is usable.
example data from a course with flipped classroom & blended learning
exam scores
activity (# of modules used)
Not a single student
using less than 10
modules passed the
course.
Most of the successful
students used 15
modules or more.

10. 10
learning dashboards @KU Leuven
interaction
self-reflection
LISSA
STUDENT
ADVISOR
STUDENT LASSI –
learning skills
REX - scoresPOS – future
students
12.450 unique students
reached!

11. 11
Student-facing learning
dashboards

12. 12
[!] Think beyond the obvious data.
• Don’t think too traditional.
• Many institutions are collecting survey
data for educational research.
LASSI questionnaire
• motivation
• concentration
• (lack of) failure anxiety
• time management
• use of test strategies
Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an
open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017).
readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo

13. 13
Does my concentration
matter?
How is my time
management?
I feel uncertain.
Is this normal?
How can I improve
my concentration?

14. 14
Dashboard learning skills
students complete LASSI
questionnaire
students receive personalized email
with invitation for dashboard
demo:
https://learninganalytics.set.kuleuven.be/static-demo-lassi/
4367 students in 26 programs
in 9 faculties @KU Leuven
4 programs @TU Delft

15. 15
3. How does this relates to
others?
2. How am I doing?
1. What is this about?
@studyProgram@
@yourScore@

16. 16
4. Why is this relevant?
5. What can I do about it?

17. 17
5. What can I do about it?

18. 18
Response
3868 students (89%)
used dashboard

19. 19
Student feedback?
http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/
How CLEAR is this info?
stars stars

20. 20
Students that click through
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
 better learning skills

21. 21
More intense users
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
 worse learning skills

22. 22
What can we learn from dashboard usage?
Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and
Microinteractions. accepted for ECTEL 2018
wavg = β0 + β1 ∗ dbuser + β2 ∗ math.hrs + β3 ∗ math.score + β4 ∗ physics + β5∗chemistry+
β6∗biology+β7∗mot+β8∗tmt+β9∗anx+β10∗tst+β11∗con+β12∗advice

23. 23
[!] Be careful with predictive
algorithms.
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection

24. 24
[!] Be careful with predictive
algorithms.
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
predicting GPA
of an individual student
Local Interpretable
Model-agnostic Explanations
(LIME)

25. 25
[!] Acceptance precedes impact.
• Involve stakeholders from the start and
value their input!
COmmunication
COoperation
• Demonstrate usefulness.
• Take care of ethics and privacy.
• Best scenario:
students & study advisors as ambassadors
COCO

26. 26
Dashboard academic achievement
additional feedback on academic
achievement
students receive personalized email
with invitation for dashboard
demo:
https://learninganalytics.set.kuleuven.be/static-demo-rex/
>12.000 students in 26 programs
in 9 faculties @KU Leuven

27. 27
Impact?
survey before intervention
 2nd year students 2016-2017
 experiences first-year feedback
 41 vragen, 5-point Likert scale
 pen & paper
dashboards
 LISSA
 LASSI (learning skills)
 3 x REX (grades)
Survey after intervention
 2nd year students 2017-2018
Under review by Assessment in Higher Education Journal

28. 28
Impact?
During the first year I received sufficient information regarding my academic achievements.
Engineering Science (p<0.001)
Under review by Assessment in Higher Education Journal
The information I received helped to position myself with respect to my peers.
Engineering Science (p<0.001)

29. 29
Impact?
The information I received made me reflect.
The information I received made me adapt my behaviour.
Under review by Assessment in Higher Education Journal

30. 30
[!] Context matters!
• available data
• national and institutional regulations
and culture
• educational vision
• educational system, size of
population ..
• …
Don’t just copy existing solutions!

31. 31
Future?
Continue and extend dashboards
@KU Leuven
using scale-up project
Transfer to other universities
LALA project!
new horizons ….

32. 32
Conclusion
learning dashboards for feedback at scale
supporting transition from SE to HE
 actionable feedback using learning dashboards
 humble but scalable approach
 traditional university settings
 involvement of stakeholders, especially practitioners
 learning dashboard create useful new learning traces!

33. 33
Project team @
Sven Charleer
AugmentHCI, Computer Science department
PhD researcher ABLE
Katrien Verbert (professor)
AugmentHCI, Computer Science department
Copromotor of STELA & ABLE
Carolien Van Soom (professor)
Leuven Engineering and Science Education Center
Head of Tutorial Services of Science
Copromotor of STELA & ABLE
Greet Langie
Leuven Engineering and Science Education Center
Vicedean (education) faculty of Engineering Technology
Promotor of readySTEMgo, copromotor of STELA & ABLE
Tinne De Laet (professor)
Leuven Engineering and Science Education Center
Head of Tutorial Services of Engineering Science
Coordinator of STELA and ABLE
Copromotor of readySTEMgo
Francisco Gutiérrez
AugmentHCI, Computer Science department
PhD researcher ABLE
Tom Broos
Leuven Engineering and Science Education Center
AugmentHCI, Computer Science department
PhD researcher STELA
Martijn Millecamp
AugmentHCI, Computer Science department
PhD researcher ABLE
Special thanks to study advisors for their cooperation, advice, feedback, and support!
♥
Maarten Pinxten (post-doc)
Leuven Engineering and Science Education Center
Head of Tutorial Services of Science
Copromotor of readySTEMgo