Learning Dashboards for Feedback at Scale

1. Learning Dashboards for feedback at scale Tinne De Laet Tinne.DeLaet@kuleuven.be @TinneDeLaet

2. largest university in Belgium, founded 1425 16 faculties → general university  55 000 students no national exam secondary schools organize own independent exams low registration fees €922,3 typically regular full-time students, 1 year no selection allowed have to except all students with secondary education diploma (except Medicine, Dentistry & Performing Arts)

3. dropoutongoing study duration How to improve student success??

4. Focus of our research in Learning Analytics 4 actionable feedback student-centered program level inclusive first-year experience institution-wide Learning Analytics actual implementation dashboards

5. positioning test started in 2013 multiple-choice test on mathematical problem solving non-obligatory and non-binding research skills and competencies underlying engineering student success not only math matters Pinxten, M., Van Soom, C., Peeters, C. et al.; At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders - any incremental validity of study strategies?; Eur J Psychol Educ (2019) 34: 45. https://doi.org/10.1007/s10212-017-0361-x European readySTEMgo project, Early identification of at-risk students in STEM; https://iiw.kuleuven.be/english/readystemgo prior academic achievement advice of teacher board of secondary education mathematical skills learning and studying skills gender socio-economic status

6. 6 [!] Feedback must be “actionable”. Warning! Male are 10% less likely to be successful. You are male. Warning! Your online activity is lagging behind. action? ? action? ? 

7. 7 [!] Start with the available data. Lots of data may eventually become available in the future … …. already start with what is available (*) (*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication. British Journal of Educational Technology, 44(4), 616-628.

8. 8 [!] Not all data is usable. example data from a traditional course with “VLE as a file system” test scores activity/week (#days) weeks of the year

9. 9 [!] Not all data is usable. example data from a course with flipped classroom & blended learning exam scores activity (# of modules used) Not a single student using less than 10 modules passed the course. Most of the successful students used 15 modules or more.

10. 10 learning dashboards @KU Leuven interaction self-reflection LISSA STUDENT ADVISOR STUDENT LASSI – learning skills REX - scoresPOS – future students 12.450 unique students reached!

11. 11 Student-facing learning dashboards

12. 12 [!] Think beyond the obvious data. • Don’t think too traditional. • Many institutions are collecting survey data for educational research. LASSI questionnaire • motivation • concentration • (lack of) failure anxiety • time management • use of test strategies Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017). readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo

13. 13 Does my concentration matter? How is my time management? I feel uncertain. Is this normal? How can I improve my concentration?

14. 14 Dashboard learning skills students complete LASSI questionnaire students receive personalized email with invitation for dashboard demo: https://learninganalytics.set.kuleuven.be/static-demo-lassi/ 4367 students in 26 programs in 9 faculties @KU Leuven 4 programs @TU Delft

15. 15 3. How does this relates to others? 2. How am I doing? 1. What is this about? @studyProgram@ @yourScore@

16. 16 4. Why is this relevant? 5. What can I do about it?

17. 17 5. What can I do about it?

18. 18 Response 3868 students (89%) used dashboard

19. 19 Student feedback? http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/ How CLEAR is this info? stars stars

20. 20 Students that click through Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.  better learning skills

21. 21 More intense users Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.  worse learning skills

22. 22 What can we learn from dashboard usage? Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018 wavg = β0 + β1 ∗ dbuser + β2 ∗ math.hrs + β3 ∗ math.score + β4 ∗ physics + β5∗chemistry+ β6∗biology+β7∗mot+β8∗tmt+β9∗anx+β10∗tst+β11∗con+β12∗advice

23. 23 [!] Be careful with predictive algorithms. http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/ • reality is complex • measurement is limited • individual circumstances • need for nuance • trigger reflection

24. 24 [!] Be careful with predictive algorithms. http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/ predicting GPA of an individual student Local Interpretable Model-agnostic Explanations (LIME)

25. 25 [!] Acceptance precedes impact. • Involve stakeholders from the start and value their input! COmmunication COoperation • Demonstrate usefulness. • Take care of ethics and privacy. • Best scenario: students & study advisors as ambassadors COCO

26. 26 Dashboard academic achievement additional feedback on academic achievement students receive personalized email with invitation for dashboard demo: https://learninganalytics.set.kuleuven.be/static-demo-rex/ >12.000 students in 26 programs in 9 faculties @KU Leuven

27. 27 Impact? survey before intervention  2nd year students 2016-2017  experiences first-year feedback  41 vragen, 5-point Likert scale  pen & paper dashboards  LISSA  LASSI (learning skills)  3 x REX (grades) Survey after intervention  2nd year students 2017-2018 Under review by Assessment in Higher Education Journal

28. 28 Impact? During the first year I received sufficient information regarding my academic achievements. Engineering Science (p<0.001) Under review by Assessment in Higher Education Journal The information I received helped to position myself with respect to my peers. Engineering Science (p<0.001)

29. 29 Impact? The information I received made me reflect. The information I received made me adapt my behaviour. Under review by Assessment in Higher Education Journal

30. 30 [!] Context matters! • available data • national and institutional regulations and culture • educational vision • educational system, size of population .. • … Don’t just copy existing solutions!

31. 31 Future? Continue and extend dashboards @KU Leuven using scale-up project Transfer to other universities LALA project! new horizons ….

32. 32 Conclusion learning dashboards for feedback at scale supporting transition from SE to HE  actionable feedback using learning dashboards  humble but scalable approach  traditional university settings  involvement of stakeholders, especially practitioners  learning dashboard create useful new learning traces!

33. 33 Project team @ Sven Charleer AugmentHCI, Computer Science department PhD researcher ABLE Katrien Verbert (professor) AugmentHCI, Computer Science department Copromotor of STELA & ABLE Carolien Van Soom (professor) Leuven Engineering and Science Education Center Head of Tutorial Services of Science Copromotor of STELA & ABLE Greet Langie Leuven Engineering and Science Education Center Vicedean (education) faculty of Engineering Technology Promotor of readySTEMgo, copromotor of STELA & ABLE Tinne De Laet (professor) Leuven Engineering and Science Education Center Head of Tutorial Services of Engineering Science Coordinator of STELA and ABLE Copromotor of readySTEMgo Francisco Gutiérrez AugmentHCI, Computer Science department PhD researcher ABLE Tom Broos Leuven Engineering and Science Education Center AugmentHCI, Computer Science department PhD researcher STELA Martijn Millecamp AugmentHCI, Computer Science department PhD researcher ABLE Special thanks to study advisors for their cooperation, advice, feedback, and support! ♥ Maarten Pinxten (post-doc) Leuven Engineering and Science Education Center Head of Tutorial Services of Science Copromotor of readySTEMgo