Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scalable, Actionable, and Ethical Learning Dashboards: a reality check

32 views

Published on

Keynote presentation at Edmedia 2018 conference: https://www.aace.org/conf/edmedia/speakers/.
Results of Erasmus+ projects ABLE (www.ableproject.eu) and STELA (www.stela-project.eu) on learning dashboards for supporting first-year students.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Scalable, Actionable, and Ethical Learning Dashboards: a reality check

  1. 1. https://mfeldstein.com/purdue-course-signals-data-issue-
  2. 2. Scalable, Actionable, and Ethical Learning Dashboards a reality check Tinne De Laet Tinne.DeLaet@kuleuven.be @TinneDeLaet
  3. 3. Who am I? Why I am here? woman phdrobotics Head Tutorial Services Engineering Science KU Leuven . associate professor mech.engine er
  4. 4. founded 1425 research-intensive university 16 faculties → general university > 55 000 students 2016-2017-2018 Europe’s most innovative university (Reuters ranking)
  5. 5. “Learning analytics is about collecting traces that learners leave behind and using those traces to improve learning.” - Erik Duval Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 5 Learning Analytics?
  6. 6. Learning Dashboards? 6Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004 “A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.” - Stephen Few
  7. 7. Successful Transition from secondary to higher Education using Learning Analytics enhance a successful transition from secondary to higher education by means of learning analytics  design and build analytics dashboards,  dashboards that go beyond identifying at-risk students, allowing actionable feedback for all students on a large scale. Achieving Benefits from Learning Analytics research strategies and practices for using learning analytics to support students during their first year at university  developing the technological aspects of learning analytics,  focuses on how learning analytics can be used to support students. 7 www.stela-project.eu @STELA_project 2015-1-UK01-KA203-013767 www.ableproject.eu @ABLE_project_eu 562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD
  8. 8. STELA ♥ ABLE 8 actionable feedback student-centered program level inclusive first-year experience institution-wide Learning Analytics actual implementation
  9. 9. [!] Feedback must be “actionable”. 9 Warning! Male students have 10% less probability to be successful. You are male. Warning! Your online activity is lagging behind. action? ? action? ? 
  10. 10. 10 awareness (self-)reflection sensemaking impact data questions answers behavior change new meaning Verbert K, Duval E, Klerkx J; Govaerts S, Santos JL (2013) Learning analytics dashboard applications. American Behavioural Scientist, 10 pages. Published online February 2013. [!] Feedback must be “actionable”.
  11. 11. learning dashboards @KU Leuven interaction self-reflection LISSA STUDENT ADVISOR STUDENT LASSI – learning skills REX - scoresPOS – future students
  12. 12. [!] Start with the available data. Lots of data may eventually become available in the future … …. already start with what is available 12 (*) (*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication. British Journal of Educational Technology, 44(4), 616-628.
  13. 13. Case study dashboard interaction student – study advisor
  14. 14. Study advisor – student conversations 14 Should I consider another program? Can I still finish the bachelor in 3 years? How should I compose my program for next year? What is the personal situation? How can I help? What is the best next step?
  15. 15. [!] Use all available expertise. 15 visualization experts practitioners / end-users researchers LA researchers first-year study success Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).
  16. 16. LISSA dashboard https://able.cs.kuleuven.be/demo-september/2016/2
  17. 17. [!] Wording matters. 17 73% chance of success 73% of students of earlier cohorts with the same study efficiency obtained the bachelor degree http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
  18. 18. LISSA: status 18 26 programs >4500 students 114 student advisors training of study advisors • Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology • http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/ • Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student dialogues. Proceedings of the 8th International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM. dashboards for three examination periods observations, interviews, questionnaires
  19. 19. LISSA: evaluation – observations 19 15 observations insights (-) factual (+) interpretative (!) reflective Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
  20. 20. Evaluation – interviews “When students see the numbers, they are surprised, but now they believe me. Before, I used my gut feeling, now I feel more certain of what I say as well”. “It’s like a main thread guiding the conversation.” “I can talk about what to do with the results, instead of each time looking for the data and puzzling it together.” “Students don’t know where to look during the conversation, and avoid eye contact. The dashboard provides them a point of focus”. “A student changed her study method in June and could now see it paid off.” LISSA supports a personal dialogue.  the level of usage depends on the experience and style of the study advisors  fact-based evidence at the side  narrative thread  key moments and student path help to reconstruct personal track “I can focus on the student’s personal path, rather than on the facts.” “Now, I can blame the dashboard and focus on collaboratively looking for the next step to take.” 20
  21. 21. LISSA: evaluation – student questionnaires 21 26 programs @KU Leuven 291 student questionnaires first examination period “Confronting, but useful” “I want to use this dashboard at home.” “Also show the sub-grades for labs, … ” “How can I know the data is trustworth?” “Can’t these visualizations be send to students?” “Crisp and clear.”
  22. 22. 22 0 0 1 1 1 1 4 2 1 4 4 3 29 21 36 37 49 42 176 112 156 132 141 169 80 155 93 116 92 72 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1. The dashboard is clarifying and surveyable. 2. The shown information regarding my study situation is correct. 3. The shown position with respect to my fellow students (histograms per exam and global… 4. A conversation with my student advisors helped me to gain insight in my study trajectory. 5. The visualisation is of added value to the conversation with the student advisor. 6. The shown information provide me insight in my current situation. Student questionnaire January 2018 (N=291) Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
  23. 23. [!] Do not oversimplify. Show uncertainty. 23 • reality is complex • measurement is limited • individual circumstances • need for nuance • trigger reflection http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
  24. 24. [!] Be careful with predictive algorithms. 24 http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/ • reality is complex • measurement is limited • individual circumstances • need for nuance • trigger reflection
  25. 25. Case study student-facing dashboards
  26. 26. [!] Start with the available data. 26 data already available? administrative (examples) student records course grades systems (examples) LMS access logs advisor meetings ) Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher education. Journal of Research in Innovative Teaching & Learning, , 1-14. demo: https://learninganalytics.set.kuleuven.be/static-demo-rex/ (en) or https://learninganalytics.set.kuleuven.be/demo/rex-1718jan-ir (nl)
  27. 27. [!] Think beyond the obvious data. 27 • Don’t think too traditional. • Many institutions are collecting survey data for educational research.
  28. 28. [!] Not all data is usable. 28 example data from a traditional course with “VLE as a file system” test scores activity/week (#days) weeks of the year
  29. 29. [!] Not all data is usable. 29 example data from a course with flipped classroom & blended learning exam scores activity (# of modules used) Not a single student using less than 10 modules passed the course. Most of the successful students used 15 modules or more.
  30. 30. [!] Keep Learning Analytics in mind when designing learning activities. 30 Learning Analytics Learning Design INFORM ENABLE If LA indeed contributes to improved learning design… … don’t make it an afterthought
  31. 31. 31 Does my concentration matter? How is my time management? I feel uncertain. Is this normal? How can I improve my concentration?
  32. 32. data already available? administrative (examples) student records course grades [!] Think beyond the obvious data. 32 systems (examples) LMS access logs advisor meetings surveys (examples) quality insurance LASSI
  33. 33. ~ 30 LASSI questions (shortened version) “Learning Skills” Example: When preparing for an exam, I create questions that I think might be included. Example: I find it difficult to maintain my concentration while doing my coursework. Example: I find it hard to stick to a study schedule. raw scores (selected 5 out of 10) CONCENTRATION MOTIVATION FAILURE ANXIETY TEST STRATEGY TIME MANAGEMENT norm scores (in Flemish HE context) Example: STRONG Example: AVERAGE Example: LOW Example: VERY STRONG Example: VERY WEAK 33  Meta cognitive abilities Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017). readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo
  34. 34. Dashboard learning skills 34 students complete LASSI questionnaire students received personalized email with invitation for dashboard 4367 students in 26 programs in 9 faculties @KU Leuven demo: https://learninganalytics.set.kuleuven.be/static-demo-lassi/ (en) of https://learninganalytics.set.kuleuven.be/demo/lassi-1718 (nl) 2 programs @TU Delft
  35. 35. Feedback model 1. What is this about? 2. How am I doing? 3. How does this relates to others? 4. Why is this relevant? 5. What can I do about it? 35
  36. 36. 36 3. How does this relates to others? 2. How am I doing? 1. What is this about? @studyProgram@ @yourScore@
  37. 37. 4. Why is this relevant? 5. What can I do about it? 37
  38. 38. 38 5. What can I do about it?
  39. 39. Response 39 3868 students (89%) used dashboard
  40. 40. Student feedback? 40 http://blog.associatie.kuleuven.be/tinnedelaet/learning-dashboard-for-actionable-feedback-on-learning-and-studying-skills/ How CLEAR is this info? stars stars
  41. 41. Students that click through Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham. 41  better learning skills
  42. 42. More intense users Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness. In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham. 42  worse learning skills
  43. 43. What can we learn from dashboard usage? 43Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018 p<1e-5 p<1e-9p-test
  44. 44. [!] Give students “the key”. 44 • Student has the key to own data. • Student takes initiative to share/discuss own data. • GDPR as opportunity!
  45. 45. [!] Acceptance precedes impact. 45 • Involve stakeholders from the start and value their input! COmmunication COoperation • Demonstrate usefulness. • Take care of ethics and privacy. • Best scenario: students & study advisors as ambassadors COCO
  46. 46. Impact? survey before intervention  2nd year students 2016-2017  experiences first-year feedback  41 vragen, 5-point Likert scale  pen & paper dashboards  LISSA  LASSI (learning skills)  3 x REX (grades) Survey after intervention  2nd year students 2017-2018
  47. 47. Impact? During the first year I received sufficient information regarding my academic achievements. 47 Engineering Science (p<0.001)
  48. 48. Impact? The information I received helped to position myself with respect to my peers. 48 Engineering Science (p<0.001)
  49. 49. Impact? 49 The information I received made me reflect. The information I received made me adapt my behaviour.
  50. 50. [!] Context matters! • available data • national and institutional regulations and culture • educational vision • educational system, size of population .. • … Don’t just copy existing LA solutions! 50
  51. 51. Summary case studies 11 findings/recommendations [!] Use all available expertise. [!] Start with the available data. [!] Look beyond the obvious data. [!] Not all data is usable. [!] Wording matters. [!] Don’t oversimplify. Show uncertainty. [!] Beware of predictive algorithms. [!] Keep Learning Analytics in mind when designing learning activities. [!] Give students “the key” to their data. [!] Acceptance precedes impact. [!] Context matters!  humble approach  small data  involvement of stakeholders, especially practitioners  actionable feedback  scalability  traditional university settings Is this Learning Analytics?
  52. 52. Future? 52 Continue and extend dashboards @KU Leuven? Transfer to other universities? extension?
  53. 53. Project team @ 53 Sven Charleer AugmentHCI, Computer Science department PhD researcher ABLE Katrien Verbert AugmentHCI, Computer Science department Copromotor of STELA & ABLE Carolien Van Soom Leuven Engineering and Science Education Center Head of Tutorial Services of Science Copromotor of STELA & ABLE Greet Langie Leuven Engineering and Science Education Center Vicedean (education) faculty of Engineering Technology Copromotor of STELA & ABLE Tinne De Laet Leuven Engineering and Science Education Center Head of Tutorial Services of Engineering Science Coordinator of STELA KU Leuven coordinator of ABLE Francisco Gutiérrez AugmentHCI, Computer Science department PhD researcher ABLE Tom Broos Leuven Engineering and Science Education Center AugmentHCI, Computer Science department PhD researcher STELA Martijn Millecamp AugmentHCI, Computer Science department PhD researcher ABLE Special thanks to study advisors for their cooperation, advice, feedback, and support! Jasper, Bart, Riet, Hilde, An, Katrien, … ♥
  54. 54. Response - REX 54 571 76 245 220 353 258 274 1060 187 283 843 242 163 38 130 120 200 151 162 647 128 230 771 269 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% clicked non clicked
  55. 55. Evaluation - REX 55 22 26 105 443 511 0 100 200 300 400 500 600 1 2 3 4 5 useful (stars) 26 29 134 441 429 0 100 200 300 400 500 600 1 2 3 4 5 clear (stars) 41 60 220 385 456 0 100 200 300 400 500 600 1 2 3 4 5 added value (stars)
  56. 56. Micro-interactions - REX 56 defaultalternative ) Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018
  57. 57. Micro-interactions 57Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Low-investment, Realistic-Return Business Cases for Learning Analytics Dashboards: Leveraging Usage Data and Microinteractions. accepted for ECTEL 2018
  58. 58. 58 2 3 1 7 2 1 10 44 23 36 3 11 64 97 81 74 36 29 150 115 126 110 119 91 61 30 56 59 128 157 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 7. The dashboard makes me more aware on my current study situation. 8. The dashboard makes me forecast the different possibilities in my future study trajectory. 9. The dashboard helps me to reflect on my past and current study behaviour or study trajectory. 10. The dashboards stimulates me to adapt my approach in my studies for the future (study behaviour or study… 11. If I will have a new conversation after one of the next examination periods, I hope that the visualisation will be… 12. I would like to consult the information on my own. Student questionnaire January 2018 (N=291) Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
  59. 59. How to determine thresholds for different groups? LISSA dashboard 59  upper and lower group: clear message  middle group as small as possible  Do not overfit! (nuance)
  60. 60. Dashboard positioning test 60https://feedback.ijkingstoets.be/ijkingstoets-10-ir/index.html (10ir0demo)

×