Confidence in Learning Analytics aka. The Pulse of Learning Analytics

4,058 views

Published on

Presentation of the paper by Drachsler & Greller on Confidence in Learning Analytics given at LAK12 conference, April 30th 2012, Vancouver, Canada

Data and survey available at:
http://bit.ly/la_survey

Published in: Education, Technology

Confidence in Learning Analytics aka. The Pulse of Learning Analytics

  1. 1. Confidence in Learning AnalyticsUnderstandings and Expectations from the Stakeholders Hendrik Drachsler & Wolfgang Greller LAK12,Vancouver, Canada, 30th April 2012 1
  2. 2. Goals of the Presentation LA Framework Survey Findings Conclusions 2
  3. 3. A view on Learning AnalyticsThe LearningAnalyticsFramework 3
  4. 4. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  5. 5. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  6. 6. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  7. 7. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  8. 8. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  9. 9. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  10. 10. Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. 4
  11. 11. Stakeholders vs. LA FrameworkOpinions fromthe stakeholderstoward thedimensions of theLA framework 5
  12. 12. Learning Analytics Questionnaire Extrapolate opinions from different target groups: (a) what is the current understandings and the expectations on LA (b) is there a common understanding of LA 6
  13. 13. Learning Analytics Questionnaire • 4 weeks available • 156 people after clean up • 121 people full records Data and survey available at: http://bit.ly/la_survey 7
  14. 14. Participants 5.8% 11.0% Higher Education8.4% K-12 Vocational Others 74.8% 8
  15. 15. Participants - Roles % 50 44% Teachers 36% Researchers37.5 L.Designers 26% Managers 25 16%12.5 0 s s s s er er er er Te ac h ea rch sig n na g Roles es L .D e Ma R 9
  16. 16. Participants - ReachResponses from 31 countries [UK (38), US (30), NL (22)] 10
  17. 17. Stakeholders data datasubjects clients 11
  18. 18. Stakeholders (a) who was expected to benefit the most from learning analytics Teachers Parents Institutions Learners 12
  19. 19. Stakeholders (a) who was expected to benefit the most from learning analytics Outcomes: 1. Teachers 2. Learners 3. Institutions 4. Parents Teachers Parents Institutions Learners 12
  20. 20. Stakeholders (b) how much will learning analytics influence bilateral relationships? Teachers Parents Institutions 13 Learners
  21. 21. Stakeholders (b) how much will learning analytics influence bilateral relationships? Outcomes: 1. Teacher-student 84% 2. Student-teacher 63% 3. Student-student 46% 4. Teacher-teacher 41% Teachers Parents Institutions 13 Learners
  22. 22. ObjectivesReflection(Glahn, 2009) 14
  23. 23. ObjectivesReflection Prediction(Glahn, 2009) 14
  24. 24. Objectives The importance of 3 generic objectives: (a) reflection (b) prediction (c) unveil hidden information 15
  25. 25. ObjectivesIn which way learning analytics will change educationalpractice in particular areas? n = 119 11% no changes at all 43% small changes 45% extensive changes 16
  26. 26. ObjectivesIn which way learning analytics will change educationalpractice in particular areas? Item=2: Timely information n 119 about learning 11% no changes at all 43% small changes Item 8: extensive changes 45% Better insights by institutions in their courses Item 5: Easier grading Item 6: Objective assessment 16
  27. 27. Educational DataDrachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets forRecommender Systems in Technology Enhanced Learning. 1st Workshop RecommnderSystems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28,2010, Barcelona, Spain. 17
  28. 28. Educational Data Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets forVerbert, K., Manouselis, in Technology Enhanced Learning.E. (accepted).Recommnder Recommender Systems N., Drachsler, H., and Duval, 1st Workshop Dataset-driven Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28,Research to Support Learning and Knowledge Analytics. Journal of Educational Technology 2010, Barcelona, Spain.& Society. 17
  29. 29. Educational Data Researcher: 1. Added context information (n=43, means 3.42) Teacher: 1. Added context information (n=52, means 3.42) Manager: 1. Sharing within the institution (n=16, means 3.63) 2. Anonymisation (n=19, means 3.53) 18
  30. 30. TechnologiesManouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). RecommenderSystems for Learning. Berlin:Springer 19
  31. 31. TechnologiesPredictionManouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). RecommenderSystems for Learning. Berlin:Springer 19
  32. 32. Technologies ReflectionPeter Kraker, ClaudiaWagner, FleurJeanquartier, StefanieN. Lindstaedt (2011):On the Way to aScience Intelligence:Visualizing TEL Tweetsfor Trend DetectionSixth EuropeanConference onTechnology EnhancedLearning (EC-TEL 2011)Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). RecommenderSystems for Learning. Berlin:Springer 19
  33. 33. TechnologiesTrust in accurate and appropriate LA results ...1.View on learning progress2. Predict learning resource 3. Assessment 4.View on engagement 5. Compare learners 6. Prediction of peers 7. Prediction of learner performance 20
  34. 34. Constraints1.Legal protection2.Privacy3.Ethics4.Ownership 21
  35. 35. ConstraintsEffect size of LA on ... not at all a little very much privacy ethics data data trans- ownership openness parency
  36. 36. Competences 23
  37. 37. Competences 1.E-literacy 2.Interpretation skills 3.Self-directedness 4.Ethical understanding 23
  38. 38. Competences 24
  39. 39. Competences Item 3: Critical reflection Item 7: Self-directedness Item 1: Numerical skills Item 5: Ethical thinking 24
  40. 40. Competences Item 3: Critical reflection Item 7: Self-directednessthat 70.2% (n=85) believed learners were NOT competent enough to Item 1: Numerical skills independently learn from learning analytics Item 5: Ethical thinking 24
  41. 41. Limitationspicture by Stephan Downes http://www.flickr.com/photos/stephen_downes/3808714505 25
  42. 42. Limitations1. Dominance of responsesfrom the HE sector2. Absence of students3. Low awareness of LA; Only surveyed innovators and early adopters4. Mainly opinions from western cultures 26
  43. 43. Survey summaryMain findingsof the surveyaccording tothe LAframework 27
  44. 44. + Stakeholders1.Main beneficiaries of LA are learnersand teachers followed by organisations2.Biggest benefits would be gained in theteacher-to-student relationship3.Learners require teacher support tolearn from LA 28
  45. 45. + Stakeholders1.Main beneficiaries of LA are learnersand teachers followed by organisations2.Biggest benefits would be gained in theteacher-to-student relationship LA asto3.Learners require teacher support tio n?learn from LA i nnova 28
  46. 46. + Objectives1. Reflection support is main objective fromthe stakeholders view ...2. ...by revealing hidden information aboutlearners 29
  47. 47. + Objectives1. Reflection support is main objective fromthe stakeholders view ...2. ...by revealing hidden information aboutlearners Reflection support only for teacher student relationship? 29
  48. 48. + Educational Data1. Context information from learners and the learningprocess2. Anonymisation is the second most important dataattribute3. Willingness to share if data is anonymised 30
  49. 49. + Educational Data1. Context information from learners and the learningprocess2. Anonymisation is the second most important dataattribute3. Willingness to share if data is anonymised Can we achieve a collection of reference datasets? 30
  50. 50. + Technologies1. Trust in LA algorithms is not well developed2. High confidence on gaining a comprehensiveview of the learning progress 31
  51. 51. + Technologies1. Trust in LA algorithms is not well developed2. High confidence on gaining a comprehensiveview of the learning progress How accurate can we measure a learning progress? 31
  52. 52. + Constraints1. Data ownership is the most important topic2. LA lead to breaches of privacy but privacy and ethical aspectsare of lesser importance3. Many organisations have ethical boards and guidelines in place 32
  53. 53. + Constraints1. Data ownership is the most important topic2. LA lead to breaches of privacy but privacy and ethical aspectsare of lesser importance3. Many organisations have ethical boards and guidelines in place Do we need new policies for data ownership and privacy? 32
  54. 54. + Competences1. Skepticism that LA will lead to more independence oflearners to control their learning process2. Training need to guide students to more self-directednessand critical reflection 33
  55. 55. + Competences1. Skepticism that LA will lead to more independence oflearners to control their learning process2. Training need to guide students to more self-directednessand critical reflection Do we need mandatory courses on statistics for the edu. sector? 33
  56. 56. Future R&D Future R&Dpicture by Tom Raftery http://www.flickr.com/photos/traftery/4773457853 34
  57. 57. Future R&D1. Encourage the LA communityto gain additional insights fromour dataset: http://bit.ly/la_survey2. Look for partners to surveypublic bodies like schoolfoundations and governmentalinstitutions3. Compare LA in differenteducational cultures and extendsurvey to eastern countries 35
  58. 58. Future R&DGroup Concept Mappingto identifyconsensus aboutparticular issuesin learninganalytics ! 36
  59. 59. Many Thanks::Questions? This presentation is available at: slideshare.com/Drachsler Email: hendrik.drachsler@ou.nl Email: wolfgang.greller@ou.nl Supporting projects: 37

×