Downloaded 84 times
Uploaded byDragan Gasevic
Learning analytics are about learning
This document discusses challenges and opportunities in learning analytics. It addresses what should be measured, issues around instrumentation, capturing interventions and social networks. It also discusses challenges around scaling qualitative analysis, temporal processes, longitudinal studies, privacy, ethics and data sharing. Overall, the document advocates that learning analytics should go beyond just measuring outcomes and page access counts, and should seek to understand learning processes, contexts and the effects of students' own decisions.
More Related Content
Similar to Learning analytics are about learning
Learning analytics are about learning
- 1. Learning Analytics isabout Learning Dragan Gasevic @dgasevic
- 2.
- 4. Scalability is possible Loweffect size of class-size John Hattie
- 5.
- 6.
- 7. Scientific American, March13, 2013 http://www.scientificamerican.com/article.cfm?id=massive-open-online-courses-transformhigher-education-and-science
- 8.
- 9. Feedback loops between studentsand instructors are missing! Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.
- 10.
- 11.
- 12.
- 13.
- 14. Predict-o-mania The same predictivemodels for everything and everyone
- 16.
- 17. Population Diversity 100% 90% 80% 70% ACCT 1(n = 746) BIOL 1 (n = 220) 60% BIOL 2 (n = 657) 50% COMM 1 (n = 499) COMP 1 (n = 242) 40% ECON 1 (n = 661) 30% GRAP 1 (n = 192) MARK 1 (n = 723) 20% MATH 1 (n = 194) 10% 0% Females International students Other Living in nonlanguage at urban home Part time student Previously enrolled to a course Early access Did not access Late access
- 18. LMS Functionality Diversity ACCT1 Light Box Gallery Forum Course Resource Turn-it-in Assignment Book Quiz Feedback Map Virtual Classroom Lesson Glossary Chat X X X X X X X BIOL 1 X X X BIOL 2 X X X X X X X X X X COMM 1 COMP 1 ECON 1 X X X X X X X X X X X X X X X X X X GRAP 1 X X X MARK 1 MATH 1 X X X X X X X X X X X X X
- 19. Predictive Power Diversity 100.00% 90.00% 80.00% 70.00% 60.00% Model1 50.00% Moodle 40.00% Model 1 + Moodle 30.00% 20.00% 10.00% 0.00% All courses ACCT 1 together BIOL 1 BIOL 2 COMM 1 COMP 1 Model 1 – demographic and socio-economic variables * - not statistically significant ECON 1 * GRAP 1 MARK 1 MATH 1
- 20. Retention is not theonly challenge It is important, of course! But, where is learning?
- 21. How do we enhancelearning if the focus is on outcomes only?
- 22.
- 23. Learning Analytics –What? Measurement, collection, analysis, and reporting of data about learners and their contexts
- 24. Learning Analytics –Why? Understanding and optimising learning and the environments in which learning occurs
- 25. Modern Educational Psychology Humanagency is central to learning Bandura, A. (1989). Human agency in social cognitive theory. American psychologist, 44(9), 1175-1184.
- 26. Winne and Hadwin'smodel of self-regulated learning
- 27.
- 28. Why does itmatter?! Challenge Metacognitive skills Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Learning: Beliefs, Techniques, and Illusions. Annual Review of Psychology, 64, 417-444. doi:10.1146/annurev-psych-113011-143823
- 29. Why does itmatter?! Challenge Information seeking skills Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medical students’ reliance on Google and Wikipedia for biomedical inquiries. British Journal of Educational Technology, 42 (2), 351-360. doi:10.1111/j.1467-8535.2009.01019.x
- 30. Why does itmatter?! Challenge Sensemaking paradox Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Sensemaking Paradox. Human– Computer Interaction, 26(1-2), 123-159. doi:10.1080/07370024.2011.556552
- 31. Why does itmatter?! Challenge Asking questions and critical thinking Graesser, A. C., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524–536..
- 32. Process and contextfocus for learning analytics needed to understand learning
- 33.
- 34. Learning Analytics Effects oflearning context External conditions (e.g., instructional design)
- 36. Cognitive presence the extentto which the participants in any particular configuration of a CoI are able to construct meaning via sustained communication Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking and Computer Conferencing: A Model and Tool to Assess Cognitive Presence. American Journal of Distance Education ,15(1), 7-23.
- 37. Effect size ofthe moderator role on critical thinking Cohen’s d = 0.66
- 38. Effect size ofan intervention on critical thinking in online discussions d = 0.95 (non-moderators) and d = 0.61 (moderators)
- 39. Cognitive presence TMA1 TMA2 TMA3 TMA4 Final Control group CognitivePresence in Online Discussions – Association w/ Grades Triggering event Exploration Integration Resolution Other -.226 -.001 .128 .201 -.028 .005 .141 .060 .027 .078 -.046 .009 .034 -.023 .113 -.050 -.037 .043 -.054 .106 -.010 .048 .113 .074 .154 ** p < 0.01; * p < 0.05
- 40. Cognitive Presence inOnline Discussions – Association w/ Grades Intervention group Control group Cognitive presence ** p TMA1 TMA2 TMA3 TMA4 Final Triggering event Exploration Integration Resolution Other Triggering event Exploration -.226 -.001 .128 .201 -.028 .149 .216 .005 .141 .060 .027 .078 -.077 .197 -.046 .009 .034 -.023 .113 -.070 .163 -.050 -.037 .043 -.054 .106 .000 .223 -.010 .048 .113 .074 .154 .016 .243 Integration .156 .396** .417** .338* .454** Resolution Other -.041 .219 .060 .046 .154 .050 .083 .075 .129 .088 < 0.01; * p < 0.05
- 41. Integration posts: effect onfinal grades 100 80 60 40 20 0 Q1 Q2 Q3 p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4 Q4
- 42. Learning Analytics Are studentsonly driven by assessments? Effects of external conditions
- 43. Self-reflections in videoannotations Course 1 (non-graded) Course 3 (graded) Course 2 (graded) Course 4 (non-graded)
- 44. Self-reflections in videoannotations 120.00 100.00 80.00 Course 1 (non-graded) Course 2a (graded) 60.00 Course 2b (graded) Course 3 (graded) 40.00 Course 4 (non-graded) 20.00 0.00 Annotation total Annotation postion Annotation postion Annotation postion Annotation postion Annotation general Q1 Q2 Q3 Q4
- 45. Self-reflections in videoannotations 1800 1600 1400 1200 Course 1 (non-graded) 1000 Course 2a (graded) Course 2b (graded) 800 Course 3 (graded) 600 Course 4 (non-graded) 400 200 0 Cognitive processes Perceptual processes Positive emotions Negative emotions Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.
- 46. Learning Analytics Effects of students’own decisions Beyond external conditions
- 47. Learner profiles –use of LMS Effect size .75 on critical thinking & academic success 3 4
- 48. Learner profiles –use of LMS 14 12 10 Triggering 8 Exploration Integration 6 Resolution Other 4 2 0 Cluster 1 Cluster 2 Cluster 3 Cluster 4 Effect size .75 on critical thinking and academic success
- 49.
- 50. Learning Analytics What tomeasure? We don’t need page access counts only! Wilson, T.D. (1999). Models in information behaviour research. Journal of Documentation, 55(3), 249 - 270, doi:10.1108/EUM0000000007145
- 51. Instrumentation About specific contextsand constructs
- 52. Instrumentation Capturing interventions Previous learningand (memory of) experience Social networks (e.g., communication, cross-class) Interaction types (e.g., transactional distances)
- 53. Motivation in Information Interaction Zhou,M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
- 54. Motivation in Information Interaction Achievementgoal orientation (2x2) Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
- 55. Technology and process ofself-regulated learning Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workplace Learning, PhD Thesis, Simon Fraser University, Surrey, BC, Canada.
- 56. Scaling up qualitativeanalysis
- 57. Temporal processes beyond codingand counting
- 58.
- 59. Generating reports and nicevisualization is not enough
- 60. Building data-driven culturein institutions Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition, and Productivity, 2011, McKinsey Global Institute, http://goo.gl/Lue3qs
- 62.
- 63. Data sharing andmobility
- 64.
Editor's Notes
- #17 http://www.census.gov/prod/2013pubs/acsbr11-14.pdf
- #29 Students generally have poor self-regulation skills:Weak metacomprehension – assessment of own knowledge – stop learning, when they don’t know enoughConfusion of the rate of learning - stop learning, when they don’t know enoughExternally-generated self-monitoring prompts – AzevedoWeak metacognitive awareness – inefficient study tactics used
- #30 Use of unreliable sources Poor querying skills
- #31 Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found
- #32 Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found
- #49 Word count: Triggering - 82.03 (55.00, 99.50)Exploration - 122.71(73.25, 149.50)Integration - 185.53 (115.00, 221.00)Resolution 291.24 (168.00, 338.00)