Learning Analytics is about
Learning
Dragan Gasevic
@dgasevic
Growing demand for
education!
Scalability is possible
Low effect size of class-size
John Hattie
Delivery
Delivery
Scientific American, March 13, 2013
http://www.scientificamerican.com/article.cfm?id=massive-open-online-courses-transform...
MOTIVATION
Feedback loops between
students and instructors
are missing!

Hattie, J., & Timperley, H. (2007). The power of feedback. R...
Learners
Registrations

Educators
Learning and
Collaborating
Learners
Registrations

Networks
Mobile
Search

Educators
Learning and
Collaborating

Networks

Videos/slides
Learners
Registrations

Networks
Mobile
Search

Educators
Learning and
Collaborating

Networks

Videos/slides
DANGER
Predict-o-mania
The same predictive models for
everything and everyone
Student diversity

http://www.census.gov/prod/2013pubs/acsbr11-14.pdf
Population Diversity
100%
90%
80%
70%

ACCT 1 (n = 746)
BIOL 1 (n = 220)

60%

BIOL 2 (n = 657)
50%

COMM 1 (n = 499)
COMP...
LMS Functionality Diversity
ACCT 1

Light Box Gallery
Forum
Course
Resource
Turn-it-in
Assignment
Book
Quiz
Feedback
Map
V...
Predictive Power Diversity
100.00%
90.00%
80.00%
70.00%
60.00%
Model 1

50.00%

Moodle
40.00%

Model 1 + Moodle

30.00%
20...
Retention is not
the only challenge
It is important, of course!

But, where is learning?
How do we
enhance learning
if the focus is on
outcomes only?
DIRECTION
Learning Analytics – What?

Measurement, collection,
analysis, and reporting of data
about learners and their contexts
Learning Analytics – Why?

Understanding and optimising
learning and the environments
in which learning occurs
Modern Educational Psychology

Human agency is
central to learning

Bandura, A. (1989). Human agency in social cognitive t...
Winne and Hadwin's model
of self-regulated learning
Knowledge society and
knowledge economy
Why does it matter?!
Challenge
Metacognitive skills

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Lear...
Why does it matter?!
Challenge
Information seeking skills

Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medi...
Why does it matter?!
Challenge
Sensemaking paradox

Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Se...
Why does it matter?!
Challenge
Asking questions and critical thinking

Graesser, A. C., & Olde, B. (2003). How does one kn...
Process and context focus
for learning analytics needed
to understand learning
OPPORTUNITIES
Learning Analytics

Effects of learning context
External conditions (e.g., instructional design)
Cognitive presence

the extent to which the participants in any
particular configuration of a CoI are able to
construct me...
Effect size of the moderator role on
critical thinking
Cohen’s d = 0.66
Effect size of an intervention on
critical thinking in online discussions
d = 0.95 (non-moderators)
and
d = 0.61 (moderato...
Cognitive presence

TMA1

TMA2

TMA3

TMA4

Final

Control group

Cognitive Presence in Online
Discussions – Association w...
Cognitive Presence in Online
Discussions – Association w/ Grades

Intervention
group

Control group

Cognitive presence

*...
Integration posts:
effect on final grades
100
80
60
40
20
0
Q1

Q2

Q3

p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4

Q4
Learning Analytics

Are students only driven by
assessments?
Effects of external conditions
Self-reflections in video annotations

Course 1
(non-graded)

Course 3
(graded)

Course 2
(graded)

Course 4
(non-graded)
Self-reflections in video annotations
120.00

100.00

80.00

Course 1 (non-graded)
Course 2a (graded)

60.00

Course 2b (g...
Self-reflections in video annotations
1800
1600
1400
1200
Course 1 (non-graded)
1000

Course 2a (graded)

Course 2b (grade...
Learning Analytics

Effects of
students’ own decisions
Beyond external conditions
Learner profiles – use of LMS
Effect size .75 on
critical thinking &
academic success

3
4
Learner profiles – use of LMS
14
12
10
Triggering

8

Exploration

Integration

6

Resolution
Other

4
2
0
Cluster 1

Clus...
CHALLENGES
Learning Analytics

What to measure?
We don’t need page access counts only!
Wilson, T.D. (1999). Models in information beh...
Instrumentation
About specific contexts and constructs
Instrumentation
Capturing interventions
Previous learning and (memory of) experience
Social networks (e.g., communication,...
Motivation in
Information Interaction

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported ver...
Motivation in
Information Interaction

Achievement goal
orientation (2x2)

Zhou, M., & Winne, P. H. (2012). Modeling acade...
Technology and
process of self-regulated learning

Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workp...
Scaling up qualitative analysis
Temporal processes
beyond coding and counting
Longitudinal studies
Generating
reports and nice visualization is
not enough
Building data-driven culture in
institutions

Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition...
Privacy and ethics
Data sharing and mobility
Thank you!
Learning analytics are about learning
Learning analytics are about learning
Learning analytics are about learning
Learning analytics are about learning
Upcoming SlideShare
Loading in...5
×

Learning analytics are about learning

6,809

Published on

The

Published in: Education, Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,809
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
74
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide
  • http://www.census.gov/prod/2013pubs/acsbr11-14.pdf
  • Students generally have poor self-regulation skills:Weak metacomprehension – assessment of own knowledge – stop learning, when they don’t know enoughConfusion of the rate of learning - stop learning, when they don’t know enoughExternally-generated self-monitoring prompts – AzevedoWeak metacognitive awareness – inefficient study tactics used
  • Use of unreliable sources Poor querying skills
  • Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found
  • Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found
  • Word count: Triggering - 82.03 (55.00, 99.50)Exploration - 122.71(73.25, 149.50)Integration - 185.53 (115.00, 221.00)Resolution 291.24 (168.00, 338.00)
  • Transcript of "Learning analytics are about learning"

    1. 1. Learning Analytics is about Learning Dragan Gasevic @dgasevic
    2. 2. Growing demand for education!
    3. 3. Scalability is possible Low effect size of class-size John Hattie
    4. 4. Delivery
    5. 5. Delivery
    6. 6. Scientific American, March 13, 2013 http://www.scientificamerican.com/article.cfm?id=massive-open-online-courses-transformhigher-education-and-science
    7. 7. MOTIVATION
    8. 8. Feedback loops between students and instructors are missing! Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.
    9. 9. Learners Registrations Educators Learning and Collaborating
    10. 10. Learners Registrations Networks Mobile Search Educators Learning and Collaborating Networks Videos/slides
    11. 11. Learners Registrations Networks Mobile Search Educators Learning and Collaborating Networks Videos/slides
    12. 12. DANGER
    13. 13. Predict-o-mania The same predictive models for everything and everyone
    14. 14. Student diversity http://www.census.gov/prod/2013pubs/acsbr11-14.pdf
    15. 15. Population Diversity 100% 90% 80% 70% ACCT 1 (n = 746) BIOL 1 (n = 220) 60% BIOL 2 (n = 657) 50% COMM 1 (n = 499) COMP 1 (n = 242) 40% ECON 1 (n = 661) 30% GRAP 1 (n = 192) MARK 1 (n = 723) 20% MATH 1 (n = 194) 10% 0% Females International students Other Living in nonlanguage at urban home Part time student Previously enrolled to a course Early access Did not access Late access
    16. 16. LMS Functionality Diversity ACCT 1 Light Box Gallery Forum Course Resource Turn-it-in Assignment Book Quiz Feedback Map Virtual Classroom Lesson Glossary Chat X X X X X X X BIOL 1 X X X BIOL 2 X X X X X X X X X X COMM 1 COMP 1 ECON 1 X X X X X X X X X X X X X X X X X X GRAP 1 X X X MARK 1 MATH 1 X X X X X X X X X X X X X
    17. 17. Predictive Power Diversity 100.00% 90.00% 80.00% 70.00% 60.00% Model 1 50.00% Moodle 40.00% Model 1 + Moodle 30.00% 20.00% 10.00% 0.00% All courses ACCT 1 together BIOL 1 BIOL 2 COMM 1 COMP 1 Model 1 – demographic and socio-economic variables * - not statistically significant ECON 1 * GRAP 1 MARK 1 MATH 1
    18. 18. Retention is not the only challenge It is important, of course! But, where is learning?
    19. 19. How do we enhance learning if the focus is on outcomes only?
    20. 20. DIRECTION
    21. 21. Learning Analytics – What? Measurement, collection, analysis, and reporting of data about learners and their contexts
    22. 22. Learning Analytics – Why? Understanding and optimising learning and the environments in which learning occurs
    23. 23. Modern Educational Psychology Human agency is central to learning Bandura, A. (1989). Human agency in social cognitive theory. American psychologist, 44(9), 1175-1184.
    24. 24. Winne and Hadwin's model of self-regulated learning
    25. 25. Knowledge society and knowledge economy
    26. 26. Why does it matter?! Challenge Metacognitive skills Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Learning: Beliefs, Techniques, and Illusions. Annual Review of Psychology, 64, 417-444. doi:10.1146/annurev-psych-113011-143823
    27. 27. Why does it matter?! Challenge Information seeking skills Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medical students’ reliance on Google and Wikipedia for biomedical inquiries. British Journal of Educational Technology, 42 (2), 351-360. doi:10.1111/j.1467-8535.2009.01019.x
    28. 28. Why does it matter?! Challenge Sensemaking paradox Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Sensemaking Paradox. Human– Computer Interaction, 26(1-2), 123-159. doi:10.1080/07370024.2011.556552
    29. 29. Why does it matter?! Challenge Asking questions and critical thinking Graesser, A. C., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524–536..
    30. 30. Process and context focus for learning analytics needed to understand learning
    31. 31. OPPORTUNITIES
    32. 32. Learning Analytics Effects of learning context External conditions (e.g., instructional design)
    33. 33. Cognitive presence the extent to which the participants in any particular configuration of a CoI are able to construct meaning via sustained communication Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking and Computer Conferencing: A Model and Tool to Assess Cognitive Presence. American Journal of Distance Education ,15(1), 7-23.
    34. 34. Effect size of the moderator role on critical thinking Cohen’s d = 0.66
    35. 35. Effect size of an intervention on critical thinking in online discussions d = 0.95 (non-moderators) and d = 0.61 (moderators)
    36. 36. Cognitive presence TMA1 TMA2 TMA3 TMA4 Final Control group Cognitive Presence in Online Discussions – Association w/ Grades Triggering event Exploration Integration Resolution Other -.226 -.001 .128 .201 -.028 .005 .141 .060 .027 .078 -.046 .009 .034 -.023 .113 -.050 -.037 .043 -.054 .106 -.010 .048 .113 .074 .154 ** p < 0.01; * p < 0.05
    37. 37. Cognitive Presence in Online Discussions – Association w/ Grades Intervention group Control group Cognitive presence ** p TMA1 TMA2 TMA3 TMA4 Final Triggering event Exploration Integration Resolution Other Triggering event Exploration -.226 -.001 .128 .201 -.028 .149 .216 .005 .141 .060 .027 .078 -.077 .197 -.046 .009 .034 -.023 .113 -.070 .163 -.050 -.037 .043 -.054 .106 .000 .223 -.010 .048 .113 .074 .154 .016 .243 Integration .156 .396** .417** .338* .454** Resolution Other -.041 .219 .060 .046 .154 .050 .083 .075 .129 .088 < 0.01; * p < 0.05
    38. 38. Integration posts: effect on final grades 100 80 60 40 20 0 Q1 Q2 Q3 p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4 Q4
    39. 39. Learning Analytics Are students only driven by assessments? Effects of external conditions
    40. 40. Self-reflections in video annotations Course 1 (non-graded) Course 3 (graded) Course 2 (graded) Course 4 (non-graded)
    41. 41. Self-reflections in video annotations 120.00 100.00 80.00 Course 1 (non-graded) Course 2a (graded) 60.00 Course 2b (graded) Course 3 (graded) 40.00 Course 4 (non-graded) 20.00 0.00 Annotation total Annotation postion Annotation postion Annotation postion Annotation postion Annotation general Q1 Q2 Q3 Q4
    42. 42. Self-reflections in video annotations 1800 1600 1400 1200 Course 1 (non-graded) 1000 Course 2a (graded) Course 2b (graded) 800 Course 3 (graded) 600 Course 4 (non-graded) 400 200 0 Cognitive processes Perceptual processes Positive emotions Negative emotions Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.
    43. 43. Learning Analytics Effects of students’ own decisions Beyond external conditions
    44. 44. Learner profiles – use of LMS Effect size .75 on critical thinking & academic success 3 4
    45. 45. Learner profiles – use of LMS 14 12 10 Triggering 8 Exploration Integration 6 Resolution Other 4 2 0 Cluster 1 Cluster 2 Cluster 3 Cluster 4 Effect size .75 on critical thinking and academic success
    46. 46. CHALLENGES
    47. 47. Learning Analytics What to measure? We don’t need page access counts only! Wilson, T.D. (1999). Models in information behaviour research. Journal of Documentation, 55(3), 249 - 270, doi:10.1108/EUM0000000007145
    48. 48. Instrumentation About specific contexts and constructs
    49. 49. Instrumentation Capturing interventions Previous learning and (memory of) experience Social networks (e.g., communication, cross-class) Interaction types (e.g., transactional distances)
    50. 50. Motivation in Information Interaction Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
    51. 51. Motivation in Information Interaction Achievement goal orientation (2x2) Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
    52. 52. Technology and process of self-regulated learning Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workplace Learning, PhD Thesis, Simon Fraser University, Surrey, BC, Canada.
    53. 53. Scaling up qualitative analysis
    54. 54. Temporal processes beyond coding and counting
    55. 55. Longitudinal studies
    56. 56. Generating reports and nice visualization is not enough
    57. 57. Building data-driven culture in institutions Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition, and Productivity, 2011, McKinsey Global Institute, http://goo.gl/Lue3qs
    58. 58. Privacy and ethics
    59. 59. Data sharing and mobility
    60. 60. Thank you!
    1. ¿Le ha llamado la atención una diapositiva en particular?

      Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

    ×