Learning Analytics & Knowledge 2011, Banff, CanadaTheory-based Learning Analytics:Notes & Examples from Learning & Sensema...
KMi: near future (3-5 years)     infrastructure for Open U, and    other knowledge-intensive orgshttp://kmi.open.ac.uk
Theory and Technology at the                                     intersection of Hypertext and                            ...
theory ~ analytics    examples                     4
theory ~ analytics                     5
?―theory-based analytics‖                    ?                           6
Premise: ANY analytic is an implicit theory of the  world, in that it is a model, selecting specific  data and claiming it...
So for ―Theory‖, let‘s include assumptions, as well asevidence-based findings, statistical models, instructional     metho...
A theory might tell you WHAT toattend to as significant/interesting        system behaviour.              ―theory-based an...
A mature theory will tell you WHY agiven pattern is significant system            behaviour.              ―theory-based an...
A mature theory validated by pedagogical  practices will tell you APPROPRIATEINTERVENTIONS to take given particular       ...
A theory can shed new light on familiar data       ―theory-based analytics‖              This might equate to reinterpreti...
A theory might also predict futurepatterns based on a causal model             ―theory-based analytics‖                   ...
Example   RAISEonline: learninganalytics in English schools                               14
RAISEonline (English schools)  Literacy, Numeracy & Science can be measured   in controlled, written examinations, designe...
Learning analytics in English schools                                        16
Learning analytics in English schools                                        17
ExampleOpen U‘s analytics                     18
OU Analytics     Years of data reveal significant patterns  between student study history, demographics                  a...
OU Analytics service: Effective Interventions Proactive measures targeted at specific points in the  student journey are ...
OU Analytics service: Predictive modelling Probability models help us to identify patterns of  success that vary between:...
Examplesensemaking in complex      problems                         22
It‘s all about Sensemaking ―Sensemaking is about such things as  placement of items into frameworks,  comprehending, redre...
Sensemaking in complexity Individual and collective cognition has known   limitations in dealing with complexity and      ...
Sensemaking in complexityComplexity ~ sensemaking                                       Focus for analytics/rec-engines?ph...
Sensemaking in complexityComplexity ~ sensemaking                                       Focus for analytics/rec-engines?ph...
Sensemaking in complexityComplexity ~ sensemaking                                       Focus for analytics/rec-engines?ph...
Sensemaking in complexityComplexity ~ sensemaking                                       Focus for analytics/rec-engines?ph...
Examplelearning-to-learn analytics                              29
 ―We are preparing students for jobs that  do not exist yet, that will use  technologies that have not been invented  yet...
Learning Power There is a set of generic dispositions and skillscharacteristic of good learners. If learners can betaught ...
Learning to Learn: 7 Dimensions of ―Learning Power‖Expert interviews + factor analysis from literature meta-review: identi...
Learning to Learn: 7 Dimensions of Learning Power                                  www.vitalhub.net/index.php?id=8
Learning to Learn: 7 Dimensions of Learning Power                                  www.vitalhub.net/index.php?id=8
Analytics tuned to Learning Power: ELLIELLI: Effective Lifelong Learning Inventory (Ruth Deakin Crick, U. Bristol)A web qu...
Analytics tuned to Learning Power: EnquiryBlogger(National Learning Futures programme: learningfutures.org)Wordpress multi...
37
EnquiryBloggers with teacher status can viewtheir cohorts plugins in their Dashboard                                      ...
Examplediscourse analytics                      39
Discourse analytics Effective learning conversations display some typical characteristics which learners can and          ...
Socio-cultural discourse analysis(Mercer et al, OU)• Disputational talk, characterised by disagreement and  individualised...
Socio-cultural discourse analysis(Mercer et al, OU)• Exploratory talk, in which partners engage critically but  constructi...
Analytics for identifying Exploratory talk     Elluminate sessions can                                                It w...
Analytics for identifying Exploratory talkFerguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Expl...
Discourse analysis with Xerox Incremental Parser (XIP)Detection of salient sentences based on rhetorical markers:BACKGROUN...
Human and machine annotation of literature    Document 1    19 sentences annotated    22 sentences annotated              ...
Analyst-defined visual connection languageDe Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytic...
— discourse-centric analyticsDe Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st ...
— discourse-centric analytics                                                                         Does the learner com...
— discourse-centric analytics                                                                            Does the learner ...
LAK 2015?...     New learning theories have emerged     which are dependent on the enabling   capabilities of online analy...
Discussion!?―theory-based analytics‖                    ?                           52
Upcoming SlideShare
Loading in …5
×

Theory-based Learning Analytics

4,433 views

Published on

Theory-based Learning Analytics: Notes & Examples from Learning & Sensemaking

Learning Analytics & Knowledge 2011, Banff, Canada

Simon Buckingham Shum

Knowledge Media Institute
Open University UK

http://simon.buckinghamshum.net
http://open.edu
@sbskmi

Published in: Education
1 Comment
4 Likes
Statistics
Notes
  • For discourse analytics see especially 41-44
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
4,433
On SlideShare
0
From Embeds
0
Number of Embeds
645
Actions
Shares
0
Downloads
108
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide
  • Finally, the ELLI approach has now developed a body of practitioner knowledge about the kinds of interventions that can be made to help stretch a learner in different dimensions. This includes knowledge of which dimensions tend to be most important or dependent. This know-how may be formalisable as patterns that can help an ELLI mentor advise, or help a learner to decide what to do next. These ELLI profiles can of course also be be embedded in learner profiles in order to match with other similar (or dissimilar) peers.
  • Theory-based Learning Analytics

    1. 1. Learning Analytics & Knowledge 2011, Banff, CanadaTheory-based Learning Analytics:Notes & Examples from Learning & SensemakingSimon Buckingham ShumKnowledge Media InstituteOpen University UKhttp://simon.buckinghamshum.nethttp://open.edu@sbskmi http://creativecommons.org/licenses/by-nc/2.0/uk 1
    2. 2. KMi: near future (3-5 years) infrastructure for Open U, and other knowledge-intensive orgshttp://kmi.open.ac.uk
    3. 3. Theory and Technology at the intersection of Hypertext and Discourse http://projects.kmi.open.ac.uk/hyperdiscourseProjects span many disciplines, ranging from basic research to applications: 3
    4. 4. theory ~ analytics examples 4
    5. 5. theory ~ analytics 5
    6. 6. ?―theory-based analytics‖ ? 6
    7. 7. Premise: ANY analytic is an implicit theory of the world, in that it is a model, selecting specific data and claiming it as an adequate proxy for something more complex ―theory-based analytics‖ 7
    8. 8. So for ―Theory‖, let‘s include assumptions, as well asevidence-based findings, statistical models, instructional methods, as well as more academic ―theories‖ ―theory-based analytics‖ The question is whether this has INTEGRITY as a meaningful indicator, and WHO/WHAT ACTS on this data 8
    9. 9. A theory might tell you WHAT toattend to as significant/interesting system behaviour. ―theory-based analytics‖ The analytics task is to meaningfully MINE from data, or ELICIT from learners, potential indicators in a computationally tractable form 9
    10. 10. A mature theory will tell you WHY agiven pattern is significant system behaviour. ―theory-based analytics‖ This might help in guiding how to MEANINGFULLY, ETHICALLY PRESENT ANALYTICS to different stakeholders, aware of how they might react to them 10
    11. 11. A mature theory validated by pedagogical practices will tell you APPROPRIATEINTERVENTIONS to take given particular learner patterns ―theory-based analytics‖ If formalizable, analytics might then be coupled with recommendation engines or adaptive system behaviours 11
    12. 12. A theory can shed new light on familiar data ―theory-based analytics‖ This might equate to reinterpreting generic web analytics through a learning lens 12
    13. 13. A theory might also predict futurepatterns based on a causal model ―theory-based analytics‖ This might be formalizable as a predictive statistical model, or an algorithm in a rec-engine 13
    14. 14. Example RAISEonline: learninganalytics in English schools 14
    15. 15. RAISEonline (English schools) Literacy, Numeracy & Science can be measured in controlled, written examinations, designed by experts for the whole country ―theory-based analytics‖ Analytics can be generated from test scores in order to meaningfully compare schools in national league tables, accounting for different contexts 15
    16. 16. Learning analytics in English schools 16
    17. 17. Learning analytics in English schools 17
    18. 18. ExampleOpen U‘s analytics 18
    19. 19. OU Analytics Years of data reveal significant patterns between student study history, demographics and outcomes ―theory-based analytics‖ We can formalize these, eg. as organisational good practices around student support, or as statistical models to reflect on actual vs predicted course outcomes 19
    20. 20. OU Analytics service: Effective Interventions Proactive measures targeted at specific points in the student journey are associated with improved retention and progression  Telephone contact with students considered to be potentially „at risk‟ before the start of their first course is associated with around a 5% improved likelihood of course completion.  Additional tutor contact mid-way through a course is associated with between 15% to 30% improved likelihood of course completion.  Additional tutor contact around course results is associated with between 10% to 25% improved likelihood of registering for a further course.  Contact with students intending to withdraw before course start is associated with retaining 4% of students on their current course.OU Student Support Review 20
    21. 21. OU Analytics service: Predictive modelling Probability models help us to identify patterns of success that vary between:  student groups  areas of curriculum  study methods Previous OU study data – quantity and results – are the best predictors of future success The results provide a more robust comparison of module pass rates and support the institution in identifying aspects of good performance that can be shared and aspects where improvement could be realisedOU Student Statistics & Surveys Team, Institute of Educational Technology 21
    22. 22. Examplesensemaking in complex problems 22
    23. 23. It‘s all about Sensemaking ―Sensemaking is about such things as placement of items into frameworks, comprehending, redressing surprise, constructing meaning, interacting in pursuit of mutual understanding, and patterning.‖ Karl Weick, 1995, p.6 Sensemaking in Organizations
    24. 24. Sensemaking in complexity Individual and collective cognition has known limitations in dealing with complexity and information overload ―theory-based analytics‖ Collective intelligence tools and analytics should be designed specifically to minimise breakdowns in sensemaking 24
    25. 25. Sensemaking in complexityComplexity ~ sensemaking Focus for analytics/rec-engines?phenomenaRisk of entrained thinking from experts who fail • Pay particular attention to exception casesto recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issuesComplex systems only seem to make sense • Use narrative theory to detect and analyseretrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are importantMuch of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance ofthrough discourse, not formal codifications quality learning relationshipsMany small signals can build over time into a • Highlight important events and connectionssignificant force/change  aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connectionsBuckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 25Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
    26. 26. Sensemaking in complexityComplexity ~ sensemaking Focus for analytics/rec-engines?phenomenaRisk of entrained thinking from experts who fail • Pay particular attention to exception casesto recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issuesComplex systems only seem to make sense • Use narrative theory to detect and analyseretrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are importantMuch of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance ofthrough discourse, not formal codifications quality learning relationshipsMany small signals can build over time into a • Highlight important events and connectionssignificant force/change  aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connectionsBuckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 26Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
    27. 27. Sensemaking in complexityComplexity ~ sensemaking Focus for analytics/rec-engines?phenomenaRisk of entrained thinking from experts who fail • Pay particular attention to exception casesto recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issuesComplex systems only seem to make sense • Use narrative theory to detect and analyseretrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are importantMuch of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance ofthrough discourse, not formal codifications, and quality learning relationshipstrust is key to flexible sensemaking when theenvironment changesMany small signals can build over time into a • Highlight important events and connectionssignificant force/change  aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connectionsBuckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 27Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
    28. 28. Sensemaking in complexityComplexity ~ sensemaking Focus for analytics/rec-engines?phenomenaRisk of entrained thinking from experts who fail • Pay particular attention to exception casesto recognise a novel phenomenon • Scaffold critical debate between contrasting perspectives on common issuesComplex systems only seem to make sense • Use narrative theory to detect and analyseretrospectively knowledge-sharing/interpretive stories • Coherent pathways through the data ocean are importantMuch of the relevant knowledge is tacit, shared • Scaffold the formation and maintenance ofthrough discourse, not formal codifications quality learning relationshipsMany small signals can build over time into a • Highlight important events and connectionssignificant force/change  aggregation and emergent patterns • Recommend based on different kinds of significant conceptual/social connectionsBuckingham Shum, S. and De Liddo, A. (2010). Collective intelligence for OER sustainability. In: OpenEd2010: 28Seventh Annual Open Education Conference, 2-4 Nov 2010, Barcelona, Spain. Eprint: http://oro.open.ac.uk/23352
    29. 29. Examplelearning-to-learn analytics 29
    30. 30.  ―We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.‖ ―Shift Happens‖ http://shifthappens.wikispaces.com 30
    31. 31. Learning Power There is a set of generic dispositions and skillscharacteristic of good learners. If learners can betaught a language for these, they can get better at ‗learning-to-learn‘ across many contexts ―theory-based analytics‖ Analytics can be generated from a self-report questionnaire, validated through good mentoring — and possibly from online behaviour, demonstrating improvements in L2L 31
    32. 32. Learning to Learn: 7 Dimensions of ―Learning Power‖Expert interviews + factor analysis from literature meta-review: identified seven dimensionsof effective “learning power”, since validated empirically with learners at many stages, agesand cultures (Deakin Crick, Broadfoot and Claxton, 2004) Being Stuck & Static Changing & Learning Data Accumulation Meaning Making Passivity Critical Curiosity Being Rule Bound Creativity Isolation & Dependence Learning Relationships Being Robotic Strategic Awareness Fragility & Dependence Resilience www.vitalhub.net/index.php?id=8
    33. 33. Learning to Learn: 7 Dimensions of Learning Power www.vitalhub.net/index.php?id=8
    34. 34. Learning to Learn: 7 Dimensions of Learning Power www.vitalhub.net/index.php?id=8
    35. 35. Analytics tuned to Learning Power: ELLIELLI: Effective Lifelong Learning Inventory (Ruth Deakin Crick, U. Bristol)A web questionnaire generates a spider diagram summarising the learner‟s self-perception: the basis for a mentored discussion and interventions Changing and learning Critical Learning Curiosity relationships Meaning Making Strategic Awareness Creativity Resilience 35Professional development in schools, colleges and business: ViTaL: http://www.vitalhub.net/vp_research-elli.htm
    36. 36. Analytics tuned to Learning Power: EnquiryBlogger(National Learning Futures programme: learningfutures.org)Wordpress multisite plugins tuning it for learning-to-learn dispositions and enquiry skillshttp://people.kmi.open.ac.uk/sbs/2011/01/digital-support-for-authentic-enquiry 36
    37. 37. 37
    38. 38. EnquiryBloggers with teacher status can viewtheir cohorts plugins in their Dashboard 38
    39. 39. Examplediscourse analytics 39
    40. 40. Discourse analytics Effective learning conversations display some typical characteristics which learners can and should be helped to master ―theory-based analytics‖ Learners‘ written, online conversations can be analysed computationally for patterns signifying weaker and stronger forms of contribution 40
    41. 41. Socio-cultural discourse analysis(Mercer et al, OU)• Disputational talk, characterised by disagreement and individualised decision making.• Cumulative talk, in which speakers build positively but uncritically on what the others have said.• Exploratory talk, in which partners engage critically but constructively with each others ideas.Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a socialmode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 41
    42. 42. Socio-cultural discourse analysis(Mercer et al, OU)• Exploratory talk, in which partners engage critically but constructively with each others ideas. • Statements and suggestions are offered for joint consideration. • These may be challenged and counter-challenged, but challenges are justified and alternative hypotheses are offered. • Partners all actively participate and opinions are sought and considered before decisions are jointly made. • Compared with the other two types, in Exploratory talk knowledge is made more publicly accountable and reasoning is more visible in the talk.Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a socialmode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 42
    43. 43. Analytics for identifying Exploratory talk Elluminate sessions can It would be useful if we could be very long – lasting for identify where learning seems to hours or even covering be taking place, so we can days of a conference recommend those sessions, and not have to sit through online chat about virtual biscuitsFerguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within 43Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    44. 44. Analytics for identifying Exploratory talkFerguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within 44Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    45. 45. Discourse analysis with Xerox Incremental Parser (XIP)Detection of salient sentences based on rhetorical markers:BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:Recent studies indicate … ... new insights provide direct … little is known … evidence ... … role … has been elusive… the previously proposed … ... we suggest a new ... approach ... Current data is insufficient …… is universally accepted ... ... results define a novel role ...CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:… unorthodox view resolves … studies ... have provided The goal of this study ...paradoxes … important advances Here, we show ...In contrast with previous Knowledge ... is crucial for ... Altogether, our results ...hypotheses ... understanding indicate... inconsistent with past findings valuable information ... from... studiesGENERALIZING: SURPRISE:... emerging as a promising We have recently observed ...approach surprisingly Ágnes Sándor & OLnet Project:Our understanding ... has grown We have identified ... unusual http://olnet.org/node/512exponentially ... The recent discovery ... suggests... growing recognition of the intriguing rolesimportance ...
    46. 46. Human and machine annotation of literature Document 1 19 sentences annotated 22 sentences annotated 11 sentences = human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences = human annotation Ágnes Sándor & OLnet Project: http://olnet.org/node/512
    47. 47. Analyst-defined visual connection languageDe Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf. 47Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    48. 48. — discourse-centric analyticsDe Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf.Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    49. 49. — discourse-centric analytics Does the learner compare his/her own ideas to that of peers, and if so, in what ways?De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf.Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    50. 50. — discourse-centric analytics Does the learner act as a broker, connecting the ideas of his/her peers, and if so, in what ways?De Liddo, A. and Buckingham Shum, S. (2011). Discourse-Centric Learning Analytics. Proc. 1st Int. Conf.Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff
    51. 51. LAK 2015?... New learning theories have emerged which are dependent on the enabling capabilities of online analytics to provide data on a huge scale… ―theory-based analytics‖ While in tandem, new categories of learning analytic and rec-engine have emerged in specific response to the need to test emerging theories with computational models and datasets 51
    52. 52. Discussion!?―theory-based analytics‖ ? 52

    ×