edfuture.net MOOC on Current/Future State of HigherEdLearning Analyticswhat are we optimizing for?                        ...
edX: “this is big data, giving us the chanceto ask big questions about learning”                                     Will ...
the plan…joined-up multi-layer analytics    an analytics ecosystem are analytics (r)evolutionary?                         ...
the convergence of  analytics layers                     4
Macro/Meso/Micro Learning Analytics                    Macro:      region/state/national/international                    ...
Macro/Meso/Micro Learning Analytics                    Macro:      region/state/national/international                    ...
For examples of each level of analytic…Buckingham Shum, S. 2012. Our Learning Analytics are Our Pedagogy. Keynote Address,...
The VLE—BI—ITS convergence                             8
As data migrates up it enriches higherlayers, normally accustomed to sparse data                        Macro:          re...
…which in turn could enrich lower layers— local patterns can be cross-validated                        Macro:          reg...
anatomy of ananalytics ecosystem                      11
A learning analytics ecosystemlearners                                 educators                                          ...
A learning analytics ecosystemlearners                                 educators                                          ...
A learning analytics ecosystemlearners      ?!*?!*                       ?!*?!*    educators                              ...
A learning analytics ecosystem data capture design team                                 dashboard                         ...
Where did the data come from?             learners                                16
Where did the data come from?                 learners                 theories              pedagogies             assess...
Where did the data come from?                 learners                                                  technologists     ...
The map is not the territoryAnalytics are not the end, but a meansThe goal is to optimize the whole system                ...
Optimize the system     for what?                      20
Same outcomes,but higher scores?  Learning Analytics as Evolutionary Technology        • more engaging       • better asse...
New outcomes wecouldn’t assess before?    Learning Analytics as   Revolutionary Technology      • learner behaviours quant...
Learning analytics for this?“We are preparing students for jobs that do not exist yet, that will use technologies that hav...
Learning analytics for this?“While employers continue to demand high academic standards, they also now want more. They wan...
Learning analytics for this?Think about the analytics products and initiativesreviewed above – where would you locate them...
Learning analytics for this?                            The Knowledge-Agency Window    co-generation                      ...
analytics grounded in the   principles of good       assessment       for learning?      (not summative assessment for    ...
Assessment for Learning              Few learning analytics arehttp://assessment-reform-group.org     currently able to ta...
Assessment for Learninghttp://assessment-reform-group.org                                     29
Assessment for Learninghttp://assessment-reform-group.org        To what extent       could automated         feedback be ...
Assessment for Learninghttp://assessment-reform-group.org                                      Can analytics              ...
Assessment for Learninghttp://assessment-reform-group.org                      How do we provide                       ana...
analytics forlearning conversations                         33
Socio-cultural discourse analysis(Mercer et al, OU)•  Disputational talk, characterised by disagreement and   individualis...
Socio-cultural discourse analysis(Mercer et al, OU)•  Exploratory talk, in which partners engage critically but   construc...
Analytics for identifying Exploratory talk        Elluminate sessions can        be very long – lasting for        hours o...
Defining indicators of Exploratory Talk  Category               Indicator  Challenge              But if, have to respond,...
Extract classified as Exploratory Talk  Time     Contribution 2:42 PM I hate talking. :-P My question was whether "gadgets...
Discourse analytics on webinar textchat                                         Given a 2.5 hour webinar, where in the liv...
Discourse analytics on webinar textchat     Given a 2.5 hour     webinar, where in the     live textchat were the     most...
KMi’s Cohere: a web deliberation platform enabling semantic social network and discourse network analytics   Rebecca is pl...
analytics forscholarly writing                    42
Discourse analysis (Xerox Incremental Parser)Detection of salient sentences in scholarly reports,based on the rhetorical s...
Human and machine analysis of a text for keycontributions             Document 1           19 sentences annotated         ...
analytics forintepersonal networking                          45
Semantic Social Network AnalyticsDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discour...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Envir...
Closing thoughts                   54
“The basic question is not                              what can we measure?                       The basic question is  ...
Our analytics promote   values, pedagogy and   assessment regimes.  Are we clear which master our analytics serve? Are weh...
Will learning analytics merely  turbocharge the current   educational paradigm?— which is so often declared   not fit for ...
…or will learning analytics  reflect what we now know about designing authentic,engaged learning, developing   the new qua...
Upcoming SlideShare
Loading in …5
×

Learning Analytics: what are we optimizing for?

2,400 views

Published on

edfuture.net MOOC on Current/Future State of HigherEd

http://edfuture.mooc.ca/archive/12/10_29_newsletter.htm

Published in: Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,400
On SlideShare
0
From Embeds
0
Number of Embeds
287
Actions
Shares
0
Downloads
43
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Learning Analytics: what are we optimizing for?

  1. 1. edfuture.net MOOC on Current/Future State of HigherEdLearning Analyticswhat are we optimizing for? Knowledge Media Institute Simon Buckingham Shum Knowledge Media Institute The Open University UK http://twitter.com/sbskmi simon.buckinghamshum.net @ 1
  2. 2. edX: “this is big data, giving us the chanceto ask big questions about learning” Will the tomorrow’s educational researcher be as helpless without an analytics infrastructure, as a geneticist without genome databases, or a physicist without CERN? 2
  3. 3. the plan…joined-up multi-layer analytics an analytics ecosystem are analytics (r)evolutionary? 3
  4. 4. the convergence of analytics layers 4
  5. 5. Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort)
  6. 6. Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Will institutions be dazzled by the dashboards, or know what questions to ask at each level?
  7. 7. For examples of each level of analytic…Buckingham Shum, S. 2012. Our Learning Analytics are Our Pedagogy. Keynote Address, Expanding Horizons 2012 Conference, 7Macquarie University, Sydney. http://www.slideshare.net/sbs/our-learning-analytics-are-our-pedagogy
  8. 8. The VLE—BI—ITS convergence 8
  9. 9. As data migrates up it enriches higherlayers, normally accustomed to sparse data Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user tracesenriches meso + macro analytics with finer-grained process data
  10. 10. …which in turn could enrich lower layers— local patterns can be cross-validated Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user traces Breadth + depth from macroenriches meso + macro analytics + meso levels could add with finer-grained process data power to micro-analytics
  11. 11. anatomy of ananalytics ecosystem 11
  12. 12. A learning analytics ecosystemlearners educators 12
  13. 13. A learning analytics ecosystemlearners educators 13
  14. 14. A learning analytics ecosystemlearners ?!*?!* ?!*?!* educators 14
  15. 15. A learning analytics ecosystem data capture design team dashboard design teamlearners ?!*?!* data curators/ translators ?!*?!* educators 15
  16. 16. Where did the data come from? learners 16
  17. 17. Where did the data come from? learners theories pedagogies assessments tools researchers / educators / instructional designers 17
  18. 18. Where did the data come from? learners technologists theories pedagogies assessments tools researchers / educators / instructional designers 18
  19. 19. The map is not the territoryAnalytics are not the end, but a meansThe goal is to optimize the whole system outcome feedback learners design Intent theories Data pedagogies assessments tools intent researchers / educators / instructional designers 19
  20. 20. Optimize the system for what? 20
  21. 21. Same outcomes,but higher scores? Learning Analytics as Evolutionary Technology • more engaging • better assessed • better outcomes • deliverable at scale 21
  22. 22. New outcomes wecouldn’t assess before? Learning Analytics as Revolutionary Technology • learner behaviours quantifiable • interpersonal networks quantifiable • discourse quantifiable • moods and dispositions quantifiable 22
  23. 23. Learning analytics for this?“We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.” “Shift Happens” http://shifthappens.wikispaces.com 23
  24. 24. Learning analytics for this?“While employers continue to demand high academic standards, they also now want more. They want people who can adapt, see connections, innovate, communicate and work with others. This is true in many areas of work. The new knowledge-based economies in particular will increasingly depend on these abilities. Many businesses are paying for courses to promote creative abilities, to teach the skills and attitudes that are now essential for economic success…” All our Futures: Creativity, culture & education, May 1999 24
  25. 25. Learning analytics for this?Think about the analytics products and initiativesreviewed above – where would you locate them on these dimensions?Creativity, Culture andEducation (2009)Changing Young Lives2012. Newcastle: CCE.http://www.creativitycultureeducation.org/changing-young-lives-2012 25
  26. 26. Learning analytics for this? The Knowledge-Agency Window co-generation Expert-led enquiry Student-led enquiry Knowledge and use Teaching as Authenticity learning design Agency Identity Repetition, Pre-scribed Knowledge Abstraction Acquisition Expert-led teaching Student-led revision Teacher agency Student agencyRuth Deakin Crick, Univ. Bristol, Centre for Systems Learning & Leadership“Pedagogy of Hope”: http://learningemergence.net/2012/09/21/pedagogy-of-hope
  27. 27. analytics grounded in the principles of good assessment for learning? (not summative assessment for grading pupils, teachers, institutions or nations) 27
  28. 28. Assessment for Learning Few learning analytics arehttp://assessment-reform-group.org currently able to take o board the richness of this original conception of assessment for learning 28
  29. 29. Assessment for Learninghttp://assessment-reform-group.org 29
  30. 30. Assessment for Learninghttp://assessment-reform-group.org To what extent could automated feedback be designed and evaluated with emotional impact in mind? 30
  31. 31. Assessment for Learninghttp://assessment-reform-group.org Can analytics identify proxies for such advanced qualities? 31
  32. 32. Assessment for Learninghttp://assessment-reform-group.org How do we provide analytics feedback that does not disempower and de- motivate struggling learners? 32
  33. 33. analytics forlearning conversations 33
  34. 34. Socio-cultural discourse analysis(Mercer et al, OU)•  Disputational talk, characterised by disagreement and individualised decision making.•  Cumulative talk, in which speakers build positively but uncritically on what the others have said.•  Exploratory talk, in which partners engage critically but constructively with each others ideas.Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a socialmode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 34
  35. 35. Socio-cultural discourse analysis(Mercer et al, OU)•  Exploratory talk, in which partners engage critically but constructively with each others ideas. •  Statements and suggestions are offered for joint consideration. •  These may be challenged and counter-challenged, but challenges are justified and alternative hypotheses are offered. •  Partners all actively participate and opinions are sought and considered before decisions are jointly made. •  Compared with the other two types, in Exploratory talk knowledge is made more publicly accountable and reasoning is more visible in the talk.Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a socialmode of thinking. Journal of Applied Linguistics, 1(2), 137-168. 35
  36. 36. Analytics for identifying Exploratory talk Elluminate sessions can be very long – lasting for hours or even covering days of a conference It would be useful if we could identify where quality learning conversations seem to be taking place, so we can recommend those sessions, and not have to sit through online chat about virtual biscuitsFerguson, R. and Buckingham Shum, S. Learning analytics to identify exploratory dialogue within synchronous text chat. 361st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011)
  37. 37. Defining indicators of Exploratory Talk Category Indicator Challenge But if, have to respond, my view Critique However, I’m not sure, maybe Discussion of Have you read, more links resources Evaluation Good example, good point Explanation Means that, our goals Explicit reasoning Next step, relates to, that’s why Justification I mean, we learned, we observed Reflections of Agree, here is another, makes the perspectives of others point, take your point, your view 37
  38. 38. Extract classified as Exploratory Talk Time Contribution 2:42 PM I hate talking. :-P My question was whether "gadgets" were just basically widgets and we could embed them in various web sites, like Netvibes, Google Desktop, etc. 2:42 PM Thanks, thats great! I am sure I understood everything, but looks inspiring! 2:43 PM Yes why OU tools not generic tools? 2:43 PM Issues of interoperability 2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you can add various widgets, similar to those existing web start pages. 2:43 PM What if we end up with as many apps/gadgets as we have social networks and then we need a recommender for the apps! 2:43 PM My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model? 2:43 PM there are various different flavours of widget e.g. Google gadgets, W3C widgets etc. SocialLearn has gone for Google gadgets 38
  39. 39. Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar Sheffield, UK not as sunny but if we zoom in on a peak… See you! as yesterday - still warm bye for now! Greetings from Hong Kong bye, and thank you Morning from Wiltshire, 80 sunny here! Bye all for now 60 40 20 0 9:28 9:32 10:13 11:48 12:00 12:05 12:04 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:52 11:54 12:03 -20 -40 Average Exploratory -60Extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat.Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955
  40. 40. Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Classified as “exploratory talk” (more substantive 100 for learning) 50 0 9:28 “non- 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 -50 exploratory” Averag -100Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within SynchronousText Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955
  41. 41. KMi’s Cohere: a web deliberation platform enabling semantic social network and discourse network analytics Rebecca is playing the role of broker, connecting 2 peers’ contributions in meaningful waysDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829
  42. 42. analytics forscholarly writing 42
  43. 43. Discourse analysis (Xerox Incremental Parser)Detection of salient sentences in scholarly reports,based on the rhetorical signals authors use:BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:Recent studies indicate … ... new insights provide direct evidence ... … little is known …… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive Current data is insufficient …… is universally accepted ... ... results define a novel role ...CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:… unorthodox view resolves … studies ... have provided important The goal of this study ...paradoxes … advances Here, we show ...In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicatehypotheses ... understanding... inconsistent with past findings ... valuable information ... from studiesGENERALIZING: SURPRISE:... emerging as a promising approach We have recently observed ... surprisinglyOur understanding ... has grownexponentially ... We have identified ... unusual... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project: http://olnet.org/node/512 intriguing rolesimportance ...De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
  44. 44. Human and machine analysis of a text for keycontributions Document 1 19 sentences annotated 22 sentences annotated 11 sentences same as human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences same as human annotationhttp://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotationDe Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
  45. 45. analytics forintepersonal networking 45
  46. 46. Semantic Social Network AnalyticsDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829
  47. 47. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  48. 48. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  49. 49. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  50. 50. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  51. 51. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  52. 52. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  53. 53. Visualizing and filtering social ties in SocialLearn by topic and typeVisualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, TheNetherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon BuckinghamShum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL
  54. 54. Closing thoughts 54
  55. 55. “The basic question is not what can we measure? The basic question is what does a good education look like?” (Gardner Campbell)http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-collegehttp://lak12.wikispaces.com/Recordings 55
  56. 56. Our analytics promote values, pedagogy and assessment regimes. Are we clear which master our analytics serve? Are wehappy to be judged by them? 56
  57. 57. Will learning analytics merely turbocharge the current educational paradigm?— which is so often declared not fit for purpose… 57
  58. 58. …or will learning analytics reflect what we now know about designing authentic,engaged learning, developing the new qualities that a complex society demands? 58

×