The document discusses learning analytics and the current and future state of higher education. It covers topics such as learning analytics frameworks including macro, meso, and micro levels; the convergence of learning analytics layers; and building an analytics ecosystem involving learners, educators, and various teams. It questions whether institutions will understand how to apply analytics at different levels or be dazzled by dashboards. It also discusses using analytics to identify effective learning conversations and different types of discourse.

1. edfuture.net MOOC on Current/Future State of HigherEd
Learning Analytics
what are we optimizing for? Knowledge Media Institute
Simon Buckingham Shum
Knowledge Media Institute
The Open University UK
http://twitter.com/sbskmi
simon.buckinghamshum.net @
1

2. edX: “this is big data, giving us the chance
to ask big questions about learning”
Will the tomorrow’s
educational researcher be
as helpless without an
analytics infrastructure, as
a geneticist without
genome databases, or a
physicist without CERN? 2

3. the plan…
joined-up multi-layer analytics
an analytics ecosystem
are analytics (r)evolutionary?
3

4. the convergence of
analytics layers
4

5. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)

6. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Will institutions be dazzled by the
dashboards, or know what
questions to ask at each level?

7. For examples of each level of analytic…
Buckingham Shum, S. 2012. Our Learning Analytics are Our Pedagogy. Keynote Address, Expanding Horizons 2012 Conference, 7
Macquarie University, Sydney. http://www.slideshare.net/sbs/our-learning-analytics-are-our-pedagogy

8. The VLE—BI—ITS convergence
8

9. As data migrates up it enriches higher
layers, normally accustomed to sparse data
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Aggregation of user traces
enriches meso + macro analytics
with finer-grained process data

10. …which in turn could enrich lower layers
— local patterns can be cross-validated
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Aggregation of user traces Breadth + depth from macro
enriches meso + macro analytics + meso levels could add
with finer-grained process data power to micro-analytics

11. anatomy of an
analytics ecosystem
11

12. A learning analytics ecosystem
learners
educators
12

13. A learning analytics ecosystem
learners
educators
13

14. A learning analytics ecosystem
learners
?!*?!*
?!*?!* educators
14

15. A learning analytics ecosystem
data capture
design team
dashboard
design team
learners
?!*?!*
data curators/
translators
?!*?!* educators
15

16. Where did the data come from?
learners
16

17. Where did the data come from?
learners
theories
pedagogies
assessments
tools
researchers / educators / instructional designers 17

18. Where did the data come from?
learners
technologists
theories
pedagogies
assessments
tools
researchers / educators / instructional designers 18

19. The map is not the territory
Analytics are not the end, but a means
The goal is to optimize the whole system
outcome
feedback
learners
design
Intent theories
Data
pedagogies
assessments
tools
intent
researchers / educators / instructional designers 19

20. Optimize the system
for what?
20

21. Same outcomes,
but higher scores?
Learning Analytics as
Evolutionary Technology
• more engaging
• better assessed
• better outcomes
• deliverable at scale
21

22. New outcomes we
couldn’t assess before?
Learning Analytics as
Revolutionary Technology
• learner behaviours quantifiable
• interpersonal networks quantifiable
• discourse quantifiable
• moods and dispositions quantifiable
22

23. Learning analytics for this?
“We are preparing students for jobs
that do not exist yet, that will use
technologies that have not been
invented yet, in order to solve
problems that are not even
problems yet.”
“Shift Happens”
http://shifthappens.wikispaces.com
23

24. Learning analytics for this?
“While employers continue to demand high academic
standards, they also now want more. They want
people who can adapt, see connections,
innovate, communicate and work with
others. This is true in many areas of work. The new
knowledge-based economies in particular will
increasingly depend on these abilities. Many
businesses are paying for courses to promote creative
abilities, to teach the skills
and attitudes that
are now essential for economic
success…”
All our Futures: Creativity, culture & education, May 1999 24

25. Learning analytics for this?
Think about the analytics
products and initiatives
reviewed above – where
would you locate them
on these dimensions?
Creativity, Culture and
Education (2009)
Changing Young Lives
2012. Newcastle: CCE.
http://www.creativitycultureeducation.org/
changing-young-lives-2012 25

26. Learning analytics for this?
The Knowledge-Agency Window
co-generation
Expert-led enquiry Student-led enquiry
Knowledge
and use
Teaching as
Authenticity
learning design
Agency
Identity
Repetition,
Pre-scribed
Knowledge
Abstraction
Acquisition
Expert-led teaching Student-led revision
Teacher agency Student agency
Ruth Deakin Crick, Univ. Bristol, Centre for Systems Learning & Leadership
“Pedagogy of Hope”: http://learningemergence.net/2012/09/21/pedagogy-of-hope

27. analytics grounded in the
principles of good
assessment
for learning?
(not summative assessment for
grading pupils, teachers,
institutions or nations)
27

28. Assessment for Learning Few learning analytics are
http://assessment-reform-group.org currently able to take o
board the richness of this
original conception of
assessment for learning
28

29. Assessment for Learning
http://assessment-reform-group.org
29

30. Assessment for Learning
http://assessment-reform-group.org
To what extent
could automated
feedback be
designed and
evaluated with
emotional impact
in mind?
30

31. Assessment for Learning
http://assessment-reform-group.org
Can analytics
identify proxies
for such
advanced
qualities? 31

32. Assessment for Learning
http://assessment-reform-group.org
How do we provide
analytics feedback
that does not
disempower and de-
motivate struggling
learners?
32

33. analytics for
learning conversations
33

34. Socio-cultural discourse analysis
(Mercer et al, OU)
• Disputational talk, characterised by disagreement and
individualised decision making.
• Cumulative talk, in which speakers build positively but
uncritically on what the others have said.
• Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
34

35. Socio-cultural discourse analysis
(Mercer et al, OU)
• Exploratory talk, in which partners engage critically but
constructively with each other's ideas.
• Statements and suggestions are offered for joint consideration.
• These may be challenged and counter-challenged, but challenges are
justified and alternative hypotheses are offered.
• Partners all actively participate and opinions are sought and considered
before decisions are jointly made.
• Compared with the other two types, in Exploratory talk knowledge is made
more publicly accountable and reasoning is more visible in the talk.
Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social
mode of thinking. Journal of Applied Linguistics, 1(2), 137-168.
35

36. Analytics for identifying Exploratory talk
Elluminate sessions can
be very long – lasting for
hours or even covering
days of a conference
It would be useful if we could
identify where quality learning
conversations seem to be taking
place, so we can recommend
those sessions, and not have to
sit through online chat about
virtual biscuits
Ferguson, R. and Buckingham Shum, S. Learning analytics to identify exploratory dialogue within synchronous text chat. 36
1st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011)

37. Defining indicators of Exploratory Talk
Category Indicator
Challenge But if, have to respond, my view
Critique However, I’m not sure, maybe
Discussion of Have you read, more links
resources
Evaluation Good example, good point
Explanation Means that, our goals
Explicit reasoning Next step, relates to, that’s why
Justification I mean, we learned, we observed
Reflections of Agree, here is another, makes the
perspectives of others point, take your point, your view
37

38. Extract classified as Exploratory Talk
Time Contribution
2:42 PM I hate talking. :-P My question was whether "gadgets" were just
basically widgets and we could embed them in various web sites,
like Netvibes, Google Desktop, etc.
2:42 PM Thanks, that's great! I am sure I understood everything, but looks
inspiring!
2:43 PM Yes why OU tools not generic tools?
2:43 PM Issues of interoperability
2:43 PM The "new" SocialLearn site looks a lot like a corkboard where you
can add various widgets, similar to those existing web start pages.
2:43 PM What if we end up with as many apps/gadgets as we have social
networks and then we need a recommender for the apps!
2:43 PM My question was on the definition of the crowd in the wisdom of
crowds we acsess in the service model?
2:43 PM there are various different flavours of widget e.g. Google gadgets,
W3C widgets etc. SocialLearn has gone for Google gadgets 38

39. Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learning
conversations?
Not at the start and end of a webinar
Sheffield, UK not as sunny but if we zoom in on a peak… See you!
as yesterday - still warm
bye for now!
Greetings from Hong Kong
bye, and thank you
Morning from Wiltshire,
80
sunny here! Bye all for now
60
40
20
0
9:28
9:32
10:13
11:48
12:00
12:05
12:04
9:36
9:40
9:41
9:46
9:50
9:53
9:56
10:00
10:05
10:07
10:07
10:09
10:17
10:23
10:27
10:31
10:35
10:40
10:45
10:52
10:55
11:04
11:08
11:11
11:17
11:20
11:24
11:26
11:28
11:31
11:32
11:35
11:36
11:38
11:39
11:41
11:44
11:46
11:52
11:54
12:03
-20
-40
Average Exploratory
-60
Extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat.
Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

40. Discourse analytics on webinar
textchat
Given a 2.5 hour
webinar, where in the
live textchat were the
most effective learning
conversations?
Classified as
“exploratory
talk”
(more
substantive
100 for learning)
50
0
9:28
“non-
9:40
9:50
10:00
10:07
10:17
10:31
10:45
11:04
11:17
11:26
11:32
11:38
11:44
11:52
12:03
-50 exploratory”
Averag
-100
Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous
Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

41. KMi’s Cohere:
a web deliberation platform enabling semantic social
network and discourse network analytics
Rebecca is playing
the role of broker,
connecting 2 peers’
contributions in
meaningful ways
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

42. analytics for
scholarly writing
42

43. Discourse analysis (Xerox Incremental Parser)
Detection of salient sentences in scholarly reports,
based on the rhetorical signals authors use:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct evidence ... … little is known …
… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive
Current data is insufficient …
… is universally accepted ... ... results define a novel role ...
CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided important The goal of this study ...
paradoxes … advances Here, we show ...
In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicate
hypotheses ... understanding
... inconsistent with past findings ... valuable information ... from studies
GENERALIZING: SURPRISE:
... emerging as a promising approach We have recently observed ...
surprisingly
Our understanding ... has grown
exponentially ... We have identified ... unusual
... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project:
http://olnet.org/node/512
intriguing roles
importance ...
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

44. Human and machine analysis of a text for key
contributions
Document 1 19 sentences annotated 22 sentences annotated
11 sentences same as human annotation
Document 2 71 sentences annotated 59 sentences annotated
42 sentences same as human annotation
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

45. analytics for
intepersonal networking
45

46. Semantic Social Network Analytics
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

47. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

48. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

49. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

50. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

51. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

52. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

53. Visualizing and filtering social ties in
SocialLearn by topic and type
Visualising Social Learning in the SocialLearn Environment. Bieke Schreurs and Maarten de Laat (Open University, The
Netherlands), Chris Teplovs (Problemshift Inc. and University of Windsor), Rebecca Ferguson and Simon Buckingham
Shum (Open University UK), SoLAR Storm webinar, Open University UK. http://bit.ly/UaFhbL

54. Closing thoughts
54

55. “The basic question is not
what can we measure?
The basic question is
what does a good education look
like?”
(Gardner Campbell)
http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college
http://lak12.wikispaces.com/Recordings 55

56. Our analytics promote
values, pedagogy and
assessment regimes.
Are we clear which master
our analytics serve? Are we
happy to be judged by them?
56

57. Will learning analytics merely
turbocharge the current
educational paradigm?
— which is so often declared
not fit for purpose…
57

58. …or will learning analytics
reflect what we now know
about designing authentic,
engaged learning, developing
the new qualities that a
complex society demands?
58